Description of problem:

Version-Release number of selected component (if applicable):

How reproducible: Very reproducible ~90% of logins to Wayland will start vino-server when it should not

Steps to Reproduce:

1. Create a new user

2. Login new user to Gnome using Wayland Server, then logout to establish profile, then logout.

2. Login the user to Gnome using X11 Server, then enable remote desktop sharing, then logout.

3. Login the user to Gnome using Wayland Server, then enable remote desktop sharing ( it may already be enabled )

System will still show vino-server process running even though it shouldn't

$ ps -ef | grep vino
$ env | grep XDG_SESSION_TYPE

EXAMPLE:

Actual results:

[shadowy@localhost ~]$ ps -ef | grep vino ; env | grep XDG_SESSION_TYPE
shadowy 13739 10671 0 14:22 ? 00:00:00 /usr/libexec/vino-server <--- Should not be running under Wayland
shadowy 14211 14149 0 14:22 pts/0 00:00:00 grep --color=auto vino

XDG_SESSION_TYPE=wayland <--- Session type is clearly Wayland, vino-server shoudl only run when this is "X11"

Expected results:

When logging into Gnome using Wayland where remote desktop screen sharing enabled, vino-server should not be running.

See Documentation here: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/using_the_desktop_environment_in_rhel_8/accessing-the-desktop-remotely_using-the-desktop-environment-in-rhel-8

EXCERPT:

In an X11 session, it uses the vino component.

In a Wayland session, it uses the gnome-remote-desktop component.

Additional info:

You will also see the following "Failed to mlock memory" errors in: [ /var/log/messages ] as well as vino-server and gnome-remote-desktop-daemon messages

LOG EXCERPT:

Feb 2 14:46:33 localhost vino-server[412977]: 02/02/2022 02:46:33 PM Listening IPv6://[::]:5901
Feb 2 14:46:33 localhost vino-server[412977]: 02/02/2022 02:46:33 PM Listening IPv4://0.0.0.0:5901
Feb 2 14:46:33 localhost vino-server[412977]: 02/02/2022 02:46:33 PM Clearing securityTypes
Feb 2 14:46:33 localhost vino-server[412977]: 02/02/2022 02:46:33 PM Clearing authTypes
Feb 2 14:46:33 localhost vino-server[412977]: 02/02/2022 02:46:33 PM Advertising security type: 'TLS' (18)
Feb 2 14:46:33 localhost vino-server[412977]: 02/02/2022 02:46:33 PM Advertising authentication type: 'VNC Authentication' (2) ,== [ vino-server running under Wayland ]
Feb 2 14:47:12 localhost gnome-remote-desktop-daemon[412978]: #033[1;33m[W][001195469.595687][remote-node.c:637 client_node_port_use_buffers()] Failed to mlock memory 0x7f9823ea10c0 16448: This is not a problem but for best performance, consider increasing RLIMIT_MEMLOCK#033[0m
Feb 2 14:47:12 localhost gnome-remote-desktop-daemon[412978]: #033[1;33m[W][001195469.595708][remote-node.c:637 client_node_port_use_buffers()] Failed to mlock memory 0x7f9823e9c100 16448: This is not a problem but for best performance, consider increasing RLIMIT_MEMLOCK#033[0m
Feb 2 14:47:12 localhost gnome-remote-desktop-daemon[412978]: #033[1;33m[W][001195469.595715][remote-node.c:637 client_node_port_use_buffers()] Failed to mlock memory 0x7f9823e97140 16448: This is not a problem but for best performance, consider increasing RLIMIT_MEMLOCK#033[0m

Additional Test/Diagnostic Steps:

You can also ssh into a system you are testing and watch the [ /var/log/messages ] while logging on/off to reproduce errors:

1. tail -f /var/log/messages | grep -e "X.Org X Server" -e "Xwayland" -e "vino"

OUTPUT EXAMPLES:

Example of vino starting when it shouldn't:

Feb 3 18:05:34 localhost journal[18155]: Using X11 display :1024 for Xwayland <== [ LOGIN SCREEN ]
Feb 3 18:05:43 localhost journal[18493]: Using X11 display :0 for Xwayland <===[ LOGIN via Wayland ]
Feb 3 18:05:53 localhost vino-server[18774]: 03/02/2022 06:05:53 PM Autoprobing TCP port in (all) network interface <== [ vino starting INSIDE Wayland ]
<====================8
Feb 3 18:05:53 localhost vino-server[18774]: 03/02/2022 06:05:53 PM Advertising authentication type: 'VNC Authentication' (2) <== [ vino finishing starting INSIDE Wayland ]

Versus Vino acting appropriately:

Feb 3 18:04:48 localhost journal[16123]: Using X11 display :1024 for Xwayland <== [ LOGIN SCREEN ]
Feb 3 18:04:54 localhost journal[16451]: Using X11 display :0 for Xwayland <===[ LOGIN via Wayland ]
Feb 3 18:04:56 localhost vino-server[16737]: X11 is not detected <== [ vino NOT detecting X11 as wayaland is running and immediately exiting ]
Feb 3 18:04:56 localhost systemd[13218]: vino-server.service: Main process exited, code=exited, status=1/FAILURE
Feb 3 18:04:56 localhost systemd[13218]: vino-server.service: Failed with result 'exit-code'.

Possible Fix: This only seems to clear the X11 settings by blanking them and re-establishing the remote desktop screen sharing in Wayland only.
If the use logs into X11 and re-activates screen sharing it will "break" it again and they will have to "blank" the X11 settings.

The Following Key in question needs to be blanked out to disable vino in wayland if a previous enabled via an X11 session:

[ /org/gnome/settings-daemon/plugins/sharing/vino-server/enabled-connections ]

1. Login as using X11 server as user, disable Screen sharing, log out

2. Login as using Wayland server as user, disable Screen sharing, then run this command:

As the user logged in, NOT AS ROOT:

$ dconf reset -f /org/gnome/settings-daemon/plugins/sharing/

Then logout.

3. Login as using Wayland server as user, ENABLE Screen sharing

Then verify that "vino-server" is not running

$ ps -ef | grep vino