Weston window move to second monitor

[Terry Barnaby]

Hi,

I am developing software for an instrument embedded system using NXP 
CPU's and the Wayland/Weston GUI system.


The hardware platform has a main LCD panel and an optional plug-in HDMI 
monitor. The GUI is currently running with the Weston kiosk-shell and a 
single application is shown full screen on the LCD. Weston is configured 
so that the HDMI monitor appears to the right of the LCD screen and if 
the desktop-shell is used applications can be dragged across.


Now what I would like is for the application to be moved to the HDMI 
monitor/screen when the monitor is plugged in (resizing as needed).


With X11 I would have simply, from the application, moved its top level 
window coordinates and without a window manager or with a simple window 
manager the window would move to the area of the second screen.


Now I understand that Weston and the standard Wayland protocol's do not 
even allow a window move request to be made.


So I assume I might have to implement a whole new Weston shell, perhaps 
based on the kiosk-shell, and either invent some form of protocol to 
allow the application to request the shell to move it or get the shell 
to see HDMI plug in events and move the application and tell it to resize.


I don't have much knowledge of Wayland/Weston internals (lots of X11), 
so can anyone give me any pointers on how I might achieve this in the 
simplest/quickest way ?


Terry

Re: Weston window move to second monitor

[Terry Barnaby]

On 03/10/2022 08:25, Alexandros Frantzis wrote:

On Sun, Oct 02, 2022 at 09:13:27AM +0100, Terry Barnaby wrote:

Hi,

I am developing software for an instrument embedded system using NXP 
CPU's

and the Wayland/Weston GUI system.

The hardware platform has a main LCD panel and an optional plug-in HDMI
monitor. The GUI is currently running with the Weston kiosk-shell and a
single application is shown full screen on the LCD. Weston is 
configured so

that the HDMI monitor appears to the right of the LCD screen and if the
desktop-shell is used applications can be dragged across.

Now what I would like is for the application to be moved to the HDMI
monitor/screen when the monitor is plugged in (resizing as needed).

With X11 I would have simply, from the application, moved its top level
window coordinates and without a window manager or with a simple window
manager the window would move to the area of the second screen.

Now I understand that Weston and the standard Wayland protocol's do 
not even

allow a window move request to be made.

So I assume I might have to implement a whole new Weston shell, perhaps
based on the kiosk-shell, and either invent some form of protocol to 
allow
the application to request the shell to move it or get the shell to 
see HDMI

plug in events and move the application and tell it to resize.

I don't have much knowledge of Wayland/Weston internals (lots of X11), so
can anyone give me any pointers on how I might achieve this in the
simplest/quickest way ?

Terry

Hi Terry,

Assuming you only care about the fullscreen case, an application can
request (but not force) a fullscreen placement for a toplevel surface
with:

xdg_toplevel_set_fullscreen(xdg_toplevel, wl_output)

In your application you could listen for advertised wl_output globals
and issue a set_fullscreen request for the preferred wl_output (e.g.,
based on the wl_output/xdg_output name). Since you are only dealing with
a specific compositor under your control, you can be fairly confident
that you will get consistent behavior here, i.e., the compositor will
honor your request.

Note that kiosk-shell provides a mechanism in the .ini file to declare
the placement of applications on specific outputs. However, the output
placement decision is static, made only when the surface is first
created/committed to, not for any subsequent output reconfigurations.
Making this output placement in kiosk-shell work more dynamically
(optionally), with the shell moving the surface to the preferred output,
could possibly be a useful enhancement for some use cases.

HTH,
Alexandros


Many thanks for the reply and info.

I will have a go with xdg_toplevel_set_fullscreen() and maybe have a 
look at creating a special shell to do this as well. Actually I have got 
this working to a degree in a copy of the kiosk-shell adding code to the 
kiosk_shell_handle_output_created() and 
kiosk_shell_surface_notify_output_destroy() to move the views. But I 
can't get the shell to clear its background when I move the application 
window, indeed it doesn't seem like the kiosk-shell code is actually 
setting the background at all as looking at the code it should set it to 
grey but it is black on my system using Weston 9.0.0. I will investigate 
further.


Terry

Weston 8: kiosk-shell background is black but should be grey

[Terry Barnaby]

From what I can see the kiosk-shell in Weston 9.0 should draw a grey 
(0.5, 0.5, 0.5) solid colour background. However on my NXP embedded 
system it shows as black.


Should the kiosk-shell in Weston 9.0 actually draw grey ?

If so is there some configuration I'm missing or is there a bug with 
that release ?


Terry

Re: Weston 8: kiosk-shell background is black but should be grey

[Terry Barnaby]

On 04/10/2022 09:56, Alexandros Frantzis wrote:

On Tue, Oct 04, 2022 at 08:55:22AM +0100, Terry Barnaby wrote:

 From what I can see the kiosk-shell in Weston 9.0 should draw a grey (0.5,
0.5, 0.5) solid colour background. However on my NXP embedded system it
shows as black.

Should the kiosk-shell in Weston 9.0 actually draw grey ?

If so is there some configuration I'm missing or is there a bug with that
release ?

Terry

Hi Terry,

kiosk-shell until some point after 9.0.0 (until commit 73aaf14e to be
precise), has a hardcoded grey background. I just verified this works
fine with 9.0.0, at least with an Intel GPU.

The aforementioned commit changes the default background to black and
allows reading a background color value from the .ini file, like
desktop-shell does.

If you are not seeing a grey background, could it be the case that the
Weston version you are using is not actually pure 9.0.0 but is based on
some commit after 9.0.0 (so really version 9.0.9x) that also includes the
background-color support?

The first lines of your Weston log should provide information about the
exact version/build you are running. What do the lines say?

In the email subject you wrote "Weston 8", is this a typo?

Thanks,
Alexandros


Hi Alexandros,

Thanks for the reply, Yes, the "Weston 8" in the subject was a typo, sorry.

The Weston 9 used is that in the NXP Yocto hardknot release.

Actually looking at the Yocto sources for that, it is actually based on 
an NXP GIT at: git://source.codeaurora.org/external/imx/weston-imx.git, 
e73c641e076aa68e8ef69dbd56c7a252117b2c83 with some additional minor 
patches. I assumed it was based directly on the upstream Weston GIT. So 
as the background colour should be grey in the upstream 9.0.0 release I 
suspect some issues in the NXP modifications and/or hardware interface.


So I will ask on the NXP forums on this.

But just for info:

Looking at the kiosk-shell code it does not have the config file 
background-color support. In general I have no issues with Weston on 
this platform, its just when trying to implement a switch of the output 
device for a full screen application, I need to redraw the background on 
the old output and I found that this wasn't working and it actually 
should have been grey anyway.


The first log lines say:

Date: 2022-10-04 BST
[10:39:31.437] weston 9.0.0
   https://wayland.freedesktop.org
   Bug reports to: 
https://gitlab.freedesktop.org/wayland/weston/issues/

   Build: lf-5.10.52-2.1.0-rc2-0-ge73c641e+
[10:39:31.451] Command line: /usr/bin/weston 
--log=/run/user/0/weston.log --modules=systemd-notify.so
[10:39:31.452] OS: Linux, 5.10.52-lts-5.10.y-ds200i-7-ge17e4a8d84dd, 
#34 SMP PREEMPT Wed Aug 31 09:58:42 BST 2022, aarch64

[10:39:31.461] Using config file '/etc/xdg/weston/weston.ini'
[10:39:31.462] Output repaint window is 16 ms maximum.
[10:39:31.469] Loading module '/usr/lib/libweston-9/drm-backend.so'
[10:39:31.487] initializing drm backend
[10:39:31.496] logind: session control granted
[10:39:32.513] dbus connection send fail and will retry
[10:39:32.518] using /dev/dri/card1
[10:39:32.518] DRM: supports atomic modesetting
[10:39:32.518] DRM: does not support GBM modifiers


/etc/xdg/westong.conf has:

[core]
#gbm-format=argb
#gbm-format=rgb565
idle-time=0
use-g2d=1
xwayland=true
repaint-window=16
#enable-overlay-view=1

# DS200i startup
shell=kiosk-shell.so
#shell=ds200i-shell.so
#shell=fullscreen-shell.so

[shell]
#size=1920x1080

[libinput]
touchscreen_calibrator=true

#[output]
#name=HDMI-A-1
#mode=1920x1080@60
#transform=rotate-90

#[output]
#name=HDMI-A-2
#mode=off
#   WIDTHxHEIGHT    Resolution size width and height in pixels
#   off Disables the output
#   preferred   Uses the preferred mode
#   current Uses the current crt controller mode
#transform=rotate-90

[screen-share]
command=@bindir@/weston --backend=rdp-backend.so 
--shell=fullscreen-shell.so --no-clients-resize


[launcher]
icon=/home/root/.nxp-demo-experience/icon/icon_demo_launcher.png
path=/usr/bin/demoexperience

[launcher]
icon=/usr/share/weston/terminal.png
path=/usr/bin/weston-terminal

Terry

Re: Weston 8: kiosk-shell background is black but should be grey

[Terry Barnaby]

On 05/10/2022 09:41, Marius Vlad wrote:

On Tue, Oct 04, 2022 at 11:07:11AM +0100, Terry Barnaby wrote:

On 04/10/2022 09:56, Alexandros Frantzis wrote:

On Tue, Oct 04, 2022 at 08:55:22AM +0100, Terry Barnaby wrote:

  From what I can see the kiosk-shell in Weston 9.0 should draw a grey (0.5,
0.5, 0.5) solid colour background. However on my NXP embedded system it
shows as black.

Should the kiosk-shell in Weston 9.0 actually draw grey ?

If so is there some configuration I'm missing or is there a bug with that
release ?

Terry

Hi Terry,

kiosk-shell until some point after 9.0.0 (until commit 73aaf14e to be
precise), has a hardcoded grey background. I just verified this works
fine with 9.0.0, at least with an Intel GPU.

The aforementioned commit changes the default background to black and
allows reading a background color value from the .ini file, like
desktop-shell does.

If you are not seeing a grey background, could it be the case that the
Weston version you are using is not actually pure 9.0.0 but is based on
some commit after 9.0.0 (so really version 9.0.9x) that also includes the
background-color support?

The first lines of your Weston log should provide information about the
exact version/build you are running. What do the lines say?

In the email subject you wrote "Weston 8", is this a typo?

Thanks,
Alexandros

Hi Alexandros,

Thanks for the reply, Yes, the "Weston 8" in the subject was a typo, sorry.

The Weston 9 used is that in the NXP Yocto hardknot release.

Actually looking at the Yocto sources for that, it is actually based on an
NXP GIT at: git://source.codeaurora.org/external/imx/weston-imx.git,
e73c641e076aa68e8ef69dbd56c7a252117b2c83 with some additional minor patches.
I assumed it was based directly on the upstream Weston GIT. So as the
background colour should be grey in the upstream 9.0.0 release I suspect
some issues in the NXP modifications and/or hardware interface.

So I will ask on the NXP forums on this.

I see you're using g2d, maybe try turning that off and see if it's still
black? Would be really nice if could update this thread once you find
out the issue, it could help others too.


I actually did that earlier this morning after looking at the NXP Weston 
code, and the background colour worked after that, so definitely an NXP 
Weston 9 issue with the g2d mods.


I have asked on the NXP forums, but I don't hold out much hope of any info.

I will probably try their latest 10 or "main" version somehow, but I am 
wary of dependency hell through the whole Linux->application stack 
especially as we are just about there with this project apart from not 
being able to move an application to a second monitor :(.




But just for info:

Looking at the kiosk-shell code it does not have the config file
background-color support. In general I have no issues with Weston on this

Support for changing the background was added, like Alexandros said,
in 9.0.x which made in into weston 10.

Fyi, code that added background intially had an issue where the two
channels were swapped, but was rectififed a bit later on
(but that's fixed in weston 10).


I have looked at that code, and it "appears" to function the same as the 
default grey code apart from getting the config parameter.


All I need to do at this stage is find some way of clearing the output 
on the first monitor when the application moves as its old screen GUI 
display is left rendered on that monitor after my code moves the view x 
pos to the second monitor (output).




platform, its just when trying to implement a switch of the output device
for a full screen application, I need to redraw the background on the old
output and I found that this wasn't working and it actually should have been
grey anyway.

The first log lines say:

Date: 2022-10-04 BST
[10:39:31.437] weston 9.0.0
    https://wayland.freedesktop.org
    Bug reports to:
https://gitlab.freedesktop.org/wayland/weston/issues/
    Build: lf-5.10.52-2.1.0-rc2-0-ge73c641e+
[10:39:31.451] Command line: /usr/bin/weston --log=/run/user/0/weston.log
--modules=systemd-notify.so
[10:39:31.452] OS: Linux, 5.10.52-lts-5.10.y-ds200i-7-ge17e4a8d84dd, #34
SMP PREEMPT Wed Aug 31 09:58:42 BST 2022, aarch64
[10:39:31.461] Using config file '/etc/xdg/weston/weston.ini'
[10:39:31.462] Output repaint window is 16 ms maximum.
[10:39:31.469] Loading module '/usr/lib/libweston-9/drm-backend.so'
[10:39:31.487] initializing drm backend
[10:39:31.496] logind: session control granted
[10:39:32.513] dbus connection send fail and will retry
[10:39:32.518] using /dev/dri/card1
[10:39:32.518] DRM: supports atomic modesetting
[10:39:32.518] DRM: does not support GBM modifiers


/etc/xdg/westong.conf has:

[core]
#gbm-format=argb
#gbm-format=rgb565
idle-time=0
use-g2d=1
xwayland=true
repaint-window=16
#enable-overlay-view=1

# DS200i startup
shell=kiosk-s

Wayland debugging with Qtwayland, gstreamer waylandsink, wayland-lib and Weston

[Terry Barnaby]

Hi,

We are developing a video processing system that runs on an NXP imx8 
processor using a Yocto embedded Linux system that has Qt6, GStreamer, 
Wayland and Weston.


We are having a problem displaying the video stream from GStreamer on a 
QWidget. In the past we had this working with Qt5 and older GStreamer, 
Wayland and Weston.


A simple test program also shows the issue on Fedora37 with QT6 and 
KDE/Plasma/Wayland.


The technique we are using is to get the Wayland surface from the 
QWidget is using (It has been configured to use a Qt::WA_NativeWindow) 
and pass this to the GStreamer's waylandsink which should then update 
this surface with video frames (via hardware). This works when the 
QWidget is a top level Window widget (QWidget(0)), but if this QWidget 
is below others in the hierarchy no video is seen and the gstreamer 
pipeline line is stalled.


It appears that waylandsink does:

Creates a surface callback:

  callback = wl_surface_frame (surface);

  wl_callback_add_listener (callback, &frame_callback_listener, self);

Then adds a buffer to a surface:

    gst_wl_buffer_attach (buffer, priv->video_surface_wrapper);
    wl_surface_set_buffer_scale (priv->video_surface_wrapper, priv->scale);
    wl_surface_damage_buffer (priv->video_surface_wrapper, 0, 0, 
G_MAXINT32, G_MAXINT32);

    wl_surface_commit (priv->video_surface_wrapper);

But never gets a callback and just sits in a loop awaiting that callback.

I assume that the surface waylandsink is using, which is created using 
the original QWidget surface (sub-surface ? with window ?) is not 
"active" for some reason.



I am trying to debug this, but this graphics stack is quite complicated 
with waylandsink, qtwayland, wayland-lib and Weston not to mention the 
NXP hardware levels. My thoughts are that it is something qtwayland is 
doing with the surface stack or thread locking issues (gstreamer uses 
separate threads). I also don't understand Wayland or Weston in detail. 
So some questions:


1. Anyone seen something like this ?

2. Anyone any idea one where to look ?

3. Given the wl_surface in the Qt app or in waylandsink is there a way I 
can print out its state and the surface hierarchy easily ?


4. Any idea on any debug methods to use ?

Cheers

Terry

Re: Wayland debugging with Qtwayland, gstreamer waylandsink, wayland-lib and Weston

[Terry Barnaby]

Hi Marius,

Many thanks for the info.

Some notes/questions below:

Terry
On 22/02/2024 17:49, Marius Vlad wrote:

Hi,
On Thu, Feb 22, 2024 at 03:21:01PM +, Terry Barnaby wrote:

Hi,

We are developing a video processing system that runs on an NXP imx8
processor using a Yocto embedded Linux system that has Qt6, GStreamer,
Wayland and Weston.

We are having a problem displaying the video stream from GStreamer on a
QWidget. In the past we had this working with Qt5 and older GStreamer,
Wayland and Weston.

A simple test program also shows the issue on Fedora37 with QT6 and
KDE/Plasma/Wayland.

I'm tempted to say if this happens on a desktop with the same Qt version and
other compositors to be an issue with Qt rather than waylandsink or
the compositor. Note that on NXP they have their own modified Weston version.


That is my current feeling and is one reason why I tried it on Fedora 
with whatever Wayland compositor KDE/Plasma is using.




The technique we are using is to get the Wayland surface from the QWidget is
using (It has been configured to use a Qt::WA_NativeWindow) and pass this to
the GStreamer's waylandsink which should then update this surface with video
frames (via hardware). This works when the QWidget is a top level Window
widget (QWidget(0)), but if this QWidget is below others in the hierarchy no
video is seen and the gstreamer pipeline line is stalled.

So the assumption is that aren't there other widgets which obscures this
one, when you move it below others?


My simple test example has two QWidgets with the one for video being 
created as a child of the first so it should be above all others. I have 
even tried drawing in it to make sure and it displays its Qt drawn 
contents fine, just not the video stream.




It appears that waylandsink does:

Creates a surface callback:

   callback = wl_surface_frame (surface);

   wl_callback_add_listener (callback, &frame_callback_listener, self);

Then adds a buffer to a surface:

     gst_wl_buffer_attach (buffer, priv->video_surface_wrapper);
     wl_surface_set_buffer_scale (priv->video_surface_wrapper, priv->scale);
     wl_surface_damage_buffer (priv->video_surface_wrapper, 0, 0, G_MAXINT32,
G_MAXINT32);
     wl_surface_commit (priv->video_surface_wrapper);

But never gets a callback and just sits in a loop awaiting that callback.

I assume that the surface waylandsink is using, which is created using the
original QWidget surface (sub-surface ? with window ?) is not "active" for
some reason.

Possibly when QWidget is below in hierarcy to be a child of of a parent,
as described in 
https://wayland.app/protocols/xdg-shell#xdg_toplevel:request:set_parent,
so I assume to have a different surface than the parent one. This would
be easy to determine with WAYLAND_DEBUG. Seems unlikely to a itself a
sub-surface of a surface.


I haven't really got the gist of whats going on, but waylandsink 
certainly creates a subsurface from the QWidget surface, in fact it 
seems to create a few things.


I assume a subsurface is used so the video can be displayed in that 
subsurface separately from the parent (de synced from it).





I am trying to debug this, but this graphics stack is quite complicated with
waylandsink, qtwayland, wayland-lib and Weston not to mention the NXP
hardware levels. My thoughts are that it is something qtwayland is doing
with the surface stack or thread locking issues (gstreamer uses separate
threads). I also don't understand Wayland or Weston in detail. So some
questions:

1. Anyone seen something like this ?

Someone else reported something similar but that by causing damage,
or moving pointer to make the video sub-surface to show up:
https://gitlab.freedesktop.org/wayland/weston/-/issues/843.


Thanks, I will have a look. Moving the mouse cursor in my case (at least 
with Weston) does not affect things.




2. Anyone any idea one where to look ?

3. Given the wl_surface in the Qt app or in waylandsink is there a way I can
print out its state and the surface hierarchy easily ?

In Weston there's something called scene-graph. You can grab it by
starting Weston with with the --debug argument, then you can print
with `weston-debug scene-graph` command. A more recent Weston version
would indent sub-surfaces by their (main) surface parent.


Thanks, that could be useful.



4. Any idea on any debug methods to use ?

WAYLAND_DEBUG=1 as env variable.


Any idea on how to get a surfaces ID from a C pointer so I can match up 
the QtWidget/waylandsink surface with the Wayland debug output ?




Cheers

Terry

Re: Wayland debugging with Qtwayland, gstreamer waylandsink, wayland-lib and Weston

[Terry Barnaby]

I have tried using "weston-debug scene-graph" and I am coming to the 
conclusion that qtwayland 6.5.0 is not really using native Wayland 
surfaces when Qt::WA_NativeWindow is used. From what I can see (and I 
could easily be wrong) the Wayland protocol shows wl_surfaces being 
created and two QWidget's QPlatformNativeInterface 
nativeResourceForWindow("surface", windowHandle()) function does return 
different wl_surface pointers but even at the QWidget level (ignoring 
gstreamer), a QPainter paint into each of these QWidgets actually uses 
Wayland to draw into just the one top level surface and "weston-debug 
scene-graph" shows only one application xdg_toplevel surface and no 
subsurfaces. I don't know how to determine the Wayland surface ID from a 
wl_surface pointer unfortunately to really check this.


If my Video QWidget(0) is a top level QWidget, then video is shown and 
"weston-debug scene-graph" shows the application xdg_toplevel and two 
wl_subsurfaces as children.


Unfortunately I think "weston-debug scene-graph" only shows surfaces 
that are actually "active" so I can't see all of the surfaces that 
Weston actually knows about (is there a method of doing this ?).


My feeling is that although Qtwayland is creating native surfaces, it 
actually only uses the one top level one and presumably doesn't 
"activate" (set a role, do something ?) with the other surfaces.


Does anyone know a good list/place where I can ask such detailed 
qtwayland questions ?


I guess I can work around this by manually creating a Wayland subsurface 
from the Qt top level surface and handing that to waylandsink and then 
manage this subsurface, like hiding, showing and resizing, when the 
QWidget is hidden/shown/resized.


Or could there be a way of "activating" the child QWidget's Wayland 
surface ?




On 22/02/2024 18:44, Terry Barnaby wrote:

Hi Marius,

Many thanks for the info.

Some notes/questions below:

Terry
On 22/02/2024 17:49, Marius Vlad wrote:

Hi,
On Thu, Feb 22, 2024 at 03:21:01PM +, Terry Barnaby wrote:

Hi,

We are developing a video processing system that runs on an NXP imx8
processor using a Yocto embedded Linux system that has Qt6, GStreamer,
Wayland and Weston.

We are having a problem displaying the video stream from GStreamer on a
QWidget. In the past we had this working with Qt5 and older GStreamer,
Wayland and Weston.

A simple test program also shows the issue on Fedora37 with QT6 and
KDE/Plasma/Wayland.
I'm tempted to say if this happens on a desktop with the same Qt 
version and

other compositors to be an issue with Qt rather than waylandsink or
the compositor. Note that on NXP they have their own modified Weston 
version.


That is my current feeling and is one reason why I tried it on Fedora 
with whatever Wayland compositor KDE/Plasma is using.



The technique we are using is to get the Wayland surface from the 
QWidget is
using (It has been configured to use a Qt::WA_NativeWindow) and pass 
this to
the GStreamer's waylandsink which should then update this surface 
with video
frames (via hardware). This works when the QWidget is a top level 
Window
widget (QWidget(0)), but if this QWidget is below others in the 
hierarchy no

video is seen and the gstreamer pipeline line is stalled.

So the assumption is that aren't there other widgets which obscures this
one, when you move it below others?


My simple test example has two QWidgets with the one for video being 
created as a child of the first so it should be above all others. I 
have even tried drawing in it to make sure and it displays its Qt 
drawn contents fine, just not the video stream.




It appears that waylandsink does:

Creates a surface callback:

   callback = wl_surface_frame (surface);

   wl_callback_add_listener (callback, &frame_callback_listener, self);

Then adds a buffer to a surface:

 gst_wl_buffer_attach (buffer, priv->video_surface_wrapper);
 wl_surface_set_buffer_scale (priv->video_surface_wrapper, 
priv->scale);
 wl_surface_damage_buffer (priv->video_surface_wrapper, 0, 0, 
G_MAXINT32,

G_MAXINT32);
 wl_surface_commit (priv->video_surface_wrapper);

But never gets a callback and just sits in a loop awaiting that 
callback.


I assume that the surface waylandsink is using, which is created 
using the
original QWidget surface (sub-surface ? with window ?) is not 
"active" for

some reason.

Possibly when QWidget is below in hierarcy to be a child of of a parent,
as described in 
https://wayland.app/protocols/xdg-shell#xdg_toplevel:request:set_parent,

so I assume to have a different surface than the parent one. This would
be easy to determine with WAYLAND_DEBUG. Seems unlikely to a itself a
sub-surface of a surface.


I haven't really got the gist of whats going on, but waylandsink 
certainly creates a subsurface from the QWidget surface, 

Re: Wayland debugging with Qtwayland, gstreamer waylandsink, wayland-lib and Weston

[Terry Barnaby]

Hi David,

Many thanks for the reply and the info on how to get the ID.

I have added a basic example with some debug output at: 
https://portal.beam.ltd.uk/public//test016-qt6-video-example.tar.gz


If there are any ideas of things I could look at/investigate I am all ears!

In a previous email I stated:
I have tried using "weston-debug scene-graph" and I am coming to the 
conclusion that qtwayland 6.5.0 is not really using native Wayland 
surfaces when Qt::WA_NativeWindow is used. From what I can see (and I 
could easily be wrong) the Wayland protocol shows wl_surfaces being 
created and two QWidget's QPlatformNativeInterface 
nativeResourceForWindow("surface", windowHandle()) function does 
return different wl_surface pointers but even at the QWidget level 
(ignoring gstreamer), a QPainter paint into each of these QWidgets 
actually uses Wayland to draw into just the one top level surface and 
"weston-debug scene-graph" shows only one application xdg_toplevel 
surface and no subsurfaces. I don't know how to determine the Wayland 
surface ID from a wl_surface pointer unfortunately to really check this.


If my Video QWidget(0) is a top level QWidget, then video is shown and 
"weston-debug scene-graph" shows the application xdg_toplevel and two 
wl_subsurfaces as children.


Unfortunately I think "weston-debug scene-graph" only shows surfaces 
that are actually "active" so I can't see all of the surfaces that 
Weston actually knows about (is there a method of doing this ?).


My feeling is that although Qtwayland is creating native surfaces, it 
actually only uses the one top level one and presumably doesn't 
"activate" (set a role, do something ?) with the other surfaces.


Does anyone know a good list/place where I can ask such detailed 
qtwayland questions ?


I guess I can work around this by manually creating a Wayland 
subsurface from the Qt top level surface and handing that to 
waylandsink and then manage this subsurface, like hiding, showing and 
resizing, when the QWidget is hidden/shown/resized.


Or could there be a way of "activating" the child QWidget's Wayland 
surface ? 



Terry

On 23/02/2024 08:35, David Edmundson wrote:

On Fri, Feb 23, 2024 at 6:15 AM Terry Barnaby  wrote:

I don't know how to determine the Wayland surface ID from a
wl_surface pointer unfortunately to really check this.

wl_proxy_get_id(static_cast(myWlSurface));



Possibly when QWidget is below in hierarcy to be a child of of a parent,
as described in

That's fine.

A QWidget with WA_NativeWindow will create a QWindow with a parent. A
QWindow with a parent will create a subsurface in wayland terms.
But it is a subsurface where Qt is managing it and you're also
committing on it, which can be a bit confusing and going through
widgets to create a subsurface isn't really needed.
There's a bunch of other options there.


---

Can you link your test app. You can send me a private email and I'll
take a look.  It doesn't seem like a core wayland problem more a
Qt/application setup issue so far. Then we can follow it up on Qt's
Jira if there is a Qt issue.

David Edmundson  - QtWayland Maintainer

Re: Wayland debugging with Qtwayland, gstreamer waylandsink, wayland-lib and Weston

[Terry Barnaby]

Hi,

I have investigated a bit further. I have built my own Weston server to 
run under X11 on Fedora37 so I can add printf's and debug more easily 
than using a cross compiled iMX8mp target system etc. I added a new 
dsmc-shell to Weston which is identical to kiosk-shell (apart from 
names) so I could play with that.


When It run my simple QWidget test program (test016-qt6-video-example) 
which creates one top level QWidget with a child QWidget to display the 
Gstreamer video in it, I see the following surfaces/subsurfaces when 
desktop_surface_committed() is called in the dsmc-shell.


Surface: 16 1044,620 mapped: 1 opaque: 0
 View: 0x29182b0
 Surface: 18 0,0 mapped: 0 opaque: 0
  Surface: 44 640,480 mapped: 1 opaque: 0
  Surface: 18 0,0 mapped: 0 opaque: 0
 Surface: 17 0,0 mapped: 0 opaque: 0
 Surface: 16 1044,620 mapped: 1 opaque: 0

Surface 16 is used by the top level QWidget, surface 18 is used by the 
Video QWidget and surface 44 is the GStreamer video display surface (I 
think!). This list is generated traversing the weston_surface's views 
and subsurface_list lists. The mapped is the "is_mapped" field of the 
surface.
Only the top level weston_surface has a weston_view in the views list it 
that is any relevance. "weston-debug scene-graph" only shows the tope 
most surface, no subsurfaces.


As mentioned before, If I use QPainter to draw into the video QWidget it 
actually draws into the top QWidgets 16 surface using Wayland protocol. 
I would have thought it would draw into its own QWidget surface 18 as it 
has Qt::WA_NativeWindow set ?


I assume that Qtwayland is not "activating" the video QWidget's surface 
or using it for some reason (send subsurface expose events ?) ?



I note in the qtwayland change logs, for the earlier QT5 for subsurface 
changes:
dist/changes-5.6.3: - [QTBUG-52118] Fixed subsurface positions sometimes 
not being committed.
dist/changes-5.11.2: - [QTBUG-69643] Fixed a bug where subsurfaces would 
not be rendered if clients added them before a WaylandQuickItem was 
created for the parent surface
dist/changes-5.12.0: - [QTBUG-49809] Added support for 
wl_subsurface.place_above and place_below in WaylandQuickItem.
dist/changes-5.15.2: - [QTBUG-86176] We now send subsurface expose 
events when a different toplevel (such as a dialog) is configured.


Could any of these be related ?

Terry
On 23/02/2024 09:29, Terry Barnaby wrote:

Hi David,

Many thanks for the reply and the info on how to get the ID.

I have added a basic example with some debug output at: 
https://portal.beam.ltd.uk/public//test016-qt6-video-example.tar.gz


If there are any ideas of things I could look at/investigate I am all 
ears!


In a previous email I stated:
I have tried using "weston-debug scene-graph" and I am coming to the 
conclusion that qtwayland 6.5.0 is not really using native Wayland 
surfaces when Qt::WA_NativeWindow is used. From what I can see (and I 
could easily be wrong) the Wayland protocol shows wl_surfaces being 
created and two QWidget's QPlatformNativeInterface 
nativeResourceForWindow("surface", windowHandle()) function does 
return different wl_surface pointers but even at the QWidget level 
(ignoring gstreamer), a QPainter paint into each of these QWidgets 
actually uses Wayland to draw into just the one top level surface and 
"weston-debug scene-graph" shows only one application xdg_toplevel 
surface and no subsurfaces. I don't know how to determine the Wayland 
surface ID from a wl_surface pointer unfortunately to really check this.


If my Video QWidget(0) is a top level QWidget, then video is shown 
and "weston-debug scene-graph" shows the application xdg_toplevel and 
two wl_subsurfaces as children.


Unfortunately I think "weston-debug scene-graph" only shows surfaces 
that are actually "active" so I can't see all of the surfaces that 
Weston actually knows about (is there a method of doing this ?).


My feeling is that although Qtwayland is creating native surfaces, it 
actually only uses the one top level one and presumably doesn't 
"activate" (set a role, do something ?) with the other surfaces.


Does anyone know a good list/place where I can ask such detailed 
qtwayland questions ?


I guess I can work around this by manually creating a Wayland 
subsurface from the Qt top level surface and handing that to 
waylandsink and then manage this subsurface, like hiding, showing and 
resizing, when the QWidget is hidden/shown/resized.


Or could there be a way of "activating" the child QWidget's Wayland 
surface ? 



Terry

On 23/02/2024 08:35, David Edmundson wrote:
On Fri, Feb 23, 2024 at 6:15 AM Terry Barnaby  
wrote:

I don't know how to determine the Wayland surface ID from a
wl_surface pointer unfortunately to really check this.

wl_proxy_get_id(static_cast(myWlSurface));


Possibly when QWidget is below in h

Re: Wayland debugging with Qtwayland, gstreamer waylandsink, wayland-lib and Weston

[Terry Barnaby]

Hi Pekka,

Thanks for the response. Notes below:

Terry

On 26/02/2024 13:28, Pekka Paalanen wrote:

On Sun, 25 Feb 2024 08:04:30 +
Terry Barnaby  wrote:


Hi,

I have investigated a bit further. I have built my own Weston server to
run under X11 on Fedora37 so I can add printf's and debug more easily
than using a cross compiled iMX8mp target system etc. I added a new
dsmc-shell to Weston which is identical to kiosk-shell (apart from
names) so I could play with that.

When It run my simple QWidget test program (test016-qt6-video-example)
which creates one top level QWidget with a child QWidget to display the
Gstreamer video in it, I see the following surfaces/subsurfaces when
desktop_surface_committed() is called in the dsmc-shell.

Surface: 16 1044,620 mapped: 1 opaque: 0
   View: 0x29182b0
   Surface: 18 0,0 mapped: 0 opaque: 0
    Surface: 44 640,480 mapped: 1 opaque: 0
    Surface: 18 0,0 mapped: 0 opaque: 0
   Surface: 17 0,0 mapped: 0 opaque: 0
   Surface: 16 1044,620 mapped: 1 opaque: 0

Btw. the sub-surface list also contains the parent weston_surface in
it, that's why surface 18 and 16 show up twice, I guess. It's used for
z-ordering.


Yes, that was my view.





Surface 16 is used by the top level QWidget, surface 18 is used by the
Video QWidget and surface 44 is the GStreamer video display surface (I
think!). This list is generated traversing the weston_surface's views
and subsurface_list lists. The mapped is the "is_mapped" field of the
surface.
Only the top level weston_surface has a weston_view in the views list it
that is any relevance. "weston-debug scene-graph" only shows the tope
most surface, no subsurfaces.

Right.

A sub-surface requires its parent surface to be mapped in order to show
up on screen. This applies recursively, so surface 18 not being mapped
prevents surface 44 from showing up.

IIRC, more or less only "fully mapped" weston_surfaces (as in, if it's
a sub-surface, the whole sub-surface ancestry path up to a top-level is
mapped) have a weston_view. weston_view defines where on screen a
weston_surface will be presented, so without a view it cannot show up.


Ok. What Wayland API requests cause a surface to actually be mapped 
(Sorry don't really know Wayland protocol) ?


The higher level surfaces are created/managed by QtWayland, but maybe I 
can override somehow.






As mentioned before, If I use QPainter to draw into the video QWidget it
actually draws into the top QWidgets 16 surface using Wayland protocol.
I would have thought it would draw into its own QWidget surface 18 as it
has Qt::WA_NativeWindow set ?

I assume that Qtwayland is not "activating" the video QWidget's surface
or using it for some reason (send subsurface expose events ?) ?


If that's true, then it is very relevant. A sub-surface becomes mapped
and visible when:

- its parent surface is mapped and visible, and

- the parent surface is committed after the sub-surface has been
   created and associated, and

- if the sub-surface is in synchronized mode, there also needs to be a
   parent surface commit after every sub-surface commit you want to
   become visible. So if you do the first sub-surface sync-mode commit
   with a buffer after the parent has already committed the
   sub-surface's creation, the parent surface needs too commit again.

This applies recursively, too, and you have a sub-sub-surface there.

Do you actually need to sub-surface in the middle? Have you tried
without it?


I am not doing anything with Wayland directly. Qt is managing the higher 
level surfaces/subsurfaces and then GStreamers waylandsink is passed one 
of these Qt managed surfaces and it creates the subsurface 44. Looking 
at waylandsink it should set this subsurface to be desynced so it can 
draw into this surface without synchronising to the parents surface 
managed by Qt.


All I am trying to do is use the technique as mentioned in various 
forums/lists to get GStreamer to display a video such that it "appears" 
where a QWidget is on the screen. Mind you all of this info is very 
rough and ready. For X11 it appears stable, but for Qt/Wayland the info, 
and it appears functionality, is not quite all there.


When you say a sub-surface in the middle I presume you mean the surface 
18 of the lower QWidget ? Well I have tried using the top most QWidget's 
surface 16 and the video is displayed, although all over the 
application. I really need to manage this surface so it can be raised, 
lowered and resizes amongst the other QWidgets somehow. I have tried 
using direct Wayland API calls to create a subsurface manually from the 
top surface but so far I have just got protocol errors while trying 
this. It may be my bad Wayland client code or how it is interfering with 
Qt's Wayland interface.


I have even tried using a separate top level surface. Unfortunately as 
the standard Wayland protocols do not allow

Re: Wayland debugging with Qtwayland, gstreamer waylandsink, wayland-lib and Weston

[Terry Barnaby]

Hi Pekka,

Some questions below:

Thanks

Terry
On 26/02/2024 15:56, Pekka Paalanen wrote:

Ok. What Wayland API requests cause a surface to actually be mapped

(Sorry don't really know Wayland protocol) ?

Hi Terry,

the basic protocol object is wl_surface. The wl_surface needs to be
given a "role". This is a specific protocol term. xdg_surface and
xdg_toplevel can give a wl_surface the role of a top-level window,
which means it can get mapped when you play by the rules set in the
xdg_toplevel specification.

Sub-surface is another role.

So the rules are always role specific, but at some point they all
require content on the wl_surface, which is given with the attach
request followed by a commit request. Role rules specify when and how
that can be done.


Yes, I have heard that. But what I don't knoe is from the client:

1. How do I find out the surfaces role ?
2. How would I set the surface to have a role such that it would be
   mapped and thus visible ? Just wondering if I can work around what I
   think is a QtWayland bug/issue/feature to make sure by second
   Widgets surface is mapped/visible so that the waylandsink subsurface
   can work. With X11 there were API calls to change the Windows state
   and I was looking for something similar with Wayland.

I need to find some way to actually display video, simply and 
efficiently on an embedded platform, in a Qt application in the year 2024 :)


I have tried lots of work arounds but none have worked due to either Qt 
issues, Wayland restrictions, Gstreamer restrictions, Weston 
issues/restrictions, NXP hardware engine issues/restrictions etc. Any 
ideas gratefully received!






The higher level surfaces are created/managed by QtWayland, but maybe I
can override somehow.


That does not feel like a good idea to me. But I also cannot really
help you, because this all seems to be pointing at Qt which I know
nothing about.


Yes, thats probably true. But I need to get this to work somehow, even 
if a kludge for now.





  

As mentioned before, If I use QPainter to draw into the video QWidget it
actually draws into the top QWidgets 16 surface using Wayland protocol.
I would have thought it would draw into its own QWidget surface 18 as it
has Qt::WA_NativeWindow set ?

This question seems to be the essence. If Qt worked like you expected,
then I think the whole program would work.

However, there is no need (from Wayland perspective) to have a
wl_surface as "surface 18" in the middle. What would be best is if you
could somehow have that "middle" QWidget but without it's own
wl_surface. Have the QWidget control the GStreamer wl_surface position
through wl_subsurface interface, while GStreamer plays the video
through wl_surface interface.

Wayland does not relay sub-surface resizing or positioning between two
client-side components at all. There is not even a way query a
sub-surface position. So the positioning and sizing is all done in
Qt<->GStreamer somehow without Wayland in between. Only the end result
gets sent over Wayland to display: Qt sets up the position and
GStreamer sets up the size and content.


I think this middle surface is needed so that Qt can manage the 
"Windows" at this level, like raise, lower, resize et. and Wayland 
sink's subsurface that is below this is separate and can be de-synced 
for the video display etc. I normally (on X11 and with Qt5/Wayland), 
respond to QWidget resizes and use the Gstreamer API calls to 
position/resize the waylandsink's sub-surface. This all works quite 
nicely under X11 and worked (although not nicely) under Qt5/Wayland.






I assume that Qtwayland is not "activating" the video QWidget's surface
or using it for some reason (send subsurface expose events ?) ?
  

If that's true, then it is very relevant. A sub-surface becomes mapped
and visible when:

- its parent surface is mapped and visible, and

- the parent surface is committed after the sub-surface has been
created and associated, and

- if the sub-surface is in synchronized mode, there also needs to be a
parent surface commit after every sub-surface commit you want to
become visible. So if you do the first sub-surface sync-mode commit
with a buffer after the parent has already committed the
sub-surface's creation, the parent surface needs too commit again.

This applies recursively, too, and you have a sub-sub-surface there.

Do you actually need to sub-surface in the middle? Have you tried
without it?

I am not doing anything with Wayland directly. Qt is managing the higher
level surfaces/subsurfaces and then GStreamers waylandsink is passed one
of these Qt managed surfaces and it creates the subsurface 44. Looking
at waylandsink it should set this subsurface to be desynced so it can
draw into this surface without synchronising to the parents surface
managed by Qt.

Right, and desync is not enough if its parent is not mapped.


All I am trying to do is use the technique as mentioned in various
forums/lists to get GStreame

Re: Wayland debugging with Qtwayland, gstreamer waylandsink, wayland-lib and Weston

[Terry Barnaby]

Hi Pekka,


Did you try making the "middle" QWidget *not* have a wl_surface of its
own?
Hack on Qt, then? Sorry, but I don't understand this insistence that
what sounds like a Qt bug must be workaround-able via Wayland.
Hmm, that does not sound right to me, but then again, I don't know Qt.

Wayland certainly does not impose such demand.


Well the way this is supposed to work, I believe, is:

1. There are two separate systems in operation here: Qt doing the
   general GUI and GStreamer waylandsink displaying the video. These
   systems know nothing of one another.
2. The link between these two systems is a Wayland surface in the
   Wayland server. QWidget will manage this surface (raise, lower,
   position etc.) and can draw into it if it wants.
3. Waylandsink creates a subsurface of that QWidget Wayland surface,
   sets it to be de-synced and and then proceeds to draw into this at
   the video frame rate.
4. There's quite a lot of hardware engine working going on in the
   background. For example video buffers may be in special memory like
   in a video or 2D hardware engine pipeline etc. Qt may be using
   separate 3D engine hardware etc.

I am not experienced with Wayland, but I think a "middle" surface is 
needed so this can be moved, raised,/lowered etc. relative to the 
applications main QWidgets and the waylandsink does not need to know 
about this (apart from resizes). Another option would be to modify 
waylandsink to do the necessary things with its subsurface. But having a 
separate shared surface from the Qt applications main drawing surface 
seems safer and I am trying to keep with what I think is the accepted 
method with minimal changes to upstream code.


This Gstreamer video display method came from the older X11 way of doing 
this with XWindows.


As stated the reason this is not working with Qt6/Wayland/Weston is 
probably a Qt6 bug/issue/feature. However a way to understand what is 
happening is to look at the shared Wayland level and maybe there is a 
way with Wayland protocol commands of overcoming the issue so I can work 
around the problem I am having in a short time (timescales!) before a 
more proper fix is worked out. For example in X11 an XMapWindow() or 
XRaiseWindow() request or positioning/size requests may have worked and 
I wondered if I could do the same sort of thing in Wayland.


Even if the QtWayland issue is fixed, I may have to do something at the 
Wayland level as I'm not sure if subsurfaces are effectively moved, 
raised/lowered etc. when their parent surface is changed Wayland.


Anyway as David has suggested, I have raised an issue on the Qt Jira 
bugs list at: https://bugreports.qt.io/browse/QTBUG-122941.


Terry


On 29/02/2024 13:39, Pekka Paalanen wrote:

On Wed, 28 Feb 2024 18:04:28 +
Terry Barnaby  wrote:


Hi Pekka,

Some questions below:

Thanks

Terry
On 26/02/2024 15:56, Pekka Paalanen wrote:

Ok. What Wayland API requests cause a surface to actually be mapped

(Sorry don't really know Wayland protocol) ?

Hi Terry,

the basic protocol object is wl_surface. The wl_surface needs to be
given a "role". This is a specific protocol term. xdg_surface and
xdg_toplevel can give a wl_surface the role of a top-level window,
which means it can get mapped when you play by the rules set in the
xdg_toplevel specification.

Sub-surface is another role.

So the rules are always role specific, but at some point they all
require content on the wl_surface, which is given with the attach
request followed by a commit request. Role rules specify when and how
that can be done.

Yes, I have heard that. But what I don't knoe is from the client:

  1. How do I find out the surfaces role ?

It is what you (or Qt, or Gst) set it to. There is no way to query it
(or any other thing) back by Wayland.

If you look at a protocol dump (e.g. WAYLAND_DEBUG=client in
environment), you can could follow the protocol messages and trace back
what the role was set to.


  2. How would I set the surface to have a role such that it would be
 mapped and thus visible ? Just wondering if I can work around what I
 think is a QtWayland bug/issue/feature to make sure by second
 Widgets surface is mapped/visible so that the waylandsink subsurface
 can work. With X11 there were API calls to change the Windows state
 and I was looking for something similar with Wayland.

There is no simple answer to this. You pick a role you need, and then
play by the protocol spec.

You do not have any surfaces without roles, though, so this would not
help you anyway. Roles cannot be changed, only set once per wl_surface
life time. Sub-surface is a role.


I need to find some way to actually display video, simply and
efficiently on an embedded platform, in a Qt application in the year 2024 :)

I have tried lots of work arounds but none have worked due to either Qt
issues, Wayland restrictions, Gstreamer restrictions, Weston
issues/re

Re: Wayland debugging with Qtwayland, gstreamer waylandsink, wayland-lib and Weston

[Terry Barnaby]

While I am trying to investigate my issue in the QtWayland arena via the 
Qt Jira Bug system, I thought I would try taking Qt out of the equation 
to simplify the application a bit more to try and gain some 
understanding of what is going on and how this should all work.


So I have created a pure GStreamer/Wayland/Weston application to test 
out how this should work. This is at: 
https://portal.beam.ltd.uk/public//test022-wayland-video-example.tar.gz


This tries to implement a C++ Widget style application using native 
Wayland. It is rough and could easily be doing things wrong wrt Wayland. 
However it does work to a reasonable degree.


However, I appear to see the same sort of issue I see with my Qt based 
system in that when a subsurface of a subsurface is used, the Gstreamer 
video is not seen.


This example normally (UseWidgetTop=0) has a top level xdg_toplevel 
desktop surface (Gui), a subsurface to that (Video) and then waylandsink 
creates a subsurface to that which it sets to de-sync mode.


When this example is run with UseWidgetTop=0 the video frames from 
gstreamer are only shown shown when the top subsurface is manually 
committed with gvideo->update() every second, otherwise the video 
pipeline is stalled. Waylandsink is stuck in a loop awaiting a Wayland 
callback after committing its subsurface.
If the Video Widget is a top level widget (UseWidgetTop=1) this works 
fine (only one subsurface deep ?).


I have tried using both Gstreamer's waylandsink and glimagesink 
elements, both show the same issue.


This seems to suggest that the de-synced subsurface system is not 
working properly with Weston, I miss-understand how this should work or 
I have a program error.

This has been tested on Fedora37 running Weston 10.0.1 under KDE/Plasma/X11.

1. Should de-synced subsurfaces under other subsurfaces work under 
Weston 10.0.1 ?


2. Do I miss-understand how this should work ?

3. Do I have some coding issue (sorry the code is a bit complex with 
Wayland callbacks and C++ timers etc) ?

Re: Wayland debugging with Qtwayland, gstreamer waylandsink, wayland-lib and Weston

[Terry Barnaby]

On 04/03/2024 09:41, Pekka Paalanen wrote:

On Mon, 4 Mar 2024 08:12:10 +
Terry Barnaby  wrote:


While I am trying to investigate my issue in the QtWayland arena via the
Qt Jira Bug system, I thought I would try taking Qt out of the equation
to simplify the application a bit more to try and gain some
understanding of what is going on and how this should all work.

So I have created a pure GStreamer/Wayland/Weston application to test
out how this should work. This is at:
https://portal.beam.ltd.uk/public//test022-wayland-video-example.tar.gz

This tries to implement a C++ Widget style application using native
Wayland. It is rough and could easily be doing things wrong wrt Wayland.
However it does work to a reasonable degree.

However, I appear to see the same sort of issue I see with my Qt based
system in that when a subsurface of a subsurface is used, the Gstreamer
video is not seen.

This example normally (UseWidgetTop=0) has a top level xdg_toplevel
desktop surface (Gui), a subsurface to that (Video) and then waylandsink
creates a subsurface to that which it sets to de-sync mode.

When this example is run with UseWidgetTop=0 the video frames from
gstreamer are only shown shown when the top subsurface is manually
committed with gvideo->update() every second, otherwise the video
pipeline is stalled.

This is intentional. From wl_subsurface specification:

   Even if a sub-surface is in desynchronized mode, it will behave as
   in synchronized mode, if its parent surface behaves as in
   synchronized mode. This rule is applied recursively throughout the
   tree of surfaces. This means, that one can set a sub-surface into
   synchronized mode, and then assume that all its child and grand-child
   sub-surfaces are synchronized, too, without explicitly setting them.

This is derived from the design decision that a wl_surface and its
immediate sub-surfaces form a seamlessly integrated unit that works
like a single wl_surface without sub-surfaces would. wl_subsurface
state is state in the sub-surface's parent, so that the parent controls
everything as if there was just a single wl_surface. If the parent sets
its sub-surface as desynchronized, it explicitly gives the sub-surface
the permission to update on screen regardless of the parent's updates.
When the sub-surface is in synchronized mode, the parent surface wants
to be updated in sync with the sub-surface in an atomic fashion.

When your surface stack looks like:

- main surface A, top-level, root surface (implicitly desynchronized)
   - sub-surface B, synchronized
 - sub-surface C, desynchronized

Updates to surface C are immediately made part of surface B, because
surface C is in desynchronized mode. If B was the root surface, all C
updates would simply go through.

However, surface B is a part of surface A, and surface B is in
synchronized mode. This means that the client wants surface A updates to
be explicit and atomic. Nothing must change on screen until A is
explicitly committed itself. So any update to surface B requires a
commit on surface A to become visible. Surface C does not get to
override the atomicity requirement of surface A updates.

This has been designed so that software component A can control surface
A, and delegate a part of surface A to component B which happens to the
using a sub-surface: surface B. If surface B parts are further
delegated to another component C, then component A can still be sure
that nothing updates on surface A until it says so. Component A sets
surface B to synchronized to ensure that.

That's the rationale behind the Wayland design.


Thanks,
pq


Ah, thanks for the info, that may be why this is not working even in Qt 
then.


This seems a dropoff in Wayland to me. If a software module wants to 
display Video into an area on the screen at its own rate, setting that 
surface to de-synced mode is no use in the general case with this 
policy. I would have thought that if a subsurface was explicitly set to 
de-synced mode then that would be honoured. I can't see a usage case for 
it to be ignored and its commits synchronised up the tree ?


So is there a way to actually display Video on a subsurface many levels 
deep in a surface hierarchy, would setting all of the surfaces up to the 
subsurface just below the desktop top level one work (although not ideal 
as it would mean overriding other software modules surfaces at the 
Wayland level) ?


Or can desynced subsurfaces really only work to one level deep ?

If it is just one subsurface level deep that video can be displayed, I 
guess I will have to get GStreamers waylandsink to create its subsurface 
off the top most surface and add calls to manage its surface from my 
app. Or maybe get waylandsinks subsurface and manipulate it behind 
waylandsinks back. Not sure what this will do to the Qt level though. 
Using the QWidgets subsurface as its base should have allowed isolation 
to a degree between the Qt and w

Re: Wayland debugging with Qtwayland, gstreamer waylandsink, wayland-lib and Weston

[Terry Barnaby]

On 04/03/2024 14:14, Pekka Paalanen wrote:

On Mon, 4 Mar 2024 13:24:56 +
Terry Barnaby  wrote:


On 04/03/2024 09:41, Pekka Paalanen wrote:

On Mon, 4 Mar 2024 08:12:10 +
Terry Barnaby  wrote:
  

While I am trying to investigate my issue in the QtWayland arena via the
Qt Jira Bug system, I thought I would try taking Qt out of the equation
to simplify the application a bit more to try and gain some
understanding of what is going on and how this should all work.

So I have created a pure GStreamer/Wayland/Weston application to test
out how this should work. This is at:
https://portal.beam.ltd.uk/public//test022-wayland-video-example.tar.gz

This tries to implement a C++ Widget style application using native
Wayland. It is rough and could easily be doing things wrong wrt Wayland.
However it does work to a reasonable degree.

However, I appear to see the same sort of issue I see with my Qt based
system in that when a subsurface of a subsurface is used, the Gstreamer
video is not seen.

This example normally (UseWidgetTop=0) has a top level xdg_toplevel
desktop surface (Gui), a subsurface to that (Video) and then waylandsink
creates a subsurface to that which it sets to de-sync mode.

When this example is run with UseWidgetTop=0 the video frames from
gstreamer are only shown shown when the top subsurface is manually
committed with gvideo->update() every second, otherwise the video
pipeline is stalled.

This is intentional. From wl_subsurface specification:

Even if a sub-surface is in desynchronized mode, it will behave as
in synchronized mode, if its parent surface behaves as in
synchronized mode. This rule is applied recursively throughout the
tree of surfaces. This means, that one can set a sub-surface into
synchronized mode, and then assume that all its child and grand-child
sub-surfaces are synchronized, too, without explicitly setting them.

This is derived from the design decision that a wl_surface and its
immediate sub-surfaces form a seamlessly integrated unit that works
like a single wl_surface without sub-surfaces would. wl_subsurface
state is state in the sub-surface's parent, so that the parent controls
everything as if there was just a single wl_surface. If the parent sets
its sub-surface as desynchronized, it explicitly gives the sub-surface
the permission to update on screen regardless of the parent's updates.
When the sub-surface is in synchronized mode, the parent surface wants
to be updated in sync with the sub-surface in an atomic fashion.

When your surface stack looks like:

- main surface A, top-level, root surface (implicitly desynchronized)
- sub-surface B, synchronized
  - sub-surface C, desynchronized

Updates to surface C are immediately made part of surface B, because
surface C is in desynchronized mode. If B was the root surface, all C
updates would simply go through.

However, surface B is a part of surface A, and surface B is in
synchronized mode. This means that the client wants surface A updates to
be explicit and atomic. Nothing must change on screen until A is
explicitly committed itself. So any update to surface B requires a
commit on surface A to become visible. Surface C does not get to
override the atomicity requirement of surface A updates.

This has been designed so that software component A can control surface
A, and delegate a part of surface A to component B which happens to the
using a sub-surface: surface B. If surface B parts are further
delegated to another component C, then component A can still be sure
that nothing updates on surface A until it says so. Component A sets
surface B to synchronized to ensure that.

That's the rationale behind the Wayland design.


Thanks,
pq

Ah, thanks for the info, that may be why this is not working even in Qt
then.

This seems a dropoff in Wayland to me. If a software module wants to
display Video into an area on the screen at its own rate, setting that
surface to de-synced mode is no use in the general case with this
policy.

It is of use, if you don't have unnecessary sub-surfaces in synchronized
mode in between, or you set all those extra sub-surfaces to
desynchronized as well.


Well they may not be necessary from the Wayland perspective, but from 
the higher level software they are useful to modularise/separate/provide 
a join for the software modules especially when software modules are 
separate like Qt and GStreamer.






I would have thought that if a subsurface was explicitly set to
de-synced mode then that would be honoured. I can't see a usage case for
it to be ignored and its commits synchronised up the tree ?

Resizing the window is the main use case.

In order to resize surface A, you also need to resize and paint surface
B, and for surface B you also need to resize and paint surface C. Then
you need to guarantee that all the updates from surface C, B and A are
applied atomically on screen.

Either you have c

Re: Wayland debugging with Qtwayland, gstreamer waylandsink, wayland-lib and Weston

[Terry Barnaby]

On 04/03/2024 15:50, Pekka Paalanen wrote:

On Mon, 4 Mar 2024 14:51:52 +
Terry Barnaby  wrote:


On 04/03/2024 14:14, Pekka Paalanen wrote:

On Mon, 4 Mar 2024 13:24:56 +
Terry Barnaby  wrote:
  

On 04/03/2024 09:41, Pekka Paalanen wrote:

On Mon, 4 Mar 2024 08:12:10 +
Terry Barnaby  wrote:
 

While I am trying to investigate my issue in the QtWayland arena via the
Qt Jira Bug system, I thought I would try taking Qt out of the equation
to simplify the application a bit more to try and gain some
understanding of what is going on and how this should all work.

So I have created a pure GStreamer/Wayland/Weston application to test
out how this should work. This is at:
https://portal.beam.ltd.uk/public//test022-wayland-video-example.tar.gz

This tries to implement a C++ Widget style application using native
Wayland. It is rough and could easily be doing things wrong wrt Wayland.
However it does work to a reasonable degree.

However, I appear to see the same sort of issue I see with my Qt based
system in that when a subsurface of a subsurface is used, the Gstreamer
video is not seen.

This example normally (UseWidgetTop=0) has a top level xdg_toplevel
desktop surface (Gui), a subsurface to that (Video) and then waylandsink
creates a subsurface to that which it sets to de-sync mode.

When this example is run with UseWidgetTop=0 the video frames from
gstreamer are only shown shown when the top subsurface is manually
committed with gvideo->update() every second, otherwise the video
pipeline is stalled.

This is intentional. From wl_subsurface specification:

 Even if a sub-surface is in desynchronized mode, it will behave as
 in synchronized mode, if its parent surface behaves as in
 synchronized mode. This rule is applied recursively throughout the
 tree of surfaces. This means, that one can set a sub-surface into
 synchronized mode, and then assume that all its child and grand-child
 sub-surfaces are synchronized, too, without explicitly setting them.

This is derived from the design decision that a wl_surface and its
immediate sub-surfaces form a seamlessly integrated unit that works
like a single wl_surface without sub-surfaces would. wl_subsurface
state is state in the sub-surface's parent, so that the parent controls
everything as if there was just a single wl_surface. If the parent sets
its sub-surface as desynchronized, it explicitly gives the sub-surface
the permission to update on screen regardless of the parent's updates.
When the sub-surface is in synchronized mode, the parent surface wants
to be updated in sync with the sub-surface in an atomic fashion.

When your surface stack looks like:

- main surface A, top-level, root surface (implicitly desynchronized)
 - sub-surface B, synchronized
   - sub-surface C, desynchronized

Updates to surface C are immediately made part of surface B, because
surface C is in desynchronized mode. If B was the root surface, all C
updates would simply go through.

However, surface B is a part of surface A, and surface B is in
synchronized mode. This means that the client wants surface A updates to
be explicit and atomic. Nothing must change on screen until A is
explicitly committed itself. So any update to surface B requires a
commit on surface A to become visible. Surface C does not get to
override the atomicity requirement of surface A updates.

This has been designed so that software component A can control surface
A, and delegate a part of surface A to component B which happens to the
using a sub-surface: surface B. If surface B parts are further
delegated to another component C, then component A can still be sure
that nothing updates on surface A until it says so. Component A sets
surface B to synchronized to ensure that.

That's the rationale behind the Wayland design.


Thanks,
pq

Ah, thanks for the info, that may be why this is not working even in Qt
then.

This seems a dropoff in Wayland to me. If a software module wants to
display Video into an area on the screen at its own rate, setting that
surface to de-synced mode is no use in the general case with this
policy.

It is of use, if you don't have unnecessary sub-surfaces in synchronized
mode in between, or you set all those extra sub-surfaces to
desynchronized as well.

Well they may not be necessary from the Wayland perspective, but from
the higher level software they are useful to modularise/separate/provide
a join for the software modules especially when software modules are
separate like Qt and GStreamer.

Sorry to hear that.


I would have thought that if a subsurface was explicitly set to
de-synced mode then that would be honoured. I can't see a usage case for
it to be ignored and its commits synchronised up the tree ?

Resizing the window is the main use case.

In order to resize surface A, you also need to resize and paint surface
B, and for surface B you also need to resize and paint surface C. The

Re: Wayland debugging with Qtwayland, gstreamer waylandsink, wayland-lib and Weston

[Terry Barnaby]

On 05/03/2024 12:26, Pekka Paalanen wrote:

On Mon, 4 Mar 2024 17:59:25 +
Terry Barnaby  wrote:


On 04/03/2024 15:50, Pekka Paalanen wrote:

On Mon, 4 Mar 2024 14:51:52 +
Terry Barnaby  wrote:

On 04/03/2024 14:14, Pekka Paalanen wrote:

On Mon, 4 Mar 2024 13:24:56 +
Terry Barnaby  wrote:

On 04/03/2024 09:41, Pekka Paalanen wrote:

On Mon, 4 Mar 2024 08:12:10 +
Terry Barnaby  wrote:
While I am trying to investigate my issue in the QtWayland 
arena via the
Qt Jira Bug system, I thought I would try taking Qt out of the 
equation

to simplify the application a bit more to try and gain some
understanding of what is going on and how this should all work.

So I have created a pure GStreamer/Wayland/Weston application 
to test

out how this should work. This is at:
https://portal.beam.ltd.uk/public//test022-wayland-video-example.tar.gz

This tries to implement a C++ Widget style application using native
Wayland. It is rough and could easily be doing things wrong wrt 
Wayland.

However it does work to a reasonable degree.

However, I appear to see the same sort of issue I see with my 
Qt based
system in that when a subsurface of a subsurface is used, the 
Gstreamer

video is not seen.

This example normally (UseWidgetTop=0) has a top level xdg_toplevel
desktop surface (Gui), a subsurface to that (Video) and then 
waylandsink

creates a subsurface to that which it sets to de-sync mode.

When this example is run with UseWidgetTop=0 the video frames from
gstreamer are only shown shown when the top subsurface is manually
committed with gvideo->update() every second, otherwise the video
pipeline is stalled.

This is intentional. From wl_subsurface specification:

Even if a sub-surface is in desynchronized mode, it will behave as
in synchronized mode, if its parent surface behaves as in
synchronized mode. This rule is applied recursively throughout the
tree of surfaces. This means, that one can set a sub-surface into
synchronized mode, and then assume that all its child and 
grand-child

sub-surfaces are synchronized, too, without explicitly setting them.

This is derived from the design decision that a wl_surface and its
immediate sub-surfaces form a seamlessly integrated unit that works
like a single wl_surface without sub-surfaces would. wl_subsurface
state is state in the sub-surface's parent, so that the parent 
controls
everything as if there was just a single wl_surface. If the 
parent sets
its sub-surface as desynchronized, it explicitly gives the 
sub-surface
the permission to update on screen regardless of the parent's 
updates.
When the sub-surface is in synchronized mode, the parent surface 
wants

to be updated in sync with the sub-surface in an atomic fashion.

When your surface stack looks like:

- main surface A, top-level, root surface (implicitly 
desynchronized)

- sub-surface B, synchronized
- sub-surface C, desynchronized

Updates to surface C are immediately made part of surface B, because
surface C is in desynchronized mode. If B was the root surface, 
all C

updates would simply go through.

However, surface B is a part of surface A, and surface B is in
synchronized mode. This means that the client wants surface A 
updates to

be explicit and atomic. Nothing must change on screen until A is
explicitly committed itself. So any update to surface B requires a
commit on surface A to become visible. Surface C does not get to
override the atomicity requirement of surface A updates.

This has been designed so that software component A can control 
surface
A, and delegate a part of surface A to component B which happens 
to the

using a sub-surface: surface B. If surface B parts are further
delegated to another component C, then component A can still be sure
that nothing updates on surface A until it says so. Component A sets
surface B to synchronized to ensure that.

That's the rationale behind the Wayland design.


Thanks,
pq
Ah, thanks for the info, that may be why this is not working even 
in Qt

then.

This seems a dropoff in Wayland to me. If a software module wants to
display Video into an area on the screen at its own rate, setting 
that

surface to de-synced mode is no use in the general case with this
policy.
It is of use, if you don't have unnecessary sub-surfaces in 
synchronized

mode in between, or you set all those extra sub-surfaces to
desynchronized as well.

Well they may not be necessary from the Wayland perspective, but from
the higher level software they are useful to 
modularise/separate/provide

a join for the software modules especially when software modules are
separate like Qt and GStreamer.

Sorry to hear that.

I would have thought that if a subsurface was explicitly set to
de-synced mode then that would be honoured. I can't see a usage 
case for

it to be ignored and its commits synchronised up the tree ?

Resizing the window is the main use case.

In order to resize surface A, you also need to resize and paint 
surface

B, and

Re: Wayland debugging with Qtwayland, gstreamer waylandsink, wayland-lib and Weston

[Terry Barnaby]

On 08/03/2024 15:23, Pekka Paalanen wrote:

On Fri, 8 Mar 2024 14:50:30 +
Terry Barnaby  wrote:


On 05/03/2024 12:26, Pekka Paalanen wrote:

On Mon, 4 Mar 2024 17:59:25 +
Terry Barnaby  wrote:
  

...


I would have thought it better/more useful to have a Wayland API call
like "stopCommiting" so that an application can sort things out for this
and other things, providing more application control. But I really have
only very limited knowledge of the Wayland system. I just keep hitting
its restrictions.
  

Right, Wayland does not work that way. Wayland sees any client as a
single entity, regardless of its internal composition of libraries and
others.

When Wayland delivers any event, whether it is an explicit resize event
or an input event (or maybe the client just spontaneously decides to),
that causes the client to want to resize a window, it is then up to the
client itself to make sure it resizes everything it needs to, and keeps
everything atomic so that the end user does not see glitches on screen.

Sub-surfaces' synchronous mode was needed to let clients batch the
updates of multiple surfaces into a single atomic commit. It is the
desync mode that was a non-mandatory add-on. The synchronous mode was
needed, because there was no other way to batch multiple
wl_surface.commit requests to apply simultaneously guaranteed. Without
it, if you commit surface A and then surface B, nothing will guarantee
that the compositor would not show A updated and B not on screen for a
moment.

Wayland intentionally did not include any mechanism in its design
intended for communication between a single client's internal
components. Why use a display server as an IPC middle-man for something
that should be process-internal communication. After all, Wayland is
primarily a protocol - inter-process communication.

Well as you say it is up to the client to perform all of the surface
resize work. So it seems to me, if the client had an issue with pixel
perfect resizing it could always set any of its desynced surfaces to
sync mode, or just stop the update to them, while it resizes. I don't
see why Wayland needs to ignore the clients request to set a subsurface
desynced down the tree.

You're right, but it's in the spec now. I've gained a bit more
experience in the decade after writing the sub-surface spec.

You can still work around it by setting all sub-surfaces always desync.


Oh you wrote it, thanks for the work!

So maybe time for version n+1 then :)

Actually allowing sub/subsurfaces to work in desync should not break any 
existing clients as they cannot use it yet. Obviously new clients 
written for it would not work on older Wayland servers though.


Its difficult to desync all the higher surfaces in a Qt or probably 
other Widget set application, they are controlled by Qt and Qt does not 
give you access to the subsurfaces it has created. It would be better to 
have had a wl_surface_set_desync(wl_surface*) rather than a 
wl_subsurface_set_desync(wl_subsurface*).


With clients using lots of libraries/subsystems it is better to not use 
their internal workings unless you have to. Normally you try and work at 
the least common denominator, in this case the Wayland display system as 
that is the shared module they both use (at least when driving a Wayland 
display server). This is why it is nice to have a surface that is almost 
totally independent of others and just is shown/not shown, is over/below 
etc. other surfaces like an XWindow. The Wayland surfaces are mainly 
this as far as I can see, apart from this desync mode although maybe 
there are others.


I have asked in the Qt forums if they could provide some sort of API to 
allow the setting of desync up the tree, but this may not happen and it 
might be difficult for them as it could mess up their applications 
rendering. It also does not match other display system API's that they 
support. The higher level QWidgets ideally need synced surfaces, its 
just the Video surfaces that need desync. Really I think this is the 
Wayland servers job.






In fact does it return an error to the client
when the Wayland server ignores this command ?

There is no "return error" in Wayland. Either a request succeeds, or
the client is disconnected with an error. It's all asynchronous, too.

Any possibility for graceful failure must be designed into protocol
extensions at one step higher level. If there is room for a graceful
failure, it will be mentioned in the XML spec with explicit messages to
communicate it.

Which command do you mean?


I meant the wl_subsurface_set_desync() API call on a sub/subsurface that 
doesn't work. As no errors were returned it took a long time to find out 
why things weren't working, just some lower level threads locked up.


Personally I think these sort of occasional, performance irrelevant, 
types of methods/requests/commands should be synchronous (maybe under an 
as

Re: Wayland debugging with Qtwayland, gstreamer waylandsink, wayland-lib and Weston

[Terry Barnaby]

On 05/03/2024 12:26, Pekka Paalanen wrote:

On Mon, 4 Mar 2024 17:59:25 +
Terry Barnaby  wrote:


On 04/03/2024 15:50, Pekka Paalanen wrote:

On Mon, 4 Mar 2024 14:51:52 +
Terry Barnaby  wrote:
  

On 04/03/2024 14:14, Pekka Paalanen wrote:

On Mon, 4 Mar 2024 13:24:56 +
Terry Barnaby  wrote:
 

On 04/03/2024 09:41, Pekka Paalanen wrote:

On Mon, 4 Mar 2024 08:12:10 +
Terry Barnaby  wrote:


While I am trying to investigate my issue in the QtWayland arena via the
Qt Jira Bug system, I thought I would try taking Qt out of the equation
to simplify the application a bit more to try and gain some
understanding of what is going on and how this should all work.

So I have created a pure GStreamer/Wayland/Weston application to test
out how this should work. This is at:
https://portal.beam.ltd.uk/public//test022-wayland-video-example.tar.gz

This tries to implement a C++ Widget style application using native
Wayland. It is rough and could easily be doing things wrong wrt Wayland.
However it does work to a reasonable degree.

However, I appear to see the same sort of issue I see with my Qt based
system in that when a subsurface of a subsurface is used, the Gstreamer
video is not seen.

This example normally (UseWidgetTop=0) has a top level xdg_toplevel
desktop surface (Gui), a subsurface to that (Video) and then waylandsink
creates a subsurface to that which it sets to de-sync mode.

When this example is run with UseWidgetTop=0 the video frames from
gstreamer are only shown shown when the top subsurface is manually
committed with gvideo->update() every second, otherwise the video
pipeline is stalled.

This is intentional. From wl_subsurface specification:

  Even if a sub-surface is in desynchronized mode, it will behave as
  in synchronized mode, if its parent surface behaves as in
  synchronized mode. This rule is applied recursively throughout the
  tree of surfaces. This means, that one can set a sub-surface into
  synchronized mode, and then assume that all its child and grand-child
  sub-surfaces are synchronized, too, without explicitly setting them.

This is derived from the design decision that a wl_surface and its
immediate sub-surfaces form a seamlessly integrated unit that works
like a single wl_surface without sub-surfaces would. wl_subsurface
state is state in the sub-surface's parent, so that the parent controls
everything as if there was just a single wl_surface. If the parent sets
its sub-surface as desynchronized, it explicitly gives the sub-surface
the permission to update on screen regardless of the parent's updates.
When the sub-surface is in synchronized mode, the parent surface wants
to be updated in sync with the sub-surface in an atomic fashion.

When your surface stack looks like:

- main surface A, top-level, root surface (implicitly desynchronized)
  - sub-surface B, synchronized
- sub-surface C, desynchronized

Updates to surface C are immediately made part of surface B, because
surface C is in desynchronized mode. If B was the root surface, all C
updates would simply go through.

However, surface B is a part of surface A, and surface B is in
synchronized mode. This means that the client wants surface A updates to
be explicit and atomic. Nothing must change on screen until A is
explicitly committed itself. So any update to surface B requires a
commit on surface A to become visible. Surface C does not get to
override the atomicity requirement of surface A updates.

This has been designed so that software component A can control surface
A, and delegate a part of surface A to component B which happens to the
using a sub-surface: surface B. If surface B parts are further
delegated to another component C, then component A can still be sure
that nothing updates on surface A until it says so. Component A sets
surface B to synchronized to ensure that.

That's the rationale behind the Wayland design.


Thanks,
pq

Ah, thanks for the info, that may be why this is not working even in Qt
then.

This seems a dropoff in Wayland to me. If a software module wants to
display Video into an area on the screen at its own rate, setting that
surface to de-synced mode is no use in the general case with this
policy.

It is of use, if you don't have unnecessary sub-surfaces in synchronized
mode in between, or you set all those extra sub-surfaces to
desynchronized as well.

Well they may not be necessary from the Wayland perspective, but from
the higher level software they are useful to modularise/separate/provide
a join for the software modules especially when software modules are
separate like Qt and GStreamer.

Sorry to hear that.
  

I would have thought that if a subsurface was explicitly set to
de-synced mode then that would be honoured. I can't see a usage case for
it to be ignored and its commits synchronised up the tree ?

Resizing the window is the main use case.

In order to resize