Re: Full-motion zero-copy screen capture in Weston

[Pekka Paalanen]

On Thu, 30 May 2024 13:40:15 +
"Hoosier, Matt"  wrote:

> Okay, interesting thoughts on all of that.
> 
> I'm not sure how far I'm going to get toward a complete overhaul of
> the capture mechanism. Maybe it would be useful to know a couple
> things about the current one (weston_output_capture_v1), so that I
> don't commit mistakes that somebody already considered and guarded
> against:
> 
> * Why was it explicitly picked only for still shots? I can image that
> it wouldn't have been a whole lot more work to design this API with a
> bounce-buffering scheme rather than a setup/do-it-once/destroy model.
> 

There was no need for streaming, as it was purely for the test suite.

The test suite is very particular about when and how the shot is taken:
it must be produced by the defined source, and it must be produced on
the very next repaint of the output, exactly once.

Did you mean a buffer pool (queueing/dequeueing) rather than
bounce-buffering? Sure.

> * Why was wl_shm explicitly picked for the buffer type? I was
> thinking here that it would have worked equally well to specify that
> the client must supply a linear-layout buffer manufactured through
> zwp_linux_dmabuf. This would be eligible for direct targeting by the
> GL renderer/KMS writeback, but also can be mapped with gbm_bo_map()
> in the compositor's Pixman backend and/or a client who wants to map
> it to the CPU upon delivery.

Because the test suite specifically needs CPU access to the screenshot.
There was no use yet for dmabuf, and GL-renderer was already doing
glReadPixels() for screenshots.

IOW, all the limitations are just "was not needed yet".

Note, that re-using or extending the protocol extension
weston_capture_v1 for streaming outputs for non-test-suite use cases
may not be the best idea. If the interface needs to be a Wayland
extension, then maybe something from the wlr extensions would suit
better. OTOH, for e.g. Pipewire there would not be a Wayland extension
used at all.

The internal API (weston_output_capture) is very much geared for
weston_capture_v1 only. The internal API might take improving rather
than weston_capture_v1.


Thanks,
pq


pgpGdEI0WVF9N.pgp
Description: OpenPGP digital signature

Re: Full-motion zero-copy screen capture in Weston

[Hoosier, Matt]

Hi,

Yeah. I agree that although I can prototype something quick and dirty here as a 
change to weston_output_capture_v1, it's probably not a good fit there.

Drew or Simon, does either of 
https://github.com/swaywm/wlroots/blob/master/protocol/wlr-screencopy-unstable-v1.xml
 or 
https://github.com/swaywm/wlroots/blob/master/protocol/wlr-export-dmabuf-unstable-v1.xml
 strike you as a good fit for the use-case of getting a streamed copy of an 
output? They seem very similar – not sure what differentiates them from each 
other.

My goal is to implement this screen capture with a guarantee that the copy 
comes from a KMS writeback connector. I know this sounds like an odd thing to 
insist on. Let's say that in my industry, the system is rarely only running 
Linux directly on the bare metal. Using the writeback hardware on the display 
controller allows to get a copy of content from all the virtual machines in the 
system.

Frankly, weston_output_capture_v1's model that clients allocate the buffers 
would make it very difficult to support efficient screen capture for more than 
one simultaneous client. You can only target one writeback framebuffer per page 
flip. Having the compositor manage the buffers' lifetimes and just publishing 
out handles (in the style of those two wlr extensions) to those probably fits 
better.

-Matt


From: Pekka Paalanen
Sent: Friday, May 31, 2024 3:02 AM
To: Hoosier, Matt
Cc: Marius Vlad; wayland-devel@lists.freedesktop.org
Subject: Re: Full-motion zero-copy screen capture in Weston

On Thu, 30 May 2024 13:40:15 +
"Hoosier, Matt"  wrote:

> Okay, interesting thoughts on all of that.
>
> I'm not sure how far I'm going to get toward a complete overhaul of
> the capture mechanism. Maybe it would be useful to know a couple
> things about the current one (weston_output_capture_v1), so that I
> don't commit mistakes that somebody already considered and guarded
> against:
>
> * Why was it explicitly picked only for still shots? I can image that
> it wouldn't have been a whole lot more work to design this API with a
> bounce-buffering scheme rather than a setup/do-it-once/destroy model.
>

There was no need for streaming, as it was purely for the test suite.

The test suite is very particular about when and how the shot is taken:
it must be produced by the defined source, and it must be produced on
the very next repaint of the output, exactly once.

Did you mean a buffer pool (queueing/dequeueing) rather than
bounce-buffering? Sure.

> * Why was wl_shm explicitly picked for the buffer type? I was
> thinking here that it would have worked equally well to specify that
> the client must supply a linear-layout buffer manufactured through
> zwp_linux_dmabuf. This would be eligible for direct targeting by the
> GL renderer/KMS writeback, but also can be mapped with gbm_bo_map()
> in the compositor's Pixman backend and/or a client who wants to map
> it to the CPU upon delivery.

Because the test suite specifically needs CPU access to the screenshot.
There was no use yet for dmabuf, and GL-renderer was already doing
glReadPixels() for screenshots.

IOW, all the limitations are just "was not needed yet".

Note, that re-using or extending the protocol extension
weston_capture_v1 for streaming outputs for non-test-suite use cases
may not be the best idea. If the interface needs to be a Wayland
extension, then maybe something from the wlr extensions would suit
better. OTOH, for e.g. Pipewire there would not be a Wayland extension
used at all.

The internal API (weston_output_capture) is very much geared for
weston_capture_v1 only. The internal API might take improving rather
than weston_capture_v1.


Thanks,
pq

Re: Full-motion zero-copy screen capture in Weston

[Simon Ser]

On Friday, May 31st, 2024 at 15:26, Hoosier, Matt  
wrote:

> Drew or Simon, does either of 
> https://github.com/swaywm/wlroots/blob/master/protocol/wlr-screencopy-unstable-v1.xml
>  or 
> https://github.com/swaywm/wlroots/blob/master/protocol/wlr-export-dmabuf-unstable-v1.xml
>  strike you as a good fit for the use-case of getting a streamed copy of an 
> output? They seem very similar – not sure what differentiates them from each 
> other.

screencopy is designed around client-allocated buffers. For
compositor-allocated buffers, export-dmabuf is what's used in the
wlroots ecosystem.

> My goal is to implement this screen capture with a guarantee that the copy 
> comes from a KMS writeback connector. I know this sounds like an odd thing to 
> insist on. Let's say that in my industry, the system is rarely only running 
> Linux directly on the bare metal. Using the writeback hardware on the display 
> controller allows to get a copy of content from all the virtual machines in 
> the system.
> Frankly, weston_output_capture_v1's model that clients allocate the buffers 
> would make it very difficult to support efficient screen capture for more 
> than one simultaneous client. You can only target one writeback framebuffer 
> per page flip. Having the compositor manage the buffers' lifetimes and just 
> publishing out handles (in the style of those two wlr extensions) to those 
> probably fits better.

IIRC Weston is not interested in supporting the wlroots protocol, they'd
prefer to use e.g. PipeWire. Should be possible to pass the result of
the writeback into a PipeWire stream.

> From: Pekka Paalanen
> Sent: Friday, May 31, 2024 3:02 AM
> To: Hoosier, Matt
> Cc: Marius Vlad; wayland-devel@lists.freedesktop.org
> Subject: Re: Full-motion zero-copy screen capture in Weston
> On Thu, 30 May 2024 13:40:15 +
> "Hoosier, Matt"  wrote:
> 
> > Okay, interesting thoughts on all of that.
> >
> > I'm not sure how far I'm going to get toward a complete overhaul of
> > the capture mechanism. Maybe it would be useful to know a couple
> > things about the current one (weston_output_capture_v1), so that I
> > don't commit mistakes that somebody already considered and guarded
> > against:
> >
> > * Why was it explicitly picked only for still shots? I can image that
> > it wouldn't have been a whole lot more work to design this API with a
> > bounce-buffering scheme rather than a setup/do-it-once/destroy model.
> >
> 
> There was no need for streaming, as it was purely for the test suite.
> 
> The test suite is very particular about when and how the shot is taken:
> it must be produced by the defined source, and it must be produced on
> the very next repaint of the output, exactly once.
> 
> Did you mean a buffer pool (queueing/dequeueing) rather than
> bounce-buffering? Sure.
> 
> > * Why was wl_shm explicitly picked for the buffer type? I was
> > thinking here that it would have worked equally well to specify that
> > the client must supply a linear-layout buffer manufactured through
> > zwp_linux_dmabuf. This would be eligible for direct targeting by the
> > GL renderer/KMS writeback, but also can be mapped with gbm_bo_map()
> > in the compositor's Pixman backend and/or a client who wants to map
> > it to the CPU upon delivery.
> 
> Because the test suite specifically needs CPU access to the screenshot.
> There was no use yet for dmabuf, and GL-renderer was already doing
> glReadPixels() for screenshots.
> 
> IOW, all the limitations are just "was not needed yet".
> 
> Note, that re-using or extending the protocol extension
> weston_capture_v1 for streaming outputs for non-test-suite use cases
> may not be the best idea. If the interface needs to be a Wayland
> extension, then maybe something from the wlr extensions would suit
> better. OTOH, for e.g. Pipewire there would not be a Wayland extension
> used at all.
> 
> The internal API (weston_output_capture) is very much geared for
> weston_capture_v1 only. The internal API might take improving rather
> than weston_capture_v1.
> 
> 
> Thanks,
> pq

Re: Full-motion zero-copy screen capture in Weston

[Andri Yngvason]

Hi Matt,

fös., 31. maí 2024 kl. 13:26 skrifaði Hoosier, Matt :
>
> Hi,
>
> Yeah. I agree that although I can prototype something quick and dirty here as 
> a change to weston_output_capture_v1, it's probably not a good fit there.
>
> Drew or Simon, does either of 
> https://github.com/swaywm/wlroots/blob/master/protocol/wlr-screencopy-unstable-v1.xml
>  or 
> https://github.com/swaywm/wlroots/blob/master/protocol/wlr-export-dmabuf-unstable-v1.xml
>  strike you as a good fit for the use-case of getting a streamed copy of an 
> output? They seem very similar – not sure what differentiates them from each 
> other.
>

I'd say ext-screencopy-v1 is pretty close to being complete. It is
designed for both single shot capture and screen casting. It has 2
acks and it has been implemented in wlroots, cosmic and wayvnc.

I would not recommend wlr-export-dmabuf for anything as it suffers
from synchronisation issues. There are no guarantees for life-times of
the dmabufs. wlr-screencopy works well enough, but ext-screencopy-v1
is better.

See: 
https://gitlab.freedesktop.org/wayland/wayland-protocols/-/merge_requests/124

> My goal is to implement this screen capture with a guarantee that the copy 
> comes from a KMS writeback connector. I know this sounds like an odd thing to 
> insist on. Let's say that in my industry, the system is rarely only running 
> Linux directly on the bare metal. Using the writeback hardware on the display 
> controller allows to get a copy of content from all the virtual machines in 
> the system.

Although the protocol is called "screencopy", it does not need to be
implemented via copying. I don't think anything precludes drm
write-back or rendering directly to the client-submitted buffers.

Regards,
Andri

Re: Full-motion zero-copy screen capture in Weston

[Simon Ser]

On Friday, May 31st, 2024 at 16:39, Andri Yngvason  wrote:

> Hi Matt,
> 
> fös., 31. maí 2024 kl. 13:26 skrifaði Hoosier, Matt matt.hoos...@garmin.com:
> 
> > Hi,
> > 
> > Yeah. I agree that although I can prototype something quick and dirty here 
> > as a change to weston_output_capture_v1, it's probably not a good fit there.
> > 
> > Drew or Simon, does either of 
> > https://github.com/swaywm/wlroots/blob/master/protocol/wlr-screencopy-unstable-v1.xml
> >  or 
> > https://github.com/swaywm/wlroots/blob/master/protocol/wlr-export-dmabuf-unstable-v1.xml
> >  strike you as a good fit for the use-case of getting a streamed copy of an 
> > output? They seem very similar – not sure what differentiates them from 
> > each other.
> 
> 
> I'd say ext-screencopy-v1 is pretty close to being complete. It is
> designed for both single shot capture and screen casting. It has 2
> acks and it has been implemented in wlroots, cosmic and wayvnc.
> 
> I would not recommend wlr-export-dmabuf for anything as it suffers
> from synchronisation issues. There are no guarantees for life-times of
> the dmabufs. wlr-screencopy works well enough, but ext-screencopy-v1
> is better.

For fixing the sync issue, there is:
https://gitlab.freedesktop.org/wlroots/wlr-protocols/-/merge_requests/121

But not enough interest it seems.

> See: 
> https://gitlab.freedesktop.org/wayland/wayland-protocols/-/merge_requests/124
> 
> > My goal is to implement this screen capture with a guarantee that the copy 
> > comes from a KMS writeback connector. I know this sounds like an odd thing 
> > to insist on. Let's say that in my industry, the system is rarely only 
> > running Linux directly on the bare metal. Using the writeback hardware on 
> > the display controller allows to get a copy of content from all the virtual 
> > machines in the system.
> 
> 
> Although the protocol is called "screencopy", it does not need to be
> implemented via copying. I don't think anything precludes drm
> write-back or rendering directly to the client-submitted buffers.

Not sure that's true… With screencopy the client necessarily provides
buffers that the compositor copies into.

Re: Full-motion zero-copy screen capture in Weston

[Simon Ser]

On Friday, May 31st, 2024 at 16:45, Simon Ser  wrote:

> > See: 
> > https://gitlab.freedesktop.org/wayland/wayland-protocols/-/merge_requests/124
> > 
> > > My goal is to implement this screen capture with a guarantee that the 
> > > copy comes from a KMS writeback connector. I know this sounds like an odd 
> > > thing to insist on. Let's say that in my industry, the system is rarely 
> > > only running Linux directly on the bare metal. Using the writeback 
> > > hardware on the display controller allows to get a copy of content from 
> > > all the virtual machines in the system.
> > 
> > Although the protocol is called "screencopy", it does not need to be
> > implemented via copying. I don't think anything precludes drm
> > write-back or rendering directly to the client-submitted buffers.
> 
> Not sure that's true… With screencopy the client necessarily provides
> buffers that the compositor copies into.

Correction: Andri said that it should be possible to pass client-allocated
buffers to the writeback connector. There might be allocation considerations
to take care of, but it might work fine indeed.

Re: Full-motion zero-copy screen capture in Weston

[Hoosier, Matt]

Thanks, understood. I would expect that I need to take some care in allocating 
a buffer that my DRM driver accepts for writeback, in this usage.

Do I interpret right that the wlroots screencopy extension wants the client to 
create a fresh zwlr_screencopy_frame_v1 for each successive frame? I wonder 
then:


  *   Whether this might lead to the possibility of dropped frames, if the 
client doesn’t manage to send back the requests to start capture for frame N+1 
soon enough after the delivery of the copied buffer for frame N; and
  *   Will the explicit request for each frame via zwlr_screencopy_frame_v1:: 
copy(), result in artificial redraws to satisfy the request? Ideally I would 
just like to receive a passive copy of each frame that naturally got drawn by 
the compositor. Perhaps with one artificial redraw at the very beginning of the 
screencopy stream just to have a defined starting point.

-Matt

From: Simon Ser 
Date: Friday, May 31, 2024 at 10:16 AM
To: Andri Yngvason 
Cc: "Hoosier, Matt" , Pekka Paalanen 
, "s...@cmpwn.com" , Marius Vlad 
, "wayland-devel@lists.freedesktop.org" 

Subject: Re: Full-motion zero-copy screen capture in Weston

On Friday, May 31st, 2024 at 16: 45, Simon Ser  wrote: > 
> See: https: //urldefense. com/v3/__https: //gitlab. freedesktop. 
org/wayland/wayland-protocols/-/merge_requests/124__;!!EJc4YC3iFmQ!WaOP607PgstX4CpWrn4DV6X4TIDRrqQW5XEZfczyVDFZdkpmCfNDh0lUQetO5TuCK5ct4NVlCqcAz4-8ehlw$


On Friday, May 31st, 2024 at 16:45, Simon Ser  wrote:



> > See: 
> > https://urldefense.com/v3/__https://gitlab.freedesktop.org/wayland/wayland-protocols/-/merge_requests/124__;!!EJc4YC3iFmQ!WaOP607PgstX4CpWrn4DV6X4TIDRrqQW5XEZfczyVDFZdkpmCfNDh0lUQetO5TuCK5ct4NVlCqcAz4-8ehlw$

> >

> > > My goal is to implement this screen capture with a guarantee that the 
> > > copy comes from a KMS writeback connector. I know this sounds like an odd 
> > > thing to insist on. Let's say that in my industry, the system is rarely 
> > > only running Linux directly on the bare metal. Using the writeback 
> > > hardware on the display controller allows to get a copy of content from 
> > > all the virtual machines in the system.

> >

> > Although the protocol is called "screencopy", it does not need to be

> > implemented via copying. I don't think anything precludes drm

> > write-back or rendering directly to the client-submitted buffers.

>

> Not sure that's true… With screencopy the client necessarily provides

> buffers that the compositor copies into.



Correction: Andri said that it should be possible to pass client-allocated

buffers to the writeback connector. There might be allocation considerations

to take care of, but it might work fine indeed.