On 05/03/2016 09:53 AM, Daniel Stone wrote:
Hi James,
On 3 May 2016 at 17:07, James Jones <[email protected]> wrote:
On 04/29/2016 03:07 PM, Daniel Stone wrote:
With new Wayland protocol, patches to all Wayland compositors to send
proper
hints to clients using this protocol, improvements to GBM, and updates to
both of these when new GPU architectures introduced new requirements,
what
you describe could do anything streams can do. However, then the problem
will have been solved only in the context of top-of-tree Wayland and
Weston.
This doesn't require explicit/new compositor interaction at all.
Extensions can be done within the gbm/EGL bundle itself (via
EGL_WL_bind_wayland_display), so you're only changing one DSO (or DSO
bundle), and the API usage there today does seem to stand up. Given
that the protocol is private - I'm certainly not advocating for a
DRI2-style all-things-to-all-hardware standard protocol to communicate
this - and that it's localised in a vendor bundle, it seems completely
widely applicable to me. As someone who's writing this from
Mutter/Wayland/GBM, I'm certainly not interested in Weston-only
solutions.
No, the necessary extensions can not be contained within the binding. There
is not enough information within the driver layer alone. Something needs to
tell the driver when the configuration changes (E.g., the consumer of a
wayland surface switches from a texture to a plane) and what the new
configuration is. This would trigger the protocol notifications &
subsequent optimization within the driver. By the nature of their API,
streams would require the compositor to take action on such configuration
changes, and streams can discover the new configuration. Something
equivalent would be required to make this work in the GBM+wl_drm/EGL case.
I don't think this is the case. As I went through with Andy, we
_already_ have intent expressed in the GBM case, in the exact same way
that EGLStreams does: consider gbm_bo_import as equivalent for
attaching to an EGLOutput(Layer) consumer, and EGLImage import +
TargetTexture2D as equivalent for attaching a gltexture consumer.
"Will be used for display on device X" is not sufficient information, as
Daniel Vetter outlined.
This
is the exact same proxy for intent to display, and in fact the GBM
approach is slightly more flexible, because it allows you to both do
direct scanout as well as GPU composition (e.g. if you're
capturing/streaming at the same time as display).
Again though, without stream-retargeting, this is not something which
exists in Streams today, and doing so is going to require more
extensions: more code in your driver, more code in every
implementation. GBM today, for all its faults, does not require
further API extension to make this work.
Agreed. We're working on similar flexibility for streams via an
EGLSwitch muxing extension. As mentioned above, GBM would require API
extensions and driver updates to reach the expressiveness of streams as
well though.
Further, as a driver vendor, the idea of requiring even in-driver
platform-specific modifications for this sounds undesirable. If it was
something that could be contained entirely within GBM, that would be
interesting. However, distributing the architecture-specific code
throughout the window-system specific code in the driver means a lot more
maintenance burden in a world with X, Chrome OS, Wayland, and several
others.
This would hold true if Streams was a perfect encapsulation, but I
don't really see how doing so adds any burden over layering the
winsys/platform layer over Streams in the first place. I mean, you've
written Wayland bindings for Streams in the first place ... how would
this be too much different? Even if the protocol is designed to be the
perfect transport for Streams, you _still_ need transport bindings to
your target protocol.
We wrote the wayland protocol as an example of what is possible using
streams, and we intend to open-source it. Presumably window-system
authors would write the protocol for other windowing systems. Further,
since streams would encapsulate all the device-specific stuff, the
protocol library wouldn't require as much maintenance as a
driver-specific protocol library.
In a world with only Wayland, yes, we'd be doing slightly more work to
bootstrap streams support than we would to support GBM+wayland.
However, other windowing systems and stream use cases exist.
What streams exposes is intended to lower the amount of stuff hidden in
drivers, not increase it. Streams is a generic swapchain mechanism
exposed to any user, whereas we would need to write something
proprietary (maybe open source, maybe closed source, but NVIDIA-specific
none the less) for each window system to get equivalent performance if
we pushed the abstraction to a lower level.
Certainly there are, but then again, there are far more usecases than
EGL. Looking at media playback, Vulkan, etc, where you don't have EGL
yet need to solve the same problems.
EGLStreams, Vulkan swapchains, and (for example) VDPAU presentation queues
are all varying levels of abstraction on top of the same thing within the
driver: a presentation engine or buffer queue, depending on whether the
target is a physical output or a compositor. These API-level components can
be hooked up to eachother as long as the lower-level details are fully
contained within the driver abstraction. A Vulkan swapchain can be
internally implemented as an EGLStream producer, for example. In fact,
Vulkan swapchains borrow many ideas directly and indirectly from EGLStream.
Indeed, I noted the similarity, but primarily for the device_swapchain
extension.
I agree, and I'm not arguing this to be on the application or
compositor side either. I believe the GBM and HWC suggestions are
entirely doable, and further that these problems will need to be
solved outside EGL anyway, for the other usecases. My worry - quite
aside from how vendors who struggle to produce a conformant EGL 1.4
implementation today will ever implement the complexity of Streams,
though this isn't your problem - is that EGL is really the wrong place
to be solving this.
Could you elaborate on what the other usecases are? If you mean the
Vulkan/media playback cases mentioned above, then I don't see what is
fundamentally wrong about using EGL as a backend within the window system
for those. If a Vulkan application needs to display on an EGL+GLES-based
Wayland compositor, there will be some point where a transition is made from
Vulkan -> EGL+GLES regardless.
Media falls down because currently there is no zerocopy binding from
either hardware or software media decode engines. Perhaps not the case
on your hardware, unusually blessed with a great deal of memory
bandwidth, but a great many devices physically cannot cope with a
single copy in the pipeline, given the ratio of content size to memory
bandwidth. Doing this in EGL would require a 'draw' step which simply
presented an existing buffer - a step which would unnecessarily
involve the GPU if the pipeline is direct from decode to scanout - or
it would involve having every media engine write their own bindings to
the Streams protocol.
Right. Streams are meant to support lot's of different producers and
consumers.
There are also incredibly exacting timing requirements for media
display, which the Streams model of 'single permanently fixed latency'
does not even come close to achieving. So for that you'd need another
extension, to report actual achieved timings back. Wayland today
fulfills these requirements with the zlinux_dmabuf and
presentation_timing protocols, with the original hardware timings fed
back through KMS.
Would it be reasonable to support such existing extensions while using
streams?
I think it's large enough that it warrants a split of gl-renderer and
compositor-drm, rather than trying to shoehorn them into the same
file. There's going to be quite some complexity hiding between the
synchronise-with-client-event-stream and direct-scanout boxes, that
will push it over the limit of what's tractable. Those files are
already pretty huge and complex.
Would it be better to wait until such complexities arise in future patches
and split the files at that point, or would you prefer we split the backends
now? Perhaps I'm just more optimistic about the complexity, but it seems
like it would be easier to evaluate once that currently-hypothetical portion
of the code exists.
Well, there were quite a few issues with the previous set of patches,
and honestly I'm expecting just resolving those to bring enough
complexity to require a three-way split (common, Streams, and
EGLImage/GBM), let alone the features you're talking about solving
with Streams: direct scanout via retargeting of Streams, etc.
I share the hope, and maybe with the WSI and Streams available, we can
design future window systems and display control APIs towards
something like that. But at the moment, the impedance mismatch between
Streams and the (deliberately very different) Wayland and KMS APIs is
already fairly glaring. The winsys support is absolutely trivial to
write, and with winsys interactions only getting more featureful and
complex, such will the common stream protocol have to be.
If I was starting from the position of the EGL ideal: that everything
is EGL, and the only external interactions are creating native types
for it, then I would surely arrive at the same position as you. But
everything we've seen so far - and again, ChromeOS have taken this to
a much further extent - has been chipping away at EGL, rather than
putting more into it, and this has been for the better.
The direction ChromeOS is taking is even more problematic, and I'd hate to
see it being held up as an example of proper design direction. We spent a
good deal of time working with Google to support ChromeOS and ended up
essentially allowing them to punch through the driver abstraction via very
opaque EGL extensions that no engineer besides the extension authors could
be expected to use correctly, and embed HW-specific knowledge within some
component of ChromeOS, such that it will likely only run optimally on a
single generation of our hardware and will need to be revisited. That's the
type of problem we're trying to avoid here. ChromeOS has made other design
compromises that cost us (and I suspect other vendors) 10-20% performance
across the board to optimize for a very specific use case (I.e., a browser)
and within very constrained schedules. It is not the right direction for
OS<->graphics driver interactions to evolve.
Direction and extent are two very different things: I largely agree
with their direction (less encapsulation inside vendor drivers), and
disagree on the extent to which they've taken it.
That's a very good point. I agree minimal encapsulation is a good goal.
I don't think that's a difference we'll ever resolve though.
I believe thus far we've all tried to focus objectively on specific issues,
proposed solutions for them, and the merits of those solutions. Weston and
the other Wayland compositors I'm aware of are based on EGL at the moment,
so regardless of its merits as an API it doesn't seem problematic purely
from a dependency standpoint to add EGLStream as an option next to the
existing EGLImage and EGLDisplay+GBM paths. I'm certainly willing to
continue discussing the merits of EGL on a broader scale, but does that
discussion need to block the patches proposed here?
Every additional codepath has its cost. Even if you just look at
Mutter and Weston in a vacuum, it seems like it'll be quite the large
patchset(s) by the time it's done, let alone extending it out to all
the other compositors. This is a patchset which will need constant
care and feeding: if it's not tested, it's broken. Right now, there is
only one Streams implementation available, which is in a driver whose
legal status is seen to be sufficiently problematic that it is not
generally distributed by downstreams, which requires a whole set of
external kernel patches to run. So even getting it to run is
non-trivial.
But then we'd have to do that in such a way that it was generally
available, else any refactoring or changes we wanted to do internally
would have to be blocked on testing/review from someone who knew that
backend well enough. Either that, or it would just get broken.
Introducing these codepaths has a very, very, real cost to the
projects you're talking about.
If there were an open source implementation of streams, would that
affect your view?
Agreed, all new code, and especially new significant branches in code
has costs. However, a balance always needs to be struck.
You could quite rightly point to the Raspberry Pi DispManX backend as
an example of the same, and you'd be right. And that's why I'm
extremely enthused about how their new KMS/GBM driver allows us to
nuke the entire backend from orbit, and reduce our testing load by
shifting them to the generic driver.
I hope we can avoid an entirely forked compositor-drm/eglstream (and
especially gl-renderer) for these reasons. The majority of the code is
still common and would be exercised using either path.
Thanks,
-James
Cheers,
Daniel
_______________________________________________
wayland-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/wayland-devel
