Re: [RFC] Plane color pipeline KMS uAPI
On Tue, May 09, 2023 at 08:22:30PM +, Simon Ser wrote: > On Tuesday, May 9th, 2023 at 21:53, Dave Airlie wrote: > > > There are also other vendor side effects to having this in userspace. > > > > Will the library have a loader? > > Will it allow proprietary plugins? > > Will it allow proprietary reimplementations? > > What will happen when a vendor wants distros to ship their > > proprietary fork of said library? > > > > How would NVIDIA integrate this with their proprietary stack? > > Since all color operations exposed by KMS are standard, the library > would just be a simple one: no loader, no plugin, no proprietary pieces, > etc. > There might be pipelines/color-ops only exposed by proprietary out of tree drivers; the operation types and semantics should ideally be defined upstream, but the code paths would in practice be vendor specific, potentially without any upstream driver using them. It should be clear whether an implementation that makes such a pipeline work is in scope for the upstream library. The same applies to the kernel; it must be clear whether pipeline elements that potentially will only be exposed by out of tree drivers will be acceptable upstream, at least as documented operations. Jonas
Re: [RFC] Plane color pipeline KMS uAPI
On 05/05, Joshua Ashton wrote:
> Some corrections and replies inline.
>
> On Fri, 5 May 2023 at 12:42, Pekka Paalanen wrote:
> >
> > On Thu, 04 May 2023 15:22:59 +
> > Simon Ser wrote:
> >
> > > Hi all,
> > >
> > > The goal of this RFC is to expose a generic KMS uAPI to configure the
> > > color
> > > pipeline before blending, ie. after a pixel is tapped from a plane's
> > > framebuffer and before it's blended with other planes. With this new uAPI
> > > we
> > > aim to reduce the battery life impact of color management and HDR on
> > > mobile
> > > devices, to improve performance and to decrease latency by skipping
> > > composition on the 3D engine. This proposal is the result of discussions
> > > at
> > > the Red Hat HDR hackfest [1] which took place a few days ago. Engineers
> > > familiar with the AMD, Intel and NVIDIA hardware have participated in the
> > > discussion.
> >
> > Hi Simon,
> >
> > this is an excellent write-up, thank you!
> >
> > Harry's question about what constitutes UAPI is a good one for danvet.
> >
> > I don't really have much to add here, a couple inline comments. I think
> > this could work.
> >
> > >
> > > This proposal takes a prescriptive approach instead of a descriptive
> > > approach.
> > > Drivers describe the available hardware blocks in terms of low-level
> > > mathematical operations, then user-space configures each block. We decided
> > > against a descriptive approach where user-space would provide a high-level
> > > description of the colorspace and other parameters: we want to give more
> > > control and flexibility to user-space, e.g. to be able to replicate
> > > exactly the
> > > color pipeline with shaders and switch between shaders and KMS pipelines
> > > seamlessly, and to avoid forcing user-space into a particular color
> > > management
> > > policy.
> > >
> > > We've decided against mirroring the existing CRTC properties
> > > DEGAMMA_LUT/CTM/GAMMA_LUT onto KMS planes. Indeed, the color management
> > > pipeline can significantly differ between vendors and this approach cannot
> > > accurately abstract all hardware. In particular, the availability,
> > > ordering and
> > > capabilities of hardware blocks is different on each display engine. So,
> > > we've
> > > decided to go for a highly detailed hardware capability discovery.
> > >
> > > This new uAPI should not be in conflict with existing standard KMS
> > > properties,
> > > since there are none which control the pre-blending color pipeline at the
> > > moment. It does conflict with any vendor-specific properties like
> > > NV_INPUT_COLORSPACE or the patches on the mailing list adding AMD-specific
> > > properties. Drivers will need to either reject atomic commits configuring
> > > both
> > > uAPIs, or alternatively we could add a DRM client cap which hides the
> > > vendor
> > > properties and shows the new generic properties when enabled.
> > >
> > > To use this uAPI, first user-space needs to discover hardware
> > > capabilities via
> > > KMS objects and properties, then user-space can configure the hardware
> > > via an
> > > atomic commit. This works similarly to the existing KMS uAPI, e.g. planes.
> > >
> > > Our proposal introduces a new "color_pipeline" plane property, and a new
> > > KMS
> > > object type, "COLOROP" (short for color operation). The "color_pipeline"
> > > plane
> > > property is an enum, each enum entry represents a color pipeline
> > > supported by
> > > the hardware. The special zero entry indicates that the pipeline is in
> > > "bypass"/"no-op" mode. For instance, the following plane properties
> > > describe a
> > > primary plane with 2 supported pipelines but currently configured in
> > > bypass
> > > mode:
> > >
> > > Plane 10
> > > ├─ "type": immutable enum {Overlay, Primary, Cursor} = Primary
> > > ├─ …
> > > └─ "color_pipeline": enum {0, 42, 52} = 0
> > >
> > > The non-zero entries describe color pipelines as a linked list of COLOROP
> > > KMS
> > > objects. The entry value is an object ID pointing to the head of the
> > > linked
> > > list (the first operation in the color pipeline).
> > >
> > > The new COLOROP objects also expose a number of KMS properties. Each has a
> > > type, a reference to the next COLOROP object in the linked list, and other
> > > type-specific properties. Here is an example for a 1D LUT operation:
> > >
> > > Color operation 42
> > > ├─ "type": enum {Bypass, 1D curve} = 1D curve
> > > ├─ "1d_curve_type": enum {LUT, sRGB, PQ, BT.709, HLG, …} = LUT
> > > ├─ "lut_size": immutable range = 4096
> > > ├─ "lut_data": blob
> > > └─ "next": immutable color operation ID = 43
> > >
> > > To configure this hardware block, user-space can fill a KMS blob with
> > > 4096 u32
> > > entries, then set "lut_data" to the blob ID. Other color operation types
> > > might
> > > have different properties.
> > >
> > > Here is another example with a 3D LUT:
> > >
> > > Color operation 42
> > > ├
Re: [RFC] Plane color pipeline KMS uAPI
On Tue, 09 May 2023 20:22:30 + Simon Ser wrote: > On Tuesday, May 9th, 2023 at 21:53, Dave Airlie wrote: > > > There are also other vendor side effects to having this in userspace. > > > > Will the library have a loader? > > Will it allow proprietary plugins? > > Will it allow proprietary reimplementations? > > What will happen when a vendor wants distros to ship their > > proprietary fork of said library? > > > > How would NVIDIA integrate this with their proprietary stack? > > Since all color operations exposed by KMS are standard, the library > would just be a simple one: no loader, no plugin, no proprietary pieces, > etc. Hi, that's certainly the long term goal, and *if* Linux software can in any way guide hardware design, then I believe it is an achievable goal. I understand "standard" as something that is widely implemented in various hardware rather than only "well-defined and documented and free to implement in any hardware if its vendor cared". However, like I mentioned in my other reply to Steven, I expect there will be a time period when each hardware has custom processing blocks no other hardware (same or different vendor) has. I might not call them outright proprietary though, because in order have them exposed via UAPI, the mathematical model of the processing block must be documented with its UAPI. This means there cannot be secrets on what the hardware does, which means there cannot be a requirement for secret sauce in userspace either. I wonder if we can also require new COLOROP elements to be freely implementable by anyone anywhere in any way one wants? Or do kernel maintainers just need to NAK proposals for elements that might not be that free? Anything that is driver-chosen or automatic can also be proprietary, because today's KMS UAPI rules do not require documenting how automatic features work, e.g. the existing YUV-to-RGB conversion. Hardware could have whatever wild skin tone improvement algorithms hidden in there for example. In this new proposal, there cannot be undocumented behaviour. Dave, if we went with a descriptive UAPI model, everything behind it could be proprietary and secret. That's not open in the least. On Wed, 10 May 2023 at 00:31, Harry Wentland wrote: > > I am debating whether we need to be serious about a userspace library > (or maybe a user-mode driver) to provide an abstraction from the > descriptive to the prescriptive model. HW vendors need a way to provide > timely support for new HW generations without requiring updates to a > large number of compositors. Drivers can always map old COLOROP elements to new style hardware blocks if they can achieve the same mathematical operation up to whatever precision was promised before. I think that should be the main form of supporting hardware evolution. Then also add new alternative COLOROP elements that can better utilize the hardware block. Naturally that means that COLOROP elements must be designed to be somewhat generic to have a reasonable life time. They cannot be extremely tightly married to the hardware implementation that might cease to exist in the very next hardware revision. Let's say some vendor has a hardware block that does a series of operations in an optimized fashion, perhaps with hardwired constants. This is exposed as a custom COLOROP element. The next hardware revision no longer has this block, but it has a bunch of new blocks that can produce the exact same result. The driver for this hardware can expose two different pipelines: one using the old COLOROP element, and another using a bunch of other COLOROP elements which exposes the new flexibility of the hardware design better. If userspace chooses the former pipeline, the driver just programs the bunch of blocks to behave accordingly. Hopefully the other COLOROP elements will be more standard than the old element. Over time, I hope this causes an evolution where hardware implements only the most standard COLOROP elements, and special-case compound elements will eventually fall out of use over the decades. Thanks, pq pgpUy4CmvGeOE.pgp Description: OpenPGP digital signature
Re: [RFC] Plane color pipeline KMS uAPI
On Wed, 10 May 2023 09:59:21 +0200 Jonas Ådahl wrote: > On Tue, May 09, 2023 at 08:22:30PM +, Simon Ser wrote: > > On Tuesday, May 9th, 2023 at 21:53, Dave Airlie wrote: > > > > > There are also other vendor side effects to having this in userspace. > > > > > > Will the library have a loader? > > > Will it allow proprietary plugins? > > > Will it allow proprietary reimplementations? > > > What will happen when a vendor wants distros to ship their > > > proprietary fork of said library? > > > > > > How would NVIDIA integrate this with their proprietary stack? > > > > Since all color operations exposed by KMS are standard, the library > > would just be a simple one: no loader, no plugin, no proprietary pieces, > > etc. > > > > There might be pipelines/color-ops only exposed by proprietary out of > tree drivers; the operation types and semantics should ideally be > defined upstream, but the code paths would in practice be vendor > specific, potentially without any upstream driver using them. It should > be clear whether an implementation that makes such a pipeline work is in > scope for the upstream library. > > The same applies to the kernel; it must be clear whether pipeline > elements that potentially will only be exposed by out of tree drivers > will be acceptable upstream, at least as documented operations. In my opinion, a COLOROP element definition can be accepted in the upstream kernel documentation only if there is also an upstream driver implementing it. It does not need to be a "direct" hardware implementation, it could also be the upstream driver mapping the COLOROP to whatever hardware block or block chain it has. For the userspace library I don't know. I am puzzled whether people want to allow proprietary components or deny them. Thanks, pq pgph6jVOIdejR.pgp Description: OpenPGP digital signature
[ANNOUNCE] weston 11.0.93
Hi all, This is the RC1 release for Weston 12.0.0. Full commit history below. Marius Vlad (1): build: bump to version 11.0.93 for the RC1 release Philipp Zabel (1): libweston: consolidate 'Using GL/Pixman renderer' log message git tag: 11.0.93 https://gitlab.freedesktop.org/wayland/weston/-/releases/11.0.93/downloads/weston-11.0.93.tar.xz SHA256: df2c84abbf7cc448176ac4b7f2556c03d35989930556cb326c5aa44759534336 weston-11.0.93.tar.xz SHA512: 65ca25cac2ea230d345a9e6489676fde59e620d910f7213858207b00621ae9416f24b5ea50868673d6de438e0a341800a4171c453b093a8f511077991cd80d7a weston-11.0.93.tar.xz PGP: https://gitlab.freedesktop.org/wayland/weston/-/releases/11.0.93/downloads/weston-11.0.93.tar.xz.sig
