Re: [RFC PATCH 01/10] drm/doc/rfc: Describe why prescriptive color pipeline is needed
On Fri, 8 Sep 2023 11:02:26 -0400 Harry Wentland wrote: > Signed-off-by: Harry Wentland > Cc: Ville Syrjala > Cc: Pekka Paalanen > Cc: Simon Ser > Cc: Harry Wentland > Cc: Melissa Wen > Cc: Jonas Ådahl > Cc: Sebastian Wick > Cc: Shashank Sharma > Cc: Alexander Goins > Cc: Joshua Ashton > Cc: Michel Dänzer > Cc: Aleix Pol > Cc: Xaver Hugl > Cc: Victoria Brekenfeld > Cc: Daniel Vetter > Cc: Uma Shankar > Cc: Naseer Ahmed > Cc: Christopher Braga > --- > Documentation/gpu/rfc/color_pipeline.rst | 278 +++ > 1 file changed, 278 insertions(+) > create mode 100644 Documentation/gpu/rfc/color_pipeline.rst Hi Harry, it's really nice to see this! Sebastian started on the backward/forward compatibility, so I'll comment on everything else here, and leave the compatibility for that thread. > diff --git a/Documentation/gpu/rfc/color_pipeline.rst > b/Documentation/gpu/rfc/color_pipeline.rst > new file mode 100644 > index ..bfa4a8f12087 > --- /dev/null > +++ b/Documentation/gpu/rfc/color_pipeline.rst > @@ -0,0 +1,278 @@ > + > +Linux Color Pipeline API > + > + > +What problem are we solving? > + > + > +We would like to support pre-, and post-blending complex color > transformations +in display controller hardware > +in order to allow for HW-supported HDR use-cases, as well as to provide > support > +to color-managed applications, such as video or image editors. > + > +While it is possible to support an HDR output on HW supporting the Colorspace > +and HDR Metadata drm_connector properties that requires the compositor or > +application to render and compose the content into one final buffer intended > for > +display. Doing so is costly. I think a tiny re-wording would make it easier to read: +~While i~*I*t is possible to support an HDR output on HW supporting the Colorspace +and HDR Metadata drm_connector properties*, but* that requires the compositor or +application to render and compose the content into one final buffer intended for +display. Doing so is costly. deletion ~~ addition ** > + > +Most modern display HW offers various 1D LUTs, 3D LUTs, matrices, and other > +operations to support color transformations. These operations are often > +implemented in fixed-function HW and therefore much more power efficient than > +performing similar operations via shaders or CPU. > + > +We would like to make use of this HW functionality to support complex color > +transformations with no, or minimal CPU or shader load. > + > + > +How are other OSes solving this problem? > + > + > +The most widely supported use-cases regard HDR content, whether video or > +gaming. > + > +Most OSes will specify the source content format (color gamut, encoding > transfer > +function, and other metadata, such as max and average light levels) to a > driver. > +Drivers will then program their fixed-function HW accordingly to map from a > +source content buffer's space to a display's space. > + > +When fixed-function HW is not available the compositor will assemble a > shader to > +ask the GPU to perform the transformation from the source content format to > the > +display's format. > + > +A compositor's mapping function and a driver's mapping function are usually > +entirely separate concepts. On OSes where a HW vendor has no insight into > +closed-source compositor code such a vendor will tune their color management > +code to visually match the compositor's. On other OSes, where both mapping > +functions are open to an implementer they will ensure both mappings match. > + I'd add, assuming it's true: This results in mapping algorithm lock-in, meaning that no-one alone can experiment with or introduce new mapping algorithms and achieve consistent results regardless of which implementation path is taken. > + > +Why is Linux different? > +=== > + > +Unlike other OSes, where there is one compositor for one or more drivers, on > +Linux we have a many-to-many relationship. Many compositors; many drivers. > +In addition each compositor vendor or community has their own view of how > +color management should be done. This is what makes Linux so beautiful. > + > +This means that a HW vendor can now no longer tune their driver to one > +compositor, as tuning it to one will almost inevitably make it look very > +different from another compositor's color mapping. This is easy to misunderstand as "all Linux desktops will get your colors different so you cannot have a consistent look in an app". That might trigger a few flamewars, even though it is a true goal in essence. Maybe "almost inevitably" should be worded much more uncertain. Maybe just "could make it look fairly different". Much of color management is about user preferences. Different desktops may have different sets of tunables, like different monitors and TV have different tunables and c
Re: [RFC PATCH 01/10] drm/doc/rfc: Describe why prescriptive color pipeline is needed
On Fri, 8 Sep 2023 16:38:44 -0400 Harry Wentland wrote: > On 2023-09-08 15:30, Sebastian Wick wrote: > > Hey Harry, > > > > Thank you and Simon for this great document. Really happy about it, but > > obviously I've got a few notes and questions inline. > > > > On Fri, Sep 08, 2023 at 11:02:26AM -0400, Harry Wentland wrote: > >> Signed-off-by: Harry Wentland > >> Cc: Ville Syrjala > >> Cc: Pekka Paalanen > >> Cc: Simon Ser > >> Cc: Harry Wentland > >> Cc: Melissa Wen > >> Cc: Jonas Ådahl > >> Cc: Sebastian Wick > >> Cc: Shashank Sharma > >> Cc: Alexander Goins > >> Cc: Joshua Ashton > >> Cc: Michel Dänzer > >> Cc: Aleix Pol > >> Cc: Xaver Hugl > >> Cc: Victoria Brekenfeld > >> Cc: Daniel Vetter > >> Cc: Uma Shankar > >> Cc: Naseer Ahmed > >> Cc: Christopher Braga > >> --- > >> Documentation/gpu/rfc/color_pipeline.rst | 278 +++ > >> 1 file changed, 278 insertions(+) > >> create mode 100644 Documentation/gpu/rfc/color_pipeline.rst > >> > >> diff --git a/Documentation/gpu/rfc/color_pipeline.rst > >> b/Documentation/gpu/rfc/color_pipeline.rst > >> new file mode 100644 > >> index ..bfa4a8f12087 > >> --- /dev/null > >> +++ b/Documentation/gpu/rfc/color_pipeline.rst > >> @@ -0,0 +1,278 @@ > >> + > >> +Linux Color Pipeline API > >> + > >> + > >> +What problem are we solving? > >> + > >> + > >> +We would like to support pre-, and post-blending complex color > >> transformations > >> +in order to allow for HW-supported HDR use-cases, as well as to provide > >> support > >> +to color-managed applications, such as video or image editors. > >> + > >> +While it is possible to support an HDR output on HW supporting the > >> Colorspace > >> +and HDR Metadata drm_connector properties that requires the compositor or > >> +application to render and compose the content into one final buffer > >> intended for > >> +display. Doing so is costly. > >> + > >> +Most modern display HW offers various 1D LUTs, 3D LUTs, matrices, and > >> other > >> +operations to support color transformations. These operations are often > >> +implemented in fixed-function HW and therefore much more power efficient > >> than > >> +performing similar operations via shaders or CPU. > >> + > >> +We would like to make use of this HW functionality to support complex > >> color > >> +transformations with no, or minimal CPU or shader load. > >> + > >> + > >> +How are other OSes solving this problem? > >> + > >> + > >> +The most widely supported use-cases regard HDR content, whether video or > >> +gaming. > >> + > >> +Most OSes will specify the source content format (color gamut, encoding > >> transfer > >> +function, and other metadata, such as max and average light levels) to a > >> driver. > >> +Drivers will then program their fixed-function HW accordingly to map from > >> a > >> +source content buffer's space to a display's space. > >> + > >> +When fixed-function HW is not available the compositor will assemble a > >> shader to > >> +ask the GPU to perform the transformation from the source content format > >> to the > >> +display's format. > >> + > >> +A compositor's mapping function and a driver's mapping function are > >> usually > >> +entirely separate concepts. On OSes where a HW vendor has no insight into > >> +closed-source compositor code such a vendor will tune their color > >> management > >> +code to visually match the compositor's. On other OSes, where both mapping > >> +functions are open to an implementer they will ensure both mappings match. > >> + > >> + > >> +Why is Linux different? > >> +=== > >> + > >> +Unlike other OSes, where there is one compositor for one or more drivers, > >> on > >> +Linux we have a many-to-many relationship. Many compositors; many drivers. > >> +In addition each compositor vendor or community has their own view of how > >> +color management should be done. This is what makes Linux so beautiful. > >> + > >> +This means that a HW vendor can now no longer tune their driver to one > >> +compositor, as tuning it to one will almost inevitably make it look very > >> +different from another compositor's color mapping. > >> + > >> +We need a better solution. > >> + > >> + > >> +Descriptive API > >> +=== > >> + > >> +An API that describes the source and destination colorspaces is a > >> descriptive > >> +API. It describes the input and output color spaces but does not describe > >> +how precisely they should be mapped. Such a mapping includes many minute > >> +design decision that can greatly affect the look of the final result. > >> + > >> +It is not feasible to describe such mapping with enough detail to ensure > >> the > >> +same result from each implementation. In fact, these mappings are a very > >> active > >> +research area. > >> + > >> + > >> +Prescriptive API > >> + > >> + >
Questions about object ID lifetimes
Forgive the long post. Tl;dr: what are the rules of object ID lifetime and reuse in the Wayland protocol? I am attempting to understand the rules of object ID lifetime within the Wayland protocol in order to construct Wayland middleware (similar to some of the tools featured on https://wayland.freedesktop.org/extras.html). I could not find a comprehensive discussion of the details online. If one exists, I would greatly appreciate a link! Middleware tools that wish to decode Wayland messages sent between the compositor and its clients need to maintain an accurate mapping between object ID and object interface (type). This is needed because the wire protocol's message header includes only the target object ID and an opcode that is relative to the object's type (the message header also includes the message length - about which I also have questions - to be pursued later...). The message (request or event) and its argument encoding can only be determined if the object ID -> type and type + opcode -> message mappings are accurately maintained. The type + opcode -> message mapping is static and can be extracted offline from the protocol XML files. Since object IDs can be reused, it is important for the middleware to understand when an ID can be reused and when it cannot be to avoid errors in the ID -> type mapping. Because the Wayland protocol is asynchronous, any message that implies destruction of an object should be acknowledged by the receiver before the destructed object's ID is reused. Fortunately, certain events and requests have been tagged as destructors in the protocol descriptions! Also fortunately, it appears (based on reading the wl_resource_destroy code in wayland-server.c) that for many object IDs, specifically for IDs of objects created by a client request (the ID appears as a new ID arg of a request, and is thus in the client side of the ID range) and for which the client makes a destructor request, the compositor will always send a wl_display::delete_id event (assuming the display_resource still exists for the client, which apparently would only not be the case after the client connection is severed) to acknowledge the destructor request. Any attempt to reuse that ID prior to the wl_display::delete_id event can lead to confusion, and should be avoided. Reuse of the ID after the wl_display::delete_id event should not result in any confusion. [BTW: for the purpose of this discussion, an object is "created" when it is introduced into a protocol message for the first time via a new_id argument. It does not refer to the actual allocation of the object in memory or to its initialization.] However, the other cases are not as easy to identify. The other cases are: 1. an object created by a client request that has destructor events 2. an object created by the compositor It might be true that case 1 does not exist. Is there a general rule against that such cases would never be considered in future expansions of the Wayland protocol? For objects created by the compositor, there are 2 subcases: 2a. objects with only destructor events 2b. objects with destructor requests Again, it might be the case that 2b does not exist, as it is analogous to case 1 above. But, is there a general rule against such future cases as well? Combining 1 and 2b, is there a general rule that says that only the object creator can initiate an object's destruction (unprovoked by the other side of the protocol)? For object IDs created by the compositor and with only destructor events (case 2a), it may be necessary to understand the details of each interface in question to decide when the ID can be reused, as there is no universal destructor acknowledgement request comparable to the wl_display::delete_id event. A requirement to understand the details to that level would make middleware development more difficult. Insert extreme sadness emoji here. Thankfully, it seems that destructor events are themselves acknowledgements of requests for destruction by the client (such as wp_drm_lease_device_v1::released event destructor vs. wp_drm_lease_device_v1::release request), or involve objects with a very limited lifetime and usage, such as callbacks (wp_presentation_feedback, zwp_linux_buffer_release, and zwp_fullscreen_shell_mode_feedback_v1). These limited lifetime/usage objects are created with the knowledge that all messages for them are destructor events, and that they are not involved in any other messages (as targets or arguments). Hence their destruction needs no further acknowledgement because the request for destruction was implied by their creation. The destructor event is the acknowledgement of that request. Is this a general rule: that a destructor event is is always the acknowledgement of a (perhaps implied) destruction request? So there may be two general simple rules that the middleware can follow to maintain a proper ID -> type mapping through ID reuse cycles: 1. reuse of ID is allowed after w
