Re: [RFC PATCH 01/10] drm/doc/rfc: Describe why prescriptive color pipeline is needed
On Thu, 19 Oct 2023 10:56:29 -0400
Harry Wentland wrote:
> On 2023-09-13 07:29, Pekka Paalanen wrote:
> > On Fri, 8 Sep 2023 11:02:26 -0400
> > Harry Wentland wrote:
> >
> >> Signed-off-by: Harry Wentland
...
> >> +COLOR_PIPELINE Plane Property
> >> +=
> >> +
> >> +Because we don't have existing KMS color properties in the pre-blending
> >> +portion of display pipelines (i.e. on drm_planes) we are introducing
> >> +color pipelines here first. Eventually we'll want to use the same
> >> +concept for the post-blending portion, i.e. drm_crtcs.
> >
> > This paragraph might fit better in a cover letter.
> >
> >> +
> >> +Color Pipelines are created by a driver and advertised via a new
> >> +COLOR_PIPELINE enum property on each plane. Values of the property
> >> +always include '0', which is the default and means all color processing
> >> +is disabled. Additional values will be the object IDs of the first
> >> +drm_colorop in a pipeline. A driver can create and advertise none, one,
> >> +or more possible color pipelines. A DRM client will select a color
> >> +pipeline by setting the COLOR PIPELINE to the respective value.
> >> +
> >> +In the case where drivers have custom support for pre-blending color
> >> +processing those drivers shall reject atomic commits that are trying to
> >> +set both the custom color properties, as well as the COLOR_PIPELINE
> >
> > s/set/use/ because one of them could be carried-over state from
> > previous commits while not literally set in this one.
> >
> >> +property.
> >> +
> >> +An example of a COLOR_PIPELINE property on a plane might look like this::
> >> +
> >> +Plane 10
> >> +├─ "type": immutable enum {Overlay, Primary, Cursor} = Primary
> >> +├─ …
> >> +└─ "color_pipeline": enum {0, 42, 52} = 0
> >
> > Enum values are string names. I presume the intention here is that the
> > strings will never need to be parsed, and the uint64_t is always equal
> > to the string representation, right?
> >
> > That needs a statement here. It differs from all previous uses of
> > enums, and e.g. requires a little bit of new API in libweston's
> > DRM-backend to handle since it has its own enums referring to the
> > string names that get mapped to the uint64_t per owning KMS object.
> >
>
> I'm currently putting the DRM object ID in the "value" and use the
> "name" as a descriptive name.
Would that string name be UAPI? I mean, if userspace hardcodes and
looks for that name, will that keep working? If it's descriptive then I
would assume not, but for every enum existing so far the string name is
UAPI.
> > struct drm_mode_property_enum {
> > __u64 value;
> > char name[DRM_PROP_NAME_LEN];
> > };
>
> This works well in IGT and gives us a nice descriptive name for
> debugging, but I could consider changing this if it'd simplify
> people's lives.
It's nice if we can have a description string for each pipeline, but
according to KMS UAPI conventions so far, userspace would look for the
string name. I'm worried that could backfire to the kernel.
Or, maybe we do want to make the string UAPI and accept that some
userspace might look for it rather than an arrangement of colorops?
Matching colorop sequences is "hard". A developer looking at pipelines,
picking one, and hardcoding its name as a preferred choice would be too
easy. "Works for my cards." IOW, if there is a useful looking string
name, we can be sure that someone will depend on it.
Thanks,
pq
> >> +References
> >> +==
> >> +
> >> +1.
> >> https://lore.kernel.org/dri-devel/QMers3awXvNCQlyhWdTtsPwkp5ie9bze_hD5nAccFW7a_RXlWjYB7MoUW_8CKLT2bSQwIXVi5H6VULYIxCdgvryZoAoJnC5lZgyK1QWn488=@emersion.fr/
> >> \ No newline at end of file
> >
>
pgpLGO3j2JiRX.pgp
Description: OpenPGP digital signature
Re: [RFC PATCH 01/10] drm/doc/rfc: Describe why prescriptive color pipeline is needed
On Thu, 19 Oct 2023 10:56:40 -0400 Harry Wentland wrote: > On 2023-10-10 12:13, Melissa Wen wrote: > > O 09/08, Harry Wentland wrote: > >> Signed-off-by: Harry Wentland ... > > Also, with this new plane API in place, I understand that we will > > already need think on how to deal with the mixing between old drm color > > properties (color encoding and color range) and these new way of setting > > plane color properties. IIUC, Pekka asked a related question about it > > when talking about CRTC automatic RGB->YUV (?) > > > > We'll still need to confirm whether we'll want to deprecate these > existing properties. If we do that we'd want a client prop. Things > should still work without deprecating them, if drivers just pick up > after the initial encoding and range CSC. > > But realistically it might be better to deprecate them and turn them > into explicit colorops. The existing properties would need to be explicitly reflected in the new pipelines anyway, otherwise there would always be doubt at which point of a pipeline the old properties apply, and they might even need to change positions between pipelines. I think it is simply easier to just hide all old color related properties when userspace sets the client-cap to enable pipelines. The problem is to make sure to hide all old properties on all drivers that support the client-cap. As a pipeline must be complete (describe everything that happens to pixel values), it's going to be a flag day per driver. Btw. the plane FB YUV->RGB conversion needs a colorop in every pipeline as well. Maybe it's purely informative and non-configurable, keyed by FB pixel format, but still. We also need a colorop to represent sample filtering, e.g. bilinear, like I think Sebastian may have mentioned in the past. Everything before the sample filter happens "per tap" as Joshua Ashton put it, and everything after it happens on the sample that was computed as a weighted average of the filter tap inputs (texels). There could be colorops other than sample filtering that operate on more than one sample at a time, like blur or sharpness. There could even be colorops that change the image size like adding padding that the following colorop hardware requires, and then yet another colorop that clips that padding away. For an example, see https://lists.freedesktop.org/archives/dri-devel/2023-October/427015.html If that padding and its color can affect the pipeline results of the pixels near the padding (e.g. some convolution is applied with them, which may be the reason why padding is necessary to begin with), then it would be best to model it. Thanks, pq pgpFg0JV7oOAT.pgp Description: OpenPGP digital signature
Re: Wayland Governance Meeting - Oct 30 / Nov 01
On Thu, 19 Oct 2023 23:58:18 +0200 Carlos Garnacho wrote: > Hi, > > I would like to propose a meeting about the color management protocol > [1] after next week (it's late to schedule for next, plus there's > people still likely at XDC). After talking with GIMP maintainers, I > think this topic might welcome some higher bandwidth place to discuss. > The doodle is at [2] to poll for the best day/time, the final date > will be announced by Wednesday 25th. > > The meeting will be held at GNOME servers [3], anyone is free to join > to listen if they want to, anonymously or not. > > The notes for this meeting, along with the ones from the previous meetings can > be found on the wayland-protocols wiki page [4]. > > Cheers, > Carlos > > [1] > https://gitlab.freedesktop.org/wayland/wayland-protocols/-/merge_requests/14 > [2] https://nuudel.digitalcourage.de/QYCvHEj5KR4HADfD Hi, what's the timezone of the proposed times? Europe will switch away from DST on Oct 29, so right before the proposed times. Thanks, pq > [3] https://meet.gnome.org/car-oot-bcf-kt4 > [4] https://gitlab.freedesktop.org/wayland/wayland-protocols/-/wikis/meetings pgpK3pCnchPfJ.pgp Description: OpenPGP digital signature
Re: [RFC PATCH v2 06/17] drm/doc/rfc: Describe why prescriptive color pipeline is needed
Thanks for continuing to work on this! On Thu, Oct 19, 2023 at 05:21:22PM -0400, Harry Wentland wrote: > v2: > - Update colorop visualizations to match reality (Sebastian, Alex Hung) > - Updated wording (Pekka) > - Change BYPASS wording to make it non-mandatory (Sebastian) > - Drop cover-letter-like paragraph from COLOR_PIPELINE Plane Property >section (Pekka) > - Use PQ EOTF instead of its inverse in Pipeline Programming example > (Melissa) > - Add "Driver Implementer's Guide" section (Pekka) > - Add "Driver Forward/Backward Compatibility" section (Sebastian, Pekka) > > Signed-off-by: Harry Wentland > Cc: Ville Syrjala > Cc: Pekka Paalanen > Cc: Simon Ser > Cc: Harry Wentland > Cc: Melissa Wen > Cc: Jonas Ådahl > Cc: Sebastian Wick > Cc: Shashank Sharma > Cc: Alexander Goins > Cc: Joshua Ashton > Cc: Michel Dänzer > Cc: Aleix Pol > Cc: Xaver Hugl > Cc: Victoria Brekenfeld > Cc: Sima > Cc: Uma Shankar > Cc: Naseer Ahmed > Cc: Christopher Braga > Cc: Abhinav Kumar > Cc: Arthur Grillo > Cc: Hector Martin > Cc: Liviu Dudau > Cc: Sasha McIntosh > --- > Documentation/gpu/rfc/color_pipeline.rst | 347 +++ > 1 file changed, 347 insertions(+) > create mode 100644 Documentation/gpu/rfc/color_pipeline.rst > > diff --git a/Documentation/gpu/rfc/color_pipeline.rst > b/Documentation/gpu/rfc/color_pipeline.rst > new file mode 100644 > index ..af5f2ea29116 > --- /dev/null > +++ b/Documentation/gpu/rfc/color_pipeline.rst > @@ -0,0 +1,347 @@ > + > +Linux Color Pipeline API > + > + > +What problem are we solving? > + > + > +We would like to support pre-, and post-blending complex color > +transformations in display controller hardware in order to allow for > +HW-supported HDR use-cases, as well as to provide support to > +color-managed applications, such as video or image editors. > + > +It is possible to support an HDR output on HW supporting the Colorspace > +and HDR Metadata drm_connector properties, but that requires the > +compositor or application to render and compose the content into one > +final buffer intended for display. Doing so is costly. > + > +Most modern display HW offers various 1D LUTs, 3D LUTs, matrices, and other > +operations to support color transformations. These operations are often > +implemented in fixed-function HW and therefore much more power efficient than > +performing similar operations via shaders or CPU. > + > +We would like to make use of this HW functionality to support complex color > +transformations with no, or minimal CPU or shader load. > + > + > +How are other OSes solving this problem? > + > + > +The most widely supported use-cases regard HDR content, whether video or > +gaming. > + > +Most OSes will specify the source content format (color gamut, encoding > transfer > +function, and other metadata, such as max and average light levels) to a > driver. > +Drivers will then program their fixed-function HW accordingly to map from a > +source content buffer's space to a display's space. > + > +When fixed-function HW is not available the compositor will assemble a > shader to > +ask the GPU to perform the transformation from the source content format to > the > +display's format. > + > +A compositor's mapping function and a driver's mapping function are usually > +entirely separate concepts. On OSes where a HW vendor has no insight into > +closed-source compositor code such a vendor will tune their color management > +code to visually match the compositor's. On other OSes, where both mapping > +functions are open to an implementer they will ensure both mappings match. > + > +This results in mapping algorithm lock-in, meaning that no-one alone can > +experiment with or introduce new mapping algorithms and achieve > +consistent results regardless of which implementation path is taken. > + > +Why is Linux different? > +=== > + > +Unlike other OSes, where there is one compositor for one or more drivers, on > +Linux we have a many-to-many relationship. Many compositors; many drivers. > +In addition each compositor vendor or community has their own view of how > +color management should be done. This is what makes Linux so beautiful. > + > +This means that a HW vendor can now no longer tune their driver to one > +compositor, as tuning it to one could make it look fairly different from > +another compositor's color mapping. > + > +We need a better solution. > + > + > +Descriptive API > +=== > + > +An API that describes the source and destination colorspaces is a descriptive > +API. It describes the input and output color spaces but does not describe > +how precisely they should be mapped. Such a mapping includes many minute > +design decision that can greatly affect the look of the final result. > + > +It is not feasible to describe such mapping with enough detail to ens
Re: [RFC PATCH v2 06/17] drm/doc/rfc: Describe why prescriptive color pipeline is needed
On Fri, 20 Oct 2023 16:22:56 +0200 Sebastian Wick wrote: > Thanks for continuing to work on this! > > On Thu, Oct 19, 2023 at 05:21:22PM -0400, Harry Wentland wrote: > > v2: > > - Update colorop visualizations to match reality (Sebastian, Alex Hung) > > - Updated wording (Pekka) > > - Change BYPASS wording to make it non-mandatory (Sebastian) > > - Drop cover-letter-like paragraph from COLOR_PIPELINE Plane Property > >section (Pekka) > > - Use PQ EOTF instead of its inverse in Pipeline Programming example > > (Melissa) > > - Add "Driver Implementer's Guide" section (Pekka) > > - Add "Driver Forward/Backward Compatibility" section (Sebastian, Pekka) > > ... > > +Driver Forward/Backward Compatibility > > += > > + > > +As this is uAPI drivers can't regress color pipelines that have been > > +introduced for a given HW generation. New HW generations are free to > > +abandon color pipelines advertised for previous generations. > > +Nevertheless, it can be beneficial to carry support for existing color > > +pipelines forward as those will likely already have support in DRM > > +clients. > > + > > +Introducing new colorops to a pipeline is fine, as long as they can be > > +disabled or are purely informational. DRM clients implementing support > > +for the pipeline can always skip unknown properties as long as they can > > +be confident that doing so will not cause unexpected results. > > + > > +If a new colorop doesn't fall into one of the above categories > > +(bypassable or informational) the modified pipeline would be unusable > > +for user space. In this case a new pipeline should be defined. > > How can user space detect an informational element? Should we just add a > BYPASS property to informational elements, make it read only and set to > true maybe? Or something more descriptive? Read-only BYPASS set to true would be fine by me, I guess. I think we also need a definition of "informational". Counter-example 1: a colorop that represents a non-configurable YUV<->RGB conversion. Maybe it determines its operation from FB pixel format. It cannot be set to bypass, it cannot be configured, and it will alter color values. Counter-example 2: image size scaling colorop. It might not be configurable, it is controlled by the plane CRTC_* and SRC_* properties. You still need to understand what it does, so you can arrange the scaling to work correctly. (Do not want to scale an image with PQ-encoded values as Josh demonstrated in XDC.) Counter-example 3: image sampling colorop. Averages FB originated color values to produce a color sample. Again do not want to do this with PQ-encoded values. Thanks, pq pgpK1q01CpKCG.pgp Description: OpenPGP digital signature
Re: [RFC PATCH v2 06/17] drm/doc/rfc: Describe why prescriptive color pipeline is needed
On 2023-10-20 10:57, Pekka Paalanen wrote: > On Fri, 20 Oct 2023 16:22:56 +0200 > Sebastian Wick wrote: > >> Thanks for continuing to work on this! >> >> On Thu, Oct 19, 2023 at 05:21:22PM -0400, Harry Wentland wrote: >>> v2: >>> - Update colorop visualizations to match reality (Sebastian, Alex Hung) >>> - Updated wording (Pekka) >>> - Change BYPASS wording to make it non-mandatory (Sebastian) >>> - Drop cover-letter-like paragraph from COLOR_PIPELINE Plane Property >>>section (Pekka) >>> - Use PQ EOTF instead of its inverse in Pipeline Programming example >>> (Melissa) >>> - Add "Driver Implementer's Guide" section (Pekka) >>> - Add "Driver Forward/Backward Compatibility" section (Sebastian, Pekka) >>> > > ... > >>> +Driver Forward/Backward Compatibility >>> += >>> + >>> +As this is uAPI drivers can't regress color pipelines that have been >>> +introduced for a given HW generation. New HW generations are free to >>> +abandon color pipelines advertised for previous generations. >>> +Nevertheless, it can be beneficial to carry support for existing color >>> +pipelines forward as those will likely already have support in DRM >>> +clients. >>> + >>> +Introducing new colorops to a pipeline is fine, as long as they can be >>> +disabled or are purely informational. DRM clients implementing support >>> +for the pipeline can always skip unknown properties as long as they can >>> +be confident that doing so will not cause unexpected results. >>> + >>> +If a new colorop doesn't fall into one of the above categories >>> +(bypassable or informational) the modified pipeline would be unusable >>> +for user space. In this case a new pipeline should be defined. >> >> How can user space detect an informational element? Should we just add a >> BYPASS property to informational elements, make it read only and set to >> true maybe? Or something more descriptive? > > Read-only BYPASS set to true would be fine by me, I guess. > Don't you mean set to false? An informational element will always do something, so it can't be bypassed. > I think we also need a definition of "informational". > > Counter-example 1: a colorop that represents a non-configurable Not sure what's "counter" for these examples? > YUV<->RGB conversion. Maybe it determines its operation from FB pixel > format. It cannot be set to bypass, it cannot be configured, and it > will alter color values. > > Counter-example 2: image size scaling colorop. It might not be > configurable, it is controlled by the plane CRTC_* and SRC_* > properties. You still need to understand what it does, so you can > arrange the scaling to work correctly. (Do not want to scale an image > with PQ-encoded values as Josh demonstrated in XDC.) > IMO the position of the scaling operation is the thing that's important here as the color pipeline won't define scaling properties. > Counter-example 3: image sampling colorop. Averages FB originated color > values to produce a color sample. Again do not want to do this with > PQ-encoded values. > Wouldn't this only happen during a scaling op? Harry > > Thanks, > pq
Re: Wayland Governance Meeting - Oct 30 / Nov 01
Hi! On 10/20/23 12:44, Pekka Paalanen wrote: On Thu, 19 Oct 2023 23:58:18 +0200 Carlos Garnacho wrote: Hi, I would like to propose a meeting about the color management protocol [1] after next week (it's late to schedule for next, plus there's people still likely at XDC). After talking with GIMP maintainers, I think this topic might welcome some higher bandwidth place to discuss. The doodle is at [2] to poll for the best day/time, the final date will be announced by Wednesday 25th. The meeting will be held at GNOME servers [3], anyone is free to join to listen if they want to, anonymously or not. The notes for this meeting, along with the ones from the previous meetings can be found on the wayland-protocols wiki page [4]. Cheers, Carlos [1] https://gitlab.freedesktop.org/wayland/wayland-protocols/-/merge_requests/14 [2] https://nuudel.digitalcourage.de/QYCvHEj5KR4HADfD Hi, what's the timezone of the proposed times? Europe will switch away from DST on Oct 29, so right before the proposed times. I just answered the poll assuming CET since Carlos is also in this timezone. If the times were not given in CET, indeed please tell us. :-) Jehan Thanks, pq [3] https://meet.gnome.org/car-oot-bcf-kt4 [4] https://gitlab.freedesktop.org/wayland/wayland-protocols/-/wikis/meetings -- ZeMarmot open animation film http://film.zemarmot.net Liberapay: https://liberapay.com/ZeMarmot/ Patreon: https://patreon.com/zemarmot Tipeee: https://www.tipeee.com/zemarmot
