RE: [RFC 00/33] Add Support for Plane Color Pipeline
> -Original Message- > From: Harry Wentland > Sent: Wednesday, August 30, 2023 12:56 AM > To: Shankar, Uma ; intel-...@lists.freedesktop.org; > dri- > de...@lists.freedesktop.org > Cc: wayland-devel@lists.freedesktop.org; Ville Syrjala > ; Pekka Paalanen > ; > Simon Ser ; Melissa Wen ; Jonas Ådahl > ; Sebastian Wick ; Shashank > Sharma ; Alexander Goins ; > Naseer Ahmed ; Christopher Braga > > Subject: Re: [RFC 00/33] Add Support for Plane Color Pipeline > > +CC Naseer and Chris, FYI > > See https://patchwork.freedesktop.org/series/123024/ for whole series. > > On 2023-08-29 12:03, Uma Shankar wrote: > > Introduction > > > > > > Modern hardwares have various color processing capabilities both at > > pre-blending and post-blending phases in the color pipeline. > > The current drm implementation exposes only the post-blending color > > hardware blocks. Support for pre-blending hardware is missing. > > There are multiple use cases where pre-blending color hardware will be > > useful: > > a) Linearization of input buffers encoded in various transfer > >functions. > > b) Color Space conversion > > c) Tone mapping > > d) Frame buffer format conversion > > e) Non-linearization of buffer(apply transfer function) > > f) 3D Luts > > > > and other miscellaneous color operations. > > > > Hence, there is a need to expose the color capabilities of the > > hardware to user-space. This will help userspace/middleware to use > > display hardware for color processing and blending instead of doing it > > through GPU shaders. > > > > Thanks, Uma, for sending this. I've been working on something similar but you > beat > me to it. :) Thanks Harry for the useful feedback and overall collaboration on this so far. > > > > Work done so far and relevant references > > > > > > Some implementation is done by Intel and AMD/Igalia to address the same. > > Broad consensus is there that we need a generic API at drm core to > > suffice the use case of various HW vendors. Below are the links > > capturing the discussion so far. > > > > Intel's Plane Color Implementation: > > https://patchwork.freedesktop.org/series/90825/ > > AMD's Plane Color Implementation: > > https://patchwork.freedesktop.org/series/116862/ > > > > > > Hackfest conclusions > > > > > > HDR/Color Hackfest was organised by Redhat to bring all the industry > > stakeholders together and converge on a common uapi expectations. > > Participants from Intel, AMD, Nvidia, Collabora, Redhat, Igalia and > > other prominent user-space developers and maintainers. > > > > Discussions happened on the uapi expectations, opens, nature of > > hardware of multiple hardware vendors, challenges in generalizing the > > same and the path forward. Consensus was made that drm core should > > implement descriptive APIs and not go with prescriptive APIs. DRM core > > should just expose the hardware capabilities; enabling, customizing > > and programming the same should be done by the user-space. Driver should > > just > honor the user space request without doing any operations internally. > > > > Thanks to Simon Ser, for nicely documenting the design consensus and > > an UAPI RFC which can be referred to here: > > > > https://lore.kernel.org/dri-devel/QMers3awXvNCQlyhWdTtsPwkp5ie9bze_hD5 > > > nAccFW7a_RXlWjYB7MoUW_8CKLT2bSQwIXVi5H6VULYIxCdgvryZoAoJnC5lZgyK1Q > Wn48 > > 8=@emersion.fr/ > > > > > > Design considerations > > = > > > > Following are the important aspects taken into account while designing > > the current RFC > > proposal: > > > > 1. Individual HW blocks can be muxed. (e.g. out of two HW blocks only > > one > can be used) > > 2. Position of the HW block in the pipeline can be programmable > > 3. LUTs can be one dimentional or three dimentional > > 4. Number of LUT entries can vary across platforms > > 5. Precision of LUT entries can vary across platforms > > 6. Distribution of LUT entries may vary. e.g Mutli-segmented, > > Logarithmic, > >Piece-Wise Linear(PWL) etc > > 7. There can be parameterized/non-parameterized fixed function HW > blocks. > >e.g. Just a hardware bit, to convert from one color space to another. > > 8. Custom non-standard HW implementation. > > 9. Leaving scope for some vendor defined pescriptive implementation if > required. > > 10.Scope to handle any modification in hardware as technology evolves > > > > The current proposal takes into account the above considerations while > > keeping the implementation as generic as possible leaving scope for future > additions or modifications. > > > > This proposal is also in line to the details mentioned by Simon's RFC > > covering all the aspects discussed in hackfest. > > > > > > Outline of the implementation > > > > > > Each Color Hardware block will be represented by a data structure > > d
RE: [RFC 01/33] drm/doc/rfc: Add RFC document for proposed Plane Color Pipeline
> -Original Message- > From: Harry Wentland > Sent: Wednesday, August 30, 2023 1:10 AM > To: Shankar, Uma ; intel-...@lists.freedesktop.org; > dri- > de...@lists.freedesktop.org > Cc: Borah, Chaitanya Kumar ; wayland- > de...@lists.freedesktop.org > Subject: Re: [RFC 01/33] drm/doc/rfc: Add RFC document for proposed Plane > Color > Pipeline > > > > On 2023-08-29 12:03, Uma Shankar wrote: > > Add the documentation for the new proposed Plane Color Pipeline. > > > > Co-developed-by: Chaitanya Kumar Borah > > > > Signed-off-by: Chaitanya Kumar Borah > > Signed-off-by: Uma Shankar > > --- > > .../gpu/rfc/plane_color_pipeline.rst | 394 ++ > > 1 file changed, 394 insertions(+) > > create mode 100644 Documentation/gpu/rfc/plane_color_pipeline.rst > > > > diff --git a/Documentation/gpu/rfc/plane_color_pipeline.rst > > b/Documentation/gpu/rfc/plane_color_pipeline.rst > > new file mode 100644 > > index ..60ce515b6ea7 > > --- /dev/null > > +++ b/Documentation/gpu/rfc/plane_color_pipeline.rst > > @@ -0,0 +1,394 @@ > > +=== > > + Plane Color Pipeline: A UAPI proposal > > +=== > > + > > +To build the proposal on, lets take the premise of a color pipeline > > +as shown below. > > + > > + +---+ > > + |RAM| > > + | +--++-++-+ | > > + | | FB 1 || FB 2 || FB N| | > > + | +--++-++-+ | > > + +---+ > > + | Plane Color Hardware Block | > > + ++ > > + | +---v-+ +---v---+ +---v--+ | > > + | | Plane A | | Plane B | | Plane N | | > > + | | Pre-CSC | | Pre-CSC | | Pre-CSC | | > > + | +---+-+ +---+---+ +---+--+ | > > + | | | || > > + | +---v-+ +---v---+ +---v--+ | > > + | |Plane A | | Plane B | | Plane N | | > > + | |CSC/CTM | | CSC/CTM | | CSC/CTM | | > > + | +---+-+ ++--+ ++-+ | > > + | | | | | > > + | +---v-+ +v--+ +v-+ | > > + | | Plane A | | Plane B | | Plane N | | > > + | |Post-CSC | | Post-CSC | | Post-CSC | | > > + | +---+-+ ++--+ ++-+ | > > + | | | | | > > + ++ > > ++--v--v---v---| > > +|| || > > +|| Pipe Blender|| > > ++++ > > +||| > > +|+---v--+ | > > +|| Pipe Pre-CSC| | > > +|| | | > > +|+---+--+ | > > +||Pipe Color | > > +|+---v--+ Hardware| > > +|| Pipe CSC/CTM| | > > +|| | | > > +|+---+--+ | > > +||| > > +|+---v--+ | > > +|| Pipe Post-CSC | | > > +|| | | > > +|+---+--+ | > > +||| > > ++-+ > > + | > > + v > > +Pipe Output > > + > > +Each plane consists of the following color blocks > > + * Pre-CSC : This block can used to linearize the input frame buffer data. > > + The linear data then can be further acted on by the following > > + color hardware blocks in the display hardware pipeline > > + > > + * CSC/CTM: Used to program color transformation matrix, this block is used > > +to perform color space conversions like BT2020 to BT709 or > > BT601 > > +etc. This block acts on the linearized data coming from the > > +Pre-CSC HW block. > > + > > + * Post-CSC: This HW block can be used to non-linearize frame buffer data > > to > > + match the sink. Another use case of it could be to perform > > Tone > > + mapping for HDR use-cases. > > + > > +Data from multiple planes will then be fed to pipe/crtc where it will get > > blended. > > +There is a similar set of HW blocks available at pipe/crtc level > > +which acts on this blended data. > > + > > +Below is a sample usecase fo video playback with sub-titles and > > +playback controls > > + > > +┌┐┌─┐ ┌─┐┌─┐ > > +│FB1 ││PRE
Re: [RFC 01/33] drm/doc/rfc: Add RFC document for proposed Plane Color Pipeline
On Wed, 30 Aug 2023 08:59:36 + "Shankar, Uma" wrote: > > -Original Message- > > From: Harry Wentland > > Sent: Wednesday, August 30, 2023 1:10 AM > > To: Shankar, Uma ; intel-...@lists.freedesktop.org; > > dri- > > de...@lists.freedesktop.org > > Cc: Borah, Chaitanya Kumar ; wayland- > > de...@lists.freedesktop.org > > Subject: Re: [RFC 01/33] drm/doc/rfc: Add RFC document for proposed Plane > > Color > > Pipeline > > > > > > > > On 2023-08-29 12:03, Uma Shankar wrote: > > > Add the documentation for the new proposed Plane Color Pipeline. > > > > > > Co-developed-by: Chaitanya Kumar Borah > > > > > > Signed-off-by: Chaitanya Kumar Borah > > > Signed-off-by: Uma Shankar > > > --- > > > .../gpu/rfc/plane_color_pipeline.rst | 394 ++ > > > 1 file changed, 394 insertions(+) > > > create mode 100644 Documentation/gpu/rfc/plane_color_pipeline.rst > > > > > > diff --git a/Documentation/gpu/rfc/plane_color_pipeline.rst > > > b/Documentation/gpu/rfc/plane_color_pipeline.rst > > > new file mode 100644 > > > index ..60ce515b6ea7 > > > --- /dev/null > > > +++ b/Documentation/gpu/rfc/plane_color_pipeline.rst ... Hi Uma! > > > +This color pipeline is then packaged within a blob for the user space > > > +to retrieve it. Details can be found in the next section > > > + > > > > Not sure I like blobs that contain other blob ids. > > It provides flexibility and helps with just one interface to userspace. Its > easy to handle and > manage once we get the hang of it 😊. > > We can clearly define the steps of parsing and data structures to be used > while interpreting > and parsing the blobs. Don't forget extendability. Possibly every single struct will need some kind of versioning, and then it's not simple to parse anymore. Add to that new/old kernel vs. old/new userspace, and it seems a bit nightmarish to design. Also since it's records inside a single blob, it's like a new file format: every record needs a standard header that allows skipping it appropriately if userspace does not understand it, or you need a standard index telling where everything is. Making all records the same size would waste space, and extendability requires variable size. I also would not assume that we can declare a standard set of blocks and that nothing else will be needed. The existing hardware is too diverse for that from what I have understood. I assume that some hardware have blocks unique to them, and they want to at least expose that functionality through a UAPI that allows at least generic enumeration of functionality, even if it needs specialized userspace code to actually make use of. If we go with +struct drm_color_op { + enum color_op_block name; + enum color_op_type type; + u32 blob_id; + u32 private_flags; +}; as in your proposal, I believe it can work (sorry, looking further down, I have assumed too much of 'type'), but the enumerations will become long, and the details blob_id is still specific to 'type'. This is unavoidable, but we can still choose the form between blobs and properties, integers and strings. I have a feeling that introspection will be valuable here, to help people understand what their hardware could do if they had the code to use it. 'name' and 'type' being integers require a translation table to strings before they are readable, and it would be best if the kernel itself provided that translation. I don't understand how 'private_flags' could be useful. There must not be any "hidden" features. Everything a block can be programmed to do via this UAPI must be clearly documented, there cannot be anything private. If two hardware versions of a block differ in a meaningful or significant way, they need to be exposed as different types of blocks. OTOH, if one goes with a (new) DRM object with string named properties model, all that struct versioning and file format hassle has mostly a clear and well-understood solution. We only need to define the rules of how userspace needs to deal with properties or values it does not understand, so that the kernel can keep adding more. Therefore, I'm not yet convinced with the "all blobs" design. > > > +Exposing a color pipeline to user space > > > +=== > > > + > > > +To advertise the available color pipelines, an immutable ENUM > > > +property "GET_COLOR_PIPELINE" is introduced. > > > +This enum property has blob id's as values. With each blob id > > > +representing a distinct color pipeline based on underlying HW > > > +capabilities and their respective combinations. > > > + > > > +The following output of drm_info [1], shows how color pipelines are > > > +visible to userspace. > > > + > > > +├───Plane 0 > > > +│ ├───Object ID: 31 > > > +│ ├───CRTCs: {0} > > > +│ ├───Legacy info > > > +... > > > +│ ├───"GET_COLOR_PIPELINE" (immutable): enum {no color > > > pipeline, > > > +
Re: [RFC 02/33] drm: Add color operation structure
On Tue, 29 Aug 2023 21:33:51 +0530 Uma Shankar wrote: > From: Chaitanya Kumar Borah > > Each Color Hardware block will be represented uniquely > in the color pipeline. Define the structure to represent > the same. > > These color operations will form the building blocks of > a color pipeline which best represents the underlying > Hardware. Color operations can be re-arranged, substracted > or added to create distinct color pipelines to accurately > describe the Hardware blocks present in the display engine. > > Co-developed-by: Uma Shankar > Signed-off-by: Uma Shankar > Signed-off-by: Chaitanya Kumar Borah > --- > include/uapi/drm/drm_mode.h | 72 + > 1 file changed, 72 insertions(+) > > diff --git a/include/uapi/drm/drm_mode.h b/include/uapi/drm/drm_mode.h > index ea1b639bcb28..882479f41745 100644 > --- a/include/uapi/drm/drm_mode.h > +++ b/include/uapi/drm/drm_mode.h > @@ -943,6 +943,78 @@ struct hdr_output_metadata { > }; > }; > > +/** > + * enum color_op_block > + * > + * Enums to identify hardware color blocks. > + * > + * @DRM_CB_PRE_CSC: LUT before the CTM unit > + * @DRM_CB_CSC: CTM hardware supporting 3x3 matrix > + * @DRM_CB_POST_CSC: LUT after the CTM unit > + * @DRM_CB_3D_LUT: LUT hardware with coefficients for all > + * color components > + * @DRM_CB_PRIVATE: Vendor specific hardware unit. Vendor > + * can expose a custom hardware by defining a > + * color operation block with this name as > + * identifier This naming scheme does not seem to work. It assumes a far too rigid pipeline, just like the old KMS property design. What if you have two other operations between PRE_CSC and CSC? What sense do PRE_CSC and POST_CSC make if you don't happen to have a CSC operation? What if a driver put POST_CSC before PRE_CSC in its pipeline? What if your CSC is actually a series of three independent operations, and in addition you have PRE_CSC and POST_CSC? 3D_LUT is an operation category, not a name. The same could be said about private. Given that all these are also UAPI, do we also need protect old userspace from seeing values it does not understand? > + */ > +enum color_op_block { > + DRM_CB_INVAL = -1, > + > + DRM_CB_PRE_CSC = 0, > + DRM_CB_CSC, > + DRM_CB_POST_CSC, > + DRM_CB_3D_LUT, > + > + /* Any new generic hardware block can be updated here */ > + > + /* > + * PRIVATE is kept at 255 to make it future proof and leave > + * scope for any new addition > + */ > + DRM_CB_PRIVATE = 255, > + DRM_CB_MAX = DRM_CB_PRIVATE, > +}; > + > +/** > + * enum color_op_type > + * > + * These enums are to identify the mathematical operation that > + * a hardware block is capable of. > + * @CURVE_1D: It represents a one dimensional lookup table > + * @CURVE_3D: Represents lut value for each color component for 3d lut > capable hardware > + * @MATRIX: It represents co-efficients for a CSC/CTM matrix hardware > + * @FIXED_FUNCTION: To enable and program any custom fixed function hardware > unit > + */ > +enum color_op_type { > + CURVE_1D, > + CURVE_3D, > + MATRIX, > + FIXED_FUNCTION, My assumption was that a color_op_type would clearly and uniquely define the mathematical model of the operation and the UABI structure of the parameter blob. That means we need different values for uniform vs. exponentially vs. programmable distributed 1D LUT, etc. If there is a 1D curve with pre-programmed (fixed and named) curves, we need to enumerate all the curve types somehow. Probably each fixed curve type should not be a different operation type, because that would explode the number of alternative pipelines. A 3D curve in my mind is a function {x,y,z} = f(t), while I suspect you meant a 3D LUT which is a {x,y,z} = f(t,u,v) - a 3-vector field in three dimensional space. A matrix element could be with or without an offset vector I guess. FIXED_FUNCTION would need to be replaced with e.g. your example VENDORXXX_BT602_TO_BT2020 to work. Have I missed something, how did you intend this to work? Thanks, pq > +}; > + > +/** > + * @struct drm_color_op > + * > + * This structure is used to represent the capability of > + * individual color hardware blocks. > + * > + * @name: a standardized enum to identify the color hardware block > + * @type: The type of mathematical operation it can perform > + * @blob_id: Id pointing to a blob containing information about > + * the hardware block which advertizes its capabilities > + * to the userspace. It can be an optional field depending > + * on the members "name" and "type". > + * @private_flags: This can be used to provide vendor specific hints > + * to user space > + */ > +struct drm_color_op { > + enum color_op_block name; > + enum color_op_type type; > + __u32 blob_id; > + __u32 private_flags; > +}; > + > /** > * DRM_MODE_PAGE_FLIP
Re: [RFC 00/33] Add Support for Plane Color Pipeline
On Wed, Aug 30, 2023 at 08:47:37AM +, Shankar, Uma wrote: > > > > -Original Message- > > From: Harry Wentland > > Sent: Wednesday, August 30, 2023 12:56 AM > > To: Shankar, Uma ; intel-...@lists.freedesktop.org; > > dri- > > de...@lists.freedesktop.org > > Cc: wayland-devel@lists.freedesktop.org; Ville Syrjala > > ; Pekka Paalanen > > ; > > Simon Ser ; Melissa Wen ; Jonas Ådahl > > ; Sebastian Wick ; Shashank > > Sharma ; Alexander Goins ; > > Naseer Ahmed ; Christopher Braga > > > > Subject: Re: [RFC 00/33] Add Support for Plane Color Pipeline > > > > +CC Naseer and Chris, FYI > > > > See https://patchwork.freedesktop.org/series/123024/ for whole series. > > > > On 2023-08-29 12:03, Uma Shankar wrote: > > > Introduction > > > > > > > > > Modern hardwares have various color processing capabilities both at > > > pre-blending and post-blending phases in the color pipeline. > > > The current drm implementation exposes only the post-blending color > > > hardware blocks. Support for pre-blending hardware is missing. > > > There are multiple use cases where pre-blending color hardware will be > > > useful: > > > a) Linearization of input buffers encoded in various transfer > > > functions. > > > b) Color Space conversion > > > c) Tone mapping > > > d) Frame buffer format conversion > > > e) Non-linearization of buffer(apply transfer function) > > > f) 3D Luts > > > > > > and other miscellaneous color operations. > > > > > > Hence, there is a need to expose the color capabilities of the > > > hardware to user-space. This will help userspace/middleware to use > > > display hardware for color processing and blending instead of doing it > > > through GPU shaders. > > > > > > > Thanks, Uma, for sending this. I've been working on something similar but > > you beat > > me to it. :) > > Thanks Harry for the useful feedback and overall collaboration on this so far. > > > > > > > Work done so far and relevant references > > > > > > > > > Some implementation is done by Intel and AMD/Igalia to address the same. > > > Broad consensus is there that we need a generic API at drm core to > > > suffice the use case of various HW vendors. Below are the links > > > capturing the discussion so far. > > > > > > Intel's Plane Color Implementation: > > > https://patchwork.freedesktop.org/series/90825/ > > > AMD's Plane Color Implementation: > > > https://patchwork.freedesktop.org/series/116862/ > > > > > > > > > Hackfest conclusions > > > > > > > > > HDR/Color Hackfest was organised by Redhat to bring all the industry > > > stakeholders together and converge on a common uapi expectations. > > > Participants from Intel, AMD, Nvidia, Collabora, Redhat, Igalia and > > > other prominent user-space developers and maintainers. > > > > > > Discussions happened on the uapi expectations, opens, nature of > > > hardware of multiple hardware vendors, challenges in generalizing the > > > same and the path forward. Consensus was made that drm core should > > > implement descriptive APIs and not go with prescriptive APIs. DRM core > > > should just expose the hardware capabilities; enabling, customizing > > > and programming the same should be done by the user-space. Driver should > > > just > > honor the user space request without doing any operations internally. > > > > > > Thanks to Simon Ser, for nicely documenting the design consensus and > > > an UAPI RFC which can be referred to here: > > > > > > https://lore.kernel.org/dri-devel/QMers3awXvNCQlyhWdTtsPwkp5ie9bze_hD5 > > > > > nAccFW7a_RXlWjYB7MoUW_8CKLT2bSQwIXVi5H6VULYIxCdgvryZoAoJnC5lZgyK1Q > > Wn48 > > > 8=@emersion.fr/ > > > > > > > > > Design considerations > > > = > > > > > > Following are the important aspects taken into account while designing > > > the current RFC > > > proposal: > > > > > > 1. Individual HW blocks can be muxed. (e.g. out of two HW blocks only > > > one > > can be used) > > > 2. Position of the HW block in the pipeline can be programmable > > > 3. LUTs can be one dimentional or three dimentional > > > 4. Number of LUT entries can vary across platforms > > > 5. Precision of LUT entries can vary across platforms > > > 6. Distribution of LUT entries may vary. e.g Mutli-segmented, > > > Logarithmic, > > > Piece-Wise Linear(PWL) etc > > > 7. There can be parameterized/non-parameterized fixed function HW > > blocks. > > > e.g. Just a hardware bit, to convert from one color space to another. > > > 8. Custom non-standard HW implementation. > > > 9. Leaving scope for some vendor defined pescriptive implementation if > > required. > > > 10.Scope to handle any modification in hardware as technology evolves > > > > > > The current proposal takes into account the above considerations while > > > keeping the implementation as generic as possible leaving scope for future > > additions or modifications. > > > > > > T
[ANNOUNCE] libei 1.1.0 RC1
libei 1.1.0 RC1 is now available. libei is a library to send Emulated Input (EI) to a matching Emulated Input Server (EIS) which can receive those events with libeis, also part of this project. libei uses GitLab releases, for tarballs please see: https://gitlab.freedesktop.org/libinput/libei/-/releases/1.0.901 Notable fixes in this release include: - ei_device_get_region_at() and its libeis equivalent was added as a helper to obtain a device's region at a given point. this is a library-only change. - ei_region_get_mapping_id() returns the mapping id previously set by the EIS implementation via eis_region_set_mapping_id(). This mapping id can be used to identify external resources that have a relationship with this region, e.g. a pipewire stream that uses the same mapping id. This is a protocol addition and bumps the ei_device protocol to version 2. Note that libei(s) was accidentally leaking some symbols from internal utility functions due to a missing compiler flag. These symbols are no longer there but ABI checkers will complain when updating to this version - those errors can be ignored. The ei-scanner protocol parser now requires Python 3.9. It did already anyway, but now it'll actually fail during meson setup instead of just blowing up with Python exceptions during build. Other than that, a bunch of fixes and tidies, in particular the s390x builds and test runs should now work correctly. Not planning on another RC - unless something major comes up expect 1.1.0 next week somewhen. Peter Hutterer (20): meson.build: bump to use gnu11 over gnu99 Add ei_device_get_region_at() to obtain a region for a point util: fix iobuf_take_fd invalid memmove for multiple fds util: fix iobuf_append_fd OOB when we have too many fds Add a mapping_id to the regions proto: require Python 3.9 test: rework the oeffis dbus tests to be pytest-compatible test: close the sockets on exit util: add a strv_from_mem() helper function test: log the buffers after reading from them test: ensure munit debug messages are visible on failure meson: build libutil with hidden symbols visibility CI: update the commit for the last ABI break test: rename a variable to avoid clashes brei: avoid pointer casts for 32/64 bit values util: switch iobuf to uint8_t to avoid sign issues brei: copy protocol strings out of the protocol buffers test: use munit_assert_uint64 for 64-bit values test: fix 64-bit pointer access for the object ids libei 1.1.0rc1 signature.asc Description: PGP signature