Re: [RFC PATCH v2 06/17] drm/doc/rfc: Describe why prescriptive color pipeline is needed
On Wed, 25 Oct 2023 15:16:08 -0500 (CDT)
Alex Goins wrote:
> Thank you Harry and all other contributors for your work on this. Responses
> inline -
>
> On Mon, 23 Oct 2023, Pekka Paalanen wrote:
>
> > On Fri, 20 Oct 2023 11:23:28 -0400
> > Harry Wentland wrote:
> >
> > > On 2023-10-20 10:57, Pekka Paalanen wrote:
> > > > On Fri, 20 Oct 2023 16:22:56 +0200
> > > > Sebastian Wick wrote:
> > > >
> > > >> Thanks for continuing to work on this!
> > > >>
> > > >> On Thu, Oct 19, 2023 at 05:21:22PM -0400, Harry Wentland wrote:
> > > >>> v2:
> > > >>> - Update colorop visualizations to match reality (Sebastian, Alex
> > > >>> Hung)
> > > >>> - Updated wording (Pekka)
> > > >>> - Change BYPASS wording to make it non-mandatory (Sebastian)
> > > >>> - Drop cover-letter-like paragraph from COLOR_PIPELINE Plane Property
> > > >>>section (Pekka)
> > > >>> - Use PQ EOTF instead of its inverse in Pipeline Programming example
> > > >>> (Melissa)
> > > >>> - Add "Driver Implementer's Guide" section (Pekka)
> > > >>> - Add "Driver Forward/Backward Compatibility" section (Sebastian,
> > > >>> Pekka)
> > > >
> > > > ...
> > > >
> > > >>> +An example of a drm_colorop object might look like one of these::
> > > >>> +
> > > >>> +/* 1D enumerated curve */
> > > >>> +Color operation 42
> > > >>> +├─ "TYPE": immutable enum {1D enumerated curve, 1D LUT, 3x3
> > > >>> matrix, 3x4 matrix, 3D LUT, etc.} = 1D enumerated curve
> > > >>> +├─ "BYPASS": bool {true, false}
> > > >>> +├─ "CURVE_1D_TYPE": enum {sRGB EOTF, sRGB inverse EOTF, PQ EOTF,
> > > >>> PQ inverse EOTF, …}
> > > >>> +└─ "NEXT": immutable color operation ID = 43
>
> I know these are just examples, but I would also like to suggest the
> possibility
> of an "identity" CURVE_1D_TYPE. BYPASS = true might get different results
> compared to setting an identity in some cases depending on the hardware. See
> below for more on this, RE: implicit format conversions.
>
> Although NVIDIA hardware doesn't use a ROM for enumerated curves, it came up
> in
> offline discussions that it would nonetheless be helpful to expose enumerated
> curves in order to hide the vendor-specific complexities of programming
> segmented LUTs from clients. In that case, we would simply refer to the
> enumerated curve when calculating/choosing segmented LUT entries.
That's a good idea.
> Another thing that came up in offline discussions is that we could use
> multiple
> color operations to program a single operation in hardware. As I understand
> it,
> AMD has a ROM-defined LUT, followed by a custom 4K entry LUT, followed by an
> "HDR Multiplier". On NVIDIA we don't have these as separate hardware stages,
> but
> we could combine them into a singular LUT in software, such that you can
> combine
> e.g. segmented PQ EOTF with night light. One caveat is that you will lose
> precision from the custom LUT where it overlaps with the linear section of the
> enumerated curve, but that is unavoidable and shouldn't be an issue in most
> use-cases.
Indeed.
> Actually, the current examples in the proposal don't include a multiplier
> color
> op, which might be useful. For AMD as above, but also for NVIDIA as the
> following issue arises:
>
> As discussed further below, the NVIDIA "degamma" LUT performs an implicit
> fixed
> point to FP16 conversion. In that conversion, what fixed point 0x maps
> to in floating point varies depending on the source content. If it's SDR
> content, we want the max value in FP16 to be 1.0 (80 nits), subject to a
> potential boost multiplier if we want SDR content to be brighter. If it's HDR
> PQ
> content, we want the max value in FP16 to be 125.0 (10,000 nits). My
> assumption
> is that this is also what AMD's "HDR Multiplier" stage is used for, is that
> correct?
It would be against the UAPI design principles to tag content as HDR or
SDR. What you can do instead is to expose a colorop with a multiplier of
1.0 or 125.0 to match your hardware behaviour, then tell your hardware
that the input is SDR or HDR to get the expected multiplier. You will
never know what the content actually is, anyway.
Of course, if we want to have a arbitrary multiplier colorop that is
somewhat standard, as in, exposed by many drivers to ease userspace
development, you can certainly use any combination of your hardware
features you need to realize the UAPI prescribed mathematical operation.
Since we are talking about floating-point in hardware, a multiplier
does not significantly affect precision.
In order to mathematically define all colorops, I believe it is
necessary to define all colorops in terms of floating-point values (as
in math), even if they operate on fixed-point or integer. By this I
mean that if the input is 8 bpc unsigned integer pixel format for
instance, 0 raw pixel channel value is mapped to 0.0 and 255 is mapped
to 1.0, and the color pipeline starts with [0.0, 1.0], not [0, 255]
domain. We have to agree
Re: [RFC PATCH v2 06/17] drm/doc/rfc: Describe why prescriptive color pipeline is needed
On Thu, Oct 26, 2023 at 11:57:47AM +0300, Pekka Paalanen wrote:
> On Wed, 25 Oct 2023 15:16:08 -0500 (CDT)
> Alex Goins wrote:
>
> > Thank you Harry and all other contributors for your work on this. Responses
> > inline -
> >
> > On Mon, 23 Oct 2023, Pekka Paalanen wrote:
> >
> > > On Fri, 20 Oct 2023 11:23:28 -0400
> > > Harry Wentland wrote:
> > >
> > > > On 2023-10-20 10:57, Pekka Paalanen wrote:
> > > > > On Fri, 20 Oct 2023 16:22:56 +0200
> > > > > Sebastian Wick wrote:
> > > > >
> > > > >> Thanks for continuing to work on this!
> > > > >>
> > > > >> On Thu, Oct 19, 2023 at 05:21:22PM -0400, Harry Wentland wrote:
> > > > >>> v2:
> > > > >>> - Update colorop visualizations to match reality (Sebastian, Alex
> > > > >>> Hung)
> > > > >>> - Updated wording (Pekka)
> > > > >>> - Change BYPASS wording to make it non-mandatory (Sebastian)
> > > > >>> - Drop cover-letter-like paragraph from COLOR_PIPELINE Plane
> > > > >>> Property
> > > > >>>section (Pekka)
> > > > >>> - Use PQ EOTF instead of its inverse in Pipeline Programming
> > > > >>> example (Melissa)
> > > > >>> - Add "Driver Implementer's Guide" section (Pekka)
> > > > >>> - Add "Driver Forward/Backward Compatibility" section (Sebastian,
> > > > >>> Pekka)
> > > > >
> > > > > ...
> > > > >
> > > > >>> +An example of a drm_colorop object might look like one of these::
> > > > >>> +
> > > > >>> +/* 1D enumerated curve */
> > > > >>> +Color operation 42
> > > > >>> +├─ "TYPE": immutable enum {1D enumerated curve, 1D LUT, 3x3
> > > > >>> matrix, 3x4 matrix, 3D LUT, etc.} = 1D enumerated curve
> > > > >>> +├─ "BYPASS": bool {true, false}
> > > > >>> +├─ "CURVE_1D_TYPE": enum {sRGB EOTF, sRGB inverse EOTF, PQ
> > > > >>> EOTF, PQ inverse EOTF, …}
> > > > >>> +└─ "NEXT": immutable color operation ID = 43
> >
> > I know these are just examples, but I would also like to suggest the
> > possibility
> > of an "identity" CURVE_1D_TYPE. BYPASS = true might get different results
> > compared to setting an identity in some cases depending on the hardware. See
> > below for more on this, RE: implicit format conversions.
> >
> > Although NVIDIA hardware doesn't use a ROM for enumerated curves, it came
> > up in
> > offline discussions that it would nonetheless be helpful to expose
> > enumerated
> > curves in order to hide the vendor-specific complexities of programming
> > segmented LUTs from clients. In that case, we would simply refer to the
> > enumerated curve when calculating/choosing segmented LUT entries.
>
> That's a good idea.
>
> > Another thing that came up in offline discussions is that we could use
> > multiple
> > color operations to program a single operation in hardware. As I understand
> > it,
> > AMD has a ROM-defined LUT, followed by a custom 4K entry LUT, followed by an
> > "HDR Multiplier". On NVIDIA we don't have these as separate hardware
> > stages, but
> > we could combine them into a singular LUT in software, such that you can
> > combine
> > e.g. segmented PQ EOTF with night light. One caveat is that you will lose
> > precision from the custom LUT where it overlaps with the linear section of
> > the
> > enumerated curve, but that is unavoidable and shouldn't be an issue in most
> > use-cases.
>
> Indeed.
>
> > Actually, the current examples in the proposal don't include a multiplier
> > color
> > op, which might be useful. For AMD as above, but also for NVIDIA as the
> > following issue arises:
> >
> > As discussed further below, the NVIDIA "degamma" LUT performs an implicit
> > fixed
> > point to FP16 conversion. In that conversion, what fixed point 0x
> > maps
> > to in floating point varies depending on the source content. If it's SDR
> > content, we want the max value in FP16 to be 1.0 (80 nits), subject to a
> > potential boost multiplier if we want SDR content to be brighter. If it's
> > HDR PQ
> > content, we want the max value in FP16 to be 125.0 (10,000 nits). My
> > assumption
> > is that this is also what AMD's "HDR Multiplier" stage is used for, is that
> > correct?
>
> It would be against the UAPI design principles to tag content as HDR or
> SDR. What you can do instead is to expose a colorop with a multiplier of
> 1.0 or 125.0 to match your hardware behaviour, then tell your hardware
> that the input is SDR or HDR to get the expected multiplier. You will
> never know what the content actually is, anyway.
>
> Of course, if we want to have a arbitrary multiplier colorop that is
> somewhat standard, as in, exposed by many drivers to ease userspace
> development, you can certainly use any combination of your hardware
> features you need to realize the UAPI prescribed mathematical operation.
>
> Since we are talking about floating-point in hardware, a multiplier
> does not significantly affect precision.
>
> In order to mathematically define all colorops, I believe it is
> necessary to define all colorops in terms of floating-point
Re: [RFC PATCH v2 06/17] drm/doc/rfc: Describe why prescriptive color pipeline is needed
On Thu, 26 Oct 2023, Sebastian Wick wrote:
> On Thu, Oct 26, 2023 at 11:57:47AM +0300, Pekka Paalanen wrote:
> > On Wed, 25 Oct 2023 15:16:08 -0500 (CDT)
> > Alex Goins wrote:
> >
> > > Thank you Harry and all other contributors for your work on this.
> > > Responses
> > > inline -
> > >
> > > On Mon, 23 Oct 2023, Pekka Paalanen wrote:
> > >
> > > > On Fri, 20 Oct 2023 11:23:28 -0400
> > > > Harry Wentland wrote:
> > > >
> > > > > On 2023-10-20 10:57, Pekka Paalanen wrote:
> > > > > > On Fri, 20 Oct 2023 16:22:56 +0200
> > > > > > Sebastian Wick wrote:
> > > > > >
> > > > > >> Thanks for continuing to work on this!
> > > > > >>
> > > > > >> On Thu, Oct 19, 2023 at 05:21:22PM -0400, Harry Wentland wrote:
> > > > > >>> v2:
> > > > > >>> - Update colorop visualizations to match reality (Sebastian,
> > > > > >>> Alex Hung)
> > > > > >>> - Updated wording (Pekka)
> > > > > >>> - Change BYPASS wording to make it non-mandatory (Sebastian)
> > > > > >>> - Drop cover-letter-like paragraph from COLOR_PIPELINE Plane
> > > > > >>> Property
> > > > > >>>section (Pekka)
> > > > > >>> - Use PQ EOTF instead of its inverse in Pipeline Programming
> > > > > >>> example (Melissa)
> > > > > >>> - Add "Driver Implementer's Guide" section (Pekka)
> > > > > >>> - Add "Driver Forward/Backward Compatibility" section
> > > > > >>> (Sebastian, Pekka)
> > > > > >
> > > > > > ...
> > > > > >
> > > > > >>> +An example of a drm_colorop object might look like one of these::
> > > > > >>> +
> > > > > >>> +/* 1D enumerated curve */
> > > > > >>> +Color operation 42
> > > > > >>> +├─ "TYPE": immutable enum {1D enumerated curve, 1D LUT, 3x3
> > > > > >>> matrix, 3x4 matrix, 3D LUT, etc.} = 1D enumerated curve
> > > > > >>> +├─ "BYPASS": bool {true, false}
> > > > > >>> +├─ "CURVE_1D_TYPE": enum {sRGB EOTF, sRGB inverse EOTF, PQ
> > > > > >>> EOTF, PQ inverse EOTF, …}
> > > > > >>> +└─ "NEXT": immutable color operation ID = 43
> > >
> > > I know these are just examples, but I would also like to suggest the
> > > possibility
> > > of an "identity" CURVE_1D_TYPE. BYPASS = true might get different results
> > > compared to setting an identity in some cases depending on the hardware.
> > > See
> > > below for more on this, RE: implicit format conversions.
> > >
> > > Although NVIDIA hardware doesn't use a ROM for enumerated curves, it came
> > > up in
> > > offline discussions that it would nonetheless be helpful to expose
> > > enumerated
> > > curves in order to hide the vendor-specific complexities of programming
> > > segmented LUTs from clients. In that case, we would simply refer to the
> > > enumerated curve when calculating/choosing segmented LUT entries.
> >
> > That's a good idea.
> >
> > > Another thing that came up in offline discussions is that we could use
> > > multiple
> > > color operations to program a single operation in hardware. As I
> > > understand it,
> > > AMD has a ROM-defined LUT, followed by a custom 4K entry LUT, followed by
> > > an
> > > "HDR Multiplier". On NVIDIA we don't have these as separate hardware
> > > stages, but
> > > we could combine them into a singular LUT in software, such that you can
> > > combine
> > > e.g. segmented PQ EOTF with night light. One caveat is that you will lose
> > > precision from the custom LUT where it overlaps with the linear section
> > > of the
> > > enumerated curve, but that is unavoidable and shouldn't be an issue in
> > > most
> > > use-cases.
> >
> > Indeed.
> >
> > > Actually, the current examples in the proposal don't include a multiplier
> > > color
> > > op, which might be useful. For AMD as above, but also for NVIDIA as the
> > > following issue arises:
> > >
> > > As discussed further below, the NVIDIA "degamma" LUT performs an implicit
> > > fixed
> > > point to FP16 conversion. In that conversion, what fixed point 0x
> > > maps
> > > to in floating point varies depending on the source content. If it's SDR
> > > content, we want the max value in FP16 to be 1.0 (80 nits), subject to a
> > > potential boost multiplier if we want SDR content to be brighter. If it's
> > > HDR PQ
> > > content, we want the max value in FP16 to be 125.0 (10,000 nits). My
> > > assumption
> > > is that this is also what AMD's "HDR Multiplier" stage is used for, is
> > > that
> > > correct?
> >
> > It would be against the UAPI design principles to tag content as HDR or
> > SDR. What you can do instead is to expose a colorop with a multiplier of
> > 1.0 or 125.0 to match your hardware behaviour, then tell your hardware
> > that the input is SDR or HDR to get the expected multiplier. You will
> > never know what the content actually is, anyway.
Right, I didn't mean to suggest that we should tag content as HDR or SDR in the
UAPI, just relating to the end result in the pipe, ultimately it would be
determined by the multiplier color op.
> >
> > Of course, if we want to have a arbitrary multiplier colorop that is
> > somewha
