Re: [PATCH v1] dynamic_debug: add support for logs destination
On Tue, 10 Oct 2023 10:06:02 -0600 jim.cro...@gmail.com wrote: > since I name-dropped you all, Hi everyone, I'm really happy to see this topic being developed! I've practically forgot about it myself, but the need for it has not diminished at all. I didn't understand much of the conversation, so I'll just reiterate what I would use it for, as a Wayland compositor developer. I added a few more cc's to get better coverage of DRM and Wayland compositor developers. > On Tue, Oct 10, 2023 at 10:01 AM wrote: > > > > On Mon, Oct 9, 2023 at 4:47 PM Łukasz Bartosik wrote: ... > > > I don't have a real life use case to configure different trace > > > instance for each callsite. > > > I just tried to be as much flexible as possible. > > > > > > > Ive come around to agree - I looked back at some old threads > > (that I was a part of, and barely remembered :-} > > > > At least Sean Paul, Lyude, Simon Ser, Pekka Paalanen > > have expressed a desire for a "flight-recorder" > > it'd be hard to say now that 2-3 such buffers would always be enough, > > esp as theres a performance reason for having your own. A Wayland compositor has roughly three important things where the kernel debugs might come in handy: - input - DRM KMS - DRM GPU rendering DRM KMS is the one I've been thinking of in the flight recorder context the most, because KMS hardware varies a lot, and there is plenty of room for both KMS drivers and KMS userspace to go wrong. The usual result is your display doesn't work, so the system is practically unusable to the end user. In the wild, the simplest or maybe the only way out of that may be a reboot, maybe an automated one (e.g. digital signage). In order to debug such problems, we would need both compositor logs and the relevant kernel debug messages. For example, Weston already has a flight recorder framework of its own, so we have the compositor debug logs. It would be useful to get the selected kernel debug logs in the same place. It could be used for automated or semi-manual bug reporting, for example, making the administrator or end user life much easier reporting issues. Since this is usually a production environment, and the Wayland compositor runs without root privileges, we need something that works with that. We would likely want the kernel debug messages in the compositor to combine and order them properly with the compositor debug messages. It's quite likely that developers would like to pick and choose which kernel debug messages might be interesting enough to record, to avoid excessive log flooding. The flight recorder in Weston is fixed size to avoid running out of memory or disk space. I can also see that Weston could have debugging options that affect which kernel debug messages it subscribes to. We can have a reasonable default setup that allows us to pinpoint the problem area and figure out most problems, and if needed, we could ask the administrator pass another debug option to Weston. It helps if there is just one place to configure everything about the compositor. This implies that it would be really nice to have userspace subscriber specific debug message streams from the kernel, or a good way to filter the messages we want. A Wayland compositor would not be interested in file system or wireless debugs for example, but another system component might be. There is also a security aspect of which component is allowed to see which messages in case they could contain anything sensitive (input debug could contain typed passwords). Configuring the kernel debug message selection for our debug message stream can and probably should require elevated privileges, and we can likely solve that in userspace with a daemon or such, to allow the Wayland compositor to run as a regular user. Thinking of desktop systems, and especially physically multi-seat systems: - there can be multiple different Wayland compositors running simultaneously - each of them may want debug messages only from a specific DRM KMS device instance, and not from others - each of them may have a different idea of which debug messages are important - because DRM KMS leasing is a thing, different compositor instances could be using the same DRM KMS device instance simultaneously; since this is specific to DRM KMS, and it should be harmless to get a little too much DRM KMS debug (that is, from the whole device instead of just the leased parts), it may not be worth to consider splitting debug message streams this far. If userspace is offered some standardised fields in kernel debug structures, then userspace could do some filtering on its own too, but I guess it would be better to filter at the source and not need that. There is also an anti-goal. The kernel debug message contents are specifically not machine-parsable. I very much do not want to impose debug strings as ABI, they are for human (and AI?) readers only. As a summary, here are the most important requirements first: - usable in production a
Re: [PATCH v1] dynamic_debug: add support for logs destination
On Wed, Oct 11, 2023 at 11:48:16AM +0300, Pekka Paalanen wrote: > On Tue, 10 Oct 2023 10:06:02 -0600 > jim.cro...@gmail.com wrote: > > > since I name-dropped you all, > > Hi everyone, > > I'm really happy to see this topic being developed! I've practically > forgot about it myself, but the need for it has not diminished at all. > > I didn't understand much of the conversation, so I'll just reiterate > what I would use it for, as a Wayland compositor developer. > > I added a few more cc's to get better coverage of DRM and Wayland > compositor developers. > > > On Tue, Oct 10, 2023 at 10:01 AM wrote: > > > > > > On Mon, Oct 9, 2023 at 4:47 PM Łukasz Bartosik wrote: > > > > > ... > > > > > I don't have a real life use case to configure different trace > > > > instance for each callsite. > > > > I just tried to be as much flexible as possible. > > > > > > > > > > Ive come around to agree - I looked back at some old threads > > > (that I was a part of, and barely remembered :-} > > > > > > At least Sean Paul, Lyude, Simon Ser, Pekka Paalanen > > > have expressed a desire for a "flight-recorder" > > > it'd be hard to say now that 2-3 such buffers would always be enough, > > > esp as theres a performance reason for having your own. > > A Wayland compositor has roughly three important things where the kernel > debugs might come in handy: > - input > - DRM KMS > - DRM GPU rendering > > DRM KMS is the one I've been thinking of in the flight recorder context > the most, because KMS hardware varies a lot, and there is plenty of > room for both KMS drivers and KMS userspace to go wrong. The usual > result is your display doesn't work, so the system is practically > unusable to the end user. In the wild, the simplest or maybe the only > way out of that may be a reboot, maybe an automated one (e.g. digital > signage). In order to debug such problems, we would need both > compositor logs and the relevant kernel debug messages. > > For example, Weston already has a flight recorder framework of its own, > so we have the compositor debug logs. It would be useful to get the > selected kernel debug logs in the same place. It could be used for > automated or semi-manual bug reporting, for example, making the > administrator or end user life much easier reporting issues. > > Since this is usually a production environment, and the Wayland > compositor runs without root privileges, we need something that works > with that. We would likely want the kernel debug messages in the > compositor to combine and order them properly with the compositor debug > messages. > > It's quite likely that developers would like to pick and choose which > kernel debug messages might be interesting enough to record, to avoid > excessive log flooding. The flight recorder in Weston is fixed size to > avoid running out of memory or disk space. I can also see that Weston > could have debugging options that affect which kernel debug messages it > subscribes to. We can have a reasonable default setup that allows us to > pinpoint the problem area and figure out most problems, and if needed, > we could ask the administrator pass another debug option to Weston. It > helps if there is just one place to configure everything about the > compositor. > > This implies that it would be really nice to have userspace subscriber > specific debug message streams from the kernel, or a good way to filter > the messages we want. A Wayland compositor would not be interested in > file system or wireless debugs for example, but another system > component might be. There is also a security aspect of which component is > allowed to see which messages in case they could contain anything > sensitive (input debug could contain typed passwords). > > Configuring the kernel debug message selection for our debug message > stream can and probably should require elevated privileges, and we can > likely solve that in userspace with a daemon or such, to allow the > Wayland compositor to run as a regular user. > > Thinking of desktop systems, and especially physically multi-seat systems: > - there can be multiple different Wayland compositors running simultaneously > - each of them may want debug messages only from a specific DRM KMS > device instance, and not from others > - each of them may have a different idea of which debug messages are important > - because DRM KMS leasing is a thing, different compositor instances > could be using the same DRM KMS device instance simultaneously; since > this is specific to DRM KMS, and it should be harmless to get a > little too much DRM KMS debug (that is, from the whole device instead > of just the leased parts), it may not be worth to consider splitting > debug message streams this far. > > If userspace is offered some standardised fields in kernel debug > structures, then userspace could do some filtering on its own too, but I > guess it would be better to filter at the source and not need that. > > There is al
Re: [RFC PATCH 01/10] drm/doc/rfc: Describe why prescriptive color pipeline is needed
On Tue, 10 Oct 2023 15:13:46 -0100
Melissa Wen wrote:
> O 09/08, Harry Wentland wrote:
> > Signed-off-by: Harry Wentland
> > Cc: Ville Syrjala
> > Cc: Pekka Paalanen
> > Cc: Simon Ser
> > Cc: Harry Wentland
> > Cc: Melissa Wen
> > Cc: Jonas Ådahl
> > Cc: Sebastian Wick
> > Cc: Shashank Sharma
> > Cc: Alexander Goins
> > Cc: Joshua Ashton
> > Cc: Michel Dänzer
> > Cc: Aleix Pol
> > Cc: Xaver Hugl
> > Cc: Victoria Brekenfeld
> > Cc: Daniel Vetter
> > Cc: Uma Shankar
> > Cc: Naseer Ahmed
> > Cc: Christopher Braga
> > ---
> > Documentation/gpu/rfc/color_pipeline.rst | 278 +++
> > 1 file changed, 278 insertions(+)
> > create mode 100644 Documentation/gpu/rfc/color_pipeline.rst
> >
> > diff --git a/Documentation/gpu/rfc/color_pipeline.rst
> > b/Documentation/gpu/rfc/color_pipeline.rst
> > new file mode 100644
> > index ..bfa4a8f12087
> > --- /dev/null
> > +++ b/Documentation/gpu/rfc/color_pipeline.rst
...
> > +Color Pipeline Discovery
> > +
> > +
> > +A DRM client wanting color management on a drm_plane will:
> > +
> > +1. Read all drm_colorop objects
> > +2. Get the COLOR_PIPELINE property of the plane
> > +3. iterate all COLOR_PIPELINE enum values
> > +4. for each enum value walk the color pipeline (via the NEXT pointers)
> > + and see if the available color operations are suitable for the
> > + desired color management operations
> > +
> > +An example of chained properties to define an AMD pre-blending color
> > +pipeline might look like this::
>
> Hi Harry,
>
> Thanks for sharing this proposal. Overall I think it's very aligned with
> Simon's description of the generic KMS color API. I think it's a good
> start point and we can refine over iterations. My general questions have
> already been pointed out by Sebastian and Pekka (mainly regarding the
> BYPASS property).
>
> I still have some doubts on how to fit these set of colorops with some
> AMD corners cases as below:
>
> > +
> > +Plane 10
> > +├─ "type": immutable enum {Overlay, Primary, Cursor} = Primary
> > +└─ "color_pipeline": enum {0, 42} = 0
> > +Color operation 42 (input CSC)
> > +├─ "type": enum {Bypass, Matrix} = Matrix
> > +├─ "matrix_data": blob
> > +└─ "next": immutable color operation ID = 43
>
> IIUC, for input CSC, there are currently two possiblities, or we use
> `drm_plane_color_encoding` and `drm_plane_color range` to get
> pre-defined coefficients or we set a custom matrix here, right? If so, I
> think we need some kind of pre-defined matrix option?
>
> Also, with this new plane API in place, I understand that we will
> already need think on how to deal with the mixing between old drm color
> properties (color encoding and color range) and these new way of setting
> plane color properties. IIUC, Pekka asked a related question about it
> when talking about CRTC automatic RGB->YUV (?)
I didn't realize color encoding and color range KMS plane properties
even existed. There is even color space on rockchip!
https://drmdb.emersion.fr/properties?object-type=4008636142
That list has even more conflicts: DEGAMMA_MODE, EOTF, FEATURE,
NV_INPUT_COLORSPACE, SCALING_FILTER, WATERMARK, alpha, GLOBAL_ALPHA,
brightness, colorkey, contrast, and more. I hope most of them are
actually from downstream drivers.
I think they should be forbidden to be used together with the new
pipeline UAPI. Mixing does not work in the long run, it would be
undefined at what position do the old properties apply in a pipeline.
Apparently, we already need a DRM client cap for the new color pipeline
UAPI to hide these legacy things.
This is different from "CRTC automatic RGB->YUV", because the CRTC
thing is chosen silently by the driver and there is nothing after it.
Those old plane properties are explicitly programmed by userspace.
Thanks,
pq
> > +Color operation 43
> > +├─ "type": enum {Scaling} = Scaling
> > +└─ "next": immutable color operation ID = 44
> > +Color operation 44 (DeGamma)
> > +├─ "type": enum {Bypass, 1D curve} = 1D curve
> > +├─ "1d_curve_type": enum {sRGB, PQ, …} = sRGB
> > +└─ "next": immutable color operation ID = 45
> > +Color operation 45 (gamut remap)
> > +├─ "type": enum {Bypass, Matrix} = Matrix
> > +├─ "matrix_data": blob
> > +└─ "next": immutable color operation ID = 46
> > +Color operation 46 (shaper LUT RAM)
> > +├─ "type": enum {Bypass, 1D curve} = 1D curve
> > +├─ "1d_curve_type": enum {LUT} = LUT
> > +├─ "lut_size": immutable range = 4096
> > +├─ "lut_data": blob
> > +└─ "next": immutable color operation ID = 47
>
> For shaper and blend LUT RAM, that the driver supports pre-defined
> curves and custom LUT at the same time but the resulted LUT is a
> combination of those, how to make this behavior clear? Could this
> behavior be described by two colorop in a row: for example, one for
> shaper TF and,just after, another for shaper LUT or would
Re: [PATCH v1] dynamic_debug: add support for logs destination
śr., 11 paź 2023 o 11:42 Daniel Vetter napisał(a): > > On Wed, Oct 11, 2023 at 11:48:16AM +0300, Pekka Paalanen wrote: > > On Tue, 10 Oct 2023 10:06:02 -0600 > > jim.cro...@gmail.com wrote: > > > > > since I name-dropped you all, > > > > Hi everyone, > > > > I'm really happy to see this topic being developed! I've practically > > forgot about it myself, but the need for it has not diminished at all. > > It's good to hear you guys are also interested in this feature. > > I didn't understand much of the conversation, so I'll just reiterate > > what I would use it for, as a Wayland compositor developer. > > > > I added a few more cc's to get better coverage of DRM and Wayland > > compositor developers. > > > > > On Tue, Oct 10, 2023 at 10:01 AM wrote: > > > > > > > > On Mon, Oct 9, 2023 at 4:47 PM Łukasz Bartosik > > > > wrote: > > > > ... > > > > > > > I don't have a real life use case to configure different trace > > > > > instance for each callsite. > > > > > I just tried to be as much flexible as possible. > > > > > > > > > > > > > Ive come around to agree - I looked back at some old threads > > > > (that I was a part of, and barely remembered :-} > > > > > > > > At least Sean Paul, Lyude, Simon Ser, Pekka Paalanen > > > > have expressed a desire for a "flight-recorder" > > > > it'd be hard to say now that 2-3 such buffers would always be enough, > > > > esp as theres a performance reason for having your own. > > > > A Wayland compositor has roughly three important things where the kernel > > debugs might come in handy: > > - input > > - DRM KMS > > - DRM GPU rendering > > > > DRM KMS is the one I've been thinking of in the flight recorder context > > the most, because KMS hardware varies a lot, and there is plenty of > > room for both KMS drivers and KMS userspace to go wrong. The usual > > result is your display doesn't work, so the system is practically > > unusable to the end user. In the wild, the simplest or maybe the only > > way out of that may be a reboot, maybe an automated one (e.g. digital > > signage). In order to debug such problems, we would need both > > compositor logs and the relevant kernel debug messages. > > > > For example, Weston already has a flight recorder framework of its own, > > so we have the compositor debug logs. It would be useful to get the > > selected kernel debug logs in the same place. It could be used for > > automated or semi-manual bug reporting, for example, making the > > administrator or end user life much easier reporting issues. > > > > Since this is usually a production environment, and the Wayland > > compositor runs without root privileges, we need something that works > > with that. We would likely want the kernel debug messages in the > > compositor to combine and order them properly with the compositor debug > > messages. > > > > It's quite likely that developers would like to pick and choose which > > kernel debug messages might be interesting enough to record, to avoid > > excessive log flooding. The flight recorder in Weston is fixed size to > > avoid running out of memory or disk space. I can also see that Weston > > could have debugging options that affect which kernel debug messages it > > subscribes to. We can have a reasonable default setup that allows us to > > pinpoint the problem area and figure out most problems, and if needed, > > we could ask the administrator pass another debug option to Weston. It > > helps if there is just one place to configure everything about the > > compositor. > > > > This implies that it would be really nice to have userspace subscriber > > specific debug message streams from the kernel, or a good way to filter > > the messages we want. A Wayland compositor would not be interested in > > file system or wireless debugs for example, but another system > > component might be. There is also a security aspect of which component is > > allowed to see which messages in case they could contain anything > > sensitive (input debug could contain typed passwords). > > > > Configuring the kernel debug message selection for our debug message > > stream can and probably should require elevated privileges, and we can > > likely solve that in userspace with a daemon or such, to allow the > > Wayland compositor to run as a regular user. > > > > Thinking of desktop systems, and especially physically multi-seat systems: > > - there can be multiple different Wayland compositors running simultaneously > > - each of them may want debug messages only from a specific DRM KMS > > device instance, and not from others > > - each of them may have a different idea of which debug messages are > > important > > - because DRM KMS leasing is a thing, different compositor instances > > could be using the same DRM KMS device instance simultaneously; since > > this is specific to DRM KMS, and it should be harmless to get a > > little too much DRM KMS debug (that is, from the whole device instead > > of just the leased parts), it may n
Re: [RFC PATCH 01/10] drm/doc/rfc: Describe why prescriptive color pipeline is needed
On Wed, Oct 11, 2023 at 12:20 PM Pekka Paalanen wrote:
>
> On Tue, 10 Oct 2023 15:13:46 -0100
> Melissa Wen wrote:
>
> > O 09/08, Harry Wentland wrote:
> > > Signed-off-by: Harry Wentland
> > > Cc: Ville Syrjala
> > > Cc: Pekka Paalanen
> > > Cc: Simon Ser
> > > Cc: Harry Wentland
> > > Cc: Melissa Wen
> > > Cc: Jonas Ådahl
> > > Cc: Sebastian Wick
> > > Cc: Shashank Sharma
> > > Cc: Alexander Goins
> > > Cc: Joshua Ashton
> > > Cc: Michel Dänzer
> > > Cc: Aleix Pol
> > > Cc: Xaver Hugl
> > > Cc: Victoria Brekenfeld
> > > Cc: Daniel Vetter
> > > Cc: Uma Shankar
> > > Cc: Naseer Ahmed
> > > Cc: Christopher Braga
> > > ---
> > > Documentation/gpu/rfc/color_pipeline.rst | 278 +++
> > > 1 file changed, 278 insertions(+)
> > > create mode 100644 Documentation/gpu/rfc/color_pipeline.rst
> > >
> > > diff --git a/Documentation/gpu/rfc/color_pipeline.rst
> > > b/Documentation/gpu/rfc/color_pipeline.rst
> > > new file mode 100644
> > > index ..bfa4a8f12087
> > > --- /dev/null
> > > +++ b/Documentation/gpu/rfc/color_pipeline.rst
>
> ...
>
> > > +Color Pipeline Discovery
> > > +
> > > +
> > > +A DRM client wanting color management on a drm_plane will:
> > > +
> > > +1. Read all drm_colorop objects
> > > +2. Get the COLOR_PIPELINE property of the plane
> > > +3. iterate all COLOR_PIPELINE enum values
> > > +4. for each enum value walk the color pipeline (via the NEXT pointers)
> > > + and see if the available color operations are suitable for the
> > > + desired color management operations
> > > +
> > > +An example of chained properties to define an AMD pre-blending color
> > > +pipeline might look like this::
> >
> > Hi Harry,
> >
> > Thanks for sharing this proposal. Overall I think it's very aligned with
> > Simon's description of the generic KMS color API. I think it's a good
> > start point and we can refine over iterations. My general questions have
> > already been pointed out by Sebastian and Pekka (mainly regarding the
> > BYPASS property).
> >
> > I still have some doubts on how to fit these set of colorops with some
> > AMD corners cases as below:
> >
> > > +
> > > +Plane 10
> > > +├─ "type": immutable enum {Overlay, Primary, Cursor} = Primary
> > > +└─ "color_pipeline": enum {0, 42} = 0
> > > +Color operation 42 (input CSC)
> > > +├─ "type": enum {Bypass, Matrix} = Matrix
> > > +├─ "matrix_data": blob
> > > +└─ "next": immutable color operation ID = 43
> >
> > IIUC, for input CSC, there are currently two possiblities, or we use
> > `drm_plane_color_encoding` and `drm_plane_color range` to get
> > pre-defined coefficients or we set a custom matrix here, right? If so, I
> > think we need some kind of pre-defined matrix option?
Seems reasonable. If they are mutually exclusive you might want to
expose 2 different pipelines for it.
> > Also, with this new plane API in place, I understand that we will
> > already need think on how to deal with the mixing between old drm color
> > properties (color encoding and color range) and these new way of setting
> > plane color properties. IIUC, Pekka asked a related question about it
> > when talking about CRTC automatic RGB->YUV (?)
Indeed, good catch! I listed some of them in my proposal more than one
year ago but completely forgot about them already.
>
> I didn't realize color encoding and color range KMS plane properties
> even existed. There is even color space on rockchip!
>
> https://drmdb.emersion.fr/properties?object-type=4008636142
>
> That list has even more conflicts: DEGAMMA_MODE, EOTF, FEATURE,
> NV_INPUT_COLORSPACE, SCALING_FILTER, WATERMARK, alpha, GLOBAL_ALPHA,
> brightness, colorkey, contrast, and more. I hope most of them are
> actually from downstream drivers.
>
> I think they should be forbidden to be used together with the new
> pipeline UAPI. Mixing does not work in the long run, it would be
> undefined at what position do the old properties apply in a pipeline.
>
> Apparently, we already need a DRM client cap for the new color pipeline
> UAPI to hide these legacy things.
Agreed. We'll need one cap for planes and one in the future for CRTCs then.
>
> This is different from "CRTC automatic RGB->YUV", because the CRTC
> thing is chosen silently by the driver and there is nothing after it.
> Those old plane properties are explicitly programmed by userspace.
>
>
> Thanks,
> pq
>
> > > +Color operation 43
> > > +├─ "type": enum {Scaling} = Scaling
> > > +└─ "next": immutable color operation ID = 44
> > > +Color operation 44 (DeGamma)
> > > +├─ "type": enum {Bypass, 1D curve} = 1D curve
> > > +├─ "1d_curve_type": enum {sRGB, PQ, …} = sRGB
> > > +└─ "next": immutable color operation ID = 45
> > > +Color operation 45 (gamut remap)
> > > +├─ "type": enum {Bypass, Matrix} = Matrix
> > > +├─ "matrix_data": blob
> > > +└─ "next": immutable color operation ID = 46
> > > +Color operation 46 (shaper L
FDBuild Helps You Building Entire Wayland And X11 Graphic Stacks From Source
I recently created the first public release of FDBuild, a tool that I have been using over the last years to build all the things I needed to easily switch between hacking on Mesa, Wayland, XServer, etc. I believe it offers a very convenient workflow for that because you basically only need to remember a single command "fdbuild" and you are able to rebuild many projects or only a single one. You can get the tool easily with pip: https://pypi.org/project/fdbuild/ And you can read more about it in this blog post: https://subdiff.org/blog/2023/fdbuild-01-released
