Re: [PATCH v1] dynamic_debug: add support for logs destination
On Wed, 11 Oct 2023 11:42:24 +0200 Daniel Vetter wrote: > On Wed, Oct 11, 2023 at 11:48:16AM +0300, Pekka Paalanen wrote: > > On Tue, 10 Oct 2023 10:06:02 -0600 > > jim.cro...@gmail.com wrote: > > > > > since I name-dropped you all, > > > > Hi everyone, > > > > I'm really happy to see this topic being developed! I've practically > > forgot about it myself, but the need for it has not diminished at all. > > > > I didn't understand much of the conversation, so I'll just reiterate > > what I would use it for, as a Wayland compositor developer. > > > > I added a few more cc's to get better coverage of DRM and Wayland > > compositor developers. > > > > > On Tue, Oct 10, 2023 at 10:01 AM wrote: > > > > > > > > On Mon, Oct 9, 2023 at 4:47 PM Łukasz Bartosik > > > > wrote: > > > > ... > > > > > > > I don't have a real life use case to configure different trace > > > > > instance for each callsite. > > > > > I just tried to be as much flexible as possible. > > > > > > > > > > > > > Ive come around to agree - I looked back at some old threads > > > > (that I was a part of, and barely remembered :-} > > > > > > > > At least Sean Paul, Lyude, Simon Ser, Pekka Paalanen > > > > have expressed a desire for a "flight-recorder" > > > > it'd be hard to say now that 2-3 such buffers would always be enough, > > > > esp as theres a performance reason for having your own. > > > > A Wayland compositor has roughly three important things where the kernel > > debugs might come in handy: > > - input > > - DRM KMS > > - DRM GPU rendering > > > > DRM KMS is the one I've been thinking of in the flight recorder context > > the most, because KMS hardware varies a lot, and there is plenty of > > room for both KMS drivers and KMS userspace to go wrong. The usual > > result is your display doesn't work, so the system is practically > > unusable to the end user. In the wild, the simplest or maybe the only > > way out of that may be a reboot, maybe an automated one (e.g. digital > > signage). In order to debug such problems, we would need both > > compositor logs and the relevant kernel debug messages. > > > > For example, Weston already has a flight recorder framework of its own, > > so we have the compositor debug logs. It would be useful to get the > > selected kernel debug logs in the same place. It could be used for > > automated or semi-manual bug reporting, for example, making the > > administrator or end user life much easier reporting issues. > > > > Since this is usually a production environment, and the Wayland > > compositor runs without root privileges, we need something that works > > with that. We would likely want the kernel debug messages in the > > compositor to combine and order them properly with the compositor debug > > messages. > > > > It's quite likely that developers would like to pick and choose which > > kernel debug messages might be interesting enough to record, to avoid > > excessive log flooding. The flight recorder in Weston is fixed size to > > avoid running out of memory or disk space. I can also see that Weston > > could have debugging options that affect which kernel debug messages it > > subscribes to. We can have a reasonable default setup that allows us to > > pinpoint the problem area and figure out most problems, and if needed, > > we could ask the administrator pass another debug option to Weston. It > > helps if there is just one place to configure everything about the > > compositor. > > > > This implies that it would be really nice to have userspace subscriber > > specific debug message streams from the kernel, or a good way to filter > > the messages we want. A Wayland compositor would not be interested in > > file system or wireless debugs for example, but another system > > component might be. There is also a security aspect of which component is > > allowed to see which messages in case they could contain anything > > sensitive (input debug could contain typed passwords). > > > > Configuring the kernel debug message selection for our debug message > > stream can and probably should require elevated privileges, and we can > > likely solve that in userspace with a daemon or such, to allow the > > Wayland compositor to run as a regular user. > > > > Thinking of desktop systems, and especially physically multi-seat systems: > > - there can be multiple different Wayland compositors running simultaneously > > - each of them may want debug messages only from a specific DRM KMS > > device instance, and not from others > > - each of them may have a different idea of which debug messages are > > important > > - because DRM KMS leasing is a thing, different compositor instances > > could be using the same DRM KMS device instance simultaneously; since > > this is specific to DRM KMS, and it should be harmless to get a > > little too much DRM KMS debug (that is, from the whole device instead > > of just the leased parts), it may not be worth to consider sp
Re: [PATCH v1] dynamic_debug: add support for logs destination
On Thu, Oct 12, 2023 at 11:55:48AM +0300, Pekka Paalanen wrote: > On Wed, 11 Oct 2023 11:42:24 +0200 > Daniel Vetter wrote: > > > On Wed, Oct 11, 2023 at 11:48:16AM +0300, Pekka Paalanen wrote: > > > On Tue, 10 Oct 2023 10:06:02 -0600 > > > jim.cro...@gmail.com wrote: > > > > > > > since I name-dropped you all, > > > > > > Hi everyone, > > > > > > I'm really happy to see this topic being developed! I've practically > > > forgot about it myself, but the need for it has not diminished at all. > > > > > > I didn't understand much of the conversation, so I'll just reiterate > > > what I would use it for, as a Wayland compositor developer. > > > > > > I added a few more cc's to get better coverage of DRM and Wayland > > > compositor developers. > > > > > > > On Tue, Oct 10, 2023 at 10:01 AM wrote: > > > > > > > > > > On Mon, Oct 9, 2023 at 4:47 PM Łukasz Bartosik > > > > > wrote: > > > > > > ... > > > > > > > > > I don't have a real life use case to configure different trace > > > > > > instance for each callsite. > > > > > > I just tried to be as much flexible as possible. > > > > > > > > > > > > > > > > Ive come around to agree - I looked back at some old threads > > > > > (that I was a part of, and barely remembered :-} > > > > > > > > > > At least Sean Paul, Lyude, Simon Ser, Pekka Paalanen > > > > > have expressed a desire for a "flight-recorder" > > > > > it'd be hard to say now that 2-3 such buffers would always be enough, > > > > > esp as theres a performance reason for having your own. > > > > > > A Wayland compositor has roughly three important things where the kernel > > > debugs might come in handy: > > > - input > > > - DRM KMS > > > - DRM GPU rendering > > > > > > DRM KMS is the one I've been thinking of in the flight recorder context > > > the most, because KMS hardware varies a lot, and there is plenty of > > > room for both KMS drivers and KMS userspace to go wrong. The usual > > > result is your display doesn't work, so the system is practically > > > unusable to the end user. In the wild, the simplest or maybe the only > > > way out of that may be a reboot, maybe an automated one (e.g. digital > > > signage). In order to debug such problems, we would need both > > > compositor logs and the relevant kernel debug messages. > > > > > > For example, Weston already has a flight recorder framework of its own, > > > so we have the compositor debug logs. It would be useful to get the > > > selected kernel debug logs in the same place. It could be used for > > > automated or semi-manual bug reporting, for example, making the > > > administrator or end user life much easier reporting issues. > > > > > > Since this is usually a production environment, and the Wayland > > > compositor runs without root privileges, we need something that works > > > with that. We would likely want the kernel debug messages in the > > > compositor to combine and order them properly with the compositor debug > > > messages. > > > > > > It's quite likely that developers would like to pick and choose which > > > kernel debug messages might be interesting enough to record, to avoid > > > excessive log flooding. The flight recorder in Weston is fixed size to > > > avoid running out of memory or disk space. I can also see that Weston > > > could have debugging options that affect which kernel debug messages it > > > subscribes to. We can have a reasonable default setup that allows us to > > > pinpoint the problem area and figure out most problems, and if needed, > > > we could ask the administrator pass another debug option to Weston. It > > > helps if there is just one place to configure everything about the > > > compositor. > > > > > > This implies that it would be really nice to have userspace subscriber > > > specific debug message streams from the kernel, or a good way to filter > > > the messages we want. A Wayland compositor would not be interested in > > > file system or wireless debugs for example, but another system > > > component might be. There is also a security aspect of which component is > > > allowed to see which messages in case they could contain anything > > > sensitive (input debug could contain typed passwords). > > > > > > Configuring the kernel debug message selection for our debug message > > > stream can and probably should require elevated privileges, and we can > > > likely solve that in userspace with a daemon or such, to allow the > > > Wayland compositor to run as a regular user. > > > > > > Thinking of desktop systems, and especially physically multi-seat systems: > > > - there can be multiple different Wayland compositors running > > > simultaneously > > > - each of them may want debug messages only from a specific DRM KMS > > > device instance, and not from others > > > - each of them may have a different idea of which debug messages are > > > important > > > - because DRM KMS leasing is a thing, different compositor instances > > > could be using
Re: [PATCH v1] dynamic_debug: add support for logs destination
On Thu, 12 Oct 2023 11:53:52 +0200 Daniel Vetter wrote: > On Thu, Oct 12, 2023 at 11:55:48AM +0300, Pekka Paalanen wrote: > > On Wed, 11 Oct 2023 11:42:24 +0200 > > Daniel Vetter wrote: > > > > > On Wed, Oct 11, 2023 at 11:48:16AM +0300, Pekka Paalanen wrote: ... > > > > - all selections tailored separately for each userspace subscriber > > > > (- per open device file description selection of messages) > > > > > > Again this feels like a userspace problem. Sessions could register what > > > kind of info they need for their session, and something like journald can > > > figure out how to record it all. > > > > Only if the kernel actually attaches all the required information to > > the debug messages *in machine readable form* so that userspace > > actually can do the filtering. And that makes *that* information UABI. > > Maybe that's fine? I wouldn't know. > > Well if you configure the filters to go into separate ringbuffers for each > session (or whatever you want to split) it also becomes uapi. It's a different UAPI: filter configuration vs. message structure. I don't mind which it is, I just suspect one is easier to maintain and extend than the other. > Also I'd say that for the first cut just getting the logs out on demand > should be good enough, multi-gpu (or multi-compositor) systems are a step > further. We can figure those out when we get there. This reminds me of what you recently said in IRC about a very different topic: swick[m], tell this past me roughly 10 years ago, would have been easy to add into the design back when there was no driver code yet I just want to mention today everything I can see as useful. It's up to the people doing the actual work to decide what they include and how. Thanks, pq pgpFa4cSNlIVv.pgp Description: OpenPGP digital signature
Re: [PATCH v1] dynamic_debug: add support for logs destination
On Thu, Oct 12, 2023 at 01:39:44PM +0300, Pekka Paalanen wrote: > On Thu, 12 Oct 2023 11:53:52 +0200 > Daniel Vetter wrote: > > > On Thu, Oct 12, 2023 at 11:55:48AM +0300, Pekka Paalanen wrote: > > > On Wed, 11 Oct 2023 11:42:24 +0200 > > > Daniel Vetter wrote: > > > > > > > On Wed, Oct 11, 2023 at 11:48:16AM +0300, Pekka Paalanen wrote: > > ... > > > > > > - all selections tailored separately for each userspace subscriber > > > > > (- per open device file description selection of messages) > > > > > > > > Again this feels like a userspace problem. Sessions could register what > > > > kind of info they need for their session, and something like journald > > > > can > > > > figure out how to record it all. > > > > > > Only if the kernel actually attaches all the required information to > > > the debug messages *in machine readable form* so that userspace > > > actually can do the filtering. And that makes *that* information UABI. > > > Maybe that's fine? I wouldn't know. > > > > Well if you configure the filters to go into separate ringbuffers for each > > session (or whatever you want to split) it also becomes uapi. > > It's a different UAPI: filter configuration vs. message structure. I > don't mind which it is, I just suspect one is easier to maintain and > extend than the other. > > > Also I'd say that for the first cut just getting the logs out on demand > > should be good enough, multi-gpu (or multi-compositor) systems are a step > > further. We can figure those out when we get there. > > This reminds me of what you recently said in IRC about a very different > topic: > >swick[m], tell this past me roughly 10 years ago, would > have been easy to add into the design back when there was no > driver code yet > > I just want to mention today everything I can see as useful. It's up to > the people doing the actual work to decide what they include and how. I actually pondered this a bit more today, and I think even with hindsight the atomic design we ended up with was probably rather close to optimal. Sure there's a bunch of things that would have been nice to include, but another very hard requirement of atomic was that it's feasible to convert current drivers over to it. And I think going full free-standing state structures with unlimited (at least at the design level) queue depth would have been a bridge too far. The hacks and conversion helpers are all gone by now, but "you can just peek at the object struct to get your state" was a huge help in reducing the conversion churn. But it definitely resulted in a big price we're still paying. tldr I don't think getting somewhere useful, even if somewhat deficient, is bad. -Sima -- Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch
Re: [PATCH v1] dynamic_debug: add support for logs destination
> If you want the kernel to keep separate flight recorders I guess we could > add that, but I don't think it currently exists for the dyndbg stuff at > least. Maybe a flight recorder v2 feature, once the basics are in. > dyndbg has +pwrites to syslog +T would separately independently write the same to global trace This would allow graceful switchover to tracefs, without removing logging from dmesg, where most folks (and any monitor tools) would expect it. Lukas (iiuc) wants to steer each site to just 1 destination. Or maybe (in addition to +p > syslog) one trace destination, either global via events, or a separate tracebuf Im ambivalent, but thinking the smooth rollover from syslog to trace might be worth having to ease migration / weaning off syslog. And we have a 4 byte hole in struct _ddebug we could just use. Unless the align 8 is optional on 32-bits, I think we're never gonna close the hole anywhere. is align 8 a generic expression of an architectural simplifying constraint ? or a need for 1-7 ptr offsets ? > > That's my idea of it. It is interesting to see how far the requirements > > can be reasonably realised. > > I think aside from the "make it available directly to unpriviledged > userspace" everything sounds reasonable and doable. > > More on the process side of things, I think Jim is very much looking for > acks and tested-by by people who are interested in better drm logging > infra. That should help that things are moving in a direction that's > actually useful, even when it's not yet entirely complete. > yes, please. Now posted at https://lore.kernel.org/lkml/20231012172137.3286566-1-jim.cro...@gmail.com/T/#t Lukas, I managed to miss your email in the send phase. please consider yourself a direct recipient :-) thanks everyone > Cheers, Sima > -- > Daniel Vetter > Software Engineer, Intel Corporation > http://blog.ffwll.ch
RE: [RFC 01/33] drm/doc/rfc: Add RFC document for proposed Plane Color Pipeline
> -Original Message- > From: Christopher Braga > Sent: Friday, September 8, 2023 1:39 AM > To: Shankar, Uma ; intel-...@lists.freedesktop.org; > dri- > de...@lists.freedesktop.org > Cc: Borah, Chaitanya Kumar ; wayland- > de...@lists.freedesktop.org > Subject: Re: [RFC 01/33] drm/doc/rfc: Add RFC document for proposed Plane > Color Pipeline > > > > On 8/29/2023 12:03 PM, Uma Shankar wrote: > > Add the documentation for the new proposed Plane Color Pipeline. > > > > Co-developed-by: Chaitanya Kumar Borah > > > > Signed-off-by: Chaitanya Kumar Borah > > Signed-off-by: Uma Shankar > > --- > > .../gpu/rfc/plane_color_pipeline.rst | 394 ++ > > 1 file changed, 394 insertions(+) > > create mode 100644 Documentation/gpu/rfc/plane_color_pipeline.rst > > > > diff --git a/Documentation/gpu/rfc/plane_color_pipeline.rst > > b/Documentation/gpu/rfc/plane_color_pipeline.rst > > new file mode 100644 > > index ..60ce515b6ea7 > > --- /dev/null > > +++ b/Documentation/gpu/rfc/plane_color_pipeline.rst > > @@ -0,0 +1,394 @@ > > +=== > > + Plane Color Pipeline: A UAPI proposal > > +=== > > + > > +To build the proposal on, lets take the premise of a color pipeline > > +as shown below. > > + > Hi Uma, > Thanks for posting this. A few comments, with some echoing the sentiments of > other commenters on the patch set. Hi Christopher, Thanks for your inputs on the series and my apologies for a late reply. > > + +---+ > > + |RAM| > > + | +--++-++-+ | > > + | | FB 1 || FB 2 || FB N| | > > + | +--++-++-+ | > > + +---+ > > + | Plane Color Hardware Block | > > + ++ > > + | +---v-+ +---v---+ +---v--+ | > > + | | Plane A | | Plane B | | Plane N | | > > + | | Pre-CSC | | Pre-CSC | | Pre-CSC | | > > + | +---+-+ +---+---+ +---+--+ | > > + | | | || > > + | +---v-+ +---v---+ +---v--+ | > > + | |Plane A | | Plane B | | Plane N | | > > + | |CSC/CTM | | CSC/CTM | | CSC/CTM | | > > + | +---+-+ ++--+ ++-+ | > > + | | | | | > > + | +---v-+ +v--+ +v-+ | > > + | | Plane A | | Plane B | | Plane N | | > > + | |Post-CSC | | Post-CSC | | Post-CSC | | > > + | +---+-+ ++--+ ++-+ | > > + | | | | | > > + ++ > > ++--v--v---v---| > > +|| || > > +|| Pipe Blender|| > > ++++ > > +||| > > +|+---v--+ | > > +|| Pipe Pre-CSC| | > > +|| | | > > +|+---+--+ | > > +||Pipe Color | > > +|+---v--+ Hardware| > > +|| Pipe CSC/CTM| | > > +|| | | > > +|+---+--+ | > > +||| > > +|+---v--+ | > > +|| Pipe Post-CSC | | > > +|| | | > > +|+---+--+ | > > +||| > > ++-+ > > + | > > + v > > +Pipe Output > > + > > +Each plane consists of the following color blocks > > + * Pre-CSC : This block can used to linearize the input frame buffer data. > > + The linear data then can be further acted on by the following > > + color hardware blocks in the display hardware pipeline > > + > > + * CSC/CTM: Used to program color transformation matrix, this block is used > > +to perform color space conversions like BT2020 to BT709 or > > BT601 > > +etc. This block acts on the linearized data coming from the > > +Pre-CSC HW block. > > + > > + * Post-CSC: This HW block can be used to non-linearize frame buffer data > > to > > + match the sink. Another use case of it could be to perform > > Tone > > + mapping for HDR use-cases. > > + > > +Data from multiple planes will then be fed to pipe/crtc where it will get > blended. > > +There is a similar set of HW blocks available at pipe/crtc level > > +which acts on this blended d
