[Mesa-dev] [RFC 0/2] GuC submission / DRM scheduler integration plan + new uAPI
Subject and patches say it all. v2: Address comments, patches has details of changes Signed-off-by: Matthew Brost Matthew Brost (2): drm/doc/rfc: i915 GuC submission / DRM scheduler drm/doc/rfc: i915 new parallel submission uAPI plan Documentation/gpu/rfc/i915_parallel_execbuf.h | 144 ++ Documentation/gpu/rfc/i915_scheduler.rst | 136 + Documentation/gpu/rfc/index.rst | 4 + 3 files changed, 284 insertions(+) create mode 100644 Documentation/gpu/rfc/i915_parallel_execbuf.h create mode 100644 Documentation/gpu/rfc/i915_scheduler.rst -- 2.28.0 ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
[Mesa-dev] [RFC 1/2] drm/doc/rfc: i915 GuC submission / DRM scheduler
Add entry for i915 GuC submission / DRM scheduler integration plan. Follow up patch with details of new parallel submission uAPI to come. v2: (Daniel Vetter) - Expand explaination of why bonding isn't supported for GuC submission - CC some of the DRM scheduler maintainers - Add priority inheritance / boosting use case - Add reasoning for removing in order assumptions (Daniel Stone) - Add links to priority spec Cc: Christian König Cc: Luben Tuikov Cc: Alex Deucher Cc: Steven Price Cc: Jon Bloomfield Cc: Jason Ekstrand Cc: Dave Airlie Cc: Daniel Vetter Cc: Jason Ekstrand Cc: dri-de...@lists.freedesktop.org Signed-off-by: Matthew Brost --- Documentation/gpu/rfc/i915_scheduler.rst | 85 Documentation/gpu/rfc/index.rst | 4 ++ 2 files changed, 89 insertions(+) create mode 100644 Documentation/gpu/rfc/i915_scheduler.rst diff --git a/Documentation/gpu/rfc/i915_scheduler.rst b/Documentation/gpu/rfc/i915_scheduler.rst new file mode 100644 index ..7faa46cde088 --- /dev/null +++ b/Documentation/gpu/rfc/i915_scheduler.rst @@ -0,0 +1,85 @@ += +I915 GuC Submission/DRM Scheduler Section += + +Upstream plan += +For upstream the overall plan for landing GuC submission and integrating the +i915 with the DRM scheduler is: + +* Merge basic GuC submission + * Basic submission support for all gen11+ platforms + * Not enabled by default on any current platforms but can be enabled via + modparam enable_guc + * Lots of rework will need to be done to integrate with DRM scheduler so + no need to nit pick everything in the code, it just should be + functional, no major coding style / layering errors, and not regress + execlists + * Update IGTs / selftests as needed to work with GuC submission + * Enable CI on supported platforms for a baseline + * Rework / get CI heathly for GuC submission in place as needed +* Merge new parallel submission uAPI + * Bonding uAPI completely incompatible with GuC submission, plus it has + severe design issues in general, which is why we want to retire it no + matter what + * New uAPI adds I915_CONTEXT_ENGINES_EXT_PARALLEL context setup step + which configures a slot with N contexts + * After I915_CONTEXT_ENGINES_EXT_PARALLEL a user can submit N batches to + a slot in a single execbuf IOCTL and the batches run on the GPU in + paralllel + * Initially only for GuC submission but execlists can be supported if + needed +* Convert the i915 to use the DRM scheduler + * GuC submission backend fully integrated with DRM scheduler + * All request queues removed from backend (e.g. all backpressure + handled in DRM scheduler) + * Resets / cancels hook in DRM scheduler + * Watchdog hooks into DRM scheduler + * Lots of complexity of the GuC backend can be pulled out once + integrated with DRM scheduler (e.g. state machine gets + simplier, locking gets simplier, etc...) + * Execlist backend will do the minimum required to hook in the DRM + scheduler so it can live next to the fully integrated GuC backend + * Legacy interface + * Features like timeslicing / preemption / virtual engines would + be difficult to integrate with the DRM scheduler and these + features are not required for GuC submission as the GuC does + these things for us + * ROI low on fully integrating into DRM scheduler + * Fully integrating would add lots of complexity to DRM + scheduler + * Port i915 priority inheritance / boosting feature in DRM scheduler + * Used for i915 page flip, may be useful to other DRM drivers as + well + * Will be an optional feature in the DRM scheduler + * Remove in-order completion assumptions from DRM scheduler + * Even when using the DRM scheduler the backends will handle + preemption, timeslicing, etc... so it is possible for jobs to + finish out of order + * Pull out i915 priority levels and use DRM priority levels + * Optimize DRM scheduler as needed + +New uAPI for basic GuC submission += +No major changes are required to the uAPI for basic GuC submission. The only +change is a new scheduler attribute: I915_SCHEDULER_CAP_STATIC_PRIORITY_MAP. +This attribute indicates the 2k i915 user priority levels are statically mapped +into 3 levels as follows: + +* -1k to -1 Low priority +* 0 Medium priority +* 1 to 1k High priority + +This is needed because the GuC only has 4 priority bands. The highest priority +band is reserved wit
[Mesa-dev] [RFC 2/2] drm/doc/rfc: i915 new parallel submission uAPI plan
Add entry fpr i915 new parallel submission uAPI plan.
v2:
(Daniel Vetter):
- Expand logical order explaination
- Add dummy header
- Only allow N BBs in execbuf IOCTL
- Configure parallel submission per slot not per gem context
Cc: Tvrtko Ursulin
Cc: Tony Ye
CC: Carl Zhang
Cc: Daniel Vetter
Cc: Jason Ekstrand
Signed-off-by: Matthew Brost
---
Documentation/gpu/rfc/i915_parallel_execbuf.h | 144 ++
Documentation/gpu/rfc/i915_scheduler.rst | 53 ++-
2 files changed, 196 insertions(+), 1 deletion(-)
create mode 100644 Documentation/gpu/rfc/i915_parallel_execbuf.h
diff --git a/Documentation/gpu/rfc/i915_parallel_execbuf.h
b/Documentation/gpu/rfc/i915_parallel_execbuf.h
new file mode 100644
index ..8c64b983ccad
--- /dev/null
+++ b/Documentation/gpu/rfc/i915_parallel_execbuf.h
@@ -0,0 +1,144 @@
+#define I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT 2 /* see
i915_context_engines_parallel_submit */
+
+/*
+ * i915_context_engines_parallel_submit:
+ *
+ * Setup a slot to allow multiple BBs to be submitted in a single execbuf
IOCTL.
+ * Those BBs will then be scheduled to run on the GPU in parallel. Multiple
+ * hardware contexts are created internally in the i915 run these BBs. Once a
+ * slot is configured for N BBs only N BBs can be submitted in each execbuf
+ * IOCTL and this is implict behavior (e.g. the user doesn't tell the execbuf
+ * IOCTL there are N BBs, the execbuf IOCTL know how many BBs there are based
on
+ * the slots configuration).
+ *
+ * Their are two currently defined ways to control the placement of the
+ * hardware contexts on physical engines: default behavior (no flags) and
+ * I915_PARALLEL_IMPLICT_BONDS (a flag). More flags may be added the in the
+ * future as new hardware / use cases arise. Details of how to use this
+ * interface below above the flags.
+ *
+ * Returns -EINVAL if hardware context placement configuration invalid or if
the
+ * placement configuration isn't supported on the platform / submission
+ * interface.
+ * Returns -ENODEV if extension isn't supported on the platform / submission
+ * inteface.
+ */
+struct i915_context_engines_parallel_submit {
+ struct i915_user_extension base;
+
+ __u16 engine_index; /* slot for parallel engine */
+ __u16 width;/* number of contexts per parallel engine */
+ __u16 num_siblings; /* number of siblings per context */
+ __u16 mbz16;
+/*
+ * Default placement behvavior (currently unsupported):
+ *
+ * Rather than restricting parallel submission to a single class with a
+ * logically contiguous placement (I915_PARALLEL_IMPLICT_BONDS), add a mode
that
+ * enables parallel submission across multiple engine classes. In this case
each
+ * context's logical engine mask indicates where that context can placed. It is
+ * implied in this mode that all contexts have mutual exclusive placement (e.g.
+ * if one context is running CS0 no other contexts can run on CS0).
+ *
+ * Example 1 pseudo code:
+ * CSX[Y] = engine class X, logical instance Y
+ * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE
+ * set_engines(INVALID)
+ * set_parallel(engine_index=0, width=2, num_siblings=2,
+ * engines=CS0[0],CS0[1],CS1[0],CS1[1])
+ *
+ * Results in the following valid placements:
+ * CS0[0], CS1[0]
+ * CS0[0], CS1[1]
+ * CS0[1], CS1[0]
+ * CS0[1], CS1[1]
+ *
+ * This can also be though of as 2 virtual engines:
+ * VE[0] = CS0[0], CS0[1]
+ * VE[1] = CS1[0], CS1[1]
+ *
+ * Example 2 pseudo code:
+ * CS[X] = generic engine of same class, logical instance X
+ * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE
+ * set_engines(INVALID)
+ * set_parallel(engine_index=0, width=2, num_siblings=3,
+ * engines=CS[0],CS[1],CS[2],CS[0],CS[1],CS[2])
+ *
+ * Results in the following valid placements:
+ * CS[0], CS[1]
+ * CS[0], CS[2]
+ * CS[1], CS[0]
+ * CS[1], CS[2]
+ * CS[2], CS[0]
+ * CS[2], CS[1]
+ *
+ *
+ * This can also be though of as 2 virtual engines:
+ * VE[0] = CS[0], CS[1], CS[2]
+ * VE[1] = CS[0], CS[1], CS[2]
+
+ * This enables a use case where all engines are created equally, we don't care
+ * where they are scheduled, we just want a certain number of resources, for
+ * those resources to be scheduled in parallel, and possibly across multiple
+ * engine classes.
+ */
+
+/*
+ * I915_PARALLEL_IMPLICT_BONDS - Create implict bonds between each context.
+ * Each context must have the same number sibling and bonds are implictly
create
+ * of the siblings.
+ *
+ * All of the below examples are in logical space.
+ *
+ * Example 1 pseudo code:
+ * CS[X] = generic engine of same class, logical instance X
+ * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE
+ * set_engines(INVALID)
+ * set_parallel(engine_index=0, width=2, num_siblings=1,
+ * engines=CS[0],CS[1], flags=I915_PARALLEL_IMPLICT_BONDS)
+ *
+ * Results in the following valid pl
Re: [Mesa-dev] [RFC 2/2] drm/doc/rfc: i915 new parallel submission uAPI plan
On Wed, May 19, 2021 at 01:45:39PM +0200, Christian König wrote:
> Oh, yeah we call that gang submit on the AMD side.
>
> Had already some internal discussions how to implement this, but so far
> couldn't figure out how to cleanly introduce that into the DRM scheduler.
>
> Can you briefly describe in a few words how that is supposed to work on the
> Intel side?
>
Sure, I've done a quick PoC internally and have been able to hook this
into the DRM scheduler.
Basically each BB still maps to a single job as each job is somewhat
unique (e.g. each job has its own ring, lrc, seqno, etc...). However all
the jobs configured to run in parallel map to a single sched_entity
which maintains the order each job was generated from the execbuf IOCTL
(1 - N). When the backend receives jobs 1 to N - 1 it basically just
updates some internal state. When the backend sees job N (last job) it
actually does the submit for jobs 1 - N which with GuC submission is a
simple command moving the LRC tail of the N jobs.
Daniel has suggested that we create a single job for the NN BBs but that
would be huge rework to the internals of the i915 and likely won't
happen by the time this code first lands.
Also worth noting one way a job isn't really a treated individually is
the excl slot with dma-resv. In that case we create a composite fence of
all jobs (dma_fence_array).
Matt
> Thanks,
> Christian.
>
> Am 19.05.21 um 01:58 schrieb Matthew Brost:
> > Add entry fpr i915 new parallel submission uAPI plan.
> >
> > v2:
> > (Daniel Vetter):
> >- Expand logical order explaination
> >- Add dummy header
> >- Only allow N BBs in execbuf IOCTL
> >- Configure parallel submission per slot not per gem context
> >
> > Cc: Tvrtko Ursulin
> > Cc: Tony Ye
> > CC: Carl Zhang
> > Cc: Daniel Vetter
> > Cc: Jason Ekstrand
> > Signed-off-by: Matthew Brost
> > ---
> > Documentation/gpu/rfc/i915_parallel_execbuf.h | 144 ++
> > Documentation/gpu/rfc/i915_scheduler.rst | 53 ++-
> > 2 files changed, 196 insertions(+), 1 deletion(-)
> > create mode 100644 Documentation/gpu/rfc/i915_parallel_execbuf.h
> >
> > diff --git a/Documentation/gpu/rfc/i915_parallel_execbuf.h
> > b/Documentation/gpu/rfc/i915_parallel_execbuf.h
> > new file mode 100644
> > index ..8c64b983ccad
> > --- /dev/null
> > +++ b/Documentation/gpu/rfc/i915_parallel_execbuf.h
> > @@ -0,0 +1,144 @@
> > +#define I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT 2 /* see
> > i915_context_engines_parallel_submit */
> > +
> > +/*
> > + * i915_context_engines_parallel_submit:
> > + *
> > + * Setup a slot to allow multiple BBs to be submitted in a single execbuf
> > IOCTL.
> > + * Those BBs will then be scheduled to run on the GPU in parallel. Multiple
> > + * hardware contexts are created internally in the i915 run these BBs.
> > Once a
> > + * slot is configured for N BBs only N BBs can be submitted in each execbuf
> > + * IOCTL and this is implict behavior (e.g. the user doesn't tell the
> > execbuf
> > + * IOCTL there are N BBs, the execbuf IOCTL know how many BBs there are
> > based on
> > + * the slots configuration).
> > + *
> > + * Their are two currently defined ways to control the placement of the
> > + * hardware contexts on physical engines: default behavior (no flags) and
> > + * I915_PARALLEL_IMPLICT_BONDS (a flag). More flags may be added the in the
> > + * future as new hardware / use cases arise. Details of how to use this
> > + * interface below above the flags.
> > + *
> > + * Returns -EINVAL if hardware context placement configuration invalid or
> > if the
> > + * placement configuration isn't supported on the platform / submission
> > + * interface.
> > + * Returns -ENODEV if extension isn't supported on the platform /
> > submission
> > + * inteface.
> > + */
> > +struct i915_context_engines_parallel_submit {
> > + struct i915_user_extension base;
> > +
> > + __u16 engine_index; /* slot for parallel engine */
> > + __u16 width;/* number of contexts per parallel engine */
> > + __u16 num_siblings; /* number of siblings per context */
> > + __u16 mbz16;
> > +/*
> > + * Default placement behvavior (currently unsupported):
> > + *
> > + * Rather than restricting parallel submission to a single class with a
> > + * logically contiguous placement (I915_PARALLEL_IMPLICT_BONDS), add a
> > mode that
> > + * enables parallel submission across multiple engine classes. In this
> >
Re: [Mesa-dev] [Intel-gfx] [RFC 2/2] drm/doc/rfc: i915 new parallel submission uAPI plan
On Wed, May 19, 2021 at 01:10:04PM +0200, Daniel Vetter wrote:
> On Tue, May 18, 2021 at 04:58:30PM -0700, Matthew Brost wrote:
> > Add entry fpr i915 new parallel submission uAPI plan.
> >
> > v2:
> > (Daniel Vetter):
> > - Expand logical order explaination
> > - Add dummy header
> > - Only allow N BBs in execbuf IOCTL
> > - Configure parallel submission per slot not per gem context
> >
> > Cc: Tvrtko Ursulin
> > Cc: Tony Ye
> > CC: Carl Zhang
> > Cc: Daniel Vetter
> > Cc: Jason Ekstrand
> > Signed-off-by: Matthew Brost
> > ---
> > Documentation/gpu/rfc/i915_parallel_execbuf.h | 144 ++
> > Documentation/gpu/rfc/i915_scheduler.rst | 53 ++-
> > 2 files changed, 196 insertions(+), 1 deletion(-)
> > create mode 100644 Documentation/gpu/rfc/i915_parallel_execbuf.h
> >
> > diff --git a/Documentation/gpu/rfc/i915_parallel_execbuf.h
> > b/Documentation/gpu/rfc/i915_parallel_execbuf.h
> > new file mode 100644
> > index ..8c64b983ccad
> > --- /dev/null
> > +++ b/Documentation/gpu/rfc/i915_parallel_execbuf.h
> > @@ -0,0 +1,144 @@
> > +#define I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT 2 /* see
> > i915_context_engines_parallel_submit */
> > +
> > +/*
> > + * i915_context_engines_parallel_submit:
> > + *
> > + * Setup a slot to allow multiple BBs to be submitted in a single execbuf
> > IOCTL.
> > + * Those BBs will then be scheduled to run on the GPU in parallel. Multiple
> > + * hardware contexts are created internally in the i915 run these BBs.
> > Once a
> > + * slot is configured for N BBs only N BBs can be submitted in each execbuf
> > + * IOCTL and this is implict behavior (e.g. the user doesn't tell the
> > execbuf
> > + * IOCTL there are N BBs, the execbuf IOCTL know how many BBs there are
> > based on
> > + * the slots configuration).
> > + *
> > + * Their are two currently defined ways to control the placement of the
> > + * hardware contexts on physical engines: default behavior (no flags) and
> > + * I915_PARALLEL_IMPLICT_BONDS (a flag). More flags may be added the in the
> > + * future as new hardware / use cases arise. Details of how to use this
> > + * interface below above the flags.
> > + *
> > + * Returns -EINVAL if hardware context placement configuration invalid or
> > if the
> > + * placement configuration isn't supported on the platform / submission
> > + * interface.
> > + * Returns -ENODEV if extension isn't supported on the platform /
> > submission
> > + * inteface.
> > + */
> > +struct i915_context_engines_parallel_submit {
> > + struct i915_user_extension base;
> > +
> > + __u16 engine_index; /* slot for parallel engine */
> > + __u16 width;/* number of contexts per parallel engine */
> > + __u16 num_siblings; /* number of siblings per context */
> > + __u16 mbz16;
>
> Ok the big picture looks reasonable now, the flags still confuse me.
>
Yea, it is a bit confusing.
> > +/*
> > + * Default placement behvavior (currently unsupported):
> > + *
> > + * Rather than restricting parallel submission to a single class with a
> > + * logically contiguous placement (I915_PARALLEL_IMPLICT_BONDS), add a
> > mode that
> > + * enables parallel submission across multiple engine classes. In this
> > case each
> > + * context's logical engine mask indicates where that context can placed.
> > It is
> > + * implied in this mode that all contexts have mutual exclusive placement
> > (e.g.
> > + * if one context is running CS0 no other contexts can run on CS0).
> > + *
> > + * Example 1 pseudo code:
> > + * CSX[Y] = engine class X, logical instance Y
> > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE
> > + * set_engines(INVALID)
> > + * set_parallel(engine_index=0, width=2, num_siblings=2,
> > + * engines=CS0[0],CS0[1],CS1[0],CS1[1])
> > + *
> > + * Results in the following valid placements:
> > + * CS0[0], CS1[0]
> > + * CS0[0], CS1[1]
> > + * CS0[1], CS1[0]
> > + * CS0[1], CS1[1]
> > + *
> > + * This can also be though of as 2 virtual engines:
> > + * VE[0] = CS0[0], CS0[1]
> > + * VE[1] = CS1[0], CS1[1]
> > + *
> > + * Example 2 pseudo code:
> > + * CS[X] = generic engine of same class, logical instance X
> > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE
> > + * set_engines(INVALID)
> > + * set_parallel(engi
Re: [Mesa-dev] [Intel-gfx] [RFC 2/2] drm/doc/rfc: i915 new parallel submission uAPI plan
On Thu, May 20, 2021 at 11:54:25AM +0200, Daniel Vetter wrote:
> On Wed, May 19, 2021 at 7:19 PM Matthew Brost wrote:
> >
> > On Wed, May 19, 2021 at 01:10:04PM +0200, Daniel Vetter wrote:
> > > On Tue, May 18, 2021 at 04:58:30PM -0700, Matthew Brost wrote:
> > > > Add entry fpr i915 new parallel submission uAPI plan.
> > > >
> > > > v2:
> > > > (Daniel Vetter):
> > > > - Expand logical order explaination
> > > > - Add dummy header
> > > > - Only allow N BBs in execbuf IOCTL
> > > > - Configure parallel submission per slot not per gem context
> > > >
> > > > Cc: Tvrtko Ursulin
> > > > Cc: Tony Ye
> > > > CC: Carl Zhang
> > > > Cc: Daniel Vetter
> > > > Cc: Jason Ekstrand
> > > > Signed-off-by: Matthew Brost
> > > > ---
> > > > Documentation/gpu/rfc/i915_parallel_execbuf.h | 144 ++
> > > > Documentation/gpu/rfc/i915_scheduler.rst | 53 ++-
> > > > 2 files changed, 196 insertions(+), 1 deletion(-)
> > > > create mode 100644 Documentation/gpu/rfc/i915_parallel_execbuf.h
> > > >
> > > > diff --git a/Documentation/gpu/rfc/i915_parallel_execbuf.h
> > > > b/Documentation/gpu/rfc/i915_parallel_execbuf.h
> > > > new file mode 100644
> > > > index ..8c64b983ccad
> > > > --- /dev/null
> > > > +++ b/Documentation/gpu/rfc/i915_parallel_execbuf.h
> > > > @@ -0,0 +1,144 @@
> > > > +#define I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT 2 /* see
> > > > i915_context_engines_parallel_submit */
> > > > +
> > > > +/*
> > > > + * i915_context_engines_parallel_submit:
> > > > + *
> > > > + * Setup a slot to allow multiple BBs to be submitted in a single
> > > > execbuf IOCTL.
> > > > + * Those BBs will then be scheduled to run on the GPU in parallel.
> > > > Multiple
> > > > + * hardware contexts are created internally in the i915 run these BBs.
> > > > Once a
> > > > + * slot is configured for N BBs only N BBs can be submitted in each
> > > > execbuf
> > > > + * IOCTL and this is implict behavior (e.g. the user doesn't tell the
> > > > execbuf
> > > > + * IOCTL there are N BBs, the execbuf IOCTL know how many BBs there
> > > > are based on
> > > > + * the slots configuration).
> > > > + *
> > > > + * Their are two currently defined ways to control the placement of the
> > > > + * hardware contexts on physical engines: default behavior (no flags)
> > > > and
> > > > + * I915_PARALLEL_IMPLICT_BONDS (a flag). More flags may be added the
> > > > in the
> > > > + * future as new hardware / use cases arise. Details of how to use this
> > > > + * interface below above the flags.
> > > > + *
> > > > + * Returns -EINVAL if hardware context placement configuration invalid
> > > > or if the
> > > > + * placement configuration isn't supported on the platform / submission
> > > > + * interface.
> > > > + * Returns -ENODEV if extension isn't supported on the platform /
> > > > submission
> > > > + * inteface.
> > > > + */
> > > > +struct i915_context_engines_parallel_submit {
> > > > + struct i915_user_extension base;
> > > > +
> > > > + __u16 engine_index; /* slot for parallel engine */
> > > > + __u16 width;/* number of contexts per parallel engine */
> > > > + __u16 num_siblings; /* number of siblings per context */
> > > > + __u16 mbz16;
> > >
> > > Ok the big picture looks reasonable now, the flags still confuse me.
> > >
> >
> > Yea, it is a bit confusing.
> >
> > > > +/*
> > > > + * Default placement behvavior (currently unsupported):
> > > > + *
> > > > + * Rather than restricting parallel submission to a single class with a
> > > > + * logically contiguous placement (I915_PARALLEL_IMPLICT_BONDS), add a
> > > > mode that
> > > > + * enables parallel submission across multiple engine classes. In this
> > > > case each
> > > > + * context's logical engine mask indicates where that context can
> > > > placed. It is
> > > > + * implied in this mode that all contexts h
Re: [Mesa-dev] [RFC 2/2] drm/doc/rfc: i915 new parallel submission uAPI plan
On Thu, May 20, 2021 at 01:11:59PM +0200, Christian König wrote: > Am 19.05.21 um 18:51 schrieb Matthew Brost: > > On Wed, May 19, 2021 at 01:45:39PM +0200, Christian König wrote: > > > Oh, yeah we call that gang submit on the AMD side. > > > > > > Had already some internal discussions how to implement this, but so far > > > couldn't figure out how to cleanly introduce that into the DRM scheduler. > > > > > > Can you briefly describe in a few words how that is supposed to work on > > > the > > > Intel side? > > > > > Sure, I've done a quick PoC internally and have been able to hook this > > into the DRM scheduler. > > > > Basically each BB still maps to a single job as each job is somewhat > > unique (e.g. each job has its own ring, lrc, seqno, etc...). However all > > the jobs configured to run in parallel map to a single sched_entity > > which maintains the order each job was generated from the execbuf IOCTL > > (1 - N). When the backend receives jobs 1 to N - 1 it basically just > > updates some internal state. When the backend sees job N (last job) it > > actually does the submit for jobs 1 - N which with GuC submission is a > > simple command moving the LRC tail of the N jobs. > > > > Daniel has suggested that we create a single job for the NN BBs but that > > would be huge rework to the internals of the i915 and likely won't > > happen by the time this code first lands. > > > > Also worth noting one way a job isn't really a treated individually is > > the excl slot with dma-resv. In that case we create a composite fence of > > all jobs (dma_fence_array). > > Yeah, that's something we have discussed as well. > > How do you prevent the scheduler from over committing to a single ring > buffer in this scenario? > Each job has its own ring, the execbuf IOCTL throttles itself for each job if there isn't space in the ring. This is exactly the same as non-parallel submits. I think this is what you were asking? If not, maybe try explaining the question a bit more. Matt > Christian. > > > > > Matt > > > > > Thanks, > > > Christian. > > > > > > Am 19.05.21 um 01:58 schrieb Matthew Brost: > > > > Add entry fpr i915 new parallel submission uAPI plan. > > > > > > > > v2: > > > >(Daniel Vetter): > > > > - Expand logical order explaination > > > > - Add dummy header > > > > - Only allow N BBs in execbuf IOCTL > > > > - Configure parallel submission per slot not per gem context > > > > > > > > Cc: Tvrtko Ursulin > > > > Cc: Tony Ye > > > > CC: Carl Zhang > > > > Cc: Daniel Vetter > > > > Cc: Jason Ekstrand > > > > Signed-off-by: Matthew Brost > > > > --- > > > >Documentation/gpu/rfc/i915_parallel_execbuf.h | 144 > > > > ++ > > > >Documentation/gpu/rfc/i915_scheduler.rst | 53 ++- > > > >2 files changed, 196 insertions(+), 1 deletion(-) > > > >create mode 100644 Documentation/gpu/rfc/i915_parallel_execbuf.h > > > > > > > > diff --git a/Documentation/gpu/rfc/i915_parallel_execbuf.h > > > > b/Documentation/gpu/rfc/i915_parallel_execbuf.h > > > > new file mode 100644 > > > > index ..8c64b983ccad > > > > --- /dev/null > > > > +++ b/Documentation/gpu/rfc/i915_parallel_execbuf.h > > > > @@ -0,0 +1,144 @@ > > > > +#define I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT 2 /* see > > > > i915_context_engines_parallel_submit */ > > > > + > > > > +/* > > > > + * i915_context_engines_parallel_submit: > > > > + * > > > > + * Setup a slot to allow multiple BBs to be submitted in a single > > > > execbuf IOCTL. > > > > + * Those BBs will then be scheduled to run on the GPU in parallel. > > > > Multiple > > > > + * hardware contexts are created internally in the i915 run these BBs. > > > > Once a > > > > + * slot is configured for N BBs only N BBs can be submitted in each > > > > execbuf > > > > + * IOCTL and this is implict behavior (e.g. the user doesn't tell the > > > > execbuf > > > > + * IOCTL there are N BBs, the execbuf IOCTL know how many BBs there > > > > are based on > > > > + * the slots configuration). > > > > + * > > >
Re: [Mesa-dev] [Intel-gfx] [RFC 2/2] drm/doc/rfc: i915 new parallel submission uAPI plan
On Fri, May 21, 2021 at 10:35:37AM +0200, Christian König wrote: > Am 20.05.21 um 23:38 schrieb Jason Ekstrand: > > On Thu, May 20, 2021 at 10:46 AM Matthew Brost > > wrote: > > > On Thu, May 20, 2021 at 01:11:59PM +0200, Christian König wrote: > > > > Am 19.05.21 um 18:51 schrieb Matthew Brost: > > > > > On Wed, May 19, 2021 at 01:45:39PM +0200, Christian König wrote: > > > > > > Oh, yeah we call that gang submit on the AMD side. > > > > > > > > > > > > Had already some internal discussions how to implement this, but so > > > > > > far > > > > > > couldn't figure out how to cleanly introduce that into the DRM > > > > > > scheduler. > > > > > > > > > > > > Can you briefly describe in a few words how that is supposed to > > > > > > work on the > > > > > > Intel side? > > On Intel, we actually have two cases which don't fit the current > > drm/scheduler model well: balanced and bonded. > > > > In the balanced model, we want to submit a batch which can go to any > > one of some set of engines and we don't care which. It's up to the > > kernel to pick an engine. Imagine you had 64 identical HW compute > > queues, for instance. This could be done by making all the identical > > engines share a single drm_gpu_scheduler and round-robin around the HW > > queues or something. I don't know that we strictly need drm/scheduler > > to be aware of it but it might be nice if it grew support for this > > mode so we could maintain a 1:1 relationship between HW queues and > > drm_gpu_schedulers. That said, I'm not sure how this would play with > > GuC queues so maybe it doesn't help? > > Oh, we do have support for load balancing like that. > > When you call drm_sched_entity_init() you can give a list of > drm_gpu_scheduler object to use round robing for scheduling. > > New jobs are then scheduler to the drm_gpu_scheduler instance which is idle > or rather the least busy one. > > > The bonded model is like your ganged, I think. We want to submit N > > batches to run in parallel. And they actually have to be executing on > > the GPU simultaneously and not just sort-of at similar times. We need > > this for video. There are also potential use-cases in Vulkan or even > > GL that might be able to use this. One difference with the balanced > > mode is that bonds don't, strictly speaking, need to be on the same > > type of engine. Imagine, for instance, a 3D batch with a parallel > > compute batch doing vertex pre-processing. > > > > I'm pretty sure the bonded case is something that the mobile drivers > > (panfrost, etc.) would like as well for doing Vulkan on tilers where > > you often have to have two command buffers running in parallel. > > They're currently doing it by submitting a giant pile of batches where > > they split the batch and add sync primitives every time some GL call > > requires them to sync between fragment and vertex pipes. > > Yeah, we have exactly the same problem as well. > > But so far every model we discussed has some drawbacks and it is rather hard > for the scheduler to guarantee that stuff runs at the same time. > > So if you got any ideas how to cleanly implement that then they would be > rather welcomed. > Everything Jason said about our submission modes is correct for execlists, we have balanced + bonded models which is tightly coupled with that backend. Fortunately with GuC submission most of this complexity goes away as the GuC handles this for us. e.g. For balanced when we register a context we just give it a mask of which physical engines a context is allowed to run on. For parallel we register N contexts in a single command with the placement information + submit to all the contexts with a command which moves the tails in LRCs for us. We really don't need to bake any of this into the DRM scheduler for GuC submission. Execlists is different story but I think our plan is to do the minimum possible to plum that into the DRM scheduler (e.g. leave the balanced / bonded code in the backend). Could we update the DRM scheduler to understand balanced (seems like already does to some extent) and bonded? Yes we could but IMO the ROI on that is low for Intel. The DRM scheduler is quite clean at the moment compared to our near incomprehensible execlist backend. Execlists are basically a legacy interface so pushing features which are only required for that backend to the DRM scheduler doesn't make sense to me. That being said, if this is something AMD needs in the DRM sch
Re: [Mesa-dev] [Intel-gfx] [RFC 2/2] drm/doc/rfc: i915 new parallel submission uAPI plan
On Thu, May 20, 2021 at 09:44:59PM +0200, Daniel Vetter wrote:
> On Thu, May 20, 2021 at 08:10:59AM -0700, Matthew Brost wrote:
> > On Thu, May 20, 2021 at 11:54:25AM +0200, Daniel Vetter wrote:
> > > On Wed, May 19, 2021 at 7:19 PM Matthew Brost
> > > wrote:
> > > >
> > > > On Wed, May 19, 2021 at 01:10:04PM +0200, Daniel Vetter wrote:
> > > > > On Tue, May 18, 2021 at 04:58:30PM -0700, Matthew Brost wrote:
> > > > > > Add entry fpr i915 new parallel submission uAPI plan.
> > > > > >
> > > > > > v2:
> > > > > > (Daniel Vetter):
> > > > > > - Expand logical order explaination
> > > > > > - Add dummy header
> > > > > > - Only allow N BBs in execbuf IOCTL
> > > > > > - Configure parallel submission per slot not per gem context
> > > > > >
> > > > > > Cc: Tvrtko Ursulin
> > > > > > Cc: Tony Ye
> > > > > > CC: Carl Zhang
> > > > > > Cc: Daniel Vetter
> > > > > > Cc: Jason Ekstrand
> > > > > > Signed-off-by: Matthew Brost
> > > > > > ---
> > > > > > Documentation/gpu/rfc/i915_parallel_execbuf.h | 144
> > > > > > ++
> > > > > > Documentation/gpu/rfc/i915_scheduler.rst | 53 ++-
> > > > > > 2 files changed, 196 insertions(+), 1 deletion(-)
> > > > > > create mode 100644 Documentation/gpu/rfc/i915_parallel_execbuf.h
> > > > > >
> > > > > > diff --git a/Documentation/gpu/rfc/i915_parallel_execbuf.h
> > > > > > b/Documentation/gpu/rfc/i915_parallel_execbuf.h
> > > > > > new file mode 100644
> > > > > > index ..8c64b983ccad
> > > > > > --- /dev/null
> > > > > > +++ b/Documentation/gpu/rfc/i915_parallel_execbuf.h
> > > > > > @@ -0,0 +1,144 @@
> > > > > > +#define I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT 2 /* see
> > > > > > i915_context_engines_parallel_submit */
> > > > > > +
> > > > > > +/*
> > > > > > + * i915_context_engines_parallel_submit:
> > > > > > + *
> > > > > > + * Setup a slot to allow multiple BBs to be submitted in a single
> > > > > > execbuf IOCTL.
> > > > > > + * Those BBs will then be scheduled to run on the GPU in parallel.
> > > > > > Multiple
> > > > > > + * hardware contexts are created internally in the i915 run these
> > > > > > BBs. Once a
> > > > > > + * slot is configured for N BBs only N BBs can be submitted in
> > > > > > each execbuf
> > > > > > + * IOCTL and this is implict behavior (e.g. the user doesn't tell
> > > > > > the execbuf
> > > > > > + * IOCTL there are N BBs, the execbuf IOCTL know how many BBs
> > > > > > there are based on
> > > > > > + * the slots configuration).
> > > > > > + *
> > > > > > + * Their are two currently defined ways to control the placement
> > > > > > of the
> > > > > > + * hardware contexts on physical engines: default behavior (no
> > > > > > flags) and
> > > > > > + * I915_PARALLEL_IMPLICT_BONDS (a flag). More flags may be added
> > > > > > the in the
> > > > > > + * future as new hardware / use cases arise. Details of how to use
> > > > > > this
> > > > > > + * interface below above the flags.
> > > > > > + *
> > > > > > + * Returns -EINVAL if hardware context placement configuration
> > > > > > invalid or if the
> > > > > > + * placement configuration isn't supported on the platform /
> > > > > > submission
> > > > > > + * interface.
> > > > > > + * Returns -ENODEV if extension isn't supported on the platform /
> > > > > > submission
> > > > > > + * inteface.
> > > > > > + */
> > > > > > +struct i915_context_engines_parallel_submit {
> > > > > > + struct i915_user_extension base;
> > > > > > +
> > > > > > + __u16 engine_index; /* slot for parallel engine */
> > > > > > + __u16 width;/*
Re: [Mesa-dev] [Intel-gfx] [RFC 2/2] drm/doc/rfc: i915 new parallel submission uAPI plan
On Thu, May 20, 2021 at 09:41:20PM +0200, Daniel Vetter wrote:
> On Thu, May 20, 2021 at 11:57:44AM +0100, Tvrtko Ursulin wrote:
> >
> > On 20/05/2021 10:54, Daniel Vetter wrote:
> > > On Wed, May 19, 2021 at 7:19 PM Matthew Brost
> > > wrote:
> > > >
> > > > On Wed, May 19, 2021 at 01:10:04PM +0200, Daniel Vetter wrote:
> > > > > On Tue, May 18, 2021 at 04:58:30PM -0700, Matthew Brost wrote:
> > > > > > Add entry fpr i915 new parallel submission uAPI plan.
> > > > > >
> > > > > > v2:
> > > > > > (Daniel Vetter):
> > > > > >- Expand logical order explaination
> > > > > >- Add dummy header
> > > > > >- Only allow N BBs in execbuf IOCTL
> > > > > >- Configure parallel submission per slot not per gem context
> > > > > >
> > > > > > Cc: Tvrtko Ursulin
> > > > > > Cc: Tony Ye
> > > > > > CC: Carl Zhang
> > > > > > Cc: Daniel Vetter
> > > > > > Cc: Jason Ekstrand
> > > > > > Signed-off-by: Matthew Brost
> > > > > > ---
> > > > > > Documentation/gpu/rfc/i915_parallel_execbuf.h | 144
> > > > > > ++
> > > > > > Documentation/gpu/rfc/i915_scheduler.rst | 53 ++-
> > > > > > 2 files changed, 196 insertions(+), 1 deletion(-)
> > > > > > create mode 100644 Documentation/gpu/rfc/i915_parallel_execbuf.h
> > > > > >
> > > > > > diff --git a/Documentation/gpu/rfc/i915_parallel_execbuf.h
> > > > > > b/Documentation/gpu/rfc/i915_parallel_execbuf.h
> > > > > > new file mode 100644
> > > > > > index ..8c64b983ccad
> > > > > > --- /dev/null
> > > > > > +++ b/Documentation/gpu/rfc/i915_parallel_execbuf.h
> > > > > > @@ -0,0 +1,144 @@
> > > > > > +#define I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT 2 /* see
> > > > > > i915_context_engines_parallel_submit */
> > > > > > +
> > > > > > +/*
> > > > > > + * i915_context_engines_parallel_submit:
> > > > > > + *
> > > > > > + * Setup a slot to allow multiple BBs to be submitted in a single
> > > > > > execbuf IOCTL.
> > > > > > + * Those BBs will then be scheduled to run on the GPU in parallel.
> > > > > > Multiple
> > > > > > + * hardware contexts are created internally in the i915 run these
> > > > > > BBs. Once a
> > > > > > + * slot is configured for N BBs only N BBs can be submitted in
> > > > > > each execbuf
> > > > > > + * IOCTL and this is implict behavior (e.g. the user doesn't tell
> > > > > > the execbuf
> > > > > > + * IOCTL there are N BBs, the execbuf IOCTL know how many BBs
> > > > > > there are based on
> > > > > > + * the slots configuration).
> > > > > > + *
> > > > > > + * Their are two currently defined ways to control the placement
> > > > > > of the
> > > > > > + * hardware contexts on physical engines: default behavior (no
> > > > > > flags) and
> > > > > > + * I915_PARALLEL_IMPLICT_BONDS (a flag). More flags may be added
> > > > > > the in the
> > > > > > + * future as new hardware / use cases arise. Details of how to use
> > > > > > this
> > > > > > + * interface below above the flags.
> > > > > > + *
> > > > > > + * Returns -EINVAL if hardware context placement configuration
> > > > > > invalid or if the
> > > > > > + * placement configuration isn't supported on the platform /
> > > > > > submission
> > > > > > + * interface.
> > > > > > + * Returns -ENODEV if extension isn't supported on the platform /
> > > > > > submission
> > > > > > + * inteface.
> > > > > > + */
> > > > > > +struct i915_context_engines_parallel_submit {
> > > > > > + struct i915_user_extension base;
> > > > > > +
> > > > > > + __u16 engine_index; /* slot for parallel engine */
> > > > > > + __u16 width;
Re: [Mesa-dev] [Intel-gfx] [RFC 2/2] drm/doc/rfc: i915 new parallel submission uAPI plan
On Fri, May 21, 2021 at 01:00:54PM +0100, Tvrtko Ursulin wrote:
>
> On 19/05/2021 00:58, Matthew Brost wrote:
> > Add entry fpr i915 new parallel submission uAPI plan.
> >
> > v2:
> > (Daniel Vetter):
> >- Expand logical order explaination
> >- Add dummy header
> >- Only allow N BBs in execbuf IOCTL
> >- Configure parallel submission per slot not per gem context
> >
> > Cc: Tvrtko Ursulin
> > Cc: Tony Ye
> > CC: Carl Zhang
> > Cc: Daniel Vetter
> > Cc: Jason Ekstrand
> > Signed-off-by: Matthew Brost
> > ---
> > Documentation/gpu/rfc/i915_parallel_execbuf.h | 144 ++
> > Documentation/gpu/rfc/i915_scheduler.rst | 53 ++-
> > 2 files changed, 196 insertions(+), 1 deletion(-)
> > create mode 100644 Documentation/gpu/rfc/i915_parallel_execbuf.h
> >
> > diff --git a/Documentation/gpu/rfc/i915_parallel_execbuf.h
> > b/Documentation/gpu/rfc/i915_parallel_execbuf.h
> > new file mode 100644
> > index ..8c64b983ccad
> > --- /dev/null
> > +++ b/Documentation/gpu/rfc/i915_parallel_execbuf.h
> > @@ -0,0 +1,144 @@
> > +#define I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT 2 /* see
> > i915_context_engines_parallel_submit */
> > +
> > +/*
> > + * i915_context_engines_parallel_submit:
> > + *
> > + * Setup a slot to allow multiple BBs to be submitted in a single execbuf
> > IOCTL.
> > + * Those BBs will then be scheduled to run on the GPU in parallel. Multiple
> > + * hardware contexts are created internally in the i915 run these BBs.
> > Once a
> > + * slot is configured for N BBs only N BBs can be submitted in each execbuf
> > + * IOCTL and this is implict behavior (e.g. the user doesn't tell the
> > execbuf
> > + * IOCTL there are N BBs, the execbuf IOCTL know how many BBs there are
> > based on
> > + * the slots configuration).
>
> 1)
> Expand the term slot here with "slot in the context engine map" least once
> for clarity.
>
Sure.
> 2)
> About where execbuf will implicitly be finding batches - suggest to also
> cover first/last flag here. I know you have it in the readme but I think it
> is good if uapi header is as self-contained as possible.
>
Yep, good idea.
> > + *
> > + * Their are two currently defined ways to control the placement of the
> > + * hardware contexts on physical engines: default behavior (no flags) and
> > + * I915_PARALLEL_IMPLICT_BONDS (a flag). More flags may be added the in the
> > + * future as new hardware / use cases arise. Details of how to use this
> > + * interface below above the flags.
> > + *
> > + * Returns -EINVAL if hardware context placement configuration invalid or
> > if the
> > + * placement configuration isn't supported on the platform / submission
> > + * interface.
> > + * Returns -ENODEV if extension isn't supported on the platform /
> > submission
> > + * inteface.
> > + */
> > +struct i915_context_engines_parallel_submit {
> > + struct i915_user_extension base;
> > +
> > + __u16 engine_index; /* slot for parallel engine */
> > + __u16 width;/* number of contexts per parallel engine */
> > + __u16 num_siblings; /* number of siblings per context */
> > + __u16 mbz16;
> > +/*
> > + * Default placement behvavior (currently unsupported):
> > + *
> > + * Rather than restricting parallel submission to a single class with a
> > + * logically contiguous placement (I915_PARALLEL_IMPLICT_BONDS), add a
> > mode that
>
> What do you mean with logically contiguous here? It sounds ambiguous versus
> logical vs "normal" engine instance numbers.
>
This is a little backwards. I think when I wrote this originally the implicit
placement comments were first. I can reword this.
> > + * enables parallel submission across multiple engine classes. In this
> > case each
> > + * context's logical engine mask indicates where that context can placed.
> > It is
> > + * implied in this mode that all contexts have mutual exclusive placement
> > (e.g.
> > + * if one context is running CS0 no other contexts can run on CS0).
>
> I think talk about logical context and its mask is too implementation detail
> at the uapi level. Instead I would suggest more userspace programmer centric
> description.
Ok, can you give me suggestion? Writing DOC isn't really my strength.
>
> > + *
> > + * Example 1 pseudo code:
> > + * CSX[Y] = engine class X, logical instance Y
> > + * INVALID = I915_EN
[Mesa-dev] [RFC PATCH 1/2] drm/doc/rfc: i915 GuC submission / DRM scheduler
Add entry for i915 GuC submission / DRM scheduler integration plan. Follow up patch with details of new parallel submission uAPI to come. v2: (Daniel Vetter) - Expand explaination of why bonding isn't supported for GuC submission - CC some of the DRM scheduler maintainers - Add priority inheritance / boosting use case - Add reasoning for removing in order assumptions (Daniel Stone) - Add links to priority spec Cc: Christian König Cc: Luben Tuikov Cc: Alex Deucher Cc: Steven Price Cc: Jon Bloomfield Cc: Jason Ekstrand Cc: Dave Airlie Cc: Daniel Vetter Cc: Jason Ekstrand Cc: dri-de...@lists.freedesktop.org Signed-off-by: Matthew Brost --- Documentation/gpu/rfc/i915_scheduler.rst | 85 Documentation/gpu/rfc/index.rst | 4 ++ 2 files changed, 89 insertions(+) create mode 100644 Documentation/gpu/rfc/i915_scheduler.rst diff --git a/Documentation/gpu/rfc/i915_scheduler.rst b/Documentation/gpu/rfc/i915_scheduler.rst new file mode 100644 index ..7faa46cde088 --- /dev/null +++ b/Documentation/gpu/rfc/i915_scheduler.rst @@ -0,0 +1,85 @@ += +I915 GuC Submission/DRM Scheduler Section += + +Upstream plan += +For upstream the overall plan for landing GuC submission and integrating the +i915 with the DRM scheduler is: + +* Merge basic GuC submission + * Basic submission support for all gen11+ platforms + * Not enabled by default on any current platforms but can be enabled via + modparam enable_guc + * Lots of rework will need to be done to integrate with DRM scheduler so + no need to nit pick everything in the code, it just should be + functional, no major coding style / layering errors, and not regress + execlists + * Update IGTs / selftests as needed to work with GuC submission + * Enable CI on supported platforms for a baseline + * Rework / get CI heathly for GuC submission in place as needed +* Merge new parallel submission uAPI + * Bonding uAPI completely incompatible with GuC submission, plus it has + severe design issues in general, which is why we want to retire it no + matter what + * New uAPI adds I915_CONTEXT_ENGINES_EXT_PARALLEL context setup step + which configures a slot with N contexts + * After I915_CONTEXT_ENGINES_EXT_PARALLEL a user can submit N batches to + a slot in a single execbuf IOCTL and the batches run on the GPU in + paralllel + * Initially only for GuC submission but execlists can be supported if + needed +* Convert the i915 to use the DRM scheduler + * GuC submission backend fully integrated with DRM scheduler + * All request queues removed from backend (e.g. all backpressure + handled in DRM scheduler) + * Resets / cancels hook in DRM scheduler + * Watchdog hooks into DRM scheduler + * Lots of complexity of the GuC backend can be pulled out once + integrated with DRM scheduler (e.g. state machine gets + simplier, locking gets simplier, etc...) + * Execlist backend will do the minimum required to hook in the DRM + scheduler so it can live next to the fully integrated GuC backend + * Legacy interface + * Features like timeslicing / preemption / virtual engines would + be difficult to integrate with the DRM scheduler and these + features are not required for GuC submission as the GuC does + these things for us + * ROI low on fully integrating into DRM scheduler + * Fully integrating would add lots of complexity to DRM + scheduler + * Port i915 priority inheritance / boosting feature in DRM scheduler + * Used for i915 page flip, may be useful to other DRM drivers as + well + * Will be an optional feature in the DRM scheduler + * Remove in-order completion assumptions from DRM scheduler + * Even when using the DRM scheduler the backends will handle + preemption, timeslicing, etc... so it is possible for jobs to + finish out of order + * Pull out i915 priority levels and use DRM priority levels + * Optimize DRM scheduler as needed + +New uAPI for basic GuC submission += +No major changes are required to the uAPI for basic GuC submission. The only +change is a new scheduler attribute: I915_SCHEDULER_CAP_STATIC_PRIORITY_MAP. +This attribute indicates the 2k i915 user priority levels are statically mapped +into 3 levels as follows: + +* -1k to -1 Low priority +* 0 Medium priority +* 1 to 1k High priority + +This is needed because the GuC only has 4 priority bands. The highest priority +band is reserved wit
[Mesa-dev] [RFC PATCH 0/2] GuC submission / DRM scheduler integration plan + new uAPI
Subject and patches say it all. v2: Address comments, patches have details of changes v3: Address comments, patches have details of changes Signed-off-by: Matthew Brost Matthew Brost (2): drm/doc/rfc: i915 GuC submission / DRM scheduler drm/doc/rfc: i915 new parallel submission uAPI plan Documentation/gpu/rfc/i915_parallel_execbuf.h | 145 ++ Documentation/gpu/rfc/i915_scheduler.rst | 136 Documentation/gpu/rfc/index.rst | 4 + 3 files changed, 285 insertions(+) create mode 100644 Documentation/gpu/rfc/i915_parallel_execbuf.h create mode 100644 Documentation/gpu/rfc/i915_scheduler.rst -- 2.28.0 ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
[Mesa-dev] [RFC PATCH 2/2] drm/doc/rfc: i915 new parallel submission uAPI plan
Add entry for i915 new parallel submission uAPI plan.
v2:
(Daniel Vetter):
- Expand logical order explaination
- Add dummy header
- Only allow N BBs in execbuf IOCTL
- Configure parallel submission per slot not per gem context
v3:
(Marcin Ślusarz):
- Lot's of typos / bad english fixed
(Tvrtko Ursulin):
- Consistent pseudo code, clean up wording in descriptions
Cc: Tvrtko Ursulin
Cc: Tony Ye
CC: Carl Zhang
Cc: Daniel Vetter
Cc: Jason Ekstrand
Signed-off-by: Matthew Brost
---
Documentation/gpu/rfc/i915_parallel_execbuf.h | 145 ++
Documentation/gpu/rfc/i915_scheduler.rst | 55 ++-
2 files changed, 198 insertions(+), 2 deletions(-)
create mode 100644 Documentation/gpu/rfc/i915_parallel_execbuf.h
diff --git a/Documentation/gpu/rfc/i915_parallel_execbuf.h
b/Documentation/gpu/rfc/i915_parallel_execbuf.h
new file mode 100644
index ..20de206e3ab4
--- /dev/null
+++ b/Documentation/gpu/rfc/i915_parallel_execbuf.h
@@ -0,0 +1,145 @@
+#define I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT 2 /* see
i915_context_engines_parallel_submit */
+
+/*
+ * i915_context_engines_parallel_submit:
+ *
+ * Setup a slot in the context engine map to allow multiple BBs to be submitted
+ * in a single execbuf IOCTL. Those BBs will then be scheduled to run on the
GPU
+ * in parallel. Multiple hardware contexts are created internally in the i915
+ * run these BBs. Once a slot is configured for N BBs only N BBs can be
+ * submitted in each execbuf IOCTL and this is implicit behavior e.g. The user
+ * doesn't tell the execbuf IOCTL there are N BBs, the execbuf IOCTL know how
+ * many BBs there are based on the slots configuration. The N BBs are the last
N
+ * buffer objects for first N if I915_EXEC_BATCH_FIRST is set.
+ *
+ * There are two currently defined ways to control the placement of the
+ * hardware contexts on physical engines: default behavior (no flags) and
+ * I915_PARALLEL_IMPLICIT_BONDS (a flag). More flags may be added the in the
+ * future as new hardware / use cases arise. Details of how to use this
+ * interface above the flags field in this structure.
+ *
+ * Returns -EINVAL if hardware context placement configuration is invalid or if
+ * the placement configuration isn't supported on the platform / submission
+ * interface.
+ * Returns -ENODEV if extension isn't supported on the platform / submission
+ * inteface.
+ */
+struct i915_context_engines_parallel_submit {
+ struct i915_user_extension base;
+
+ __u16 engine_index; /* slot for parallel engine */
+ __u16 width;/* number of contexts per parallel engine */
+ __u16 num_siblings; /* number of siblings per context */
+ __u16 mbz16;
+/*
+ * Default placement behavior (currently unsupported):
+ *
+ * Allow BBs to be placed on any available engine instance. In this case each
+ * context's engine mask indicates where that context can be placed. It is
+ * implied in this mode that all contexts have mutual exclusive placement.
+ * e.g. If one context is running CSX[0] no other contexts can run on CSX[0]).
+ *
+ * Example 1 pseudo code:
+ * CSX,Y[N] = generic engine class X or Y, logical instance N
+ * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE
+ * set_engines(INVALID)
+ * set_parallel(engine_index=0, width=2, num_siblings=2,
+ * engines=CSX[0],CSX[1],CSY[0],CSY[1])
+ *
+ * Results in the following valid placements:
+ * CSX[0], CSY[0]
+ * CSX[0], CSY[1]
+ * CSX[1], CSY[0]
+ * CSX[1], CSY[1]
+ *
+ * This can also be thought of as 2 virtual engines described by 2-D array in
+ * the engines the field:
+ * VE[0] = CSX[0], CSX[1]
+ * VE[1] = CSY[0], CSY[1]
+ *
+ * Example 2 pseudo code:
+ * CSX[Y] = generic engine of same class X, logical instance N
+ * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE
+ * set_engines(INVALID)
+ * set_parallel(engine_index=0, width=2, num_siblings=3,
+ * engines=CSX[0],CSX[1],CSX[2],CSX[0],CSX[1],CSX[2])
+ *
+ * Results in the following valid placements:
+ * CSX[0], CSX[1]
+ * CSX[0], CSX[2]
+ * CSX[1], CSX[0]
+ * CSX[1], CSX[2]
+ * CSX[2], CSX[0]
+ * CSX[2], CSX[1]
+ *
+ * This can also be thought of as 2 virtual engines described by 2-D array in
+ * the engines the field:
+ * VE[0] = CSX[0], CSX[1], CSX[2]
+ * VE[1] = CSX[0], CSX[1], CSX[2]
+
+ * This enables a use case where all engines are created equally, we don't care
+ * where they are scheduled, we just want a certain number of resources, for
+ * those resources to be scheduled in parallel, and possibly across multiple
+ * engine classes.
+ */
+
+/*
+ * I915_PARALLEL_IMPLICIT_BONDS - Create implicit bonds between each context.
+ * Each context must have the same number of sibling and bonds are implicitly
+ * created between each set of siblings.
+ *
+ * Example 1 pseudo code:
+ * CSX[N] = generic engine of same class X, logical instance N
+ * INVALID = I915_ENGINE_CLASS_INVALID,
Re: [Mesa-dev] [Intel-gfx] [RFC PATCH 2/2] drm/doc/rfc: i915 new parallel submission uAPI plan
On Fri, Jun 04, 2021 at 07:59:05PM +0200, Daniel Vetter wrote:
> On Wed, May 26, 2021 at 04:33:57PM -0700, Matthew Brost wrote:
> > Add entry for i915 new parallel submission uAPI plan.
> >
> > v2:
> > (Daniel Vetter):
> > - Expand logical order explaination
> > - Add dummy header
> > - Only allow N BBs in execbuf IOCTL
> > - Configure parallel submission per slot not per gem context
> > v3:
> > (Marcin Ślusarz):
> > - Lot's of typos / bad english fixed
> > (Tvrtko Ursulin):
> > - Consistent pseudo code, clean up wording in descriptions
> >
> > Cc: Tvrtko Ursulin
> > Cc: Tony Ye
> > CC: Carl Zhang
> > Cc: Daniel Vetter
> > Cc: Jason Ekstrand
> > Signed-off-by: Matthew Brost
> > ---
> > Documentation/gpu/rfc/i915_parallel_execbuf.h | 145 ++
> > Documentation/gpu/rfc/i915_scheduler.rst | 55 ++-
> > 2 files changed, 198 insertions(+), 2 deletions(-)
> > create mode 100644 Documentation/gpu/rfc/i915_parallel_execbuf.h
> >
> > diff --git a/Documentation/gpu/rfc/i915_parallel_execbuf.h
> > b/Documentation/gpu/rfc/i915_parallel_execbuf.h
> > new file mode 100644
> > index ..20de206e3ab4
> > --- /dev/null
> > +++ b/Documentation/gpu/rfc/i915_parallel_execbuf.h
> > @@ -0,0 +1,145 @@
> > +#define I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT 2 /* see
> > i915_context_engines_parallel_submit */
> > +
> > +/*
> > + * i915_context_engines_parallel_submit:
>
> So the idea is to make these kerneldoc and pull them into the rfc section.
> Then when we merge, move them to the real uapi section, like what Matt has
> done for lmem.
>
Yep, will fix in next rev.
> > + *
> > + * Setup a slot in the context engine map to allow multiple BBs to be
> > submitted
> > + * in a single execbuf IOCTL. Those BBs will then be scheduled to run on
> > the GPU
> > + * in parallel. Multiple hardware contexts are created internally in the
> > i915
> > + * run these BBs. Once a slot is configured for N BBs only N BBs can be
> > + * submitted in each execbuf IOCTL and this is implicit behavior e.g. The
> > user
> > + * doesn't tell the execbuf IOCTL there are N BBs, the execbuf IOCTL know
> > how
> > + * many BBs there are based on the slots configuration. The N BBs are the
> > last N
> > + * buffer objects for first N if I915_EXEC_BATCH_FIRST is set.
>
> s/for/or/
>
> > + *
> > + * There are two currently defined ways to control the placement of the
> > + * hardware contexts on physical engines: default behavior (no flags) and
> > + * I915_PARALLEL_IMPLICIT_BONDS (a flag). More flags may be added the in
> > the
> > + * future as new hardware / use cases arise. Details of how to use this
> > + * interface above the flags field in this structure.
> > + *
> > + * Returns -EINVAL if hardware context placement configuration is invalid
> > or if
> > + * the placement configuration isn't supported on the platform / submission
> > + * interface.
> > + * Returns -ENODEV if extension isn't supported on the platform /
> > submission
> > + * inteface.
> > + */
> > +struct i915_context_engines_parallel_submit {
> > + struct i915_user_extension base;
> > +
> > + __u16 engine_index; /* slot for parallel engine */
>
> Kernel doc here for the inline comments too.
>
Yep.
> > + __u16 width;/* number of contexts per parallel engine */
> > + __u16 num_siblings; /* number of siblings per context */
> > + __u16 mbz16;
> > +/*
> > + * Default placement behavior (currently unsupported):
> > + *
> > + * Allow BBs to be placed on any available engine instance. In this case
> > each
> > + * context's engine mask indicates where that context can be placed. It is
> > + * implied in this mode that all contexts have mutual exclusive placement.
> > + * e.g. If one context is running CSX[0] no other contexts can run on
> > CSX[0]).
> > + *
> > + * Example 1 pseudo code:
> > + * CSX,Y[N] = generic engine class X or Y, logical instance N
> > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE
> > + * set_engines(INVALID)
> > + * set_parallel(engine_index=0, width=2, num_siblings=2,
> > + * engines=CSX[0],CSX[1],CSY[0],CSY[1])
> > + *
> > + * Results in the following valid placements:
> > + * CSX[0], CSY[0]
> > + * CSX[0], CSY[1]
> > + * CSX[1], CSY[0]
> > + * CSX[1], CSY[1]
> > + *
> > + * Th
[Mesa-dev] [PATCH 0/2] GuC submission / DRM scheduler integration plan + new uAPI
Subject and patches say it all. v2: Address comments, patches have details of changes v3: Address comments, patches have details of changes v4: Address comments, patches have details of changes Signed-off-by: Matthew Brost Matthew Brost (2): drm/doc/rfc: i915 GuC submission / DRM scheduler drm/doc/rfc: i915 new parallel submission uAPI plan Documentation/gpu/rfc/i915_parallel_execbuf.h | 117 ++ Documentation/gpu/rfc/i915_scheduler.rst | 148 ++ Documentation/gpu/rfc/index.rst | 4 + 3 files changed, 269 insertions(+) create mode 100644 Documentation/gpu/rfc/i915_parallel_execbuf.h create mode 100644 Documentation/gpu/rfc/i915_scheduler.rst -- 2.28.0 ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
[Mesa-dev] [PATCH 1/2] drm/doc/rfc: i915 GuC submission / DRM scheduler
Add entry for i915 GuC submission / DRM scheduler integration plan. Follow up patch with details of new parallel submission uAPI to come. v2: (Daniel Vetter) - Expand explaination of why bonding isn't supported for GuC submission - CC some of the DRM scheduler maintainers - Add priority inheritance / boosting use case - Add reasoning for removing in order assumptions (Daniel Stone) - Add links to priority spec v4: (Tvrtko) - Add TODOs section (Daniel Vetter) - Pull in 1 line from following patch Cc: Christian König Cc: Luben Tuikov Cc: Alex Deucher Cc: Steven Price Cc: Jon Bloomfield Cc: Jason Ekstrand Cc: Dave Airlie Cc: Daniel Vetter Cc: Jason Ekstrand Cc: dri-de...@lists.freedesktop.org Signed-off-by: Matthew Brost Reviewed-by: Daniel Vetter Acked-by: Dave Airlie --- Documentation/gpu/rfc/i915_scheduler.rst | 91 Documentation/gpu/rfc/index.rst | 4 ++ 2 files changed, 95 insertions(+) create mode 100644 Documentation/gpu/rfc/i915_scheduler.rst diff --git a/Documentation/gpu/rfc/i915_scheduler.rst b/Documentation/gpu/rfc/i915_scheduler.rst new file mode 100644 index ..7acd386a6b49 --- /dev/null +++ b/Documentation/gpu/rfc/i915_scheduler.rst @@ -0,0 +1,91 @@ += +I915 GuC Submission/DRM Scheduler Section += + +Upstream plan += +For upstream the overall plan for landing GuC submission and integrating the +i915 with the DRM scheduler is: + +* Merge basic GuC submission + * Basic submission support for all gen11+ platforms + * Not enabled by default on any current platforms but can be enabled via + modparam enable_guc + * Lots of rework will need to be done to integrate with DRM scheduler so + no need to nit pick everything in the code, it just should be + functional, no major coding style / layering errors, and not regress + execlists + * Update IGTs / selftests as needed to work with GuC submission + * Enable CI on supported platforms for a baseline + * Rework / get CI heathly for GuC submission in place as needed +* Merge new parallel submission uAPI + * Bonding uAPI completely incompatible with GuC submission, plus it has + severe design issues in general, which is why we want to retire it no + matter what + * New uAPI adds I915_CONTEXT_ENGINES_EXT_PARALLEL context setup step + which configures a slot with N contexts + * After I915_CONTEXT_ENGINES_EXT_PARALLEL a user can submit N batches to + a slot in a single execbuf IOCTL and the batches run on the GPU in + paralllel + * Initially only for GuC submission but execlists can be supported if + needed +* Convert the i915 to use the DRM scheduler + * GuC submission backend fully integrated with DRM scheduler + * All request queues removed from backend (e.g. all backpressure + handled in DRM scheduler) + * Resets / cancels hook in DRM scheduler + * Watchdog hooks into DRM scheduler + * Lots of complexity of the GuC backend can be pulled out once + integrated with DRM scheduler (e.g. state machine gets + simplier, locking gets simplier, etc...) + * Execlists backend will minimum required to hook in the DRM scheduler + * Legacy interface + * Features like timeslicing / preemption / virtual engines would + be difficult to integrate with the DRM scheduler and these + features are not required for GuC submission as the GuC does + these things for us + * ROI low on fully integrating into DRM scheduler + * Fully integrating would add lots of complexity to DRM + scheduler + * Port i915 priority inheritance / boosting feature in DRM scheduler + * Used for i915 page flip, may be useful to other DRM drivers as + well + * Will be an optional feature in the DRM scheduler + * Remove in-order completion assumptions from DRM scheduler + * Even when using the DRM scheduler the backends will handle + preemption, timeslicing, etc... so it is possible for jobs to + finish out of order + * Pull out i915 priority levels and use DRM priority levels + * Optimize DRM scheduler as needed + +TODOs for GuC submission upstream += + +* Need an update to GuC firmware / i915 to enable error state capture +* Open source tool to decode GuC logs +* Public GuC spec + +New uAPI for basic GuC submission += +No major changes are required to the uAPI for basic GuC submission. The only +change is a new scheduler attribute: I915_SCHEDULER_CAP_STATIC_PRIORITY_MAP. +This attr
[Mesa-dev] [PATCH 2/2] drm/doc/rfc: i915 new parallel submission uAPI plan
Add entry for i915 new parallel submission uAPI plan.
v2:
(Daniel Vetter):
- Expand logical order explaination
- Add dummy header
- Only allow N BBs in execbuf IOCTL
- Configure parallel submission per slot not per gem context
v3:
(Marcin Ślusarz):
- Lot's of typos / bad english fixed
(Tvrtko Ursulin):
- Consistent pseudo code, clean up wording in descriptions
v4:
(Daniel Vetter)
- Drop flags
- Add kernel doc
- Reword a few things / fix typos
(Tvrtko)
- Reword a few things / fix typos
Cc: Tvrtko Ursulin
Cc: Tony Ye
CC: Carl Zhang
Cc: Daniel Vetter
Cc: Jason Ekstrand
Signed-off-by: Matthew Brost
Acked-by: Daniel Vetter
---
Documentation/gpu/rfc/i915_parallel_execbuf.h | 117 ++
Documentation/gpu/rfc/i915_scheduler.rst | 59 -
2 files changed, 175 insertions(+), 1 deletion(-)
create mode 100644 Documentation/gpu/rfc/i915_parallel_execbuf.h
diff --git a/Documentation/gpu/rfc/i915_parallel_execbuf.h
b/Documentation/gpu/rfc/i915_parallel_execbuf.h
new file mode 100644
index ..c22af3a359e4
--- /dev/null
+++ b/Documentation/gpu/rfc/i915_parallel_execbuf.h
@@ -0,0 +1,117 @@
+#define I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT 2 /* see
i915_context_engines_parallel_submit */
+
+/**
+ * struct drm_i915_context_engines_parallel_submit - Configure engine for
+ * parallel submission.
+ *
+ * Setup a slot in the context engine map to allow multiple BBs to be submitted
+ * in a single execbuf IOCTL. Those BBs will then be scheduled to run on the
GPU
+ * in parallel. Multiple hardware contexts are created internally in the i915
+ * run these BBs. Once a slot is configured for N BBs only N BBs can be
+ * submitted in each execbuf IOCTL and this is implicit behavior e.g. The user
+ * doesn't tell the execbuf IOCTL there are N BBs, the execbuf IOCTL knows how
+ * many BBs there are based on the slot's configuration. The N BBs are the last
+ * N buffer objects or first N if I915_EXEC_BATCH_FIRST is set.
+ *
+ * The default placement behavior is to create implicit bonds between each
+ * context if each context maps to more than 1 physical engine (e.g. context is
+ * a virtual engine). Also we only allow contexts of same engine class and
these
+ * contexts must be in logically contiguous order. Examples of the placement
+ * behavior described below. Lastly, the default is to not allow BBs to
+ * preempted mid BB rather insert coordinated preemption on all hardware
+ * contexts between each set of BBs. Flags may be added in the future to change
+ * bott of these default behaviors.
+ *
+ * Returns -EINVAL if hardware context placement configuration is invalid or if
+ * the placement configuration isn't supported on the platform / submission
+ * interface.
+ * Returns -ENODEV if extension isn't supported on the platform / submission
+ * inteface.
+ *
+ * .. code-block::
+ *
+ * Example 1 pseudo code:
+ * CS[X] = generic engine of same class, logical instance X
+ * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE
+ * set_engines(INVALID)
+ * set_parallel(engine_index=0, width=2, num_siblings=1,
+ * engines=CS[0],CS[1])
+ *
+ * Results in the following valid placement:
+ * CS[0], CS[1]
+ *
+ * Example 2 pseudo code:
+ * CS[X] = generic engine of same class, logical instance X
+ * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE
+ * set_engines(INVALID)
+ * set_parallel(engine_index=0, width=2, num_siblings=2,
+ * engines=CS[0],CS[2],CS[1],CS[3])
+ *
+ * Results in the following valid placements:
+ * CS[0], CS[1]
+ * CS[2], CS[3]
+ *
+ * This can also be thought of as 2 virtual engines described by 2-D array
+ * in the engines the field with bonds placed between each index of the
+ * virtual engines. e.g. CS[0] is bonded to CS[1], CS[2] is bonded to
+ * CS[3].
+ * VE[0] = CS[0], CS[2]
+ * VE[1] = CS[1], CS[3]
+ *
+ * Example 3 pseudo code:
+ * CS[X] = generic engine of same class, logical instance X
+ * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE
+ * set_engines(INVALID)
+ * set_parallel(engine_index=0, width=2, num_siblings=2,
+ * engines=CS[0],CS[1],CS[1],CS[3])
+ *
+ * Results in the following valid and invalid placements:
+ * CS[0], CS[1]
+ * CS[1], CS[3] - Not logical contiguous, return -EINVAL
+ */
+struct drm_i915_context_engines_parallel_submit {
+ /**
+* @base: base user extension.
+*/
+ struct i915_user_extension base;
+
+ /**
+* @engine_index: slot for parallel engine
+*/
+ __u16 engine_index;
+
+ /**
+* @width: number of contexts per parallel engine
+*/
+ __u16 width;
+
+ /**
+* @num_siblings: number of siblings per context
+*/
+ __u16 num_siblings;
+
+ /**
+* @mbz16
Re: [Mesa-dev] [Intel-gfx] [RFC PATCH 2/2] drm/doc/rfc: i915 new parallel submission uAPI plan
On Thu, Jun 17, 2021 at 06:46:48PM +0200, Daniel Vetter wrote:
> Sorry I'm behind on mails ...
>
Aren't we all.
> On Fri, Jun 11, 2021 at 12:50:29PM -0700, Matthew Brost wrote:
> > On Fri, Jun 04, 2021 at 07:59:05PM +0200, Daniel Vetter wrote:
> > > On Wed, May 26, 2021 at 04:33:57PM -0700, Matthew Brost wrote:
> > > > Add entry for i915 new parallel submission uAPI plan.
> > > >
> > > > v2:
> > > > (Daniel Vetter):
> > > > - Expand logical order explaination
> > > > - Add dummy header
> > > > - Only allow N BBs in execbuf IOCTL
> > > > - Configure parallel submission per slot not per gem context
> > > > v3:
> > > > (Marcin Ślusarz):
> > > > - Lot's of typos / bad english fixed
> > > > (Tvrtko Ursulin):
> > > > - Consistent pseudo code, clean up wording in descriptions
> > > >
> > > > Cc: Tvrtko Ursulin
> > > > Cc: Tony Ye
> > > > CC: Carl Zhang
> > > > Cc: Daniel Vetter
> > > > Cc: Jason Ekstrand
> > > > Signed-off-by: Matthew Brost
> > > > ---
> > > > Documentation/gpu/rfc/i915_parallel_execbuf.h | 145 ++
> > > > Documentation/gpu/rfc/i915_scheduler.rst | 55 ++-
> > > > 2 files changed, 198 insertions(+), 2 deletions(-)
> > > > create mode 100644 Documentation/gpu/rfc/i915_parallel_execbuf.h
> > > >
> > > > diff --git a/Documentation/gpu/rfc/i915_parallel_execbuf.h
> > > > b/Documentation/gpu/rfc/i915_parallel_execbuf.h
> > > > new file mode 100644
> > > > index ..20de206e3ab4
> > > > --- /dev/null
> > > > +++ b/Documentation/gpu/rfc/i915_parallel_execbuf.h
> > > > @@ -0,0 +1,145 @@
> > > > +#define I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT 2 /* see
> > > > i915_context_engines_parallel_submit */
> > > > +
> > > > +/*
> > > > + * i915_context_engines_parallel_submit:
> > >
> > > So the idea is to make these kerneldoc and pull them into the rfc section.
> > > Then when we merge, move them to the real uapi section, like what Matt has
> > > done for lmem.
> > >
> >
> > Yep, will fix in next rev.
> >
> > > > + *
> > > > + * Setup a slot in the context engine map to allow multiple BBs to be
> > > > submitted
> > > > + * in a single execbuf IOCTL. Those BBs will then be scheduled to run
> > > > on the GPU
> > > > + * in parallel. Multiple hardware contexts are created internally in
> > > > the i915
> > > > + * run these BBs. Once a slot is configured for N BBs only N BBs can be
> > > > + * submitted in each execbuf IOCTL and this is implicit behavior e.g.
> > > > The user
> > > > + * doesn't tell the execbuf IOCTL there are N BBs, the execbuf IOCTL
> > > > know how
> > > > + * many BBs there are based on the slots configuration. The N BBs are
> > > > the last N
> > > > + * buffer objects for first N if I915_EXEC_BATCH_FIRST is set.
> > >
> > > s/for/or/
> > >
> > > > + *
> > > > + * There are two currently defined ways to control the placement of the
> > > > + * hardware contexts on physical engines: default behavior (no flags)
> > > > and
> > > > + * I915_PARALLEL_IMPLICIT_BONDS (a flag). More flags may be added the
> > > > in the
> > > > + * future as new hardware / use cases arise. Details of how to use this
> > > > + * interface above the flags field in this structure.
> > > > + *
> > > > + * Returns -EINVAL if hardware context placement configuration is
> > > > invalid or if
> > > > + * the placement configuration isn't supported on the platform /
> > > > submission
> > > > + * interface.
> > > > + * Returns -ENODEV if extension isn't supported on the platform /
> > > > submission
> > > > + * inteface.
> > > > + */
> > > > +struct i915_context_engines_parallel_submit {
> > > > + struct i915_user_extension base;
> > > > +
> > > > + __u16 engine_index; /* slot for parallel engine */
> > >
> > > Kernel doc here for the inline comments too.
> > >
> >
> > Yep.
> >
> > > > + __u16 width;/* number of context
