Re: Mesa >= 23.3.x and python 2.6 ...
On 2024-01-19 12:24:24, Stefan Dirsch wrote:
> Hi Jordan
>
> Thanks for digging into this!
>
> On Fri, Jan 19, 2024 at 12:10:37PM -0800, Jordan Justen wrote:
> > On 2024-01-18 04:37:52, Stefan Dirsch wrote:
> > > Hi
> > >
> > > I noticed that with version 23.3.x Mesa no longer can be built with python
> > > 2.6. It still worked with Mesa 23.2.1.
> >
> > As mentioned in other emails, this was typo where 3.6 was intended.
> >
> > >
> > > It fails with
> > >
> > > [ 95s] Traceback (most recent call last):
> > > [ 95s] File "../src/intel/genxml/gen_bits_header.py", line 23, in
> > >
> > > [ 95s] import intel_genxml
> > > [ 95s] File
> > > "/home/abuild/rpmbuild/BUILD/mesa-23.3.3/src/intel/genxml/intel_
> > > genxml.py", line 5
> > > [ 95s] from __future__ import annotations
> > > [ 95s] ^
> > > [ 95s] SyntaxError: future feature annotations is not defined
> > >
> >
> > I guess this code first appeared in Dylan's:
> >
> > 4fd2e15855d ("intel/genxml: add type annotations to gen_sort_tags.py")
> >
> > and then became part of the standard tests a few commits later in:
> >
> > 1f0a0a46d97 ("meson: run genxml sort tests")
> >
> > back in Oct 2022. So, I guess at that point 'ninja test' would have
> > failed with Python 3.6.
> >
> > Then, I suppose I propagated this to being used on every build in:
> >
> > 0495f952d48 ("intel/genxml: Add genxml_import.py script")
> >
> > in Sept 2023.
>
> Thanks. This explains why I've found this code already in older releases, but
> it didn't fail for me yet. You said tests. Is this just a test, I could
> disable (as dirty hack)? I was assuming it would generate code ...
In 0495f952d48, I moved in to a common file, and essentially, now it's
used by our script that runs during the build in addition to the test.
It was "fun" finding a way to get python 3.6 :), but after that, I
think I found a way to make Python 3.6 work. I guess you can try it
out:
https://gitlab.freedesktop.org/jljusten/mesa/-/commits/intel-genxml-python3.6
In my light testing, Python 3.6 through 3.13 seemed to work. Python
3.5 did *not* work.
-Jordan
Re: Mesa >= 23.3.x and python 2.6 ...
Hi Jordan
On Sat, Jan 20, 2024 at 01:47:58AM -0800, Jordan Justen wrote:
> On 2024-01-19 12:24:24, Stefan Dirsch wrote:
> > Hi Jordan
> >
> > Thanks for digging into this!
> >
> > On Fri, Jan 19, 2024 at 12:10:37PM -0800, Jordan Justen wrote:
> > > On 2024-01-18 04:37:52, Stefan Dirsch wrote:
> > > > Hi
> > > >
> > > > I noticed that with version 23.3.x Mesa no longer can be built with
> > > > python
> > > > 2.6. It still worked with Mesa 23.2.1.
> > >
> > > As mentioned in other emails, this was typo where 3.6 was intended.
> > >
> > > >
> > > > It fails with
> > > >
> > > > [ 95s] Traceback (most recent call last):
> > > > [ 95s] File "../src/intel/genxml/gen_bits_header.py", line 23, in
> > > >
> > > > [ 95s] import intel_genxml
> > > > [ 95s] File
> > > > "/home/abuild/rpmbuild/BUILD/mesa-23.3.3/src/intel/genxml/intel_
> > > > genxml.py", line 5
> > > > [ 95s] from __future__ import annotations
> > > > [ 95s] ^
> > > > [ 95s] SyntaxError: future feature annotations is not defined
> > > >
> > >
> > > I guess this code first appeared in Dylan's:
> > >
> > > 4fd2e15855d ("intel/genxml: add type annotations to gen_sort_tags.py")
> > >
> > > and then became part of the standard tests a few commits later in:
> > >
> > > 1f0a0a46d97 ("meson: run genxml sort tests")
> > >
> > > back in Oct 2022. So, I guess at that point 'ninja test' would have
> > > failed with Python 3.6.
> > >
> > > Then, I suppose I propagated this to being used on every build in:
> > >
> > > 0495f952d48 ("intel/genxml: Add genxml_import.py script")
> > >
> > > in Sept 2023.
> >
> > Thanks. This explains why I've found this code already in older releases,
> > but
> > it didn't fail for me yet. You said tests. Is this just a test, I could
> > disable (as dirty hack)? I was assuming it would generate code ...
>
> In 0495f952d48, I moved in to a common file, and essentially, now it's
> used by our script that runs during the build in addition to the test.
>
> It was "fun" finding a way to get python 3.6 :), but after that, I
> think I found a way to make Python 3.6 work. I guess you can try it
> out:
>
> https://gitlab.freedesktop.org/jljusten/mesa/-/commits/intel-genxml-python3.6
>
> In my light testing, Python 3.6 through 3.13 seemed to work. Python
> 3.5 did *not* work.
Wow! Thanks a lot! Indeed these two patches fix this build issue for me! :-)
In addition I needed to add the attached patch to fix some more errors I saw
with a python 3.6 build. With that I can build again Mesa 23.3.3.
Thanks,
Stefan
Public Key available
--
Stefan Dirsch (Res. & Dev.) SUSE Software Solutions Germany GmbH
Tel: 0911-740 53 0Frankenstraße 146
FAX: 0911-740 53 479 D-90461 Nürnberg
http://www.suse.deGermany
Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman
(HRB 36809, AG Nürnberg)
--- mesa-23.3.3/src/nouveau/nvidia-headers/class_parser.py.orig 2024-01-20 12:47:36.464476669 +0100
+++ mesa-23.3.3/src/nouveau/nvidia-headers/class_parser.py 2024-01-20 13:04:30.212851443 +0100
@@ -9,6 +9,16 @@
from mako.template import Template
+def removeprefix(s, prefix):
+if s.startswith(prefix):
+return s[len(prefix):]
+return s
+
+def removesuffix(s, suffix):
+if s.endswith(suffix):
+return s[:-len(suffix)]
+return s
+
METHOD_ARRAY_SIZES = {
'BIND_GROUP_CONSTANT_BUFFER': 16,
'CALL_MME_DATA' : 256,
@@ -273,7 +283,7 @@
if ":" in list[2]:
state = 1
elif teststr in list[1]:
-curmthd.field_defs[curfield][list[1].removeprefix(teststr)] = list[2]
+curmthd.field_defs[curfield][removeprefix(list[1], teststr)] = list[2]
else:
state = 1
@@ -283,7 +293,7 @@
if ("0x" in list[2]):
state = 1
else:
-field = list[1].removeprefix(teststr)
+field = removeprefix(list[1], teststr)
bitfield = list[2].split(":")
curmthd.field_name_start[field] = bitfield[1]
curmthd.field_name_end[field] = bitfield[0]
@@ -304,13 +314,13 @@
is_array = 0
if (':' in list[2]):
continue
-name = list[1].removeprefix(teststr)
+name = removeprefix(list[1], teststr)
if name.endswith("(i)"):
is_array = 1
-name = name.removesuffix("(i)")
+name = removesuffix(name, "(i)")
if name.endswith("(j)"):
is_array = 1
-name = na
Re: Future direction of the Mesa Vulkan runtime (or "should we build a new gallium?")
Hello Faith and everyfrogy! I've been developing a new Vulkan driver for Mesa — Terakan, for AMD TeraScale Evergreen and Northern Islands GPUs — since May of 2023. You can find it in amd/terascale/vulkan on the Terakan branch of my fork at Triang3l/mesa. While it currently lacks many of the graphical features, the architecture of state management, meta, and descriptors, has already largely been implemented in its code. I'm overall relatively new to Mesa, in the past having contributed the fragment shader interlock implementation to RADV that included working with the state management, but never having written a Gallium driver, or a Vulkan driver in the ANV copy-pasting era, so this may be a somewhat fresh — although quite conservative — take on this. Due to various hardware and kernel driver differences (bindings being individually loaded into fixed slots as part of the command buffer state, the lack of command buffer chaining in the kernel resulting in having to reapply all of the state when the size of the hardware command buffer exceeds the HW/KMD limits), I've been designing the architecture of my Vulkan driver largely from scratch, without using the existing Mesa drivers as a reference. Unfortunately, it seems like we ended up going in fundamentally opposite directions in our designs, so I'd say that I'm much more scared about this approach than I am excited about it. My primary concerns about this architecture can be summarized into two categories: • The obligation to manage pipeline and dynamic state in the common representation — essentially mostly the same Vulkan function call arguments, but with an additional layer for processing pNext and merging pipeline and dynamic state — restricts the abilities of drivers to optimize state management for specific hardware. Most importantly, it hampers precompiling of state in pipeline objects. In state management, this would make Mesa Vulkan implementations closer not even to Gallium, but to the dreaded OpenGL. • Certain parts of the common code are designed around assumptions about the majority of the hardware, however some devices may have large architectural differences in specific areas, and trying to adapt the way of programming such hardware subsystems results in having to write suboptimal algorithms, as well as sometimes artificially restricting the VkPhysicalDeviceLimits the device can report. An example from my driver is the meaning of a pipeline layout on fixed-slot TeraScale. Because it uses flat binding indices throughout all sets (sets don't exist in the hardware at all), it needs base offsets for each set within the stage's bindings — which are precomputed at pipeline layout creation. This is fundamentally incompatible with MR !27024's direction to remove the concept of a pipeline layout — and if the common vkCmdBindDescriptorSets makes the VK_KHR_maintenance6 layout-object-less path the only available one, it would add a lot of overhead by making it necessary to recompute the offsets at every bind. I think what we need to consider about pipeline state (in the broader sense, including both state objects and dynamic state) is that it inherently has very different properties from anything the common runtime already covers. What most of the current objects in the common runtime have in common is that they: • Are largely hardware-independent and can work everywhere the same way. • Either: • Provide a complex solution to a large-scale problem, essentially being sort of advanced "middleware". Examples are WSI, synchronization, pipeline cache, secondary command buffer emulation, render pass emulation. • Or, solve a trivial task in a way that's non-intrusive towards algorithms employed by the drivers — such as managing object handles, invoking allocators, reference-counting descriptor set and pipeline layouts, pooling VkCommandBuffer instances. • Rarely influence the design of "hot path" functions, such as changes to pipeline state and bindings. On the other hand, pipeline state: 1. Is entirely hardware-specific. 2. Is modified very frequently — making up the majority of command buffer recording time. 3. Can be precompiled in pipeline objects — and that's highly desirable due to the previous point. Because of 1, there's almost nothing in the pipeline state that the common runtime can help share between drivers. Yes, it can potentially be used to automate running some NIR passes for baking static state into shaders, but currently it looks like the runtime is going in a somewhat different direction, and that needs only some helper functions invoked at pipeline creation time. Aside from that, I can't see it being able to be useful for anything other than merging static and dynamic state into a single structure. For drivers where developers would prefer this approach for various reasons (prototyping simplicity, or staying at the near-original-Vulkan level of abstract
