Re: [Mesa-dev] [ANNOUNCE] mesa 20.0.5
I want to sum up what happened from my perspective, I think it could be useful to improve the process: 1) There was a regression in 20.* (https://gitlab.freedesktop.org/mesa/mesa/-/issues/2758) 2) I "fixed" the regression and broke non-iris drivers (https://gitlab.freedesktop.org/mesa/mesa/-/commit/d684fb37bfbc47d098158cb03c0672119a4469fe) 3) I "fixed" the new regression of fix 2) (https://gitlab.freedesktop.org/mesa/mesa/-/commit/829013d0cad0fa2513b32ae07cf8d745f6e5c62d) 4) After that, it appeared that due to a bug in piglit, Intel CI didn't run piglit tests which gave me a false sense of commit's correctness (https://gitlab.freedesktop.org/mesa/mesa/-/issues/2815) 5) I aimed to have a fix before the next minor release on 2020-04-29 by consulting the release calendar. 6) I accidentally saw 20.0.5 being released with one of the two of my commits. I see multiple failure points: 1) Me not carefully examining all code paths, because at least one that failed should have been obvious to test. 2) CI not communicating that piglit tests were not executed. Again, I could have seen this, examined CI results, but did not. 3) After restoration of CI capabilities test what added to "expected failure" and this may have contributed to regression being missed when testing the release. I'm not sure about this part so correct me if I'm wrong. 4) I didn't know about this release and that this release was help up for the fix of 2758. 5) There were now window between announcing the scope of the release and release itself. Since I knew about regression I could have notified about it. Also there is no milestone for minor releases so it's problematic to link issue and release. It's a second release in a row with clear regression crept in. I believe that we can use this to improve the process and safeguard against such regressions in the future. P.S. I'm preparing and will test a final fix which will be sent soon. Danylo On 23.04.20 07:40, Dylan Baker wrote: Quoting Ilia Mirkin (2020-04-22 15:47:59) On Wed, Apr 22, 2020 at 6:39 PM Danylo Piliaiev wrote: I'm sorry for this trouble. However looking at it I think: maybe something could be changed about applying patches to stable to safeguard against such issues. We used to get pre-announcements a few days prior to a release which would allow developers to look things over, which would allow us to notice things like that. Not sure when this got dropped. Cheers, -ilia That was dropped in favor of a live staging branch that is updated daily (at least on week days). Dylan ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] [ANNOUNCE] mesa 20.0.5
Danylo Piliaiev writes: > I want to sum up what happened from my perspective, I think it could be > useful to improve the process: > > 1) There was a regression in 20.* > (https://gitlab.freedesktop.org/mesa/mesa/-/issues/2758) > 2) I "fixed" the regression and broke non-iris drivers > (https://gitlab.freedesktop.org/mesa/mesa/-/commit/d684fb37bfbc47d098158cb03c0672119a4469fe) > 3) I "fixed" the new regression of fix 2) > (https://gitlab.freedesktop.org/mesa/mesa/-/commit/829013d0cad0fa2513b32ae07cf8d745f6e5c62d) > 4) After that, it appeared that due to a bug in piglit, Intel CI didn't > run piglit tests which gave me a false sense of commit's correctness > (https://gitlab.freedesktop.org/mesa/mesa/-/issues/2815) > 5) I aimed to have a fix before the next minor release on 2020-04-29 by > consulting the release calendar. > 6) I accidentally saw 20.0.5 being released with one of the two of my > commits. > > I see multiple failure points: > 1) Me not carefully examining all code paths, because at least one that > failed should have been obvious to test. You missed one important failure point: 1a) untested piglit commits pushed to master which disabled the test suite. I was impressed by how quickly regressions were added to master in the few days that piglit was disabled in CI. At least 4 regressions in as many days. > 2) CI not communicating that piglit tests were not executed. Again, I > could have seen this, examined CI results, but did not. This was my fault. Over 5 years ago, I change the CI harness to ignore errors thrown by piglit when it is run on an empty test set. There are valid CI use cases (when bisecting) where the CI will attempt to run a small test set on older platforms that do not support any of the tests. A more defensive implementation would check that an empty test set is allowed only in the specific/expected use case. > 3) After restoration of CI capabilities test what added to "expected > failure" and this may have contributed to regression > being missed when testing the release. I'm not sure about this part > so correct me if I'm wrong. You are correct. CI cannot discern the context of a text failure automatically. Typically, test failures on the stable branch are not release blockers. Test failures which represent regressions are written up as gitlab issues and tracked there. Which brings up another failure point: 3a) Bisected regressions tagged with Fixes or mesa-stable are automatically applied to Mesa's release branch. This failure point has burned us many times, most recently with the 20.0 regression fixed by 2120f106e0e. Mesa currently has no mechanism for blocking a release with a gitlab issue. This current example is tagged with "bisected" and "regression", but the important distinction is that the bisected commit ALSO has tags which apply it to stable releases. Mesa's release process does not include a check of bugs that have been written up in gitlab (https://www.mesa3d.org/releasing.html). My own opinion is that gitlab's issues are unusable for this purpose, due to its lack of search functionality. I have found no way to audit gitlab issues leading up to a release. Gitlab's issues may work well for developing on master, but they are not as good as Bugzilla for managing releases. > 4) I didn't know about this release and that this release was help up > for the fix of 2758. > 5) There were now window between announcing the scope of the release and > release itself. Since I knew about regression > I could have notified about it. Also there is no milestone for minor > releases so it's problematic to link issue and release. > > It's a second release in a row with clear regression crept in. I believe > that we can use this to improve the process and > safeguard against such regressions in the future. Does anyone have recommendations for how to use Gitlab to verify that there are no identified-but-unfixed bugs in a pending release? > P.S. I'm preparing and will test a final fix which will be sent soon. > > Danylo > > On 23.04.20 07:40, Dylan Baker wrote: >> Quoting Ilia Mirkin (2020-04-22 15:47:59) >>> On Wed, Apr 22, 2020 at 6:39 PM Danylo Piliaiev >>> wrote: I'm sorry for this trouble. However looking at it I think: maybe something could be changed about applying patches to stable to safeguard against such issues. >>> We used to get pre-announcements a few days prior to a release which >>> would allow developers to look things over, which would allow us to >>> notice things like that. Not sure when this got dropped. >>> >>> Cheers, >>> >>>-ilia >> That was dropped in favor of a live staging branch that is updated daily (at >> least on week days). >> >> Dylan > > ___ > mesa-dev mailing list > mesa-dev@lists.freedesktop.org > https://lists.freedesktop.org/mailman/listinfo/mesa-dev ___ mesa-dev mailing list m
Re: [Mesa-dev] [ANNOUNCE] mesa 20.0.5
On 2020-04-23 5:14 p.m., Mark Janes wrote: > Danylo Piliaiev writes: > > 3a) Bisected regressions tagged with Fixes or mesa-stable are > automatically applied to Mesa's release branch. > > This failure point has burned us many times, most recently with the 20.0 > regression fixed by 2120f106e0e. > > Mesa currently has no mechanism for blocking a release with a gitlab > issue. This current example is tagged with "bisected" and "regression", > but the important distinction is that the bisected commit ALSO has tags > which apply it to stable releases. > > Mesa's release process does not include a check of bugs that have been > written up in gitlab (https://www.mesa3d.org/releasing.html). My own > opinion is that gitlab's issues are unusable for this purpose, due to > its lack of search functionality. I have found no way to audit gitlab > issues leading up to a release. > > Gitlab's issues may work well for developing on master, but they are not > as good as Bugzilla for managing releases. > >> 4) I didn't know about this release and that this release was help up >> for the fix of 2758. >> 5) There were now window between announcing the scope of the release and >> release itself. Since I knew about regression >> I could have notified about it. Also there is no milestone for minor >> releases so it's problematic to link issue and release. >> >> It's a second release in a row with clear regression crept in. I believe >> that we can use this to improve the process and >> safeguard against such regressions in the future. > > Does anyone have recommendations for how to use Gitlab to verify that > there are no identified-but-unfixed bugs in a pending release? I'd say GitLab milestones could be used to address the issues you raised above: Create a milestone for each release, and only cut the release once all issues and MRs assigned to it have been dealt with. -- Earthling Michel Dänzer | https://redhat.com Libre software enthusiast | Mesa and X developer ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] [ANNOUNCE] mesa 20.0.5
Michel Dänzer writes: > On 2020-04-23 5:14 p.m., Mark Janes wrote: >> >> Does anyone have recommendations for how to use Gitlab to verify that >> there are no identified-but-unfixed bugs in a pending release? > > I'd say GitLab milestones could be used to address the issues you raised > above: Create a milestone for each release, and only cut the release > once all issues and MRs assigned to it have been dealt with. Milestones have been used to track progress toward recent releases. It is non-trivial to audit all open bugs to determine which ones must be assigned to a milestone. Bugs are opened long before milestones are created. ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] [ANNOUNCE] mesa 20.0.5
On 2020-04-23 6:19 p.m., Mark Janes wrote: > Michel Dänzer writes: >> On 2020-04-23 5:14 p.m., Mark Janes wrote: >>> >>> Does anyone have recommendations for how to use Gitlab to verify that >>> there are no identified-but-unfixed bugs in a pending release? >> >> I'd say GitLab milestones could be used to address the issues you raised >> above: Create a milestone for each release, and only cut the release >> once all issues and MRs assigned to it have been dealt with. > > Milestones have been used to track progress toward recent releases. It > is non-trivial to audit all open bugs to determine which ones must be > assigned to a milestone. Bugs are opened long before milestones are > created. Of course there are more complicated cases, but the particular case which spawned this thread should have been pretty straightforward to handle with a 20.0.5 milestone. -- Earthling Michel Dänzer | https://redhat.com Libre software enthusiast | Mesa and X developer ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] [ANNOUNCE] mesa 20.0.5
On Thu, Apr 23, 2020 at 08:14:59AM -0700, Mark Janes wrote: Danylo Piliaiev writes: I want to sum up what happened from my perspective, I think it could be useful to improve the process: 1) There was a regression in 20.* (https://gitlab.freedesktop.org/mesa/mesa/-/issues/2758) 2) I "fixed" the regression and broke non-iris drivers (https://gitlab.freedesktop.org/mesa/mesa/-/commit/d684fb37bfbc47d098158cb03c0672119a4469fe) 3) I "fixed" the new regression of fix 2) (https://gitlab.freedesktop.org/mesa/mesa/-/commit/829013d0cad0fa2513b32ae07cf8d745f6e5c62d) 4) After that, it appeared that due to a bug in piglit, Intel CI didn't run piglit tests which gave me a false sense of commit's correctness (https://gitlab.freedesktop.org/mesa/mesa/-/issues/2815) 5) I aimed to have a fix before the next minor release on 2020-04-29 by consulting the release calendar. 6) I accidentally saw 20.0.5 being released with one of the two of my commits. I see multiple failure points: 1) Me not carefully examining all code paths, because at least one that failed should have been obvious to test. You missed one important failure point: 1a) untested piglit commits pushed to master which disabled the test suite. I was impressed by how quickly regressions were added to master in the few days that piglit was disabled in CI. At least 4 regressions in as many days. 2) CI not communicating that piglit tests were not executed. Again, I could have seen this, examined CI results, but did not. This was my fault. Over 5 years ago, I change the CI harness to ignore errors thrown by piglit when it is run on an empty test set. There are valid CI use cases (when bisecting) where the CI will attempt to run a small test set on older platforms that do not support any of the tests. A more defensive implementation would check that an empty test set is allowed only in the specific/expected use case. I fixed the (intel) CI by implementing a check for this (and thanks Dylan for the piglit patch to help facilitate this check), all other piglit failures should be immediate and visible from now on. -Clayton signature.asc Description: PGP signature ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] [ANNOUNCE] mesa 20.0.5
Quoting Michel Dänzer (2020-04-23 09:24:45) > On 2020-04-23 6:19 p.m., Mark Janes wrote: > > Michel Dänzer writes: > >> On 2020-04-23 5:14 p.m., Mark Janes wrote: > >>> > >>> Does anyone have recommendations for how to use Gitlab to verify that > >>> there are no identified-but-unfixed bugs in a pending release? > >> > >> I'd say GitLab milestones could be used to address the issues you raised > >> above: Create a milestone for each release, and only cut the release > >> once all issues and MRs assigned to it have been dealt with. > > > > Milestones have been used to track progress toward recent releases. It > > is non-trivial to audit all open bugs to determine which ones must be > > assigned to a milestone. Bugs are opened long before milestones are > > created. > > Of course there are more complicated cases, but the particular case > which spawned this thread should have been pretty straightforward to > handle with a 20.0.5 milestone. > Indeed, I like the milestone approach and I've gone ahead and created the milestone. Dylan signature.asc Description: signature ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
[Mesa-dev] [Announce] Milestones for each stable release
Due to a recent spat of regressions in the mesa stable branch and some miscommunication about the status of these issues, we're going to experiment with using a gitlab milestone for each stable release. If there are issues or MRs that need to be resolved before the next release please ensure that they're on that milestone. A milestone for 20.0.6 (the next scheduled release) is opened here: https://gitlab.freedesktop.org/mesa/mesa/-/milestones/13 Thanks, Dylan signature.asc Description: signature ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] [ANNOUNCE] mesa 20.0.5
On 23.04.20 19:56, Dylan Baker wrote: Quoting Michel Dänzer (2020-04-23 09:24:45) On 2020-04-23 6:19 p.m., Mark Janes wrote: Michel Dänzer writes: On 2020-04-23 5:14 p.m., Mark Janes wrote: Does anyone have recommendations for how to use Gitlab to verify that there are no identified-but-unfixed bugs in a pending release? I'd say GitLab milestones could be used to address the issues you raised above: Create a milestone for each release, and only cut the release once all issues and MRs assigned to it have been dealt with. Milestones have been used to track progress toward recent releases. It is non-trivial to audit all open bugs to determine which ones must be assigned to a milestone. Bugs are opened long before milestones are created. Of course there are more complicated cases, but the particular case which spawned this thread should have been pretty straightforward to handle with a 20.0.5 milestone. Indeed, I like the milestone approach and I've gone ahead and created the milestone. Yes, even if milestones don't alleviate all issues - at least it is better than nothing, thanks! ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] [ANNOUNCE] mesa 20.0.5
While we talked about the general communication issues, I want to finish with this particular one. Is this release going out as normal and the fix will be in the next 20.0.6 on 2020-04-29? The current scope of the regression is non-iris drivers or iris compiled without llvm. The regression will result in crash of ARB_vp shaders which don't use at least one vertex attribute or one particular case of fixed-function shader. On the brighter side it seams there weren't many such cases in the wild, we saw only UT2004. Danylo On 23.04.20 07:45, Dylan Baker wrote: This is particularly problematic, as that solved a regression in 20.0.4, and we held up the release for that fix. The second patch is in the staging branch and I can do an emergency release, but if there's another patch that's needed then that doesn't help. I'm not sure that reverting the original offending patch is an option either. Dylan Quoting Danylo Piliaiev (2020-04-22 15:38:59) My commit "st/mesa: Update shader info of ffvp/ARB_vp after translation to NIR" did introduce regression and its fix, which was merged (but isn't in this release even though there is "fixes" tag), didn't fully fix it. See https://gitlab.freedesktop.org/mesa/mesa/-/issues/2815 This commit may break usage of fixed function and ARB_vp shaders. I thought that I would fix it until the next release and I wasn't expected this one. I'm sorry for this trouble. However looking at it I think: maybe something could be changed about applying patches to stable to safeguard against such issues. On 23.04.20 00:54, Dylan Baker wrote: Hi list, I'd like to announce the availability of mesa 20.0.5. It's one week late due to a number of issues, including a regression in mesa, a regression in piglit, and some CI trouble. I'm still planning to make 20.0.6 next week at it's secheduled time. This is a pretty big release, as it contiains 3 weeks rather than just two weeks of changes. We have the normal spattering of changes, with the AMD and Intel drivers receiving the majority of the work. Dylan Shortlog Arcady Goldmints-Orlov (1): nir: Lower returns correctly inside nested loops Bas Nieuwenhuizen (3): radv: Store 64-bit availability bools if requested. radv: Consider maximum sample distances for entire grid. radv: Use correct buffer count with variable descriptor set sizes. D Scott Phillips (1): util/sparse_array: don't stomp head's counter on pop operations Daniel Stone (1): EGL: Add eglSetDamageRegionKHR to GLVND dispatch list Danylo Piliaiev (1): st/mesa: Update shader info of ffvp/ARB_vp after translation to NIR Dave Airlie (2): draw: free the NIR IR. llvmpipe/nir: free the nir shader Dylan Baker (8): .pick_status.json: Update to 089e1fb287eb9b70c191091128ed5ba7edd2960a .pick_status.json: Update to 65e2eaa4d3a7095ac438fafb09d1e36a4210966e .pick_status.json: Update to 28d36d26c2212276e1238fad8f0b12caab97fee8 .pick_status.json: Update to acf7e73be54c7f1cc52fcc9be38a9df26849200e .pick_status.json: Update to 13ce637f1b28381e72470763ff5e39dd3c562476 .pick_status.json: Update to c3c1f4d6bcc210408f8b180727d269838b38193b docs: Add relnotes for 20.0.5 VERSION: bump for 20.0.5 Emil Velikov (4): glx: set the loader_logger early and for everyone egl/drm: reinstate (kms_)swrast support Revert "egl/dri2: Don't dlclose() the driver on dri2_load_driver_common failure" glx: omit loader_loader() for macOS Eric Anholt (1): ci: Remove LLVM from ARM test drivers. Eric Engestrom (1): docs/relnotes: add sha256sum for 20.0.4 Hyunjun Ko (1): nir: fix wrong assignment to buffer in xfb_varyings_info Ilia Mirkin (1): nv50: don't try to upload MSAA settings for BUFFER textures Jason Ekstrand (5): anv/image: Use align_u64 for image offsets nir/load_store_vectorize: Fix shared atomic info spirv: Handle OOB vector extract operations intel: Add _const versions of prog_data cast helpers anv: Report correct SLM size Jose Maria Casanova Crespo (1): v3d: Primitive Counts Feedback needs an extra 32-bit padding. Juan A. Suarez Romero (2): intel/compiler: store the FS inputs in WM prog data anv/pipeline: allow more than 16 FS inputs Karol Herbst (2): clover: fix build with single library clang build Revert "nvc0: fix line width on GM20x+" Lionel Landwerlin (7): iris: properly free resources on BO allocation failure iris: share buffer managers accross screens iris: make resources take a ref on the screen object i965:
Re: [Mesa-dev] [ANNOUNCE] mesa 20.0.5
Michel Dänzer writes: > On 2020-04-23 6:19 p.m., Mark Janes wrote: >> Michel Dänzer writes: >>> On 2020-04-23 5:14 p.m., Mark Janes wrote: Does anyone have recommendations for how to use Gitlab to verify that there are no identified-but-unfixed bugs in a pending release? >>> >>> I'd say GitLab milestones could be used to address the issues you raised >>> above: Create a milestone for each release, and only cut the release >>> once all issues and MRs assigned to it have been dealt with. >> >> Milestones have been used to track progress toward recent releases. It >> is non-trivial to audit all open bugs to determine which ones must be >> assigned to a milestone. Bugs are opened long before milestones are >> created. > > Of course there are more complicated cases, but the particular case > which spawned this thread should have been pretty straightforward to > handle with a 20.0.5 milestone. I do not agree that release milestones are helpful for this category of bugs. The majority of stable point releases will have zero issues in a release milestone. Opening/closing empty milestones all the time is a lot of busy work. Milestones are helpful for *initial* releases of stable branches, not point releases. Even for initial releases important use cases for milestones are not supported by gitlab. As an example, we may be able to verify that a specific test is regressed with an A/B test of the previous release -- and perhaps identify the commit that regressed it. How can we find if an issue exists for this test? You cannot: - search for issues mentioning a test name (unless it is in the title) - search for issues mentioning a commit - subscribe to issue comments in a way that would let you search offline How can we audit new issues for items that have not been triaged since the last release? The only workflow that I can imagine is to sort all issues by "last updated" and read through every open issue changed in the past 3 months. Of course, the list changes as you read through it. I'm contrasting this with Bugzilla, where we could subscribe to bug comments and read through them on a daily basis. At release time, I could have confidence that I had seen all bugs that might affect end users. The labels "bisected" and "regression" serve as a good indicator of bugs that should block release, assuming someone has by chance applied the labels correctly. For point releases, adding a "stable" label may tell us when an issue blocks point releases as well. Any issue related to a commit that CC's stable would get this label. Personally I think this label will not solve the problems either, because it requires careful monitoring of issues to notice regressions which cc stable. The information needed to *automate verification* of stable releases exists, within gitlab and the git log. However, a sane and robust release process cannot be implemented on top of gitlab's pitiful search interface. ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] [ANNOUNCE] mesa 20.0.5
I don't think this was preventable given how many things went wrong. Sometimes we have to accept that shit happens. We can fix that in the next release. Marek On Thu., Apr. 23, 2020, 07:01 Danylo Piliaiev, wrote: > I want to sum up what happened from my perspective, I think it could be > useful to improve the process: > > 1) There was a regression in 20.* > (https://gitlab.freedesktop.org/mesa/mesa/-/issues/2758) > 2) I "fixed" the regression and broke non-iris drivers > ( > https://gitlab.freedesktop.org/mesa/mesa/-/commit/d684fb37bfbc47d098158cb03c0672119a4469fe > ) > 3) I "fixed" the new regression of fix 2) > ( > https://gitlab.freedesktop.org/mesa/mesa/-/commit/829013d0cad0fa2513b32ae07cf8d745f6e5c62d > ) > 4) After that, it appeared that due to a bug in piglit, Intel CI didn't > run piglit tests which gave me a false sense of commit's correctness >(https://gitlab.freedesktop.org/mesa/mesa/-/issues/2815) > 5) I aimed to have a fix before the next minor release on 2020-04-29 by > consulting the release calendar. > 6) I accidentally saw 20.0.5 being released with one of the two of my > commits. > > I see multiple failure points: > 1) Me not carefully examining all code paths, because at least one that > failed should have been obvious to test. > 2) CI not communicating that piglit tests were not executed. Again, I > could have seen this, examined CI results, but did not. > 3) After restoration of CI capabilities test what added to "expected > failure" and this may have contributed to regression > being missed when testing the release. I'm not sure about this part > so correct me if I'm wrong. > 4) I didn't know about this release and that this release was help up > for the fix of 2758. > 5) There were now window between announcing the scope of the release and > release itself. Since I knew about regression > I could have notified about it. Also there is no milestone for minor > releases so it's problematic to link issue and release. > > It's a second release in a row with clear regression crept in. I believe > that we can use this to improve the process and > safeguard against such regressions in the future. > > P.S. I'm preparing and will test a final fix which will be sent soon. > > Danylo > > On 23.04.20 07:40, Dylan Baker wrote: > > Quoting Ilia Mirkin (2020-04-22 15:47:59) > >> On Wed, Apr 22, 2020 at 6:39 PM Danylo Piliaiev > >> wrote: > >>> I'm sorry for this trouble. However looking at it I think: maybe > something could be > >>> changed about applying patches to stable to safeguard against such > issues. > >> We used to get pre-announcements a few days prior to a release which > >> would allow developers to look things over, which would allow us to > >> notice things like that. Not sure when this got dropped. > >> > >> Cheers, > >> > >>-ilia > > That was dropped in favor of a live staging branch that is updated daily > (at > > least on week days). > > > > Dylan > > ___ > mesa-dev mailing list > mesa-dev@lists.freedesktop.org > https://lists.freedesktop.org/mailman/listinfo/mesa-dev > ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
