Re: [Mesa-dev] gitlab.fd.o financial situation and impact on services
On Friday 2020-02-28 08:59, Daniel Stone wrote: > >I believe that in January, we had $2082 of network cost (almost >entirely egress; ingress is basically free) and $1750 of >cloud-storage cost (almost all of which was download). That's based >on 16TB of cloud-storage (CI artifacts, container images, file >uploads, Git LFS) egress and 17.9TB of other egress (the web service >itself, repo activity). Projecting that out [×12 for a year] gives >us roughly $45k of network activity alone, I had come to a similar conclusion a few years back: It is not very economic to run ephemereal buildroots (and anything like it) between two (or more) "significant locations" of which one end is located in a Large Cloud datacenter like EC2/AWS/etc. As for such usecases, me and my surrounding peers have used (other) offerings where there is 50 TB free network/month, and yes that may have entailed doing more adminning than elsewhere - but an admin appreciates $2000 a lot more than a corporation, too. ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] [Intel-gfx] gitlab.fd.o financial situation and impact on services
Hi All, I know there's been a lot of discussion already, but I wanted to respond to Daniel's original post. I joined GitLab earlier this month as their new Open Source Program Manager [1] and wanted to introduce myself here since I’ll be involved from the GitLab side as we work together to problem-solve the financial situation here. My role at GitLab is to help make it easier for Open Source organizations to migrate (by helping to smooth out some of the current pain points), and to help advocate internally for changes to the product and our workflows to make GitLab better for Open Source orgs. We want to make sure that our Open Source community feels supported beyond just migration. As such, I’ll be running the GitLab Open Source Program [2]. My background is that I’m the former President and Chairperson of the GNOME Foundation, which is one of the earliest Free Software projects to migrate to GitLab. GNOME initially faced some limitations with the CI runner costs too, but thanks to generous support from donors, has no longer experienced those issues in recent times. I know there's already a working relationship between our communities, but it could be good to examine what GNOME and KDE have done and see if there's anything we can apply here. We've reached out to Daniel Stone, our main contact for the freedesktop.org migration, and he has gotten us in touch with Daniel V. and the X.Org Foundation Board to learn more about what's already been done and what we can do next. Please bear with me as I continue to get ramped up in my new job, but I’d like to offer as much support as possible with this issue. We’ll be exploring ways for GitLab to help make sure there isn’t a gap in coverage during the time that freedesktop looks for sponsors. I know that on GitLab’s side, supporting our Open Source user community is a priority. Best, Nuritzi [1] https://about.gitlab.com/company/team/#nuritzi [2] https://about.gitlab.com/handbook/marketing/community-relations/opensource-program/ On Fri, Feb 28, 2020 at 1:22 PM Daniel Vetter wrote: > On Fri, Feb 28, 2020 at 9:31 PM Dave Airlie wrote: > > > > On Sat, 29 Feb 2020 at 05:34, Eric Anholt wrote: > > > > > > On Fri, Feb 28, 2020 at 12:48 AM Dave Airlie > wrote: > > > > > > > > On Fri, 28 Feb 2020 at 18:18, Daniel Stone > wrote: > > > > > > > > > > On Fri, 28 Feb 2020 at 03:38, Dave Airlie > wrote: > > > > > > b) we probably need to take a large step back here. > > > > > > > > > > > > Look at this from a sponsor POV, why would I give X.org/fd.o > > > > > > sponsorship money that they are just giving straight to google > to pay > > > > > > for hosting credits? Google are profiting in some minor way from > these > > > > > > hosting credits being bought by us, and I assume we aren't > getting any > > > > > > sort of discounts here. Having google sponsor the credits costs > google > > > > > > substantially less than having any other company give us money > to do > > > > > > it. > > > > > > > > > > The last I looked, Google GCP / Amazon AWS / Azure were all pretty > > > > > comparable in terms of what you get and what you pay for them. > > > > > Obviously providers like Packet and Digital Ocean who offer > bare-metal > > > > > services are cheaper, but then you need to find someone who is > going > > > > > to properly administer the various machines, install decent > > > > > monitoring, make sure that more storage is provisioned when we need > > > > > more storage (which is basically all the time), make sure that the > > > > > hardware is maintained in decent shape (pretty sure one of the fd.o > > > > > machines has had a drive in imminent-failure state for the last few > > > > > months), etc. > > > > > > > > > > Given the size of our service, that's a much better plan (IMO) than > > > > > relying on someone who a) isn't an admin by trade, b) has a million > > > > > other things to do, and c) hasn't wanted to do it for the past > several > > > > > years. But as long as that's the resources we have, then we're > paying > > > > > the cloud tradeoff, where we pay more money in exchange for fewer > > > > > problems. > > > > > > > > Admin for gitlab and CI is a full time role anyways. The system is > > > > definitely not self sustaining without time being put in by you and > > > > anholt still. If we have $75k to burn on credits, and it was diverted > > > > to just pay an admin to admin the real hw + gitlab/CI would that not > > > > be a better use of the money? I didn't know if we can afford $75k for > > > > an admin, but suddenly we can afford it for gitlab credits? > > > > > > As I think about the time that I've spent at google in less than a > > > year on trying to keep the lights on for CI and optimize our > > > infrastructure in the current cloud environment, that's more than the > > > entire yearly budget you're talking about here. Saying "let's just > > > pay for people to do more work instead of paying for full-service > > > cloud" is not a cost optimizatio
Re: [Mesa-dev] [Intel-gfx] gitlab.fd.o financial situation and impact on services
On Fri, Feb 28, 2020 at 11:00 AM Rob Clark wrote: > > On Fri, Feb 28, 2020 at 3:43 AM Michel Dänzer wrote: > > > > On 2020-02-28 10:28 a.m., Erik Faye-Lund wrote: > > > > > > We could also do stuff like reducing the amount of tests we run on each > > > commit, and punt some testing to a per-weekend test-run or someting > > > like that. We don't *need* to know about every problem up front, just > > > the stuff that's about to be released, really. The other stuff is just > > > nice to have. If it's too expensive, I would say drop it. > > > > I don't agree that pre-merge testing is just nice to have. A problem > > which is only caught after it lands in mainline has a much bigger impact > > than one which is already caught earlier. > > > > one thought.. since with mesa+margebot we effectively get at least > two(ish) CI runs per MR, ie. one when it is initially pushed, and one > when margebot rebases and tries to merge, could we leverage this to > have trimmed down pre-margebot CI which tries to just target affected > drivers, with margebot doing a full CI run (when it is potentially > batching together multiple MRs)? > > Seems like a way to reduce our CI runs with a good safety net to > prevent things from slipping through the cracks. Here are a couple more hopefully constructive but possibly bogus ideas: 1. Suggest people put their CI farms behind a squid transparent caching proxy. There seem to be many HowTo's on the internet for doing this and it shouldn't be terribly hard. Maybe GitLab uses too much HTTPS and that messes things up? If not, this would cut downloads to one-per-farm rather than one-per-machine 2. Add -Dstrip=true to the meson config. We want asserts but do we really need those debug symbols? Quick testing on my machine, it seems to reduce the size of build artifacts by about 60% Feel free to tell the peanut gallery (me) why I'm wrong. :-) --Jason ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] [Intel-gfx] gitlab.fd.o financial situation and impact on services
On Fri, 2020-02-28 at 10:43 +, Daniel Stone wrote: > On Fri, 28 Feb 2020 at 10:06, Erik Faye-Lund > wrote: > > On Fri, 2020-02-28 at 11:40 +0200, Lionel Landwerlin wrote: > > > Yeah, changes on vulkan drivers or backend compilers should be > > > fairly > > > sandboxed. > > > > > > We also have tools that only work for intel stuff, that should > > > never > > > trigger anything on other people's HW. > > > > > > Could something be worked out using the tags? > > > > I think so! We have the pre-defined environment variable > > CI_MERGE_REQUEST_LABELS, and we can do variable conditions: > > > > https://docs.gitlab.com/ee/ci/yaml/#onlyvariablesexceptvariables > > > > That sounds like a pretty neat middle-ground to me. I just hope > > that > > new pipelines are triggered if new labels are added, because not > > everyone is allowed to set labels, and sometimes people forget... > > There's also this which is somewhat more robust: > https://gitlab.freedesktop.org/mesa/mesa/merge_requests/2569 My 20 cents: 1. I think we should completely disable running the CI on MRs which are marked WIP. Speaking from personal experience, I usually make a lot of changes to my MRs before they are merged, so it is a waste of CI resources. 2. Maybe we could take this one step further and only allow the CI to be only triggered manually instead of automatically on every push. 3. I completely agree with Pierre-Eric on MR 2569, let's not run the full CI pipeline on every change, only those parts which are affected by the change. It not only costs money, but is also frustrating when you submit a change and you get unrelated failures from a completely unrelated driver. Best regards, Timur ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] [Intel-gfx] gitlab.fd.o financial situation and impact on services
Le samedi 29 février 2020 à 19:14 +0100, Timur Kristóf a écrit : > On Fri, 2020-02-28 at 10:43 +, Daniel Stone wrote: > > On Fri, 28 Feb 2020 at 10:06, Erik Faye-Lund > > wrote: > > > On Fri, 2020-02-28 at 11:40 +0200, Lionel Landwerlin wrote: > > > > Yeah, changes on vulkan drivers or backend compilers should be > > > > fairly > > > > sandboxed. > > > > > > > > We also have tools that only work for intel stuff, that should > > > > never > > > > trigger anything on other people's HW. > > > > > > > > Could something be worked out using the tags? > > > > > > I think so! We have the pre-defined environment variable > > > CI_MERGE_REQUEST_LABELS, and we can do variable conditions: > > > > > > https://docs.gitlab.com/ee/ci/yaml/#onlyvariablesexceptvariables > > > > > > That sounds like a pretty neat middle-ground to me. I just hope > > > that > > > new pipelines are triggered if new labels are added, because not > > > everyone is allowed to set labels, and sometimes people forget... > > > > There's also this which is somewhat more robust: > > https://gitlab.freedesktop.org/mesa/mesa/merge_requests/2569 > > My 20 cents: > > 1. I think we should completely disable running the CI on MRs which are > marked WIP. Speaking from personal experience, I usually make a lot of > changes to my MRs before they are merged, so it is a waste of CI > resources. In the mean time, you can help by taking the habit to use: git push -o ci.skip CI is in fact run for all branches that you push. When we (GStreamer Project) started our CI we wanted to limit this to MR, but haven't found a good way yet (and Gitlab is not helping much). The main issue is that it's near impossible to use gitlab web API from a runner (requires private key, in an all or nothing manner). But with the current situation we are revisiting this. The truth is that probably every CI have lot of room for optimization, but it can be really time consuming. So until we have a reason to, we live with inefficiency, like over sized artifact, unused artifacts, over-sized docker image, etc. Doing a new round of optimization is obviously a clear short term goals for project, including GStreamer project. We have discussions going on and are trying to find solutions. Notably, we would like to get rid of the post merge CI, as in a rebase flow like we have in GStreamer, it's a really minor risk. > > 2. Maybe we could take this one step further and only allow the CI to > be only triggered manually instead of automatically on every push. > > 3. I completely agree with Pierre-Eric on MR 2569, let's not run the > full CI pipeline on every change, only those parts which are affected > by the change. It not only costs money, but is also frustrating when > you submit a change and you get unrelated failures from a completely > unrelated driver. That's a much more difficult goal then it looks like. Let each projects manage their CI graph and content, as each case is unique. Running more tests, or building more code isn't the main issue as the CPU time is mostly sponsored. The data transfers between the cloud of gitlab and the runners (which are external), along to sending OS image to Lava labs is what is likely the most expensive. As it was already mention in the thread, what we are missing now, and being worked on, is per group/project statistics that give us the hotspot so we can better target the optimization work. > > Best regards, > Timur > > ___ > gstreamer-devel mailing list > gstreamer-de...@lists.freedesktop.org > https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] [Intel-gfx] gitlab.fd.o financial situation and impact on services
On Sat, 2020-02-29 at 14:46 -0500, Nicolas Dufresne wrote: > > > > 1. I think we should completely disable running the CI on MRs which > > are > > marked WIP. Speaking from personal experience, I usually make a lot > > of > > changes to my MRs before they are merged, so it is a waste of CI > > resources. > > In the mean time, you can help by taking the habit to use: > > git push -o ci.skip Thanks for the advice, I wasn't aware such an option exists. Does this also work on the mesa gitlab or is this a GStreamer only thing? How hard would it be to make this the default? > That's a much more difficult goal then it looks like. Let each > projects > manage their CI graph and content, as each case is unique. Running > more > tests, or building more code isn't the main issue as the CPU time is > mostly sponsored. The data transfers between the cloud of gitlab and > the runners (which are external), along to sending OS image to Lava > labs is what is likely the most expensive. > > As it was already mention in the thread, what we are missing now, and > being worked on, is per group/project statistics that give us the > hotspot so we can better target the optimization work. Yes, would be nice to know what the hotspot is, indeed. As far as I understand, the problem is not CI itself, but the bandwidth needed by the build artifacts, right? Would it be possible to not host the build artifacts on the gitlab, but rather only the place where the build actually happened? Or at least, only transfer the build artifacts on-demand? I'm not exactly familiar with how the system works, so sorry if this is a silly question. ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] [Intel-gfx] gitlab.fd.o financial situation and impact on services
On Sat, Feb 29, 2020 at 3:47 PM Timur Kristóf wrote: > > On Sat, 2020-02-29 at 14:46 -0500, Nicolas Dufresne wrote: > > > > > > 1. I think we should completely disable running the CI on MRs which > > > are > > > marked WIP. Speaking from personal experience, I usually make a lot > > > of > > > changes to my MRs before they are merged, so it is a waste of CI > > > resources. > > > > In the mean time, you can help by taking the habit to use: > > > > git push -o ci.skip > > Thanks for the advice, I wasn't aware such an option exists. Does this > also work on the mesa gitlab or is this a GStreamer only thing? Mesa is already set up so that it only runs on MRs and branches named ci-* (or maybe it's ci/*; I can't remember). > How hard would it be to make this the default? I strongly suggest looking at how Mesa does it and doing that in GStreamer if you can. It seems to work pretty well in Mesa. --Jason > > That's a much more difficult goal then it looks like. Let each > > projects > > manage their CI graph and content, as each case is unique. Running > > more > > tests, or building more code isn't the main issue as the CPU time is > > mostly sponsored. The data transfers between the cloud of gitlab and > > the runners (which are external), along to sending OS image to Lava > > labs is what is likely the most expensive. > > > > As it was already mention in the thread, what we are missing now, and > > being worked on, is per group/project statistics that give us the > > hotspot so we can better target the optimization work. > > Yes, would be nice to know what the hotspot is, indeed. > > As far as I understand, the problem is not CI itself, but the bandwidth > needed by the build artifacts, right? Would it be possible to not host > the build artifacts on the gitlab, but rather only the place where the > build actually happened? Or at least, only transfer the build artifacts > on-demand? > > I'm not exactly familiar with how the system works, so sorry if this is > a silly question. > > ___ > mesa-dev mailing list > mesa-dev@lists.freedesktop.org > https://lists.freedesktop.org/mailman/listinfo/mesa-dev ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] [Intel-gfx] gitlab.fd.o financial situation and impact on services
Le samedi 29 février 2020 à 15:54 -0600, Jason Ekstrand a écrit : > On Sat, Feb 29, 2020 at 3:47 PM Timur Kristóf wrote: > > On Sat, 2020-02-29 at 14:46 -0500, Nicolas Dufresne wrote: > > > > 1. I think we should completely disable running the CI on MRs which > > > > are > > > > marked WIP. Speaking from personal experience, I usually make a lot > > > > of > > > > changes to my MRs before they are merged, so it is a waste of CI > > > > resources. > > > > > > In the mean time, you can help by taking the habit to use: > > > > > > git push -o ci.skip > > > > Thanks for the advice, I wasn't aware such an option exists. Does this > > also work on the mesa gitlab or is this a GStreamer only thing? > > Mesa is already set up so that it only runs on MRs and branches named > ci-* (or maybe it's ci/*; I can't remember). > > > How hard would it be to make this the default? > > I strongly suggest looking at how Mesa does it and doing that in > GStreamer if you can. It seems to work pretty well in Mesa. You are right, they added CI_MERGE_REQUEST_SOURCE_BRANCH_NAME in 11.6 (we started our CI a while ago). But there is even better now, ou can do: only: refs: - merge_requests Thanks for the hint, I'll suggest that. I've lookup some of the backend of mesa, I think it's really nice, though there is a lot of concept that won't work in a multi-repo CI. Again, I need to refresh on what was moved from the enterprise to the community version in this regard, > > --Jason > > > > > That's a much more difficult goal then it looks like. Let each > > > projects > > > manage their CI graph and content, as each case is unique. Running > > > more > > > tests, or building more code isn't the main issue as the CPU time is > > > mostly sponsored. The data transfers between the cloud of gitlab and > > > the runners (which are external), along to sending OS image to Lava > > > labs is what is likely the most expensive. > > > > > > As it was already mention in the thread, what we are missing now, and > > > being worked on, is per group/project statistics that give us the > > > hotspot so we can better target the optimization work. > > > > Yes, would be nice to know what the hotspot is, indeed. > > > > As far as I understand, the problem is not CI itself, but the bandwidth > > needed by the build artifacts, right? Would it be possible to not host > > the build artifacts on the gitlab, but rather only the place where the > > build actually happened? Or at least, only transfer the build artifacts > > on-demand? > > > > I'm not exactly familiar with how the system works, so sorry if this is > > a silly question. > > > > ___ > > mesa-dev mailing list > > mesa-dev@lists.freedesktop.org > > https://lists.freedesktop.org/mailman/listinfo/mesa-dev ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
[Mesa-dev] [RFC PATCH] mesa: fix _mesa_draw_nonzero_divisor_bits to return nonzero divisors
The bitmask is _EffEnabledNonZeroDivisor, so no need to invert it before
returning.
Fixes: fd6636ebc06d (st/mesa: simplify determination whether a draw needs
min/max index)
Signed-off-by: Ilia Mirkin
---
I haven't done any extensive testing on this, but it does seem to fix a
regression on nv50 where the limits were not being supplied. (And
there's an assert to make sure that they are.)
src/mesa/main/arrayobj.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/mesa/main/arrayobj.h b/src/mesa/main/arrayobj.h
index 19ab65b3242..3efcd577ae5 100644
--- a/src/mesa/main/arrayobj.h
+++ b/src/mesa/main/arrayobj.h
@@ -221,7 +221,7 @@ _mesa_draw_nonzero_divisor_bits(const struct gl_context
*ctx)
{
const struct gl_vertex_array_object *const vao = ctx->Array._DrawVAO;
assert(vao->NewArrays == 0);
- return ~vao->_EffEnabledNonZeroDivisor & ctx->Array._DrawVAOEnabledAttribs;
+ return vao->_EffEnabledNonZeroDivisor & ctx->Array._DrawVAOEnabledAttribs;
}
--
2.24.1
___
mesa-dev mailing list
mesa-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] [RFC PATCH] mesa: fix _mesa_draw_nonzero_divisor_bits to return nonzero divisors
Rb. I'm on the phone.
M.
On Sat., Feb. 29, 2020, 22:53 Ilia Mirkin, wrote:
> The bitmask is _EffEnabledNonZeroDivisor, so no need to invert it before
> returning.
>
> Fixes: fd6636ebc06d (st/mesa: simplify determination whether a draw needs
> min/max index)
> Signed-off-by: Ilia Mirkin
> ---
>
> I haven't done any extensive testing on this, but it does seem to fix a
> regression on nv50 where the limits were not being supplied. (And
> there's an assert to make sure that they are.)
>
> src/mesa/main/arrayobj.h | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/src/mesa/main/arrayobj.h b/src/mesa/main/arrayobj.h
> index 19ab65b3242..3efcd577ae5 100644
> --- a/src/mesa/main/arrayobj.h
> +++ b/src/mesa/main/arrayobj.h
> @@ -221,7 +221,7 @@ _mesa_draw_nonzero_divisor_bits(const struct
> gl_context *ctx)
> {
> const struct gl_vertex_array_object *const vao = ctx->Array._DrawVAO;
> assert(vao->NewArrays == 0);
> - return ~vao->_EffEnabledNonZeroDivisor &
> ctx->Array._DrawVAOEnabledAttribs;
> + return vao->_EffEnabledNonZeroDivisor &
> ctx->Array._DrawVAOEnabledAttribs;
> }
>
>
> --
> 2.24.1
>
>
___
mesa-dev mailing list
mesa-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] [Intel-gfx] gitlab.fd.o financial situation and impact on services
For Mesa, we could run CI only when Marge pushes, so that it's a strictly pre-merge CI. Marek On Sat., Feb. 29, 2020, 17:20 Nicolas Dufresne, wrote: > Le samedi 29 février 2020 à 15:54 -0600, Jason Ekstrand a écrit : > > On Sat, Feb 29, 2020 at 3:47 PM Timur Kristóf > wrote: > > > On Sat, 2020-02-29 at 14:46 -0500, Nicolas Dufresne wrote: > > > > > 1. I think we should completely disable running the CI on MRs which > > > > > are > > > > > marked WIP. Speaking from personal experience, I usually make a lot > > > > > of > > > > > changes to my MRs before they are merged, so it is a waste of CI > > > > > resources. > > > > > > > > In the mean time, you can help by taking the habit to use: > > > > > > > > git push -o ci.skip > > > > > > Thanks for the advice, I wasn't aware such an option exists. Does this > > > also work on the mesa gitlab or is this a GStreamer only thing? > > > > Mesa is already set up so that it only runs on MRs and branches named > > ci-* (or maybe it's ci/*; I can't remember). > > > > > How hard would it be to make this the default? > > > > I strongly suggest looking at how Mesa does it and doing that in > > GStreamer if you can. It seems to work pretty well in Mesa. > > You are right, they added CI_MERGE_REQUEST_SOURCE_BRANCH_NAME in 11.6 > (we started our CI a while ago). But there is even better now, ou can > do: > > only: > refs: > - merge_requests > > Thanks for the hint, I'll suggest that. I've lookup some of the backend > of mesa, I think it's really nice, though there is a lot of concept > that won't work in a multi-repo CI. Again, I need to refresh on what > was moved from the enterprise to the community version in this regard, > > > > > --Jason > > > > > > > > That's a much more difficult goal then it looks like. Let each > > > > projects > > > > manage their CI graph and content, as each case is unique. Running > > > > more > > > > tests, or building more code isn't the main issue as the CPU time is > > > > mostly sponsored. The data transfers between the cloud of gitlab and > > > > the runners (which are external), along to sending OS image to Lava > > > > labs is what is likely the most expensive. > > > > > > > > As it was already mention in the thread, what we are missing now, and > > > > being worked on, is per group/project statistics that give us the > > > > hotspot so we can better target the optimization work. > > > > > > Yes, would be nice to know what the hotspot is, indeed. > > > > > > As far as I understand, the problem is not CI itself, but the bandwidth > > > needed by the build artifacts, right? Would it be possible to not host > > > the build artifacts on the gitlab, but rather only the place where the > > > build actually happened? Or at least, only transfer the build artifacts > > > on-demand? > > > > > > I'm not exactly familiar with how the system works, so sorry if this is > > > a silly question. > > > > > > ___ > > > mesa-dev mailing list > > > mesa-dev@lists.freedesktop.org > > > https://lists.freedesktop.org/mailman/listinfo/mesa-dev > > ___ > mesa-dev mailing list > mesa-dev@lists.freedesktop.org > https://lists.freedesktop.org/mailman/listinfo/mesa-dev > ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
