[Numpy-discussion] Re: Numpy wheels: who maintains the builds and who pays for it?
On Thu, Feb 22, 2024 at 7:03 PM wrote: > Hi folks, > > My name is Sean and I'm the author of several GIS packages using Numpy: > Fiona, Rasterio, and Shapely. Hi Sean, thanks for this very good question, and for all your work on GIS packages. > I've followed Numpy's trail when it comes to wheel building for many years > and now I'm seeking advice on how to prioritize platforms to support and > how to pay for the labor and computing that it takes to build wheels and > maintain the infrastructure over time. I'm probably best placed to answer your questions, because I've both been involved in NumPy build & packaging for a long time and am responsible for overseeing a significant fraction of the funded work on NumPy as well as coordinating unrestricted funding coming in (mostly via Tidelift, as you can see at https://opencollective.com/numpy). I'll do my best to accurately represent the situation for NumPy. Your questions are challenging though, so if you want a higher-bandwidth conversation I'd be happy to chat. Or we can use part of a community meeting for this, since I imagine other folks may be interested in this topic as well. > Fiona and Rasterio have an order of magnitude more C library dependencies > than Numpy, via GDAL (https://gdal.org/), which is almost more of an OS > than a library. > Dealing with NumPy's BLAS dependency is already a large amount of work, so I don't envy your task. PyPI really isn't well-suited to that many C libraries (as I'm sure you know); for a long time the geospatial stack was only usable from conda-forge, where packaging is a much easier task. I'm not sure that was a terrible situation - there are a couple of domains like that where things just get too challenging. So if you want to do something much more restricted than NumPy for platforms to support with wheels, that seems perfectly okay. > I found a thread in the archive about adding musllinux wheels, but it > wasn't clear to me how the work gets done, who does, and how it gets paid > for. Of all work on NumPy, the funded part has increased steadily. Until ~2016 that fraction was zero, and now a lot of the heavy lifting is funded work - 9 out of 10 of the top 10 committers over the past 1.5 years get paid for at least a part of their time spent on NumPy. This is supported in several ways (partially documented at https://numpy.org/about, but that's a bit out of date): 1. a number of grants received over the years, from: Moore and Sloan Foundations (>$1M), the Chan Zuckerberg Institute (>$1M), and NASA (~$400k) 2. maintainers employed by companies who allow those maintainers to spend part of their day job time on NumPy: - Quansight (Matti, Nathan, Rohit, Mateusz, Melissa, me) - NVIDIA (Sebastian - long-time maintainer, now ~2 years at NVIDIA) - Intel (Raghuveer, contributor for several years, just gained commit rights) - Arm (Chris, contributor for ~1 year, just gained commit rights) - I'm not sure if I should list Berkeley here too; folks at Berkeley contributed a lot in the past, not sure if that was all grant-funded or if there was unrestricted BIDS money to support NumPy. 3. unrestricted project funds, obtained from individual and corporate (Tidelift (>$100k), Bloomberg ($10k)) donations, which support Sayed's Developer in Residence position: https://blog.scientific-python.org/numpy/fellowship-program/. 4. contracts for work on NumPy from clients of Quansight (and maybe other companies, that is hard to know) that aligned with the NumPy project roadmap. Noteworthy mentions here for the Sovereign Tech Fund, which supported packaging-related work ( https://www.sovereigntechfund.de/tech/openblas), and the D. E. Shaw group, which supported recent work on string ufuncs. That said, *funding for packaging work is still quite challenging*. While the above is an impressive list of funding, the vast majority of funders do care about what they fund, and "keep the package installable" or "do general maintenance work" typically doesn't do well in grant applications. Funders have improved in this regard, and the ones mentioned in (1) above do allow a general maintenance bucket which is some percentage of an overall grant. The people who are doing most of the work on packaging and wheel build CI jobs for NumPy are Matti Picus, Andrew Nelson and myself. Andrew's time is unfunded, for Matti and me I'd say a significant fraction of the time working on this topic is also unfunded. Does NumFOCUS support pay a maintainer to do it? No, NumFOCUS does the admin for our project funds, but doesn't supply funding to NumPy. It also doesn't structurally support any other open source projects with direct funding, with the exception of its Small Development Grant program - which is meant for smaller one-off projects (amounts in the $2k - $10k range) rather than part-time or full-time employment. > Are Numpy maintainers adding new platform builds as part of their day > jobs? In general, no. This has alway
[Numpy-discussion] ANN: DataLab v0.12
Hello everyone, We're thrilled to announce the release of DataLab v0.12! https://datalab-platform.github.io DataLab is an open-source platform for scientific and technical data processing and visualization, based on NumPy, SciPy, ... and Qt. Once again, this release is the result of the feedback and contributions of our early adopters. We're grateful for your support and look forward to your continued insights. 💥 New Features: - New tour (when starting DataLab for the first time) and demo features. - New "Text file import assistant" feature for easily importing data from arbitrary text files. - New statistics calculations, intensity profile graphical features, ... 🚀 New Binder environment to test DataLab online without installing anything: https://mybinder.org/v2/gh/DataLab-Platform/DataLab/binder-environments?urlpath=git-pull%3Frepo%3Dhttps%253A%252F%252Fgithub.com%252FDataLab-Platform%252FDataLab%26urlpath%3Ddesktop%252F%26branch%3Dbinder-environments 📚 Documentation: - New text tutorials are available: "Measuring Laser Beam Size", "DataLab and Spyder: a perfect match" - New "Contributing" section explaining how to contribute to DataLab, whether you are a developer or not - New "Macros" section explaining how to create and use macros in DataLab 💡 You have a project or a use case in mind? (we can even write a tutorial for you...) You want to contribute? We'd love to hear from you! So please, don't hesitate to contact us or to open an issue ( https://github.com/Codra-Ingenierie-Informatique/DataLab/issues/new/choose) on our GitHub project for any request or question. Thank you for taking the time to read my announcement. Looking forward to your thoughts. Best, Pierre Raybaut ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Need help in numpy building from source on windows.
detailed proble can be found at - https://stackoverflow.com/questions/78059816/issues-in-buildingnumpy-from-source-on-windows ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Re: Need help in numpy building from source on windows.
On Mon, Feb 26, 2024 at 3:26 PM rajoraganesh--- via NumPy-Discussion < numpy-discussion@python.org> wrote: > detailed proble can be found at - > https://stackoverflow.com/questions/78059816/issues-in-buildingnumpy-from-source-on-windows Quick answers: 1. Please don't build 1.26.0, but 1.26.4. The CBLAS detection issue you are hitting there has been made more robust in 1.26.2-3, so it should go away 2. Please feel free to open an issue on https://github.com/numpy/numpy/issues for build issues like this. That's better than the mailing list. Cheers, Ralf > ___ > NumPy-Discussion mailing list -- numpy-discussion@python.org > To unsubscribe send an email to numpy-discussion-le...@python.org > https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ > Member address: ralf.gomm...@gmail.com > ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Re: ANN: DataLab v0.12
Hey, I am currently working on a project related to Lunga Sound Classification. Can we have a datalab tutorial on Health data visualisation. On Mon, Feb 26, 2024, 7:51 PM Pierre Raybaut wrote: > Hello everyone, > > We're thrilled to announce the release of DataLab v0.12! > https://datalab-platform.github.io > > DataLab is an open-source platform for scientific and technical data > processing and visualization, based on NumPy, SciPy, ... and Qt. > > Once again, this release is the result of the feedback and contributions > of our early adopters. We're grateful for your support and look forward to > your continued insights. > > 💥 New Features: > > - New tour (when starting DataLab for the first time) and demo features. > - New "Text file import assistant" feature for easily importing data from > arbitrary text files. > - New statistics calculations, intensity profile graphical features, ... > > 🚀 New Binder environment to test DataLab online without installing > anything: > > https://mybinder.org/v2/gh/DataLab-Platform/DataLab/binder-environments?urlpath=git-pull%3Frepo%3Dhttps%253A%252F%252Fgithub.com%252FDataLab-Platform%252FDataLab%26urlpath%3Ddesktop%252F%26branch%3Dbinder-environments > > 📚 Documentation: > - New text tutorials are available: "Measuring Laser Beam Size", "DataLab > and Spyder: a perfect match" > - New "Contributing" section explaining how to contribute to DataLab, > whether you are a developer or not > - New "Macros" section explaining how to create and use macros in DataLab > > 💡 You have a project or a use case in mind? (we can even write a tutorial > for you...) You want to contribute? We'd love to hear from you! > So please, don't hesitate to contact us or to open an issue ( > https://github.com/Codra-Ingenierie-Informatique/DataLab/issues/new/choose) > on our GitHub project for any request or question. > > Thank you for taking the time to read my announcement. Looking forward to > your thoughts. > > Best, > Pierre Raybaut > ___ > NumPy-Discussion mailing list -- numpy-discussion@python.org > To unsubscribe send an email to numpy-discussion-le...@python.org > https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ > Member address: rakshitsingh...@gmail.com > ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
