Hi Ralf,
Thank you very much for the response. I apologize for my delay. I misread the
subscription defaults for this list and was waiting for something in my inbox.
You mentioned measuring platform usage. How does the project do that? Analysis
of PyPI download stats?
I feel like rasterio and fiona are in a weird position where making a host of C
libraries installable via pip is one of their top features (which is why
projects like rio-xarray and geopandas use them), so that packaging is equally
large of a concern/cost as maintenance of its code. I'm ever getting requests
for new platform support from people who don't otherwise engage with the
projects. They clearly want wheels to use with their specialized CI systems at
work and that's all. You've given me a lot to think about, and I appreciate it.
Good luck with the 2.0.0 release! I'm working to be sure that my projects will
be compatible.
Yours,
Sean
Ralf Gommers wrote:
> On Thu, Feb 22, 2024 at 7:03 PM wrote:
> > Hi folks,
> > My name is Sean and I'm the author of several GIS packages using Numpy:
> > Fiona, Rasterio, and Shapely.
> > Hi Sean, thanks for this very good question, and for all your work on GIS
> packages.
> > I've followed Numpy's trail when it comes to wheel building for many years
> > and now I'm seeking advice on how to prioritize platforms to support and
> > how to pay for the labor and computing that it takes to build wheels and
> > maintain the infrastructure over time.
> > I'm probably best placed to answer your questions, because I've both been
> involved in NumPy build & packaging for a long time and am responsible for
> overseeing a significant fraction of the funded work on NumPy as well as
> coordinating unrestricted funding coming in (mostly via Tidelift, as you
> can see at https://opencollective.com/numpy). I'll do my best to accurately
> represent the situation for NumPy. Your questions are challenging though,
> so if you want a higher-bandwidth conversation I'd be happy to chat. Or we
> can use part of a community meeting for this, since I imagine other folks
> may be interested in this topic as well.
> > Fiona and Rasterio have an order of magnitude more C library dependencies
> > than Numpy, via GDAL (https://gdal.org/), which is almost more of an OS
> > than a library.
> > Dealing with NumPy's BLAS dependency is already a large amount of work, so
> I don't envy your task. PyPI really isn't well-suited to that many C
> libraries (as I'm sure you know); for a long time the geospatial stack was
> only usable from conda-forge, where packaging is a much easier task. I'm
> not sure that was a terrible situation - there are a couple of domains like
> that where things just get too challenging. So if you want to do something
> much more restricted than NumPy for platforms to support with wheels, that
> seems perfectly okay.
> > I found a thread in the archive about adding musllinux wheels, but it
> > wasn't clear to me how the work gets done, who does, and how it gets paid
> > for.
> > Of all work on NumPy, the funded part has increased steadily. Until ~2016
> that fraction was zero, and now a lot of the heavy lifting is funded work -
> 9 out of 10 of the top 10 committers over the past 1.5 years get paid for
> at least a part of their time spent on NumPy. This is supported in several
> ways (partially documented at https://numpy.org/about, but that's a bit out
> of date):
>
> 1. a number of grants received over the years, from: Moore and Sloan
> Foundations (>$1M), the Chan Zuckerberg Institute (>$1M), and NASA (~$400k)
> 2. maintainers employed by companies who allow those maintainers to spend
> part of their day job time on NumPy:
> - Quansight (Matti, Nathan, Rohit, Mateusz, Melissa, me)
> - NVIDIA (Sebastian - long-time maintainer, now ~2 years at NVIDIA)
> - Intel (Raghuveer, contributor for several years, just gained commit
> rights)
> - Arm (Chris, contributor for ~1 year, just gained commit rights)
> - I'm not sure if I should list Berkeley here too; folks at Berkeley
> contributed a lot in the past, not sure if that was all grant-funded or if
> there was unrestricted BIDS money to support NumPy.
> 3. unrestricted project funds, obtained from individual and corporate
> (Tidelift (>$100k), Bloomberg ($10k)) donations, which support Sayed's
> Developer in Residence position:
> https://blog.scientific-python.org/numpy/fellowship-program/.
> 4. contracts for work on NumPy from clients of Quansight (and maybe other
> companies, that is hard to know) that aligned with the NumPy project
> roadmap. Noteworthy mentions here for the Sovereign Tech Fund, which
> supported packaging-related work (
> https://www.sovereigntechfund.de/tech/openblas), and the D. E. Shaw group,
> which supported recent work on string ufuncs.
>
> That said, *funding for packaging work is still quite challenging*. While
> the above is an impressive list of funding, the vast majority of funders do
> care about what
