[Numpy-discussion] Re: Endorsing SPECs 1, 6, 7, and 8

[Nathan via NumPy-Discussion]

Thanks for clarifying! In that case I think endorsing SPEC 7 makes sense.

On Tue, Oct 8, 2024 at 3:08 PM Robert Kern  wrote:

> On Tue, Oct 8, 2024 at 8:36 AM Nathan via NumPy-Discussion <
> numpy-discussion@python.org> wrote:
>
>>
>> Since the legacy RNG interface cannot be deprecated and we encourage
>> downstream to use it in tests according to the text of NEP 19, I'm not sure
>> about the text in SPEC 7 that talks about deprecating using legacy RNGs. Or
>> are you saying that we have now reached the point where we can update NEP
>> 19 to encourage moving away from the legacy interface?
>>
>
>  We have already always encouraged people to move away from the legacy
> interface in their APIs. SPEC 7 recommends a principled way for downstream
> projects to implement that move.
>
> NEP 19 acknowledged that sometimes one might still have a use case for
> creating a legacy RandomState object and calling it in their tests to
> generate test data (but not otherwise pass that RandomState object to the
> code under test), but that's not what SPEC 7 addresses. NEP 19 doesn't
> really actively recommend the use of RandomState for this purpose, just
> acknowledges that it's a valid use case that numpy will continue to support
> even while we push for the exclusive use of Generator inside of
> library/program code. NEP 19 doesn't need an update for us to endorse SPEC
> 7 (whether it needs one, separately, to clarify its intent is another
> question).
>
> --
> Robert Kern
>
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Endorsing SPECs 1, 6, 7, and 8

[Nathan via NumPy-Discussion]

Regarding thread safety - that's not a problem. At least for Python 3.13,
the GIL is temporarily re-enabled during imports. That won't necessarily be
true in the future, but separately CPython also uses per-module locks on
import, so there shouldn't be any issues with threads simultaneously
importing submodules.

It looks like we already implement lazy-loading for e.g. linalg, fft,
random, and other submodules. Does that lazy-loading mechanism conform to
the SPEC? If not, should it?

The keys to the castle SPEC makes sense to me, I'm fine with endorsing it.
I believe that all of NumPy's online accounts are already spread out over
multiple maintainers, so presumably we don't actually need to do much here
to implement it?

Since the legacy RNG interface cannot be deprecated and we encourage
downstream to use it in tests according to the text of NEP 19, I'm not sure
about the text in SPEC 7 that talks about deprecating using legacy RNGs. Or
are you saying that we have now reached the point where we can update NEP
19 to encourage moving away from the legacy interface? From the text of NEP
19 regarding the legacy RNG interface:

> This NEP does not propose that these requirements remain in perpetuity.
After we have experience with the new PRNG subsystem, we can and should
revisit these issues in future NEPs.

I don't have a problem with SPEC 8, although I suspect there might be a
fair bit of work to get NumPy's CI to match the suggestions in the SPEC.



On Tue, Oct 8, 2024 at 2:08 PM Joren Hammudoglu via NumPy-Discussion <
numpy-discussion@python.org> wrote:

> Is SPEC 1 thread-safe enough for py313+nogil?
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: nathan12...@gmail.com
>
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: What to do with np.matrix

[Nathan via NumPy-Discussion]

Here's a github code search for the string "np.matrix":

https://github.com/search?q=%22np.matrix%22&type=code

First, if you narrow down to just Python code, there are almost 60 thousand
results, which is quite high, much higher than we we're comfortable with
for outright removals for NumPy 2.0.

Compared with code searches I did in service of the NumPy 2.0 API changes,
this returns a lot of repositories in the flavor of "someone's homework
assignments" rather than "core scientific python package" or "package owned
by a billion dollar corporation".

So, it's good that "important" packages don't seem to use np.matrix much,
but also it's bad given that the code that *does* seem to use it is
probably infrequently or poorly tested, and will require a lengthy
deprecation period to catch, if the authors are inclined to do anything
about it at all.

In that case, I think moving things to an external pypi package along with
a long-lived shim in NumPy that points people to the pypi package is
probably the least disruptive thing to do, if we're going to do anything.

-Nathan
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: NumPy 2.2.0 Released

[Nathan via NumPy-Discussion]

Improvements to the promoters for some of the string ufuncs:
https://github.com/numpy/numpy/pull/27636

Support for stringdtype arrays in the type hints and typing support for the
string ufuncs:
https://github.com/numpy/numpy/pull/27470

If you have a particular improvement you’re looking for I’d love to hear
more.

On Sun, Dec 8, 2024 at 3:01 PM Neal Becker via NumPy-Discussion <
numpy-discussion@python.org> wrote:

> Where can I find more information on improvements to stringdtype?
>
> On Sun, Dec 8, 2024, 11:25 AM Charles R Harris via NumPy-Discussion <
> numpy-discussion@python.org> wrote:
>
>> Hi All,
>>
>> On behalf of the NumPy team, I'm pleased to announce the release of NumPy
>> 2.2.0. The NumPy 2.2.0 release is a short release that brings us back
>> into sync with the usual twice yearly release cycle. There have been a
>> number of small cleanups, as well as work bringing the new StringDType to
>> completion and improving support for free threaded Python. Highlights are:
>>
>>- New functions `matvec` and `vecmat`, see below.
>>- Many improved annotations.
>>- Improved support for the new StringDType.
>>- Improved support for free threaded Python
>>- Fixes for f2py
>>
>> This release supports Python 3.10-3.13. Wheels can be downloaded from
>> PyPI ; source archives, release
>> notes, and wheel hashes are available on Github
>> .
>>
>> Cheers,
>>
>> Charles Harris
>> ___
>> NumPy-Discussion mailing list -- numpy-discussion@python.org
>> To unsubscribe send an email to numpy-discussion-le...@python.org
>> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
>> Member address: ndbeck...@gmail.com
>>
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: nathan12...@gmail.com
>
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] New GitHub issue UI

[Nathan via NumPy-Discussion]

Hi all,

GitHub is rolling out the new UI for issues, which includes a lot of new
opportunities to reorganize our backlog. More detail on the changelog blog:
https://github.blog/changelog/2025-01-13-evolving-github-issues-public-preview/

In particular, there is now much richer support for tracking issues by
marking issues as "sub-issues". We can also (finally) get rid of the issue
category labels - GitHub now has support for "issue types".

If someone with triage rights would like to take this on, it would be a
nice project to try to go through the backlog and update things to use the
new system, as well as the bot that auto-applies labels. You could probably
use a script rather than doing it manually.

-Nathan
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Wondering if there is interest in a "variable convolution" feature in numpy?

[Nathan via NumPy-Discussion]

NumPy probably isn’t the right spot for this - we’re very conservative
about adding new functionality to NumPy that might also live in SciPy.
SciPy has convolution functionality but I’m not sure if they would want
greenfield code for this. Definitely worth asking the SciPy developers.

That said, have you considered publishing and promoting your own package on
PyPI and conda-forge? It’s a bit of work to get everything set up, but at
least these days you can “publish” the work (in an academic sense)
relatively straightforwardly with a Journal of Open Source Software
submission.

See also the PyOpenSci guide, which has extensive guidance for writing and
publishing packages for general consumption:

https://www.pyopensci.org/python-package-guide/index.html

On Wed, Jun 4, 2025 at 6:32 AM cantor.duster--- via NumPy-Discussion <
numpy-discussion@python.org> wrote:

> Hello,
>
> My team and I (especially @Arqu1100) have been working on energy-dependent
> convolutions for a nuclear physics application:
> https://github.com/det-lab/energyDependentColvolve.
>
> We're looking to release this code either as a standalone library or as
> part of a library because we ran into quite a few issues when writing the
> code and would like to help out other groups who need to do this "simple"
> calculation.
>
> This code is definitely not ready for a pull request, but if there's any
> interest in this feature we're happy to create one.  @Arqu1100 has worked
> particularly hard on creating test cases. The only existing library we
> found that does what we do here is varconvolve, which we haven't been able
> to verify against our test cases. Other examples of code that does a
> similar job are referenced on Stack Overflow as being embedded into ldscat (
> https://stackoverflow.com/questions/18624005/how-do-i-perform-a-convolution-in-python-with-a-variable-width-gaussian
> ).
>
> If there's a better place to ask the question, please let me know.  Thanks
> all!
>
> Amy Roberts
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3//lists/numpy-discussion.python.org
> Member address: nathan12...@gmail.com
>
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3//lists/numpy-discussion.python.org
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Addition of eigenvalue functions

[Nathan via NumPy-Discussion]

If functionality is available in SciPy we usually don’t consider adding it
to NumPy. That rules out adding eig.

Is there any reason why polyeig doesn’t make sense to add to SciPy instead
of NumPy? Generally if functionality makes sense to add to SciPy that’s
where we point people to.

On Thu, Jun 12, 2025 at 6:39 AM waqar jamali via NumPy-Discussion <
numpy-discussion@python.org> wrote:

> NumPy currently lacks a generalized eigenvalue function such as eig(A, B)
> or polyeig(A, B).
>
> These functions are essential for several algorithms, including the
> Criss-Cross algorithm and various eigenvalue problems. In particular,
> large-scale problems in control theory are often reduced to subspace
> problems where MATLAB routines like eig(A, B) and polyeig are widely used
> in research. Therefore, I believe that adding such functionality to NumPy
> would be highly beneficial.
>
> I have submitted a pull request here.
>
> https://github.com/numpy/numpy/pull/29163
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3//lists/numpy-discussion.python.org
> Member address: nathan12...@gmail.com
>
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3//lists/numpy-discussion.python.org
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Tricky ufunc implementation question

[Nathan via NumPy-Discussion]

If a NumPy array is shared between two threads, NumPy doesn’t do anything
to synchronize array access. This is true in all Python versions and build
configurations - since NumPy releases the GIL during most array operations
whether or not you’re using free-threaded Python doesn’t change much except
for e.g. object arrays, which do hold the GIL.

See:
https://numpy.org/doc/stable/reference/thread_safety.html

IMO you probably shouldn’t try to enforce more strict thread safety than
NumPy itself does.

We didn’t add any locking to support free-threaded Python because it’s
always worked like this, and introducing locking might lead to performance
bottlenecks in read-only multithreaded applications and would substantially
increase NumPy’s internal complexity.

Long-term, I’d like to see more effort put towards adding stronger
guarantees around freezing arrays. I also want to look closer at adding
runtime checks to detect races and report them. One example: you could
imagine each array having an internal “version” counter that is incremented
every time the array is mutated. Doing an atomic read on the version before
and after a mutation should hopefully have a small overhead compared with
the rest of NumPy, and we could report runtime errors when arrays are
mutated “underneath” a thread doing an operation.

The devil is in the details though - there are *a lot* of ways to mutate
NumPy arrays. This also doesn’t consider the buffer protocol or accessing
arrays via third-party C extensions. See e.g. Alex Gaynor’s blog post on
this from the perspective of Rust and PyO3:

https://alexgaynor.net/2022/oct/23/buffers-on-the-edge/

On Thu, Jul 3, 2025 at 5:50 AM Benjamin Root via NumPy-Discussion <
numpy-discussion@python.org> wrote:

> On a related note, does numpy's gufunc mechanism provide any thread
> safety, or is the responsibility on the extension writer to do that? For
> simple numpy array inputs, I would think that I don't have to worry about
> free-threaded python messing things up (unless I have a global state), I'm
> wondering if something like dask array inputs could mess up calls to a
> thread-unsafe function.
>
> If it is on the extension writer, are there any examples on how to do
> that? Are there other guarantees (or lack thereof) that a gufunc writer
> should be aware of? How about reorderability? gufuncs operates on
> subarrays, so wouldn't dask inputs that are chunked potentially operate on
> the chunks in any order they like?
>
> Thanks,
> Ben Root
>
>
> On Tue, Jul 1, 2025 at 4:26 PM Benjamin Root  wrote:
>
>> Warren,
>>
>> The examples in ufunclab helped clear up a few things and I was able to
>> experiment and get a working gufunc! Thank you for your help!
>>
>> Ben Root
>>
>> On Fri, Jun 27, 2025 at 8:54 PM Benjamin Root 
>> wrote:
>>
>>> Warren,
>>>
>>> I'm fine with implementing it in C. I just didn't think gufuncs were for
>>> me. I couldn't tell from the description if it would be for my usecase
>>> since I wasn't looping over subarrays, and I didn't see any good examples.
>>> Maybe the documentation could be clearer. I'll have a look at your examples.
>>>
>>> I did try that signature with np.vectorize() with the signature keyword
>>> argument, but it didn't seem to work. Maybe it didn't work for the reasons
>>> in that open issue.
>>>
>>> Thank you,
>>> Ben Root
>>>
>>> On Fri, Jun 27, 2025 at 8:03 PM Warren Weckesser via NumPy-Discussion <
>>> numpy-discussion@python.org> wrote:
>>>
 On Fri, Jun 27, 2025 at 5:29 PM Benjamin Root via NumPy-Discussion
  wrote:
 >
 > I'm looking at a situation where I like to wrap a C++ function that
 takes two doubles as inputs, and returns an error code, a position vector,
 and a velocity vector so that I essentially would have a function signature
 of (N), (N) -> (N), (N, 3), (N, 3). When I try to use np.vectorize() or
 np.frompyfunc() on the python version of this function, I keep running into
 issues where it wants to make the outputs into object arrays of tuples. And
 looking at utilizing PyUFunc_FromFuncAndData, it isn't clear to me how I
 can tell it to expect those two output arrays to have a size 3 outer
 dimension.
 >
 > Are ufuncs the wrong thing here? How should I go about this? Is it
 even possible?

 Ben,

 It looks like the simplest signature for your core operation would be
 (),()->(),(3),(3), with broadcasting taking care of higher dimensional
 inputs.  Because not all the core shapes are scalars, that would
 require a *generalized* ufunc (gufunc).  There is an open issue
 (https://github.com/numpy/numpy/issues/14020) with a request for a
 function to generate a gufunc from a Python function.

 numba has the @guvectorize decorator, but I haven't use it much, and
 in my few quick attempts just now, it appeared to not accept fixed
 integer sizes in the output shape.  But wait to see if any numba gurus
 respond with a definitive a