Do the JVM-based pythons solve any threading issues? Plain parallel java
seems indispensable.
Bill
--
Phobrain.com
On 2025-07-03 05:05, Nathan via NumPy-Discussion wrote:
> If a NumPy array is shared between two threads, NumPy doesn't do anything to
> synchronize array access. This is true in all Python versions and build
> configurations - since NumPy releases the GIL during most array operations
> whether or not you're using free-threaded Python doesn't change much except
> for e.g. object arrays, which do hold the GIL.
>
> See:
> https://numpy.org/doc/stable/reference/thread_safety.html
>
> IMO you probably shouldn't try to enforce more strict thread safety than
> NumPy itself does.
>
> We didn't add any locking to support free-threaded Python because it's always
> worked like this, and introducing locking might lead to performance
> bottlenecks in read-only multithreaded applications and would substantially
> increase NumPy's internal complexity.
>
> Long-term, I'd like to see more effort put towards adding stronger guarantees
> around freezing arrays. I also want to look closer at adding runtime checks
> to detect races and report them. One example: you could imagine each array
> having an internal "version" counter that is incremented every time the array
> is mutated. Doing an atomic read on the version before and after a mutation
> should hopefully have a small overhead compared with the rest of NumPy, and
> we could report runtime errors when arrays are mutated "underneath" a thread
> doing an operation.
>
> The devil is in the details though - there are *a lot* of ways to mutate
> NumPy arrays. This also doesn't consider the buffer protocol or accessing
> arrays via third-party C extensions. See e.g. Alex Gaynor's blog post on this
> from the perspective of Rust and PyO3:
>
> https://alexgaynor.net/2022/oct/23/buffers-on-the-edge/
>
> On Thu, Jul 3, 2025 at 5:50 AM Benjamin Root via NumPy-Discussion
> <numpy-discussion@python.org> wrote:
>
> On a related note, does numpy's gufunc mechanism provide any thread safety,
> or is the responsibility on the extension writer to do that? For simple numpy
> array inputs, I would think that I don't have to worry about free-threaded
> python messing things up (unless I have a global state), I'm wondering if
> something like dask array inputs could mess up calls to a thread-unsafe
> function.
>
> If it is on the extension writer, are there any examples on how to do that?
> Are there other guarantees (or lack thereof) that a gufunc writer should be
> aware of? How about reorderability? gufuncs operates on subarrays, so
> wouldn't dask inputs that are chunked potentially operate on the chunks in
> any order they like?
>
> Thanks,
> Ben Root
>
> On Tue, Jul 1, 2025 at 4:26 PM Benjamin Root <ben.v.r...@gmail.com> wrote:
>
> Warren,
>
> The examples in ufunclab helped clear up a few things and I was able to
> experiment and get a working gufunc! Thank you for your help!
>
> Ben Root
>
> On Fri, Jun 27, 2025 at 8:54 PM Benjamin Root <ben.v.r...@gmail.com> wrote:
>
> Warren,
>
> I'm fine with implementing it in C. I just didn't think gufuncs were for me.
> I couldn't tell from the description if it would be for my usecase since I
> wasn't looping over subarrays, and I didn't see any good examples. Maybe the
> documentation could be clearer. I'll have a look at your examples.
>
> I did try that signature with np.vectorize() with the signature keyword
> argument, but it didn't seem to work. Maybe it didn't work for the reasons in
> that open issue.
>
> Thank you,
> Ben Root
>
> On Fri, Jun 27, 2025 at 8:03 PM Warren Weckesser via NumPy-Discussion
> <numpy-discussion@python.org> wrote: On Fri, Jun 27, 2025 at 5:29 PM Benjamin
> Root via NumPy-Discussion
> <numpy-discussion@python.org> wrote:
>>
>> I'm looking at a situation where I like to wrap a C++ function that takes
>> two doubles as inputs, and returns an error code, a position vector, and a
>> velocity vector so that I essentially would have a function signature of
>> (N), (N) -> (N), (N, 3), (N, 3). When I try to use np.vectorize() or
>> np.frompyfunc() on the python version of this function, I keep running into
>> issues where it wants to make the outputs into object arrays of tuples. And
>> looking at utilizing PyUFunc_FromFuncAndData, it isn't clear to me how I can
>> tell it to expect those two output arrays to have a size 3 outer dimension.
>>
>> Are ufuncs the wrong thing here? How should I go about this? Is it even
>> possible?
>
> Ben,
>
> It looks like the simplest signature for your core operation would be
> (),()->(),(3),(3), with broadcasting taking care of higher dimensional
> inputs. Because not all the core shapes are scalars, that would
> require a *generalized* ufunc (gufunc). There is an open issue
> (https://github.com/numpy/numpy/issues/14020) with a request for a
> function to generate a gufunc from a Python function.
>
> numba has the @guvectorize decorator, but I haven't use it much, and
> in my few quick attempts just now, it appeared to not accept fixed
> integer sizes in the output shape. But wait to see if any numba gurus
> respond with a definitive answer about whether or not it can handle
> the shape signature (),()->(),(3),(3).
>
> You could implement the gufunc in a C or C++ extension module, if you
> don't mind the additional development effort and packaging hassle. I
> know that works--I've implemented quite a few gufuncs in ufunclab
> (https://github.com/WarrenWeckesser/ufunclab).
>
> Warren
>
>>
>> Thanks in advance,
>> Ben Root
>> _______________________________________________
>> NumPy-Discussion mailing list -- numpy-discussion@python.org
>> To unsubscribe send an email to numpy-discussion-le...@python.org
>> https://mail.python.org/mailman3//lists/numpy-discussion.python.org
>> Member address: warren.weckes...@gmail.com
> _______________________________________________
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3//lists/numpy-discussion.python.org
> Member address: ben.v.r...@gmail.com
_______________________________________________
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3//lists/numpy-discussion.python.org
Member address: nathan12...@gmail.com
_______________________________________________
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3//lists/numpy-discussion.python.org
Member address: bross_phobr...@sonic.net
_______________________________________________
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3//lists/numpy-discussion.python.org
Member address: arch...@mail-archive.com
