[Numpy-discussion] Re: Policy on AI-generated code
On Thu, Jul 04, 2024 at 04:18:03PM +0100, Matthew Brett wrote: > I feel sure we would want to avoid GPL code if the copyright holders > felt that we were abusing their license - regardless of whether the > court felt the copyright was realistically enforceable. Apologies for probably stating the obvious, but BSD code also requires attribution either in the code itself or in the docs. I'm told that Bing Copilot often displays links to the origin of the generated code like Stackoverflow. So some tools do "know" where the code came from and recognize the general copyright issue. Stefan Krah ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Re: Policy on AI-generated code
On Thu, Jul 04, 2024 at 03:46:02PM +, Rohit Goswami wrote: > Doesn't the project adopting wording of this kind "pass the buck" onto the > maintainers? I think it depends. NetBSD's AI policy mentions the responsibility of the committers: https://www.netbsd.org/developers/commit-guidelines.html Gentoo mentions the contributors: https://wiki.gentoo.org/wiki/Project:Council/AI_policy The latter is quite common in CLAs in general. Also, for a very long time PyPI had a policy that put the entire responsibility for complying with U.S. cryptography export regulations on the uploader, no matter where the uploader was from. I was told in no uncertain terms that this policy was just and that it would protect the PSF (protection of uploaders was not a concern). Stefan Krah ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
Re: [Numpy-discussion] A little about XND
On Sun, Jun 17, 2018 at 08:47:02PM -0400, Marten van Kerkwijk wrote: > More of a detailed question, but as we are currently thinking about > extending the signature of gufuncs (i.e., things like `(m,n),(n,p)->(m,p)` > for matrix multiplication), and as you must have thought about this for > libgufunc, could you point me to how one would document the signature in > your new system? (I briefly tried but there's no docs yet and I couldn't > immediately find it in the code). The docs are a bit scattered across the three libraries, here is something about types and pattern matching: http://ndtypes.readthedocs.io/en/latest/ndtypes/types.html http://ndtypes.readthedocs.io/en/latest/ndtypes/pattern-matching.html A couple of example signatures: https://github.com/plures/gumath/blob/5f1f6de3d2c9a003b9dfb224fe09c63ae81bf18b/libgumath/extending/quaternion.c#L121 https://github.com/plures/gumath/blob/5f1f6de3d2c9a003b9dfb224fe09c63ae81bf18b/libgumath/extending/pdist.c#L115 The function signature for float64-specialized matrix multiplication is: "... * N * M * float64, ... * M * P * float64 -> ... * N * P * float64" The function signature for generic matrix multiplication is: "... * N * M * T, ... * M * P * T -> ... * N * P * T" A function that only accepts scalars: "... * N * M * Scalar, ... * M * P * Scalar -> ... * N * P * Scalar" A couple of observations: Functions are multimethods, so function dispatch on concrete arguments works by trying to locate a matching kernel. For example, if only the above "float64" kernel is present, all other dtypes will fail. Casting --- It is still under debate how we handle casting. The current examples libgumath/kernels simply generate *all* signatures that allow exact casting of the input for a specific function. This is feasible for unary and binary kernels, but could lead to case explosion for functions with many arguments. The kernel writer however is always free to use the above type variable or Scalar signatures and handle casting inside the kernel. Explicit gufuncs Gufuncs are explicit and require leading ellipses. A signature of "N * M * float64" is not a gufunc and does not allow outer dimensions. Disable broadcasting "D... * N * M * float64, D... * M * P * float64 -> D... * N * P * float64" Dimension variables match a sequence of dimensions, so in the above example all outer dimensions must be exactly the same. Non-symbolic matches "... * 2 * 3 * int8" only accepts "2 * 3 * int8" as the inner dimensions. Sorry for the long mail, I hope this clears up a bit what function signatures generally look like. Stefan Krah ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] A little about XND
Hi Marten,
On Mon, Jun 18, 2018 at 12:34:03PM -0400, Marten van Kerkwijk wrote:
> That looks quite nice and expressive. In the context of a discussion we
> have been having about describing `matmul/@` and possibly broadcastable
> dimensions, I think from your description it sounds like one would describe
> `@` with multiple functions (the multiple dispatch we have been (are?)
> considering as well):
>
>
> "... * N * M * T, ... * M * P * T -> ... * N * P * T"
> "M * T, ... * M * P * T -> ... P * T"
> "... * N * M * T, M * T -> ... * N * T"
> "M * T, M * T -> T"
Yes, that's the way, and the outer dimensions (the part matched by the
ellipsis) are always broadcast like in NumPy.
> Is there a way to describe broadcasting? The sample case we've come up
> with is a function that calculates a weighted mean. This might take
> (values, sigmas) and return (mean, sigma_mean), which would imply a
> signature like:
>
> "... N * T, ... N * T -> ... * T, ... * T"
>
> But would your signature allow indicating that one could pass in a single
> sigma? I.e., broadcast the second 1 to N if needed?
Actually I came across this today when implementing optimized matching
for binary functions.
I wanted the faster kernel
"... * N * int64, ... * N * int64 -> ... * N * int64"
to also match e.g. the input
"int64, 10 * int64".
The generic datashape spec would forbid this, but perhaps the '?' that
you propose in nep-0020 would offer a way out of this for ndtypes.
It's a bit confusing for datashape, since there is already a questionmark
for missing variable dimensions (that have shape==0 in the data).
>>> ndt("var * ?var * int64")
ndt("var * ?var * int64")
This would be the type for e.g. [[0], None, [1,2,3]].
But for symbolic dimensions (which only match fixed dimensions) perhaps this
"... * ?N * int64, ... * ?N * int64 -> ... * ?N * int64"
or, as in the NEP,
"... * N? * int64, ... * N? * int64 -> ... * N? * int64"
should mean "At least one input has ndim >= 1, broadcast as necessary".
This still means that for the "all ndim==0" case one would need an
additional kernel "int64, int64 -> int64".
> I realize that this is no longer about describing precisely what the
> function doing the calculation expects, but rather what an upper level is
> allowed to do before calling the function (i.e., take a dimension of 1 and
> broadcast it).
Yes, for datashape the problem is that it also allows non-broadcastable
signatures like "N * float64", really the same as "double x[]" in C.
But the '?' with occasionally one additional kernel for ndim==0 could
solve this.
Stefan Krah
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Re: Better compatibility of the Python scientific/data stack with fast Python interpreters
On Tue, Apr 29, 2025 at 11:20:28AM +0200, PIERRE AUGIER wrote: > - slow as soon as the code involves Python extensions (because extensions > work through an emulation layer, like cpyext in PyPy) [...] > Moreover, unfortunately, HPy does not currently receive as much care as it > should. Does HPy solve the speed issue for C extensions? I thought it was more of a compatibility layer so that things similar to CFFI could be used automatically. CFFI/PyPy on the other hand was slower for the _decimal extension than CPython. CPython's C-extension speed is actually quite reasonable. The only garbage collected language that achieves the _same_ extension speed that I know of is OCaml (which is quite an achievement). SBCL Lisp, which has a great compiler for pure Lisp, also does not have really fast extensions. Does the Graal VM solve this issue? Stefan Krah ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Re: Better compatibility of the Python scientific/data stack with fast Python interpreters
On Wed, May 07, 2025 at 03:52:33PM +0200, PIERRE AUGIER via NumPy-Discussion
wrote:
> > "If the goal is to run psycopg3 fast in PyPy, there are at least three
> > paths:
> >
> >use a cffi backend: this is likely to be the fastest one
> >(in case psycopg uses Cython and doesn't call directly any CPython C
> > API): use
> >the Cython HPy backend, once it's ready
> >(in case psycopg uses the CPython C API directly): rewrite it to use HPy
> >instead."
[cut]
> Could you explain a bit more? Or provide a link? CFFI with PyPy? or CFFI with
> CPython?
Yes: _if_ the above 2022 quote from the psycopg3 issue is still valid,
and my benchmarks show that the CPython C-API is faster than the PyPy
CFFI, then (according to the quote), HPy is slower than PyPy CFFI and
therefore much slower than CPython C-API.
All of that under the assumption that there are many API calls.
Stefan Krah
> Is there a proper reproducible benchmark?
Here are the benchmarks for _decimal:
bench.py
import time
import platform
if platform.python_implementation() == "CPython":
import _decimal as C
else: # PyPy
import __decimal as C
def pi(D):
lasts, t, s, n, na, d, da = D(0), D(3), D(3), D(1), D(0), D(0), D(24)
while s != lasts:
lasts = s
n, na = n+na, na+8
d, da = d+da, da+32
t = (t * n) / d
s += t
return s
print("\n#
==")
print("# Calculating pi, 1 iterations")
print("#
==\n")
prec = 9
print("\nPrecision: %d decimal digits\n" % prec)
start = time.time()
C.getcontext().prec = prec
for i in range(1):
x = pi(C.Decimal)
print("result: %s" % str(x))
print("time: %fs\n" % (time.time()-start))
### NOTE: only execute git clean -xdf in a throwaway repository! ###
git clone https://github.com/python/cpython
cd cpython
git checkout v3.9.0
./configure && make -j32
cd ..
./cpython/python bench.py
[cut]
time: 0.169412s
cd cpython
git clean -xdf
git checkout v3.13.0
./configure && make -j32
cd ..
./cpython/python bench.py
[cut]
time: 0.238405s # Massive slowdown compared to 3.9.
wget https://downloads.python.org/pypy/pypy3.11-v7.3.19-linux64.tar.bz2
tar xvf pypy3.11-v7.3.19-linux64.tar.bz2
cd pypy3.11-v7.3.19-linux64/lib/pypy3.11/
../../bin/pypy _decimal_build.py # Builds _decimal_cffi using libmpdec.
cd ../../..
./pypy3.11-v7.3.19-linux64/bin/pypy bench.py
[cut]
time: 2.433634s
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com
[Numpy-discussion] Re: Better compatibility of the Python scientific/data stack with fast Python interpreters
On Wed, Apr 30, 2025 at 02:08:36PM +0200, PIERRE AUGIER wrote: > It seems to me that a strategy based on HPy would give practical benefices > for users in a much shorter time (typically few years) than just waiting for > CPython C API evolution. It would be nice to use alternative interpreters, but I still see conflicting messages about the performance of HPy: https://github.com/psycopg/psycopg/issues/154 "If the goal is to run psycopg3 fast in PyPy, there are at least three paths: use a cffi backend: this is likely to be the fastest one (in case psycopg uses Cython and doesn't call directly any CPython C API): use the Cython HPy backend, once it's ready (in case psycopg uses the CPython C API directly): rewrite it to use HPy instead." CFFI however is slower, e.g. for _decimal, than the native CPython C-API. (At least it was in version 3.9, _decimal has been slowed down significantly since I left.) _decimal of course operates on scalars and has many API calls, so maybe for NumPy this is not relevant except for small arrays. Or perhaps HPy has evolved in the meantime (the above GitHub thread is from 2022). Stefan Krah ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Re: Better compatibility of the Python scientific/data stack with fast Python interpreters
On Thu, May 15, 2025 at 03:41:52PM +0200, PIERRE AUGIER via NumPy-Discussion wrote: > == PyPy HPy univ / CPy native (time ratio, smaller is better) > == > TestModule::test_noargs 0.50 > TestModule::test_onearg_None 0.60 > TestModule::test_onearg_int 0.65 > TestModule::test_varargs 0.93 > TestModule::test_call_with_tuple 2.21 > TestModule::test_call_with_tuple_and_dict 1.50 > TestModule::test_allocate_int 0.67 > TestModule::test_allocate_tuple 6.33 > TestType::test_allocate_obj 5.89 > TestType::test_method_lookup 0.05 > TestType::test_noargs 0.85 > TestType::test_onearg_None0.95 > TestType::test_onearg_int 0.97 > TestType::test_varargs1.18 > TestType::test_len0.48 > TestType::test_getitem0.73 > TestHeapType::test_allocate_obj_and_survive 5.11 > TestHeapType::test_allocate_obj_and_die 4.07 > TestHeapType::test_method_lookup 0.05 > TestHeapType::test_noargs 0.84 > TestHeapType::test_onearg_None0.93 > TestHeapType::test_onearg_int 0.96 > TestHeapType::test_varargs1.20 > TestHeapType::test_len0.50 > TestHeapType::test_getitem0.78 Thanks, that does look encouraging except for object creation times. I think hpy could benefit from smaller benchmark examples. For example, I didn't find one with PyNumberMethods except in numpy-hpy, which does some numpy-specific dark magic. This is the benchmark I tried to achieve but gave up after 30 min: === # # Fake should be an immutable object that uses tp_new but not tp_init and # has one nb_add number method. The method can cheat and just return a new # Fake() without doing anything. The point of the benchmark is to test the # speed of nb_add and object creation. # from fake import Fake a = Fake() b = Fake() for _ in range(1000): x = a + b y = b + x a = x b = y print(y) # In case optimizing compilers have gotten smart and eliminate the loop. === Stefan Krah ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
