[Numpy-discussion] Need help extending a NumPy array in C

[lpsmith]

Hello!  I am maintaining a C++ codebase with extensive ties to Python bindings 
(via SWIG).  One of the features of the code is that we define (in C) a 
subclass of a NumPy Array.  Everything worked until we started getting this 
message with numpy 1.23:

   RuntimeError: Object of type  appears to be C subclassed 
NumPy array, void scalar, or allocated in a non-standard way.NumPy reserves the 
right to change the size of these structures. Projects are required to take 
this into account by either recompiling against a specific NumPy version or 
padding the struct and enforcing a maximum NumPy version.

My problem is that I don't know how do to either of those things.  I would have 
assumed that whatever I compiled against, it would always be compiled against a 
specific NumPy version, and I also assumed that 'enforcing a maximum NumPy 
version' would happen in requirements.txt for the Python package, but that also 
seems to not be the case.  Any hints?  Thank you!

-Lucian Smith
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Need help extending a NumPy array in C

[Matti Picus]

On 8/7/22 01:37, lpsm...@uw.edu wrote:


Hello!  I am maintaining a C++ codebase with extensive ties to Python bindings 
(via SWIG).  One of the features of the code is that we define (in C) a 
subclass of a NumPy Array.  Everything worked until we started getting this 
message with numpy 1.23:

RuntimeError: Object of type  appears to be C 
subclassed NumPy array, void scalar, or allocated in a non-standard way.NumPy 
reserves the right to change the size of these structures. Projects are required to 
take this into account by either recompiling against a specific NumPy version or 
padding the struct and enforcing a maximum NumPy version.

My problem is that I don't know how do to either of those things.  I would have 
assumed that whatever I compiled against, it would always be compiled against a 
specific NumPy version, and I also assumed that 'enforcing a maximum NumPy 
version' would happen in requirements.txt for the Python package, but that also 
seems to not be the case.  Any hints?  Thank you!

-Lucian Smith



Did you rerun SWIG to regenerate the bindings before the 1.23 build?

Matti

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Need help extending a NumPy array in C

[Sebastian Berg]

On Thu, 2022-07-07 at 22:37 +, lpsm...@uw.edu wrote:
> Hello!  I am maintaining a C++ codebase with extensive ties to Python
> bindings (via SWIG).  One of the features of the code is that we
> define (in C) a subclass of a NumPy Array.  Everything worked until
> we started getting this message with numpy 1.23:
> 
>    RuntimeError: Object of type  appears to be C
> subclassed NumPy array, void scalar, or allocated in a non-standard
> way.NumPy reserves the right to change the size of these structures.
> Projects are required to take this into account by either recompiling
> against a specific NumPy version or padding the struct and enforcing
> a maximum NumPy version.
> 
> My problem is that I don't know how do to either of those things.  I
> would have assumed that whatever I compiled against, it would always
> be compiled against a specific NumPy version, and I also assumed that
> 'enforcing a maximum NumPy version' would happen in requirements.txt
> for the Python package, but that also seems to not be the case.  Any
> hints?  Thank you!


This should only happen if you compile against NumPy 1.19 and then run
your code on NumPy 1.20+.  (The warning is quite a lot older than 1.23
so I am surprised about you mentioning 1.23 specifically.)

Of course you also have to compile with the oldest NumPy version you
wish to support, so that would be a normal thing.

If you don't find the pattern for extending the `PyArrayObject` struct
maybe you can share a bit of the code (we can do that off-list if
necessary).
In principle the warning could be triggered by any bad memory access,
but it seems unlikely...


The normal reason for this (or what the warning tries to inform you
about), is that somewhere in the code you should have the following:

typedef struct {
PyArrayObject;
/* some information you add: */
char *names;
} NamedArrayObject;

PyTypeObject NamedArray_Type = {
.tp_basicsize = sizeof(NamedArrayObject);
};


A quick hack would be just to add `void *numpy_reserved[2];` after the
`PyArrayObject`, maybe as:

typedef struct {
void *reserved[2];
} numpy_array_padding;

typedef struct {
PyArrayObject;
numpy_array_padding;  /* space for newer NumPy */
char *names;
} NamedArrayObject;

Then you can add a runtime check at module initialization:

sizeof(PyArrayObject) + sizeof(numpy_array+padding)
>= PyArray_Type.tp_basicsize

To raise an error if a similar incompatibility arises in the future.
(I say `void *reserved[2]` because we appended a single `void *`
twice.)

That padding could be done at run-time as well, but that is a bit
trickier and maybe more hassle than worthwhile.

Hope this helps fixing it quickly!

Cheers,

Sebastian


> 
> -Lucian Smith
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: sebast...@sipsolutions.net
> 



signature.asc
Description: This is a digitally signed message part
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Maintainers Track at EuroScipy 2022

[Adrin]

Hi Folks,

We're organizing the maintainers' track at EuroScipy 2022, happening in
Basel at the end of August this year: https://www.euroscipy.org/2022/.

This track is a venue for maintainers to meet one another and meet
maintainers of other projects, interact, and exchange ideas. With the works
going into the scientific python community and SPECs(
https://scientific-python.org/), we think it would be nice for folks who
are going to be around at EuroScipy to meet and discuss issues.

You can see the previous track at the last EuroScipy here:
https://www.euroscipy.org/2019/maintainers.html

We would really appreciate it if you could come, be there, and contribute
in sessions if you wish. We're also looking for ideas/proposals for
sessions.

We have two concrete questions:
- Are there any maintainers from your project who are attending the
conference?
- Do you have a proposal for a session for the track?

Best,
On Behalf of The EuroScipy Committee,
Adrin
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] NumPy 1.23.1 released

[Charles R Harris]

Hi All,

On behalf of the NumPy team, I'm pleased to announce the release of NumPy
1.23.1. NumPy 1.23.1 is a maintenance release that fixes bugs discovered
after the 1.23.0 release. Notable fixes are:

   - Fix searchsorted for float16 NaNs
   - Fix compilation on Apple M1
   - Fix KeyError in crackfortran operator support (Slycot)

The Python versions supported for this release are 3.8-3.10. Wheels can be
downloaded from PyPI ; source
archives, release notes, and wheel hashes are available on Github
.

*Contributors*

A total of 7 people contributed to this release.  People with a "+" by their
names contributed a patch for the first time.

   - Charles Harris
   - Matthias Koeppe +
   - Pranab Das +
   - Rohit Goswami
   - Sebastian Berg
   - Serge Guelton
   - Srimukh Sripada +

*Pull requests merged*

A total of 8 pull requests were merged for this release.

   - #21866: BUG: Fix discovered MachAr (still used within valgrind)
   - #21867: BUG: Handle NaNs correctly for float16 during sorting
   - #21868: BUG: Use ``keepdims`` during normalization in ``np.average``
   and...
   - #21869: DOC: mention changes to ``max_rows`` behaviour in
   ``np.loadtxt``
   - #21870: BUG: Reject non integer array-likes with size 1 in delete
   - #21949: BLD: Make can_link_svml return False for 32bit builds on x86_64
   - #21951: BUG: Reorder extern "C" to only apply to function
   declarations...
   - #21952: BUG: Fix KeyError in crackfortran operator support

Cheers,

Charles Harris
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com