[Numpy-discussion] Re: How does Numpy model non-float arrays?

[Matti Picus]

On 22/8/23 02:25, Dylon Edwards wrote:


It is my understanding that Numpy accelerates array operations with BLAS where 
possible, but BLAS does not support all the dtypes that Numpy does. How does 
Numpy model non-float arrays like arrays of dtype=bool or dtype=object?


Numpy only uses BLAS where it can. Where it cannot it doesn't use BLAS. 
Is there a particular operation you are interested in?


Matti

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: JSON format for multi-dimensional data

[Matti Picus]

On 20/2/24 01:24, phili...@loco-labs.io wrote:


Hi community,

This memo is a proposal to implement a compact and reversible (lossless 
round-trip) JSON interface for multi-dimensional data and in particular for 
Numpy (see issue #12481). The links to the documents are at the end of the memo.

The JSON-NTV (Named and Typed value) format is a JSON format which integrates a 
notion of type. This format has also been implemented for tabular data (see 
NTV-pandas package available in the pandas ecosystem and the PDEP12 
specification). .

The use of this format has the following advantages:
- Taking into account data types not known to Numpy,
- Reversible format (lossless round-trip)
- Interoperability with other tools for tabular or multi-dimensional data (e.g. 
pandas, Xarray)
- Ease of sharing Json format
- Binary coding possible (e.g. CBOR format)
- Format integrating data of different nature

The associated Jupyter Notebook presents some key points of this proposal 
(first draft):

Summary:
   - introduction
   - benefits
   - multi-dimensionnal data
   - Multi-dimensional types
   - Format JSON
   - Using the NTV format
   - Equivalence of tabular format and multidimensional format
   - Astropy specific points
   - Units and quantities
   - Coordinates
   - Tables
   - Other structures

This subject seems important to me (in particular for interoperability issues) 
and I would like to have your feedback before working on the implementation.
Especially,
- do you think this “semantic” format is interesting to use?
- do you have any particular expectations or subjects that I need to study 
beforehand?
- do you have any examples or test cases to offer me?
And of course, any type of remark and comment is welcome.

Thanks in advance !

links:
- Jupyter notebook : 
https://nbviewer.org/github/loco-philippe/Environmental-Sensing/blob/main/python/Tests/numpy_tests.ipynb
- JSON-NTV format : https://www.ietf.org/archive/id/draft-thomy-json-ntv-02.html
- JSON-NTV overview : 
https://nbviewer.org/github/loco-philippe/NTV/blob/main/example/example_ntv.ipynb
- NTV tabular format : 
https://www.ietf.org/archive/id/draft-thomy-ntv-tab-00.html#name-tabular-structure
- NTV-pandas package : 
https://github.com/loco-philippe/ntv-pandas/blob/main/README.md
- NTV-pandas examples : 
https://nbviewer.org/github/loco-philippe/ntv-pandas/blob/main/example/example_ntv_pandas.ipynb
- Pandas specification - PDEP12 : 
https://pandas.pydata.org/pdeps/0012-compact-and-reversible-JSON-interface.html


There is an open issue [1] about such a format, is this the same or 
different?



We discussed this at the latest triage meeting. While interoperability 
is one of NumPy's goals, and something we care deeply about, we were not 
sure how this initiative will play out. Perhaps, like the Pandas 
package, it should live outside NumPy for a while until some wider 
consensus could emerge. We did have a few questions about the standard:


- How does it handle sharing data? NumPy can handle very large ndarrays, 
and a read-only container with a shared memory location, like in DLPack 
[0] seems more natural than a format that precludes sharing data.


- Is there a size limitation either on the data or on the number of 
dimensions? Could this format represent, for instance, data with more 
than 100 dimensions, which could not be mapped back to NumPy.



Matti


[0] https://dmlc.github.io/dlpack/latest/

[1] https://github.com/numpy/numpy/issues/12481

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: builing numpy on windows

[Matti Picus]

On 27/2/24 06:09, Ganesh Rajora via NumPy-Discussion wrote:

Hi Team,

I am Ganesh working for an MNC here in India and I am working on 
customised Python where I build set of python modules with python.


I do not install them directly from web but I build each and 
everything from source code.  this is because security concerns at the 
organisation.


In similar line I want to get some help from you on building numpy on 
windows, as I did not find any direct reference on how to do that on 
any of numpy's official web. I am facing lots of issues to build that, 
If you could help me out with it or could give me a right point of 
contact to discuss the issue would be great help.


I have posted the issue here on stackoverflow -

Issues in buildingnumpy from source on Windows 




Thanks,
Ganesh



I posted on the stack overflow link. My guess is you are not building an 
OpenBLAS MSVC import library, but using MSVC to build NumPy.


Matti

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Change in behaviour: broadcasting rules for np.solve

[Matti Picus]

API changes should hit the mailing list, so I am copying from PR 
https://github.com/numpy/numpy/pull/25914 by Aaron Meurer:





Previously the np.linalg.solve documentation stated:

|a : (..., M, M) array_like Coefficient matrix. b : {(..., M,), (..., M, 
K)}, array_like |


however, this is inherently ambiguous. For example, if a has shape (2, 
2, 2) and b has shape (2, 2), b could be treated as a (2,) stack of (2,) 
column vectors, in which case the result should have shape (2, 2), or as 
a single 2x2 matrix, in which case, the result should have shape (2, 2, 2).



NumPy resolved this ambiguity in a confusing way, which was to treat b 
as (..., M) whenever b.ndim == a.ndim - 1, and as (..., M, K) otherwise.



A much more consistent way to handle this ambiguity is to treat b as a 
single vector if and only if it is 1-dimensional, i.e., use


|b : {(M,), (..., M, K)}, array_like |

This is consistent with, for instance, the matmul operator, which only 
uses the special 1-D vector logic if an operand is exactly 
1-dimensional, and treats the operands as (stacks of) 2-D matrices 
otherwise. The PR  updates np.linalg.solve() to use this behavior.



This is a backwards compatibility break, as any instance where the b 
array has more than one dimension and exactly one fewer dimension than 
the a array will now use the matrix logic, potentially returning a 
different result with a different shape. The previous behavior can be 
manually emulated with something like


np.solve(a,b[...,None])[...,0]

since b as a (M,) vector is equivalent to b as a (M, 1) matrix (or the 
user could manually import and use the internal solve1() gufunc which 
implements the b-as-vector logic).



This change aligns the solve() function with the array API, which 
resolves this broadcasting ambiguity in this way. See

https://data-apis.org/array-api/latest/extensions/generated/array_api.linalg.solve.html#array_api.linalg.solve





Matti

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Arrays of variable itemsize

[Matti Picus]

I am not sure what kind of a scheme would support various-sized native 
ints. Any scheme that puts pointers in the array is going to be worse: 
the pointers will be 64-bit. You could store offsets to data, but then 
you would need to store both the offsets and the contiguous data, nearly 
doubling your storage. What shape are your arrays, that would be the 
minimum size of the offsets?


Matti


On 13/3/24 18:15, Dom Grigonis wrote:
By the way, I think I am referring to integer arrays. (Or integer part 
of floats.)


I don’t think what I am saying sensibly applies to floats as they are.

Although, new float type could base its integer part on such concept.

—

Where I am coming from is that I started to hit maximum bounds on 
integer arrays, where most of values are very small and some become 
very large. And I am hitting memory limits. And I don’t have many 
zeros, so sparse arrays aren’t an option.


Approximately:
90% of my arrays could fit into `np.uint8`
1% requires `np.uint64`
the rest 9% are in between.

And there is no predictable order where is what, so splitting is not 
an option either.




On 13 Mar 2024, at 17:53, Nathan  wrote:

Yes, an array of references still has a fixed size width in the array 
buffer. You can think of each entry in the array as a pointer to some 
other memory on the heap, which can be a dynamic memory allocation.


There's no way in NumPy to support variable-sized array elements in 
the array buffer, since that assumption is key to how numpy 
implements strided ufuncs and broadcasting.,


On Wed, Mar 13, 2024 at 9:34 AM Dom Grigonis  
wrote:


Thank you for this.

I am just starting to think about these things, so I appreciate
your patience.

But isn’t it still true that all elements of an array are still
of the same size in memory?

I am thinking along the lines of per-element dynamic memory
management. Such that if I had array [0, 1e1], the first
element would default to reasonably small size in memory.


On 13 Mar 2024, at 16:29, Nathan  wrote:

It is possible to do this using the new DType system.

Sebastian wrote a sketch for a DType backed by the GNU
multiprecision float library:
https://github.com/numpy/numpy-user-dtypes/tree/main/mpfdtype

It adds a significant amount of complexity to store data outside
the array buffer and introduces the possibility of
use-after-free and dangling reference errors that are impossible
if the array does not use embedded references, so that’s the
main reason it hasn’t been done much.

On Wed, Mar 13, 2024 at 8:17 AM Dom Grigonis
 wrote:

Hi all,

Say python’s builtin `int` type. It can be as large as
memory allows.

np.ndarray on the other hand is optimized for vectorization
via strides, memory structure and many things that I
probably don’t know. Well the point is that it is convenient
and efficient to use for many things in comparison to
python’s built-in list of integers.

So, I am thinking whether something in between exists? (And
obviously something more clever than np.array(dtype=object))

Probably something similar to `StringDType`, but for
integers and floats. (It’s just my guess. I don’t know
anything about `StringDType`, but just guessing it must be
better than np.array(dtype=object) in combination with
np.vectorize)

Regards,
dgpb

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to
numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: nathan12...@gmail.com

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: dom.grigo...@gmail.com


___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: nathan12...@gmail.com

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: dom.grigo...@gmail.com



___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: matti.pi...@gmail.com

[Numpy-discussion] Re: Accelerated NumPy with Weld compiler

[Matti Picus]

On 20/3/24 15:13, Hemant Singh via NumPy-Discussion wrote:

Stanford and MIT jointly developed the Weld language and compiler to run NumPy 
and Pandas significantly faster, 10x-250x.
My company has changed the compiler to be production ready. Anyone interested 
in trialing this compiler, please contact me with your work email.

Best,

Hemant



Hi Hemant. Weld seems like a cool technology. It seems that the 
opensource weld repo [0] has been dormant for the past couple of years. 
Will you be contributing back to that? We would like to encourage free 
access to technology, but understand people need to make a living. Do 
you have a model for licensing or sharing your compiler?


Matti


[0] https://github.com/weld-project/weld

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Moving the weekly traige/community meetings

[Matti Picus]

Could we move the weekly community/triage meetings one hour later? Some 
participants have a permanent conflict, and the current time is 
inconvenient for my current time zone.


Matti

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Making show_runtime and show_config enable file output

[Matti Picus]

On 09/04/2024 2:42, Matan Addam wrote:


Hello,

I find the information printed by the above mentioned functions to be useful 
for understanding performance context on installed machines, as well as 
variability across machines when troubleshooting. How would the maintainers 
view a pull request adding to those functions the option of directing their 
output to a file?

These functions, at least as built on my machine using numpy 1.24.4, using 
python's print and pprint for their outputs, both of which functions allowing 
arguments for redirecting their output to a file. Adding this option may enable 
recording the information to files without redirecting all of stdout.

What would your position be?

Or are they actually a facade built upon installation by dynamically generated 
code, which yields different function implementations on different platforms?

It could be otherwise nice to provide access to a dict of the data for the more 
general purpose, which would enable all desiderata of interest leveraging this 
information.

Kind regards,
Matan



Makes sense to me to output a json-formatted file

Matti

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: New feature

[Matti Picus]

On 25/04/2024 23:16, Alexei Lisitsa wrote:


Waiting for answer



What kind of answer would you like? I took a look at numpy_list[0] and 
if it serves your needs, that is great, but I don't think such ndarray 
generation routines should be added to NumPy until they become more 
commonly known, used, and "voted" popular. I would suggest you


- add some examples to the README[1] to demonstrate its utility. I found 
I had to drill down to the code[1] to see what it actually does.


- make it a stand-alone package and not part of NumPy's code base

Matti


[0] 
https://github.com/alessionuovo/mionumpy/tree/list_operations/numpy_extensions


[1] 
https://github.com/alessionuovo/mionumpy/blob/list_operations/numpy_extensions/numpy_list/list_op.py


___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Please consider dropping Python 3.9 support for Numpy 2.0

[Matti Picus]

On 05/05/2024 11:32, Mark Harfouche wrote:

I know this is somewhat the 11th hour for the numpy 2.0 release but as 
downstream packager and user of numpy, I would really like to ask that 
Numpy strives to adhere to NEP29, and SPEC0


SEPC0 makes it pretty clear that Python 3.9 should not be included in 
any "future" versions.


Leading by example in this case will help downstream make the hard 
decision to adhere to the compatibility guidelines from SPEC0.


Thank you for considering this last minute request. I know it adds 
work at this stage.


Mark



I think NumPy should not be the leader in dropping versions, rather 
should be one of the more conservative packages since other packages 
depend on it. We have indeed dropped 3.9 on HEAD, and will not be 
supporting it for 2.1, but to me it makes sense to support it for the 
large 2.0 release.



Matti

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: New feature

[Matti Picus]

For a start you could share the link for the stand-alone package here, 
with a short description of what it does.


Additional steps would be to write a blog post about it, write it up on 
reddit's python subreddit, give a talk about it at a conference.


Matti


On 25/04/2024 23:36, Alexei Lisitsa wrote:

Ok, thanks package stand alone I did but how I advertise it?
Maybe you as expert can say some thing that is needed , something to 
develop I want some new experience




___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Build NumPy with Debugging Symbols with Meson

[matti picus]

On 15/05/2024 23:55, Robert McLeod wrote:
|
| Hi everyone,
|
| Is there a gist or similar guide anywhere on the steps required to
build NumPy with debugging symbols on the Windows platform using the
new Meson build system?


In general, I would try to avoid using python_d.exe. It requires the
entire software stack to be rebuilt with that python, which can be
quite painful (as you discovered). Instead, you can build only the
project you wish to debug with release-with-debug-symbols (on windows,
these are `*.pdb` files), which will allow you to run the exe under a
debugger.

Another tip for debugging: add the line `import pdb;pdb.set_trace()`
to your python code, which will stop the process under test. Then you
can fire up visual studio, and in the debugger menu choose
attach-to-process and connect to the correct PID to start debugging
compiled code. If you do this after `import numpy`, you should be able
to navigate to the code you wish to debug in an editor window and set
a breakpoint. Then go back to your python terminal and continue the
run.

Matti
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Unexpected return values for np.mod with x2=np.inf and similar

[Matti Picus]

What operating system?


If I recall correctly, NumPy tries to be compatible with CPython for 
these edge cases.



The actual implementation is a bit scattered. I think it would be nice 
if we could have an "explain" decorator to ufuncs that would return the 
name of the inner loop used in practice to aid in debugging and 
teaching. Until then your best bet is to build NumPy locally with debug 
information and use a debugger, but even that can be challenging at times.


Matti


On 07/06/2024 21:10, jesse.live...@gmail.com wrote:

Hi all,

I ran into an odd edge-case with np.mod and was wondering if this is the 
expected behavior, and if so why. This is on a fresh install of python 3.10.14 
with numpy 1.26.4 from conda-forge.
...
Any ideas why these are the return values? I had a hard time tracking down 
where in the numpy source np.mod was coming from.
Jesse
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: matti.pi...@gmail.com
https://github.com/numpy/numpy/blob/main/numpy/_core/src/umath/loops_modulo.dispatch.c.src#L557

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Generate documentation for the 2.0.0 release

[Matti Picus]

I will make a PR to https://github.com/numpy/doc

Matti


On 16/06/2024 19:04, Charles R Harris wrote:
PS, you will need to check out `.spin/cmds.py` from the main branch in 
order to build the documentation.


Chuck

On Sun, Jun 16, 2024 at 9:59 AM Charles R Harris 
 wrote:


Hi All,

Could someone running Python < 3.12 generate the documentation for
the NumPy 2.0.0 release. My native Python is 3.12, which works
fine except that there is no distutils. Trying to run the build
under 3.11, after hard coding the Python version in the
doc/Makefile, runs into various problems, and there are probably
some subtle Python version mismatches buried in the scripts
somewhere that I don't want to spend time chasing down at the moment.

Chuck


___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: matti.pi...@gmail.com

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Windows 11 arm64 wheel

[Matti Picus]

On 12/08/2024 11:55, slobodan.miletic--- via NumPy-Discussion wrote:


As the bug fix is merged we are starting the investigation on the CI job.
I have a few questions about this:
1) Are there some additional instructions for making and running the numpy CI 
jobs and cross compilation available in the documentation?
2) Do we need to have arm64 scipy-openblas released to build numpy wheel?
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: matti.pi...@gmail.com



1. You can look at .github/workflows/linux_qemu.yml that use qemu/docker 
to run.  AFAIK there is no documentation about cross-compilation.


2. It would be preferable to build a scipy-openblas wheel but we could 
use a "vanilla" build of OpenBLAS as well. We do expect to use 64-bit 
interfaces, but scipy does not do so yet, so we would need two builds. I 
don't know how @gholke compiles OpenBLAS/NumPy and where there is a 
gfortran for windows arm64, maybe worthwhile asking?



Matti

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Welcome Joren Hammudoglu to the NumPy Maintainers Team

[matti picus]

Welcome Joren, thanks for all the work
Matti


On Mon, Aug 19, 2024 at 1:41 PM Sebastian Berg
 wrote:
>
> Hi all,
>
> please join me in welcoming Joren (https://github.com/jorenham) to the
> NumPy maintainers team.
>
> Joren has done a lot of work recently contributing, reviewing, and
> maintaining typing related improvements to NumPy.
> We are looking forward to see new momentum to improve NumPy typing.
>
> Thanks for all the contributions!
>
> Cheers,
>
> Sebastian
>
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: matti.pi...@gmail.com
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Documentation for the 2.0.0 release.

[matti picus]

Done. Thanks for the release.
Matti

On Mon, 19 Aug 2024 at 02:59, Charles R Harris 
wrote:
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: What should remain on PyPi

[matti picus]

I would prefer we never delete packages once we upload them to PyPI,
unless there are security issues with them. As Sean demonstrated,
someone somewhere is going to be using them, and deleting packages
will inevitably break something.
Matti


On Tue, Sep 3, 2024 at 7:44 PM Sean Gillies  wrote:
>
> Hi Chuck,
>
> I've got a version of a package on PyPI that requires Numpy 2.0.0rc1 at build 
> time. Not the best decision in hindsight, but I assumed that Numpy was the 
> kind of project that wouldn't remove published distributions unless there 
> were security issues. It had not up today, right? Would it be possible to 
> restore 2.0.0rc1?
>
> On Tue, Sep 3, 2024 at 9:20 AM Charles R Harris  
> wrote:
>>
>> Hi All,
>>
>> I just got through deleting a bunch of pre-releases on PyPi and it occurred 
>> to me that we should have a policy as to what releases should be kept. I 
>> think that reproducibility requires that we keep all the major and micro 
>> versions, but if so, we should make that an official guarantee. Perhaps a 
>> short NEP? This might even qualify for an SPEC. Thoughts?
>>
>> Chuck
>
>
> --
> Sean Gillies
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: matti.pi...@gmail.com
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] make nditer also a context manager

[Matti Picus]

When an nditer uses certain op_flags[0], like updateifcopy or readwrite 
or copy, the operands (which are ndarray views into the original data) 
must use the UPDATEIFCOPY flag. The meaning of this flag is to allocate 
temporary memory to hold the modified data and make the original data 
readonly. When the caller is finished with the nditer, the temporary 
memory is written back to to the original data and the original data's 
readwrite status is restored. The trigger to resolve the temporary data 
is currently via the deallocate function of the nditer, thus the call 
becomes something like


i = nditer(a, op_flags=)
#do something with i
i = None # trigger writing data from i to a

This IMO violates the "explicit is better" philosophy, and has the 
additional disadvantage of relying on refcounting semantics to trigger 
the data resolution, which does not work on PyPy.


I have a pending pull request[1] to add a private numpy API function 
PyArray_ResolveUpdateIfCopy, which allows triggering the data resolution 
in a more explicit way. The new API function is added at the end of the 
functions like take or put with non-contiguous arguments, explicit tests 
have also been added . The only user-facing effect is to allow using an 
nditer as a context manager, so the lines above would become


with nditer(a, op_flags=) as i:
#do something with i
# data is written back when exiting

The pull request passes all tests on CPython. The last commit makes the 
use of a nditer context manager mandatory on PyPy if the UPDATEIFCOPY 
semantics are triggered, while allowing existing code to function 
without a warning on CPython.


Note that np.nested_iters[2] is problematic, in that it currently 
returns a tuple of nidters, which AFAICT cannot be made into a context 
manager, so __enter__ and __exit__ must be called manually for the nditers.


At some future point we could decide to deprecate the non-context 
managed use on CPython as well, following a cycle of first issuing a 
deprecation warning for a few release versions.


Any thoughts? Does making nditer a context manager make sense? How 
widespread is use of nested_iters in live code (it does not appear in 
the official NumPy documentation)

Thanks,
Matti

[0] https://docs.scipy.org/doc/numpy/reference/generated/numpy.nditer.html
[1] https://github.com/numpy/numpy/pull/9269
[2] 
https://github.com/numpy/numpy/blob/master/numpy/core/tests/test_nditer.py#L2344 


___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] nditer and updateifcopy semantics - advice needed

[Matti Picus]

I filed issue 9714 trying to get some feedback on what to do with 
updateifcopy semantics and user-exposed nditer.
For those who are unfamiliar with the issue see below for a short 
summary, issue 7054 for a lengthy discussion, or pull request 9639 
(which is still not merged).


As I mention in the issue, I am willing to put in the work to make the 
magical update done in the last line of this snippet more explicit:


a = arange(24, dtype='i = nditer(a, ['buffered'], order='F', casting='unsafe', 
op_dtypes='>f8', buffersize=5)

j = i.copy()
i = None #  HERE

but need some direction from the community. Possible solutions:

1. nditer is rarely used, just deprecate updateifcopy use on iterands 
and raise an exception


2. make nditer into a context manager, so the code would become explicit

a = arange(24, dtype='with nditer(a, ['buffered'], order='F', casting='unsafe', 
op_dtypes='>f8', buffersize=5) as i:

    j = i.copy()

3. something else?

Any opinions?
Matti

-
what are updateifcopy semantics? When a temporary copy or work buffer is 
required, NumPy can (ab)use the base attribute of an ndarray by


   - creating a copy of the data from the base array

   - mark the base array read-only

Then when the temporary buffer is "no longer needed"

   - the data is copied back

   - the original base array is marked read-write

The trigger for the "no longer needed" decision before pull request 9639 
is in the dealloc function.
That is not generally a place to do useful work, especially on PyPy 
which can call dealloc much later.
Pull request 9639 adds an explicit PyArray_ResolveWritebackIfCopy api 
function, and recommends calling it explicitly before dealloc.


The only place this change is visible to the python-level user is in 
nditer.
C-API users will need to adapt their code to use the new API function, 
with a deprecation cycle that is backwardly compatible on CPython.

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] NumPy-Discussion Digest, Vol 134, Issue 10

[Matti Picus]

  
  


On 04/11/17 18:00,
  numpy-discussion-requ...@python.org wrote:


  Date: Fri, 3 Nov 2017 20:56:38 -0600
From: Charles R Harris 
To: numpy-discussion 
Subject: [Numpy-discussion] NumPy 1.14 branch.
Message-ID:
	
Content-Type: text/plain; charset="utf-8"

Hi All,

I'd like to branch NumPy 1.14 soon. Before doing so, I'd like to make sure
at a minimum that

1) Changes in array print formatting are done.
2) Proposed deprecations have been make.

If there are other things that folks see as essential, now is the time to
speak up.

Chuckn



I would like to get some help with review/merge of the UPDATEIFCOPY
deprecation in 
https://github.com/numpy/numpy/pull/9639
It deprecates resolving UPDATEIFCOPY semantics in the ndarray
__dealloc__, which makes the resolution less magical and also makes
NumPy compliant with non-refcount garbage collection strategies
(like PyPy's GC).
The thought is to warn in Numpy 1.14, and remove in 1.15, so if we
miss this milestone it might be a while until one clear pain point
with PyPy can be resolved.

Matti
  

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] deprecate updateifcopy in nditer operand flags?

[Matti Picus]

I filed issue 9714 https://github.com/numpy/numpy/issues/9714 and wrote 
a mail in September trying to get some feedback on what to do with 
updateifcopy semantics and user-exposed nditer.

It garnered no response, so I am trying again.
For those who are unfamiliar with the issue see below for a short 
summary and issue 7054 for a lengthy discussion.
Note that pull request 9639 which should be merged very soon changes the 
magical UPDATEIFCOPY into WRITEBACKIFCOPY, and hopefully will appear in 
NumPy 1.14.


As I mention in the issue, there is a magical update done in this 
snippet in the next-to-the-last line:


|a = np.arange(24, dtype='f8').reshape(2, 3, 4).T i = np.nditer(a, [], 
[['readwrite', 'updateifcopy']], casting='same_kind', 
op_dtypes=[np.dtype('f4')]) # Check that UPDATEIFCOPY is activated 
i.operands[0][2, 1, 1] = -12.5 assert a[2, 1, 1] != -12.5 i = None # 
magic!!! assert a[2, 1, 1] == -12.5|



Not only is this magic very implicit, it relies on refcount semantics 
and thus does not work on PyPy.

Possible solutions:

1. nditer is rarely used, just deprecate updateifcopy use on operands

2. make nditer into a context manager, so the code would become explicit

|a = np.arange(24, dtype='f8').reshape(2, 3, 4).T with np.nditer(a, [], 
[['readwrite', 'updateifcopy']], casting='same_kind', 
op_dtypes=[np.dtype('f4')]) as i: # Check that WRITEBACKIFCOPY is 
activated i.operands[0][2, 1, 1] = -12.5 assert a[2, 1, 1] != -12.5 
assert a[2, 1, 1] == -12.5 # a is modified in i.__exit__|



3. something else?

Any opinions? Does anyone use nditer in production code?
Matti

-
what are updateifcopy semantics? When a temporary copy or work buffer is 
required, NumPy can (ab)use the base attribute of an ndarray by


   - creating a copy of the data from the base array

   - mark the base array read-only

Then when the temporary buffer is "no longer needed"

   - the data is copied back

   - the original base array is marked read-write

The trigger for the "no longer needed" decision before pull request 9639 
is in the dealloc function.
That is not generally a place to do useful work, especially on PyPy 
which can call dealloc much later.
Pull request 9639 adds an explicit PyArray_ResolveWritebackIfCopy api 
function, and recommends calling it explicitly before dealloc.


The only place this change is visible to the python-level user is in 
nditer.
C-API users will need to adapt their code to use the new API function, 
with a deprecation cycle that is backwardly compatible on CPython.

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] deprecate updateifcopy in nditer operand, flags?

[Matti Picus]

  
  


  Date: Wed, 8 Nov 2017 18:41:03 +0200
From: Matti Picus 
To: numpy-discussion@python.org
Subject: [Numpy-discussion] deprecate updateifcopy in nditer operand
	flags?
Message-ID: 
Content-Type: text/plain; charset=utf-8; format=flowed

I filed issue 9714 https://github.com/numpy/numpy/issues/9714 and wrote 
a mail in September trying to get some feedback on what to do with 
updateifcopy semantics and user-exposed nditer.
It garnered no response, so I am trying again.
For those who are unfamiliar with the issue see below for a short 
summary and issue 7054 for a lengthy discussion.
Note that pull request 9639 which should be merged very soon changes the 
magical UPDATEIFCOPY into WRITEBACKIFCOPY, and hopefully will appear in 
NumPy 1.14.

As I mention in the issue, there is a magical update done in this 
snippet in the next-to-the-last line:

|a = np.arange(24, dtype='f8').reshape(2, 3, 4).T i = np.nditer(a, [], 
[['readwrite', 'updateifcopy']], casting='same_kind', 
op_dtypes=[np.dtype('f4')]) # Check that UPDATEIFCOPY is activated 
i.operands[0][2, 1, 1] = -12.5 assert a[2, 1, 1] != -12.5 i = None # 
magic!!! assert a[2, 1, 1] == -12.5|


Formatting

a = np.arange(24, dtype='f8').reshape(2, 3, 4).T

i = np.nditer(a, [], [['readwrite', 'updateifcopy']],
casting='same_kind',

   op_dtypes=[np.dtype('f4')])

# Check that WRITEBACKIFCOPY is activated

i.operands[0][2, 1, 1] = -12.5

assert a[2, 1, 1] != -12.5

i=None   # magic

assert a[2, 1, 1] == -12.5


  

Not only is this magic very implicit, it relies on refcount semantics 
and thus does not work on PyPy.
Possible solutions:

1. nditer is rarely used, just deprecate updateifcopy use on operands

2. make nditer into a context manager, so the code would become explicit

|a = np.arange(24, dtype='f8').reshape(2, 3, 4).T with np.nditer(a, [], 
[['readwrite', 'updateifcopy']], casting='same_kind', 
op_dtypes=[np.dtype('f4')]) as i: # Check that WRITEBACKIFCOPY is 
activated i.operands[0][2, 1, 1] = -12.5 assert a[2, 1, 1] != -12.5 
assert a[2, 1, 1] == -12.5 # a is modified in i.__exit__|


Formatting


a = np.arange(24, dtype='f8').reshape(2, 3, 4).T

with np.nditer(a, [], [['readwrite', 'updateifcopy']],
casting='same_kind',

   op_dtypes=[np.dtype('f4')]) as i:

    # Check that WRITEBACKIFCOPY is activated

    i.operands[0][2, 1, 1] = -12.5

    assert a[2, 1, 1] != -12.5

assert a[2, 1, 1] == -12.5   # a is modified in
i.__exit__

  3. something else?

Any opinions? Does anyone use nditer in production code?
Matti

-
what are updateifcopy semantics? When a temporary copy or work buffer is 
required, NumPy can (ab)use the base attribute of an ndarray by

 ?? - creating a copy of the data from the base array

 ?? - mark the base array read-only

Then when the temporary buffer is "no longer needed"

 ?? - the data is copied back

 ?? - the original base array is marked read-write

The trigger for the "no longer needed" decision before pull request 9639 
is in the dealloc function.
That is not generally a place to do useful work, especially on PyPy 
which can call dealloc much later.
Pull request 9639 adds an explicit PyArray_ResolveWritebackIfCopy api 
function, and recommends calling it explicitly before dealloc.

The only place this change is visible to the python-level user is in 
nditer.
C-API users will need to adapt their code to use the new API function, 
with a deprecation cycle that is backwardly compatible on CPython.





  

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] deprecate updateifcopy in nditer operand, flags?

[Matti Picus]

On 10/11/17 12:25, numpy-discussion-requ...@python.org wrote:

Date: Fri, 10 Nov 2017 02:25:19 -0800
From: Nathaniel Smith
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] deprecate updateifcopy in nditer
operand, flags?
Message-ID:

Content-Type: text/plain; charset="UTF-8"

On Wed, Nov 8, 2017 at 2:13 PM, Allan Haldane  wrote:

On 11/08/2017 03:12 PM, Nathaniel Smith wrote:

- We could adjust the API so that there's some explicit operation to
trigger the final writeback. At the Python level this would probably
mean that we start supporting the use of nditer as a context manager,
and eventually start raising an error if you're in one of the "unsafe"
case and not using the context manager form. At the C level we
probably need some explicit "I'm done with this iterator now" call.

One question is which cases exactly should produce warnings/eventually
errors. At the Python level, I guess the simplest rule would be that
if you have any write/readwrite arrays in your iterator, then you have
to use a 'with' block. At the C level, it's a little trickier, because
it's hard to tell up-front whether someone has updated their code to
call a final cleanup function, and it's hard to emit a warning/error
on something that*doesn't*  happen. (You could print a warning when
the nditer object is GCed if the cleanup function wasn't called, but
you can't raise an error there.) I guess the only reasonable option is
to deprecate NPY_ITER_READWRITE and NP_ITER_WRITEONLY, and make people
switch to passing new flags that have the same semantics but also
promise that the user has updated their code to call the new cleanup
function.

Seems reasonable.

When people use the Nditer C-api, they (almost?) always call
NpyIter_Dealloc when they're done. Maybe that's a place to put a warning
for C-api users. I think you can emit a warning there since that
function calls the GC, not the other way around.

It looks like you've already discussed the possibilities of putting
things in NpyIter_Dealloc though, and it could be tricky, but if we only
need a warning maybe there's a way.
https://github.com/numpy/numpy/pull/9269/files/6dc0c65e4b2ea67688d6b617da3a175cd603fc18#r127707149

Oh, hmm, yeah, on further examination there are some more options here.

I had missed that for some reason NpyIter isn't actually a Python
object, so actually it's never subject to GC and you always need to
call NpyIter_Deallocate when you are finished with it. So that's a
natural place to perform writebacks. We don't even need a warning.
(Which is good, because warnings can be set to raise errors, and while
the docs say that NpyIter_Deallocate can fail, in fact it never has
been able to in the past and none of the code in numpy or the examples
in the docs actually check the return value. Though I guess in theory
writeback can also fail so I suppose we need to start returning
NPY_FAIL in that case. But it should be vanishingly rare in practice,
and it's not clear if anyone is even using this API outside of numpy.)

And for the Python-level API, there is the option of performing the
final writeback when the iterator is exhausted. The downside to this
is that if someone only goes half-way through the iteration and then
aborts (e.g. by raising an exception), then the last round of
writeback won't happen. But maybe that's fine, or at least better than
forcing the use of 'with' blocks everywhere? If we do this then I
think we'd at least want to make sure that the writeback really never
happens, as opposed to happening at some random later point when the
Python iterator object is GCed. But I'd appreciate if anyone would
express a preference between these:-)

-n

-- Nathaniel J. Smith -- https://vorpus.org
We cannot assume that the call to NPyIter_Deallocate() can resolve 
writebackifcopy semantics. NPyIter_Copy() will return a new iterator 
(after Py_INCREF ing the operands), so when either the original or the 
copy is deallocated the operand's writeback buffer may still be needed. 
So at the C level the user must resolve the writback when the last copy 
of the iterator is deallocated.


At the python level we can force the use of a context manager and 
prohibit use of a suspicious (one with writebackifcopy semantics) nditer 
outside of a context manager. As for non-exhausted nditers, IMO using a 
context manager makes it very clear when the writeback resolution is 
meant to happen. Do we really want to support a use case where someone 
creates an iterator, uses it partially, then needs to think carefully 
about whether the operand changes will be resolved?

Matti

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Where to discuss NEPs (was: Re: new NEP: np.AbstractArray and np.asabstractarray)

[Matti Picus]

  
  
On 3/9/2018 8:26 AM, Ralf Gommers wrote:

  
So my suggestion is discussion should happen on the list, and
NEP
updates should be merged promptly, or just self-merged. Sound
good?
  
  
  Agreed that overall (1) is better than (2), rejected NEPs
should be visible. However there's no need for super-quick
self-merge, and I think it would be counter-productive. 

Instead, just send a PR, leave it open for some discussion, and
update for detailed comments (as well as long in-depth
discussions that only a couple of people care about) in the
Github UI and major ones on the list. Once it's stabilized a
bit, then merge with status "Draft" and update once in a while.
I think this is also much more in like with what python-dev
does, I have seen substantial discussion on Github and have not
seen quick self-merges.

  
  Ralf

  

Agreed. Perhaps we could have some kind of guideline to prevent
flooding the mailing list. 
For instance "if you find yourself responding for the third time in
one day, perhaps the mailing list is not the best medium for
discussion and using the GitHub GUI may be preferable"
Matti
  

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] NEP sprint: 21 and 22 March

[matti picus]

I would love to join but I will be at the PyPy yearly sprint in Switzerland
from Saturday to Wednesday, and traveling back to Israel on Thursday. I can
join virtually Wednesday, my evening will be your morning. I begin
traveling Thurs morning which is sometime Wed afternoon for you and will be
offline until I arrive home around 20:00 Israel time, which is Thurs
morning.

Matti

On Fri, 16 Mar 2018 at 00:29, Stefan van der Walt 
wrote:

> Hi everyone,
>
> A quick reminder of the NEP sprint that will happen at Berkeley next
> Wednesday and Thursday.  Please let me know if you are interested in
> joining.
>
> Best regards
> Stéfan
>
> On Fri, 09 Mar 2018 15:26:38 -0800, Stefan van der Walt wrote:
> > Hi everyone,
> >
> > As you may have noticed, there's been quite a bit of movement recently
> > around NumPy Enhancement Proposals---on setting specifications,
> > building infrastructure, as well as writing new proposals.
> >
> > To further support this work, we will be hosting an informal NEP
> > sprint at Berkeley on 21 and 22 March.  Our aim is to bring core
> > contributors and interested community members together to discuss
> > proposal ideas, write up new NEPs, and polish existing ones.
> >
> > Some potential topics of discussion are:
> >
> >  - Duck arrays
> >  - Array concatenation
> >  - Random number generator seed versioning
> >  - User defined dtypes
> >  - Deprecation pathways for `np.matrix`
> >  - What to do about nditer?
> >
> > All community members are welcome to attend.  If you are a core
> > contributor, we may be able to fund some travel costs as well; please
> > let me know.
> >
> > Best regards
> > Stéfan
> > ___
> > NumPy-Discussion mailing list
> > NumPy-Discussion@python.org
> > https://mail.python.org/mailman/listinfo/numpy-discussion
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] nditer as a context manager

[Matti Picus]

|Hello all, PR #9998 (https://github.com/numpy/numpy/pull/9998/) proposes 
an update to the nditer API, both C and python. The issue (link) is that |||sometimes nditer uses temp arrays via the "writeback" mechanism, the 
data is copied back to the original arrays "when finished". However 
"when finished" was implemented using nditer deallocation. |This 
mechanism is implicit and unclear, and relies on refcount semantics 
which do not work on non-refcount python implementations like PyPY. It 
also leads to lines of code like "iter=None" to trigger the writeback 
resolution. On the c-api level the agreed upon solution is to add a new |||`NpyIter_Close` function in C, this is to be called before 
`NpyIter_Dealloc`. The reviewers and I would like to ask the wider NumPy 
community for opinions about the proposed python-level solution: 
|turning the python nditer object into a context manager. This way 
"writeback" occurs at context manager exit via a call to 
`NpyIter_Close`, instead of like before when it occurred at nditer 
deallocation (which might not happen until much later in Pypy, and could 
be delayed by GC even in Cpython). Another solution that was rejected 
(https://github.com/numpy/numpy/pull/10184) was to add an nditer.close() 
python-level function that would not require a context manager It was 
felt that this is more error-prone since it requires users to add the 
line for each iterator created. The back-compat issues are that: 1. We 
are adding a new function to the numpy API, `NpyIter_Close` (pretty 
harmless) 2. We want people to update their C code using nditer, to call 
`NpyIter_Close` before  they call `NpyIter_Dealloc` and will start 
raising a deprecation warning if misuse is detected 3. We want people to 
update their Python code to use the nditer object as a context manager, 
and will warn if they do not. We tried to minimize back-compat issues, 
in the sense that old code (which didn't work in PyPy anyway) will still 
work, although it will now emit deprecation warnings. In the future we 
also plan to raise an error if an nditer is used in Python without a 
context manager (when it should have been). For C code, we plan to leave 
the deprecation warning in place probably forever, as we can only detect 
the deprecated behavior in the deallocator, where exceptions cannot be 
raised. Anybody who uses nditers should take a look and please reply if 
it seems the change will be too painful. For more details, please see 
the updated docs in that PR Matti (and reviewers) |


___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] nditer as a context manager (reformatted?)

[Matti Picus]

Hello all, PR #9998 (https://github.com/numpy/numpy/pull/9998/) proposes 
an update to the nditer API, both C and python. The issue 
(https://github.com/numpy/numpy/issues/9714) is that sometimes nditer 
uses temp arrays via the "writeback" mechanism, the data is copied back 
to the original arrays "when finished". However "when finished" was 
implemented using nditer deallocation.


This mechanism is implicit and unclear, and relies on refcount semantics 
which do not work on non-refcount python implementations like PyPY. It 
also leads to lines of code like "iter=None" to trigger the writeback 
resolution.


On the c-api level the agreed upon solution is to add a new 
`NpyIter_Close` function in C, this is to be called before 
`NpyIter_Dealloc`.


The reviewers and I would like to ask the wider NumPy community for 
opinions about the proposed python-level solution: turning the python 
nditer object into a context manager. This way "writeback" occurs at 
context manager exit via a call to `NpyIter_Close`, instead of like 
before when it occurred at `nditer` deallocation (which might not happen 
until much later in Pypy, and could be delayed by GC even in Cpython).


Another solution that was rejected 
(https://github.com/numpy/numpy/pull/10184) was to add an nditer.close() 
python-level function that would not require a context manager It was 
felt that this is more error-prone since it requires users to add the 
line for each iterator created.


The back-compat issues are that:

1. We are adding a new function to the numpy API, `NpyIter_Close` 
(pretty harmless)


2. We want people to update their C code using nditer, to call 
`NpyIter_Close` before  they call `NpyIter_Dealloc` and will start 
raising a deprecation warning if misuse is detected


3. We want people to update their Python code to use the nditer object 
as a context manager, and will warn if they do not.


We tried to minimize back-compat issues, in the sense that old code 
(which didn't work in PyPy anyway) will still work, although it will now 
emit deprecation warnings. In the future we also plan to raise an error 
if an nditer is used in Python without a context manager (when it should 
have been). For C code, we plan to leave the deprecation warning in 
place probably forever, as we can only detect the deprecated behavior in 
the deallocator, where exceptions cannot be raised.


Anybody who uses nditers should take a look and please reply if it seems 
the change will be too painful.


For more details, please see the updated docs in that PR

Matti (and reviewers)
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Introduction: NumPy developers at BIDS

[Matti Picus]

On 08/04/18 21:02, Eric Firing wrote:

On 2018/04/07 9:19 PM, Stefan van der Walt wrote:

We would love community input on identifying the best areas & issues to
pay attention to,


Stefan,

What is the best way to provide this, and how will the decisions be made?

Eric
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion


Hi. I feel very lucky to be able to dedicate the next phase of my career to 
working on NumPy. Even though BIDS has hired me, I view myself as working for 
the community, in an open and transparent way. In thinking about how to help 
make NumPy contributors more productive, we laid out these tasks:

- triage open issues and pull requests, picking up some of the long-standing 
issues and trying to resolve them

- help with code review
 
- review and suggest improvements to the NumPy documentation


- if needed, help with releases and infrastructure maintenance tasks

Down the road, the next level of things would be

- setting up a benchmark site like speed.python.org

- add more downstream package testing to the NumPy CI so we can verify that new 
releases work with packages such as scipy, scikit-learn, astropy

To document my work, I have set up a wikihttps://github.com/mattip/numpy/wiki  
that lists some longer-term tasks and ideas. I look forward to meeting and 
working with Tyler as well as SciPy2018 where there will be both a BOF meeting 
to discuss NumPy and a two-day sprint.

BIDS is ultimately responsible to the funders to make sure my work achieves the 
goals Stefan laid out, but I am going to try to be as responsive as possible to 
any input from the wider community, either directly (mattip on github and 
#numpy on IRC), via email, or this mailing list.

Matti

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Introduction: NumPy developers at BIDS

[Matti Picus]

On 18/04/18 21:21, Stefan van der Walt wrote:

Hi Matthew,

On Wed, 18 Apr 2018 16:42:49 +0100, Matthew Brett wrote:

I was thinking about the engage community part, because it seems to me
it would be good to spend time on that first, and if it was me, I
think I'd go for more regular public meetings / discussions at this
stage rather than less.

Right, so what do you think of the suggested monthly developer meeting,
to start off with.  Why don't we try it and see how much interest there
is?

Best regards,
Stéfan

Let's try holding a video conference on Wed April 25, noon to one 
Berkeley time. Details are on the Trello card here 
https://trello.com/c/mTcHBmqq . If there are particular topics you would 
like to bring up please add them as a comment on the card.


Matti
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] NumPy sprint May 24-25 at BIDS and virtual meetup tomorrow

[Matti Picus]

We will take advantage of a few NumPy developers being at Berkeley to 
hold a two day sprint May 24-25 
https://scisprints.github.io/#may-numpy-developer-sprint.
Everyone is welcome, drop me a note if you are thinking of coming so we 
can estimate numbers.


As previously announced, I am hosting a video conference open discussion 
Wed April 25 12:00-13:00 PDT at https://meet.google.com/bmv-fbob-ezp as 
part of a community building effort.
If there are specific topics you wish to bring up, either contact me, 
add a line to the Trello card here https://trello.com/c/mTcHBmqq, or 
bring them up at the discussion.


Thanks,
Matti Picus
.
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Extending ufunc signature syntax for matmul, frozen dimensions

[Matti Picus]

In looking to solve issue #9028 "no way to override matmul/@ if 
__array_ufunc__ is set", it seems there is consensus around the idea of 
making matmul a true gufunc, but matmul can behave differently for 
different combinations of array and vector:


(n,k),(k,m)->(n,m)
(n,k),(k) -> (n)
(k),(k,m)->(m)

Currently there is no way to express that in the ufunc signature. The 
proposed solution to issue #9029 is to extend the meaning of a signature 
so "syntax like (n?,k),(k,m?)->(n?,m?) could mean that n and m are 
optional dimensions; if missing in the input, they're treated as 1, and 
then dropped from the output" Additionally, there is an open pull 
request #5015 "Add frozen dimensions to gufunc signatures" to allow 
signatures like '(3),(3)->(3)'.


I would like extending ufunc signature handling to implement both these 
ideas, in a way that would be backwardly-compatible with the publicly 
exposed PyUFuncObject. PyUFunc_FromFuncAndDataAndSignature is used to 
allocate and initialize a PyUFuncObject, are there downstream projects 
that allocate their own PyUFuncObject not via 
PyUFunc_FromFuncAndDataAndSignature? If so, we could use one of the 
"reserved" fields, or extend the meaning of the "identity" field to 
allow version detection. Any thoughts?


Any other thoughts about extending the signature syntax?

Thanks,
Matti
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] summary of "office Hours" open discusison April 25

[Matti Picus]

Office Hours 25April 2018 12:00 -13:00 PDT

Present: Matti Picus, Allan Haldane, Ralf Gommers, Matthew Brett, Tyler
Reddy, Stéfan van der Walt, Hameer Abbasi

Some of the people were not present for the entire discussion, audio was a
little flaky at times.

Topics:

Grant background overview

Matti has been browsing through issues and pull-requests to try to get a
handle on common themes and community pain points.

- Policy questions:
  - Do we close duplicate issues? (answer - Yes, referencing the other issue,
as long as they are true duplicates )
  - Do we close tutorial-like issues that are documented?(answer - Yes, maybe
improving documentation)
- Common theme - there are many issues about overflow, mainly about int32.
  Maybe add a mode or command switch for warning on int32 overflow?

- Requested topic for discussion - improving CI and MacOS testing
  - How to filter CI issues on github? There is a component:build label but
it is not CI specific
  - What about MacOS testing - should it be sending notices? (answer -
Probably)
  - Running ASV benchmarking (https://asv.readthedocs.io/en/latest/). It is
done with SciPy, but it is fragile, not done nightly; need ability to run
branches more robustly documentation on SciPy site
https://github.com/scipy/scipy/tree/master/benchmarks
- Hameer: f2py during testing is the system one, not the internal one

Most of the remaining discussion was a meta-discussion about how the
community will continue to decide priorities and influence how the full-time
developers spend their time.

- Setting up a community-driven roadmap would be useful
- Be aware of the risks of having devoted developer time on a
  community project
- Influence can be subtle: ideally, community writes roadmap, instead
  of simply commenting on proposal
- Can we distill past lessons to inform future decisions?
- In general, how to determine community priorities?
- Constant communication paramount, looks like things are going in
  the right direction.

Furher resources to consider:
- How did Jupyter organize their roadmap (ask Brian Granger)?
- How did Pandas run the project with a full time maintainer (Jeff Reback)?
- Can we copy other projects' management guidelines?

We did not set a time for another online discussion, since it was felt
that maybe near/during the sprint in May would be appropriate.

I apologize for any misrepresentation.

Matti Picus

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Extending ufunc signature syntax for matmul, frozen dimensions

[Matti Picus]

On 01/05/18 01:45, Allan Haldane wrote:

On 04/29/2018 05:46 AM, Matti Picus wrote:

In looking to solve issue #9028 "no way to override matmul/@ if
__array_ufunc__ is set", it seems there is consensus around the idea of
making matmul a true gufunc, but matmul can behave differently for
different combinations of array and vector:

(n,k),(k,m)->(n,m)
(n,k),(k) -> (n)
(k),(k,m)->(m)

Currently there is no way to express that in the ufunc signature. The
proposed solution to issue #9029 is to extend the meaning of a signature
so "syntax like (n?,k),(k,m?)->(n?,m?) could mean that n and m are
optional dimensions; if missing in the input, they're treated as 1, and
then dropped from the output" Additionally, there is an open pull
request #5015 "Add frozen dimensions to gufunc signatures" to allow
signatures like '(3),(3)->(3)'.

How much harder would it be to implement multiple-dispatch for gufunc
signatures, instead of modifying the signature to include `?` ?

There was some discussion of this last year:

http://numpy-discussion.10968.n7.nabble.com/Changes-to-generalized-ufunc-core-dimension-checking-tp42618p42638.html

That sounded like a clean solution to me, although I'm a bit ignorant of
the gufunc internals and the compatibility constraints.

I assume gufuncs already have code to match the signature to the array
dims, so it sounds fairly straightforward (I say without looking at any
code) to do this in a loop over alternate signatures until one works.

Allan
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

I will take a look at multiple-dispatch for gufuncs.
The discussion also suggests using an axis kwarg when calling a gufunc 
for which there is PR #1108 (https://github.com/numpy/numpy/pull/11018) 
discussion).


Matti
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Extending ufunc signature syntax for matmul, frozen dimensions

[Matti Picus]

On 01/05/18 00:38, Eric Wieser wrote:


I think I’m -1 on this - this just makes things harder on the 
implementers of |_array_ufunc__| who now might have to work out which 
signature matches. I’d prefer the solution where |np.matmul| is a 
wrapper around one of three gufuncs (or maybe just around one with 
axis insertion) - this is similar to how np.linalg already works.


Eric

​

On Mon, 30 Apr 2018 at 14:34 Stephan Hoyer <mailto:sho...@gmail.com>> wrote:


On Sun, Apr 29, 2018 at 2:48 AM Matti Picus mailto:matti.pi...@gmail.com>> wrote:

The proposed solution to issue #9029 is to extend the meaning
of a signature so "syntax like (n?,k),(k,m?)->(n?,m?) could
mean that n and m are optional dimensions; if missing in the
input, they're treated as 1, and
then dropped from the output"


I agree that this is an elegant fix for matmul, but are there
other use-cases for "optional dimensions" in gufuncs?

It feels a little wrong to add gufunc features if we can only
think of one function that can use them.
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org <mailto:NumPy-Discussion@python.org>
https://mail.python.org/mailman/listinfo/numpy-discussion



___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion


I will try to prototype this solution and put it up for comment, 
alongside the multi-signature one.

Matti
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Extending ufunc signature syntax for matmul, frozen dimensions

[Matti Picus]

On 01/05/18 21:08, Marten van Kerkwijk wrote:

Just for completeness: there are *four* gufuncs (matmat, matvec,
vecmat, and vecvec).

I remain torn about the best way forward. The main argument against
using them inside matmul is that in order to decide which of the four
to use, matmul has to have access to the `shape` of the arguments.
This meants that means that `__array_ufunc__` cannot be used to
override `matmul` (or `@`) for any object which does not have a shape.
 From that perspective, multiple signatures is definitely a more
elegant solution.

An advantage of the separate solution is that they are useful
independently of whether they are used internally in `matmul`; though,
then again, with a multi-signature matmul, these would be trivially
created as convenience functions.

-- Marten
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion
My goal is to solve issue #9028, "no way to override matmul/@ if 
__array_ufunc__ is set on other". Maybe I am too focused on that, it 
seems shape does not come into play here.


Given a call to matmul(self, other) it appears to me that the decision 
to commit to self.matmul or to call other.__array_ufunc__("__call__", 
self.matmul, ...) is independent of the shapes and needs only nin and nout.

In other words, the implementation of matmul becomes (simplified):

(matmul(self, other) called)->

    (use __array_ufunc__ and nin and nout to decide whether to defer to 
other's __array_ufunc__ via PyUFunc_CheckOverride which implements NEP13) ->


    (yes: call other.__array_ufunc__ as for any other ufunc),

    (no: call matmul like we currently do, no more __aray__ufunc__ 
testing needed)


So the two avenues I am trying are

1) make matmul a gufunc and then it will automatically use the 
__array_ufunc__ machinery without any added changes, but this requires 
expanding the meaning of a signature to allow dispatch


2) generalize the __array_ufunc__ machinery to handle some kind of 
wrapped function, the wrapper knows about nin and nout and calls 
PyUFunc_CheckOverride, which would allow matmul to work unchanged and 
might support other functions as well.


The issue of whether matmat, vecmat, matvec, and vecvec are functions, 
gufuncs accessible from userland, or not defined at all is secondary to 
the current issue of overriding matmul , we can decide that in the future.
If we do create ufuncs for these variants, calling a.vecmat(other) for 
instance will still resolve to other's __array_ufunc__ without needing 
to explore other's shape.


I probably misunderstood what you were driving at because I am so 
focused on this particular issue.


Matti
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] NumPy sprint May 24-25 at BIDS

[Matti Picus]

A reminder - we will take advantage of a few NumPy developers being at 
Berkeley to hold a two day sprint May 24-25 
https://scisprints.github.io/#may-numpy-developer-sprint 
.
We invite any core contributors who would like to attend and can help if 
needed with travel and accomodations.


Stefan and Matti
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] NumPy sprint May 24-25 at BIDS

[Matti Picus]

On 10/05/18 04:44, Charles R Harris wrote:
On Wed, May 9, 2018 at 2:33 PM, Matti Picus <mailto:matti.pi...@gmail.com>> wrote:


A reminder - we will take advantage of a few NumPy developers
being at Berkeley to hold a two day sprint May 24-25
https://scisprints.github.io/#may-numpy-developer-sprint
<https://scisprints.github.io/#may-numpy-developer-sprint>
<https://scisprints.github.io/#may-numpy-developer-sprint
<https://scisprints.github.io/#may-numpy-developer-sprint>>.
We invite any core contributors who would like to attend and can
help if needed with travel and accomodations.

Stefan and Matti
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org <mailto:NumPy-Discussion@python.org>
https://mail.python.org/mailman/listinfo/numpy-discussion
<https://mail.python.org/mailman/listinfo/numpy-discussion>


Hi Matti,

I need to know some details:

 1. Where is the meeting
 2. When is the meeting
 3. Where are good places to stay
 4. Is there a recommended airport

I expect the university has a page somewhere with useful information 
for visitors, a link would be helpful. I've been to SV several times, 
but the last time I was in Berkeley was in 1969 :)


Chuck


We will meet at the BIDS inside the Berkeley campus, more info on 
travel, location and accommodation is here 
https://bids.berkeley.edu/about/directions-and-travel


Matti
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] NumPy sprint May 24-25 at BIDS

[Matti Picus]

On 09/05/18 13:33, Matti Picus wrote:
A reminder - we will take advantage of a few NumPy developers being at 
Berkeley to hold a two day sprint May 24-25 
https://scisprints.github.io/#may-numpy-developer-sprint 
<https://scisprints.github.io/#may-numpy-developer-sprint>.
We invite any core contributors who would like to attend and can help 
if needed with travel and accomodations.


Stefan and Matti
So far I know about Stefan, Nathaniel, Chuck and me. Things will work 
better if we can get organized ahead of time. Anyone else planning on 
attending for both days or part of the sprint, please drop me a line. If 
there are any issues, pull requests, NEPs, or ideas you would like us to 
work on please let me know, or add it to the Trello card 
https://trello.com/c/fvSYkm2w


Matti
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] matmul as a ufunc

[Matti Picus]

I have made progress with resolving the issue that matmul, the operation 
which implements `a @ b`, is not a ufunc [2]. Discussion on the issue, 
which prevents the __array_ufunc__ mechanism for overriding matmul on 
subclasses of ndarray, yeilded two approaches:


- create a wrapper that can convince the ufunc mechanism to call 
__array_ufunc__ even on functions that are not true ufuncs


- expand the semantics of core signatures so that a single matmul ufunc 
can implement matrix-matrix, vector-matrix, matrix-vector, and 
vector-vector multiplication.


I have put up prototypes of both approaches as pr 11061 [0] and 11133 
[1], they are WIP to prove the concept and are a bit rough. Either 
approach can be made to work.


Which is preferable?

What are the criteria we should use to judge the relative merits 
(backward compatibility, efficiency, code clarity, enabling further 
enhancements, ...) of the approaches?


Matti

[0] https://github.com/numpy/numpy/pull/11061
[1] https://github.com/numpy/numpy/pull/11133
[2] https://github.com/numpy/numpy/issues/9028
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Splitting MaskedArray into a separate package

[Matti Picus]

MaskedArray is a strange but useful creature. This NEP proposes to 
distribute it as a separate package under the NumPy brand.


As I understand the process, a proposed NEP should be first discussed 
here to gauge general acceptance, then after that the details should be 
discussed on the pull request itself 
https://github.com/numpy/numpy/pull/11146.


Here is the motivation section from the NEP:


MaskedArrays are a sub-class of the NumPy ``ndarray`` that adds
masking capabilities, i.e. the ability to ignore or hide certain array
values during computation.

While historically convenient to distribute this class inside of NumPy,
improved packaging has made it possible to distribute it separately
without difficulty.

Motivations for this move include:

 * Focus: the NumPy package should strive to only include the
   `ndarray` object, and the essential utilities needed to manipulate
   such arrays.
 * Complexity: the MaskedArray implementation is non-trivial, and imposes
   a significant maintenance burden.
 * Compatibility: MaskedArray objects, being subclasses of `ndarrays`,
   often cause complications when being used with other packages.
   Fixing these issues is outside the scope of NumPy development.

This NEP proposes a deprecation pathway through which MaskedArrays
would still be accessible to users, but no longer as part of the core
package.


Any thoughts?

Matti and Stefan


___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Allowing broadcasting of code dimensions in generalized ufuncs

[Matti Picus]

On 31/05/18 08:23, Allan Haldane wrote:

Re: implenetation complexity, I just want to bring up multiple-dispatch
signatures again, where the new signature syntax would just be to join
some signatures together with "|", and try them in order until one works.

I'm not convinced it's better myself, I just wanted to make sure we 
are aware of it. The translation from the current proposed syntax 
would be:


  Current Syntax    Multiple-dispatch syntax

(n|1),(n|1)->()  <===>   (n),(n)->() | (n),()->() | (),(n)->()


(m?,n),(n,p?)->(m?,p?)  <===> (m,n),(n,p)->(m,p) |
  (n),(n,p)->(p) |
  (m,n),(n)->(m) |
  (n),(n)->()

...
Cheers,
Allan


I am -1 on multiple signatures. We may revisit this in time, but for now 
I find the minimal intrusiveness of the current changes appealing, 
especially as it requires few to no changes whatsoever to the inner loop 
function. Multiple dispatch could easily break that model by allowing 
very different signatures to be aggregated into a single ufunc, leading 
to unhandled edge cases and strange segfaults. It also seems to me that 
looping over all signatures might slow down ufunc calling, leading to 
endless variations of strategies of optimizing signature ordering.


Matti
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] A roadmap for NumPy - longer term planning

[Matti Picus]

At the recent NumPy sprint at BIDS (thanks to those who made the trip) 
we spent some time brainstorming about a roadmap for NumPy, in the 
spirit of similar work that was done for Jupyter. The idea is that a 
document with wide community acceptance can guide the work of the 
full-time developer(s), and be a source of ideas for expanding 
development efforts.


I put the document up at 
https://github.com/numpy/numpy/wiki/NumPy-Roadmap, and hope to discuss 
it at a BOF session during SciPy in the middle of July in Austin.


Eventually it could become a NEP or formalized in another way.

Matti
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] NEP: Dispatch Mechanism for NumPy’s high level API

[Matti Picus]

On 05/06/18 14:11, Stephan Hoyer wrote:
On Tue, Jun 5, 2018 at 12:35 PM Marten van Kerkwijk 
mailto:m.h.vankerkw...@gmail.com>> wrote:


Things would, I think, make much more sense if
`ndarray.__array_ufunc__` (or `*_function__`) actually *were* the
implementation for array-only. But while that is something I'd
like to eventually get to, it seems out of scope for the current
discussion.


If this is a desirable end-state, we should at least consider it now 
while we are designing the __array_function__ interface.


With the current proposal, I think this would be nearly impossible. 
The challenge is that ndarray.__array_function__ would somehow need to 
call the non-overloaded version of the provided function provided that 
no other arguments overload __array_function__. However, currently 
don't expose this information in any way.


Some ways this could be done (including some of your prior suggestions):
- Add a coerce=True argument to all NumPy functions, which could be 
used by non-overloaded implementations.
- A separate namespace for non-overloaded functions (e.g., 
numpy.array_only).
- Adding another argument to the __array_function__ interface to 
explicitly provide the non-overloaded implementation (e.g., func_impl).


I don't like any of these options and I'm not sure I agree with your 
goal, but the NEP should make clear that we are precluding this 
possibility.




What is the difference between the `func` provided as the first argument 
to `__array_function__` and `__array_ufunc__` and the "non-overloaded 
version of the provided function"?


This NEP calls it an "arbitrary callable".
In `__array_ufunc__` it turns out people count on it being exactly the 
`np.ufunc`.


Matti
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] 1.14.5 bugfix release

[Matti Picus]

  
  
If there is a desire to do a bug-fix release 1.14.5 I would like to
try my hand at releasing it, using doc/RELEASE_WALKTHROUGH.rst.txt.
There were a few issues around compiling 1.14.4 on alpine and
NetBSD.
Since 1.15 will probably be released soon, do we continue to push
these kind of bug fixes releases to 1.14.x?
Matti
  

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] SciPy 2018

[Matti Picus]

On 12/06/18 14:26, Charles R Harris wrote:

Hi All,

Thought I'd raise the topic of meeting up at SciPy 2018. I wasn't 
planning on registering for the main conference, but would be happy to 
fly down for a couple of days if we plan on a meetup during sprints or 
some other point in the conference schedule.


Chuck


There will be a NumPy sprint July 14-15. I have requested a BOF room.
For the BOF, I hoped to continue the discussion of the NumPy roadmap 
https://github.com/numpy/numpy/wiki/NumPy-Roadmap as well as provide a 
forum to meet in person.


Matti
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Permissions to upload to PyPI

[Matti Picus]

Almost ready to finish the 1.14.5 release, but it seems I need 
permissions to upload to PyPI (makes sense). My user name there is 
mattip. Can someone help out?

Matti
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] NumPy 1.14.5 released

[Matti Picus]

Hi All,

I am pleased to announce the release of NumPy 14.4.5.

This is a bugfix release for bugs reported following the 1.14.4 release. The
most significant fixes are:

* fixes for compilation errors on alpine and NetBSD

The Python versions supported in this release are 2.7 and 3.4 - 3.6. The 
Python

3.6 wheels available from PIP are built with Python 3.6.2 and should be
compatible with all previous versions of Python 3.6. The source releases 
were

cythonized with Cython 0.28.2 and should work for the upcoming Python 3.7.

Contributors


A total of 1 person contributed to this release.  People with a "+" by their
names contributed a patch for the first time.

* Charles Harris

Pull requests merged


A total of 2 pull requests were merged for this release.

* `#11274 `__: BUG: Correct 
use of NPY_UNUSED.
* `#11294 `__: BUG: Remove 
extra trailing parentheses.


Cheers,
Matti
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Circle CI moving from 1.0 to 2.0

[Matti Picus]

I stumbled across this notice (only seems to appear in a failed build)

"This project is currently running on CircleCI 1.0 which will no longer 
be supported after August 31, 2018. Please start migrating this project 
to CircleCI 2.0 ."


Here is the original link https://circleci.com/gh/numpy/numpy/2080

Is this an artifact that can be ignored or do we need to migrate, if so 
has anyone already done it for their project?

Matti
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] rackspace ssl certificates

[Matti Picus]

  
  
On 19/06/18 09:58, Charles R Harris wrote:

  

  

  

  > What I was curious about is
that there were no more "daily" builds of
> master.

  

Is that right?  That there were daily builds of master,
on Appveyor?
I don't know how those worked, I only recently got cron
permission ...
  
  
  
  No, but there used to be daily builds on travis. They
stopped 8 days ago, https://travis-ci.org/MacPython/numpy-wheels/builds.
  
  
  Chuck

  

  
  
maybe one of these commits
https://github.com/MacPython/numpy-wheels/commits/daily

Matti
  

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Remove sctypeNA and typeNA from numpy core

[Matti Picus]

numpy.core has many ways to catalogue dtype names: sctypeDict, typeDict 
(which is precisely sctypeDict), typecodes, and typename. We also 
generate sctypeNA and typeNA but, as issue 11241 shows, it is sometimes 
wrong. They are also not documented and never used inside numpy. Instead 
of fixing it, I propose to remove sctypeNA and typeNA.


Any thoughts or objections?
Matti
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Remove sctypeNA and typeNA from numpy core

[Matti Picus]

On 21/06/18 09:25, Matti Picus wrote:
numpy.core has many ways to catalogue dtype names: sctypeDict, 
typeDict (which is precisely sctypeDict), typecodes, and typename. We 
also generate sctypeNA and typeNA but, as issue 11241 shows, it is 
sometimes wrong. They are also not documented and never used inside 
numpy. Instead of fixing it, I propose to remove sctypeNA and typeNA.


Any thoughts or objections?
Matti

Whoops  11340 (not 11241) which has been merged.
Matti
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] rackspace ssl certificates

[Matti Picus]

On 19/06/18 10:57, Matthew Brett wrote:

Hi,

On Tue, Jun 19, 2018 at 6:27 PM, Matti Picus  wrote:

On 19/06/18 09:58, Charles R Harris wrote:

What I was curious about is that there were no more "daily" builds of
master.

Is that right?  That there were daily builds of master, on Appveyor?
I don't know how those worked, I only recently got cron permission ...


No, but there used to be daily builds on travis. They stopped 8 days ago,
https://travis-ci.org/MacPython/numpy-wheels/builds.

Oops - yes - sorry - I retired the 'daily' branch, in favor of
'master', but forgot to update the Travis-CI settings.

Done now.

Cheers,

Matthew

FWIW, still no daily builds at 
https://travis-ci.org/MacPython/numpy-wheels/builds

Matti
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Revised NEP-18, __array_function__ protocol

[Matti Picus]

On 28/06/18 17:18, Stephan Hoyer wrote:
On Thu, Jun 28, 2018 at 1:12 PM Marten van Kerkwijk 
mailto:m.h.vankerkw...@gmail.com>> wrote:


For C classes like the ufuncs, it seems `__self__` is defined for
methods as well (at least, `np.add.reduce.__self__` gives
`np.add`), but not a `__func__`. There is a `__name__`
(="reduce"), though, which means that I think one can still
retrieve what is needed (obviously, this also means
`__array_ufunc__` could have been simpler...)


Good point!

I guess this means we should encourage using __name__ rather than 
__func__. I would not want to preclude refactoring classes from Python 
to C/Cython.



___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion
There was opposition to that in a PR I made to provide a wrapper around 
matmul to turn it into a ufunc. It would have left the __name__ but 
changed the __func__.

https://github.com/numpy/numpy/pull/11061#issuecomment-387468084
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Looking for description/insight/documentation on matmul

[Matti Picus]

On 09/07/18 09:48, jeff saremi wrote:
Is there any resource available or anyone who's able to describe 
matmul operation of matrices when n > 2?


The only description i can find is: "If either argument is N-D, N > 2, 
it is treated as a stack of matrices residing in the last two indexes 
and broadcast accordingly." which is very cryptic to me.

Could someone break this down please?
when a [2 3 5 6] is multiplied by a [7 8 9] what are the resulting 
dimensions? is there one answer to that? Is it deterministic?
What does "residing in the last two indices" mean? What is broadcast 
and where?

thanks
jeff



You could do

np.matmul(np.ones((2, 3, 4, 5, 6)), np.ones((2, 3, 4, 6, 7))).shape

which yields (2, 3, 4, 5, 7).

When ndim >= 2 in both operands, matmul uses the last two dimensions as 
(..., n, m) @ (, m, p) -> (..., n, p). Note the repeating "m", so 
your example would not work: n1=5, m1=6 in the first operand and m2=8, 
p2=9 in the second so m1 != m2.


The "broadcast" refers only to the "..." dimensions, if in either of the 
operands you replace the 2 or 3 or 4 with 1 then that operand will 
broadcast (repeat itself) across that dimension to fit the other 
operand. Also if one of the three first dimensions is missing in one of 
the operands it will broadcast.


When ndim < 2 for one of the operands only, it will be interpreted as 
"m", and the other dimension "n" or "p" will not appear on the output, 
so the signature is (..., n, m),(m) -> (..., n) or (m),(..., m, p)->(..., p)


When ndim < 2 for both of the operands, it is the same as  a dot product 
and will produce a scalar.


You didn't ask, but I will complete the picture: np.dot is different for 
the case of n>=2. The result will extend (combine? broadcast across?) 
both sets of ... dimensions, so


np.dot(np.ones((2,3,4,5,6)), np.ones((8, 9, 6, 7))).shape

which yields (2, 6, 4, 5, 8, 9, 7). The (2, 3, 4) dimensions are 
followed by (8, 9)


Matti
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Fwd: [NumFOCUS Projects] NumFOCUS Summit Registration

[Matti Picus]

On 06/07/18 00:54, Ralf Gommers wrote:

Hi all,

In September the NumFOCUS Summit for 2018 will be held in New York. 
NumPy can send one representative (or a couple, but costs are only 
covered for one person). We opened this opportunity up to members of 
the Steering Council first, however it doesn't look like anyone is in 
the position to attend. Therefore we'd now like to ask if anyone else 
would like to attend.


Rules of the game: you do need a strong affiliation with the project 
(e.g. you have commit right, have been contributing for a while, have 
been hired to work on NumPy, etc.); in case multiple people are 
interested then the NumPy Steering Council will make a decision.


If you're interested, please reply here or off-list.

Cheers,
Ralf



I would like to put my name forward.
Matti



-- Forwarded message --
From: *Leah Silen* mailto:l...@numfocus.org>>
Date: Wed, Jun 27, 2018 at 4:35 PM
Subject: [NumFOCUS Projects] NumFOCUS Summit Registration
To: proje...@numfocus.org 


*

Hi all,


We’d like to share some additional details on this year’s NumFOCUS 
Project Summit.



The Sustainability Workshop portion of the Summit (Sept 22-23) is an 
opportunity for projects to focus on creating roadmaps and developing 
strategies for long-term sustainability. This year we would like your 
help in generating the content and driving the direction of the 
workshop. We’ll be reaching out for your suggestions on specific areas 
you would like to see addressed.



The Project Forum for Core Developers and Critical Users(Sep 24-25) is 
a meeting, open to the public, for critical users to directly interact 
with the core developers of NumFOCUS projects. The goal is to connect 
you with resources to advance your community roadmaps while beginning 
a dialogue with key stakeholders. The event will be limited to 150 
attendees. Again, we will be giving participants (both projects and 
users) an opportunity to drive the content and agenda.



NumFOCUS will cover travel expenses for one representative per 
sponsored project.This includes an airfare allotment of $600 and hotel 
accommodations for 4 nights in New York City. The airfare amount is an 
average; we understand international travel costs will exceed this 
amount. Please contact us if you would like to have more than one rep 
attend. When choosing who will attend, we ask that you prioritize 
members of your leadership team or steering committee as this will 
help your project get the most benefit out of the Summit experience.



In order to book hotel rooms, it is urgent that we know who will be 
attending as soon as possible.We’ve set up the following site for 
project leads to register: 



Once again, if you’re interested in working on any portion of the 
summit, we would love your input and leadership in setting the agenda. 
Please reach out to sum...@numfocus.org if 
you would like to be involved.




NumFOCUS Summit Committee-


Andy Terrel

James Powell

Leah Silen

Gina Helfrich

Jim Weiss


*
---
Leah Silen
Executive Director, NumFOCUS
l...@numfocus.org 
512-222-5449 






___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] we held an impromptu dtype brainstorming sesison at SciPy

[Matti Picus]

The stars all aligned properly and some of the steering committee 
suggested we put together a quick brainstorming session over what to do 
with dtypes.
About 20 people joined in the discussion which was very productive. We 
began with user stories and design requirements, and asked some present 
to spend 5 minutes and create a straw-man implementation of what their 
dream dtype implementation would contain. The resulting document 
https://github.com/numpy/numpy/wiki/Dtype-Brainstorming will serve as 
the basis for a future NEP and more work toward a better, 
user-extensible dtype.


More comments are welcome, the discussion is only at the beginning stages.

Matti
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Commit privileges

[Matti Picus]

On 19/07/18 14:08, Charles R Harris wrote:

Hi All,

The NumPy Steering Council has been looking at commit rights for the 
NumPy developers hired at BIDS. We would like them to be able to label 
PRs, close issues, and merge simple fixes; doing that requires commit 
privileges. OTOH, it is also the case that people paid to work on 
NumPy don't automatically receive commit privileges. So it is a bit a 
quandary and we don't seem to have an official document codifying the 
giving of commit privileges, and the Github privileges are rather 
coarse grained, pretty much all or nothing for a given repository.


The situation has also caused us to rethink commit privileges in 
general, perhaps we are being too selective. So there is some interest 
in offering commit privileges more freely, with the understanding that 
they are needed for many of the mundane tasks required to maintain 
NumPy, but that new people should be conservative in their exercise of 
the privilege. Given the reality of the Github environment, such a 
system needs be honor based, but would allow more people an 
opportunity to participate at a deeper level.


So in line with that, we are going to give both of the BIDS workers 
commit privileges, but also extend the option of commit privileges for 
issue triage and other such things to the community at large. If you 
have contributed to NumPy and are interested in having commit rights, 
please respond to this post, but bear in mind that this is an 
experiment and that things might change if the system is abused.


Chuck



I rephrased this mail as an addition to the governance document, see 
https://github.com/numpy/numpy/pull/11609


Matti
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] NEP 15, 20 implementations waiting for review

[Matti Picus]

Two largish pull requests that implement approved NEPS are waiting for 
review:


https://github.com/numpy/numpy/pull/11175 for expanded gufunc signatures 
(NEP 20)


https://github.com/numpy/numpy/pull/10915 for merging multiarray and 
umath c-extension modules (NEP 15)


I realize reviewer time is precious, and appreciate that these PRs are 
perhaps more complex to review than others, but it would be nice to keep 
the good momentum we have in the NEP process moving forward.

Thanks,
Matti
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Taking back control of the #numpy irc channel

[Matti Picus]

Over the past few days spambots have been hitting freenode's IRC 
channels[0, 1]. It turns out the #numpy channel has no operator, so we 
cannot make the channel mode "|+q $~a"[2] - i.e. only registered 
freenode users can talk but anyone can listen.


I was in touch with the freenode staff, they requested that someone from 
the steering council reach out to them at ||proje...@freenode.net, here 
is the quote from the discussion:


"
it's pretty much a matter of them sending an email telling us who they'd 
like to represent them on freenode, which channels and cloak namespaces 
they want, and any info we might need on the project

"

In the mean time they set the channel mode appropriately, so this is 
also a notice that if you want to chat on the #numpy IRC channel you 
need to register.


Hope someone from the council picks this up and reaches out to them, and 
will decide who is to able to become channel operators (the recommended 
practice is to use it like sudo, only assume the role when needed then 
turn it back off).


Matti

[0] https://freenode.net/news/spambot-attack
[1] https://freenode.net/news/spam-shake
[2] https://nedbatchelder.com/blog/201808/fighting_spam_on_freenode.html
|
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] pytest, fixture and parametrize

[Matti Picus]

On 08/08/18 17:34, Juan Nunez-Iglesias wrote:
A NumPy document on preferred testing practices would be very valuable 
even to projects beyond NumPy, just like HOW_TO_DOCUMENT.txt was 
valuable to the entire ecosystem.


We have these guidelines 
http://www.numpy.org/devdocs/reference/testing.html, or if you prefer 
the source https://github.com/numpy/numpy/blob/master/doc/TESTS.rst.txt. 
It was updated for pytest in the 1.15 release, but could use some more 
editing and refinement.

Matti
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

[Matti Picus]

  
  
On 15/08/18 19:44, Matthew Brett wrote:

  My suspicion is, to the extent
that Matti and Tyler can devote time and energy to shepherding the
discussion, these will become quicker and more productive.

Since my name was mentioned ..
Even if we could implement pull requests immediately when issues are
reported, and reply to all mails, careful review and community
consensus take time. See for instance the merge-umath-and-multiarray
PR #10915 (NEP 15) and the generalized-ufunc PR #11175 (NEP20).
Progress within NumPy is and always will be slow. If there is an
expectation from the community that we take a more active role in
moving things forward, that is feedback we need to hear, hopefully
with concrete suggestions, preferably in a separate email thread.

Whatever direction __array_function__ or other protocols take, we
should create a page of links to implementations of
__array_function__ (or for that matter a dtype, gufunc, ndarray, or
an __array_ufunc__) so that we can gather data on how these are
being used. It would also mean we could test downstream packages
when changes are proposed, and we would know who to reach out to
when issues arise.

Matti
  

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

[Matti Picus]

On 29/08/18 10:37, Nathaniel Smith wrote:

it's easy to imagine scenarios where the
people being broken aren't the ones who had a chance to read the docs
– e.g. if a major package starts relying on __array_function__, then
it's all*their*  users who we'd be breaking, even though they had
nothing to do with it.
This is a packaging problem. This proposal is intended for use by other 
"major packages", not so much for end-users. We would have much more 
trouble if we were proposing a broad change to something like indexing 
or the random number module (see those NEPs). If we break one of those 
major packages, it is on them to pin the version of NumPy they can work 
with. In my opinion very few end users will be implementing their own 
ndarray classes with `__array_function__`. While we will get issue 
reports, we can handle them much as we do the MKL or OpenBLAS ones - 
pinpoint the problem and urge users to complain to those packages.


Other than adding a warning, I am not sure what the concrete proposal is 
here. To not accept the NEP?

Matti
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Adding `max_rows` to `loadtxt'

[Matti Picus]

A new contributor submitted a PR[0] to add `max_rows`[1] to `loadtxt`, 
like is done in 'genfromtxt', (which is used under the hood for 
'ndfromtxt', 'mafromtxt', and

'recfromtxt`).  Any thoughts?

[0] https://github.com/numpy/numpy/pull/11962
[1] Well, actually `maxlines`, but I asked to change it to `max_rows` to 
be consistent with `genfromtxt`.


Matti
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Search not working on www.numpy.org/devdocs/search.html

[Matti Picus]

I can enter a search term (say `ndarray`) in 
www.numpy.org/devdocs/search.html, but the result is empty. It worked 
yesterday. My firefox javascript debug console for the remote search says:


jQuery.Deferred exception: Search is not defined 
@http://www.numpy.org/devdocs/search.html?q=ndarray:34:25 
j@http://www.numpy.org/devdocs/_static/jquery.js:2:29997 
g/http://www.numpy.org/devdocs/_static/jquery.js:2:30313 undefined


If I build the documentation locally, search emits a different error but 
still works:


XML Parsing Error: not well-formed Location: 
file:///.../doc/build/html/searchindex.js Line Number 1, Column 16:


Searching on scipy.org, for instance 
https://docs.scipy.org/doc/numpy/search.html?q=ndarray&check_keywords=yes&area=default 
works with no errors on the console


Any hints?
Matti
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Adding a hex version like PY_VERSION_HEX

[Matti Picus]

In PR 12074 https://github.com/numpy/numpy/pull/12074 I propose adding a 
function `version.get_numpy_version_as_hex()` which returns a hex value 
to represent the current NumPy version MAJOR.MINOR.MICRO where


v = hex(MAJOR << 24 | MINOR << 16 | MICRO)

so the current 1.15.0 would become '0x10f'. I also made this 
avaiable via C through `get_hex_version`. The hex version is based on 
the |PY_VERSION_HEX| macro from CPython.


Currently we have a ABI version and an API version for the numpy C-API. 
We only increment those for updated or breaking changes in the NumPy 
C-API, but not for


- changes in behavior, especially in python code

- changes in sizes of outward-facing structures like PyArray_Desc

Occasionally it is desirable to determine backward compatibility from 
the runtime version, rather than from the ABI or API versions, and 
having it as a single value makes the comparison in C easy. For instance 
this may be convenient when there is suspicion that older header files 
may have been used to create or manipulate an object directly in C (or 
via a cython optimization), and we want to verify the version used to 
create the object, or when we may want to verify de-serialized objects. 
The `numpy.lib._version.NumpyVersion` class enables version comparison 
in python, but I would prefer a single value that can be stored in a C 
struct as an integer type.


Since this is an enhancement proposal, I am bringing the idea to the 
mailing list for reactions.


Matti
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Adding a hex version like PY_VERSION_HEX

[Matti Picus]

On 05/10/18 11:46, Jerome Kieffer wrote:

On Fri, 5 Oct 2018 11:31:20 +0300
Matti Picus  wrote:


In PR 12074 https://github.com/numpy/numpy/pull/12074 I propose adding a
function `version.get_numpy_version_as_hex()` which returns a hex value
to represent the current NumPy version MAJOR.MINOR.MICRO where

v = hex(MAJOR << 24 | MINOR << 16 | MICRO)

+1

We use it in our code and it is a good practice, much better then 0.9.0>0.10.0 !

We added some support for dev, alpha, beta, RC and final versions in
https://github.com/silx-kit/silx/blob/master/version.py

Cheers,

Thanks. I think at this point I will change the proposal to

v = hex(MAJOR << 24 | MINOR << 16 | MICRO << 8)

which leaves room for future enhancement with "release level" and "serial" as 
the lower bits.

Matti

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Determining NPY_ABI_VERSION statically in compiled extensions

[Matti Picus]

On 08/10/18 23:31, Robert T. McGibbon wrote:
Is anyone aware of any tricks that can be played with tools like 
`readelf`, `nm` or `dlopen` / `dlsym` in order to statically determine 
what version of numpy a fully-compiled C extension (for example, found 
inside a wheel) was compiled against? Even if it only worked with 
relatively new versions of numpy, that would be fine.


I'm interested in creating something similar to 
https://github.com/pypa/auditwheel that could statically check for 
compatibility between wheel files and python installations, in 
situations where the metadata about how they were compiled is missing.

--
-Robert

NPY_ABI_VERSION is exposed in C as PyArray_GetNDArrayCVersion and 
NPY_API_VERSION is exposed in C as PyArray_GetNDArrayCFeatureVersion. 
These are not incremented for every NumPy release, see the documentation 
in numpy/core/setup_common.py.


The numpy.__version__ is determined by a python file numpy/version.py, 
which is probably what you want to use.


There is an open Issue to better reveal compile time info 
https://github.com/numpy/numpy/issues/10983


Matti
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Approving NEP 27 - Historical discussion of 0-D arrays

[Matti Picus]

In PR 12166 https://github.com/numpy/numpy/pull/12166 we revived an old 
wiki document discussing the implementation of 0-dimensional arrays. 
This became informational NEP-27 
http://www.numpy.org/neps/nep-0027-zero-rank-arrarys.html. There was 
fruitful discussion of the NEP and the need for both 0-D arrays and 
scalars on the PR comments. The NEP itself is informational and freezes 
the information to the 2006 discussion, noting that "some of the 
information here is dated, for instance indexing of 0-D arrays now is 
now implemented and does not error."



I would like to submit the NEP for discussion and approval.

Matti

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Removing priority labels from github

[Matti Picus]

We currently have highest, high, normal, low, and lowest priority labels 
for github issues/PRs. At the recent status meeting, we proposed 
consolidating these to a single "high" priority label. Anything "low" 
priority should be merged or closed since it will be quickly forgotten, 
and no "normal" tag is needed.



With that, we (the BIDS team) would like to encourage reviewers to use 
the "high" priority tag to indicate things we should be working on.


Any objections or thoughts?


Matti (in the names of Tyler and Stefan)


___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] asanyarray vs. asarray

[Matti Picus]

  
  
Was there discussion around which of `asarray` or asanyarray` to
  prefer? PR 11162, https://github.com/numpy/numpy/pull/11162,
  proposes `asanyarray` in place of `asarray` at the entrance to `_quantile_ureduce_func` to preserve ndarray
  subclasses. Should we be looking into changing all the
  `asarray` calls into `asanyarray`?


Matti


  

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Reminder: weekly status meeting

[Matti Picus]

Hi everyone,

The team at BIDS meets once a week to discuss progress, priorities, and
roadblocks.  While our priorities are broadly determined by the project
roadmap [0], we would like to provide an opportunity for the community
to give more regular and detailed feedback on our work.

We therefore invite you to join us for our weekly calls,
each **Wednesday from 12:00 to 13:00 Pacific Time**.

Detail of the next meeting (2018-10-24) is given in the agenda [1],
which is a living document. Feel free to add topics you wish to discuss.

We hope to see you there!
Best regards,
Stéfan, Tyler, Matti

[0]https://www.numpy.org/neps/index.html
[1]https://hackmd.io/5WZ6VwQKSbSR_4Ng65pUFw?both

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Attribute hiding APIs for PyArrayObject

[Matti Picus]

TL;DR - should we revert the attribute-hiding constructs in 
ndarraytypes.h and unify PyArrayObject_fields with PyArrayObject?



Background


NumPy 1.8 deprecated direct access to PyArrayObject fields. It made 
PyArrayObject "opaque", and hid the fields behind a PyArrayObject_fields 
structure 
https://github.com/numpy/numpy/blob/v1.15.3/numpy/core/include/numpy/ndarraytypes.h#L659 
with a comment about moving this to a private header. In order to access 
the fields, users are supposed to use PyArray_FIELDNAME functions, like 
PyArray_DATA and PyArray_NDIM. It seems there were thoughts at the time 
that numpy might move away from a C-struct based


underlying data structure. Other changes were also made to enum names, 
but those are relatively painless to find-and-replace.



NumPy has a mechanism to manage deprecating APIs, C users define 
NPY_NO_DEPRICATED_API to a desired level, say NPY_1_8_API_VERSION, and 
can then access the API "as if" they were using NumPy 1.8. Users who do 
not define NPY_NO_DEPRICATED_API get a warning when compiling, and 
default to the pre-1.8 API (aliasing of PyArrayObject to 
PyArrayObject_fields and direct access to the C struct fields). This is 
convenient for downstream users, both since the new API does not provide 
much added value, and it is much easier to write a->nd than 
PyArray_NDIM(a). For instance, pandas uses direct assignment to the data 
field for fast json parsing 
https://github.com/pandas-dev/pandas/blob/master/pandas/_libs/src/ujson/python/JSONtoObj.c#L203 
via chunks. Working around the new API in pandas would require more 
engineering. Also, for example, cython has a mechanism to transpile 
python code into C, mapping slow python attribute lookup to fast C 
struct field access 
https://cython.readthedocs.io/en/latest/src/userguide/extension_types.html#external-extension-types



In a parallel but not really related universe, cython recently upgraded 
the object mapping so that we can quiet the annoying "size changed" 
runtime warning https://github.com/numpy/numpy/issues/11788 without 
requiring warning filters, but that requires updating the numpy.pxd file 
provided with cython, and it was proposed that NumPy actually vendor its 
own file rather than depending on the cython one 
(https://github.com/numpy/numpy/issues/11803).



The problem


We have now made further changes to our API. In NumPy 1.14 we changed 
UPDATEIFCOPY to WRITEBACKIFCOPY, and in 1.16 we would like to deprecate 
PyArray_SetNumericOps and PyArray_GetNumericOps. The strange warning 
when NPY_NO_DEPRICATED_API is annoying. The new API cannot be supported 
by cython without some deep surgery 
(https://github.com/cython/cython/pull/2640). When I tried dogfooding an 
updated numpy.pxd for the only cython code in NumPy, mtrand.pxy, I came 
across some of these issues (https://github.com/numpy/numpy/pull/12284). 
Forcing the new API will require downstream users to refactor code or 
re-engineer constructs, as in the pandas example above.



The question


Is the attribute-hiding effort worth it? Should we give up, revert the 
PyArrayObject/PyArrayObject_fields division and allow direct access from 
C to the numpy internals? Is there another path forward that is less 
painful?



Matti

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Reminder: weekly status meeting 31.10 at 12:00 pacific time

[Matti Picus]

  
  
The draft agenda is at
  https://hackmd.io/D3I3CdO2T9ipZ2g5uAChcA?both.
Everyone is invited to join.


Matti, Tyler and Stefan

  

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Prep for NumPy 1.16.0 branch

[Matti Picus]

On 4/11/18 8:04 pm, Charles R Harris wrote:

Hi All,

Time to begin looking forward to the NumPy 1.16.x branch. I think 
there are three main topics to address:


 1. current PRs that need review and merging,
 2. critical fixes that need to be made,
 3. status of `__array_function__`.

The last probably needs some discussion. `__array_fuction__` seems to 
be working at this point, but does cause noticeable slowdowns in some 
function calls. I don't know if those slowdowns are significant in 
practice, the only way to discover that may be to make the release, or 
at least thorough testing of the release candidates, but we should at 
least discuss them and possible workarounds if needed. Trying to have 
things in good shape is important because 1.16.x will be the last 
release that supports Python 2.7, and even though we will be 
maintaining it for the next year, that will be easier if it is stable. 
As to the first two topics, I think we should try to be conservative 
at this point and look mostly for bug fixes and documentation updates.


Thoughts?

Chuck



Beyond things with the 1.16 milestone, it would be nice to address the 
structured array cleanup 
https://gist.github.com/ahaldane/6cd44886efb449f9c8d5ea012747323b and to 
get the matmul-as-ufunc https://github.com/numpy/numpy/pull/12219 merged



Matti

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Weekly status meeting 8.11 at 12:00 pacific time

[Matti Picus]

We will be holding our weekly BIDS NumPy status meeting on Thurs Nov 8 
at noon pacific time. We moved the meeting to Thursday because of a 
scheduling conflict. Please join us.


The draft agenda, along with details of how to join, is up at 
https://hackmd.io/TTurMvviSkarcxf8vURq-Q?both


Previous sessions' notes are available at 
https://github.com/BIDS-numpy/docs/tree/master/status_meetings



Matti, Tyler and Stefan

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] asarray/anyarray; matrix/subclass

[Matti Picus]

On 9/11/18 5:09 pm, Nathaniel Smith wrote:

On Fri, Nov 9, 2018 at 4:59 PM, Stephan Hoyer  wrote:

On Fri, Nov 9, 2018 at 6:46 PM Nathaniel Smith  wrote:

But matrix isn't the only problem with asanyarray. np.ma also violates
Liskov. No doubt there are other problematic ndarray subclasses out
there too...


Please forgive my ignorance (I don't really use mask arrays), but how
specifically do masked arrays violate Liskov? In most cases shouldn't they
work the same as base numpy arrays, except with operations keeping track of
masks?

Since many operations silently skip over masked values, the
computation semantics are different. For example, in a regular array,
sum()/size() == mean(), but with a masked array these are totally
different operations. So if you have code that was written for regular
arrays, but pass in a masked array, there's a solid chance that it
will silently return nonsensical results.

(This is why it's better for NAs to propagate by default.)

-n



Echos of the discussions in neps 12, 24, 25, 26. http://www.numpy.org/neps


Matti

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Weekly status meeting Nov 14 at 12:00 Pacific time

[Matti Picus]

We will be holding our weekly BIDS NumPy status meeting on Wed Nov 14
at noon Pacific time. Please join us.

The draft agenda, along with details of how to join, is up at
https://hackmd.io/H0x6z5uYSgex2FA6p5nlvw?both

Previous sessions' notes are available at
https://github.com/BIDS-numpy/docs/tree/master/status_meetings


Matti, Tyler and Stefan

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Weekly status meeting Nov 28 at 12:00 Pacific time

[Matti Picus]

We will be holding our weekly BIDS NumPy status meeting on Wed Nov 28
at noon Pacific time. Please join us.

The draft agenda, along with details of how to join, is up at
https://hackmd.io/Gn1ymjwkRjm9WVY5Cgbwsw?both

Previous sessions' notes are available at
https://github.com/BIDS-numpy/docs/tree/master/status_meetings


Matti, Tyler and Stefan
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Reminder: Numpy dev meeting Fri-Sat Nov 30-Dec 1

[Matti Picus]

We will be meeting at BIDS 9:00AM Friday for a two-day NumPy developer 
meeting. All are welcome, if you haven't already please let Stefan know 
you are coming so we can plan for space. A tentative schedule is here



https://hackmd.io/gFqjPUSvSmm-0gmBDbrTBw?both


Feel free to add content (just tag it with your name).


Tyler, Matti, Stefan.

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] There will be no developer status meeting this week

[Matti Picus]

I am traveling, so we decided to cancel this week's developer online 
meeting. The next one will be Wed Dec 19.



Matti

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Reminder: weekly status meeting Dec 12 at 12:00 pacific time

[Matti Picus]

The draft agenda is at https://hackmd.io/Gn1ymjwkRjm9WVY5Cgbwsw?both

Everyone is invited to join.


Matti, Tyler and Stefan
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Reminder: weekly status meeting Dec 19 at 12:00 pacific time

[Matti Picus]

The draft agenda is at https://hackmd.io/Gn1ymjwkRjm9WVY5Cgbwsw?both

This will be our last meeting of 2018.

The BIDS team
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Warn or immidiately change readonly flag on broadcast_arrays return value?

[Matti Picus]

In PR 12609 https://github.com/numpy/numpy/pull/12609 I added code to 
emit a DepricationWarning when broadcast_arrays returns an array where 
the output is repeated. While this is a minimal fix to the problem, 
perhaps we should consider making the output readonly immediately instead?



- A deprecation cycle requires two changes to downstream user's code: 
one to filter the deprecation warning, and another when we actually make 
the change


- Writing to the repeated data will cause errors now.


What do you think, should we change the behaviour at all, and if so 
should we depricate it over two releases or change it immediately?



The original issue is here https://github.com/numpy/numpy/issues/2705


Matti

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] three-dim array

[Matti Picus]

On 27/12/18 3:21 am, Benjamin Root wrote:
Ewww, kinda wish that would be an error... It would be too easy for a 
typo to get accepted this way.


On Wed, Dec 26, 2018 at 1:59 AM Eric Wieser 
mailto:wieser.eric%2bnu...@gmail.com>> 
wrote:


In the latest version of numpy, this runs without an error,
although may or may not be what you want:

|In [1]: np.array([[1,2],[[1,2],[3,4]]]) Out[1]: array([[1, 2],
[list([1, 2]), list([3, 4])]], dtype=object) |

Here the result is a 2x2 array, where some elements are numbers
and others are lists.

​



Specify the dtype explicitly: `dtype=int` or so, then NumPy will refuse 
to create a ragged array.



There has been occasional discussion of `dtype='not-object'`, but I 
don't think it resulted in an issue or PR.


Matti

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Debugging NumPy development build in conda environment

[Matti Picus]

On 4/1/19 3:34 pm, Lars Grueter wrote:

Unfortunately the last two commands
stopped working out of the blue with my existing environment and fail with


RuntimeError: Broken toolchain: cannot link a simple C program



The compiler error is a few lines above this (copied below). When I have 
gotten that kind of output, it is because I am using a 32 bit compiler 
on 64 bit or the opposite.




C compiler: gcc -pthread -B 
/home/lg/.miniconda3/envs/dev-numpy/compiler_compat -Wl,--sysroot=/ 
-Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC


compile options: '-Inumpy/core/src/common -Inumpy/core/src -Inumpy/core 
-Inumpy/core/src/npymath -Inumpy/core/src/multiarray 
-Inumpy/core/src/umath -Inumpy/core/src/npysort 
-I/home/lg/.miniconda3/envs/dev-numpy/include/python3.7m -c'

gcc: _configtest.c
gcc -pthread -B /home/lg/.miniconda3/envs/dev-numpy/compiler_compat 
-Wl,--sysroot=/ _configtest.o -o _configtest
/home/lg/.miniconda3/envs/dev-numpy/compiler_compat/ld: _configtest.o: 
unable to initialize decompress status for section .debug_info
/home/lg/.miniconda3/envs/dev-numpy/compiler_compat/ld: _configtest.o: 
unable to initialize decompress status for section .debug_info
/home/lg/.miniconda3/envs/dev-numpy/compiler_compat/ld: _configtest.o: 
unable to initialize decompress status for section .debug_info
/home/lg/.miniconda3/envs/dev-numpy/compiler_compat/ld: _configtest.o: 
unable to initialize decompress status for section .debug_info

_configtest.o: file not recognized: file format not recognized



Can you compile anything with that gcc?


Matti

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Reminder: weekly status meeting Jan 9 at 12:00 pacific time

[Matti Picus]

The draft agenda is at https://hackmd.io/D_0X2QCjRpS-ENiFWokc-g?both#

There is a section for community suggested topics, feel free to join the 
conversation and add in topics that need attention.



The BIDS team

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Reminder: weekly status meeting Jan 16 at 12:00 pacific time

[Matti Picus]

The draft agenda is at https://hackmd.io/D_0X2QCjRpS-ENiFWokc-g?both#

There is a section for community suggested topics, feel free to join the 
conversation and add in topics that need attention.



The BIDS team
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Reminder: weekly status meeting Dec 19 at 12:00 pacific time

[Matti Picus]

The draft agenda is at https://hackmd.io/6N3r7yUtSHqUsijC-CEbJA?both

Everyone is invited to join.

Matti, Tyler and Stefan
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy-discussion

[Matti Picus]

On 29/1/19 9:22 am, Yuping Wang wrote:

Dear Nmupy developers and users:
        I am a new user of Numpy ,I have encountered a question about 
Numpy recently, which need your help. The question is below:




     I don  know why there are negative numbers
     can somebody explain to me why these happens
Thanks in advance!

NumPy ndarrays are typed. Since you do not specify a type in arange, 
numpy assumes you want `a` to be the type of 2000. I assume you are on a 
system where numpy converts 2000 to an `int32`, so if you ask what is 
the type of a: (`a.dtype`) it will show `int32`, meaning the values in a 
are interpreted as if they are 32 bit integers.



2000*2000*2000 overflows a 32 bit signed integer, so it wraps around to 
a negative value.



The way around this is to specify what type you wish to use: 
`a=np.arange(2000, dtype=float)**3` or `a=np.arange(2000, 
dtype='int64`)**3 will not overflow. For floats, most CPUs/compilers 
provide an indication when float values overflow, so we can raise a 
warning appropriately. Unfortunately, detecting overflows with ints has 
no hardware support, so adding a warning would mean checking each value 
before a calculation, causing an unacceptable slow down.



Where did you start learning about NumPy? Perhaps you could suggest they 
add this explanation (or if it is in our documentation point out where 
you think would be appropriate) since it seems many people have problems 
with dtypes and overflow.



Matti

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Reminder: weekly status meeting Jan 30 at 12:00 pacific time

[Matti Picus]

The draft agenda is at https://hackmd.io/UnnvbPNpSm-gRpbfTK7cmg?both#

There is a section for community suggested topics, feel free to join the 
conversation and add in topics that need attention.



The BIDS team
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Reminder: weekly status meeting Tues Feb 19 12:00 PST

[Matti Picus]

This week the status meeting will be on Tues Feb 19 at 12:00 PST, not at 
the usual time



The draft agenda is at https://hackmd.io/TtqnUDvPQgaGeej7zR8seA?both

Everyone is invited to join.


Matti, Tyler and Stefan
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Reminder: weekly status meeting Feb 27 at 12:00 pacific time

[Matti Picus]

The draft agenda is at https://hackmd.io/TtqnUDvPQgaGeej7zR8seA?both#


There is a section for community suggested topics, feel free to join the 
conversation and add in topics that need attention.


Past meeting notes can be seen at 
https://github.com/BIDS-numpy/docs/tree/master/status_meetings



The BIDS team

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Removing undocumented __buffer__ attribute lookup

[Matti Picus]

In digging around the code, I found a gem in PyArray_FromBuffer (exposed 
to python as numpy.frombuffer). If a PyObject* does not have a 
tp_as_buffer->bf_getbuffer function, we check if the python object has a 
__buffer__ attribute. If so we use that as buf in 
PyObject_GetBuffer(buf, ...).



This seems to stem back to the original numerics code, where getBuffer 
would look up the attribute and call it as a method. PyArray_FromBuffer 
does not call the attribute as a method, it simply passes it on to 
PyObject_GetBuffer, which will then raise an error saying it cannot 
convert a method. You can try this out by creating a class with a 
__buffer__ method and calling numpy.frombuffer on it.



I submitted a pull request to remove the code. Since it is undocumented 
and (as far as I can tell) broken, I do not think we need a deprecation 
cycle.



More details, including links to the original numeric code from 2005, in 
the PR https://github.com/numpy/numpy/pull/13049



Any thoughts or objections?


Matti

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Outreachy applicant

[Matti Picus]

On 12/3/19 2:22 pm, asish kr wrote:
Hi,  i am an outreachy applicant and would like to contribute to this 
project.  Is there anyone who could guide me here. Thanks





I responded privately to the applicant. More information about our 
Outreachy mentoring is here 
https://www.outreachy.org/may-2019-august-2019-outreachy-internships/communities/numpy/improve-c-api-documentation-for-numpy/cfp



Matti

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Testing your contributions

[Matti Picus]

On 16/3/19 9:59 am, Sandy wrote:

Hey there,

I am new to numpy. I just wanted to know, suppose I make a code
contribution to the repo https://github.com/numpy/numpy how do I test
its working?
___



Welcome.

You should add a test.

Running `python runtest.py` will build and run all the tests, including 
your new one.


More information is available at 
www.numpy.org/devdocs/reference/testing.html


More information about a typical development workflow is at 
www.numpy.org/devdocs/dev/gitwash/development_workflow.html



Please pinpoint unclear documentation by opening an issue 
https://github.com/numpy/numpy/issues.



In general, you will get a more sympathetic response by trying something 
first and reporting what didn't work or what was unclear.



Matti

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Outreachy applicant: Improve C API Documentation for Numpy

[Matti Picus]

On 17/3/19 3:33 pm, Nisha Aggarwal wrote:

Hello,
I am an Outreachy applicant interested in the project: Improve C API 
Documentation for Numpy, but I am unsure of how to proceed. Can 
someone guide me here?

Thanks



Hi and welcome.

The relevant issues have the DOC tag, you can see them via

https://github.com/numpy/numpy/issues?q=is%3Aissue+is%3Aopen+doc+label%3A%2204+-+Documentation%22

Especially relevant are

https://github.com/numpy/numpy/issues/13013 which I think someone is 
already working on


https://github.com/numpy/numpy/pull/13104 which can be broken into a 
number of smaller PRs


https://github.com/numpy/numpy/issues/12632

https://github.com/numpy/numpy/issues/12385

and

https://github.com/numpy/numpy/pull/12097 which is a wip to add a 
version selector. This is not really connected with python or numpy, and 
is a pure javascript issue.



Our workflow is based on github, for an introduction see 
http://www.numpy.org/devdocs/dev/gitwash/development_workflow.html.


For instructions on building the documentation, which uses sphinx, see, 
http://www.numpy.org/devdocs/docs/index.html?highlight=documentation



Matti
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Outreachy Program Aspirant

[Matti Picus]

On 20/3/19 12:20 pm, rishab lamba wrote:


Hi, I am Rishab Lamba undergoing my B.Tech from BVCOE, India. As an 
applicant from Outreachy, I was going through this project 
https://www.outreachy.org/may-2019-august-2019-outreachy-internships/communities/numpy/#improve-c-api-documentation-for-numpy 
and would like to work on it. I was not aware of such a program and 
hence cutting it really close to the deadline but I still hope if I 
can make contributions as much as possible to the numpy repository. 
For this, any guidance for submitting the PRs will be appreciated



Hi and welcome.

The relevant issues have the DOC tag, you can see them via

https://github.com/numpy/numpy/issues?q=is%3Aissue+is%3Aopen+doc+label%3A%2204+-+Documentation%22 



Especially relevant are

https://github.com/numpy/numpy/issues/13013 which I think someone is 
already working on


https://github.com/numpy/numpy/pull/13104 which can be broken into a 
number of smaller PRs


https://github.com/numpy/numpy/issues/12632

https://github.com/numpy/numpy/issues/12385

and

https://github.com/numpy/numpy/pull/12097 which is a wip to add a 
version selector. This is not really connected with python or numpy, and 
is a pure javascript issue.



Our workflow is based on github, for an introduction see 
http://www.numpy.org/devdocs/dev/gitwash/development_workflow.html.


For instructions on building the documentation, which uses sphinx, see, 
http://www.numpy.org/devdocs/docs/index.html?highlight=documentation


Feel free to reach out to me directly with further questions.
Matti
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] prpp report—please help review!

[Matti Picus]

On 20/3/19 10:09 pm, Stefan van der Walt wrote:

Hi everyone,

During the community call today, we drew up a list of open PR counts 
per developer.  We are working towards bringing down their number 
(around 240 today), and would appreciate it if you could help us 
review and/or make decisions.


NumPy also has 1760 open issues 
; many of these are probably 
invalid—but just clicking through them all takes a lot of time.  So if 
you want to grab 5 random ones and check their status, that would be 
very helpful.


Best regards,
Stéfan



One thing that might help is if we could work through the 12 PRs that 
have a "Needs Decision" label 
https://github.com/numpy/numpy/pulls?q=is%3Aopen+is%3Apr+label%3A%2254+-+Needs+decision%22. 
These are waiting for someone to say Yay or Nay.



Matti

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Introducing Myself

[Matti Picus]

On 25/3/19 4:08 pm, Vrinda Narayan wrote:

Heyy,
My name is Vrinda Narayan and I'm an Outreachy Applicant. I'm a CSE 
undergraduate and I know python, C, JavaScript and git. I am 
interested in contributing to the Numpy project Improve C API 
documentation for Numpy.
I have used Numpy for various projects in my college and I really look 
forward to working with Numpy and contributing more to this community.

Warm regards
Vrinda Narayan



Hi Vrinda Narayan


To get you started it would be great if you can try to work on an issue
in numpy and create a pull request (PR) to fix it. That way we can get
to know you a bit and you get to know numpy and our workflow.

New contributors might find it easiest to work on the documentation tag/label,
you can see them via:
https://github.com/numpy/numpy/issues?q=is%3Aissue+is%3Aopen+doc+label%3A%2204+-+Documentation%22

Some relevant issues (note some of these may already have someone
interested in them):

https://github.com/numpy/numpy/issues/13013  which probably someone
is already working on

https://github.com/numpy/numpy/issues/13114  which can be broken into a
number of smaller PRs

https://github.com/numpy/numpy/issues/12632

https://github.com/numpy/numpy/issues/12385

and

https://github.com/numpy/numpy/pull/12097  which is a wip to add a
version selector. This is not really connected with python or numpy,
and is a pure javascript issue.

Our workflow is based on github, for an introduction see
http://www.numpy.org/devdocs/dev/gitwash/development_workflow.html.

For instructions on building the documentation, which uses sphinx, see,
http://www.numpy.org/devdocs/docs/index.html?highlight=documentation  


I hope you can find something to get you started.

Try to work through the process and reach out when you get stuck with
specific questions: "I tried X, expected Y, but got Z".

Feel free to reach out to Sebastian Berg (cc`ed on this mail) or me directly if 
you have any
questions. The issues themselves are also a good place to ask if you
wish to work on one.

Best Regards,
Matti

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion