Re: [Numpy-discussion] NumPy 1.12.0b1 released

[Matthew Brett]

Hi,

On Thu, Nov 17, 2016 at 3:24 PM, Matti Picus  wrote:
> Congrats to all on the release.Two questions:
>
> Is there a guide to building standard wheels for NumPy?

I don't think so - there is a repository that we use to build the
wheels, that has the Windows, OSX and manyllinux recipes for the
standard CPython build:

https://github.com/MacPython/numpy-wheelso

If you can work out a way to automate the PyPy builds and tests -
especially using the same repo - that would be very useful.

> Assuming I can build standardized PyPy 2.7 wheels for Ubuntu, Win32 and
> OSX64, how can I get them blessed and uploaded to PyPI?

If you can automate the build and tests, I'm guessing there will be no
objections - but it's not my call...

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Status of NAs for integer arrays

[Ralf Gommers]

On Fri, Nov 18, 2016 at 8:43 PM, Dmitrii Izgurskii 
wrote:

> Hi,
>
> I would like to know where I can follow the progress of what's being done
> on implementing missing values in integer arrays. Some threads I found are
> from 2012. Is really nothing being done on this issue?
>

Not that I'm aware of, no.

Ralf
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] NumPy 1.12.0b1 released

[Ralf Gommers]

On Fri, Nov 18, 2016 at 9:08 PM, Matthew Brett 
wrote:

> Hi,
>
> On Thu, Nov 17, 2016 at 3:24 PM, Matti Picus 
> wrote:
> > Congrats to all on the release.Two questions:
> >
> > Is there a guide to building standard wheels for NumPy?
>
> I don't think so - there is a repository that we use to build the
> wheels, that has the Windows, OSX and manyllinux recipes for the
> standard CPython build:
>
> https://github.com/MacPython/numpy-wheelso
>
> If you can work out a way to automate the PyPy builds and tests -
> especially using the same repo - that would be very useful.
>
> > Assuming I can build standardized PyPy 2.7 wheels for Ubuntu, Win32 and
> > OSX64, how can I get them blessed and uploaded to PyPI?
>
> If you can automate the build and tests, I'm guessing there will be no
> objections - but it's not my call...
>

I'm in favor, assuming that the wheel tags and PyPy backwards compatibility
situation is OK. Can't really find any examples. What I mean is that for
CPython wheels contain tags like "cp27" or "cp35". PyPy wheels should have
tags "pp". Are the PyPy cpyext layer and the  defined
such that a new PyPy release won't break older wheels?

Ralf
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] ANN: xtensor and xtensor-python

[Sylvain Corlay]

Hi all,

We are pleased to announce the release of the xtensor
 library and its python bindings
xtensor-python , by Johan
Mabille (@JohanMabille) and Sylvain Corlay (@SylvainCorlay).

1. xtensor

xtensor  is a C++ template library
for manipulating multi-dimensional array expressions.

It provides

   - an API following the idioms of the *C++ standard library*.
   - an extensible expression system enabling lazy broadcasting and universal
   functions.
   - python bindings for operating inplace on numpy arrays in your C++
   thanks to Python's buffer protocol and pybind.

More details on lazy computing with xtensor are available below.

2. xtensor-cookiecutter

Besides, xtensor  and xtensor-python
, we provide a cookiecutter
template project to help extension authors get started. The
xtensor-cookiecutter  generates
a simple project for a Python C++ extension with

   - a working setup.py compiling the extension module
   - a few example of functions making use of xtensor exposed to python,
   and the example of a vectorized function exposed to python.
   - all the boilerplate for the setup of the unit testing and generation
   of the HTML documentation for these examples.

3.
You can try it now!

You can try the xtensor live on the project website
. The Try it Now button is powered by

   - The Cling C++ interpreter.
   - The Jupyter notebook.
   - The Binder project.

--
--
Getting
started

xtensor requires a modern C++ compiler supporting C++14. The following C+
compilers are supported:

   - On Windows platforms, Visual C++ 2015 Update 2, or more recent
   - On Unix platforms, gcc 4.9 or a recent version of Clang


Installation

xtensor and xtensor-python are header-only libraries. We provide packages
for the conda package manager.

conda install -c conda-forge xtensor # installs xtensor
conda install -c conda-forge xtensor-python  # installs xtensor and
xtensor-python


Usage
Basic
Usage

Initialize a 2-D array and compute the sum of one of its rows and a 1-D
array.

#include 
#include "xtensor/xarray.hpp"
#include "xtensor/xio.hpp"

xt::xarray arr1
  {{1.0, 2.0, 3.0},
   {2.0, 5.0, 7.0},
   {2.0, 5.0, 7.0}};

xt::xarray arr2
  {5.0, 6.0, 7.0};

xt::xarray res = xt::make_xview(arr1, 1) + arr2;

std::cout << res;

Outputs:

{7, 11, 14}

Initialize a 1-D array and reshape it inplace.

#include 
#include "xtensor/xarray.hpp"
#include "xtensor/xio.hpp"

xt::xarray arr
  {1, 2, 3, 4, 5, 6, 7, 8, 9};

arr.reshape({3, 3});

std::cout << arr;

Outputs:

{{1, 2, 3},
 {4, 5, 6},
 {7, 8, 9}}

Lazy
Broadcasting with xtensor

We can operate on arrays of different shapes of dimensions in an
elementwise fashion. Broadcasting rules of xtensor are similar to those of
numpy  and libdynd .
Broadcasting
rules

In an operation involving two arrays of different dimensions, the array
with the lesser dimensions is broadcast across the leading dimensions of
the other.

For example, if A has shape (2, 3), and B has shape (4, 2, 3), the result
of a broadcasted operation with A and B has shape (4, 2, 3).

   (2, 3) # A
(4, 2, 3) # B
-
(4, 2, 3) # Result

The same rule holds for scalars, which are handled as 0-D expressions. If
matched up dimensions of two input arrays are different, and one of them
has size 1, it is broadcast to match the size of the other. Let's say B has
the shape (4, 2, 1) in the previous example, so the broadcasting happens as
follows:

   (2, 3) # A
(4, 2, 1) # B
-
(4, 2, 3) # Result

Universal
functions, Laziness and Vectorization

With xtensor, if x, y and z are arrays of *broadcastable shapes*, the
return type of an expression such as x + y * sin(z) is not an array. It is
an xexpression object offering th

Re: [Numpy-discussion] NumPy 1.12.0b1 released

[Peter Cock]

I have a related question to Matti's,

Do you have any recommendations for building standard wheels
for 3rd party Python libraries which use both the NumPy Python
and C API?

e.g. Do we need to do anything special given the NumPy C API
itself is versioned? Does it matter compiler chain should we use?

Thanks

Peter

On Thu, Nov 17, 2016 at 11:24 PM, Matti Picus  wrote:
> Congrats to all on the release.Two questions:
>
> Is there a guide to building standard wheels for NumPy?
>
> Assuming I can build standardized PyPy 2.7 wheels for Ubuntu, Win32 and
> OSX64, how can I get them blessed and uploaded to PyPI?
>
> Matti
>
>
> On 17/11/16 07:47, numpy-discussion-requ...@scipy.org wrote:
>>
>> Date: Wed, 16 Nov 2016 22:47:39 -0700
>> From: Charles R Harris
>> To: numpy-discussion, SciPy Users List
>> ,  SciPy Developers
>> List,
>> python-announce-l...@python.org
>> Subject: [Numpy-discussion] NumPy 1.12.0b1 released.
>>
>> Hi All,
>>
>> I'm pleased to annouce the release of NumPy 1.12.0b1. This release
>> supports  Python 2.7 and 3.4 - 3.6 and is the result of 388 pull requests
>> submitted by 133 contributors. It is quite sizeable and rather than put
>> the
>> release notes inline I've attached them as a file and they may also be
>> viewed at Github.
>> Zip files and tarballs may also be found the Github link. Wheels and
>> source
>> archives may be downloaded from PyPI, which is the recommended method.
>>
>> This release is a large collection of fixes, enhancements, and
>> improvements
>> and it is difficult to select just a few as highlights. However, the
>> following enhancements may be of particular interest
>>
>> - Order of operations in ``np.einsum`` now can be optimized for large
>> speed improvements.
>> - New ``signature`` argument to ``np.vectorize`` for vectorizing with
>> core dimensions.
>> - The ``keepdims`` argument was added to many functions.
>> - Support for PyPy 2.7 v5.6.0 has been added. While not complete, this
>> is a milestone for PyPy's C-API compatibility layer.
>>
>> Thanks to all,
>>
>> Chuck
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] NumPy 1.12.0b1 released

[Nathan Goldbaum]

Since the NumPy API is forwards compatible, you should use the oldest
version of NumPy you would like to support to build your wheels with. The
wheels will then work with any future NumPy versions.

On Fri, Nov 18, 2016 at 9:30 AM Peter Cock 
wrote:

> I have a related question to Matti's,
>
> Do you have any recommendations for building standard wheels
> for 3rd party Python libraries which use both the NumPy Python
> and C API?
>
> e.g. Do we need to do anything special given the NumPy C API
> itself is versioned? Does it matter compiler chain should we use?
>
> Thanks
>
> Peter
>
> On Thu, Nov 17, 2016 at 11:24 PM, Matti Picus 
> wrote:
> > Congrats to all on the release.Two questions:
> >
> > Is there a guide to building standard wheels for NumPy?
> >
> > Assuming I can build standardized PyPy 2.7 wheels for Ubuntu, Win32 and
> > OSX64, how can I get them blessed and uploaded to PyPI?
> >
> > Matti
> >
> >
> > On 17/11/16 07:47, numpy-discussion-requ...@scipy.org wrote:
> >>
> >> Date: Wed, 16 Nov 2016 22:47:39 -0700
> >> From: Charles R Harris
> >> To: numpy-discussion, SciPy Users List
> >> ,  SciPy Developers
> >> List,
> >> python-announce-l...@python.org
> >> Subject: [Numpy-discussion] NumPy 1.12.0b1 released.
> >>
> >> Hi All,
> >>
> >> I'm pleased to annouce the release of NumPy 1.12.0b1. This release
> >> supports  Python 2.7 and 3.4 - 3.6 and is the result of 388 pull
> requests
> >> submitted by 133 contributors. It is quite sizeable and rather than put
> >> the
> >> release notes inline I've attached them as a file and they may also be
> >> viewed at Github >.
> >> Zip files and tarballs may also be found the Github link. Wheels and
> >> source
> >> archives may be downloaded from PyPI, which is the recommended method.
> >>
> >> This release is a large collection of fixes, enhancements, and
> >> improvements
> >> and it is difficult to select just a few as highlights. However, the
> >> following enhancements may be of particular interest
> >>
> >> - Order of operations in ``np.einsum`` now can be optimized for
> large
> >> speed improvements.
> >> - New ``signature`` argument to ``np.vectorize`` for vectorizing
> with
> >> core dimensions.
> >> - The ``keepdims`` argument was added to many functions.
> >> - Support for PyPy 2.7 v5.6.0 has been added. While not complete,
> this
> >> is a milestone for PyPy's C-API compatibility layer.
> >>
> >> Thanks to all,
> >>
> >> Chuck
> >
> >
> > ___
> > NumPy-Discussion mailing list
> > NumPy-Discussion@scipy.org
> > https://mail.scipy.org/mailman/listinfo/numpy-discussion
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] NumPy 1.12.0b1 released

[Peter Cock]

Thanks Nathan,

That makes sense (compile using the oldest version of NumPy
we wish to support).

The information on https://github.com/MacPython/numpy-wheels
will probably be very useful too (I've been meaning to try out
appveyor at some point for Windows builds/testing).

Regards,

Peter

On Fri, Nov 18, 2016 at 2:46 PM, Nathan Goldbaum  wrote:
> Since the NumPy API is forwards compatible, you should use the oldest
> version of NumPy you would like to support to build your wheels with. The
> wheels will then work with any future NumPy versions.
>
> On Fri, Nov 18, 2016 at 9:30 AM Peter Cock 
> wrote:
>>
>> I have a related question to Matti's,
>>
>> Do you have any recommendations for building standard wheels
>> for 3rd party Python libraries which use both the NumPy Python
>> and C API?
>>
>> e.g. Do we need to do anything special given the NumPy C API
>> itself is versioned? Does it matter compiler chain should we use?
>>
>> Thanks
>>
>> Peter
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] NumPy 1.12.0b1 released

[Nathaniel Smith]

On Nov 18, 2016 01:14, "Ralf Gommers"  wrote:
>
>
>
> On Fri, Nov 18, 2016 at 9:08 PM, Matthew Brett 
wrote:
>>
>> Hi,
>>
>> On Thu, Nov 17, 2016 at 3:24 PM, Matti Picus 
wrote:
>> > Congrats to all on the release.Two questions:
>> >
>> > Is there a guide to building standard wheels for NumPy?
>>
>> I don't think so - there is a repository that we use to build the
>> wheels, that has the Windows, OSX and manyllinux recipes for the
>> standard CPython build:
>>
>> https://github.com/MacPython/numpy-wheelso
>>
>> If you can work out a way to automate the PyPy builds and tests -
>> especially using the same repo - that would be very useful.
>>
>> > Assuming I can build standardized PyPy 2.7 wheels for Ubuntu, Win32 and
>> > OSX64, how can I get them blessed and uploaded to PyPI?
>>
>> If you can automate the build and tests, I'm guessing there will be no
>> objections - but it's not my call...
>
>
> I'm in favor, assuming that the wheel tags and PyPy backwards
compatibility situation is OK. Can't really find any examples. What I mean
is that for CPython wheels contain tags like "cp27" or "cp35". PyPy wheels
should have tags "pp". Are the PyPy cpyext layer and the
 defined such that a new PyPy release won't break older wheels?

Another thing to think about is that 1.12 on pypy won't pass its test suite
(though it's close), and we're not yet testing new PRs on pypy, so no
guarantees about 1.13 yet. I think on balance these probably aren't reasons
*not* to upload wheels, but it's a funny place where we're talking about
providing "official" builds even though it's not an "officially supported
platform". So we will at least want to be clear about that. And someone
will have to handle the bug reports about the test suite failing :-).

-n
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] PyPy wheels (was: NumPy 1.12.0b1 released)

[Ralf Gommers]

On Sat, Nov 19, 2016 at 5:24 AM, Nathaniel Smith  wrote:

> Another thing to think about is that 1.12 on pypy won't pass its test
> suite (though it's close), and we're not yet testing new PRs on pypy, so no
> guarantees about 1.13 yet. I think on balance these probably aren't reasons
> *not* to upload wheels, but it's a funny place where we're talking about
> providing "official" builds even though it's not an "officially supported
> platform". So we will at least want to be clear about that. And someone
> will have to handle the bug reports about the test suite failing :-).
>

Those are good points. We could run PyPy on TravisCI; the PyPy install and
numpy build aren't difficult anymore.

Handling bug reports is mostly checking if it's PyPy specific, and if so
refer to the PyPy tracker I'd think. It is some work, but given that PyPy
has finally chosen a way to support Numpy that's not a dead end and has
come quite a long way quite quickly, taking on that bit of extra work as
Numpy maintainers is a good time investment imho.

Many bug reports will go straight to PyPy though I expect, because often
that is the obvious place to go. This is what I just got from downloading
the OS X PyPy binary and pip installing numpy master:

Python version 2.7.12 (aff251e54385, Nov 09 2016, 17:25:49)[PyPy 5.6.0 with
GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)]
nose version 1.3.7
.S...ES.RPython
traceback:
  File "pypy_interpreter.c", line 43348, in
BuiltinCodePassThroughArguments1_funcrun_obj
  File "pypy_module_cpyext_4.c", line 16627, in
generic_cpy_call__StdObjSpaceConst_funcPtr_SomeI_17
Fatal RPython error: AssertionError
Abort trap: 6

So guess I'll go find out where that issue tracker is:)

Ralf


On Nov 18, 2016 01:14, "Ralf Gommers"  wrote:
> >
> >
> >
> > On Fri, Nov 18, 2016 at 9:08 PM, Matthew Brett 
> wrote:
> >>
> >> Hi,
> >>
> >> On Thu, Nov 17, 2016 at 3:24 PM, Matti Picus 
> wrote:
> >> > Congrats to all on the release.Two questions:
> >> >
> >> > Is there a guide to building standard wheels for NumPy?
> >>
> >> I don't think so - there is a repository that we use to build the
> >> wheels, that has the Windows, OSX and manyllinux recipes for the
> >> standard CPython build:
> >>
> >> https://github.com/MacPython/numpy-wheelso
> >>
> >> If you can work out a way to automate the PyPy builds and tests -
> >> especially using the same repo - that would be very useful.
> >>
> >> > Assuming I can build standardized PyPy 2.7 wheels for Ubuntu, Win32
> and
> >> > OSX64, how can I get them blessed and uploaded to PyPI?
> >>
> >> If you can automate the build and tests, I'm guessing there will be no
> >> objections - but it's not my call...
> >
> >
> > I'm in favor, assuming that the wheel tags and PyPy backwards
> compatibility situation is OK. Can't really find any examples. What I mean
> is that for CPython wheels contain tags like "cp27" or "cp35". PyPy wheels
> should have tags "pp". Are the PyPy cpyext layer and the
>  defined such that a new PyPy release won't break older wheels?
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] PyPy wheels (was: NumPy 1.12.0b1 released)

[Nathaniel Smith]

On Nov 18, 2016 3:30 PM, "Ralf Gommers"  wrote:
>
>
>
> On Sat, Nov 19, 2016 at 5:24 AM, Nathaniel Smith  wrote:
>>
>> Another thing to think about is that 1.12 on pypy won't pass its test
suite (though it's close), and we're not yet testing new PRs on pypy, so no
guarantees about 1.13 yet. I think on balance these probably aren't reasons
*not* to upload wheels, but it's a funny place where we're talking about
providing "official" builds even though it's not an "officially supported
platform". So we will at least want to be clear about that. And someone
will have to handle the bug reports about the test suite failing :-).
>
>
> Those are good points. We could run PyPy on TravisCI; the PyPy install
and numpy build aren't difficult anymore.

I'm not sure how useful this until the test suite is passing, though, just
because of how travis-ci works.

The main outstanding issue needs some work on the numpy side: basically
UPDATEIFCOPY, as currently conceived, just can't work reliably on pypy,
because it assumes that Python objects get deterministically deallocated as
soon as their reference count drops to zero, and pypy doesn't have
reference counts.

I think fixing this should be straightforward enough, in case anyone wants
to work on it. Every user of UPDATEIFCOPY already has to be aware of the
reference counts and know when the pseudo-"view" array is supposed to be
deallocated. So I think we should define a new UPDATEIFCOPY2 flag, which
acts like the current UPDATEIFCOPY, except that instead of using __del__ to
do the writeback, there's an explicit API call you have to make, like
PyArray_UpdateIfCopy2_Writeback, that checks for the flag and does the
writeback if set. Then we should transition to using this internally, and
probably deprecate UPDATEIFCOPY (though we may never be able to get rid of
it entirely).

-n
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] PyPy wheels (was: NumPy 1.12.0b1 released)

[Ralf Gommers]

On Sat, Nov 19, 2016 at 2:53 PM, Nathaniel Smith  wrote:

> On Nov 18, 2016 3:30 PM, "Ralf Gommers"  wrote:
> >
> >
> >
> > On Sat, Nov 19, 2016 at 5:24 AM, Nathaniel Smith  wrote:
> >>
> >> Another thing to think about is that 1.12 on pypy won't pass its test
> suite (though it's close), and we're not yet testing new PRs on pypy, so no
> guarantees about 1.13 yet. I think on balance these probably aren't reasons
> *not* to upload wheels, but it's a funny place where we're talking about
> providing "official" builds even though it's not an "officially supported
> platform". So we will at least want to be clear about that. And someone
> will have to handle the bug reports about the test suite failing :-).
> >
> >
> > Those are good points. We could run PyPy on TravisCI; the PyPy install
> and numpy build aren't difficult anymore.
>
> I'm not sure how useful this until the test suite is passing, though, just
> because of how travis-ci works.
>
It's not hard to skip the currently failing tests only on a single run in
the build matrix. That would keep TravisCI green and ensure there's no
regressions.

Ralf



> The main outstanding issue needs some work on the numpy side: basically
> UPDATEIFCOPY, as currently conceived, just can't work reliably on pypy,
> because it assumes that Python objects get deterministically deallocated as
> soon as their reference count drops to zero, and pypy doesn't have
> reference counts.
>
> I think fixing this should be straightforward enough, in case anyone wants
> to work on it. Every user of UPDATEIFCOPY already has to be aware of the
> reference counts and know when the pseudo-"view" array is supposed to be
> deallocated. So I think we should define a new UPDATEIFCOPY2 flag, which
> acts like the current UPDATEIFCOPY, except that instead of using __del__ to
> do the writeback, there's an explicit API call you have to make, like
> PyArray_UpdateIfCopy2_Writeback, that checks for the flag and does the
> writeback if set. Then we should transition to using this internally, and
> probably deprecate UPDATEIFCOPY (though we may never be able to get rid of
> it entirely).
>
> -n
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion