[Numpy-discussion] ANN: PyTables 3.3.0

[Tom Kooij]

===
 Announcing PyTables 3.3.0
===

We are happy to announce PyTables 3.3.0.


What's new
==

- Single codebase Python 2 and 3 support (PR #493).
- Internal Blosc version updated to 1.11.1 (closes :issue:`541`)
- Full BitShuffle support for new Blosc versions (>= 1.8).
- It is now possible to remove all rows from a table.
- It is now possible to read reference types by dereferencing them as
  numpy array of objects (closes :issue:`518` and :issue:`519`).
  Thanks to Ehsan Azar
- Fixed Windows 32 and 64-bit builds.

In case you want to know more in detail what has changed in this
version, please refer to: http://www.pytables.org/release_notes.html

You can install it via pip or download a source package with generated
PDF and HTML docs from:
https://github.com/PyTables/PyTables/releases/tag/v3.3.0

For an online version of the manual, visit:
http://www.pytables.org/usersguide/index.html


What it is?
===

PyTables is a library for managing hierarchical datasets and
designed to efficiently cope with extremely large amounts of data with
support for full 64-bit file addressing.  PyTables runs on top of
the HDF5 library and NumPy package for achieving maximum throughput and
convenient use.  PyTables includes OPSI, a new indexing technology,
allowing to perform data lookups in tables exceeding 10 gigarows
(10**10 rows) in less than a tenth of a second.


Resources
=

About PyTables: http://www.pytables.org

About the HDF5 library: http://hdfgroup.org/HDF5/

About NumPy: http://numpy.scipy.org/


Acknowledgments
===

Thanks to many users who provided feature improvements, patches, bug
reports, support and suggestions.  See the ``THANKS`` file in the
distribution package for a (incomplete) list of contributors.  Most
specially, a lot of kudos go to the HDF5 and NumPy makers.
Without them, PyTables simply would not exist.


Share your experience
=

Let us know of any bugs, suggestions, gripes, kudos, etc. you may have.




  **Enjoy data!**

  -- The PyTables Developers

.. Local Variables:
.. mode: rst
.. coding: utf-8
.. fill-column: 72
.. End:
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] New iterator APIs (nditer / MapIter): Overlap detection in NumPy

[Sebastian Berg]

On So, 2016-09-11 at 21:21 +, Pauli Virtanen wrote:
> Hi,
> 
> In the end some further API additions turn out to appear needed:
> 

Very nice :).

> * NPY_ITER_COPY_IF_OVERLAP, NPY_ITER_OVERLAP_NOT_SAME
>   flags for NpyIter_New.
> 
> * New API function PyArray_MapIterArrayCopyIfOverlap,
>   as ufunc.at needs to check overlaps for index arrays 
>   before constructing iterators, and the parsing is done 
>   in multiarray.

I think here Nathaniels point might be right. It could be we can assume
that copying is always fine, there is probably only one or two
downstream projects using this function, plus it seems harder to create
abusing structures that actually do something useful.
It was only exposed for usage in `ufunc.at` if I remember right. I know
theano uses it though, but not sure about anyone else, maybe numba. On
the other hand It is not the worst API clutter in history.

> 
> Continuation here: https://github.com/numpy/numpy/pull/8043
> 
> 
> 
> Wed, 07 Sep 2016 18:02:59 +0200, Sebastian Berg kirjoitti:
> 
> > 
> > Hi all,
> > 
> > Pauli just opened a nice pull request [1] to add overlap detection
> > to
> > the new iterator, this means adding a new iterator flag:
> > 
> > `NPY_ITER_COPY_IF_OVERLAP`
> > 
> > If passed to the iterator (also exposed in python), the iterator
> > will
> > copy the operands such that reading and writing should only occur
> > for
> > identical operands. For now this is implemented by always copying
> > the
> > output/writable operand (this could be improved though, so I would
> > not
> > say its fixed API).
> > 
> > Since adding this flag is new API, please feel free to suggest
> > other
> > names/approaches or even decline the change ;).
> > 
> > 
> > This is basically a first step, which should be easily followed by
> > adding overlap detection to ufuncs, removing traps such as the well
> > (or
> > not so well known) `a += a.T`. Other parts of numpy may follow one
> > by
> > one.
> > 
> > The work is based on his older awesome new memory overlap detection
> > implementation.
> > 
> > If there are no comments, I will probably merge it very soon, so we
> > can
> > look at the follow up things.
> > 
> > - Sebastian
> > 
> > 
> > [1] https://github.com/numpy/numpy/
> pull/8026___
> > 
> > NumPy-Discussion mailing list NumPy-Discussion@scipy.org
> > https://mail.scipy.org/mailman/listinfo/numpy-discussion
> 
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion

signature.asc
Description: This is a digitally signed message part
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] NumPy 1.11.2rc1

[Charles R Harris]

Hi All,

I'm pleased to announce the release of Numpy 1.11.2rc1. This release
supports Python 2.6 - 2.7, and 3.2 - 3.5 and fixes bugs and regressions
found in Numpy 1.11.1.  Wheels for Linux, Windows, and OSX can be found on
PyPI. Sources are available on both PyPI and Sourceforge
.

Thanks to all who were involved in this release.

The following pull requests have been merged. PRs overridden by later
merges and trivial release notes updates have been omitted.


   -  #7736 BUG: Many functions silently drop 'keepdims' kwarg.
   -  #7738 ENH: Add extra kwargs and update doc of many MA methods.
   -  #7778 DOC: Update Numpy 1.11.1 release notes.
   -  #7793 BUG: MaskedArray.count treats negative axes incorrectly.
   -  #7816 BUG: Fix array too big error for wide dtypes.
   -  #7821 BUG: Make sure npy_mul_with_overflow_ detects overflow.
   -  #7824 MAINT: Allocate fewer bytes for empty arrays.
   -  #7847 MAINT,DOC: Fix some imp module uses and update f2py.compile
   docstring.
   -  #7849 MAINT: Fix remaining uses of deprecated Python imp module.
   -  #7851 BLD: Fix ATLAS version detection.
   -  #7896 BUG: Construct ma.array from np.array which contains padding.
   -  #7904 BUG: Fix float16 type not being called due to wrong ordering.
   -  #7917 BUG: Production install of numpy should not require nose.
   -  #7919 BLD: Fixed MKL detection for recent versions of this library.
   -  #7920 BUG: Fix for issue #7835 (ma.median of 1d).
   -  #7932 BUG: Monkey-patch _msvccompile.gen_lib_option like other
   compilers.
   -  #7939 BUG: Check for HAVE_LDOUBLE_DOUBLE_DOUBLE_LE in
   npy_math_complex.
   -  #7953 BUG: Guard against buggy comparisons in generic quicksort.
   -  #7954 BUG: Use keyword arguments to initialize Extension base class.
   -  #7955 BUG: Make sure numpy globals keep identity after reload.
   -  #7972 BUG: MSVCCompiler grows 'lib' & 'include' env strings
   exponentially.
   -  #8005 BLD: Remove __NUMPY_SETUP__ from builtins at end of setup.py.
   -  #8010 MAINT: Remove leftover imp module imports.
   -  #8020 BUG: Fix return of np.ma.count if keepdims is True and axis is
   None.
   -  #8024 BUG: Fix numpy.ma.median.
   -  #8031 BUG: Fix np.ma.median with only one non-masked value.
   -  #8044 BUG: Fix bug in NpyIter buffering with discontinuous arrays.

The following people contributed to this release. The '+' marks first time
contributors.

   - Allan Haldane
   - Bertrand Lefebvre
   - Charles Harris
   - Julian Taylor
   - Loïc Estève
   - Marshall Bockrath-Vandegrift+
   - Michael Seifert+
   - Pauli Virtanen
   - Ralf Gommers
   - Sebastian Berg
   - Shota Kawabuchi+
   - Thomas A Caswell
   - Valentin Valls+
   - Xavier Abellan Ecija+

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] New iterator APIs (nditer / MapIter): Overlap detection in NumPy

[Pauli Virtanen]

Mon, 12 Sep 2016 11:31:07 +0200, Sebastian Berg kirjoitti:
>> * NPY_ITER_COPY_IF_OVERLAP, NPY_ITER_OVERLAP_NOT_SAME
>>   flags for NpyIter_New.
>> 
>> * New API function PyArray_MapIterArrayCopyIfOverlap,
>>   as ufunc.at needs to check overlaps for index arrays before
>>   constructing iterators, and the parsing is done in multiarray.
> 
> I think here Nathaniels point might be right. It could be we can assume
> that copying is always fine, there is probably only one or two
> downstream projects using this function, plus it seems harder to create
> abusing structures that actually do something useful.
> It was only exposed for usage in `ufunc.at` if I remember right. I know
> theano uses it though, but not sure about anyone else, maybe numba. On
> the other hand It is not the worst API clutter in history.

Do you suggest that I break the PyArray_MapIterArray API?

One issue here is that the function doesn't make distinction between read-
only access and read-write access, so copying may give unnecessary 
slowdown. The second thing is that it will result to a bit uglier code, as 
I need to manage the overlap with the second operation in ufunc_at.

For NpyIter, I'd still be wary about copying by default, because it's not 
needed everywhere (the may_share_memory checks are better done earlier), 
and since the semantic change can break things inside Numpy.

Pauli

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] New iterator APIs (nditer / MapIter): Overlap detection in NumPy

[Sebastian Berg]

On Mo, 2016-09-12 at 20:22 +, Pauli Virtanen wrote:
> Mon, 12 Sep 2016 11:31:07 +0200, Sebastian Berg kirjoitti:
> > 
> > > 
> > > * NPY_ITER_COPY_IF_OVERLAP, NPY_ITER_OVERLAP_NOT_SAME
> > >   flags for NpyIter_New.
> > > 
> > > * New API function PyArray_MapIterArrayCopyIfOverlap,
> > >   as ufunc.at needs to check overlaps for index arrays before
> > >   constructing iterators, and the parsing is done in multiarray.
> > I think here Nathaniels point might be right. It could be we can
> > assume
> > that copying is always fine, there is probably only one or two
> > downstream projects using this function, plus it seems harder to
> > create
> > abusing structures that actually do something useful.
> > It was only exposed for usage in `ufunc.at` if I remember right. I
> > know
> > theano uses it though, but not sure about anyone else, maybe numba.
> > On
> > the other hand It is not the worst API clutter in history.
> Do you suggest that I break the PyArray_MapIterArray API?
> 
> One issue here is that the function doesn't make distinction between
> read-
> only access and read-write access, so copying may give unnecessary 
> slowdown. The second thing is that it will result to a bit uglier
> code, as 
> I need to manage the overlap with the second operation in ufunc_at.
> 

Yeah, was only wondering about MapIterArray, because I might get away
with the API break in the case that it works everywhere for our
internal usage. But if its not quite straight forward, there is no
point in thinking about it.

> For NpyIter, I'd still be wary about copying by default, because it's
> not 
> needed everywhere (the may_share_memory checks are better done
> earlier), 
> and since the semantic change can break things inside Numpy.
> 

Yes, I tend to agree here about it. You can always wonder whether its
still the most convenient place to do the checks (at least for a few
places), but from glancing at the code, it still seems elegant to me.
If we are concerned about making the iterator more and more complex,
maybe we can really do something else about it as well.

I am not sure whether I will manage to look at it very closely soon, so
would invite anyone to take a look; this is definitely a highlight!

- Sebastian


>   Pauli
> 
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion

signature.asc
Description: This is a digitally signed message part
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion