Re: [Numpy-discussion] using loadtxt to load a text file in to a numpy array

2014-01-22 Thread Chris Barker - NOAA Federal
On Jan 22, 2014, at 1:13 PM, Oscar Benjamin wrote: > > It's not safe to stop removing the null bytes. This is how numpy determines > the length of the strings in a dtype='S' array. The strings are not > "fixed-width" but rather have a maximum width. Exactly--but folks have told us on this list t

Re: [Numpy-discussion] (no subject)

2014-01-22 Thread Ralf Gommers
On Tue, Jan 21, 2014 at 5:46 PM, Charles R Harris wrote: > > > > On Tue, Jan 21, 2014 at 9:26 AM, jennifer stone > wrote: > >> >> >What are your interests and experience? If you use numpy, are there >>> things >>> >you would like to fix, or enhancements you would like to see? >>> >>> Chuck >>> >

Re: [Numpy-discussion] using loadtxt to load a text file in to a numpy array

2014-01-22 Thread Oscar Benjamin
On Wed, Jan 22, 2014 at 12:07:28PM -0800, Chris Barker wrote: > On Wed, Jan 22, 2014 at 2:46 AM, Oscar Benjamin > wrote: > > > BTW, as much as the fixed-width 'S' dtype doesn't really work for str in > > Python 3 it's also a poor fit for bytes since it strips trailing nulls: > > > > >>> a = np.ar

Re: [Numpy-discussion] using loadtxt to load a text file in to a numpy array

2014-01-22 Thread Chris Barker
On Wed, Jan 22, 2014 at 2:46 AM, Oscar Benjamin wrote: > BTW, as much as the fixed-width 'S' dtype doesn't really work for str in > Python 3 it's also a poor fit for bytes since it strips trailing nulls: > > >>> a = np.array(['a\0s\0', 'qwert'], dtype='S') > >>> a > array([b'a\x00s', b'qwert'], >

Re: [Numpy-discussion] using loadtxt to load a text file in to a numpy array

2014-01-22 Thread Andrew Collette
Hi Oscar, > Is it fair to say that people should really be using vlen utf-8 strings for > text? Is it problematic because of the need to interface with non-Python > libraries using the same hdf5 file? The general recommendation has been to use fixed-width strings for exactly that reason; FORTRAN

Re: [Numpy-discussion] accumulation operation

2014-01-22 Thread Julian Taylor
On 22.01.2014 18:23, Ralf Juengling wrote: > Executing the following code, > > > import numpy as np > a = np.zeros((3,)) > w = np.array([0, 1, 0, 1, 2]) > v = np.array([10.0, 1, 10.0, 2, 9]) > a[w] += v > > > > I was expecting ‘a’ to be array([20., 3., 9.]. Inst

Re: [Numpy-discussion] accumulation operation

2014-01-22 Thread Sebastian Berg
On Wed, 2014-01-22 at 17:23 +, Ralf Juengling wrote: > Executing the following code, > > > > >>> import numpy as np > > >>> a = np.zeros((3,)) > > >>> w = np.array([0, 1, 0, 1, 2]) > > >>> v = np.array([10.0, 1, 10.0, 2, 9]) > > >>> a[w] += v > > > > I was expecting ‘a’ to be array(

[Numpy-discussion] accumulation operation

2014-01-22 Thread Ralf Juengling
Executing the following code, >>> import numpy as np >>> a = np.zeros((3,)) >>> w = np.array([0, 1, 0, 1, 2]) >>> v = np.array([10.0, 1, 10.0, 2, 9]) >>> a[w] += v I was expecting 'a' to be array([20., 3., 9.]. Instead I get >>> a array([ 10., 2., 9.]) This with numpy version 1.6.1. Is ther

Re: [Numpy-discussion] fromiter cannot create array of object - was: Creating an ndarray from an iterable, over sequences

2014-01-22 Thread Sebastian Berg
On Wed, 2014-01-22 at 07:58 +0100, Dr. Leo wrote: > Hi, > > thanks. Both recarray and itertools.chain work just fine in the example > case. > > However, the real purpose of this is to read strings from a large xml > file into a pandas DataFrame. But fromiter cannot create arrays of dtype > 'objec

Re: [Numpy-discussion] using loadtxt to load a text file in to a numpy array

2014-01-22 Thread Oscar Benjamin
On Tue, Jan 21, 2014 at 06:54:33PM -0700, Andrew Collette wrote: > Hi Chris, > > > it looks from here: > > http://www.hdfgroup.org/HDF5/doc/ADGuide/WhatsNew180.html > > > > that HDF uses utf-8 for unicode strings -- so you _could_ roundtrip with a > > lot of calls to encode/decode -- which could b