Re: [Numpy-discussion] Dynamic array list implementation

[Stephan Hoyer]

We have a type similar to this (a typed list) internally in pandas, although it 
is restricted to a single dimension and far from feature complete -- it only 
has .append and a .to_array() method for converting to a 1d numpy array. Our 
version is written in Cython, and we use it for performance reasons when we 
would otherwise need to create a list of unknown length:



https://github.com/pydata/pandas/blob/v0.17.1/pandas/hashtable.pyx#L99



In my experience, it's several times faster than using a builtin list from 
Cython, which makes sense given that it needs to copy about 1/3 the data (no 
type or reference count for individual elements). Obviously, it uses 1/3 the 
space to store the data, too. We currently don't expose this object externally, 
but it could be an interesting project to adapt this code into a standalone 
project that could be more broadly useful.






Cheers,

Stephan

On Tue, Dec 22, 2015 at 8:20 PM, Chris Barker 
wrote:

> sorry for being so lazy as to not go look at the project pages, but
> This sounds like it could be really useful, and maybe supercise a coupl eof
> half-baked projects of mine. But -- what does "dynamic" mean?
> - can you append to these arrays?
> - can it support "ragged arrrays" -- it looks like it does.
>>
>> >>> L = ArrayList( [[0], [1,2], [3,4,5], [6,7,8,9]] )
>> >>> print(L)
>> [[0], [1 2], [3 4 5], [6 7 8 9]]
>>
>> so this looks like a ragged array -- but what do you get when you do:
> for row in L:
> print row
>> >>> print(L.data)
>> [0 1 2 3 4 5 6 7 8
>>
>> is .data a regular old 1-d numpy array?
 L = ArrayList( np.arange(10), [3,3,4])
>> >>> print(L)
>> [[0 1 2], [3 4 5], [6 7 8 9]]
>> >>> print(L.data)
>> [0 1 2 3 4 5 6 7 8 9]
>>
>> does an ArrayList act like a numpy array in other ways:
> L * 5
> L* some_array
> in which case, how does it do broadcasting???
> Thanks,
> -CHB
 L = ArrayList(["Hello", "world", "!"])
>> >>> print(L[0])
>> 'Hello'
>> >>> L[1] = "brave new world"
>> >>> print(L)
>> ['Hello', 'brave new world', '!']
>>
>>
>>
>> Nicolas
>>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
> -- 
> Christopher Barker, Ph.D.
> Oceanographer
> Emergency Response Division
> NOAA/NOS/OR&R(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
> chris.bar...@noaa.gov___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy-1.11.0.dev0 windows wheels compiled with mingwpy available

[Henry Gomersall]

On 23/12/15 06:08, Ralf Gommers wrote:
> 
> On Tue, Dec 22, 2015 at 9:58 AM, Henry Gomersall  > wrote:
> 
> On 23/10/15 02:14, Robert McGibbon wrote:
> > The original goal was to get MS to pay for this, on the theory that
> > they should be cleaning up their own messes, but after 6 months of
> > back-and-forth we've pretty much given up on that at this point, and
> > I'm in the process of emailing everyone I can think of who might be
> > convinced to donate some money to the cause. Maybe we should have a
> > kickstarter or something, I dunno :-).
> 
> Just noticed this. Yes a kickstarter would be good - I'd be willing to
> contribute something.
> 
> What's the current estimated cost for 3 person months?
> 
> 
> We're expecting to be able to announce some progress on this front
> during the holiday season - please hold on for a little bit.

Great, will do!

Henry (oddly excited about windows build systems)
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Dynamic array list implementation

[Nicolas P. Rougier]

Typed list in numpy would be a nice addition indeed and your cython 
implementation is nice (and small).

In my case I need to ensure a contiguous storage to allow easy upload onto the 
GPU.
But my implementation is quite slow, especially when you add one item at a time:

>>> python benchmark.py
Python list, append 10 items: 0.01161
Array list, append 10 items: 0.46854
Array list, append 10 items at once: 0.05801
Python list, prepend 10 items: 1.96168
Array list, prepend 10 items: 12.83371
Array list, append 10 items at once: 0.06002



I realize I did not answer all Chris' questions:

>>> L = ArrayList( [[0], [1,2], [3,4,5], [6,7,8,9]] )
>>> for item in L: print(item)
[0]
[1 2]
[3 4 5]
[6 7 8 9]

>>> print (type(L.data))

>>> print(L.data.dtype)
int64
>>> print(L.data.shape)
(10,)


I did not implement operations yet, but it would be a matter for transferring 
call to the underlying numpy data array.
>>> L._data *= 2
>>> print(L)
[[0], [4 8], [12 16 20], [24 28 32 36]]



> On 23 Dec 2015, at 09:34, Stephan Hoyer  wrote:
> 
> We have a type similar to this (a typed list) internally in pandas, although 
> it is restricted to a single dimension and far from feature complete -- it 
> only has .append and a .to_array() method for converting to a 1d numpy array. 
> Our version is written in Cython, and we use it for performance reasons when 
> we would otherwise need to create a list of unknown length:
> https://github.com/pydata/pandas/blob/v0.17.1/pandas/hashtable.pyx#L99
> 
> In my experience, it's several times faster than using a builtin list from 
> Cython, which makes sense given that it needs to copy about 1/3 the data (no 
> type or reference count for individual elements). Obviously, it uses 1/3 the 
> space to store the data, too. We currently don't expose this object 
> externally, but it could be an interesting project to adapt this code into a 
> standalone project that could be more broadly useful.
> 
> Cheers,
> Stephan
> 
> 
> 
> On Tue, Dec 22, 2015 at 8:20 PM, Chris Barker  wrote:
> 
> sorry for being so lazy as to not go look at the project pages, but
> 
> This sounds like it could be really useful, and maybe supercise a coupl eof 
> half-baked projects of mine. But -- what does "dynamic" mean?
> 
> - can you append to these arrays?
> - can it support "ragged arrrays" -- it looks like it does.
> 
> >>> L = ArrayList( [[0], [1,2], [3,4,5], [6,7,8,9]] )
> >>> print(L)
> [[0], [1 2], [3 4 5], [6 7 8 9]]
> 
> so this looks like a ragged array -- but what do you get when you do:
> 
> for row in L:
> print row
> 
>  
> >>> print(L.data)
> [0 1 2 3 4 5 6 7 8 
> 
> is .data a regular old 1-d numpy array?
> 
> >>> L = ArrayList( np.arange(10), [3,3,4])
> >>> print(L)
> [[0 1 2], [3 4 5], [6 7 8 9]]
> >>> print(L.data)
> [0 1 2 3 4 5 6 7 8 9]
> 
> 
> does an ArrayList act like a numpy array in other ways:
> 
> L * 5
> 
> L* some_array
> 
> in which case, how does it do broadcasting???
> 
> Thanks,
> 
> -CHB
> 
> >>> L = ArrayList(["Hello", "world", "!"])
> >>> print(L[0])
> 'Hello'
> >>> L[1] = "brave new world"
> >>> print(L)
> ['Hello', 'brave new world', '!']
> 
> 
> 
> Nicolas
> 
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
> 
> 
> 
> 
> -- 
> 
> Christopher Barker, Ph.D.
> Oceanographer
> 
> Emergency Response Division
> NOAA/NOS/OR&R(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
> 
> chris.bar...@noaa.gov
> 
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Dynamic array list implementation

[Sebastian Berg]

On Mi, 2015-12-23 at 00:34 -0800, Stephan Hoyer wrote:
> We have a type similar to this (a typed list) internally in pandas,
> although it is restricted to a single dimension and far from feature
> complete -- it only has .append and a .to_array() method for
> converting to a 1d numpy array. Our version is written in Cython, and
> we use it for performance reasons when we would otherwise need to
> create a list of unknown length:
> https://github.com/pydata/pandas/blob/v0.17.1/pandas/hashtable.pyx#L99
> 

Probably is a bit orthogonal since I guess you want/need cython, but
pythons buildin array.array should get you there pretty much as well. Of
course it requires the C typecode (though that should not be hard to
get) and does not support strings.

- Sebastian

> 
> In my experience, it's several times faster than using a builtin list
> from Cython, which makes sense given that it needs to copy about 1/3
> the data (no type or reference count for individual elements).
> Obviously, it uses 1/3 the space to store the data, too. We currently
> don't expose this object externally, but it could be an interesting
> project to adapt this code into a standalone project that could be
> more broadly useful.
> 
> 
> Cheers,
> Stephan
> 
> 
> 
> 
> On Tue, Dec 22, 2015 at 8:20 PM, Chris Barker 
> wrote:
> 
> 
> sorry for being so lazy as to not go look at the project
> pages, but
> 
> 
> This sounds like it could be really useful, and maybe
> supercise a coupl eof half-baked projects of mine. But -- what
> does "dynamic" mean?
> 
> 
> - can you append to these arrays?
> - can it support "ragged arrrays" -- it looks like it does.
> 
> 
> >>> L = ArrayList( [[0], [1,2], [3,4,5], [6,7,8,9]] )
> >>> print(L)
> [[0], [1 2], [3 4 5], [6 7 8 9]]
> so this looks like a ragged array -- but what do you get when
> you do:
> 
> 
> for row in L:
> print row
> 
> 
>  
> 
> >>> print(L.data)
> [0 1 2 3 4 5 6 7 8 
> is .data a regular old 1-d numpy array?
> 
> 
> >>> L = ArrayList( np.arange(10), [3,3,4])
> >>> print(L)
> [[0 1 2], [3 4 5], [6 7 8 9]]
> >>> print(L.data)
> [0 1 2 3 4 5 6 7 8 9]
> 
> does an ArrayList act like a numpy array in other ways:
> 
> 
> L * 5
> 
> 
> L* some_array
> 
> 
> in which case, how does it do broadcasting???
> 
> 
> Thanks,
> 
> 
> -CHB
> 
> 
> >>> L = ArrayList(["Hello", "world", "!"])
> >>> print(L[0])
> 'Hello'
> >>> L[1] = "brave new world"
> >>> print(L)
> ['Hello', 'brave new world', '!']
> 
> 
> 
> 
> Nicolas
> 
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
> 
> 
> 
> 
> 
> -- 
> 
> Christopher Barker, Ph.D.
> Oceanographer
> 
> Emergency Response Division
> NOAA/NOS/OR&R(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
> 
> chris.bar...@noaa.gov
> 
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion



signature.asc
Description: This is a digitally signed message part
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] packaging of numpy for windows

[jo]

Hi
I would like to package python + numpy and our app for windows using pynsist in 
such a way that the user does not need to compile anything. 

Although pynsist made a package numpy did not start after installation due to 
some missing libraries. Obviously including the numpy directory is not enough. 

Is there an nsi script or filelist for numpy what I should deliver ? Do I need 
msft redistributables? 

tx
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion