Re: [Numpy-discussion] [ANN] carray: an in-memory compressed data container

2010-08-21 Thread Francesc Alted
Hey Sebastian, 2010/8/21, Sebastian Haase : > Hi Francesc, > > another exciting project ... congratulations ! Thanks! > Am I correct in thinking that memmapping a carray would also be a > great speed advantage over memmapped ndarrays ? Let's say I have a > 2Gbyte ndarray memmaped over a NFS net

Re: [Numpy-discussion] [ANN] carray: an in-memory compressed data container

2010-08-21 Thread Sebastian Haase
Hi Francesc, another exciting project ... congratulations ! Am I correct in thinking that memmapping a carray would also be a great speed advantage over memmapped ndarrays ? Let's say I have a 2Gbyte ndarray memmaped over a NFS network connection, should the speed increase simply scale with the c

Re: [Numpy-discussion] [ANN] carray: an in-memory compressed data container

2010-08-20 Thread Francesc Alted
2010/8/20, Zbyszek Szmek : > OK, I've got a case where carray really shines :| > > zbys...@escher:~/python/numpy/carray-0.1.dev$ PYTHONPATH=. python > bench/concat.py numpy 80 1000 4 1 > problem size: (80) x 1000 = 10^8.90309 > time for concat: 4.806s > size of the final container: 6103.516

Re: [Numpy-discussion] [ANN] carray: an in-memory compressed data container

2010-08-20 Thread Zbyszek Szmek
OK, I've got a case where carray really shines :| zbys...@escher:~/python/numpy/carray-0.1.dev$ PYTHONPATH=. python bench/concat.py numpy 80 1000 4 1 problem size: (80) x 1000 = 10^8.90309 time for concat: 4.806s size of the final container: 6103.516 MB zbys...@escher:~/python/numpy/carra

[Numpy-discussion] [ANN] carray: an in-memory compressed data container

2010-08-20 Thread Francesc Alted
== Announcing carray 0.1 == What it is == carray is a container for numerical data that can be compressed in-memory. The compresion process is carried out internally by Blosc, a high-performance compressor that is optimized for binary data. Havin