Re: [Numpy-discussion] seeking advice on a fast string->array conversion

2010-11-20 Thread william ratcliff
Actually, I do use spec when I have synchotron experiments. But why are your files so large? On Nov 16, 2010 9:20 AM, "Darren Dale" wrote: > I am wrapping up a small package to parse a particular ascii-encoded > file format generated by a program we use heavily here at the lab. (In > the unlikely

[Numpy-discussion] seeking advice on a fast string->array conversion

2010-11-19 Thread Darren Dale
I am wrapping up a small package to parse a particular ascii-encoded file format generated by a program we use heavily here at the lab. (In the unlikely event that you work at a synchrotron, and use Certified Scientific's "spec" program, and are actually interested, the code is currently available

Re: [Numpy-discussion] seeking advice on a fast string->array conversion

2010-11-19 Thread Darren Dale
Apologies, I accidentally hit send... On Tue, Nov 16, 2010 at 9:20 AM, Darren Dale wrote: > I am wrapping up a small package to parse a particular ascii-encoded > file format generated by a program we use heavily here at the lab. (In > the unlikely event that you work at a synchrotron, and use Ce

Re: [Numpy-discussion] seeking advice on a fast string->array conversion

2010-11-19 Thread Pauli Virtanen
Tue, 16 Nov 2010 09:20:29 -0500, Darren Dale wrote: [clip] > module. So I am wondering about the performance of np.fromstring: Fromstring is slow, probably because it must work around locale- dependence of the underlying C parsing functions. Moreover, the Numpy parsing mechanism generates many in

Re: [Numpy-discussion] seeking advice on a fast string->array conversion

2010-11-19 Thread Pauli Virtanen
Tue, 16 Nov 2010 09:41:04 -0500, Darren Dale wrote: [clip] > That loop takes 0.33 seconds to execute, which is a good start. I need > some help converting this example to return an actual numpy array. Could > anyone please offer a suggestion? Easiest way is probably to use ndarray buffers and resi

Re: [Numpy-discussion] seeking advice on a fast string->array conversion

2010-11-19 Thread Darren Dale
On Tue, Nov 16, 2010 at 10:31 AM, Darren Dale wrote: > On Tue, Nov 16, 2010 at 9:55 AM, Pauli Virtanen wrote: >> Tue, 16 Nov 2010 09:41:04 -0500, Darren Dale wrote: >> [clip] >>> That loop takes 0.33 seconds to execute, which is a good start. I need >>> some help converting this example to return

Re: [Numpy-discussion] seeking advice on a fast string->array conversion

2010-11-19 Thread Darren Dale
Sorry, I accidentally hit send long before I was finished writing. But to answer your question, they contain many 2048-element multi-channel analyzer spectra. Darren On Tue, Nov 16, 2010 at 9:26 AM, william ratcliff wrote: > Actually, > I do use spec when I have synchotron experiments.  But why

Re: [Numpy-discussion] seeking advice on a fast string->array conversion

2010-11-16 Thread Christopher Barker
On 11/16/10 10:01 AM, Christopher Barker wrote: OK -- I'll whip up a test similar to yours -- stay tuned! Here's what I've done: import numpy as np from maproomlib.utility import file_scanner def gen_file(): f = file('test.dat', 'w') for i in range(1200): f.write('1 ' * 2048)

Re: [Numpy-discussion] seeking advice on a fast string->array conversion

2010-11-16 Thread Christopher Barker
On 11/16/10 8:57 AM, Darren Dale wrote: > In my case, I am making an assumption about the integrity of the file. That does make things easier, but less universal. I guess this is the whole trade-off about "reusable code". It sure it a lot easier to write code that does the one thing you need tha

Re: [Numpy-discussion] seeking advice on a fast string->array conversion

2010-11-16 Thread Darren Dale
On Tue, Nov 16, 2010 at 11:46 AM, Christopher Barker wrote: > On 11/16/10 7:31 AM, Darren Dale wrote: >> On Tue, Nov 16, 2010 at 9:55 AM, Pauli Virtanen  wrote: >>> Tue, 16 Nov 2010 09:41:04 -0500, Darren Dale wrote: >>> [clip] That loop takes 0.33 seconds to execute, which is a good start. I

Re: [Numpy-discussion] seeking advice on a fast string->array conversion

2010-11-16 Thread Christopher Barker
On 11/16/10 7:31 AM, Darren Dale wrote: > On Tue, Nov 16, 2010 at 9:55 AM, Pauli Virtanen wrote: >> Tue, 16 Nov 2010 09:41:04 -0500, Darren Dale wrote: >> [clip] >>> That loop takes 0.33 seconds to execute, which is a good start. I need >>> some help converting this example to return an actual num

Re: [Numpy-discussion] seeking advice on a fast string->array conversion

2010-11-16 Thread Darren Dale
On Tue, Nov 16, 2010 at 9:55 AM, Pauli Virtanen wrote: > Tue, 16 Nov 2010 09:41:04 -0500, Darren Dale wrote: > [clip] >> That loop takes 0.33 seconds to execute, which is a good start. I need >> some help converting this example to return an actual numpy array. Could >> anyone please offer a sugge