Sebastian Haase gmail.com> writes:
>
> I would expect a 700MB text file translate into less than 200MB of
> data - assuming that you are talking about decimal numbers (maybe
> total of 10 digits each + spaces) and saving as float32 binary.
> So the problem would "only" be the loading in - rathe
On Thu, Jul 8, 2010 at 4:46 PM, Bruce Southey wrote:
> On 07/08/2010 08:52 AM, Wes McKinney wrote:
>> On Thu, Jul 8, 2010 at 9:26 AM, Hannes Bretschneider
>> wrote:
>>
>>> Dear NumPy developers,
>>>
>>> I have to process some big data files with high-frequency
>>> financial data. I am trying to
On 07/08/2010 08:52 AM, Wes McKinney wrote:
> On Thu, Jul 8, 2010 at 9:26 AM, Hannes Bretschneider
> wrote:
>
>> Dear NumPy developers,
>>
>> I have to process some big data files with high-frequency
>> financial data. I am trying to load a delimited text file having
>> ~700 MB with ~ 10 mill
On Thu, Jul 8, 2010 at 9:26 AM, Hannes Bretschneider
wrote:
> Dear NumPy developers,
>
> I have to process some big data files with high-frequency
> financial data. I am trying to load a delimited text file having
> ~700 MB with ~ 10 million lines using numpy.genfromtxt(). The
> machine is a Debia
Dear NumPy developers,
I have to process some big data files with high-frequency
financial data. I am trying to load a delimited text file having
~700 MB with ~ 10 million lines using numpy.genfromtxt(). The
machine is a Debian Lenny server 32bit with 3GB of memory. Since
the file is just 700MB I
On Thu, Mar 4, 2010 at 19:10, kaby wrote:
>
> Hi.
> I am using numpy arrays and when constructing an array I get a "cannot
> allocate memory for thread-local data: ABORT"
> The array i'm constructing is
> zeros((numVars, 2, numVars, 2), dtype=float) Where numVars is at about 2000.
>
> I was expec
Hi.
I am using numpy arrays and when constructing an array I get a "cannot
allocate memory for thread-local data: ABORT"
The array i'm constructing is
zeros((numVars, 2, numVars, 2), dtype=float) Where numVars is at about 2000.
I was expecting the memory usage to be
2000*2000*2*2*8Bytes=128.000.
