On Tue, Aug 23, 2022 at 8:47 PM <[email protected]> wrote: > I want to calc multiple ndarrays at once and lack memory, so want to write > in chunks (here sized to GPU batch capacity). It seems there should be an > interface to write the header, then write a number of elements cyclically, > then add any closing rubric and close the file. > > Is it as simple as lib.format.write_array_header_2_0(fp, d) > then writing multiple shape(N,) arrays of float by > fp.write(item.tobytes())? >
`item.tofile(fp)` is more efficient, but yes, that's the basic scheme. There is no footer after the data. The alternative is to use `np.lib.format.open_memmap(filename, mode='w+', dtype=dtype, shape=shape)`, then assign slices sequentially to the returned memory-mapped array. A memory-mapped array is usually going to be friendlier to whatever memory limits you are running into than a nominally "in-memory" array. -- Robert Kern
_______________________________________________ NumPy-Discussion mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: [email protected]
