[Numpy-discussion] Proposal - Making ndarray object JSON serializable via standardized JData annotations

Dear numpy developers, I would like to share a proposal on making ndarray JSON serializable by default, as detailed in this github issue: https://github.com/numpy/numpy/issues/20461 briefly, my group and collaborators are working on a new NIH (National Institute of Health) funded initiative

[Numpy-discussion] Re: Proposal - Making ndarray object JSON serializable via standardized JData annotations

izable" in json.dump/dumps without needing additional modules; more importantly, it simplifies users experience in exchanging complex arrays (complex valued, sparse, special shapes) with other programming environments. Qianqian Best, Stephan On Thu, Nov 25, 2021 at 2:35 PM Qianqi

[Numpy-discussion] Re: Proposal - Making ndarray object JSON serializable via standardized JData annotations

On 11/25/21 23:00, Robert Kern wrote: We could also provide a JSONEncoder/JSONDecoder pair, too, but as I mention in one of the Github issues you link to, there are a number of different expectations that people could have for what the JSON representation of an array is. Some will want to use t

[Numpy-discussion] Re: An extension of the .npy file format

I am curious what you and other developers think about adopting JSON/binary JSON as a similarly simple, reverse-engineering-able but universally parsable array exchange format instead of designing another numpy-specific binary format. I am interested in this topic (as well as thoughts among nu

[Numpy-discussion] Re: An extension of the .npy file format

On 8/25/22 12:25, Robert Kern wrote: No one is really proposing another format, just a minor tweak to the existing NPY format. agreed. I was just following the previous comment on alternative formats (such as hdf5) and pros/cons of npy. I don't quite know what this means. My installed versi

[Numpy-discussion] Exporting numpy arrays to binary JSON (BJData) for better portability

To avoid derailing the other thread on extending .npy files, I am going to start a new thread on alternative array storage file formats using binary JSON - in case there is such a need a

[Numpy-discussion] Re: Exporting numpy arrays to binary JSON (BJData) for better portability

On 8/25/22 18:33, Neal Becker wrote: the loading time (from an nvme drive, Ubuntu 18.04, python 3.6.9, numpy 1.19.5) for each file is listed below: |0.179s  eye1e4.npy (mmap_mode=None)|| ||0.001s  eye1e4.npy (mmap_mode=r)|| ||0.718s  eye1e4_bjd_raw_ndsyntax.jdb|| ||1.4

[Numpy-discussion] Re: Exporting numpy arrays to binary JSON (BJData) for better portability

I/O overhead is really low compared with the cost of computation (all the decompression takes place in CPU cache, not memory), so in the end, this is not that surprising. Cheers! On Fri, Aug 26, 2022 at 4:26 AM Qianqian Fang wrote: On 8/25/22 18:33, Neal Becker wrote: the loa

[Numpy-discussion] Re: Exporting numpy arrays to binary JSON (BJData) for better portability

On 8/30/22 06:29, Francesc Alted wrote: Not exactly.  What we've done is to encode the header and the trailer (i.e. where the metadata is) of the frame with msgpack.  Thechunks section is where the actual data is; t