[Numpy-discussion] Re: JSON format for multi-dimensional data
On Tue, Feb 27, 2024 at 3:45 PM wrote: > Thanks Ralf, > > This answers my question about the absence of I/O Numpy format. > > There are three other points related to this format proposal: > > - integration of a semantic level above the number / character formats as > for datetime (e.g. units, point / polygon, URI, email, IP, encoding...), > - neutral format (platform independent) for multidimensional data > including multi-variables, axes, indexes and metadata, > - finally the conversion of tabular data into multi-dimensional data > (dimension greater than 2) via a neutral format. > > Do these points interest Numpy or would this rather concern applications > built on a Numpy base? > I think the latter. Cheers, Ralf ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Re: JSON format for multi-dimensional data
aside from the previously mentioned ticket https://github.com/numpy/numpy/issues/12481, I also made a similar proposal, posted in 2021 https://github.com/numpy/numpy/issues/20461 https://mail.python.org/archives/list/numpy-discussion@python.org/message/EVQW2PO64464JEN3RQXSCDP32RQDIQFW/ lightweight JSON annotations for various data structures (trees, tables, graphs), especially ND-arrays, are defined in the JData spec https://github.com/NeuroJSON/jdata/blob/master/JData_specification.md#data-annotation-keywords JSON/binary JSON annotation encoders/decoders have been implemented for Python (https://pypi.org/project/jdata/), MATLAB/Octave (https://github.com/fangq/jsonlab), JavaScript/NodeJS (https://www.npmjs.com/package/jda), as well as C++ (JSON for Modern C++, https://json.nlohmann.me/features/binary_formats/bjdata/) I have been extensively used this annotation in JSON/binary JSON in my neuroimaging data portal, https://neurojson.io/, for example, for 3D data https://neurojson.org/db/fieldtrip(atlas)/FieldTrip--Brainnetome--Brainnetome https://neurojson.org/db/fieldtrip(atlas)/FieldTrip--Brainnetome--Brainnetome#preview=$.tissue for mesh data https://neurojson.org/db/brainmeshlibrary/BrainWeb--Subject04--gm--surf https://neurojson.org/db/brainmeshlibrary/BrainWeb--Subject04--gm--surf#preview=$ the ND array supports binary data with loss-less compression. I've also implemented in a renewed thread posted in 2022, I also tested the blosc2 (https://www.blosc.org/) compression codecs and got excellent read/write speed https://mail.python.org/archives/list/numpy-discussion@python.org/thread/JIT4AIVEYJLSSHTSA7GOUBIVQLT3WPRU/#U33R5GL34OTL7EZX2VRQGOO4KUWED56M https://mail.python.org/archives/list/numpy-discussion@python.org/message/TUO7CKTQL2GGH2MIWSBH6YCO3GX4AV2O/ the blosc2 compression codes are supported in my python and matlab/C parsers. Qianqian ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Re: JSON format for multi-dimensional data
Bravo for this very comprehensive work which covers all technical/scientific data structures well ! I think we share the same goal of improving interoperability through the use of neutral formats. However, I see some differences: - I focus efforts more particularly on increasing the semantic level with a generalization and an extension of the notion of type, - I also try not to call into question what works well so that the impacts are minimal. For example, we can have a mixed JSON structure integrating a part of data in NTV format and another part outside it. Likewise for tabular data, we can go from a "format" type to a "semantic" type without significant impact for a tool like Pandas. More particularly, concerning multidimensional data, it seems to me that it is necessary not to limit oneself to the ndarray structure but that it is also necessary to integrate associated structures such as those defined in Xarray. ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Re: JSON format for multi-dimensional data
I sure like the idea of this. I agree that this should be external to numpy. At least until it becomes a standard in a sense that json itself is. And that n-dimensional array should ideally be extended to indexed structures with named dimensions and attributes. To accommodate a superset of: xarray & scipp. Regards, DG > On 28 Feb 2024, at 08:34, fangqq--- via NumPy-Discussion > wrote: > > aside from the previously mentioned ticket > https://github.com/numpy/numpy/issues/12481, I also made a similar proposal, > posted in 2021 > > https://github.com/numpy/numpy/issues/20461 > https://mail.python.org/archives/list/numpy-discussion@python.org/message/EVQW2PO64464JEN3RQXSCDP32RQDIQFW/ > > > lightweight JSON annotations for various data structures (trees, tables, > graphs), especially ND-arrays, are defined in the JData spec > > https://github.com/NeuroJSON/jdata/blob/master/JData_specification.md#data-annotation-keywords > > > JSON/binary JSON annotation encoders/decoders have been implemented for > Python (https://pypi.org/project/jdata/), MATLAB/Octave > (https://github.com/fangq/jsonlab), JavaScript/NodeJS > (https://www.npmjs.com/package/jda), as well as C++ (JSON for Modern C++, > https://json.nlohmann.me/features/binary_formats/bjdata/) > > > I have been extensively used this annotation in JSON/binary JSON in my > neuroimaging data portal, https://neurojson.io/, for example, for 3D data > > https://neurojson.org/db/fieldtrip(atlas)/FieldTrip--Brainnetome--Brainnetome > https://neurojson.org/db/fieldtrip(atlas)/FieldTrip--Brainnetome--Brainnetome#preview=$.tissue > > for mesh data > > https://neurojson.org/db/brainmeshlibrary/BrainWeb--Subject04--gm--surf > https://neurojson.org/db/brainmeshlibrary/BrainWeb--Subject04--gm--surf#preview=$ > > the ND array supports binary data with loss-less compression. I've also > implemented > > > in a renewed thread posted in 2022, I also tested the blosc2 > (https://www.blosc.org/) compression codecs and got excellent read/write speed > > https://mail.python.org/archives/list/numpy-discussion@python.org/thread/JIT4AIVEYJLSSHTSA7GOUBIVQLT3WPRU/#U33R5GL34OTL7EZX2VRQGOO4KUWED56M > https://mail.python.org/archives/list/numpy-discussion@python.org/message/TUO7CKTQL2GGH2MIWSBH6YCO3GX4AV2O/ > > the blosc2 compression codes are supported in my python and matlab/C parsers. > > > Qianqian > ___ > NumPy-Discussion mailing list -- numpy-discussion@python.org > To unsubscribe send an email to numpy-discussion-le...@python.org > https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ > Member address: dom.grigo...@gmail.com ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
