[Numpy-discussion] ANN: pvlib-0.9.2 released

2022-09-01 Thread Dr. Mark Alexander Mikofski PhD 
Dear Pythonistas and solar power enthusiasts,

The maintainers are happy to announce a new release of pvlib python:
software for simulating performance of photovoltaic solar energy systems.

*See what's new for v0.9.2:*
** *https://pvlib-python.readthedocs.io/en/stable/whatsnew.html

*Releases are available from PyPI and the conda-forge channel:*
* https://pypi.org/project/pvlib/
* https://anaconda.org/conda-forge/pvlib
* https://anaconda.org/conda-forge/pvlib-python

*Note: you can now install from conda-forge using either "pvlib" or
"pvlib-python"*

*Read the Documentation:*
* https://pvlib-python.readthedocs.io/en/stable/index.html

*Report issues & contribute:*
* https://github.com/pvlib/pvlib-python

*Highlights:*
* Albedo can be entered as a column in weather DataFrame when running a
model chain. This can be used to enter a time series instead of just
monthly values.
* A new function calc_surface_orientation() returns tracker surface tilt
and azimuth when given tracker rotation and axis tilt.
* There's a new example in the gallery to calculate backside irradiance
using pvfactors.
* Dropped support for Python-3.6 and increased required pandas to v0.25.0
* A new build system that conforms to PEP 517 & PEP518
* There were several new contributors.

*Thank you for using pvlib python!*

-- 
Mark Mikofski, PhD (2005)
*Fiat Lux*
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Exporting numpy arrays to binary JSON (BJData) for better portability

2022-09-01 Thread Francesc Alted 
Hi,

On Thu, Sep 1, 2022 at 6:18 AM Qianqian Fang  wrote:

> On 8/30/22 06:29, Francesc Alted wrote:
>
>
> Not exactly.  What we've done is to encode the header and the trailer
> (i.e. where the metadata is) of the frame with msgpack.  The chunks
> section
> 
>  is where the actual data is; this section does not follow a msgpack
> structure as such, but it is rather a sequence of data chunks and an index
> (for quickly locating the chunks).  You can easily access the header or
> trailer sections reading from the start or the end of the frame.  This way
> you don't need to update the indexes of chunks in msgpack, which can be
> expensive during data updates.
>
> This indeed prevents data to be dumped by using typical msgpack tools, but
> our sense is that users should care mostly about metainfo, and let the
> libraries to deal with the actual data in the most efficient way.
>
>
> thanks for your detailed reply. I spent the past few days reading the
> links/documentations, as well as experimenting the blosc2 meta-compressors,
> I was quite impressed by the performance of blosc2. I was also happy to see
> great alignments behind the drives for Caterva those of NeuroJSON.
>
> I have a few quick updates
>
> 1. I added blosc2 as a codec in my jdata module, as an alternative
> compressor to zlib/lzma/lz4
>
>
> https://github.com/NeuroJSON/pyjdata/commit/ce25fa53ce73bf4cbe2cff9799b5a616e2cd75cb
>

Looks good!  Although if you want to support arrays larger than 2 GB, you'd
better use a frame (as I was doing in my example; see also
https://github.com/Blosc/python-blosc2/blob/082db1d2d2ec9afac653903775e2dccac97e2bc9/examples/schunk.py).
Also, the frame is the one using the msgpack for storing the metainfo.


> 2. as I mentioned, jdata/bjdata were not optimized for speed, they contain
> many inefficient handling of numpy arrays (as I discovered); after some
> profiling, I was able to remove most of those, the run-time is now nearly
> entirely spent in compression/decompression (see attached profiler outputs
> for the `zlib` compressor benchmark)
>

Looks good.


> 3. the new jdata that supports blosc2, v0.5.0, has been tagged and
> uploaded (https://pypi.org/project/jdata)
>

That's great.  Although as said, switching from a single chunk into a frame
would allow you to store data > 2 GB.  Whether or not this a goal for you,
I don't know.


> 4. I wrote a script and compared the run times of various codecs (using
> BJData and JSON as containers) , the code can be found here
>
> https://github.com/NeuroJSON/pyjdata/blob/master/test/benchcodecs.py
>
> the save/load times tested on a Ryzen 9 3950X/Ubuntu 18.04 box (at various
> threads) are listed below (similar to your posted before)
>
>
> *- Testing npy/npz*
>   'npy','save' 0.2914195 'load' 0.1963226  'size'  80128
>   'npz','save' 2.8617918 'load' 1.9550347  'size'  813846
>
> *- Testing text-based JSON files (.jdt)** (nthread=8)...*
>   'zlib',   'save' 2.5132861 'load' 1.7221164  'size'  1084942
>   'lzma',   'save' 9.5481696 'load' 0.3865211  'size'  150738
>   'lz4','save' 0.3467197 'load' 0.5019965  'size'  4495297
>   'blosc2blosclz'save' 0.0165646 'load' 0.1143934  'size'  1092747
>   'blosc2lz4',  'save' 0.0175058 'load' 0.1015181  'size'  1090159
>   'blosc2lz4hc','save' 0.2102167 'load' 0.1053235  'size'  4315421
>   'blosc2zlib', 'save' 0.1002635 'load' 0.1188845  'size'  1270252
>   'blosc2zstd', 'save' 0.0463817 'load' 0.1017909  'size'  253176
>
> *- Testing binary JSON (BJData) files (.jdb) (nthread=8)...*
>   'zlib',   'save' 2.4401443 'load' 1.6316463  'size'  813721
>   'lzma',   'save' 9.3782029 'load' 0.3728334  'size'  113067
>   'lz4','save' 0.3389360 'load' 0.5017435  'size'  3371487
>   'blosc2blosclz'save' 0.0173912 'load' 0.1042985  'size'  819576
>   'blosc2lz4',  'save' 0.0133688 'load' 0.1030941  'size'  817635
>   'blosc2lz4hc','save' 0.1968047 'load' 0.0950071  'size'  3236580
>   'blosc2zlib', 'save' 0.1023218 'load' 0.1083922  'size'  952705
>   'blosc2zstd', 'save' 0.0468430 'load' 0.1019175  'size'  189897
>
> *- Testing binary JSON (BJData) files (.jdb) **(nthread=1)...*
>   'blosc2blosclz'save' 0.0883078 'load' 0.2432985  'size'  819576
>   'blosc2lz4',  'save' 0.0867996 'load' 0.2394990  'size'  817635
>   'blosc2lz4hc','save' 2.4794559 'load' 0.2498981  'size'  3236580
>   'blosc2zlib', 'save' 0.7477457 'load' 0.4873921  'size'  952705
>   'blosc2zstd', 'save' 0.3435547 'load' 0.3754863  'size'  189897
>
> *- Testing binary JSON (BJData) files (.jdb) **(nthread=32)...*
>   'blosc2blosclz'save' 0.0197186 'load' 0.1410989  'size'  819576
>   'blosc2lz4',  'save' 0.0168068 'load' 0.1414074  'size'  817635
>   'blosc2lz4hc','save' 0.0790011 'load' 0.0935394  'size'  3236580
>   'blosc2zlib', 'save' 0.0608818 'load' 0.0985531  'size'  952705
>   'blosc2zstd', 'save' 0.0370790 'load' 0.0945577  'size'  189897