[Numpy-discussion] ANN: NumExpr 2.8.6 Released
Hi everyone, NumExpr 2.8.6 is a release to deal with issues related to downstream `pandas` where the sanitization blacklist was hitting private variables used in their evaluate. In addition the sanitization was hitting on scientific notation. For those who do not wish to have sanitization on by default, it can be changed by setting an environment variable, `NUMEXPR_SANITIZE=0`. If you use `pandas` in your packages it is advisable you pin `numexpr >= 2.8.6` in your requirements. Project documentation is available at: http://numexpr.readthedocs.io/ Changes from 2.8.5 to 2.8.6 --- * The sanitization can be turned off by default by setting an environment variable, `set NUMEXPR_SANITIZE=0` * Improved behavior of the blacklist to avoid triggering on private variables and scientific notation numbers. What's Numexpr? --- Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python. It has multi-threaded capabilities, as well as support for Intel's MKL (Math Kernel Library), which allows an extremely fast evaluation of transcendental functions (sin, cos, tan, exp, log...) while squeezing the last drop of performance out of your multi-core processors. Look here for a some benchmarks of numexpr using MKL: https://github.com/pydata/numexpr/wiki/NumexprMKL Its only dependency is NumPy (MKL is optional), so it works well as an easy-to-deploy, easy-to-use, computational engine for projects that don't want to adopt other solutions requiring more heavy dependencies. Where I can find Numexpr? - The project is hosted at GitHub in: https://github.com/pydata/numexpr You can get the packages from PyPI as well (but not for RC releases): http://pypi.python.org/pypi/numexpr Documentation is hosted at: http://numexpr.readthedocs.io/en/latest/ Share your experience - Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy data! -- Robert McLeod robbmcl...@gmail.com robert.mcl...@hitachi-hightech.com ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Build NumPy with Debugging Symbols with Meson
Hi everyone,
Is there a gist or similar guide anywhere on the steps required to build
NumPy with debugging symbols on the Windows platform using the new Meson
build system? It seems a bit difficult because NumPy seems to expect a
`conda`-like file structure whereas if you are cloning `cpython` directly
it is different. In particular `cpython` puts all the Windows files under
the `PC` and `PCBuild` subdirectories. Also meson doesn't seem to have a
means to call `python_d.exe` directly which may be causing problems. I've
ended up building both `python.exe` and `python_d.exe` but run into some
problem finding Python in `meson.build`.
This describes my efforts to date:
# Build CPython with Debugging Symbols on Windows
```shell
git clone https://github.com/python/cpython.git
cd cpython
git switch 3.11
PCbuild\get_externals.bat
# Build the debug version `python_d.exe`
PCbuild\build.bat -e -d
# Meson calls `python.exe` and doesn't seem to have an argument to change
this.
# We could either build it, or create a symlink
PCbuild\build.bat
# ln -s PCbuild/amd64/python_d.exe PCbuild/amd64/python.exe
```
The built Python will then be located in
`/cpython/PCBuild/amd64/python_d.exe`.
```batch
set PREFIX=C:/Users/Robert/dev/cpython
set PATH=%PREFIX%;%PREFIX%/PCBuild/amd64;%PREFIX%/Scripts;%PATH%
```
Next we have to install pip:
https://docs.python.org/3/library/ensurepip.html,
meson, and cython.
```shell
python_d -m ensurepip
python_d -mpip install meson meson-python ninja cython
```
NOTE: produces `pip3.exe`, not `pip`.
# Build NumPy with debug Python
```shell
git clone https://github.com/numpy/numpy.git
cd numpy
git switch maintenance/1.26.x
git submodule update --init
```
https://mesonbuild.com/meson-python/how-to-guides/debug-builds.html
Next try and build with meson/ninja:
```shell
mkdir build-dbg
cd build-dbg
meson .. setup --buildtype=debug
--includedir=C:/Users/Robert/dev/cpython/PC
--libdir=C:/Users/Robert/dev/cpython/PCbuild/amd64
ninja
ninja install
```
`meson .. setup <...>` fails and complains that,
'''
Run-time dependency python found: NO (tried sysconfig)
..\meson.build:41:12: ERROR: Python dependency not found
'''
which corresponds to:
```meson.build
py = import('python').find_installation(pure: false)
py_dep = py.dependency()
```
I tried also editing `pyproject.toml` to add the section:
```toml
[tool.meson-python.args]
setup = ['-Dbuildtype=debug',
'-Dincludedir=C:/Users/Robert/dev/cpython/PC',
'-Dlibdir=C:/Users/Robert/dev/cpython/PCbuild/amd64']
```
and then build NumPy with pip using the debug python build:
```shell
python_d -mpip install . --no-build-isolation -Cbuild-dir=build-dbg
```
But it fails in the same fashion. Searching for issues I find this one but
it's likely in this case something related to the debug Python build is the
problem.
https://github.com/mesonbuild/meson/issues/12547
Meson installed version is 1.4.0. Any advice would be appreciated. I'm
happy to write a gist if I can get it working.
--
Robert McLeod
robbmcl...@gmail.com
robert.mcl...@hitachi-hightech.com
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com
Re: [Numpy-discussion] record data previous to Numpy use
While I'm going to bet that the fastest way to build a ndarray from ascii is with a 'io.ByteIO` stream, NumPy does have a function to load from text, `numpy.loadtxt` that works well enough for most purposes. https://docs.scipy.org/doc/numpy/reference/generated/numpy.loadtxt.html It's hard to tell from the original post if the ascii is being continuously generated or not. If it's being produced in an on-going fashion then a stream object is definitely the way to go, as the array chunks can be produced by `numpy.frombuffer()`. https://docs.python.org/3/library/io.html https://docs.scipy.org/doc/numpy/reference/generated/numpy.frombuffer.html Robert On Wed, Jul 5, 2017 at 3:21 PM, Robert Kern wrote: > On Wed, Jul 5, 2017 at 5:41 AM, wrote: > > > > Dear all > > > > I’m sorry if my question is too basic (not fully in relation to Numpy – > while it is to build matrices and to work with Numpy afterward), but I’m > spending a lot of time and effort to find a way to record data from an asci > while, and reassign it into a matrix/array … with unsuccessfully! > > > > The only way I found is to use ‘append()’ instruction involving dynamic > memory allocation. :-( > > Are you talking about appending to Python list objects? Or the np.append() > function on numpy arrays? > > In my experience, it is usually fine to build a list with the `.append()` > method while reading the file of unknown size and then converting it to an > array afterwards, even for dozens of millions of lines. The list object is > quite smart about reallocating memory so it is not that expensive. You > should generally avoid the np.append() function, though; it is not smart. > > > From my current experience under Scilab (a like Matlab scientific > solver), it is well know: > > > > Step 1 : matrix initialization like ‘np.zeros(n,n)’ > > Step 2 : record the data > > and write it in the matrix (step 3) > > > > I’m obviously influenced by my current experience, but I’m interested in > moving to Python and its packages > > > > For huge asci files (involving dozens of millions of lines), my strategy > is to work by ‘blocks’ as : > > > > Find the line index of the beginning and the end of one block (this > implies that the file is read ounce) > > Read the block > > (process repeated on the different other blocks) > > Are the blocks intrinsic parts of the file? Or are you just trying to > break up the file into fixed-size chunks? > > > I tried different codes such as bellow, but each time Python is telling > me I cannot mix iteration and record method > > > > # > > > > position = []; j=0 > > with open(PATH + file_name, "r") as rough_ data: > > for line in rough_ data: > > if my_criteria in line: > > position.append(j) ## huge blocs but limited in > number > > j=j+1 > > > > i = 0 > > blockdata = np.zeros( (size_block), dtype=np.float) > > with open(PATH + file_name, "r") as f: > > for line in itertools.islice(f,1,size_block): > > blockdata [i]=float(f.readline() ) > > For what it's worth, this is the line that is causing the error that you > describe. When you iterate over the file with the `for line in > itertools.islice(f, ...):` loop, you already have the line text. You don't > (and can't) call `f.readline()` to get it again. It would mess up the > iteration if you did and cause you to skip lines. > > By the way, it is useful to help us help you if you copy-paste the exact > code that you are running as well as the full traceback instead of > paraphrasing the error message. > > -- > Robert Kern > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > > -- Robert McLeod, Ph.D. robert.mcl...@unibas.ch robert.mcl...@bsse.ethz.ch robbmcl...@gmail.com ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Only integer scalar arrays can be converted to a scalar index
On Fri, Sep 15, 2017 at 2:37 PM, Elliot Hallmark wrote: > Nope. Numpy only works on in memory arrays. You can determine your own > chunking strategy using hdf5, or something like dask can figure that > strategy out for you. With numpy you might worry about not accidentally > making duplicates or intermediate arrays, but that's the extent of memory > optimization you can do in numpy itself. > NumPy does have it's own memory map variant on ndarray: https://docs.scipy.org/doc/numpy/reference/generated/numpy.memmap.html -- Robert McLeod, Ph.D. robbmcl...@gmail.com robbmcl...@protonmail.com ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposal of timeline for dropping Python 2.7 support
On Wed, Nov 8, 2017 at 2:50 PM, Matthew Brett wrote: > Hi, > > On Wed, Nov 8, 2017 at 7:08 PM, Julian Taylor > wrote: > > On 06.11.2017 11:10, Ralf Gommers wrote: > >> > >> > >> On Mon, Nov 6, 2017 at 7:25 AM, Charles R Harris > >> mailto:charlesr.har...@gmail.com>> wrote: > >> > >> Hi All, > >> > >> Thought I'd toss this out there. I'm tending towards better sooner > >> than later in dropping Python 2.7 support as we are starting to run > >> up against places where we would like to use Python 3 features. That > >> is particularly true on Windows where the 2.7 compiler is really old > >> and lacks C99 compatibility. > >> > >> > >> This is probably the most pressing reason to drop 2.7 support. We seem > >> to be expending a lot of effort lately on this stuff. I was previously > >> advocating being more conservative than the timeline you now propose, > >> but this is the pain point that I think gets me over the line. > > > > > > Would dropping python2 support for windows earlier than the other > > platforms a reasonable approach? > > I am not a big fan of to dropping python2 support before 2020, but I > > have no issue with dropping python2 support on windows earlier as it is > > our largest pain point. > > I wonder about this too. I can imagine there are a reasonable number > of people using older Linux distributions on which they cannot upgrade > to a recent Python 3, but is that likely to be true for Windows? > > We'd have to make sure we could persuade pypi to give the older > version for Windows, by default - I don't know if that is possible. > Pip repo names and actual module names don't have to be the same. One potential work-around would be to make a 'numpylts' repo on PyPi which is the 1.17 version with support for Python 2.7 and bug-fix releases as required. This will still cause regressions but it's a matter of modifying `requirements.txt` in downstream Python 2.7 packages and not much else. E.g. in `requirements.txt`: numpy;python_version>"3.0" numpylts; python_version<"3.0" In both cases you still call `import numpy` in the code. Robert -- Robert McLeod, Ph.D. robbmcl...@gmail.com robbmcl...@protonmail.com www.entropyreduction.al ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposal of timeline for dropping Python 2.7 support
On Mon, Nov 13, 2017 at 7:47 AM, Matthias Bussonnier < bussonniermatth...@gmail.com> wrote: > > For this to be efficient, it should be done soon enough to allow > downstream projects to adapt their requirements.txt. > > Release managers: how much more effort would it be to upload current > numpy to both numpy and numpylts? > > I'm not quite sure I see the point. you would ask downstream to change > `numpy` to `numpylts` instead of `numpy` to `numpy<2` ? > > Also I think then you have the risk of having for example pandas > saying `numpy<2` and scipy saying `numpylts` and now the pasckages are > incompatible ? The trouble is PyPi doesn't allow multiple branches. So if you upload NumPy 2.0 wheels, then you cannot turn around and upload 1.18.X bug-fix patches. At least, this is my understanding of PyPi. -- Robert McLeod, Ph.D. robbmcl...@gmail.com robbmcl...@protonmail.com www.entropyreduction.al ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] ANN: NumExpr 2.6.5
== Announcing Numexpr 2.6.5 == Hi everyone, This is primarily an incremental performance improvement release, especially with regards to improving import times of downstream packages (e.g. `pandas`, `tables`, `sympy`). Import times have been reduced from ~300 ms to ~100 ms through removing a `pkg_resources` import and making the `cpuinfo` import lazy. The maximum number of threads is now set at import-time, similar to `numba`, by setting an environment variable 'NUMEXPR_MAX_THREADS'. The runtime number of threads can still be reduced by calling `numexpr.set_num_threads(N)`. DEPRECATION WARNING: The variable `numexpr.is_cpu_amd_intel` has been set to a dummy value of `False`. This variable may be removed in the future. Project documentation is available at: http://numexpr.readthedocs.io/ Changes from 2.6.4 to 2.6.5 --- - The maximum thread count can now be set at import-time by setting the environment variable 'NUMEXPR_MAX_THREADS'. The default number of max threads was lowered from 4096 (which was deemed excessive) to 64. - A number of imports were removed (pkg_resources) or made lazy (cpuinfo) in order to speed load-times for downstream packages (such as `pandas`, `sympy`, and `tables`). Import time has dropped from about 330 ms to 90 ms. Thanks to Jason Sachs for pointing out the source of the slow-down. - Thanks to Alvaro Lopez Ortega for updates to benchmarks to be compatible with Python 3. - Travis and AppVeyor now fail if the test module fails or errors. - Thanks to Mahdi Ben Jelloul for a patch that removed a bug where constants in `where` calls would raise a ValueError. - Fixed a bug whereby all-constant power operations would lead to infinite recursion. -- Robert McLeod, Ph.D. robbmcl...@gmail.com robbmcl...@protonmail.com robert.mcl...@hitachi-hhtc.ca www.entropyreduction.al ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] ANN: Numexpr 2.6.6
== Announcing Numexpr 2.6.6 == Hi everyone, This is a bug-fix release. Thanks to Mark Dickinson for a fix to the thread barrier that occassionally suffered from spurious wakeups on MacOSX. Project documentation is available at: http://numexpr.readthedocs.io/ Changes from 2.6.5 to 2.6.6 --- - Thanks to Mark Dickinson for a fix to the thread barrier that occassionally suffered from spurious wakeups on MacOSX. What's Numexpr? --- Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python. It has multi-threaded capabilities, as well as support for Intel's MKL (Math Kernel Library), which allows an extremely fast evaluation of transcendental functions (sin, cos, tan, exp, log...) while squeezing the last drop of performance out of your multi-core processors. Look here for a some benchmarks of numexpr using MKL: https://github.com/pydata/numexpr/wiki/NumexprMKL Its only dependency is NumPy (MKL is optional), so it works well as an easy-to-deploy, easy-to-use, computational engine for projects that don't want to adopt other solutions requiring more heavy dependencies. Where I can find Numexpr? - The project is hosted at GitHub in: https://github.com/pydata/numexpr You can get the packages from PyPI as well (but not for RC releases): http://pypi.python.org/pypi/numexpr Documentation is hosted at: http://numexpr.readthedocs.io/en/latest/ Share your experience - Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy data! -- Robert McLeod, Ph.D. robbmcl...@gmail.com robbmcl...@protonmail.com robert.mcl...@hitachi-hhtc.ca www.entropyreduction.al ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] ANN: Numexpr 2.6.7
== Announcing Numexpr 2.6.7 == Hi everyone, This is a bug-fix release. Thanks to Lehman Garrison for a fix that could result in memory leak-like behavior. Project documentation is available at: http://numexpr.readthedocs.io/ Changes from 2.6.6 to 2.6.7 --- - Thanks to Lehman Garrison for finding and fixing a bug that exhibited memory leak-like behavior. The use in `numexpr.evaluate` of `sys._getframe` combined with `.f_locals` from that frame object results an extra refcount on objects in the frame that calls `numexpr.evaluate`, and not `evaluate`'s frame. So if the calling frame remains in scope for a long time (such as a procedural script where `numexpr` is called from the base frame) garbage collection would never occur. - Imports for the `numexpr.test` submodule were made lazy in the `numexpr` module. What's Numexpr? --- Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python. It has multi-threaded capabilities, as well as support for Intel's MKL (Math Kernel Library), which allows an extremely fast evaluation of transcendental functions (sin, cos, tan, exp, log...) while squeezing the last drop of performance out of your multi-core processors. Look here for a some benchmarks of numexpr using MKL: https://github.com/pydata/numexpr/wiki/NumexprMKL Its only dependency is NumPy (MKL is optional), so it works well as an easy-to-deploy, easy-to-use, computational engine for projects that don't want to adopt other solutions requiring more heavy dependencies. Where I can find Numexpr? - The project is hosted at GitHub in: https://github.com/pydata/numexpr You can get the packages from PyPI as well (but not for RC releases): http://pypi.python.org/pypi/numexpr Documentation is hosted at: http://numexpr.readthedocs.io/en/latest/ Share your experience - Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy data! -- Robert McLeod, Ph.D. robbmcl...@gmail.com robbmcl...@protonmail.com robert.mcl...@hitachi-hhtc.ca www.entropyreduction.al ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] ANN: NumExpr 2.6.8
== Announcing Numexpr 2.6.8 == Hi everyone, Our attempt to fix the memory leak in 2.6.7 had an unforseen consequence that the `f_locals` from the top-most frame is actually `f_globals`, and clearing it to fix the extra reference count deletes all global variables. Needless to say this is undesired behavior. A check has been added to prevent clearing the globals dict, tested against both `python` and `ipython`. As such, we recommend skipping 2.6.7 and upgrading straight to 2.6.8 from 2.6.6. Project documentation is available at: http://numexpr.readthedocs.io/ Changes from 2.6.7 to 2.6.8 --- - Add check to make sure that `f_locals` is not actually `f_globals` when we do the `f_locals` clear to avoid the #310 memory leak issue. - Compare NumPy versions using `distutils.version.LooseVersion` to avoid issue #312 when working with NumPy development versions. - As part of `multibuild`, wheels for Python 3.7 for Linux and MacOSX are now available on PyPI. What's Numexpr? --- Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python. It has multi-threaded capabilities, as well as support for Intel's MKL (Math Kernel Library), which allows an extremely fast evaluation of transcendental functions (sin, cos, tan, exp, log...) while squeezing the last drop of performance out of your multi-core processors. Look here for a some benchmarks of numexpr using MKL: https://github.com/pydata/numexpr/wiki/NumexprMKL Its only dependency is NumPy (MKL is optional), so it works well as an easy-to-deploy, easy-to-use, computational engine for projects that don't want to adopt other solutions requiring more heavy dependencies. Where I can find Numexpr? - The project is hosted at GitHub in: https://github.com/pydata/numexpr You can get the packages from PyPI as well (but not for RC releases): http://pypi.python.org/pypi/numexpr Documentation is hosted at: http://numexpr.readthedocs.io/en/latest/ Share your experience - Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy data! -- Robert McLeod, Ph.D. robbmcl...@gmail.com robbmcl...@protonmail.com robert.mcl...@hitachi-hhtc.ca www.entropyreduction.al ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] ANN: NumExpr 2.6.9
Hi everyone, This is a version-bump release to provide wheels for Python 3.7.1 on Windows platforms. Also Mike Toews made the handling of our environment variables more robust. Project documentation is available at: http://numexpr.readthedocs.io/ Changes from 2.6.8 to 2.6.9 --- - Thanks to Mike Toews for more robust handling of the thread-setting environment variables. - With Appveyor updating to Python 3.7.1, wheels for Python 3.7 are now available in addition to those for other OSes. What's Numexpr? --- Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python. It has multi-threaded capabilities, as well as support for Intel's MKL (Math Kernel Library), which allows an extremely fast evaluation of transcendental functions (sin, cos, tan, exp, log...) while squeezing the last drop of performance out of your multi-core processors. Look here for a some benchmarks of numexpr using MKL: https://github.com/pydata/numexpr/wiki/NumexprMKL Its only dependency is NumPy (MKL is optional), so it works well as an easy-to-deploy, easy-to-use, computational engine for projects that don't want to adopt other solutions requiring more heavy dependencies. Where I can find Numexpr? - The project is hosted at GitHub in: https://github.com/pydata/numexpr You can get the packages from PyPI as well (but not for RC releases): http://pypi.python.org/pypi/numexpr Documentation is hosted at: http://numexpr.readthedocs.io/en/latest/ Share your experience - Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy data! -- Robert McLeod, Ph.D. robbmcl...@gmail.com robbmcl...@protonmail.com robert.mcl...@hitachi-hhtc.ca www.entropyreduction.al ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 'nansqrt' function?
The `bottleneck` library is a very good package if there's some function in NumPy that you want to handle `nan`s in reductions without exploding. https://github.com/kwgoodman/bottleneck On Wed, Feb 13, 2019 at 12:35 PM Mauro Cavalcanti wrote: > Dear ALL, > > In the process of porting an existing (but abandoned) package to the > latest version of Numpy, I stumbled upon a call to a 'numpy.nansqrt' > function, which seems not to exist. > > Here is the specific code: > > def normTrans(y): > denom = np.nansqrt(np.nansum(y**2)) > return y/denom > > As far as I could find, there is no such 'nansqrt' function in the current > version of Numpy, so I suspect that the above code has not been properly > tested. > > Am I right, or that function had existed in some past version of Numpy? > > Thanks in advance for any comments or suggestions. > > Best regards, > ___ > NumPy-Discussion mailing list > NumPy-Discussion@python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -- Robert McLeod, Ph.D. robbmcl...@gmail.com robbmcl...@protonmail.com robert.mcl...@hitachi-hhtc.ca www.entropyreduction.al ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] ANN: NumExpr 2.7.0 Release
= Announcing Numexpr 2.7.0 = Hi everyone, This is a minor version bump for NumExpr. We would like to highlight the changes made in 2.6.9 (which in retrospect should have been a minor version bump), where the maximum number of threads spawned can be limited by setting the environment variable "NUMEXPR_MAX_THREADS". If this variable is not set, in 2.7.0 the historical limit of 8 threads will be used. The lack of a check caused some problems on very large hosts in cluster environments in 2.6.9. In addition, we are officially dropping Python 2.6 support in this release as we cannot perform continuous integration for it. Project documentation is available at: http://numexpr.readthedocs.io/ Changes from 2.6.9 to 2.7.0 - The default number of 'safe' threads has been restored to the historical limit of 8, if the environment variable "NUMEXPR_MAX_THREADS" has not been set. - Thanks to @eltoder who fixed a small memory leak. - Support for Python 2.6 has been dropped, as it is no longer available via TravisCI. - A typo in the test suite that had a less than rather than greater than symbol in the NumPy version check has been corrected thanks to dhomeier. - The file `site.cfg` was being accidentally included in the sdists on PyPi. It has now been excluded. What's Numexpr? --- Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python. It has multi-threaded capabilities, as well as support for Intel's MKL (Math Kernel Library), which allows an extremely fast evaluation of transcendental functions (sin, cos, tan, exp, log...) while squeezing the last drop of performance out of your multi-core processors. Look here for a some benchmarks of numexpr using MKL: https://github.com/pydata/numexpr/wiki/NumexprMKL Its only dependency is NumPy (MKL is optional), so it works well as an easy-to-deploy, easy-to-use, computational engine for projects that don't want to adopt other solutions requiring more heavy dependencies. Where I can find Numexpr? - The project is hosted at GitHub in: https://github.com/pydata/numexpr You can get the packages from PyPI as well (but not for RC releases): http://pypi.python.org/pypi/numexpr Documentation is hosted at: http://numexpr.readthedocs.io/en/latest/ Share your experience - Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy data! -- Robert McLeod robbmcl...@gmail.com robert.mcl...@hitachi-hhtc.ca ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Announcing Numexpr 2.7.1
Hi everyone, This is a version bump to add support for Python 3.8 and NumPy 1.18. We are also removing support for Python 3.4. Project documentation is available at: http://numexpr.readthedocs.io/ Changes from 2.7.0 to 2.7.1 - Python 3.8 support has been added. - Python 3.4 support is discontinued. - The tests are now compatible with NumPy 1.18. - `site.cfg.example` was updated to use the `libraries` tag instead of `mkl_libs`, which is recommended for newer version of NumPy. What's Numexpr? --- Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python. It has multi-threaded capabilities, as well as support for Intel's MKL (Math Kernel Library), which allows an extremely fast evaluation of transcendental functions (sin, cos, tan, exp, log...) while squeezing the last drop of performance out of your multi-core processors. Look here for a some benchmarks of numexpr using MKL: https://github.com/pydata/numexpr/wiki/NumexprMKL Its only dependency is NumPy (MKL is optional), so it works well as an easy-to-deploy, easy-to-use, computational engine for projects that don't want to adopt other solutions requiring more heavy dependencies. Where I can find Numexpr? - The project is hosted at GitHub in: https://github.com/pydata/numexpr You can get the packages from PyPI as well (but not for RC releases): http://pypi.python.org/pypi/numexpr Documentation is hosted at: http://numexpr.readthedocs.io/en/latest/ Share your experience - Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy data! -- Robert McLeod robbmcl...@gmail.com robert.mcl...@hitachi-hhtc.ca ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] manylinux wheels with Github Actions
Everyone, I'm looking to move `numexpr` into GitHub Actions for the combination of testing but also building wheels for deployment to pypi. I've never really been satisfied with the Appveyor/Travis combo and the documentation around Actions seems to be a lot stronger. I thought I might ask for advice on the mailing list before I jump in. Does anyone here know of good, working examples of such for scientific packages that I could crib as a guide? I would want/need to make use of manylinux1 Docker images for building Linux wheels. https://github.com/pypa/manylinux This is an example recipe that builds Cython using a custom Docker image. It's a nice starting point but it would be preferable to use the official pypa images. https://github.com/marketplace/actions/python-wheels-manylinux-build Kivy is a working example with a pretty complete test and build setup (although it's a bit over complicated for my purposes): https://github.com/kivy/kivy/tree/master/.github/workflows Anyone have any experiences to share with test and deploy via Actions? Robert -- Robert McLeod robbmcl...@gmail.com robert.mcl...@hitachi-hhtc.ca ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] ANN: NumExpr 2.7.2
Announcing NumExpr 2.7.2 Hi everyone, It's been awhile since the last update to NumExpr, mostly as the existing scientific Python tool chain for building wheels on PyPi became defunct and we have had to redevelop a new one based on `cibuildwheel` and GitHub Actions. This release also brings us support (and wheels for) Python 3.9. There have been a number of changes to enhance how NumExpr works when NumPy uses MKL as a backend. Project documentation is available at: http://numexpr.readthedocs.io/ Changes from 2.7.1 to 2.7.2 --- - Support for Python 2.7 and 3.5 is deprecated and will be discontinued when `cibuildwheels` and/or GitHub Actions no longer support these versions. - Wheels are now provided for Python 3.7, 3.5, 3.6, 3.7, 3.8, and 3.9 via GitHub Actions. - The block size is now exported into the namespace as `numexpr.__BLOCK_SIZE1__` as a read-only value. - If using MKL, the number of threads for VML is no longer forced to 1 on loading the module. Testing has shown that VML never runs in multi-threaded mode for the default BLOCKSIZE1 of 1024 elements, and forcing to 1 can have deleterious effects on NumPy functions when built with MKL. See issue #355 for details. - Use of `ndarray.tostring()` in tests has been switch to `ndarray.tobytes()` for future-proofing deprecation of `.tostring()`, if the version of NumPy is greater than 1.9. - Added a utility method `get_num_threads` that returns the (maximum) number of threads currently in use by the virtual machine. The functionality of `set_num_threads` whereby it returns the previous value has been deprecated and will be removed in 2.8.X. What's Numexpr? --- Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python. It has multi-threaded capabilities, as well as support for Intel's MKL (Math Kernel Library), which allows an extremely fast evaluation of transcendental functions (sin, cos, tan, exp, log...) while squeezing the last drop of performance out of your multi-core processors. Look here for a some benchmarks of numexpr using MKL: https://github.com/pydata/numexpr/wiki/NumexprMKL Its only dependency is NumPy (MKL is optional), so it works well as an easy-to-deploy, easy-to-use, computational engine for projects that don't want to adopt other solutions requiring more heavy dependencies. Where I can find Numexpr? - The project is hosted at GitHub in: https://github.com/pydata/numexpr You can get the packages from PyPI as well (but not for RC releases): http://pypi.python.org/pypi/numexpr Documentation is hosted at: http://numexpr.readthedocs.io/en/latest/ Share your experience - Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy data! -- Robert McLeod robbmcl...@gmail.com robert.mcl...@hitachi-hhtc.ca ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] ANN: NumExpr 2.7.3
Announcing NumExpr 2.7.3 Hi everyone, This is a maintenance release to make use of the oldest supported NumPy version when building wheels, in an effort to alleviate issues seen on Windows machines that do not have the latest Windows MSVC runtime installed. It also adds wheels built via GitHub Actions for ARMv8 platforms. Project documentation is available at: http://numexpr.readthedocs.io/ Changes from 2.7.2 to 2.7.3 --- - Pinned Numpy versions to minimum supported version in an effort to alleviate issues seen in Windows machines not having the same Windows SDK installed as was used to build the wheels. - ARMv8 wheels are now available, thanks to `odidev` for the pull request. What's Numexpr? --- Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python. It has multi-threaded capabilities, as well as support for Intel's MKL (Math Kernel Library), which allows an extremely fast evaluation of transcendental functions (sin, cos, tan, exp, log...) while squeezing the last drop of performance out of your multi-core processors. Look here for a some benchmarks of numexpr using MKL: https://github.com/pydata/numexpr/wiki/NumexprMKL Its only dependency is NumPy (MKL is optional), so it works well as an easy-to-deploy, easy-to-use, computational engine for projects that don't want to adopt other solutions requiring more heavy dependencies. Where I can find Numexpr? - The project is hosted at GitHub in: https://github.com/pydata/numexpr You can get the packages from PyPI as well (but not for RC releases): http://pypi.python.org/pypi/numexpr Documentation is hosted at: http://numexpr.readthedocs.io/en/latest/ Share your experience - Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy data! -- Robert McLeod robbmcl...@gmail.com robert.mcl...@hitachi-hhtc.ca ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Announcing NumExpr 2.8.0
Hi everyone, NumExpr 2.8.0 is released. This is mostly a version bump. We now support Python 3.10 and support for 2.7 and 3.5 has been discontinued. Project documentation is available at: http://numexpr.readthedocs.io/ Changes from 2.7.3 to 2.8.0 --- * Wheels for Python 3.10 are now provided. * Support for Python 2.7 and 3.5 has been discontinued. * All residual support for Python 2.X syntax has been removed, and therefore the setup build no longer makes calls to the `2to3` script. The `setup.py` has been refactored to be more modern. * The examples on how to link into Intel VML/MKL/oneAPI now use the dynamic library. What's Numexpr? --- Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python. It has multi-threaded capabilities, as well as support for Intel's MKL (Math Kernel Library), which allows an extremely fast evaluation of transcendental functions (sin, cos, tan, exp, log...) while squeezing the last drop of performance out of your multi-core processors. Look here for a some benchmarks of numexpr using MKL: https://github.com/pydata/numexpr/wiki/NumexprMKL Its only dependency is NumPy (MKL is optional), so it works well as an easy-to-deploy, easy-to-use, computational engine for projects that don't want to adopt other solutions requiring more heavy dependencies. Where I can find Numexpr? - The project is hosted at GitHub in: https://github.com/pydata/numexpr You can get the packages from PyPI as well (but not for RC releases): http://pypi.python.org/pypi/numexpr Documentation is hosted at: http://numexpr.readthedocs.io/en/latest/ Share your experience - Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy data! -- Robert McLeod robbmcl...@gmail.com robert.mcl...@hitachi-hightech.com ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Announcing NumExpr 2.8.1
Hi everyone, This is another maintenance release to further modernize our install script for distributions that do not include `setuptools` by default. Thanks to Antonio Valentino for the changes. Project documentation is available at: http://numexpr.readthedocs.io/ Changes from 2.8.0 to 2.8.1 --- * Fixed dependency list. * Added ``pyproject.toml`` and modernize the ``setup.py`` script. Thanks to Antonio Valentino for the PR. What's Numexpr? --- Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python. It has multi-threaded capabilities, as well as support for Intel's MKL (Math Kernel Library), which allows an extremely fast evaluation of transcendental functions (sin, cos, tan, exp, log...) while squeezing the last drop of performance out of your multi-core processors. Look here for a some benchmarks of numexpr using MKL: https://github.com/pydata/numexpr/wiki/NumexprMKL Its only dependency is NumPy (MKL is optional), so it works well as an easy-to-deploy, easy-to-use, computational engine for projects that don't want to adopt other solutions requiring more heavy dependencies. Where I can find Numexpr? - The project is hosted at GitHub in: https://github.com/pydata/numexpr You can get the packages from PyPI as well (but not for RC releases): http://pypi.python.org/pypi/numexpr Documentation is hosted at: http://numexpr.readthedocs.io/en/latest/ Share your experience - Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy data! -- Robert McLeod robbmcl...@gmail.com robert.mcl...@hitachi-hightech.com ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] ANN: NumExpr 2.8.3
Hi everyone,
Please find here another maintenance release of NumExpr. Support for Python
3.6
has been dropped to enable support for NumPy 1.23 (and by extension Python
3.11
when it is released). Wheels for ARM64 multilinux should be available again
after
troubles with GitHub Actions and Apple Silicon wheels are also now
available on
PyPi for download.
Project documentation is available at:
http://numexpr.readthedocs.io/
Changes from 2.8.1 to 2.8.2
---
* Support for Python 3.6 has been dropped due to the need to substitute the
flag
`NPY_ARRAY_WRITEBACKIFCOPY` for `NPY_ARRAY_UPDATEIFCOPY`. This flag
change was
initiated in NumPy 1.14 and finalized in 1.23. The only changes were made
to
cases where an unaligned constant was passed in with a pre-allocated
output
variable:
```
x = np.empty(5, dtype=np.uint8)[1:].view(np.int32)
ne.evaluate('3', out=x)
```
We think the risk of issues is very low, but if you are using NumExpr as
a
expression evaluation tool you may want to write a test for this edge
case.
* Thanks to Matt Einhorn (@matham) for improvements to the GitHub Actions
build process to add support for Apple Silicon and aarch64.
* Thanks to Biswapriyo Nath (@biswa96) for a fix to allow `mingw` builds on
Windows.
* There have been some changes made to not import `platform.machine()` on
`sparc`
but it is highly advised to upgrade to Python 3.9+ to avoid this issue
with
the Python core package `platform`.
What's Numexpr?
---
Numexpr is a fast numerical expression evaluator for NumPy. With it,
expressions that operate on arrays (like "3*a+4*b") are accelerated
and use less memory than doing the same calculation in Python.
It has multi-threaded capabilities, as well as support for Intel's
MKL (Math Kernel Library), which allows an extremely fast evaluation
of transcendental functions (sin, cos, tan, exp, log...) while
squeezing the last drop of performance out of your multi-core
processors. Look here for a some benchmarks of numexpr using MKL:
https://github.com/pydata/numexpr/wiki/NumexprMKL
Its only dependency is NumPy (MKL is optional), so it works well as an
easy-to-deploy, easy-to-use, computational engine for projects that
don't want to adopt other solutions requiring more heavy dependencies.
Where I can find Numexpr?
-
The project is hosted at GitHub in:
https://github.com/pydata/numexpr
You can get the packages from PyPI as well (but not for RC releases):
http://pypi.python.org/pypi/numexpr
Documentation is hosted at:
http://numexpr.readthedocs.io/en/latest/
Share your experience
-
Let us know of any bugs, suggestions, gripes, kudos, etc. you may
have.
Enjoy data!
--
Robert McLeod
robbmcl...@gmail.com
robert.mcl...@hitachi-hightech.com
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com
[Numpy-discussion] ANN: NumExpr 2.8.4 Release
Announcing NumExpr 2.8.4 Hi everyone, This is a maintenance and bug-fix release for NumExpr. In particular, now we have added Python 3.11 support. Project documentation is available at: http://numexpr.readthedocs.io/ Changes from 2.8.3 to 2.8.4 --- * Support for Python 3.11 has been added. * Thanks to Tobias Hangleiter for an improved accuracy complex `expm1` function. While it is 25 % slower, it is significantly more accurate for the real component over a range of values and matches NumPy outputs much more closely. * Thanks to Kirill Kouzoubov for a range of fixes to constants parsing that was resulting in duplicated constants of the same value. * Thanks to Mark Harfouche for noticing that we no longer need `numpy` version checks. `packaging` is no longer a requirement as a result. What's Numexpr? --- Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python. It has multi-threaded capabilities, as well as support for Intel's MKL (Math Kernel Library), which allows an extremely fast evaluation of transcendental functions (sin, cos, tan, exp, log...) while squeezing the last drop of performance out of your multi-core processors. Look here for a some benchmarks of numexpr using MKL: https://github.com/pydata/numexpr/wiki/NumexprMKL Its only dependency is NumPy (MKL is optional), so it works well as an easy-to-deploy, easy-to-use, computational engine for projects that don't want to adopt other solutions requiring more heavy dependencies. Where I can find Numexpr? - The project is hosted at GitHub in: https://github.com/pydata/numexpr You can get the packages from PyPI as well (but not for RC releases): http://pypi.python.org/pypi/numexpr Documentation is hosted at: http://numexpr.readthedocs.io/en/latest/ Share your experience - Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy data! -- Robert McLeod robbmcl...@gmail.com robert.mcl...@hitachi-hightech.com ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] ANN: NumExpr 2.8.5
Hi everyone, In 2.8.5 we have added a new function, `validate` which checks an expression `ex` for validity, for usage where the program is parsing a user input. There are also consequences for this sort of usage, since `eval(ex)` is called, and as such we do some string sanitization as described below. Project documentation is available at: http://numexpr.readthedocs.io/ Changes from 2.8.4 to 2.8.5 --- * A `validate` function has been added. This function checks the inputs, returning `None` on success or raising an exception on invalid inputs. This function was added as numerous projects seem to be using NumExpr for parsing user inputs. `re_evaluate` may be called directly following `validate`. * As an addendum to the use of NumExpr for parsing user inputs, is that NumExpr calls `eval` on the inputs. A regular expression is now applied to help sanitize the input expression string, forbidding '__', ':', and ';'. Attribute access is also banned except for '.r' for real and '.i' for imag. * Thanks to timbrist for a fix to behavior of NumExpr with integers to negative powers. NumExpr was pre-checking integer powers for negative values, which was both inefficient and caused parsing errors in some situations. Now NumExpr will simply return 0 as a result for such cases. While NumExpr generally tries to follow NumPy behavior, performance is also critical. * Thanks to peadar for some fixes to how NumExpr launches threads for embedded applications. * Thanks to de11n for making parsing of the `site.cfg` for MKL consistent among all shared platforms. What's Numexpr? --- Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python. It has multi-threaded capabilities, as well as support for Intel's MKL (Math Kernel Library), which allows an extremely fast evaluation of transcendental functions (sin, cos, tan, exp, log...) while squeezing the last drop of performance out of your multi-core processors. Look here for a some benchmarks of numexpr using MKL: https://github.com/pydata/numexpr/wiki/NumexprMKL Its only dependency is NumPy (MKL is optional), so it works well as an easy-to-deploy, easy-to-use, computational engine for projects that don't want to adopt other solutions requiring more heavy dependencies. Where I can find Numexpr? - The project is hosted at GitHub in: https://github.com/pydata/numexpr You can get the packages from PyPI as well (but not for RC releases): http://pypi.python.org/pypi/numexpr Documentation is hosted at: http://numexpr.readthedocs.io/en/latest/ Share your experience - Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy data! -- Robert McLeod robbmcl...@gmail.com robert.mcl...@hitachi-hightech.com ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
