[Numpy-discussion] Re: Setting C and linker flags for Windows build
Hi, On Thu, Jun 30, 2022 at 12:42 AM Kevin Sheppard wrote: >> >> >> Hi, >> >> I am very sorry - I feel I should know this, or be able to work it >> out, but is there a way of setting the flags to the C compiler and the >> linker, for the Numpy build, on Windows? >> >> I'm trying to set the flags for a build with Windows mingw-w64 - but I >> believe Numpy is ignoring $env:LDFLAGS, $env:CFLAGS and $env:OPT - and >> I can't see any way of setting these options from the command line. >> Am I missing something? >> >> Cheers, >> >> Matthew >> Member address: kevin.k.shepp...@gmail.com > > > I think these are all hard coded and non-changable. This was the case when I > got NumPy building with clang-cl. That was my impression too, but I was delayed by the text at: https://numpy.org/doc/stable/user/building.html#supplying-additional-compiler-flags which says: """ Additional compiler flags can be supplied by setting the OPT, FOPT (for Fortran), and CC environment variables. When providing options that should improve the performance of the code ensure that you also set -DNDEBUG so that debugging code is not executed. """ I guess we should change that text to note these do not work on Windows. I think you can supply extra Fortran flags with the `config_gc` option to `setup.py`, but I don't think the others have any effect on Windows. I was also confused by these lines in `azure-steps-windows.yml`: """ $env:CFLAGS = "-m32" $env:LDFLAGS = "-m32" """ I assume these don't actually have any effect. Cheers, Matthew ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Re: Setting C and linker flags for Windows build
On Thu, Jun 30, 2022 at 10:29 AM Matthew Brett wrote: > Hi, > > On Thu, Jun 30, 2022 at 12:42 AM Kevin Sheppard > wrote: > >> > >> > >> Hi, > >> > >> I am very sorry - I feel I should know this, or be able to work it > >> out, but is there a way of setting the flags to the C compiler and the > >> linker, for the Numpy build, on Windows? > >> > >> I'm trying to set the flags for a build with Windows mingw-w64 - but I > >> believe Numpy is ignoring $env:LDFLAGS, $env:CFLAGS and $env:OPT - and > >> I can't see any way of setting these options from the command line. > >> Am I missing something? > >> > >> Cheers, > >> > >> Matthew > > >> Member address: kevin.k.shepp...@gmail.com > > > > > > I think these are all hard coded and non-changable. This was the case > when I got NumPy building with clang-cl. > > That was my impression too, but I was delayed by the text at: > > > https://numpy.org/doc/stable/user/building.html#supplying-additional-compiler-flags > > which says: > > """ > Additional compiler flags can be supplied by setting the OPT, FOPT > (for Fortran), and CC environment variables. When providing options > that should improve the performance of the code ensure that you also > set -DNDEBUG so that debugging code is not executed. > """ > > I guess we should change that text to note these do not work on Windows. > > I think you can supply extra Fortran flags with the `config_gc` option > to `setup.py`, but I don't think the others have any effect on > Windows. > > I was also confused by these lines in `azure-steps-windows.yml`: > > """ > $env:CFLAGS = "-m32" > $env:LDFLAGS = "-m32" > """ > > I assume these don't actually have any effect. > No, the above all seem wrong (unless there has been a major regression recently), and the `-m32` flags work and are necessary. CFLAGS, FFLAGS, CXXFLAGS and LDFLAGS can be used, and append compile flags. If you want to remove flags that are already hardcoded in numpy.distutils, then that's a different story - you need to edit the numpy.distutils source code then. I just stumbled on a Windows problem with `$env:`, and the problem was Windows CI is hopelessly weird. It matters for example if you do something in a Powershell context (`ps |` in a .yml file) or outside of it. So it may be something like that. Cheers, Ralf ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Re: Setting C and linker flags for Windows build
Hi, On Thu, Jun 30, 2022 at 9:40 AM Ralf Gommers wrote: > > > > On Thu, Jun 30, 2022 at 10:29 AM Matthew Brett > wrote: >> >> Hi, >> >> On Thu, Jun 30, 2022 at 12:42 AM Kevin Sheppard >> wrote: >> >> >> >> >> >> Hi, >> >> >> >> I am very sorry - I feel I should know this, or be able to work it >> >> out, but is there a way of setting the flags to the C compiler and the >> >> linker, for the Numpy build, on Windows? >> >> >> >> I'm trying to set the flags for a build with Windows mingw-w64 - but I >> >> believe Numpy is ignoring $env:LDFLAGS, $env:CFLAGS and $env:OPT - and >> >> I can't see any way of setting these options from the command line. >> >> Am I missing something? >> >> >> >> Cheers, >> >> >> >> Matthew >> >> >> Member address: kevin.k.shepp...@gmail.com >> > >> > >> > I think these are all hard coded and non-changable. This was the case >> > when I got NumPy building with clang-cl. >> >> That was my impression too, but I was delayed by the text at: >> >> https://numpy.org/doc/stable/user/building.html#supplying-additional-compiler-flags >> >> which says: >> >> """ >> Additional compiler flags can be supplied by setting the OPT, FOPT >> (for Fortran), and CC environment variables. When providing options >> that should improve the performance of the code ensure that you also >> set -DNDEBUG so that debugging code is not executed. >> """ >> >> I guess we should change that text to note these do not work on Windows. >> >> I think you can supply extra Fortran flags with the `config_gc` option >> to `setup.py`, but I don't think the others have any effect on >> Windows. >> >> I was also confused by these lines in `azure-steps-windows.yml`: >> >> """ >> $env:CFLAGS = "-m32" >> $env:LDFLAGS = "-m32" >> """ >> >> I assume these don't actually have any effect. > > > No, the above all seem wrong (unless there has been a major regression > recently), and the `-m32` flags work and are necessary. CFLAGS, FFLAGS, > CXXFLAGS and LDFLAGS can be used, and append compile flags. If you want to > remove flags that are already hardcoded in numpy.distutils, then that's a > different story - you need to edit the numpy.distutils source code then. > > I just stumbled on a Windows problem with `$env:`, and the problem was > Windows CI is hopelessly weird. It matters for example if you do something in > a Powershell context (`ps |` in a .yml file) or outside of it. So it may be > something like that. Are you sure though - that you can append flags to the compile and link step with CFLAGS and LDFLAGS? It didn't work for me locally. And I can't see where they would have an effect, on Windows, in the code. Cheers, Matthew ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Re: Setting C and linker flags for Windows build
On Thu, Jun 30, 2022 at 10:47 AM Matthew Brett wrote: > Hi, > > On Thu, Jun 30, 2022 at 9:40 AM Ralf Gommers > wrote: > > > > > > > > On Thu, Jun 30, 2022 at 10:29 AM Matthew Brett > wrote: > >> > >> Hi, > >> > >> On Thu, Jun 30, 2022 at 12:42 AM Kevin Sheppard > >> wrote: > >> >> > >> >> > >> >> Hi, > >> >> > >> >> I am very sorry - I feel I should know this, or be able to work it > >> >> out, but is there a way of setting the flags to the C compiler and > the > >> >> linker, for the Numpy build, on Windows? > >> >> > >> >> I'm trying to set the flags for a build with Windows mingw-w64 - but > I > >> >> believe Numpy is ignoring $env:LDFLAGS, $env:CFLAGS and $env:OPT - > and > >> >> I can't see any way of setting these options from the command line. > >> >> Am I missing something? > >> >> > >> >> Cheers, > >> >> > >> >> Matthew > >> > >> >> Member address: kevin.k.shepp...@gmail.com > >> > > >> > > >> > I think these are all hard coded and non-changable. This was the > case when I got NumPy building with clang-cl. > >> > >> That was my impression too, but I was delayed by the text at: > >> > >> > https://numpy.org/doc/stable/user/building.html#supplying-additional-compiler-flags > >> > >> which says: > >> > >> """ > >> Additional compiler flags can be supplied by setting the OPT, FOPT > >> (for Fortran), and CC environment variables. When providing options > >> that should improve the performance of the code ensure that you also > >> set -DNDEBUG so that debugging code is not executed. > >> """ > >> > >> I guess we should change that text to note these do not work on Windows. > >> > >> I think you can supply extra Fortran flags with the `config_gc` option > >> to `setup.py`, but I don't think the others have any effect on > >> Windows. > >> > >> I was also confused by these lines in `azure-steps-windows.yml`: > >> > >> """ > >> $env:CFLAGS = "-m32" > >> $env:LDFLAGS = "-m32" > >> """ > >> > >> I assume these don't actually have any effect. > > > > > > No, the above all seem wrong (unless there has been a major regression > recently), and the `-m32` flags work and are necessary. CFLAGS, FFLAGS, > CXXFLAGS and LDFLAGS can be used, and append compile flags. If you want to > remove flags that are already hardcoded in numpy.distutils, then that's a > different story - you need to edit the numpy.distutils source code then. > > > > I just stumbled on a Windows problem with `$env:`, and the problem was > Windows CI is hopelessly weird. It matters for example if you do something > in a Powershell context (`ps |` in a .yml file) or outside of it. So it may > be something like that. > > Are you sure though - that you can append flags to the compile and > link step with CFLAGS and LDFLAGS? It didn't work for me locally. > And I can't see where they would have an effect, on Windows, in the > code. > Yes I'm sure, and it's even one of the few things in numpy.distutils that are tested: https://github.com/numpy/numpy/pull/14250. Maybe see if you can make that test fail when setting env vars in a couple of different ways? Cheers, Ralf ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Re: Setting C and linker flags for Windows build
On Thu, Jun 30, 2022 at 10:57 AM Ralf Gommers wrote: > > > On Thu, Jun 30, 2022 at 10:47 AM Matthew Brett > wrote: > >> Hi, >> >> On Thu, Jun 30, 2022 at 9:40 AM Ralf Gommers >> wrote: >> > >> > >> > >> > On Thu, Jun 30, 2022 at 10:29 AM Matthew Brett >> wrote: >> >> >> >> Hi, >> >> >> >> On Thu, Jun 30, 2022 at 12:42 AM Kevin Sheppard >> >> wrote: >> >> >> >> >> >> >> >> >> Hi, >> >> >> >> >> >> I am very sorry - I feel I should know this, or be able to work it >> >> >> out, but is there a way of setting the flags to the C compiler and >> the >> >> >> linker, for the Numpy build, on Windows? >> >> >> >> >> >> I'm trying to set the flags for a build with Windows mingw-w64 - >> but I >> >> >> believe Numpy is ignoring $env:LDFLAGS, $env:CFLAGS and $env:OPT - >> and >> >> >> I can't see any way of setting these options from the command line. >> >> >> Am I missing something? >> >> >> >> >> >> Cheers, >> >> >> >> >> >> Matthew >> >> >> >> >> Member address: kevin.k.shepp...@gmail.com >> >> > >> >> > >> >> > I think these are all hard coded and non-changable. This was the >> case when I got NumPy building with clang-cl. >> >> >> >> That was my impression too, but I was delayed by the text at: >> >> >> >> >> https://numpy.org/doc/stable/user/building.html#supplying-additional-compiler-flags >> >> >> >> which says: >> >> >> >> """ >> >> Additional compiler flags can be supplied by setting the OPT, FOPT >> >> (for Fortran), and CC environment variables. When providing options >> >> that should improve the performance of the code ensure that you also >> >> set -DNDEBUG so that debugging code is not executed. >> >> """ >> >> >> >> I guess we should change that text to note these do not work on >> Windows. >> >> >> >> I think you can supply extra Fortran flags with the `config_gc` option >> >> to `setup.py`, but I don't think the others have any effect on >> >> Windows. >> >> >> >> I was also confused by these lines in `azure-steps-windows.yml`: >> >> >> >> """ >> >> $env:CFLAGS = "-m32" >> >> $env:LDFLAGS = "-m32" >> >> """ >> >> >> >> I assume these don't actually have any effect. >> > >> > >> > No, the above all seem wrong (unless there has been a major regression >> recently), and the `-m32` flags work and are necessary. CFLAGS, FFLAGS, >> CXXFLAGS and LDFLAGS can be used, and append compile flags. If you want to >> remove flags that are already hardcoded in numpy.distutils, then that's a >> different story - you need to edit the numpy.distutils source code then. >> > >> > I just stumbled on a Windows problem with `$env:`, and the problem was >> Windows CI is hopelessly weird. It matters for example if you do something >> in a Powershell context (`ps |` in a .yml file) or outside of it. So it may >> be something like that. >> >> Are you sure though - that you can append flags to the compile and >> link step with CFLAGS and LDFLAGS? It didn't work for me locally. >> And I can't see where they would have an effect, on Windows, in the >> code. >> > > Yes I'm sure, and it's even one of the few things in numpy.distutils that > are tested: https://github.com/numpy/numpy/pull/14250. Maybe see if you > can make that test fail when setting env vars in a couple of different ways? > Note that the C/C++ handling can come from plain `distutils`, but it should work the same. Here's a PR that fixes the Windows wheel builds for SciPy for a Pythran issue by using CXXFLAGS: https://github.com/MacPython/scipy-wheels/pull/122. The CPython bug tracker is full of noise about this kind of env var handling, nothing conclusive as far as I know. It's of course possible that it's broken under some circumstances (or in `setuptools.distutils`). There are no tests for this in CPython it looks like. Cheers, Ralf ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Re: Setting C and linker flags for Windows build
On Thu, Jun 30, 2022 at 11:15 AM Ralf Gommers wrote: > > > On Thu, Jun 30, 2022 at 10:57 AM Ralf Gommers > wrote: > >> >> >> On Thu, Jun 30, 2022 at 10:47 AM Matthew Brett >> wrote: >> >>> Hi, >>> >>> On Thu, Jun 30, 2022 at 9:40 AM Ralf Gommers >>> wrote: >>> > >>> > >>> > >>> > On Thu, Jun 30, 2022 at 10:29 AM Matthew Brett < >>> matthew.br...@gmail.com> wrote: >>> >> >>> >> Hi, >>> >> >>> >> On Thu, Jun 30, 2022 at 12:42 AM Kevin Sheppard >>> >> wrote: >>> >> >> >>> >> >> >>> >> >> Hi, >>> >> >> >>> >> >> I am very sorry - I feel I should know this, or be able to work it >>> >> >> out, but is there a way of setting the flags to the C compiler and >>> the >>> >> >> linker, for the Numpy build, on Windows? >>> >> >> >>> >> >> I'm trying to set the flags for a build with Windows mingw-w64 - >>> but I >>> >> >> believe Numpy is ignoring $env:LDFLAGS, $env:CFLAGS and $env:OPT - >>> and >>> >> >> I can't see any way of setting these options from the command line. >>> >> >> Am I missing something? >>> >> >> >>> >> >> Cheers, >>> >> >> >>> >> >> Matthew >>> >> >>> >> >> Member address: kevin.k.shepp...@gmail.com >>> >> > >>> >> > >>> >> > I think these are all hard coded and non-changable. This was the >>> case when I got NumPy building with clang-cl. >>> >> >>> >> That was my impression too, but I was delayed by the text at: >>> >> >>> >> >>> https://numpy.org/doc/stable/user/building.html#supplying-additional-compiler-flags >>> >> >>> >> which says: >>> >> >>> >> """ >>> >> Additional compiler flags can be supplied by setting the OPT, FOPT >>> >> (for Fortran), and CC environment variables. When providing options >>> >> that should improve the performance of the code ensure that you also >>> >> set -DNDEBUG so that debugging code is not executed. >>> >> """ >>> >> >>> >> I guess we should change that text to note these do not work on >>> Windows. >>> >> >>> >> I think you can supply extra Fortran flags with the `config_gc` option >>> >> to `setup.py`, but I don't think the others have any effect on >>> >> Windows. >>> >> >>> >> I was also confused by these lines in `azure-steps-windows.yml`: >>> >> >>> >> """ >>> >> $env:CFLAGS = "-m32" >>> >> $env:LDFLAGS = "-m32" >>> >> """ >>> >> >>> >> I assume these don't actually have any effect. >>> > >>> > >>> > No, the above all seem wrong (unless there has been a major regression >>> recently), and the `-m32` flags work and are necessary. CFLAGS, FFLAGS, >>> CXXFLAGS and LDFLAGS can be used, and append compile flags. If you want to >>> remove flags that are already hardcoded in numpy.distutils, then that's a >>> different story - you need to edit the numpy.distutils source code then. >>> > >>> > I just stumbled on a Windows problem with `$env:`, and the problem was >>> Windows CI is hopelessly weird. It matters for example if you do something >>> in a Powershell context (`ps |` in a .yml file) or outside of it. So it may >>> be something like that. >>> >>> Are you sure though - that you can append flags to the compile and >>> link step with CFLAGS and LDFLAGS? It didn't work for me locally. >>> And I can't see where they would have an effect, on Windows, in the >>> code. >>> >> >> Yes I'm sure, and it's even one of the few things in numpy.distutils that >> are tested: https://github.com/numpy/numpy/pull/14250. Maybe see if you >> can make that test fail when setting env vars in a couple of different ways? >> > > Note that the C/C++ handling can come from plain `distutils`, but it > should work the same. Here's a PR that fixes the Windows wheel builds for > SciPy for a Pythran issue by using CXXFLAGS: > https://github.com/MacPython/scipy-wheels/pull/122. > > The CPython bug tracker is full of noise about this kind of env var > handling, nothing conclusive as far as I know. It's of course possible that > it's broken under some circumstances (or in `setuptools.distutils`). There > are no tests for this in CPython it looks like. > A workaround may be to append the flags you need to CC instead. > > Cheers, > Ralf > > > ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Re: Setting C and linker flags for Windows build
On Wed, Jun 29, 2022 at 5:26 PM Matthew Brett wrote: > Hi, > > I am very sorry - I feel I should know this, or be able to work it > out, but is there a way of setting the flags to the C compiler and the > linker, for the Numpy build, on Windows? > > I'm trying to set the flags for a build with Windows mingw-w64 - but I > believe Numpy is ignoring $env:LDFLAGS, $env:CFLAGS and $env:OPT - and > I can't see any way of setting these options from the command line. > Am I missing something? > > Cheers, > > Matthew > I don't know how you are using env, but variables set that way are local to the shell in which they are set. In PS that is every separate shell invocation. Chuck ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Re: Setting C and linker flags for Windows build
Hi Chuck,
On Thu, Jun 30, 2022 at 2:13 PM Charles R Harris
wrote:
>
>
>
> On Wed, Jun 29, 2022 at 5:26 PM Matthew Brett wrote:
>>
>> Hi,
>>
>> I am very sorry - I feel I should know this, or be able to work it
>> out, but is there a way of setting the flags to the C compiler and the
>> linker, for the Numpy build, on Windows?
>>
>> I'm trying to set the flags for a build with Windows mingw-w64 - but I
>> believe Numpy is ignoring $env:LDFLAGS, $env:CFLAGS and $env:OPT - and
>> I can't see any way of setting these options from the command line.
>> Am I missing something?
>>
>> Cheers,
>>
>> Matthew
>
>
> I don't know how you are using env, but variables set that way are local to
> the shell in which they are set. In PS that is every separate shell
> invocation.
Maybe you are thinking of local PS variables? $env: variables do get
passed to subshells, like exported Bash variables.
"""
# show_env.py
import os
print('MY VAR', os.environ.get('MY_VAR'))
print('LOCAL VAR', os.environ.get('LOCAL_VAR'))
"""
Then:
"""
PS C:\tmp> $env:MY_VAR = 'foo'
PS C:\tmp> $LOCAL_VAR = 'bar'
PS C:\tmp> python .\show_env.py
"""
gives
"""
MY VAR foo
LOCAL VAR None
"""
Cheers,
Matthew
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com
[Numpy-discussion] Re: Setting C and linker flags for Windows build
On Thu, Jun 30, 2022 at 11:02 AM Matthew Brett wrote: > Hi Chuck, > > On Thu, Jun 30, 2022 at 2:13 PM Charles R Harris > wrote: > > > > > > > > On Wed, Jun 29, 2022 at 5:26 PM Matthew Brett > wrote: > >> > >> Hi, > >> > >> I am very sorry - I feel I should know this, or be able to work it > >> out, but is there a way of setting the flags to the C compiler and the > >> linker, for the Numpy build, on Windows? > >> > >> I'm trying to set the flags for a build with Windows mingw-w64 - but I > >> believe Numpy is ignoring $env:LDFLAGS, $env:CFLAGS and $env:OPT - and > >> I can't see any way of setting these options from the command line. > >> Am I missing something? > >> > >> Cheers, > >> > >> Matthew > > > > > > I don't know how you are using env, but variables set that way are local > to the shell in which they are set. In PS that is every separate shell > invocation. > > Maybe you are thinking of local PS variables? $env: variables do get > passed to subshells, like exported Bash variables. > Yes. But my experience was azure-pipelines where every shell invocation was a different subprocess (I think), so setting the environment variables in one shell using env: didn't show up in the others. But I don't know what your usage is. Chuck ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Re: Setting C and linker flags for Windows build
Hi, On Thu, Jun 30, 2022 at 6:30 PM Charles R Harris wrote: > > > > On Thu, Jun 30, 2022 at 11:02 AM Matthew Brett > wrote: >> >> Hi Chuck, >> >> On Thu, Jun 30, 2022 at 2:13 PM Charles R Harris >> wrote: >> > >> > >> > >> > On Wed, Jun 29, 2022 at 5:26 PM Matthew Brett >> > wrote: >> >> >> >> Hi, >> >> >> >> I am very sorry - I feel I should know this, or be able to work it >> >> out, but is there a way of setting the flags to the C compiler and the >> >> linker, for the Numpy build, on Windows? >> >> >> >> I'm trying to set the flags for a build with Windows mingw-w64 - but I >> >> believe Numpy is ignoring $env:LDFLAGS, $env:CFLAGS and $env:OPT - and >> >> I can't see any way of setting these options from the command line. >> >> Am I missing something? >> >> >> >> Cheers, >> >> >> >> Matthew >> > >> > >> > I don't know how you are using env, but variables set that way are local >> > to the shell in which they are set. In PS that is every separate shell >> > invocation. >> >> Maybe you are thinking of local PS variables? $env: variables do get >> passed to subshells, like exported Bash variables. > > > Yes. But my experience was azure-pipelines where every shell invocation was a > different subprocess (I think), so setting the environment variables in one > shell using env: didn't show up in the others. But I don't know what your > usage is. Yes, for Azure pipelines or Github workflows, you have to do special work to persist the environment variables between steps - but here I was just running on my local machine. It's on my local machine I find that CFLAGS and LDFLAGS don't seem to affect compilation / linking. Cheers, Matthew ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Re: Setting C and linker flags for Windows build
Hi, On Thu, Jun 30, 2022 at 10:00 AM Ralf Gommers wrote: > > > > On Thu, Jun 30, 2022 at 10:47 AM Matthew Brett > wrote: >> >> Hi, >> >> On Thu, Jun 30, 2022 at 9:40 AM Ralf Gommers wrote: >> > >> > >> > >> > On Thu, Jun 30, 2022 at 10:29 AM Matthew Brett >> > wrote: >> >> >> >> Hi, >> >> >> >> On Thu, Jun 30, 2022 at 12:42 AM Kevin Sheppard >> >> wrote: >> >> >> >> >> >> >> >> >> Hi, >> >> >> >> >> >> I am very sorry - I feel I should know this, or be able to work it >> >> >> out, but is there a way of setting the flags to the C compiler and the >> >> >> linker, for the Numpy build, on Windows? >> >> >> >> >> >> I'm trying to set the flags for a build with Windows mingw-w64 - but I >> >> >> believe Numpy is ignoring $env:LDFLAGS, $env:CFLAGS and $env:OPT - and >> >> >> I can't see any way of setting these options from the command line. >> >> >> Am I missing something? >> >> >> >> >> >> Cheers, >> >> >> >> >> >> Matthew >> >> >> >> >> Member address: kevin.k.shepp...@gmail.com >> >> > >> >> > >> >> > I think these are all hard coded and non-changable. This was the case >> >> > when I got NumPy building with clang-cl. >> >> >> >> That was my impression too, but I was delayed by the text at: >> >> >> >> https://numpy.org/doc/stable/user/building.html#supplying-additional-compiler-flags >> >> >> >> which says: >> >> >> >> """ >> >> Additional compiler flags can be supplied by setting the OPT, FOPT >> >> (for Fortran), and CC environment variables. When providing options >> >> that should improve the performance of the code ensure that you also >> >> set -DNDEBUG so that debugging code is not executed. >> >> """ >> >> >> >> I guess we should change that text to note these do not work on Windows. >> >> >> >> I think you can supply extra Fortran flags with the `config_gc` option >> >> to `setup.py`, but I don't think the others have any effect on >> >> Windows. >> >> >> >> I was also confused by these lines in `azure-steps-windows.yml`: >> >> >> >> """ >> >> $env:CFLAGS = "-m32" >> >> $env:LDFLAGS = "-m32" >> >> """ >> >> >> >> I assume these don't actually have any effect. >> > >> > >> > No, the above all seem wrong (unless there has been a major regression >> > recently), and the `-m32` flags work and are necessary. CFLAGS, FFLAGS, >> > CXXFLAGS and LDFLAGS can be used, and append compile flags. If you want to >> > remove flags that are already hardcoded in numpy.distutils, then that's a >> > different story - you need to edit the numpy.distutils source code then. >> > >> > I just stumbled on a Windows problem with `$env:`, and the problem was >> > Windows CI is hopelessly weird. It matters for example if you do something >> > in a Powershell context (`ps |` in a .yml file) or outside of it. So it >> > may be something like that. >> >> Are you sure though - that you can append flags to the compile and >> link step with CFLAGS and LDFLAGS? It didn't work for me locally. >> And I can't see where they would have an effect, on Windows, in the >> code. > > > Yes I'm sure, and it's even one of the few things in numpy.distutils that are > tested: https://github.com/numpy/numpy/pull/14250. Maybe see if you can make > that test fail when setting env vars in a couple of different ways? I've put up a minimal repo: https://github.com/matthew-brett/minnpd I ran > $env:CFLAGS = "-mbad-opt" > $env:LDFLAGS="-bad-option" > python setup.py build You can see from the `build.log` file, that the build doesn't pick up either flag. Cheers, Matthew ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Re: Setting C and linker flags for Windows build
Hi, On Thu, Jun 30, 2022 at 8:14 PM Matthew Brett wrote: > > Hi, > > On Thu, Jun 30, 2022 at 10:00 AM Ralf Gommers wrote: > > > > > > > > On Thu, Jun 30, 2022 at 10:47 AM Matthew Brett > > wrote: > >> > >> Hi, > >> > >> On Thu, Jun 30, 2022 at 9:40 AM Ralf Gommers > >> wrote: > >> > > >> > > >> > > >> > On Thu, Jun 30, 2022 at 10:29 AM Matthew Brett > >> > wrote: > >> >> > >> >> Hi, > >> >> > >> >> On Thu, Jun 30, 2022 at 12:42 AM Kevin Sheppard > >> >> wrote: > >> >> >> > >> >> >> > >> >> >> Hi, > >> >> >> > >> >> >> I am very sorry - I feel I should know this, or be able to work it > >> >> >> out, but is there a way of setting the flags to the C compiler and > >> >> >> the > >> >> >> linker, for the Numpy build, on Windows? > >> >> >> > >> >> >> I'm trying to set the flags for a build with Windows mingw-w64 - but > >> >> >> I > >> >> >> believe Numpy is ignoring $env:LDFLAGS, $env:CFLAGS and $env:OPT - > >> >> >> and > >> >> >> I can't see any way of setting these options from the command line. > >> >> >> Am I missing something? > >> >> >> > >> >> >> Cheers, > >> >> >> > >> >> >> Matthew > >> >> > >> >> >> Member address: kevin.k.shepp...@gmail.com > >> >> > > >> >> > > >> >> > I think these are all hard coded and non-changable. This was the > >> >> > case when I got NumPy building with clang-cl. > >> >> > >> >> That was my impression too, but I was delayed by the text at: > >> >> > >> >> https://numpy.org/doc/stable/user/building.html#supplying-additional-compiler-flags > >> >> > >> >> which says: > >> >> > >> >> """ > >> >> Additional compiler flags can be supplied by setting the OPT, FOPT > >> >> (for Fortran), and CC environment variables. When providing options > >> >> that should improve the performance of the code ensure that you also > >> >> set -DNDEBUG so that debugging code is not executed. > >> >> """ > >> >> > >> >> I guess we should change that text to note these do not work on Windows. > >> >> > >> >> I think you can supply extra Fortran flags with the `config_gc` option > >> >> to `setup.py`, but I don't think the others have any effect on > >> >> Windows. > >> >> > >> >> I was also confused by these lines in `azure-steps-windows.yml`: > >> >> > >> >> """ > >> >> $env:CFLAGS = "-m32" > >> >> $env:LDFLAGS = "-m32" > >> >> """ > >> >> > >> >> I assume these don't actually have any effect. > >> > > >> > > >> > No, the above all seem wrong (unless there has been a major regression > >> > recently), and the `-m32` flags work and are necessary. CFLAGS, FFLAGS, > >> > CXXFLAGS and LDFLAGS can be used, and append compile flags. If you want > >> > to remove flags that are already hardcoded in numpy.distutils, then > >> > that's a different story - you need to edit the numpy.distutils source > >> > code then. > >> > > >> > I just stumbled on a Windows problem with `$env:`, and the problem was > >> > Windows CI is hopelessly weird. It matters for example if you do > >> > something in a Powershell context (`ps |` in a .yml file) or outside of > >> > it. So it may be something like that. > >> > >> Are you sure though - that you can append flags to the compile and > >> link step with CFLAGS and LDFLAGS? It didn't work for me locally. > >> And I can't see where they would have an effect, on Windows, in the > >> code. > > > > > > Yes I'm sure, and it's even one of the few things in numpy.distutils that > > are tested: https://github.com/numpy/numpy/pull/14250. Maybe see if you can > > make that test fail when setting env vars in a couple of different ways? > > I've put up a minimal repo: > > https://github.com/matthew-brett/minnpd > > I ran > > > $env:CFLAGS = "-mbad-opt" > > $env:LDFLAGS="-bad-option" > > python setup.py build > > You can see from the `build.log` file, that the build doesn't pick up > either flag. Windows also ignores the CC environment variable: $ $env:CC = "cl -mbad-opt" $ python setup.py build (no error, no sign of option) Cheers, Matthew ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Re: Setting C and linker flags for Windows build
On Thu, Jun 30, 2022 at 9:25 PM Matthew Brett wrote: > Hi, > > On Thu, Jun 30, 2022 at 8:14 PM Matthew Brett > wrote: > > > >> >> > > >> >> I guess we should change that text to note these do not work on > Windows. > > >> >> > > >> >> I think you can supply extra Fortran flags with the `config_gc` > option > > >> >> to `setup.py`, but I don't think the others have any effect on > > >> >> Windows. > > >> >> > > >> >> I was also confused by these lines in `azure-steps-windows.yml`: > > >> >> > > >> >> """ > > >> >> $env:CFLAGS = "-m32" > > >> >> $env:LDFLAGS = "-m32" > > >> >> """ > > >> >> > > >> >> I assume these don't actually have any effect. > > >> > > > >> > > > >> > No, the above all seem wrong (unless there has been a major > regression recently), and the `-m32` flags work and are necessary. CFLAGS, > FFLAGS, CXXFLAGS and LDFLAGS can be used, and append compile flags. If you > want to remove flags that are already hardcoded in numpy.distutils, then > that's a different story - you need to edit the numpy.distutils source code > then. > > >> > > > >> > I just stumbled on a Windows problem with `$env:`, and the problem > was Windows CI is hopelessly weird. It matters for example if you do > something in a Powershell context (`ps |` in a .yml file) or outside of it. > So it may be something like that. > > >> > > >> Are you sure though - that you can append flags to the compile and > > >> link step with CFLAGS and LDFLAGS? It didn't work for me locally. > > >> And I can't see where they would have an effect, on Windows, in the > > >> code. > > > > > > > > > Yes I'm sure, and it's even one of the few things in numpy.distutils > that are tested: https://github.com/numpy/numpy/pull/14250. Maybe see if > you can make that test fail when setting env vars in a couple of different > ways? > > > > I've put up a minimal repo: > > > > https://github.com/matthew-brett/minnpd > > > > I ran > > > > > $env:CFLAGS = "-mbad-opt" > > > $env:LDFLAGS="-bad-option" > > > python setup.py build > > > > You can see from the `build.log` file, that the build doesn't pick up > > either flag. > > Windows also ignores the CC environment variable: > > $ $env:CC = "cl -mbad-opt" > $ python setup.py build > Looks like a Windows-specific bug in `distutils` then (assuming the `$env` bit is correct, I still don't trust it). I don't have a Windows machine available, but on Linux this errors out quickly: $ CC="gcc -mbad-opt" python setup.py build ... gcc: error: unrecognized command line option '-mbad-opt' And same for CFLAGS, it works as advertised. Probably not worth spending more time on, since this is all deprecated code. Cheers, Ralf ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Feature request: function to get minimum and maximum values simultaneously (as a tuple)
A function to get the minimum and maximum values of an array simultaneously could be very useful, from both a convenience and performance point of view. Especially when arrays get larger the performance benefit could be significant, and even more if the array doesn't fit in L2/L3 cache or even memory. There are many cases where not either the minimum or the maximum of an array is required, but both. Think of clipping an array, getting it's range, checking for outliers, normalizing, making a plot like a histogram, etc. This function could be called aminmax() for example, and also be called like ndarray.minmax(). It should return a tuple (min, max) with the minimum and maximum values of the array, identical to calling (ndarray.min(), ndarray.max()). With such a function, numpy.ptp() and the special cases of numpy.quantile(a, q=[0,1]) and numpy.percentile(a, q=[0,100]) could also potentially be speeded up, among others. Potentially argmin and argmax could get the same treatment, being called argminmax(). There is also a very extensive post on Stack Overflow (a bit old already) with discussion and benchmarks: https://stackoverflow.com/questions/12200580/numpy-function-for-simultaneous-max-and-min ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Re: Feature request: function to get minimum and maximum values simultaneously (as a tuple)
On Thu, Jun 30, 2022, at 22:23, Ewout ter Hoeven wrote: > A function to get the minimum and maximum values of an array > simultaneously could be very useful, from both a convenience and > performance point of view. Especially when arrays get larger the > performance benefit could be significant, and even more if the array > doesn't fit in L2/L3 cache or even memory. Hi, There's an open issue asking for this feature: https://github.com/numpy/numpy/issues/9836 András > > There are many cases where not either the minimum or the maximum of an > array is required, but both. Think of clipping an array, getting it's > range, checking for outliers, normalizing, making a plot like a > histogram, etc. > > This function could be called aminmax() for example, and also be called > like ndarray.minmax(). It should return a tuple (min, max) with the > minimum and maximum values of the array, identical to calling > (ndarray.min(), ndarray.max()). > > With such a function, numpy.ptp() and the special cases of > numpy.quantile(a, q=[0,1]) and numpy.percentile(a, q=[0,100]) could > also potentially be speeded up, among others. > > Potentially argmin and argmax could get the same treatment, being > called argminmax(). > > There is also a very extensive post on Stack Overflow (a bit old > already) with discussion and benchmarks: > https://stackoverflow.com/questions/12200580/numpy-function-for-simultaneous-max-and-min > ___ > NumPy-Discussion mailing list -- numpy-discussion@python.org > To unsubscribe send an email to numpy-discussion-le...@python.org > https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ > Member address: deak.and...@gmail.com ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Re: Feature request: function to get minimum and maximum values simultaneously (as a tuple)
On 6/30/22, Ewout ter Hoeven wrote: > A function to get the minimum and maximum values of an array simultaneously > could be very useful, from both a convenience and performance point of view. > Especially when arrays get larger the performance benefit could be > significant, and even more if the array doesn't fit in L2/L3 cache or even > memory. > > There are many cases where not either the minimum or the maximum of an array > is required, but both. Think of clipping an array, getting it's range, > checking for outliers, normalizing, making a plot like a histogram, etc. > > This function could be called aminmax() for example, and also be called like > ndarray.minmax(). It should return a tuple (min, max) with the minimum and > maximum values of the array, identical to calling (ndarray.min(), > ndarray.max()). > > With such a function, numpy.ptp() and the special cases of numpy.quantile(a, > q=[0,1]) and numpy.percentile(a, q=[0,100]) could also potentially be > speeded up, among others. > > Potentially argmin and argmax could get the same treatment, being called > argminmax(). > > There is also a very extensive post on Stack Overflow (a bit old already) > with discussion and benchmarks: > https://stackoverflow.com/questions/12200580/numpy-function-for-simultaneous-max-and-min FYI, I have a fairly simple gufunc implementation of `minmax` in ufunclab (https://github.com/WarrenWeckesser/ufunclab), along with `arg_minmax`, `min_argmin` and `max_argmax`. See README.md starting here: https://github.com/WarrenWeckesser/ufunclab#minmax For those familiar with C and gufunc implementation details, you can find the implementations in https://github.com/WarrenWeckesser/ufunclab/blob/main/src/minmax/minmax_gufunc.c.src. You'll see that, as far as gufuncs go, these are not very sophisticated. They do not include implementations for all the NumPy data types, and I haven't yet spent much time on optimization. Warren > ___ > NumPy-Discussion mailing list -- numpy-discussion@python.org > To unsubscribe send an email to numpy-discussion-le...@python.org > https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ > Member address: warren.weckes...@gmail.com > ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
