Re: Scaling -fmacro-prefix-map= to thousands entries
On Wed, Oct 4, 2023 at 11:20 PM Sergei Trofimovich via Gcc
wrote:
>
> Hi gcc developers!
>
> Tl;DR:
>
> I would like to implement a scalable way to pass `-fmacro-prefix-map=`
> for `NixOS` distribution to avoid leaking build-time paths generated by
> `__FILE__` macros used by various libraries.
>
> I need some guidance what path to take to be acceptable for `gcc`
> upstream.
>
> I have a few possible solutions and wonder what I should try to upstream
> to GCC. The options I see:
>
> 1. Hardcode NixOS-specific way to mangle paths.
>
>Pros: simplest to implement, can be easily configured away if needed
>Cons: inflexible, `clang` might or might not accept the same hack
>
> 2. Extend `-fmacro-prefix-map=` (or add a new `-fmacro-prefix-map-file=`)
>to allow passing a file
>
>Pros: still not too hard to implement, generic enough to be used in
> other contexts.
>Cons: Will require client to construct the map file.
>
> 3. Have more flexible `-fmacro-prefix-map-regex=` option that allows
>patterns. Something like:
>
>
> -fmacro-prefix-map-regex=/nix/store/[a-z0-9]{32}-=/nix/store/-
>
> Pros: at least for NixOS one option will be enough to cover all
> packages as they all share above template.
> Cons: pulls some form of regex with it's can of worms including escape
> delimiters, might not be flexible enough for other use cases.
>
> 4. Something else?
>
> Which one(s) should I take to implement?
>
> More words:
>
> `NixOS` (and `nixpkgs` repository) install every software package into
> an individual directory with unique prefix. Some examples:
>
> /nix/store/y8wfrgk7br5rfz4221lfb9v8w3n0cnyd-glibc-2.37-8-dev
> /nix/store/rb3q4kcyfg77cmkiwywx2aqdd3x5ch93-libmpc-1.3.1
> /nix/store/8n240jfdmsb3lnc2qa2vb9dwk638j1lp-gmp-with-cxx-6.3.0-dev
> /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2
> ...
>
> It's a fundamental design decision to allow parallel package installs.
>
> From dependency tracking standpoint it's highly undesirable to have
> these absolute paths to be hardcoded into final executable binaries if
> they are not used at runtime.
>
> Example redundant path we would like not to have in final binaries:
>
> $ strings result/bin/nix | grep phjcmy025rd1ankw5y1b21xsdii83cyk
>
> /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2/include/nlohmann/json.hpp
>
> /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2/include/nlohmann/detail/output/serializer.hpp
>
> /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2/include/nlohmann/detail/conversions/to_chars.hpp
>
> /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2/include/nlohmann/detail/input/lexer.hpp
>
> /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2/include/nlohmann/detail/iterators/iter_impl.hpp
>
> /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2/include/nlohmann/detail/input/json_sax.hpp
>
> /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2/include/nlohmann/detail/iterators/iteration_proxy.hpp
>
> /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2/include/nlohmann/detail/input/parser.hpp
>
> Those paths are inserted via glibc's assert() uses of `__FILE__`
> directive and thus hardcode header file paths from various packages
> (like lttng-ust or nlohmann/json) into compiled binaries. Sometimes
> `__FILE__` usage is mire creating than assert().
>
> I would like to get rid of references to header files. I think
> `-fmacro-prefix-map=` are ideal for this particular use case.
>
> The prototype that creates equivalent of the following commands does
> work for smaller packages:
>
>
> -fmacro-prefix-map=/nix/store/y8wfrgk7br5rfz4221lfb9v8w3n0cnyd-glibc-2.37-8-dev=/nix/store/-glibc-2.37-8-dev
>
> -fmacro-prefix-map=/nix/store/8n240jfdmsb3lnc2qa2vb9dwk638j1lp-gmp-with-cxx-6.3.0-dev=/nix/store/-gmp-with-cxx-6.3.0-dev
>
> -fmacro-prefix-map=/nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2=/nix/store/-nlohmann_json-3.11.2
> ...
>
> The above works for small amount of options (like, 100). But around 1000
> options we start hitting linux limits on the single environment variable
> or real-world packages like `qemu` with a ton of input depends.
>
> The command-line limitations are in various places:
> - `gcc` limitation of lifting all command line options into a single
> environment variable: https://gcc.gnu.org/PR111527
> - `linux` limitation of constraining single environ variable to a value
> way below than full available environment space:
> https://lkml.org/lkml/2023/9/24/381
>
> `linux` fix would buy us 50x more budged (A Lot) but it will not help
> much other operating systems like `Darwin` where absolute environment
> limit is a lot lower than `linux`.
Re: Scaling -fmacro-prefix-map= to thousands entries
On Wed, Oct 04, 2023 at 22:19:32 +0100, Sergei Trofimovich via Gcc wrote: > The prototype that creates equivalent of the following commands does > work for smaller packages: > > > -fmacro-prefix-map=/nix/store/y8wfrgk7br5rfz4221lfb9v8w3n0cnyd-glibc-2.37-8-dev=/nix/store/-glibc-2.37-8-dev > > -fmacro-prefix-map=/nix/store/8n240jfdmsb3lnc2qa2vb9dwk638j1lp-gmp-with-cxx-6.3.0-dev=/nix/store/-gmp-with-cxx-6.3.0-dev > > -fmacro-prefix-map=/nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2=/nix/store/-nlohmann_json-3.11.2 > ... > > The above works for small amount of options (like, 100). But around 1000 > options we start hitting linux limits on the single environment variable > or real-world packages like `qemu` with a ton of input depends. Are you trying to pass this through via `CFLAGS` and friends? > The command-line limitations are in various places: > - `gcc` limitation of lifting all command line options into a single > environment variable: https://gcc.gnu.org/PR111527 > - `linux` limitation of constraining single environ variable to a value > way below than full available environment space: > https://lkml.org/lkml/2023/9/24/381 > > `linux` fix would buy us 50x more budged (A Lot) but it will not help > much other operating systems like `Darwin` where absolute environment > limit is a lot lower than `linux`. > > I already implemented [1.] in https://github.com/NixOS/nixpkgs/pull/255192 > (also attached `mangle-NIX_STORE-in-__FILE__.patch` 3.5K patch against > `master` as a proof of concept). > > What would be the best way to scale up `-fmacro-prefix-map=` up to NixOS > needs for `gcc`? I would like to implement something sensible I could > upstream. How about `CFLAGS=@macro_prefix_map.args` and writing that file in the same codepath where you generate the flags today. It'll work with just about every compiler and tools like `ccache` will understand that it is an input that affects the build and properly take the file's contents into account. --Ben
Re: Scaling -fmacro-prefix-map= to thousands entries
On Thu, Oct 05, 2023 at 09:19:15AM +0200, Richard Biener wrote:
> On Wed, Oct 4, 2023 at 11:20 PM Sergei Trofimovich via Gcc
> wrote:
> >
> > Hi gcc developers!
> >
> > Tl;DR:
> >
> > I would like to implement a scalable way to pass `-fmacro-prefix-map=`
> > for `NixOS` distribution to avoid leaking build-time paths generated by
> > `__FILE__` macros used by various libraries.
> >
> > I need some guidance what path to take to be acceptable for `gcc`
> > upstream.
> >
> > I have a few possible solutions and wonder what I should try to upstream
> > to GCC. The options I see:
> >
> > 1. Hardcode NixOS-specific way to mangle paths.
> >
> >Pros: simplest to implement, can be easily configured away if needed
> >Cons: inflexible, `clang` might or might not accept the same hack
> >
> > 2. Extend `-fmacro-prefix-map=` (or add a new `-fmacro-prefix-map-file=`)
> >to allow passing a file
> >
> >Pros: still not too hard to implement, generic enough to be used in
> > other contexts.
> >Cons: Will require client to construct the map file.
> >
> > 3. Have more flexible `-fmacro-prefix-map-regex=` option that allows
> >patterns. Something like:
> >
> >
> > -fmacro-prefix-map-regex=/nix/store/[a-z0-9]{32}-=/nix/store/-
> >
> > Pros: at least for NixOS one option will be enough to cover all
> > packages as they all share above template.
> > Cons: pulls some form of regex with it's can of worms including escape
> > delimiters, might not be flexible enough for other use cases.
> >
> > 4. Something else?
> >
> > Which one(s) should I take to implement?
> >
> > More words:
> >
> > `NixOS` (and `nixpkgs` repository) install every software package into
> > an individual directory with unique prefix. Some examples:
> >
> > /nix/store/y8wfrgk7br5rfz4221lfb9v8w3n0cnyd-glibc-2.37-8-dev
> > /nix/store/rb3q4kcyfg77cmkiwywx2aqdd3x5ch93-libmpc-1.3.1
> > /nix/store/8n240jfdmsb3lnc2qa2vb9dwk638j1lp-gmp-with-cxx-6.3.0-dev
> > /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2
> > ...
> >
> > It's a fundamental design decision to allow parallel package installs.
> >
> > From dependency tracking standpoint it's highly undesirable to have
> > these absolute paths to be hardcoded into final executable binaries if
> > they are not used at runtime.
> >
> > Example redundant path we would like not to have in final binaries:
> >
> > $ strings result/bin/nix | grep phjcmy025rd1ankw5y1b21xsdii83cyk
> >
> > /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2/include/nlohmann/json.hpp
> >
> > /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2/include/nlohmann/detail/output/serializer.hpp
> >
> > /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2/include/nlohmann/detail/conversions/to_chars.hpp
> >
> > /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2/include/nlohmann/detail/input/lexer.hpp
> >
> > /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2/include/nlohmann/detail/iterators/iter_impl.hpp
> >
> > /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2/include/nlohmann/detail/input/json_sax.hpp
> >
> > /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2/include/nlohmann/detail/iterators/iteration_proxy.hpp
> >
> > /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2/include/nlohmann/detail/input/parser.hpp
> >
> > Those paths are inserted via glibc's assert() uses of `__FILE__`
> > directive and thus hardcode header file paths from various packages
> > (like lttng-ust or nlohmann/json) into compiled binaries. Sometimes
> > `__FILE__` usage is mire creating than assert().
> >
> > I would like to get rid of references to header files. I think
> > `-fmacro-prefix-map=` are ideal for this particular use case.
> >
> > The prototype that creates equivalent of the following commands does
> > work for smaller packages:
> >
> >
> > -fmacro-prefix-map=/nix/store/y8wfrgk7br5rfz4221lfb9v8w3n0cnyd-glibc-2.37-8-dev=/nix/store/-glibc-2.37-8-dev
> >
> > -fmacro-prefix-map=/nix/store/8n240jfdmsb3lnc2qa2vb9dwk638j1lp-gmp-with-cxx-6.3.0-dev=/nix/store/-gmp-with-cxx-6.3.0-dev
> >
> > -fmacro-prefix-map=/nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2=/nix/store/-nlohmann_json-3.11.2
> > ...
> >
> > The above works for small amount of options (like, 100). But around 1000
> > options we start hitting linux limits on the single environment variable
> > or real-world packages like `qemu` with a ton of input depends.
> >
> > The command-line limitations are in various places:
> > - `gcc` limitation of lifting all command line options into a single
> > environment variable: https://gcc.gnu.org/PR111527
> > - `linux` limitation of constraining single environ variable to a v
Re: Scaling -fmacro-prefix-map= to thousands entries
On Thu, Oct 05, 2023 at 07:20:35AM -0400, Ben Boeckel wrote:
> On Wed, Oct 04, 2023 at 22:19:32 +0100, Sergei Trofimovich via Gcc wrote:
> > The prototype that creates equivalent of the following commands does
> > work for smaller packages:
> >
> >
> > -fmacro-prefix-map=/nix/store/y8wfrgk7br5rfz4221lfb9v8w3n0cnyd-glibc-2.37-8-dev=/nix/store/-glibc-2.37-8-dev
> >
> > -fmacro-prefix-map=/nix/store/8n240jfdmsb3lnc2qa2vb9dwk638j1lp-gmp-with-cxx-6.3.0-dev=/nix/store/-gmp-with-cxx-6.3.0-dev
> >
> > -fmacro-prefix-map=/nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2=/nix/store/-nlohmann_json-3.11.2
> > ...
> >
> > The above works for small amount of options (like, 100). But around 1000
> > options we start hitting linux limits on the single environment variable
> > or real-world packages like `qemu` with a ton of input depends.
>
> Are you trying to pass this through via `CFLAGS` and friends?
Roughly via `CFLAGS`. `nixpkgs` uses it's own private
`NIX_CFLAGS_COMPILE` variable which gets extended in `gcc-wrapper` shel
wrapper as expcit list of arguments to real `gcc-binary`. It's almost
like `CFLAGS` but is expected to be transaparent to most build systems.
> > The command-line limitations are in various places:
> > - `gcc` limitation of lifting all command line options into a single
> > environment variable: https://gcc.gnu.org/PR111527
> > - `linux` limitation of constraining single environ variable to a value
> > way below than full available environment space:
> > https://lkml.org/lkml/2023/9/24/381
> >
> > `linux` fix would buy us 50x more budged (A Lot) but it will not help
> > much other operating systems like `Darwin` where absolute environment
> > limit is a lot lower than `linux`.
> >
> > I already implemented [1.] in https://github.com/NixOS/nixpkgs/pull/255192
> > (also attached `mangle-NIX_STORE-in-__FILE__.patch` 3.5K patch against
> > `master` as a proof of concept).
> >
> > What would be the best way to scale up `-fmacro-prefix-map=` up to NixOS
> > needs for `gcc`? I would like to implement something sensible I could
> > upstream.
>
> How about `CFLAGS=@macro_prefix_map.args` and writing that file in the
> same codepath where you generate the flags today. It'll work with just
> about every compiler and tools like `ccache` will understand that it is
> an input that affects the build and properly take the file's contents
> into account.
That was my initial attempt. I'll duplicate my response from
https://gcc.gnu.org/pipermail/gcc/2023-October/242639.html here as is:
"""
Yeah, in theory response files would extend the limit. In practice `gcc`
always extends response files internally into a single
`COLLECT_GCC_OPTIONS` option and hits the environment variable limit
very early:
https://gcc.gnu.org/PR111527
Example reproducer:
$ for i in `seq 1 1000`; do printf -- "-fmacro-prefix-map=%0*d=%0*d\n" 200
1 200 2; done > a.rsp
$ touch a.c; gcc @a.rsp -c a.c
gcc: fatal error: cannot execute 'cc1': execv: Argument list too long
compilation terminated.
And if you want to look at the gory details:
$ strace -f -etrace=execve -s 100 -v -v -v gcc @a.rsp -c a.c
...
[pid78] execve("cc1", ["cc1", "-quiet", "a.c", "-quiet", "-dumpbase",
"a.c", "-dumpbase-ext", ".c", "-mtune=generic", "-march=x86-64",
"-fmacro-prefix-map=...=...",
"-fmacro-prefix-map=...=...",
...],
[...,
"COLLECT_GCC=gcc",
"COLLECT_GCC_OPTIONS='-fmacro-prefix-map=...=...'
'-fmacro-prefix-map=...=...' ... '-c' '-mtune=generic' '-march=x86-64'"]) = -1
E2BIG (Argument list too long)
Note how `gcc` not only expands response file into an argument list
(that is not too bad) but also duplicates the whole list as a single
`COLLECT_GCC_OPTIONS=...` environment variable with added quoting on
top.
Would be nice if `gcc` just passed response files around as is :)
"""
--
Sergei
Re: Scaling -fmacro-prefix-map= to thousands entries
On Thu, Oct 5, 2023 at 1:59 PM Sergei Trofimovich wrote:
>
> On Thu, Oct 05, 2023 at 09:19:15AM +0200, Richard Biener wrote:
> > On Wed, Oct 4, 2023 at 11:20 PM Sergei Trofimovich via Gcc
> > wrote:
> > >
> > > Hi gcc developers!
> > >
> > > Tl;DR:
> > >
> > > I would like to implement a scalable way to pass `-fmacro-prefix-map=`
> > > for `NixOS` distribution to avoid leaking build-time paths generated by
> > > `__FILE__` macros used by various libraries.
> > >
> > > I need some guidance what path to take to be acceptable for `gcc`
> > > upstream.
> > >
> > > I have a few possible solutions and wonder what I should try to upstream
> > > to GCC. The options I see:
> > >
> > > 1. Hardcode NixOS-specific way to mangle paths.
> > >
> > >Pros: simplest to implement, can be easily configured away if needed
> > >Cons: inflexible, `clang` might or might not accept the same hack
> > >
> > > 2. Extend `-fmacro-prefix-map=` (or add a new `-fmacro-prefix-map-file=`)
> > >to allow passing a file
> > >
> > >Pros: still not too hard to implement, generic enough to be used in
> > > other contexts.
> > >Cons: Will require client to construct the map file.
> > >
> > > 3. Have more flexible `-fmacro-prefix-map-regex=` option that allows
> > >patterns. Something like:
> > >
> > >
> > > -fmacro-prefix-map-regex=/nix/store/[a-z0-9]{32}-=/nix/store/-
> > >
> > > Pros: at least for NixOS one option will be enough to cover all
> > > packages as they all share above template.
> > > Cons: pulls some form of regex with it's can of worms including escape
> > > delimiters, might not be flexible enough for other use cases.
> > >
> > > 4. Something else?
> > >
> > > Which one(s) should I take to implement?
> > >
> > > More words:
> > >
> > > `NixOS` (and `nixpkgs` repository) install every software package into
> > > an individual directory with unique prefix. Some examples:
> > >
> > > /nix/store/y8wfrgk7br5rfz4221lfb9v8w3n0cnyd-glibc-2.37-8-dev
> > > /nix/store/rb3q4kcyfg77cmkiwywx2aqdd3x5ch93-libmpc-1.3.1
> > > /nix/store/8n240jfdmsb3lnc2qa2vb9dwk638j1lp-gmp-with-cxx-6.3.0-dev
> > > /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2
> > > ...
> > >
> > > It's a fundamental design decision to allow parallel package installs.
> > >
> > > From dependency tracking standpoint it's highly undesirable to have
> > > these absolute paths to be hardcoded into final executable binaries if
> > > they are not used at runtime.
> > >
> > > Example redundant path we would like not to have in final binaries:
> > >
> > > $ strings result/bin/nix | grep phjcmy025rd1ankw5y1b21xsdii83cyk
> > >
> > > /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2/include/nlohmann/json.hpp
> > >
> > > /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2/include/nlohmann/detail/output/serializer.hpp
> > >
> > > /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2/include/nlohmann/detail/conversions/to_chars.hpp
> > >
> > > /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2/include/nlohmann/detail/input/lexer.hpp
> > >
> > > /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2/include/nlohmann/detail/iterators/iter_impl.hpp
> > >
> > > /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2/include/nlohmann/detail/input/json_sax.hpp
> > >
> > > /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2/include/nlohmann/detail/iterators/iteration_proxy.hpp
> > >
> > > /nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2/include/nlohmann/detail/input/parser.hpp
> > >
> > > Those paths are inserted via glibc's assert() uses of `__FILE__`
> > > directive and thus hardcode header file paths from various packages
> > > (like lttng-ust or nlohmann/json) into compiled binaries. Sometimes
> > > `__FILE__` usage is mire creating than assert().
> > >
> > > I would like to get rid of references to header files. I think
> > > `-fmacro-prefix-map=` are ideal for this particular use case.
> > >
> > > The prototype that creates equivalent of the following commands does
> > > work for smaller packages:
> > >
> > >
> > > -fmacro-prefix-map=/nix/store/y8wfrgk7br5rfz4221lfb9v8w3n0cnyd-glibc-2.37-8-dev=/nix/store/-glibc-2.37-8-dev
> > >
> > > -fmacro-prefix-map=/nix/store/8n240jfdmsb3lnc2qa2vb9dwk638j1lp-gmp-with-cxx-6.3.0-dev=/nix/store/-gmp-with-cxx-6.3.0-dev
> > >
> > > -fmacro-prefix-map=/nix/store/phjcmy025rd1ankw5y1b21xsdii83cyk-nlohmann_json-3.11.2=/nix/store/-nlohmann_json-3.11.2
> > > ...
> > >
> > > The above works for small amount of options (like, 100). But around 1000
> > > options we start hitting linux limits on the single environment variable
> > > or real-world packages like `qemu` with a ton of input de
Re: Scaling -fmacro-prefix-map= to thousands entries
Hi,
Sergei Trofimovich via Gcc writes:
> Sounds good. Do you have any preference over specific syntax? My
> suggestions would be:
>
> 1. `-fmacro-prefix-map=file-name`: if `file-name` there is not in `key=val`
>format then treat it as file
> 2. `-fmacro-prefix-map=@file-name`: use @ as a signal to use file
> 3. `fmacro-prefix-map-file=file-name`: use a new option
>
>> Btw, I thought we have response files to deal with command-line limits,
>> why doesn't that work here? I see the driver expands response files
>> but IIRC it also builds those when the command-line gets too large
>> and uses it for the environment and the cc1 invocation? If it doesn't
>> do the latter why not fix it that way?
>
> Yeah, in theory response files would extend the limit. In practice `gcc`
> always extends response files internally into a single
> `COLLECT_GCC_OPTIONS` option and hits the environment variable limit
> very early:
>
> https://gcc.gnu.org/PR111527
Doesn't it make more sense to fix this? The issue is more general than
just this option (even if manifesting like so today).
Can the COLLECT_GCC_OPTIONS consumers deal with argfiles?
> Example reproducer:
>
> $ for i in `seq 1 1000`; do printf -- "-fmacro-prefix-map=%0*d=%0*d\n"
> 200 1 200 2; done > a.rsp
> $ touch a.c; gcc @a.rsp -c a.c
> gcc: fatal error: cannot execute 'cc1': execv: Argument list too long
> compilation terminated.
>
> And if you want to look at the gory details:
>
> $ strace -f -etrace=execve -s 100 -v -v -v gcc @a.rsp -c a.c
> ...
> [pid78] execve("cc1", ["cc1", "-quiet", "a.c", "-quiet", "-dumpbase",
> "a.c", "-dumpbase-ext", ".c", "-mtune=generic", "-march=x86-64",
> "-fmacro-prefix-map=...=...",
> "-fmacro-prefix-map=...=...",
> ...],
> [...,
> "COLLECT_GCC=gcc",
> "COLLECT_GCC_OPTIONS='-fmacro-prefix-map=...=...'
> '-fmacro-prefix-map=...=...' ... '-c' '-mtune=generic' '-march=x86-64'"]) =
> -1 E2BIG (Argument list too long)
>
> Note how `gcc` not only expands response file into an argument list
> (that is not too bad) but also duplicates the whole list as a single
> `COLLECT_GCC_OPTIONS=...` environment variable with added quoting on
> top.
>
> Would be nice if `gcc` just passed response files around as is :)
--
Arsen Arsenović
signature.asc
Description: PGP signature
Re: Complex numbers support: discussions summary
On 9/26/23 20:40, Toon Moene wrote: On 9/26/23 09:30, Richard Biener via Gcc wrote: On Mon, Sep 25, 2023 at 5:17 PM Sylvain Noiry via Gcc wrote: As I said at the end of the presentation, we have written a paper which explains our implementation in details. You can find it on the wiki page of the Cauldron (https://gcc.gnu.org/wiki/cauldron2023talks?action=AttachFile&do=view&target=Exposing+Complex+Numbers+to+Target+Back-ends+%28paper%29.pdf). Thanks for the detailed presentation at the Cauldron. My personal summary is that I'm less convinced delaying lowering is the way to go. Thanks Sylvain for the quick summary of the discussion - it helps a great deal now that the discussion is still fresh in our memory. I found time today to run some tests. First of all, the result of the gcc test harness as applied to the top of the complex/kvx branch in the https://github.com/kalray/gcc repository: https://gcc.gnu.org/pipermail/gcc-testresults/2023-October/797627.html I think there are several complex failures here that are not in "standard" 12.2 release (for x86_64-linux-gnu). I also compiled all of lapack-3.11.0 with that compiler and obtained the same results as with gcc/gfortran 13.2: --> LAPACK TESTING SUMMARY <-- Processing LAPACK Testing output found in the TESTING directory SUMMARY nb test run numerical error other error === = REAL1327023 0 (0.000%)0 (0.000%) DOUBLE PRECISION1300917 6 (0.000%)0 (0.000%) COMPLEX 786775 0 (0.000%)0 (0.000%) COMPLEX16 787842 0 (0.000%)0 (0.000%) --> ALL PRECISIONS 4202557 6 (0.000%)0 (0.000%) Kind regards, -- Toon Moene - e-mail: t...@moene.org - phone: +31 346 214290 Saturnushof 14, 3738 XG Maartensdijk, The Netherlands
Re: Scaling -fmacro-prefix-map= to thousands entries
On Thu, Oct 05, 2023 at 12:59:21 +0100, Sergei Trofimovich via Gcc wrote: > Sounds good. Do you have any preference over specific syntax? My > suggestions would be: > > 1. `-fmacro-prefix-map=file-name`: if `file-name` there is not in `key=val` >format then treat it as file > 2. `-fmacro-prefix-map=@file-name`: use @ as a signal to use file > 3. `fmacro-prefix-map-file=file-name`: use a new option Whatever is picked, please let `ccache` and `sccache` know so that they can include this argument's contents in their hashes. `distcc` and other distributed build tools need to know to send this file to wherever compilation is happening as well (though they may actually not care as they may send around preprocessed source?). Thanks, --Ben
Re: LRA for avr: Maintain live range info for pseudos assigned to FP?
On 9/7/23 07:21, senthilkumar.selva...@microchip.com wrote:
Hi,
One more execution failure for the avr target, this time from
gcc.c-torture/execute/bitfld-3.c.
Steps to reproduce
Enable LRA in avr.cc by removing TARGET_LRA_P hook, build with
$ make all-host && make install-host
and then
$ avr-gcc gcc/testsuite/gcc.c-torture/execute/bitfld-3.c -S -Os -mmcu=avr51
-fdump-rtl-all
When lra_update_fp2sp_elimination runs and pseudos assigned to the
FP have to be spilled to stack slots, they sometimes end up in a
slot that already has a reg with an overlapping live range. This is
because lra_reg_info[regno].live_ranges is NULL for such spilled
pseudos, and therefore when assign_stack_slot_num_and_sort_pseduos
checks if lra_intersected_live_ranges_p, it always returns false.
In the below reload dump, all the pseudos assigned to FP get
allocated to slot 0. The live ranges for some of them (r1241 for
e.g.) conflicts with r603 that was originally assigned to slot 0,
but they still end up in the same slot, causing the execution failure.
Sorry for the delay with the answer, Senthil. Avr is unusual target and
needs some changes in LRA but the changes improves LRA portability. So
thank you for your work on porting LRA to AVR.
The patch is ok for me. The only comment is that making calculation of
the set only once would be nice. Live range calculation in LRA can take
a lot of time, code of update_pseudo_point is hot and the worst the set
will be really used rarely but it is calculated every time.
You can commit the current patch and I'll do it by myself or, if you
want, you can modify the patch by yourself and submit it for review and
I'll review as soon as possible. Either way works for me.
diff --git a/gcc/lra-lives.cc b/gcc/lra-lives.cc
index f60e564da82..e4289f13979 100644
--- a/gcc/lra-lives.cc
+++ b/gcc/lra-lives.cc
@@ -250,7 +250,17 @@ update_pseudo_point (int regno, int point, enum point_type
type)
if (HARD_REGISTER_NUM_P (regno))
return;
- if (complete_info_p || lra_get_regno_hard_regno (regno) < 0)
+ /* Pseudos assigned to the FP register could potentially get spilled
+ to stack slots when lra_update_fp2sp_elimination runs, so keep
+ their live range info up to date, even if they aren't in memory
+ right now. */
+ int hard_regno = lra_get_regno_hard_regno (regno);
+ HARD_REG_SET set;
+ CLEAR_HARD_REG_SET(set);
+ add_to_hard_reg_set (&set, Pmode, HARD_FRAME_POINTER_REGNUM);
+
+ if (complete_info_p || hard_regno < 0
+ || overlaps_hard_reg_set_p (set, PSEUDO_REGNO_MODE (regno), hard_regno))
{
if (type == DEF_POINT)
{
gcc-11-20231005 is now available
Snapshot gcc-11-20231005 is now available on https://gcc.gnu.org/pub/gcc/snapshots/11-20231005/ and on various mirrors, see http://gcc.gnu.org/mirrors.html for details. This snapshot has been generated from the GCC 11 git branch with the following options: git://gcc.gnu.org/git/gcc.git branch releases/gcc-11 revision 001bbe581a54f4b0707f14c74425eae4481ffbf5 You'll find: gcc-11-20231005.tar.xz Complete GCC SHA256=c59884c314aef13c85775c30e244ff555c68d926662e17670e55e8c6f74d0ebf SHA1=c5d34b03a8180caf8a3d18a78fbee799cd4ad0bc Diffs from 11-20230928 are available in the diffs/ subdirectory. When a particular snapshot is ready for public consumption the LATEST-11 link is updated and a message is sent to the gcc list. Please do not use a snapshot before it has been announced that way.
Re: Scaling -fmacro-prefix-map= to thousands entries
On Thu, Oct 5, 2023 at 2:17 PM Arsen Arsenović wrote:
>
> Hi,
>
> Sergei Trofimovich via Gcc writes:
>
> > Sounds good. Do you have any preference over specific syntax? My
> > suggestions would be:
> >
> > 1. `-fmacro-prefix-map=file-name`: if `file-name` there is not in `key=val`
> >format then treat it as file
> > 2. `-fmacro-prefix-map=@file-name`: use @ as a signal to use file
> > 3. `fmacro-prefix-map-file=file-name`: use a new option
> >
> >> Btw, I thought we have response files to deal with command-line limits,
> >> why doesn't that work here? I see the driver expands response files
> >> but IIRC it also builds those when the command-line gets too large
> >> and uses it for the environment and the cc1 invocation? If it doesn't
> >> do the latter why not fix it that way?
> >
> > Yeah, in theory response files would extend the limit. In practice `gcc`
> > always extends response files internally into a single
> > `COLLECT_GCC_OPTIONS` option and hits the environment variable limit
> > very early:
> >
> > https://gcc.gnu.org/PR111527
>
> Doesn't it make more sense to fix this? The issue is more general than
> just this option (even if manifesting like so today).
>
> Can the COLLECT_GCC_OPTIONS consumers deal with argfiles?
No. The traditional consumer is collect2, nowadays the main consumer
is lto-wrapper and the lto-plugin. There's
parse_options_from_collect_gcc_options but
for example collect2 does its own parsing. There might be other tools out
in the wild interpreting COLLECT_GCC_OPTIONS.
We might make life of the tools easy if either all of COLLECT_GCC_OPTIONS
is fully expanded or it is a single @file "argument" (but with
otherwise identical,
quoted content).
Richard.
> > Example reproducer:
> >
> > $ for i in `seq 1 1000`; do printf -- "-fmacro-prefix-map=%0*d=%0*d\n"
> > 200 1 200 2; done > a.rsp
> > $ touch a.c; gcc @a.rsp -c a.c
> > gcc: fatal error: cannot execute 'cc1': execv: Argument list too long
> > compilation terminated.
> >
> > And if you want to look at the gory details:
> >
> > $ strace -f -etrace=execve -s 100 -v -v -v gcc @a.rsp -c a.c
> > ...
> > [pid78] execve("cc1", ["cc1", "-quiet", "a.c", "-quiet",
> > "-dumpbase", "a.c", "-dumpbase-ext", ".c", "-mtune=generic",
> > "-march=x86-64",
> > "-fmacro-prefix-map=...=...",
> > "-fmacro-prefix-map=...=...",
> > ...],
> > [...,
> > "COLLECT_GCC=gcc",
> > "COLLECT_GCC_OPTIONS='-fmacro-prefix-map=...=...'
> > '-fmacro-prefix-map=...=...' ... '-c' '-mtune=generic' '-march=x86-64'"]) =
> > -1 E2BIG (Argument list too long)
> >
> > Note how `gcc` not only expands response file into an argument list
> > (that is not too bad) but also duplicates the whole list as a single
> > `COLLECT_GCC_OPTIONS=...` environment variable with added quoting on
> > top.
> >
> > Would be nice if `gcc` just passed response files around as is :)
>
>
> --
> Arsen Arsenović
