[RFC] Reliable compiler specification setting (at least include/lib dirs) through the process environment
Hello GCC and Clang devs!
Unlike the traditional approach of installing system libraries into one
central location like /usr/{lib,include}, the nix package manager [1]
installs each package into it's own prefix
(e.g. /nix/store/mn9kqag3d24v6q41x747zd7n5qnalch7-zlib-1.2.8-dev). Moreover,
each package is built in its own environment determined from its
explicitly listed dependencies, regardless of what else is installed on
the system. Because not all package build scripts properly respect
CFLAGS etc., we currently wrap the compiler [2] to respect custom
environment variables like NIX_CFLAGS_COMPILE, so during the build of a
package that depends on zlib and Xlib might have NIX_CFLAGS_COMPILE set
to "-isystem
/nix/store/bl0rz2xinsm9yslghd7n5vaba86zxknh-libX11-1.6.3-dev/include -isystem
/nix/store/mn9kqag3d24v6q41x747zd7n5qnalch7-zlib-1.2.8-dev/include".
Unfortunately, as you can see if you click through the link or look
through the git history, the wrapper is quite complex (frankly, hacky)
and has evolved mostly through trial and error. Moreover, it's known to
break things like response files [3] and is generally speaking a source
of frustration. I believe the situation would be much improved if gcc
and clang supported some form of "environment-specific" configuration,
either through environment variables or, if absolutely necessary,
command line flags that can be passed unconditionally (note, for
example, that we currently parse the cc command line to see if we're
going to do any linking before deciding to pass in linking-specific
options) and clearly have the semantics we want. Ideally we would be
able to specify something on the level of abstraction of "this directory
should be treated like you would normally treat /usr" and get
e.g. /include, /lib, frameworks on OS X, etc. handled properly.
Would patches aimed at achieving this be considered for inclusion
upstream? My current thought for a first step would be an environment
variable specifying a file with command line flags that are ignored by
the compiler driver in contexts where they are inapplicable or
overridden by other command line flags, but I'm definitely open to
guidance on how this should best be achieved from your perspective. I'm
happy to do the work needed to get this in place if there is interest,
please let me know!
Thanks,
Shea Levy
[1]: https://nixos.org/nix
[2]:
https://github.com/NixOS/nixpkgs/blob/8cbdd9d0c290e294a9d783c8868e738db05c9ce2/pkgs/build-support/cc-wrapper/cc-wrapper.sh
[3]:
https://github.com/NixOS/nixpkgs/commit/a421e7bd4a28c69bded8b17888325e31554f61a1
signature.asc
Description: PGP signature
Re: [RFC] Reliable compiler specification setting (at least include/lib dirs) through the process environment
Hey Ludo’,
Amazing, more than a decade of close working with these tools and I
never knew about C_INCLUDE_PATH et al! It looks like those will solve a
huge portion of the problem.
Will look at your gcc and clang patches as well, thank you!
~Shea
Ludovic Courtès writes:
> Hi Shea,
>
> Shea Levy skribis:
>
>> Unlike the traditional approach of installing system libraries into one
>> central location like /usr/{lib,include}, the nix package manager [1]
>> installs each package into it's own prefix
>> (e.g. /nix/store/mn9kqag3d24v6q41x747zd7n5qnalch7-zlib-1.2.8-dev). Moreover,
>> each package is built in its own environment determined from its
>> explicitly listed dependencies, regardless of what else is installed on
>> the system. Because not all package build scripts properly respect
>> CFLAGS etc., we currently wrap the compiler [2] to respect custom
>> environment variables like NIX_CFLAGS_COMPILE, so during the build of a
>> package that depends on zlib and Xlib might have NIX_CFLAGS_COMPILE set
>> to "-isystem
>> /nix/store/bl0rz2xinsm9yslghd7n5vaba86zxknh-libX11-1.6.3-dev/include
>> -isystem /nix/store/mn9kqag3d24v6q41x747zd7n5qnalch7-zlib-1.2.8-dev/include".
>>
>> Unfortunately, as you can see if you click through the link or look
>> through the git history, the wrapper is quite complex (frankly, hacky)
>
>> [2]:
>> https://github.com/NixOS/nixpkgs/blob/8cbdd9d0c290e294a9d783c8868e738db05c9ce2/pkgs/build-support/cc-wrapper/cc-wrapper.sh
>
> Guix avoids the compiler wrapper altogether like this:
>
> • We use C_INCLUDE_PATH, LIBRARY_PATH, and friends:
>
> <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/gcc.scm#n296>.
>
> • We have a simple linker wrapper aimed at adding -Wl,-rpath flags:
>
> <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/ld-wrapper.in#n42>.
> The comment in that file explains why the other options considered
> were unsuitable.
>
> • We modify the built-in “lib” spec of GCC to add the necessary -L and
> -rpath flags:
>
> <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/gcc.scm#n218>.
>
> • Likewise, we tell Clang where to find libc and friends:
>
> <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/patches/clang-libc-search-path.patch>
>
> <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/llvm.scm#n161>.
>
> This is not too intrusive and more robust than wrapping everything.
>
> I suppose GCC and Clang could facilitate this by providing configure
> options to augment the “lib” spec, specify the location of libc alone,
> or something along these lines.
>
> Thoughts?
>
> Ludo’.
signature.asc
Description: PGP signature
Re: [RFC] Reliable compiler specification setting (at least include/lib dirs) through the process environment
Hi Ludo’,
Your patches look good! My biggest concern is how the ld wrapper behaves
in the presence of response files. Have you tested that?
Thanks,
Shea
Ludovic Courtès writes:
> Hi Shea,
>
> Shea Levy skribis:
>
>> Unlike the traditional approach of installing system libraries into one
>> central location like /usr/{lib,include}, the nix package manager [1]
>> installs each package into it's own prefix
>> (e.g. /nix/store/mn9kqag3d24v6q41x747zd7n5qnalch7-zlib-1.2.8-dev). Moreover,
>> each package is built in its own environment determined from its
>> explicitly listed dependencies, regardless of what else is installed on
>> the system. Because not all package build scripts properly respect
>> CFLAGS etc., we currently wrap the compiler [2] to respect custom
>> environment variables like NIX_CFLAGS_COMPILE, so during the build of a
>> package that depends on zlib and Xlib might have NIX_CFLAGS_COMPILE set
>> to "-isystem
>> /nix/store/bl0rz2xinsm9yslghd7n5vaba86zxknh-libX11-1.6.3-dev/include
>> -isystem /nix/store/mn9kqag3d24v6q41x747zd7n5qnalch7-zlib-1.2.8-dev/include".
>>
>> Unfortunately, as you can see if you click through the link or look
>> through the git history, the wrapper is quite complex (frankly, hacky)
>
>> [2]:
>> https://github.com/NixOS/nixpkgs/blob/8cbdd9d0c290e294a9d783c8868e738db05c9ce2/pkgs/build-support/cc-wrapper/cc-wrapper.sh
>
> Guix avoids the compiler wrapper altogether like this:
>
> • We use C_INCLUDE_PATH, LIBRARY_PATH, and friends:
>
> <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/gcc.scm#n296>.
>
> • We have a simple linker wrapper aimed at adding -Wl,-rpath flags:
>
> <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/ld-wrapper.in#n42>.
> The comment in that file explains why the other options considered
> were unsuitable.
>
> • We modify the built-in “lib” spec of GCC to add the necessary -L and
> -rpath flags:
>
> <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/gcc.scm#n218>.
>
> • Likewise, we tell Clang where to find libc and friends:
>
> <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/patches/clang-libc-search-path.patch>
>
> <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/llvm.scm#n161>.
>
> This is not too intrusive and more robust than wrapping everything.
>
> I suppose GCC and Clang could facilitate this by providing configure
> options to augment the “lib” spec, specify the location of libc alone,
> or something along these lines.
>
> Thoughts?
>
> Ludo’.
signature.asc
Description: PGP signature
