Making gcc -no-canonical-prefixes the default?
A quick question about -no-canonical-prefixes... By default, gcc calls realpath() on prefixes generated relative to argv[0] in the gcc driver. If gcc is held as a "symlink farm" the realpath() makes it fail (absent a lot of messy -B, -L, -isytem and so on). It complains about not finding cc1 or cc1plus in libexec. -no-canonical-prefixes turns off realpath() to make gcc work cleanly when stored this way. Does anyone know a reason why -no-canonical-prefixes could not become the gcc default? Are there gcc configurations that must have the realpath()? The flag is benign on normally laid out gcc installations. If it did become the default case, would adding a symmetrical -canonical-prefixes to turn realpath() back on be worthwhile? Thanks. -- Google UK Limited | Registered Office: Belgrave House, 76 Buckingham Palace Road, London SW1W 9TQ | Registered in England Number: 3977902
Re: Making gcc -no-canonical-prefixes the default?
On 31 January 2011 01:20, Gerald Pfeifer wrote: > On Fri, 28 Jan 2011, Ian Lance Taylor wrote: >> Some archealogy turned up this as the reason canonicalization was >> inserted: >> >> http://gcc.gnu.org/ml/gcc/2003-02/msg01121.html >> http://gcc.gnu.org/ml/gcc-patches/2003-02/msg01697.html >> >> Also relevant here is http://gcc.gnu.org/PR29931 . > > I am quite sure that the current behavior where one can have a symlink > point to an installation of GCC anywhere else (outside of any path even) > has been in place for quite a bit longer than 2003. Thanks for the notes and archaeology. My guess is that because it's easier, symlinking a single gcc binary is more common than symlinking a complete gcc tree. > On Sat, 29 Jan 2011, Dave Korn wrote: >> I think the case which is particularly common is the alternatives >> system, which has a chain of symlinks finally pointing to a real gcc. >> (That works just fine with the current default, AFAIK, although that >> may be only because the real gcc is in $PATH?) > > Nope, only the symlink(s) need to be in the path. And indeed this > usecase that you and also Joseph describe is how I've been using it > for many years. > > Given that this has een default behavior for so long (more than a > decade from what I can tell), I'd recommend not changing it. That would be as good a reason as any for not changing it. Thanks for the clarification. The patch in http://gcc.gnu.org/PR29931 uses... heuristics to detect if the gcc driver is a single symlink or part of a symlink farm (Florian Weimer's suggestion above). The patch has never been fully accepted, though; any known reason? I believe it will handle the two main use cases reliably. Combining it with -[no-]canonical-prefixes flags as overrides would cover all cases unambiguously. If heuristics aren't deemed reliable enough, another option is to set a default with a -no-canonical-prefixes-default configure flag. Less flexible because it's baked in to gcc, but simpler. On balance the PR29931 patch approach with -[no-]canonical-prefixes overrides seems to me like it will lead to least trouble in future. It also closes a long-standing gcc PR. Any dissent? Other options? Thanks. -- Google UK Limited | Registered Office: Belgrave House, 76 Buckingham Palace Road, London SW1W 9TQ | Registered in England Number: 3977902
Hints for backporting gcc 4.5 powerpc fix to gcc 4.4.3?
I'm currently trying to backport a small part of gcc 4.5 r151729 to
gcc 4.4.3. This revision fixes a problem in powerpc code generation
that leads to gcc not using lmw/stmw instructions in function prologue
and epilogues, where it could otherwise validly use them.
On the face of things, the central piece of r151729 I seem to want is just this:
Index: gcc/config/rs6000/rs6000.c
===
--- gcc/config/rs6000/rs6000.c (revision 151728)
+++ gcc/config/rs6000/rs6000.c (revision 151729)
@@ -18033,7 +18033,8 @@ static bool
no_global_regs_above (int first, bool gpr)
{
int i;
- for (i = first; i < gpr ? 32 : 64 ; i++)
+ int last = gpr ? 32 : 64;
+ for (i = first; i < last; i++)
if (global_regs[i])
return false;
return true;
Taking only that and leaving out all of the rest of r151729 lets me
build a powerpc gcc that does use lmw/stmw instructions in function
prologue and epilogues as hoped. Unfortunately it also has bad
codegen elsewhere. So it seems I need more than just this little
piece of r151729. Unfortunately, r151729 is a fairly large patch that
seems to do a number of jobs and which does not apply readily to gcc
4.4. At the moment it's not clear to me what other parts of it I
might need.
Can anyone here offer any hints or pointers on how to extract from the
r151729 diff just the few pieces needed to fix this single powerpc
codegen bug in gcc 4.4.3? Anyone recognize this issue and already
dealt with it in isolation?
Thanks.
--
Google UK Limited | Registered Office: Belgrave House, 76 Buckingham
Palace Road, London SW1W 9TQ | Registered in England Number: 3977902
Re: Hints for backporting gcc 4.5 powerpc fix to gcc 4.4.3?
On 22 March 2011 14:56, David Edelsohn wrote:
>
> On Tue, Mar 22, 2011 at 9:25 AM, Simon Baldwin wrote:
> > I'm currently trying to backport a small part of gcc 4.5 r151729 to
> > gcc 4.4.3. This revision fixes a problem in powerpc code generation
> > that leads to gcc not using lmw/stmw instructions in function prologue
> > and epilogues, where it could otherwise validly use them.
> >
> > On the face of things, the central piece of r151729 I seem to want is just
> > this:
> >
> > Index: gcc/config/rs6000/rs6000.c
> > ===
> > --- gcc/config/rs6000/rs6000.c (revision 151728)
> > +++ gcc/config/rs6000/rs6000.c (revision 151729)
> > @@ -18033,7 +18033,8 @@ static bool
> > no_global_regs_above (int first, bool gpr)
> > {
> > int i;
> > - for (i = first; i < gpr ? 32 : 64 ; i++)
> > + int last = gpr ? 32 : 64;
> > + for (i = first; i < last; i++)
> > if (global_regs[i])
> > return false;
> > return true;
> >
> > Taking only that and leaving out all of the rest of r151729 lets me
> > build a powerpc gcc that does use lmw/stmw instructions in function
> > prologue and epilogues as hoped. Unfortunately it also has bad
> > codegen elsewhere. So it seems I need more than just this little
> > piece of r151729. Unfortunately, r151729 is a fairly large patch that
> > seems to do a number of jobs and which does not apply readily to gcc
> > 4.4. At the moment it's not clear to me what other parts of it I
> > might need.
> >
> > Can anyone here offer any hints or pointers on how to extract from the
> > r151729 diff just the few pieces needed to fix this single powerpc
> > codegen bug in gcc 4.4.3? Anyone recognize this issue and already
> > dealt with it in isolation?
>
> The change to no_global_regs_above() is one of the key pieces, but
> that change exposed other latent bugs, as you have encountered. One
> needs the additional patches to the save/restore strategy routines and
> prologue/epilogue. This is why the entire patch was committed in one
> piece.
Thanks for the reply, David. I'll take another look and see if I can
abstract out just the required pieces. In practice, though, it looks
like it may be easier for me to just upgrade to gcc 4.5 or 4.6.
Certainly safer.
--
Google UK Limited | Registered Office: Belgrave House, 76 Buckingham
Palace Road, London SW1W 9TQ | Registered in England Number: 3977902
If you had a month to improve gcc build parallelization, where would you begin?
Suppose you had a month in which to reorganise gcc so that it builds its 3-stage bootstrap and runtime libraries in some massively parallel fashion, without hardware or resource constraints(*). How might you approach this? I'm looking for ideas on improving the build time of gcc itself. So far I have identified two problem areas: - Structural. In a 3-stage bootstrap, each stage depends on the output of the one before. No matter what does the actual compilation (make, jam, and so on), this constrains the options. - Mechanical. Configure scripts cause bottlenecks in the build process. Even if compilation is offloaded onto something like distcc, configures run locally and randomly throughout the complete build, rather than (say) all at once upfront. Source code compilation blocks until configure is completed. One way to improve the first might be to build the runtime libraries with stage2 and interleave with stage3 build on the assumption that comparison will pass. That probably won't produce an exciting speedup, likely modest at best. Improving the second seems trickier still. Pre-canned configure responses might help, but would be incredibly brittle. If even feasible, that is. Rewriting gcc's build to use something other than autotools is unlikely to win many friends, at least in the short term. Have there been attempts at this before? Any papers or presentations I'm not aware of that address the issue? Or maybe you've thought about this in the past and would be willing to share your ideas? Even if only as a warning to stay away... Thanks. (* Of course, in reality there are always resource constraints.) -- Google UK Limited | Registered Office: Belgrave House, 76 Buckingham Palace Road, London SW1W 9TQ | Registered in England Number: 3977902
Gcc testsuite, LD_LIBRARY_PATH, and a new libgcc_s that is incompatible with the host libc
I've recently encountered a gcc testsuite issue that is caused by the libgcc_s in the built compiler being incompatible with, but the same architecture as, the host's libc. I'm not sure what the right fix is, so I'm looking for ideas. I build gcc-4.7 for x86 on an x86 host. The bootstrap compiler is not the host compiler, and sysroot options link the built gcc-4.7 against an alternative libc, one that is newer than the host's libc. Within gcc, to support bootstrap builds the top level makefile sets LD_LIBRARY_PATH to encompass ./gcc and ./prev-gcc, and passes that setting to both build and check targets. The build creates a new libgcc_s in ./gcc, and when 'make check' executes expect via runtest this new libgcc_s is visible to the expect binary. Expect is threaded, and calls pthread_exit, which calls pthread_cancel_init, which dlopen's libgcc_s. Because of LD_LIBRARY_PATH, the host libc finds the new libgcc_s from the gcc build. In my case they're not compatible. The new libgcc_s was built against a non-host libc that is newer than the host's libc and so dlopen fails to find a versioned symbol. libc aborts the expect process: libgcc_s.so.1 must be installed for pthread_cancel to work Aborted Several check targets refuse to run -- though bizarrely, not actually fail the testsuite(!) -- because expect invoked by 'runtest --version' aborts: /bin/sh: line 10: 21052 Aborted /bin/sh -c "$runtest --version" > /dev/null 2>&1 WARNING: could not find `runtest' Firstly, has anyone else encountered this or otherwise given it any thought? And secondly, any hints for practical fixes? Thx. -- Google UK Limited | Registered Office: Belgrave House, 76 Buckingham Palace Road, London SW1W 9TQ | Registered in England Number: 3977902
Reproducible gcc builds, gfortran, and -grecord-gcc-switches
The default setting of -grecord-gcc-switches recently changed from 0 to 1: 2012-04-25 Jakub Jelinek * common.opt (flag_debug_types_section): Default to 0. ... (dwarf_record_gcc_switches): Default to 1. Because of this, by default all objects in libraries built as part of gcc now have driver command line options stored in their debug_str section. For C and C++ that's not an issue. However, Fortran has a special undocumented extra option created by its specs, "-cpp=%g.f90". As a result all the object files in libgfortran.a now have a temporary filename baked into their debug_str section. This creates a problem for build and packaging systems that are fanatical about binary reproducibility and checksums. Temporary file names differ on each compilation, so that two different builds of libgfortran.a, and by extension all of gcc, will not be bit-identical. Anyone else encountered this? Bug or feature? There are several possible and relatively easy fixes -- any thoughts or feedback on which fixes might be better, or worse? Something for mainline or for only google-specific branches? -- Google UK Limited | Registered Office: Belgrave House, 76 Buckingham Palace Road, London SW1W 9TQ | Registered in England Number: 3977902
Re: Reproducible gcc builds, gfortran, and -grecord-gcc-switches
On 16 August 2012 09:38, Gerald Pfeifer wrote:
>
> On Wed, 15 Aug 2012, Simon Baldwin wrote:
> > This creates a problem for build and packaging systems that are
> > fanatical about binary reproducibility and checksums. Temporary file
> > names differ on each compilation, so that two different builds of
> > libgfortran.a, and by extension all of gcc, will not be bit-identical.
> >
> > Anyone else encountered this? Bug or feature? There are several
> > possible and relatively easy fixes -- any thoughts or feedback on
> > which fixes might be better, or worse? Something for mainline or for
> > only google-specific branches?
>
> Based on what I am seeing with various build systems and hear some
> users say, I'd love to see this addressed on mainline.
Thanks for the note.
To make things more concrete I've appended a prototype patch below. I
don't know if any other languages are affected, but if they are this
patch should extend reasonably well to cover those also. Does anyone
have strong objections to this approach? Thanks.
--
Omit OPT_cpp_ from the Dwarf producer string in gfortran.
Gfortran uses -cpp= internally, and with -grecord_gcc_switches
this command line switch is stored by default in object files. This causes
problems with build and packaging systems that care about gcc binary
reproducibility and file checksums; the temporary file is different on each
compiler invocation.
Fixed by adding a new lang_hook to filter language-specific command line
options, and making that filter omit OPT_cpp_ for fortran.
Tested for fortran (suppresses -cpp=...) and c++ (no effect).
gcc/ChangeLog
2012-08-16 Simon Baldwin
* dwarf2out.c (gen_producer_string): Add call to lang hooks to
filter decoded options codes.
* langhooks.c (lhd_no_dwarf_record_gcc_switch): New.
* langhooks.h (no_dwarf_record_gcc_switch): New lang_hooks member.
* langhooks-def.h (lhd_no_dwarf_record_gcc_switch): New declaration.
gcc/fortran/ChangeLog
* gfortran.h (gfc_no_dwarf_record_gcc_switch): New declaration.
* f95-lang.c (LANG_HOOKS_NO_DWARF_RECORD_GCC_SWITCH): Define.
* options.c (gfc_no_dwarf_record_gcc_switch): New, omits OPT_cpp_
from the Dwarf producer string.
Index: gcc/dwarf2out.c
===
--- gcc/dwarf2out.c (revision 190442)
+++ gcc/dwarf2out.c (working copy)
@@ -18101,6 +18101,10 @@ gen_producer_string (void)
/* Ignore these. */
continue;
default:
+ if (lang_hooks.no_dwarf_record_gcc_switch (save_decoded_options[j]
+ .opt_index))
+ /* Ignore anything the language wants omitted. */
+ continue;
gcc_checking_assert (save_decoded_options[j].canonical_option[0][0]
== '-');
switch (save_decoded_options[j].canonical_option[0][1])
Index: gcc/fortran/gfortran.h
===
--- gcc/fortran/gfortran.h (revision 190442)
+++ gcc/fortran/gfortran.h (working copy)
@@ -2432,6 +2432,7 @@ bool gfc_handle_option (size_t, const ch
const struct cl_option_handlers *);
bool gfc_post_options (const char **);
char *gfc_get_option_string (void);
+bool gfc_no_dwarf_record_gcc_switch (size_t);
/* f95-lang.c */
void gfc_maybe_initialize_eh (void);
Index: gcc/fortran/f95-lang.c
===
--- gcc/fortran/f95-lang.c (revision 190442)
+++ gcc/fortran/f95-lang.c (working copy)
@@ -98,6 +98,7 @@ static tree gfc_builtin_function (tree);
#undef LANG_HOOKS_INIT_OPTIONS_STRUCT
#undef LANG_HOOKS_INIT_OPTIONS
#undef LANG_HOOKS_HANDLE_OPTION
+#undef LANG_HOOKS_NO_DWARF_RECORD_GCC_SWITCH
#undef LANG_HOOKS_POST_OPTIONS
#undef LANG_HOOKS_PARSE_FILE
#undef LANG_HOOKS_MARK_ADDRESSABLE
@@ -129,6 +130,7 @@ static tree gfc_builtin_function (tree);
#define LANG_HOOKS_INIT_OPTIONS_STRUCT gfc_init_options_struct
#define LANG_HOOKS_INIT_OPTIONS gfc_init_options
#define LANG_HOOKS_HANDLE_OPTIONgfc_handle_option
+#define LANG_HOOKS_NO_DWARF_RECORD_GCC_SWITCH gfc_no_dwarf_record_gcc_switch
#define LANG_HOOKS_POST_OPTIONSgfc_post_options
#define LANG_HOOKS_PARSE_FILE gfc_be_parse_file
#define LANG_HOOKS_TYPE_FOR_MODE gfc_type_for_mode
Index: gcc/fortran/options.c
===
--- gcc/fortran/options.c (revision 190442)
+++ gcc/fortran/options.c (working copy)
@@ -1171,3 +1171,15 @@ gfc_get_option_string (void)
result[--pos] = '\0';
return result;
}
+
+/* Return 1 to suppress this language-specific option from the Dwarf
+ producer string. */
+
+bool
+gfc_no_dwarf_record_gcc_switch (size_t scode)
+{
+ enum opt_code code = (enum opt_code) scode;
+
+ /* Suppress internal "--cpp=
Instability in successive builds of fortran mod files with f951
I'm seeing an element of "instability" in files written by f951. About one time in some tens or even hundreds of builds done with a tight loop, the file libgomp/omp_lib.mod has a different checksum from others. This is slightly problematic because it means that binary distributions of gcc can't be relied on to build bit-identically when given identical source inputs. Automated build processes that insist on full reproducibility will therefore hiccup from time to time when they run into this effect. I've looked at fortran/module.c and at libgomp/omp_lib.mod, and this is not a bug. fortran/module.c builds a tree of pointers to symbols, and then traverses that tree to serialize symbols for the module file. The tree is organized by pointer value because its function is fast pointer lookup. This means, however, that the order nodes are visited in traversal will depend on the relative values of the pointers in it. If the pointer to symbol "A" on the first run is a lower address than the pointer to symbol "B" on the first run of f951, but higher on the second run, these serialized symbols will appear in different orders in the output module files from the two runs. Neither output module file is invalid. The ordering is irrelevant. fortran/module.c clearly indicates it writes symbols in no particular order. This lack of bit-equivalence in module files across f951 invocations could be caused by instability earlier in the compiler, perhaps uninitialized memory (gcc is configured for powerpc 8540). It seems more likely though that these pointers are all allocated by malloc and malloc addresses aren't reproducible. Malloc returns pointers from regions allocated by anonymous mmap. And the kernel is under no constraint to provide "predictable" addresses for mmap (and neither is malloc, for that matter). So every now and again the pattern of allocated symbol pointers leads to a different (yet still valid) output. In a sense, looking for bit-equivalence in module files might be asking for malloc/mmap to be completely deterministic, and they may not be. I've looked at fortran/module.c to try to find any simple way to force a consistent ordering, but there doesn't seem to be anything that wouldn't lead to massive and unnecessarily invasive change here. Is there one that I'm missing? Or any known issues with instability in gcc's front end or powerpc 8540 specific code? Anyone else run across this effect? If not, does anyone have any other ideas for how I might ensure complete determinism in f951? Or is insisting on bit-equivalent files from successive gcc builds just asking too much? Thanks. -- Google UK Limited | Registered Office: Belgrave House, 76 Buckingham Palace Road, London SW1W 9TQ | Registered in England Number: 3977902
Re: Something weird with cp/decl.c switch statement
Thanks for the review. Commited as revision 125339: r125339 | simonb | 2007-06-05 11:29:42 -0700 (Tue, 05 Jun 2007) | 4 lines * decl.c (grokdeclarator): Readability change. Moved case labels into direct switch statement scope. --S Mark Mitchell wrote: Ian Lance Taylor wrote: > And Simon already sent in a tested patch for a couple of days ago: > > http://gcc.gnu.org/ml/gcc-patches/2007-06/msg00199.html This patch is OK, thanks. -- Mark Mitchell CodeSourcery [EMAIL PROTECTED] (650) 331-3385 x713
