Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c)

[Martin Liška]

On 5/20/21 5:55 PM, Jan Hubicka wrote:

Quick solution is to also modify partitioner to use the local symbol
names when doing incremental linking (those mixing in source code and
random seeds) to avoid clashes.


Good hint. I added hash based on object file name (I don't want to handle
proper string escaping) and -frandom-seed.

What do you think about the patch?
Thanks,
Martin
>From 372d2944571906932fd1419bfc51a949d67b857e Mon Sep 17 00:00:00 2001
From: Martin Liska 
Date: Fri, 21 May 2021 10:25:49 +0200
Subject: [PATCH] LTO: add lto_priv suffixfor LTO_LINKER_OUTPUT_NOLTOREL.

gcc/lto/ChangeLog:

	* lto-partition.c (privatize_symbol_name_1): Add random suffix
	based on hash of the object file and -frandom-seed.
---
 gcc/lto/lto-partition.c | 21 ++---
 1 file changed, 18 insertions(+), 3 deletions(-)

diff --git a/gcc/lto/lto-partition.c b/gcc/lto/lto-partition.c
index 15761ac9eb5..fef48c869a2 100644
--- a/gcc/lto/lto-partition.c
+++ b/gcc/lto/lto-partition.c
@@ -35,6 +35,7 @@ along with GCC; see the file COPYING3.  If not see
 #include "ipa-fnsummary.h"
 #include "lto-partition.h"
 #include "sreal.h"
+#include "toplev.h"
 
 vec ltrans_partitions;
 
@@ -941,9 +942,23 @@ privatize_symbol_name_1 (symtab_node *node, tree decl)
 
   name = maybe_rewrite_identifier (name);
   unsigned &clone_number = lto_clone_numbers->get_or_insert (name);
-  symtab->change_decl_assembler_name (decl,
-  clone_function_name (
-	  name, "lto_priv", clone_number));
+
+  char *suffix = NULL;
+  if (flag_lto_linker_output == LTO_LINKER_OUTPUT_NOLTOREL)
+{
+  hashval_t fnhash = 0;
+  if (node->lto_file_data != NULL)
+	fnhash = htab_hash_string (node->lto_file_data->file_name);
+  suffix = XNEWVEC (char, 128);
+  char sep = symbol_table::symbol_suffix_separator ();
+  sprintf (suffix, "lto_priv%c%u%c%" PRIu64, sep, fnhash, sep,
+	   (unsigned HOST_WIDE_INT)get_random_seed (false));
+}
+
+  tree clone
+= clone_function_name (name, suffix ? suffix : "lto_priv", clone_number);
+  symtab->change_decl_assembler_name (decl, clone);
+  free (suffix);
   clone_number++;
 
   if (node->lto_file_data)
-- 
2.31.1

Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c)

[Martin Liška]

On 5/20/21 2:54 PM, Richard Biener wrote:

On Thu, May 20, 2021 at 2:34 PM Martin Liška  wrote:


Hello.

I've got a patch candidate that leverages partial linking for a couple of 
selected object files.

I'm sending make all-host- jX results for my machine:

before: 3m18s (user 32m52s)
https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/1dd5eae5001295ba0230a689f7edc67284c9b742/gcc-all-host.svg

after: 2m57m (user 35m)
https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/d659b2187cf622167841efbbe6bc93cb33855fa9/gcc-all-host-partial-lto.svg

One can utilize it with:
make -j16 all-host PARTIAL_LTO=1

@Segher, Andrew: Can you please measure time improvement for your slow 
bootstrap?
One can also tweak --param=lto-partitions=16 param value.

Thoughts?


You're LTO linking multiple objects here - that's almost as if you
were doing this
for the whole of libbackend.a ... so $(OBJS)_CLFAGS += -flto and in the
libbackend.a rule do a similar partial link trick.


Yeah, apart from that one can't likely do partial linking for an archive:

$ g++ -no-pie -flto=auto --param=lto-partitions=16 -flinker-output=nolto-rel -r 
libbackend.a
collect2: fatal error: ld terminated with signal 11 [Segmentation fault], core 
dumped
compilation terminated.

while ld.bfd immediately finishes.



That gets you half of a LTO bootstrap then.

So why did you go from applying this per-file to multiple files?  Does $(LINKER)
have a proper rule to pick up a jobserver?

When upstreaming in any form you probably have to gate it on bootstrap-lto
being not active.


Sure, that's reasonable, we can likely detect a -flto option in $(COMPILE), 
right?

One more thing I face is broken dependency:
$ make clean && make -j32 PARTIAL_LTO=1

g++ -fcf-protection -fno-PIE -c   -g   -DIN_GCC -fPIC-fno-exceptions 
-fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings 
-Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute 
-Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros 
-Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. 
-I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. 
-I/home/marxin/Programming/gcc/gcc/../include 
-I/home/marxin/Programming/gcc/gcc/../libcpp/include 
-I/home/marxin/Programming/gcc/gcc/../libcody  
-I/home/marxin/Programming/gcc/gcc/../libdecnumber 
-I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber 
-I/home/marxin/Programming/gcc/gcc/../libbacktrace   -o gimple-match-lto.o -MT 
gimple-match-lto.o -MMD -MP -MF ./.deps/gimple-match-lto.TPo gimple-match.c 
-flto
g++ -fcf-protection -fno-PIE -c   -g   -DIN_GCC -fPIC-fno-exceptions 
-fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings 
-Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute 
-Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros 
-Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. 
-I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. 
-I/home/marxin/Programming/gcc/gcc/../include 
-I/home/marxin/Programming/gcc/gcc/../libcpp/include 
-I/home/marxin/Programming/gcc/gcc/../libcody  
-I/home/marxin/Programming/gcc/gcc/../libdecnumber 
-I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber 
-I/home/marxin/Programming/gcc/gcc/../libbacktrace   -o generic-match-lto.o -MT 
generic-match-lto.o -MMD -MP -MF ./.deps/generic-match-lto.TPo generic-match.c 
-flto

In file included from ./tm.h:26,
 from /home/marxin/Programming/gcc/gcc/backend.h:28,
 from /home/marxin/Programming/gcc/gcc/generic-match-head.c:23,
 from generic-match.c:4:
/home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error: 
insn-attr-common.h: No such file or directory
 2286 | #include "insn-attr-common.h"
  |  ^~~~
compilation terminated.
make: *** [Makefile:2678: generic-match-lto.o] Error 1
make: *** Waiting for unfinished jobs

In file included from ./tm.h:26,
 from /home/marxin/Programming/gcc/gcc/backend.h:28,
 from /home/marxin/Programming/gcc/gcc/gimple-match-head.c:23,
 from gimple-match.c:4:
/home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error: 
insn-attr-common.h: No such file or directory
 2286 | #include "insn-attr-common.h"
  |  ^~~~

I explicitly added:
gimple-match.o: gimple-match.c $(generated_files)
generic-match.o: generic-match.c $(generated_files)

But it's not obeyed.

Martin



Richard.


Thanks,
Martin


>From b209c7aea76ceb8eadcf5843f30299fc8041a579 Mon Sep 17 00:00:00 2001
From: Martin Liska 
Date: Thu, 20 May 2021 14:29:35 +0200
Subject: [PATCH] Try LTO partial linking.

---
 gcc/Makefile.in | 33 +
 1 file changed, 29 insertions(+), 4 deletions(-)

diff --git a/gcc/Makefile.in b/gcc/Makefile.in
index

Re: [PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c)

[David Edelsohn via Gcc]

On Fri, May 21, 2021 at 5:25 AM Martin Liška  wrote:
>
> On 5/20/21 2:54 PM, Richard Biener wrote:
> > On Thu, May 20, 2021 at 2:34 PM Martin Liška  wrote:
> >>
> >> Hello.
> >>
> >> I've got a patch candidate that leverages partial linking for a couple of 
> >> selected object files.
> >>
> >> I'm sending make all-host- jX results for my machine:
> >>
> >> before: 3m18s (user 32m52s)
> >> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/1dd5eae5001295ba0230a689f7edc67284c9b742/gcc-all-host.svg
> >>
> >> after: 2m57m (user 35m)
> >> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/d659b2187cf622167841efbbe6bc93cb33855fa9/gcc-all-host-partial-lto.svg
> >>
> >> One can utilize it with:
> >> make -j16 all-host PARTIAL_LTO=1
> >>
> >> @Segher, Andrew: Can you please measure time improvement for your slow 
> >> bootstrap?
> >> One can also tweak --param=lto-partitions=16 param value.
> >>
> >> Thoughts?
> >
> > You're LTO linking multiple objects here - that's almost as if you
> > were doing this
> > for the whole of libbackend.a ... so $(OBJS)_CLFAGS += -flto and in the
> > libbackend.a rule do a similar partial link trick.
>
> Yeah, apart from that one can't likely do partial linking for an archive:
>
> $ g++ -no-pie -flto=auto --param=lto-partitions=16 -flinker-output=nolto-rel 
> -r libbackend.a
> collect2: fatal error: ld terminated with signal 11 [Segmentation fault], 
> core dumped
> compilation terminated.
>
> while ld.bfd immediately finishes.
>
> >
> > That gets you half of a LTO bootstrap then.
> >
> > So why did you go from applying this per-file to multiple files?  Does 
> > $(LINKER)
> > have a proper rule to pick up a jobserver?
> >
> > When upstreaming in any form you probably have to gate it on bootstrap-lto
> > being not active.
>
> Sure, that's reasonable, we can likely detect a -flto option in $(COMPILE), 
> right?
>
> One more thing I face is broken dependency:
> $ make clean && make -j32 PARTIAL_LTO=1
>
> g++ -fcf-protection -fno-PIE -c   -g   -DIN_GCC -fPIC-fno-exceptions 
> -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing 
> -Wwrite-strings -Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute 
> -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros 
> -Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. 
> -I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. 
> -I/home/marxin/Programming/gcc/gcc/../include 
> -I/home/marxin/Programming/gcc/gcc/../libcpp/include 
> -I/home/marxin/Programming/gcc/gcc/../libcody  
> -I/home/marxin/Programming/gcc/gcc/../libdecnumber 
> -I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber 
> -I/home/marxin/Programming/gcc/gcc/../libbacktrace   -o gimple-match-lto.o 
> -MT gimple-match-lto.o -MMD -MP -MF ./.deps/gimple-match-lto.TPo 
> gimple-match.c -flto
> g++ -fcf-protection -fno-PIE -c   -g   -DIN_GCC -fPIC-fno-exceptions 
> -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing 
> -Wwrite-strings -Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute 
> -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros 
> -Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. 
> -I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. 
> -I/home/marxin/Programming/gcc/gcc/../include 
> -I/home/marxin/Programming/gcc/gcc/../libcpp/include 
> -I/home/marxin/Programming/gcc/gcc/../libcody  
> -I/home/marxin/Programming/gcc/gcc/../libdecnumber 
> -I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber 
> -I/home/marxin/Programming/gcc/gcc/../libbacktrace   -o generic-match-lto.o 
> -MT generic-match-lto.o -MMD -MP -MF ./.deps/generic-match-lto.TPo 
> generic-match.c -flto
>
> In file included from ./tm.h:26,
>   from /home/marxin/Programming/gcc/gcc/backend.h:28,
>   from 
> /home/marxin/Programming/gcc/gcc/generic-match-head.c:23,
>   from generic-match.c:4:
> /home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error: 
> insn-attr-common.h: No such file or directory
>   2286 | #include "insn-attr-common.h"
>|  ^~~~
> compilation terminated.
> make: *** [Makefile:2678: generic-match-lto.o] Error 1
> make: *** Waiting for unfinished jobs
>
> In file included from ./tm.h:26,
>   from /home/marxin/Programming/gcc/gcc/backend.h:28,
>   from 
> /home/marxin/Programming/gcc/gcc/gimple-match-head.c:23,
>   from gimple-match.c:4:
> /home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error: 
> insn-attr-common.h: No such file or directory
>   2286 | #include "insn-attr-common.h"
>|  ^~~~
>
> I explicitly added:
> gimple-match.o: gimple-match.c $(generated_files)
> generic-match.o: generic-match.c $(generated_files)
>
> But it's not obeyed.

Please remembe

Re: GCC 9.4 Status Report (2021-05-19), branch frozen for release

[Jason Merrill via Gcc]

Sorry, just pushed 3 patches before I noticed this.  They're safe but not
critical, should I back them out?

On Wed, May 19, 2021 at 4:06 AM Richard Biener  wrote:

>
> Status
> ==
>
> The GCC 9 branch is now frozen for the upcoming GCC 9.4 release.
> I will announce a first release candidate shortly.
>
> All changes require release manager approval now.
>
>
> Quality Data
> 
>
> Priority  #   Change from last report
> ---   ---
> P1-   1
> P2  304   -   7
> P3   23   +   7
> P4  174
> P5   23
> ---   ---
> Total P1-P3 328   -   1
> Total   525   -   1
>
>
> Previous Report
> ===
>
> https://gcc.gnu.org/pipermail/gcc/2021-April/235948.html
>
>

Re: GCC 9.4 Release Candidate available

[Iain Sandoe via Gcc]

Richard Biener  wrote:



The first release candidate for GCC 9.4 is available from

https://sourceware.org/pub/gcc/snapshots/9.4.0-RC-20210519/

and shortly its mirrors.  It has been generated from git commit
8091c46cf736124a106922ddfd1fdb99f33b0241.

I have so far bootstrapped and tested the release candidate
on {x86_64,i586,aarch64,ppc64le,s390x,riscv64,ppc,ppc64,armv7l}-linux.


(unfortunately very little time available at the moment for Darwin)
I bootstrapped/tested i686-darwin9,10 powerpc-darwin9, x86_64-darwin10..20
with no show-stoppers seen.

Iain

gcc-10-20210521 is now available

[GCC Administrator via Gcc]

Snapshot gcc-10-20210521 is now available on
  https://gcc.gnu.org/pub/gcc/snapshots/10-20210521/
and on various mirrors, see http://gcc.gnu.org/mirrors.html for details.

This snapshot has been generated from the GCC 10 git branch
with the following options: git://gcc.gnu.org/git/gcc.git branch 
releases/gcc-10 revision a7e72b282177fbaa6a852324641c871313d326da

You'll find:

 gcc-10-20210521.tar.xz   Complete GCC

  SHA256=65d44c6a5669df9eec528e3d8fbf3e5227c70db8c7ae1f7aef4dc39461e79549
  SHA1=cf54e5060b9a23342acd8f55e1ffbbaacde5895a

Diffs from 10-20210514 are available in the diffs/ subdirectory.

When a particular snapshot is ready for public consumption the LATEST-10
link is updated and a message is sent to the gcc list.  Please do not use
a snapshot before it has been announced that way.

RE: [EXTERNAL] Re: State of AutoFDO in GCC

[Eugene Rozenfeld via Gcc]

I tried following the instructions in "2.2 Build autofdo tool for gcc" in 
https://github.com/google/autofdo#readme
and got build failures:

eugene@eugene-Virtual-Machine:~/autofdo1/build$ ninja
[1/228] Building CXX object CMakeFiles/create_gcov_lib.dir/profile.cc.o
FAILED: CMakeFiles/create_gcov_lib.dir/profile.cc.o
/usr/bin/c++  -I../ -I../third_party/glog/src -I../third_party/abseil 
-I../third_party/perf_data_converter/src 
-I../third_party/perf_data_converter/src/quipper -I../util -I. 
-Ithird_party/glog -std=gnu++1z -MD -MT 
CMakeFiles/create_gcov_lib.dir/profile.cc.o -MF 
CMakeFiles/create_gcov_lib.dir/profile.cc.o.d -o 
CMakeFiles/create_gcov_lib.dir/profile.cc.o -c ../profile.cc
In file included from 
../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
 from ../sample_reader.h:18,
 from ../profile.h:15,
 from ../profile.cc:5:
../third_party/perf_data_converter/src/quipper/base/macros.h:8:0: warning: 
"DISALLOW_COPY_AND_ASSIGN" redefined
#define DISALLOW_COPY_AND_ASSIGN(TypeName) \
In file included from ../profile.h:14:0,
 from ../profile.cc:5:
../base/macros.h:114:0: note: this is the location of the previous definition
#define DISALLOW_COPY_AND_ASSIGN(TypeName) \
In file included from 
../third_party/perf_data_converter/src/quipper/perf_parser.h:18:0,
 from ../sample_reader.h:18,
 from ../profile.h:15,
 from ../profile.cc:5:
../third_party/perf_data_converter/src/quipper/base/macros.h:12:0: warning: 
"arraysize" redefined
#define arraysize(x) (sizeof(x) / sizeof(*x))
In file included from ../profile.h:14:0,
 from ../profile.cc:5:
../base/macros.h:162:0: note: this is the location of the previous definition
#define arraysize(array) (sizeof(ArraySizeHelper(array)))
In file included from 
../third_party/perf_data_converter/src/quipper/perf_parser.h:21:0,
 from ../sample_reader.h:18,
 from ../profile.h:15,
 from ../profile.cc:5:
../third_party/perf_data_converter/src/quipper/compat/proto.h:16:10: fatal 
error: perf_stat.pb.h: No such file or directory
#include "perf_stat.pb.h"
  ^~~~
compilation terminated.

What is supposed to generate perf_stat.pb.h?

Thanks,

Eugene

From: Wei Mi 
Sent: Monday, May 10, 2021 4:47 PM
To: Andi Kleen 
Cc: Hongtao Yu ; Xinliang David Li ; Jan 
Hubicka ; gcc@gcc.gnu.org; Eugene Rozenfeld 
; Wenlei He 
Subject: [EXTERNAL] Re: State of AutoFDO in GCC

https://github.com/google/autofdo
 has been updated. Now create_gcov/dump_gcov are added back and can be built 
separately.

Please look at "2.2 Build autofdo tool for gcc" in 
https://github.com/google/autofdo#readme

On Wed, Apr 28, 2021 at 10:40 PM Andi Kleen 
mailto:a...@linux.intel.com>> wrote:
>
> On Mon, Apr 26, 2021 at 06:40:56PM +, Hongtao Yu wrote:
> >Andi, thanks for pointing out the perf script issues. Can you please
> >elaborate a bit on the exact issue you have seen? We've been using
> >specific output of perf script such as mmap, LBR and callstack events
> >filtered by process id. It works fine so far but may certainly hit issues
> >in the future with extended uses.
>
> Okay I took a look at the latest autofdo now. It seems to be basically
> a LLVM project now that depends on LLVM to even build with all kinds
> of dependency hell on some old LLVM version and other packages.
>
> I guess gcc will really need a replacement that doesn't pull in
> all of LLVM if it wants to continue supporting autofdo.
>
> I'm myself unable to build now.
>
> I'm using the old version I had a git fork of and that
> was before all of this. I added a patch to make it work
> with the latest perf by ignoring increased perf_attr
> and unknown perf events.
>
> Honza please use
>
> https://github.com/andikleen/autofdo