Re: Hot and Cold Partitioning (Was: GCC 4.1 Projects)
I apologize for not responding to these messages sooner; I was out of town for a few days and only just read them. In the first place, I am a little confused about exactly what Joern is objecting to. If I am reading your emails correctly, you seem to feel that the hot/cold partitioning optimization, as currently designed, has a problem because sometimes it will increase the size of the hot section by an amount that will not be compensated for by the removal of the cold code to another section. You also seem to be expressing concerns that some branch instructions will not be able to span the distance between hot and cold sections, and it appears that you therefore don't want this optimization to be put in. It sounds as if you don't want this optimization to go in at all, but in actuality it is already there, and what I am proposing to do is fix parts of it that are still a little bit broken. As with all optimizations, hot/cold partitioning is an educated guess at how to improve the program. Therefore it will on occasion make a wrong guess. By using profiling data (at other people indicated) the number of wrong guesses will be greatly reduced, but not entirely eliminated. While most of the time it will either have no effect or will improve program performance, it can and will occasionally slow it down. This is one of the reasons that the optimization is controlled by a flag, and is not turned on by default. If you find the optimization is giving you trouble, you can always turn it off. The optimization was designed to take into account the fact that on many architectures, various branch instructions might not be able to span the distance between hot/cold sections. As others have indicated, this is done by adding a level of indirection to the jumps. This is conditioned on macros that can (should) be defined by each architecture, so the indirection won't be performed on architectures where it isn't needed. There might be some validity in the idea of modifying this optimization, in the future, to consider the size of a basic block in addition to it's "hot-ness", when deciding which partition to put it into. I expect this would not be that difficult to implement, and would probably address your concerns. However, at the moment, I would first like to get the "correctness" fixes for the hot/cold partitioning optimization into FSF mainline. But I am open to persuasion, and if the FSF community in general feels that I really ought to add the size test as well at this time, I will do so. What do other people think? -- Caroline Tice [EMAIL PROTECTED] On Feb 28, 2005, at 12:09 PM, Joern RENNECKE wrote: Dale Johannesen wrote: No, you should not turn on partitioning in situations where code size is important to you. You are missing the point. In my example, with perfect profiling data, you still end up with more code in the hot section, Yes. i.e. more pages are actually swapped in. Unless the cross-section branch is actually executed, there's no reason the unconditional jumps should get paged in, so this doesn't follow. If you separate the unconditional jumps from the rest of the function, you just have created a per-function cold section. Except for corner cases, there would have to be a lot of them to save a page of working set. And if you have that many, it will mean that the condjump can't reach. And it is still utterly pointless to put blocks into the inter-function cold section if that only makes the intra-function cold section larger. So we've come from 4 bytes, on cycle: bf 0f mov #0,rn over 6 bytes, BR issue slot during one cycle: bt L2 L1: .. L2: bra L1 mov #0,n to 10 bytes in hot part of the hot section, 12 bytes in cold part of the hot section, and another 10 to 12 bytes in the cold section, while the execution time in the hot path is now two cycles (if we manage to get a good schedule, we might execute two other instructions in these cycles, but still, this is no better than we started out with): .hotsection: bf L2 mov.w 0f,rn braf @rn nop 0: .word L2-0b L1: ... L2: mov.l 0f,rn jmp @rn nop .balign 4 0: .long L3 .coldsection L3: mov.l 0f,rn jmp @rn mov #0,rn .balign 4 0: .long L1
Re: Bootstrap failure on i686-pc-linux-gnu since 2005-04-09 20:41UTC
The patch has just been submitted to gcc-patches for approval (the tests are nearly done and appear to be passing so far). -- Caroline [EMAIL PROTECTED] On Apr 11, 2005, at 10:12 PM, Kazu Hirata wrote: Hi, I have been getting bootstrap failures on i686-pc-linux-gnu since Caroline's patch http://gcc.gnu.org/ml/gcc-cvs/2005-04/msg00541.html Errors go like so: org/xml/sax/helpers/ParserAdapter.java: In class 'org.xml.sax.helpers.ParserAdapter': org/xml/sax/helpers/ParserAdapter.java: In constructor '()': org/xml/sax/helpers/ParserAdapter.java:84: internal compiler error: Segmentation fault I have confirmed that the first patch attached to PR 20934 fixes this problem. Looking at the audit trails in PR 20934, I see that Caroline has been working on this problem. What is the status of the patch? Is there more work or refinement needed? Kazu Hirata
Re: Mainline has been broken for more than 3 days now
The patch for this has already been submitted to gcc-patches: http://gcc.gnu.org/ml/gcc-patches/2005-04/msg01240.html -- Caroline Tice [EMAIL PROTECTED] On Apr 12, 2005, at 6:04 AM, Diego Novillo wrote: I have been using this crutch for the last couple of days to be able to get mainline to bootstrap with java enabled. Index: varasm.c === RCS file: /cvs/gcc/gcc/gcc/varasm.c,v retrieving revision 1.495 diff -u -3 -p -r1.495 varasm.c --- varasm.c9 Apr 2005 20:41:49 - 1.495 +++ varasm.c10 Apr 2005 22:39:41 - @@ -308,6 +308,7 @@ in_unlikely_text_section (void) ret_val = ((in_section == in_unlikely_executed_text) || (in_section == in_named +&& cfun && cfun->unlikely_text_section_name && strcmp (in_named_name, cfun->unlikely_text_section_name) == 0)); I know we are in stage1 and all, but shouldn't the timer have started already for whatever patch broke mainline on 2005-04-10? From the breakage, it seems that this change is at fault: +2005-04-09 Caroline Tice <[EMAIL PROTECTED]> + + * bb-reorder.c (find_rarely_executed_basic_blocks_and_crossing_edges): + Remove targetm.have_named_sections test. + (fix_edges_for_rarely_executed_code): Likewise. + (insert_section_boundary_note): Likewise. + (reorder_basic_blocks): Check partitioning flag before calling + verify_hot_cold_block_grouping. + * dbxout.c (dbxout_function_end): Get hot/cold section labels from + the function struct rather than global variables. + * dwarf2out.c (COLD_TEXT_SECTION_LABEL): New macro. + (COLD_END_LABEL): Likewise [ ... ] But I'm not 100% sure that this is the case. Diego.
Re: hot/cold vs glibc
On Apr 18, 2005, at 8:35 AM, Daniel Jacobowitz wrote: Hi Caroline, You've made this change to assemble_start_function (unidiff format): + last_text_section = no_section; + in_section = no_section; resolve_unique_section (decl, 0, flag_function_sections); + + /* Switch to the correct text section for the start of the function. */ + function_section (decl); + if (flag_reorder_blocks_and_partition + && !hot_label_written) +ASM_OUTPUT_LABEL (asm_out_file, hot_section_label); Why did you need to reset in_section? This causes an extra .text to be emitted before every function. It also breaks the (ugly, non-unit-at-a-time compatible, but otherwise working) mechanism that glibc uses to generate crti.o and crtn.o, so I can no longer build a mips64-linux toolchain using HEAD. That was because, up until very recently, the call to function_section there used in_section to partially determine which section to switch to. In the code just above the bit in your email, there were switches to the hot/cold sections to properly set alignments for the function and to insert the hot/cold start labels. Those calls to switch sections also set the value for in_section. I needed to blank it out again so that function_section would go to the correct section. I believe the current implementation of function_section no longer depends on the value of in_section, so you could probably remove that line without any adverse affects. Just out of curiousity, could you be more explicit about exactly how having an extra .text breaks the mechanism? That worries me... -- Caroline [EMAIL PROTECTED]
Re: output directory for -fvtv-debug and -fvtv-counts
Hi Sandra, Thank you for fixing this documentation. The code got updated a bit after this was written and I forgot to update that. The log files no longer go into "the dump file directory on the user's machine". Instead, this option checks to see if there is an environment variable VTV_LOGS_DIR that has been defined. If that variable exists, then the log files are written there; otherwise they go into the current working directory. Let me know if you have any other questions, and thank you for doing this! -- Caroline Tice cmt...@google.com On Wed, Dec 31, 2014 at 12:03 PM, Sandra Loosemore wrote: > I noticed that the documentation for the -fvtv-* options in invoke.texi > needed a little copy-editing and markup correction. I started working on a > patch, but I'm stuck on this: > > The documentation currently says the compile-time log files for these > options are written in "the dump file directory on the user's machine". > Where, exactly, is this supposed to be? From reading the code, it looks > like the default is the current directory rather than the directory of the > output file (the default for the -fdump-* options). Is this difference > intentional? If so, can the documentation just say it's written to the > current directory? Alternatively, the option that controls this, -dumpdir, > is not documented in invoke.texi; should it be, or is this (and its > similarly undocumented friend -dumpbase) strictly an internal option for > communication between the GCC driver and subprocesses, or something like > that? > > -Sandra >
[Announce] New vtable-security branch in GCC repository
Hello, I have just created a new branch, off the google/gcc-4_6-mobile branch, for working on improvements to function pointer and vtable security. The new branch is branches/google-4_6-mobile/vtable-security. I will be maintaining the branch. As features on this branch become mature, I hope to be able to merge them into the mainline compiler. -- Caroline Tice cmt...@google.com
Build/Makefile question
Hello, I am working on a patch (which I hope to be able to submit in the next few days), and I have run into a snag that I am hoping someone can help me with. As part of this patch, I am trying to build two new *.a (static archive) libraries in libstdc++-v3/libsupc++. I have modified Makefile.am and rebuilt Makefile.in in libsupc++ to do this. I had this working just fine on GCC 4.6.3, but when I try to apply the changes to GCC ToT, I find that something in the build system seems to have changed: Not only is it building the two static archive libraries, it is ALSO then building two dynamic shared libraries (*.so files), which I do NOT want. And even worse, it's putting the wrong stuff into the .so files. I have not been able to find/isolate the code that is telling GCC to go build the .so files. As I said, this worked for me in GCC 4.6.3. Does anybody know what might be going wrong or where/how to fix this? Help, please?? -- Caroline TIce cmt...@google.com
Re: Build/Makefile question
Ok, here are the patches for the Makefile.am and Makefile.in files. I am also having trouble with the following issue: I need to make sure that one of the new libraries is linked in with libstdc++ when libstdc++ gets built. I have not been able to successfully figure out where exactly is the command that builds & links libstdc++.so.* (yes I've looked at the build; it's being built with libtool somewhere, somehow, and no matter where/how I specify the link flag to link in my new library, the libtool link command is not picking it up). -- Caroline Tice cmt...@google.com On Fri, Oct 26, 2012 at 2:28 PM, Ian Lance Taylor wrote: > On Fri, Oct 26, 2012 at 10:58 AM, Caroline Tice wrote: >> >> I am working on a patch (which I hope to be able to submit in the next >> few days), and I have run into a snag that I am hoping someone can >> help me with. >> >> As part of this patch, I am trying to build two new *.a (static >> archive) libraries in libstdc++-v3/libsupc++. I have modified >> Makefile.am and rebuilt Makefile.in in libsupc++ to do this. I had >> this working just fine on GCC 4.6.3, but when I try to apply the >> changes to GCC ToT, I find that something in the build system seems to >> have changed: Not only is it building the two static archive >> libraries, it is ALSO then building two dynamic shared libraries (*.so >> files), which I do NOT want. And even worse, it's putting the wrong >> stuff into the .so files. I have not been able to find/isolate the >> code that is telling GCC to go build the .so files. As I said, this >> worked for me in GCC 4.6.3. Does anybody know what might be going >> wrong or where/how to fix this? Help, please?? > > Kind of hard to say without seeing the patch. Look at the build log, > and make sure that libtool is being invoked with --tag disable-shared. > That should be all that is required. > > Ian Makefile.am.diff Description: Binary data Makefile.in.diff Description: Binary data
Re: Build/Makefile question
Ian Tayler (in private communication) asked that I get the part of the build log that shows the .so and .a files being built and send it to the list. Here it is. -- Caroline Tice cmt...@google.com /bin/sh ../libtool --tag CXX --tag disable-shared --mode=compile /usr/local/google2/cmtice/gcc-fsf.obj/./gcc/xgcc -shared-libgcc -B/usr/local/google2/cmtice/gcc-fsf.obj/./gcc -nostdinc++\ -L/usr/local/google2/cmtice/gcc-fsf.obj/x86_64-unknown-linux-gnu/32/libstdc++-v3/src -L/usr/local/google2/cmtice/gcc-fsf.obj/x86_64-unknown-linux-gnu/32/libstdc++-v3/src/.libs -B/usr/loca\ l/x86_64-unknown-linux-gnu/bin/ -B/usr/local/x86_64-unknown-linux-gnu/lib/ -isystem /usr/local/x86_64-unknown-linux-gnu/include -isystem /usr/local/x86_64-unknown-linux-gnu/sys-include -m\ 32 -I/usr/local/google2/cmtice/gcc-fsf/libstdc++-v3/../libgcc -I/usr/local/google2/cmtice/gcc-fsf.obj/x86_64-unknown-linux-gnu/32/libstdc++-v3/include/x86_64-unknown-linux-gnu -I/usr/local\ /google2/cmtice/gcc-fsf.obj/x86_64-unknown-linux-gnu/32/libstdc++-v3/include -I/usr/local/google2/cmtice/gcc-fsf/libstdc++-v3/libsupc++ -prefer-pic -D_GLIBCXX_SHARED -Wall -Wextra -Wwrit\ e-strings -Wcast-qual -Wabi -fdiagnostics-show-location=once -ffunction-sections -fdata-sections -frandom-seed=vtv_init.lo -g -O0 -D_GNU_SOURCE -m32 -Wl,-L../libsupc++/.libs -Wl,--wh\ ole-archive,-lvtv_init,--no-whole-archive -fvtable-verify=std -Wno-error -c ../../../../../gcc-fsf/libstdc++-v3/libsupc++/vtv_init.cc libtool: compile: /usr/local/google2/cmtice/gcc-fsf.obj/./gcc/xgcc -shared-libgcc -B/usr/local/google2/cmtice/gcc-fsf.obj/./gcc -nostdinc++ -L/usr/local/google2/cmtice/gcc-fsf.obj/x86_64-\ unknown-linux-gnu/32/libstdc++-v3/src -L/usr/local/google2/cmtice/gcc-fsf.obj/x86_64-unknown-linux-gnu/32/libstdc++-v3/src/.libs -B/usr/local/x86_64-unknown-linux-gnu/bin/ -B/usr/local/x86\ _64-unknown-linux-gnu/lib/ -isystem /usr/local/x86_64-unknown-linux-gnu/include -isystem /usr/local/x86_64-unknown-linux-gnu/sys-include -m32 -I/usr/local/google2/cmtice/gcc-fsf/libstdc++-\ v3/../libgcc -I/usr/local/google2/cmtice/gcc-fsf.obj/x86_64-unknown-linux-gnu/32/libstdc++-v3/include/x86_64-unknown-linux-gnu -I/usr/local/google2/cmtice/gcc-fsf.obj/x86_64-unknown-linux-\ gnu/32/libstdc++-v3/include -I/usr/local/google2/cmtice/gcc-fsf/libstdc++-v3/libsupc++ -D_GLIBCXX_SHARED -Wall -Wextra -Wwrite-strings -Wcast-qual -Wabi -fdiagnostics-show-location=once -f\ function-sections -fdata-sections -frandom-seed=vtv_init.lo -g -O0 -D_GNU_SOURCE -m32 -Wl,-L../libsupc++/.libs -Wl,--whole-archive,-lvtv_init,--no-whole-archive -fvtable-verify=std -Wno-er\ ror -c ../../../../../gcc-fsf/libstdc++-v3/libsupc++/vtv_init.cc -fPIC -DPIC -D_GLIBCXX_SHARED -o vtv_init.o ../../../../../gcc-fsf/libstdc++-v3/libsupc++/vtv_init.cc:49:54: warning: constructor priorities from 0 to 100 are reserved for the implementation [enabled by default] void __VLTunprotect() __attribute__((constructor(98))); ^ ../../../../../gcc-fsf/libstdc++-v3/libsupc++/vtv_init.cc:59:53: warning: constructor priorities from 0 to 100 are reserved for the implementation [enabled by default] void __VLTprotect() __attribute__((constructor(100))); ^ /bin/sh ../libtool --tag=CC --mode=link /usr/local/google2/cmtice/gcc-fsf.obj/./gcc/xgcc -B/usr/local/google2/cmtice/gcc-fsf.obj/./gcc/ -B/usr/local/x86_64-unknown-linux-gnu/bin/ -B/usr/\ local/x86_64-unknown-linux-gnu/lib/ -isystem /usr/local/x86_64-unknown-linux-gnu/include -isystem /usr/local/x86_64-unknown-linux-gnu/sys-include -m32 -g -O0 -m32 -m32 -o libvtv_init.l\ a -rpath /usr/local/lib/../lib32 vtv_init.lo libtool: link: /usr/local/google2/cmtice/gcc-fsf.obj/./gcc/xgcc -B/usr/local/google2/cmtice/gcc-fsf.obj/./gcc/ -B/usr/local/x86_64-unknown-linux-gnu/bin/ -B/usr/local/x86_64-unknown-linux-\ gnu/lib/ -isystem /usr/local/x86_64-unknown-linux-gnu/include -isystem /usr/local/x86_64-unknown-linux-gnu/sys-include -m32 -shared-m32 -m32 -m32 -Wl,-soname -Wl,libvtv_init.so.0 -o\ .libs/libvtv_init.so.0.0.0 libtool: link: (cd ".libs" && rm -f "libvtv_init.so.0" && ln -s "libvtv_init.so.0.0.0" "libvtv_init.so.0") libtool: link: (cd ".libs" && rm -f "libvtv_init.so" && ln -s "libvtv_init.so.0.0.0" "libvtv_init.so") libtool: link: ar rc .libs/libvtv_init.a vtv_init.o libtool: link: ranlib .libs/libvtv_init.a libtool: link: ( cd ".libs" && rm -f "libvtv_init.la" && ln -s "../libvtv_init.la" "libvtv_init.la" ) /bin/sh ../libtool --tag CXX --tag disable-shared --mode=compile /usr/local/google2/cmtice/gcc-fsf.obj/./gcc/xgcc -shared-libgcc -B/usr/local/google2/cmtice/gcc-fsf.obj/./gcc -nostdinc++\ -L/usr/local/google2/cmtice/gcc-fsf.obj/x86_64-unknown-linux-gnu/32/libstdc++-v3/src -L/usr/local/google2/cmtice/gc
Re: Build/Makefile question
On Mon, Oct 29, 2012 at 7:55 AM, Michael Matz wrote: > Hi, > > On Sat, 27 Oct 2012, Ian Lance Taylor wrote: > >> On Sat, Oct 27, 2012 at 1:45 PM, Caroline Tice wrote: >> > Ian Tayler (in private communication) asked that I get the part of the >> > build log that shows the .so and .a files being built and send it to >> > the list. Here it is. >> >> I see the problem. libstdc++/libsupc++/Makefile.am overrides the >> default CXXLINK to invoke libtool with --tag disable-shared. Your new >> shared libraries have only C input files, so they are being linked >> with CXXLINK, they are being linked with LINK. You need to override >> the default value of LINK. > > Actually her Makefile.am contains C++ sources: > > ! vtv_init_sources = \ > ! vtv_init.cc > ! > ! vtv_stubs_sources = \ > ! vtv_stubs.cc > > which should be enough to make it use CXXLINK, but her Makefile.in > contains: > > + libvtv_init_la_LIBADD = > + libvtv_init_la_SOURCES = vtv_init.c > + libvtv_init_la_OBJECTS = vtv_init.lo > + libvtv_stubs_la_LIBADD = > + libvtv_stubs_la_SOURCES = vtv_stubs.c > + libvtv_stubs_la_OBJECTS = vtv_stubs.lo > ... > ! SOURCES = $(libsupc___la_SOURCES) $(libsupc__convenience_la_SOURCES) \ > ! libvtv_init.c libvtv_stubs.c > > ... but then still ... > > ! vtv_init_sources = \ > ! vtv_init.cc > ! > ! vtv_stubs_sources = \ > ! vtv_stubs.cc > > So, the first hunk of Makefile.in contains references to .c files. > Caroline: did you edit Makefile.in yourself? Otherwise I don't see how > this file could have been generated by automake with this inconsistent use > of .c/.cc. Or maybe you have still .c sources lying around in your > source tree? > Actually, I did have to edit the Makefile.in slightly. When I generate it with automake, it automatically adds the lines: libvtv_init_la_LIBADD = libvtv_init_la_SOURCES = libvtv_init.c libvtv_init_la_OBJECTS = libvtv_init.lo libvtv_stubs_la_LIBADD = libvtv_stubs_la_SOURCES = libvtv_stubs.c libvtv_stubs_la_OBJECTS = libvtv_stubs.lo as well as the lines: SOURCES = $(libsupc___la_SOURCES) $(libsupc__convenience_la_SOURCES) \ libvtv_init.c libvtv_stubs.c I don't know where it's getting those source file names, but they're wrong. I went in and removed the "lib" prefix from them, but I hadn't noticed that it had made them ".c" files instead of ".cc" files. If you know how to fix the .am file so that it doesn't generate these bogus & wrong source files in the .in file, I would appreciate your sharing with me... -- Caroline cmt...@google.com > > Ciao, > Michael.
Re: Build/Makefile question
I should have mentioned sooner, but thanks, that fixed my problem! -- Caroline cmt...@google.com On Mon, Oct 29, 2012 at 10:34 AM, Ian Lance Taylor wrote: > On Mon, Oct 29, 2012 at 10:24 AM, Caroline Tice wrote: >> >> Actually, I did have to edit the Makefile.in slightly. When I >> generate it with automake, it automatically adds the lines: >> >> libvtv_init_la_LIBADD = >> libvtv_init_la_SOURCES = libvtv_init.c >> libvtv_init_la_OBJECTS = libvtv_init.lo >> libvtv_stubs_la_LIBADD = >> libvtv_stubs_la_SOURCES = libvtv_stubs.c >> libvtv_stubs_la_OBJECTS = libvtv_stubs.lo >> >> as well as the lines: >> >> SOURCES = $(libsupc___la_SOURCES) $(libsupc__convenience_la_SOURCES) \ >> libvtv_init.c libvtv_stubs.c >> >> >> >> I don't know where it's getting those source file names, but they're >> wrong. I went in and removed the "lib" prefix from them, but I hadn't >> noticed that it had made them ".c" files instead of ".cc" files. If >> you know how to fix the .am file so that it doesn't generate these >> bogus & wrong source files in the .in file, I would appreciate your >> sharing with me... > > > Hmmm, never, ever, edit the Makefile.in file. And if you do edit it, > for goodness sakes tell us. But never edit it in the first place. > >> libvtv_init_la_SOURCES = libvtv_init.c > > automake came up with that because it is the default SOURCES for a > library. It means that you didn't define the required SOURCES > variable. You can see the name of that SOURCE variable from the line > that automake introduced: it's libvt_init_la_SOURCES. > > Ian
[Announce] New gcc 4.7 vtable security branch in GCC repository.
Hello, I have just created a new branch, based on the google/gcc-4_7-mobile branch, for migrating the vtable verification feature from gcc 4.6.3 (on the google/gcc-4_6-mobile-vtable-security branch) to gcc 4.7. The new branch is branches/google/gcc-4_7-mobile-vtable-security. I will be maintaining the branch. I have already started submitting the vtable verification feature patches (hopefully to go into GCC 4.9 when it opens) and will continue both working on it in the branch, and reconciling the work there with the ToT/patches.. -- Caroline Tice cmt...@google.com
Re: [Announce] New gcc 4.7 vtable security branch in GCC repository.
On Thu, Jan 24, 2013 at 8:34 AM, Caroline Tice wrote: > Hello, > > I have just created a new branch, based on the google/gcc-4_7-mobile > branch, for migrating the vtable verification feature from gcc 4.6.3 > (on the google/gcc-4_6-mobile-vtable-security branch) to gcc 4.7. The > new branch is branches/google/gcc-4_7-mobile-vtable-security. Typo: New branch name is branches/google/gcc-4_7-mobile-vtable-verification > > I will be maintaining the branch. I have already started submitting > the vtable verification feature patches (hopefully to go into GCC 4.9 > when it opens) and will continue both working on it in the branch, and > reconciling the work there with the ToT/patches.. > > -- Caroline Tice > cmt...@google.com
