Re: That light at the end of the tunnel?

[Eric S. Raymond]

David Edelsohn :
> I realize that reverting the change would cause another regression.
> My suggestion is that the bisection of reposurgeon may be easier /
> faster and could help you pinpoint the root cause.  Then you could
> consider a different solution to the problem that you were trying to
> fix with the initial patch.  You could see explicitly how the
> reposurgeon patch fixes one problem and how it creates a different
> problem in Trunk.

Apologies for the belated reply.  I've been on semi-vacation at the
World Boardgaming Championships for the last week, working on NTPsec
stuff only.

Yes, if I were certain of the SVN revision at which conversion was last good,
I would do this. Since I'm not, and can't regress on two timelines at once,
I have to choose one.
-- 
http://www.catb.org/~esr/";>Eric S. Raymond

My work is funded by the Internet Civil Engineering Institute: https://icei.org
Please visit their site and donate: the civilization you save might be your own.

Re: That light at the end of the tunnel?

[Eric S. Raymond]

Richard Biener :
> Ok, so let me ask whether you can currently convert trunk and 
> gcc-{6,7,8}-branch
> successfully, ignoring "merges" into them (shouldn't have happened).  All 
> other
> branches can in theory be converted later if required, right?

Trunk conversionm is currently broken.  That's what makes this serious problem;
it was working before.
-- 
http://www.catb.org/~esr/";>Eric S. Raymond

My work is funded by the Internet Civil Engineering Institute: https://icei.org
Please visit their site and donate: the civilization you save might be your own.

Re: That light at the end of the tunnel?

[Michael Clark]

> On 31/07/2018, at 12:47 AM, Eric S. Raymond  wrote:
> 
> Richard Biener :
>> Ok, so let me ask whether you can currently convert trunk and 
>> gcc-{6,7,8}-branch
>> successfully, ignoring "merges" into them (shouldn't have happened).  All 
>> other
>> branches can in theory be converted later if required, right?
> 
> Trunk conversionm is currently broken.  That's what makes this serious 
> problem;
> it was working before.

Didn’t you mention the trunk conversion issue was due to a bug introduced into 
reposurgeon at some date recently as a fix for some other conversion?

Can you not roll back to a version of reposurgeon used for the last successful 
conversion? Assuming you know the date of the last successful conversion, then 
you can use the reposurgeon version from that date.

What were the problems in the last “successful” conversion that led to the 
subsequent changes in reposurgeon?

Were the conversion problems in the last successful conversion major enough 
that the resultant tree was not usable for trunk and the major development 
branches: gcc6, gcc7 and gcc8 branches? Or were the issues on older branches?

If you are bisecting reposurgeon then would you not search for the date of the 
message when you had reported successful conversion here on this list (with 
whatever remaining issues that version had), and use the version of reposurgeon 
from that date for your next trial conversion?

Last known good version of reposurgeon is the problem right? (with whatever 
outstanding issues that version had with the conversion).

Re: That light at the end of the tunnel?

[Eric S. Raymond]

Joseph Myers :
> On Mon, 23 Jul 2018, Richard Earnshaw (lists) wrote:
> 
> > So traditional git bisect is inherently serial, but we can be more
> > creative here, surely.  A single run halves the search space each time.
> > But three machines working together can split it into 4 each run, 7
> > machines into 8, etc.  You don't even need a precise 2^N - 1 to get a
> > speedup.
> 
> Exactly.  Given an appropriate recipe for testing whether the conversion 
> of history up to a given revision is OK or not, I can run tests in 
> parallel for nine different revisions on nine different machines (each 
> with 128 GB memory) at the same time as easily as running one such test.  

I'll try to work up a script that bundles the test procedure this week.

Before I do that, though, I have an idea that might reduce the test times
a lot.  It hasn't been practical to strip and topo-reduce the repository
before this, but I'm almpst finished porting repocutter to go.

> I think parallelising the bisection process is a better approach than 
> trying to convert only a subset of branches (which I don't think would 
> help with the present problem - though we can always consider killing 
> selected branches with too many mid-branch deletealls, if appropriate) or 
> waiting for a move to Go.

The conversion already kills one old branch because of an unrecoverable
malformation - I checked with the devlist about this. I agree that trying
to cut other branches is unlikely to be helpful while we have
incorrect conversion on trunk.
-- 
http://www.catb.org/~esr/";>Eric S. Raymond

My work is funded by the Internet Civil Engineering Institute: https://icei.org
Please visit their site and donate: the civilization you save might be your own.

Re: [RFC] Adding Python as a possible language and it's usage

[Joseph Myers]

On Fri, 27 Jul 2018, Paul Smith wrote:

> If Perl is already in the bootstrap set and the awk scripts are hard to
> maintain then why can't the awk scripts be rewritten in Perl instead of
> Python?  That would avoid adding more prerequisites and surely Perl is
> sufficiently expressive that it can perform these translations just as
> well as Python.

At least in the glibc community we find the current developers generally 
prefer Python for such code, so using it in place of Perl (or Awk) works 
better for maintainability now.

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: [RFC] Adding Python as a possible language and it's usage

[Joseph Myers]

On Sat, 28 Jul 2018, Ramana Radhakrishnan wrote:

> > Obviously if you're bootstrapping core packages and their build
> > dependencies, use in glibc is more or less equivalent to use in GCC.  (But
> > if build dependencies include those involved in testing, you already have
> > python as one for glibc, and Tcl for GCC, for example.)
> 
> This implies that the decision for glibc has been made. while you
> imply above that the discussion is still on going ?

Python has been used for some glibc tests for some time.  It's usage to 
replace other Perl and Awk scripts (and especially those required for the 
build) for which there is discussion - though as no-one has objected to 
such a change we may effectively have consensus.

https://sourceware.org/ml/libc-alpha/2018-07/msg00559.html

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: c plugin guidance wanted

[David Malcolm]

On Sun, 2018-07-29 at 13:41 -0700, Joe Perches wrote:
> I would like to implement a gcc plugin that can compress an
> __attribute__((format(printf, x, y))) const char array before
> storage so that the formats can be size reduced when stored and
> expanded appropriately before use.
> 
> As this is the first plugin I am trying to implement, I'm a bit
> lost in details and I am looking for guidance.
> 
> Basically:
> 
> Take the printf_format_type CONST_DECL format being verified by
> gcc/c-format-main.c and after the verification, compress the
> format and its length before storage into its .read_only section.
> 
> A custom vsnprintf routine would expand the compressed format to
> its original form before use.
> 
> All tips to necessary gcc plugin implementation details appreciated.

If I understand your idea correctly, you're wanting to write a plugin
that turns something like:

  printf ("some format string %s %i %s etc\n", arg1, arg2, arg3);

into something like:

  your_custom_printf ("cmprssd_frmt_strng", arg1, arg2, arg3);

I don't know how familiar with the internals of GCC you are.  The way
GCC stores functions internally has three phases:
* tree-like structures, coming out of the frontends, which become...
* "gimple" statements, in SSA form, in linked lists within basic blocks
in a CFG [1], which become...
* lower-level rtx instructions, initially in a CFG, which eventually
become assembler.
(I'm probably oversimplifying here).

So for any plugin pass you need to choose whether it's going to work on
the gimple form, or on the rtx form, and where to put your pass so that
it interacts well with gcc's other passes.

You can see the changing internal representation by passing in
  -fdump-tree-all -fdump-ipa-all -fdump-rtl-all
and then wading through the dumpfiles, or, for functions with non-
trivial control flow:
  -fdump-tree-all-graph -fdump-ipa-all-graph -fdump-rtl-all-graph
to get .dot files, if you have GraphViz handy.

It sounds like you can implement the above transformation by purely
manipulating one call statement, changing which function is to be
called, and manipulating the format string argument.  I believe this
could be done either on gimple, or on rtl.

Note that we already do some optimizations where e.g.
  printf ("%s", string);
get turned into:
  puts (string);

So you might want to decide how your optimization ought to interact
with those ones: would you want a "compressed_puts", or to do your
optimization first so that this one doesn't happen? etc

Hope this is helpful
Dave

[1] (strictly speaking, it takes a few passes before gimple statements
are in a CFG, and then a little later into SSA form)

Re: [RFC] Adding Python as a possible language and it's usage

[Andreas Schwab]

On Jul 30 2018, Joseph Myers  wrote:

> Python has been used for some glibc tests for some time.

Using it for tests is ok, since they are not part of the bootstrap
cycle.

Andreas.

-- 
Andreas Schwab, SUSE Labs, sch...@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: That light at the end of the tunnel?

[Eric S. Raymond]

Michael Clark :
> Can you not roll back to a version of reposurgeon used for the last 
> successful conversion? Assuming you know the date of the last successful 
> conversion, then you can use the reposurgeon version from that date.

I don't remember the date of the last successful conversion. Next time
I get one I will unambiguously record it.

All your suggestions are intelligent.  Unfortunately, I had already considered
them all myself.
-- 
http://www.catb.org/~esr/";>Eric S. Raymond

My work is funded by the Internet Civil Engineering Institute: https://icei.org
Please visit their site and donate: the civilization you save might be your own.

Re: c plugin guidance wanted

[Joe Perches]

On Mon, 2018-07-30 at 10:51 -0400, David Malcolm wrote:
> On Sun, 2018-07-29 at 13:41 -0700, Joe Perches wrote:
> > I would like to implement a gcc plugin that can compress an
> > __attribute__((format(printf, x, y))) const char array before
> > storage so that the formats can be size reduced when stored and
> > expanded appropriately before use.
> > 
> > As this is the first plugin I am trying to implement, I'm a bit
> > lost in details and I am looking for guidance.
> > 
> > Basically:
> > 
> > Take the printf_format_type CONST_DECL format being verified by
> > gcc/c-format-main.c and after the verification, compress the
> > format and its length before storage into its .read_only section.
> > 
> > A custom vsnprintf routine would expand the compressed format to
> > its original form before use.
> > 
> > All tips to necessary gcc plugin implementation details appreciated.
> 
> If I understand your idea correctly, you're wanting to write a plugin
> that turns something like:
> 
>   printf ("some format string %s %i %s etc\n", arg1, arg2, arg3);
> 
> into something like:
> 
>   your_custom_printf ("cmprssd_frmt_strng", arg1, arg2, arg3);

Thank you for the reply.

More or less, but not quite.

The idea is to use a specific __attribute__, here
format(printf, x, y), although it could be another
custom attribute, to identify which strings to store
in some compressed manner.

I want to compress some format strings and keep gcc's
ability to verify format and arg compatibility while
possibly duplicating storage for identical strings.

For example, this should store 2 strings in .rodata...

o original uncompressed "fmt: '%s'\n"
o and the compressed equivalent used only in the first
  compress_fmt call for the first argument.

$ cat t_format.c
__attribute__((format(printf, 1, 2)))
void compress_fmt(const char *fmt, ...);
void original_fmt(const char *fmt, ...);

#define FMT "fmt: '%s'\n"

void foo(void)
{
  const char *fmt = FMT;// uncompressed

  compress_fmt(FMT, fmt);   // compressed and uncompressed
  original_fmt(FMT, fmt);   // uncompressed

  compress_fmt(fmt, fmt);   // uncompressed
}
$

Relocation Truncated to Fit: R_X86_64_PC32

[R0b0t1]

Hello list,

I have been told to solve this error I should pass -fPIC and
-mcmodel=large. I believe I understand the reasons.

Sadly the error persists. Any advice?

Thanks in advance,
R0b0t1

---

Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-pc-cygwin/7.3.0/lto-wrapper.exe
Target: x86_64-pc-cygwin
Configured with:
/cygdrive/i/szsz/tmpp/gcc/gcc-7.3.0-3.x86_64/src/gcc-7.3.0/configure
--srcdir=/cygdrive/i/szsz/tmpp/gcc/gcc-7.3.0-3.x86_64/src/gcc-7.3.0
--prefix=/usr --exec-prefix=/usr --localstatedir=/var
--sysconfdir=/etc --docdir=/usr/share/doc/gcc
--htmldir=/usr/share/doc/gcc/html -C --build=x86_64-pc-cygwin
--host=x86_64-pc-cygwin --target=x86_64-pc-cygwin
--without-libiconv-prefix --without-libintl-prefix
--libexecdir=/usr/lib --enable-shared --enable-shared-libgcc
--enable-static --enable-version-specific-runtime-libs
--enable-bootstrap --enable-__cxa_atexit --with-dwarf2
--with-tune=generic
--enable-languages=ada,c,c++,fortran,lto,objc,obj-c++
--enable-graphite --enable-threads=posix --enable-libatomic
--enable-libcilkrts --enable-libgomp --enable-libitm
--enable-libquadmath --enable-libquadmath-support --disable-libssp
--enable-libada --disable-symvers --with-gnu-ld --with-gnu-as
--with-cloog-include=/usr/include/cloog-isl --without-libiconv-prefix
--without-libintl-prefix --with-system-zlib --enable-linker-build-id
--with-default-libstdcxx-abi=gcc4-compatible
--enable-libstdcxx-filesystem-ts
Thread model: posix
gcc version 7.3.0 (GCC)

Re: Relocation Truncated to Fit: R_X86_64_PC32

[R0b0t1]

On Mon, Jul 30, 2018 at 4:32 PM, R0b0t1  wrote:
> Hello list,
>
> I have been told to solve this error I should pass -fPIC and
> -mcmodel=large. I believe I understand the reasons.
>
> Sadly the error persists. Any advice?
>

It looks like I may have to recompile everything with -mcmodel=large
(https://gcc.gnu.org/bugzilla/show_bug.cgi?id=46125) which is rather
inconvenient in this case.

If anyone has more information it may help.

Cheers,
R0b0t1

Re: Relocation Truncated to Fit: R_X86_64_PC32

[Jonathan Wakely]

On Mon, 30 Jul 2018 at 22:53, R0b0t1  wrote:
>
> On Mon, Jul 30, 2018 at 4:32 PM, R0b0t1  wrote:
> > Hello list,
> >
> > I have been told to solve this error I should pass -fPIC and
> > -mcmodel=large. I believe I understand the reasons.
> >
> > Sadly the error persists. Any advice?
> >
>
> It looks like I may have to recompile everything with -mcmodel=large
> (https://gcc.gnu.org/bugzilla/show_bug.cgi?id=46125) which is rather
> inconvenient in this case.
>
> If anyone has more information it may help.

Your question is off-topic on the gcc list, I've CC'd the gcc-help
list where it belongs. Please remove gcc@ from any further replies.

-fPIC should be enough, you shouldn't need to change the code model.

Issues building gcc natively on mipsel

[martin krastev]

Hello,

I'm trying to build gcc-7.3.0 (md5 747d5010b7c6938b480bc6e4d7c4be9a of
tar.gz) natively on a MACHTYPE=mipsel-unknown-linux-gnu MIPS p5600 machine
under Debian Stretch. I'm getting an illegal instruction during libstdc++
build phase:

libtool: compile:  /home/gru/proj/gcc_build/./gcc/xgcc -shared-libgcc
-B/home/gru/proj/gcc_build/./gcc -nostdinc++
-L/home/gru/proj/gcc_build/mipsel-linux-gnu/libstdc++-v3/src
-L/home/gru/proj/gcc_build/mipsel-linux-gnu/libstdc++-v3/src/.libs
-L/home/gru/proj/gcc_build/mipsel-linux-gnu/libstdc++-v3/libsupc++/.libs
-B/usr/local/mipsel-linux-gnu/bin/ -B/usr/local/mipsel-linux-gnu/lib/
-isystem /usr/local/mipsel-linux-gnu/include -isystem
/usr/local/mipsel-linux-gnu/sys-include
-I/home/gru/proj/gcc-7.3.0/libstdc++-v3/../libgcc
-I/home/gru/proj/gcc_build/mipsel-linux-gnu/libstdc++-v3/include/mipsel-linux-gnu
-I/home/gru/proj/gcc_build/mipsel-linux-gnu/libstdc++-v3/include
-I/home/gru/proj/gcc-7.3.0/libstdc++-v3/libsupc++ -D_GLIBCXX_SHARED
-std=gnu++14 -Wall -Wextra -Wwrite-strings -Wcast-qual -Wabi
-fdiagnostics-show-location=once -ffunction-sections -fdata-sections
-frandom-seed=cow-ops.lo -g -O2 -D_GNU_SOURCE -minterlink-mips16 -c
/home/gru/proj/gcc_build/../gcc-7.3.0/libstdc++-v3/src/filesystem/cow-ops.cc
-fPIC -DPIC -D_GLIBCXX_SHARED -o cow-ops.o
In file included from
/home/gru/proj/gcc_build/mipsel-linux-gnu/libstdc++-v3/include/vector:64:0,
 from
/home/gru/proj/gcc_build/mipsel-linux-gnu/libstdc++-v3/include/experimental/bits/fs_path.h:39,
 from
/home/gru/proj/gcc_build/mipsel-linux-gnu/libstdc++-v3/include/experimental/filesystem:39,
 from
/home/gru/proj/gcc_build/../gcc-7.3.0/libstdc++-v3/src/filesystem/dir.cc:29,
 from
/home/gru/proj/gcc_build/../gcc-7.3.0/libstdc++-v3/src/filesystem/cow-dir.cc:26:
/home/gru/proj/gcc_build/mipsel-linux-gnu/libstdc++-v3/include/bits/stl_vector.h:
In member function 'void std::vector<_Tp,
_Alloc>::_M_move_assign(std::vector<_Tp, _Alloc>&&, std::true_type) [with
_Tp = std::experimental::filesystem::v1::path::_Cmpt; _Alloc =
std::allocator]':
/home/gru/proj/gcc_build/mipsel-linux-gnu/libstdc++-v3/include/bits/stl_vector.h:1531:9:
internal compiler error: Illegal instruction
  vector __tmp(get_allocator());
 ^
0x159a473 crash_signal
/home/gru/proj/gcc_build/../gcc-7.3.0/gcc/toplev.c:337
Please submit a full bug report,
with preprocessed source if appropriate.
Please include the complete backtrace with any bug report.
See  for instructions.
Makefile:523: recipe for target 'cow-dir.lo' failed
make[6]: *** [cow-dir.lo] Error 1


I have configured and launched the build as:

$ CPATH=/usr/include/mipsel-linux-gnu/
LIBRARY_PATH=/usr/lib/mipsel-linux-gnu/ $PWD/../gcc-7.3.0/configure
--enable-languages=c,c++ --with-float=hard --disable-multiarch
--target=mipsel-linux-gnu
$ CPATH=/usr/include/mipsel-linux-gnu
LIBRARY_PATH=/usr/lib/mipsel-linux-gnu make -j2

I've clearly misconfigured the build -- xgcc ended up with a wrong ISA, but
I can't figure out how. What am I missing?

Best regards,
Martin