Do we want to add -fsanitize=function?

[Martin Liška]

Hi.

The missing sanitizer reports about violations of function signatures
for indirect calls, like:

$ cat sanitize-function.cpp
#include 

void f() {}
void (*fnpointer) (int);

void save () {
  fnpointer = reinterpret_cast(reinterpret_cast(f));
}

int main(void) {
  save ();
  fnpointer (32);
}

$ clang++ sanitize-function.cpp -fsanitize=function -g && ./a.out
sanitize-function.cpp:12:3: runtime error: call to function f() through pointer 
to incorrect function type 'void (*)(int)'
/home/marxin/Programming/testcases/sanitize-function.cpp:3: note: f() defined 
here
#0 0x431c57 in main 
/home/marxin/Programming/testcases/sanitize-function.cpp:12:3
#1 0x7f6284994e0a in __libc_start_main 
/usr/src/debug/glibc-2.30-2.1.x86_64/csu/../csu/libc-start.c:308:16
#2 0x403349 in _start 
/home/abuild/rpmbuild/BUILD/glibc-2.30/csu/../sysdeps/x86_64/start.S:120

The sanitizer leverages the following UBSAN API:

void __ubsan_handle_function_type_mismatch_v1(FunctionTypeMismatchData *Data,
  ValueHandle Function,
  ValueHandle calleeRTTI,
  ValueHandle fnRTTI) {
}

Which is quite obvious API, except the last argument. The last argument is a 
pointer to RTTI
of a function pointer that will be used for indirect calls. Having a pointer to 
a fn, clang
emits the following sequence at the very beginning of a function:

void save () {
  431bb0:   eb 06   jmp431bb8 <_Z4savev+0x8>
  431bb2:   76 32   jbe431be6 
  431bb4:   e8 90 00 00 55  callq  55431c49 <_end+0x546d8979>
  431bb9:   48 89 e5mov%rsp,%rbp

so it jump +8 and the content of the next 8 bytes is actually a pointer to RTTI 
of this function.
That's how can one get RTTI of a fn pointer. The checking code then verifies 
that a dereferenced
function really contains the 431bb8 jump at the very beginning of a function.
The suggested approach is very target-dependent and quite hackish.

So my question is if we want the sanitizer? And second, do we have something 
similar that does
so explicit .text emission w/o GAS assistance?

Thanks,
Martin

Re: Do we want to add -fsanitize=function?

[Martin Liška]

On 1/14/20 12:54 PM, Jakub Jelinek wrote:

On Tue, Jan 14, 2020 at 12:36:11PM +0100, Martin Liška wrote:

The missing sanitizer reports about violations of function signatures
for indirect calls, like:

$ cat sanitize-function.cpp
#include 

void f() {}
void (*fnpointer) (int);

void save () {
   fnpointer = reinterpret_cast(reinterpret_cast(f));
}

int main(void) {
   save ();
   fnpointer (32);
}


_Z4savev:   # @_Z4savev
.cfi_startproc
.long   846595819   # 0x327606eb
.long   .L__unnamed_2-_Z4savev
# %bb.0:# %entry
...
seems to be what they emit on x86_64.  Now, wonder what they do on other
targets


Other targets are not supported :P


, and how does it play with all the other options that add stuff
to the start of functions, e.g. -fcf-protection=full (where it needs to
really start with endbr64 instruction)


Using the options one will get:

_Z4savev:   # @_Z4savev
.cfi_startproc
.long   846595819   # 0x327606eb
.long   .L__unnamed_2-_Z4savev
# %bb.0:
endbr64

So endbr64 is placed after the RTTI record.


, or the various options for
patcheable function entries, -mfentry, profiling and the like.


These work similarly, then follow the RTTI record:

_Z4savev:   # @_Z4savev
.cfi_startproc
.long   846595819   # 0x327606eb
.long   .L__unnamed_2-_Z4savev
# %bb.0:
callq   __fentry__

Martin



Jakub

Re: Do we want to add -fsanitize=function?

[Martin Liška]

On 1/14/20 1:59 PM, Jakub Jelinek wrote:

On Tue, Jan 14, 2020 at 01:57:47PM +0100, Martin Liška wrote:

seems to be what they emit on x86_64.  Now, wonder what they do on other
targets


Other targets are not supported :P


, and how does it play with all the other options that add stuff
to the start of functions, e.g. -fcf-protection=full (where it needs to
really start with endbr64 instruction)


Using the options one will get:

_Z4savev:   # @_Z4savev
.cfi_startproc
.long   846595819   # 0x327606eb
.long   .L__unnamed_2-_Z4savev
# %bb.0:
endbr64

So endbr64 is placed after the RTTI record.


Which is wrong, this will then fail on CET hardware.


Sure, which is a minor limitation. FCF is supposed to be production
security feature while UBSAN is more for a testing playground.

Martin



Jakub

Re: Do we want to add -fsanitize=function?

[Martin Liška]

On 1/14/20 3:00 PM, Jakub Jelinek wrote:

But then the compiler should just fail if you mix the two, rather than
emitting something that doesn't work at all.
Or better fix the design, so that it can grok an endbr64 together with
the following jump as another magic.


Sure. One can make an error when these 2 options are mixed together.
So the question still remains opened, do we want to implement the
sanitizer feature?

Martin

Re: Do we want to add -fsanitize=function?

[Martin Liška]

On 1/14/20 4:30 PM, Jakub Jelinek wrote:

On Tue, Jan 14, 2020 at 04:15:54PM +0100, Martin Liška wrote:

On 1/14/20 3:00 PM, Jakub Jelinek wrote:

But then the compiler should just fail if you mix the two, rather than
emitting something that doesn't work at all.
Or better fix the design, so that it can grok an endbr64 together with
the following jump as another magic.


Sure. One can make an error when these 2 options are mixed together.
So the question still remains opened, do we want to implement the
sanitizer feature?


IMHO not for GCC 10, for GCC 11, it really sounds too hackish, so unsure.


That's my impression as well that the selected approach is quite a hack.
So unless somebody is really interested I'm not planning to work
on that for GCC 11.

Martin


It should at least cover more than one arch and have these issues like CET
etc. discussed upstream.

Jakub

git: remote: *** The first line of a commit message should be a short description of the change, not a single word.

[Martin Liška]

Can you please remove the hook for user branches likes:

$ git push origin me/filter-non-common
Enumerating objects: 27, done.
Counting objects: 100% (27/27), done.
Delta compression using up to 16 threads
Compressing objects: 100% (14/14), done.
Writing objects: 100% (14/14), 1.77 KiB | 1.77 MiB/s, done.
Total 14 (delta 13), reused 0 (delta 0)
remote: *** The first line of a commit message should be a short description of 
the change, not a single word.
remote: error: hook declined to update refs/users/marxin/heads/filter-non-common
To git+ssh://gcc.gnu.org/git/gcc.git
 ! [remote rejected] me/filter-non-common -> 
refs/users/marxin/heads/filter-non-common (hook declined)
error: failed to push some refs to 'git+ssh://gcc.gnu.org/git/gcc.git'

Thanks,
Martin

Re: git: remote: *** The first line of a commit message should be a short description of the change, not a single word.

[Martin Liška]

On 1/21/20 6:30 PM, Jonathan Wakely wrote:

Whether they make it to trunk or not doesn't really change the fact
that a one-word message is poor. If it's only on your local machine,
do what you like. The hook only complains when such a commit is
published on gcc.gnu.org.


I would disagree here. I used 'WIP' as my commit message for a branch
that is an experimental and potentially fixes a PR. GCC git is a central
point for my setup, I need to push the branch in order to pull it from
a different machine and test it there.

Moreover, as Jason said, one can have multiple commits that will be
squashed anyway before a patch submission.

Martin

Re: GCC GSoC 2020: Call for mentors and project ideas

[Martin Liška]

On 1/15/20 11:45 PM, Martin Jambor wrote:

Therefore, first and foremost, I would like to ask all (moderately)
seasoned GCC contributors to consider mentoring a student this year and
ideally also come up with a project that they would like to lead.  I'm
collecting proposal on our wiki page


@David would you be interested in a analyzer topics? Seems to me
ideal for newcomers to come up with a static analyzer check?

Martin

Re: GSoC Questions

[Martin Liška]

On 2/5/20 7:21 AM, Nicholas Krause wrote:

Greetings Martin,

I won't be applied but it was good to see you at least got some possible ideas 
out of
my research from the make parts. Two questions as related to GSoC, in terms of
long term planning for my work:


Hello.



1. *Implement something similar to Clang's/-ftime-trace/*is in my view the most 
important project
for GCC multi-threading as having a real profiler in gcc proper is the biggest 
bottleneck.


Here I would definitely recommend perf which can identify bottleneck, as well 
as locking issues
and so on.


Finding
shared state and having heuristics is a real thorn without it. Is the goal just 
to support it as a
feature similar to Clang or is this intended to be a real profiler, as it seems 
so?


My intention was to do the same what clang does. The format is quite generic 
and can be easily
extended to support more.



2.*Create a general jobserver client/server library *If its planning to be a 
library then your
probably hooking it into the frontends for C/C++.  Not sure who the maintainer 
for that is
but is he aware of the possible changes? It may be good to either have him as 
another
mentor or someone to ask about the frontend parts unless you or Martin Liska 
have
that knowledge.


No, first consumer of that should be LTO WPA which runs parallel LTO LTRANS 
during linking.
I'm not targeting any hooking into FEs.

P.S. Please CC me next time if you speak about me ;)

Thanks,
Martin



Regards and good luck with GSoC,

Nick**

Re: Request for better syntax checking in lang.opt

[Martin Liška]

On 2/20/20 8:29 AM, Thomas Koenig wrote:

Hi,

having just lost a few hours on a space in lang.opt where
there was supposed to be none, leading to a new option
being silently ignored, a request:


Hello.

Sure, I can improve sanity checking. What exactly have you screwed up?



Would it be possible to improve the syntax checking for lang.opt?
It's a file that people touch only rarely, so it is likely that
they will have no experience or will already have forgotten the
gotchas they encountered the last time.


Fully agree with that!
Martin



Regards

 Thomas

Re: Request for better syntax checking in lang.opt

[Martin Liška]

On 2/20/20 7:08 PM, Thomas Koenig wrote:

Hi Martin,


Sure, I can improve sanity checking.


Thanks!


What exactly have you screwed up?

I had, in lang.opt

EnumValue
Enum (gfc_fcoarray) String (native) Value (GFC_FCOARRAY_NATIVE)

It was a bit non-obvious to me that this led to the whole sub-option
being ignored due to

*drum roll*

the space between the keywords and the opening parenthesis.  I have
internalized the GNU style guides to such an extent that I hardly
ever see the space there :-)


Hello.

I was able to write a sanity check for these kind of issues, but it does
not resolve all similar issues for other keywords. It's not easy to do it.

Martin



Regards

 Thomas


>From 440fda0ccd2211cfd0478f50cc20d0969fe8bce0 Mon Sep 17 00:00:00 2001
From: Martin Liska 
Date: Fri, 21 Feb 2020 10:40:57 +0100
Subject: [PATCH] Make more sanity checks for enums.

---
 gcc/opt-functions.awk | 13 +
 gcc/opt-read.awk  | 10 +-
 2 files changed, 18 insertions(+), 5 deletions(-)

diff --git a/gcc/opt-functions.awk b/gcc/opt-functions.awk
index 2f0442dc563..be4b9e66165 100644
--- a/gcc/opt-functions.awk
+++ b/gcc/opt-functions.awk
@@ -72,6 +72,19 @@ function opt_args(name, flags)
 	return flags
 }
 
+# If FLAGS contains a "NAME(...argument...)" flag, return the value
+# of the argument.  Print error message otherwise.
+function opt_args_non_empty(name, flags, description)
+{
+	args = opt_args(name, flags)
+	if (args == "")
+	{
+		print "Empty option argument '" name "' during parsing of: " flags >> "/dev/stderr"
+		exit 1
+	}
+	return args
+}
+
 # Return the Nth comma-separated element of S.  Return the empty string
 # if S does not contain N elements.
 function nth_arg(n, s)
diff --git a/gcc/opt-read.awk b/gcc/opt-read.awk
index a2e16f29aff..9bb9dfcf6ca 100644
--- a/gcc/opt-read.awk
+++ b/gcc/opt-read.awk
@@ -81,8 +81,8 @@ BEGIN {
 		}
 		else if ($1 == "Enum") {
 			props = $2
-			name = opt_args("Name", props)
-			type = opt_args("Type", props)
+			name = opt_args_non_empty("Name", props)
+			type = opt_args_non_empty("Type", props)
 			unknown_error = opt_args("UnknownError", props)
 			enum_names[n_enums] = name
 			enum_type[name] = type
@@ -93,9 +93,9 @@ BEGIN {
 		}
 		else if ($1 == "EnumValue")  {
 			props = $2
-			enum_name = opt_args("Enum", props)
-			string = opt_args("String", props)
-			value = opt_args("Value", props)
+			enum_name = opt_args_non_empty("Enum", props)
+			string = opt_args_non_empty("String", props)
+			value = opt_args_non_empty("Value", props)
 			val_flags = "0"
 			val_flags = val_flags \
 			  test_flag("Canonical", props, "| CL_ENUM_CANONICAL") \
-- 
2.25.0

Re: Request for better syntax checking in lang.opt

[Martin Liška]

On 2/23/20 10:47 AM, Thomas Koenig wrote:

Hi Martin,


I was able to write a sanity check for these kind of issues, but it does
not resolve all similar issues for other keywords. It's not easy to do it.


Having looked at the origina awk code, I agree.  Maybe, in the long
term, a lex/yacc grammar with a monolithic C program to write out
the headers wold be more suitable.


... or as I suggested, we could use Python. But that was not welcomed
by the community as unnecessary dependency.



Having said that, I think what you did is already quite valuable
and will save some gcc developers from a few prmature grey hairs :-)


Anyway, I'll send the patch in the next stage1.

Martin



So, I would recommend to commit as is.  Sanity checks do not have
to be perfect.

Thanks for taking this on!

Regards

 Thomas

Re: Make LTO Patch for Job Server Thread Detection Agnostic

[Martin Liška]

On 2/27/20 9:44 AM, Jonathan Wakely wrote:

Martin, the comment in your patch says "-std=c11" which should be c++11.


Sure, thanks for pointing to it.

I'm going to install the obvious patch for it.
Martin
>From 3c633731ade4cdad56f5729eaedb76cb2f727c39 Mon Sep 17 00:00:00 2001
From: Martin Liska 
Date: Mon, 2 Mar 2020 11:03:11 +0100
Subject: [PATCH] Fix typo in C++ standard version.

gcc/ChangeLog:

2020-03-02  Martin Liska  

	* lto-wrapper.c: Fix typo in comment about
	C++ standard version.
---
 gcc/lto-wrapper.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/gcc/lto-wrapper.c b/gcc/lto-wrapper.c
index 6e3f294257e..b8a35c85714 100644
--- a/gcc/lto-wrapper.c
+++ b/gcc/lto-wrapper.c
@@ -1234,7 +1234,7 @@ init_num_threads (void)
 #endif
 }
 
-/* FIXME: once using -std=c11, we can use std::thread::hardware_concurrency.  */
+/* FIXME: once using -std=c++11, we can use std::thread::hardware_concurrency.  */
 
 /* Return true when a jobserver is running and can accept a job.  */
 
-- 
2.25.1

Re: GSoC topic: Implement hot cold splitting at GIMPLE IR level

[Martin Liška]

Hello.

Thank you for idea. I would like to provide some comments about what GCC can 
currently
do and I'm curious we need something extra on top of what we do.
Right now, GCC can do hot/cold partitioning based on functions and basic 
blocks. With
a PGO profile, the optimization is quite aggressive and can save quite some code
being placed into a cold partitioning and being optimized for size. Without a 
profile,
we do a static profile guess (predict.c), where we also propagate information 
about cold
blocks (determine_unlikely_bbs). Later in RTL, we utilize the information and 
make
the real reordering (bb-reorder.c).

Martin

GCC bugzilla: REST API

[Martin Liška]

Hi.

I'm working on a script that will catch the missing email into
Bugzilla that are triggered by git commits mentioning a PR.
For that I would need the enablement of REST API that was enabled
in previous bugzilla instance.

Thank you,
Martin

Re: GCC bugzilla: REST API

[Martin Liška]

On 3/12/20 12:15 AM, Frank Ch. Eigler wrote:

Hi -


I'm working on a script that will catch the missing email into
Bugzilla that are triggered by git commits mentioning a PR.
For that I would need the enablement of REST API that was enabled
in previous bugzilla instance.


I believe this should work now.  Thanks to Joseph Myers for
pointing out that we were missing some more perl modules.
Fought with koji & etc. awhile and now bugzilla's checksetup.pl
is happy all around.


Hello.

Thank you for that. I can confirm it works fine now.

Martin



- FChE

Re: commits in Bugzilla attributed to others?

[Martin Liška]

On 3/13/20 5:03 PM, Marek Polacek wrote:

On Fri, Mar 13, 2020 at 09:56:58AM -0600, Martin Sebor wrote:

On 3/13/20 9:50 AM, Marek Polacek wrote:

On Fri, Mar 13, 2020 at 09:46:40AM -0600, Martin Sebor via Gcc wrote:

It looks as though commits with bug fixes appear in Bugzilla comments
made by others(*).  Fox instance, commit r10-7151 for PR 92071 shows
in comment #16 on the bug under Martin Liška's name.

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=92071#c16

Similarly, commits r9-8372, r10-7152, and r8-10121 show up in PR 94119
in comments with Martin's name on them:

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94119

and likewise here:

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=91913
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94163
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94154
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94063

Is this a known problem?

[*] I haven't seen this for others so maybe Martin's account got messed
up in the migration somehow?


The authors of those commits are correct, it's just that the commit-to-Bugzilla
sync is broken at the moment and Martin L. was doing it manually.


I see.  Thanks for the explanation (and to Martin for all the work
to patch up the problem while it's being solved)!


And I should've said "has been doing it manually" because I don't think it's
been fixed yet :).


Hello.

Thank you. I've just pasted last missing pieces and the service should be 
working
fine right now.

Martin



And yes, thanks Martin!

Marek

Not usable email content encoding

[Martin Liška]

Hello.

I noticed some emails reaching gcc-patc...@gcc.gnu.org use the following 
quoting:

```
 case UNGT_EXPR:
 case UNGE_EXPR:
 case UNEQ_EXPR:
+case MEM_REF:
   /* Binary operations evaluating both arguments (increment and
 =09 decrement are binary internally in GCC).  */
   orig_op0 =3D op0 =3D TREE_OPERAND (expr, 0);
@@ -435,6 +436,14 @@ c_fully_fold_internal (tree expr, bool i
 =09  || TREE_CODE (TREE_TYPE (orig_op0)) =3D=3D FIXED_POINT_TYPE)
 =09  && TREE_CODE (TREE_TYPE (orig_op1)) =3D=3D INTEGER_TYPE)
 =09warn_for_div_by_zero (loc, op1);
+  if (code =3D=3D MEM_REF
+=09  && ret !=3D expr
+=09  && TREE_CODE (ret) =3D=3D MEM_REF)
+=09{
+=09  TREE_READONLY (ret) =3D TREE_READONLY (expr);
+=09  TREE_SIDE_EFFECTS (ret) =3D TREE_SIDE_EFFECTS (expr);
+=09  TREE_THIS_VOLATILE (ret) =3D TREE_THIS_VOLATILE (expr);
+=09}
```

It's probably related to the following email tag:
Content-Transfer-Encoding: quoted-printable

The format is problematic when copying a patch.
Email example:
https://gcc.gnu.org/pipermail/gcc-patches/2020-March/542053.html

Thank you,
Martin

Re: Not usable email content encoding

[Martin Liška]

On 3/16/20 2:47 PM, Richard Earnshaw wrote:

This isn't new.  It's usually the sender that controls that, though it's often 
automatic in their email client.


Jakub told me that he sees:

Ten mail byl:
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

while the email reaching the mailing list has:

Content-Type: text/plain; charset=WINDOWS-1252
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

Martin

Re: Not usable email content encoding

[Martin Liška]

On 3/16/20 2:54 PM, Alexander Monakov wrote:

Are you trying to copy from the raw message representation?


Exactly.

Martin

Re: subversion status on gcc.gnu.org

[Martin Liška]

Hi.

The suggested approach expect that the revision belongs to master
branch. Which is not the case for branch commits like:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=91601#c11

Link:
https://gcc.gnu.org/viewcvs?rev=275301&root=gcc&view=rev

Thank you,
Martin

Re: Question about git: merging to gccgo branch

[Martin Liška]

On 1/22/20 7:38 PM, Joseph Myers wrote:

On Wed, 22 Jan 2020, Ian Lance Taylor wrote:


I don't want to send 581 e-mails.  I would be happy not sending any
e-mails at all.  I would also be happy sending 1 e-mail.


This is the issue we've discussed in
 and the messages linked
from there.

Until we have a proper fix not to send emails for the commits being
merged, I've increased that limit to 1000 for GCC, as it doesn't seem
useful to make people split the commits merged into batches of at most
100.  I've also disabled the check for the default commit message for
merge commits, since it seems that wasn't being useful in practice for
branches where merges are appropriate and a separate check prevents merge
commits on master and release branches.



Hello.

I see the same for my user branch:

$ git push origin me/marxin-gcc-benchmark-branch
Enumerating objects: 4, done.
Counting objects: 100% (4/4), done.
Delta compression using up to 16 threads
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 729 bytes | 729.00 KiB/s, done.
Total 2 (delta 1), reused 0 (delta 0), pack-reused 0
remote: *** This update introduces too many new commits (1517), which would
remote: *** trigger as many emails, exceeding the current limit (1000).
remote: *** Contact your repository adminstrator if you really meant
remote: *** to generate this many commit emails.
remote: error: hook declined to update 
refs/users/marxin/heads/marxin-gcc-benchmark-branch
To git+ssh://gcc.gnu.org/git/gcc.git
 ! [remote rejected] me/marxin-gcc-benchmark-branch -> 
refs/users/marxin/heads/marxin-gcc-benchmark-branch (hook declined)
error: failed to push some refs to 'git+ssh://gcc.gnu.org/git/gcc.git'

Can you please disable email sending for user branch? Or does it make any sense?

Thanks,
Martin

Re: Question about git: merging to gccgo branch

[Martin Liška]

On 3/30/20 7:54 PM, Joseph Myers wrote:

Email sending for user branches makes perfect sense, to make visible the
development going on.


If so, can we then come up with a naming convention that will block the email
sending. I mean something like me/PR123456-fix-that-private?
I see the limitation really baseless.

Martin

Re: subversion status on gcc.gnu.org

[Martin Liška]

On 4/2/20 2:26 PM, Jakub Jelinek via Gcc wrote:

On Thu, Apr 02, 2020 at 02:19:10PM +0200, Georg-Johann Lay wrote:

Am 20.03.20 um 18:37 schrieb Frank Ch. Eigler via Gcc:

Hi -

Both svn: and ssh+svn: now work for your archeological needs.
Further, URLs such as

https://gcc.gnu.org/viewcvs?rev=279160&root=gcc&view=rev
https://gcc.gnu.org/r123456

are mapped to gitweb searches that try to locate the matching
From-SVN: rABCDEF commit.  This way, historical URLs from bugzilla
should work.


This does not work as expected.  For example,

https://gcc.gnu.org/r2000

maps to a search and you get thousands of hits for SVN: r2000**


The gcc git svn-rev alias handles that (and also handles release and other
branches) using approx.
git log --all --grep="From-SVN: r$rev\b" | head -n 1 | awk '{print $2}'
where $rev is 2000 in your case.

Jakub



Btw. what about statically generated database (files) which can be used for 
redirection?
I'm sending an example and I have a script that can generate any file content.

Martin


svn-to-git-mapping.tar.bz2
Description: application/bzip

Re: subversion status on gcc.gnu.org

[Martin Liška]

On 4/3/20 10:54 AM, Jakub Jelinek wrote:

On Fri, Apr 03, 2020 at 10:38:17AM +0200, Martin Liška wrote:

The gcc git svn-rev alias handles that (and also handles release and other
branches) using approx.
git log --all --grep="From-SVN: r$rev\b" | head -n 1 | awk '{print $2}'
where $rev is 2000 in your case.

Jakub



Btw. what about statically generated database (files) which can be used for 
redirection?
I'm sending an example and I have a script that can generate any file content.


That can work too, but do we want a directory with 280156 files?


It's not ideal :) Maybe a modern linux filesystem will not suffer so much?
Oversees people can tell what they want.


 One way is
to split the number into halves and have files like whatever/280/151 (and
deal specially with the revisions < 1000 where it would use whatever/0/123 ).


That would be possible.


Or it can be a database query.


I can imagine a simple sqlite file that can be queried.
Basically all is possible.

Martin



Jakub

Re: subversion status on gcc.gnu.org

[Martin Liška]

On 4/3/20 11:23 AM, Martin Liška wrote:

On 4/3/20 10:54 AM, Jakub Jelinek wrote:

On Fri, Apr 03, 2020 at 10:38:17AM +0200, Martin Liška wrote:

The gcc git svn-rev alias handles that (and also handles release and other
branches) using approx.
git log --all --grep="From-SVN: r$rev\b" | head -n 1 | awk '{print $2}'
where $rev is 2000 in your case.

Jakub



Btw. what about statically generated database (files) which can be used for 
redirection?
I'm sending an example and I have a script that can generate any file content.


That can work too, but do we want a directory with 280156 files?


It's not ideal :) Maybe a modern linux filesystem will not suffer so much?
Oversees people can tell what they want.


 One way is
to split the number into halves and have files like whatever/280/151 (and
deal specially with the revisions < 1000 where it would use whatever/0/123 ).


That would be possible.


Or it can be a database query.


I can imagine a simple sqlite file that can be queried.
Basically all is possible.

Martin



Jakub





I've got it, we want to use:
http://httpd.apache.org/docs/trunk/rewrite/rewritemap.html#dbm

which allows using a map file with additional index. There's svn-to-git.txt 
file:
https://drive.google.com/file/d/1xZ7TwvqcV6zErup920OLhb09tePgSJHQ/view?usp=sharing

that should be transformed by:
$ httxt2dbm -i svn-to-git.txt -o svn-to-git.map

Martin

Re: text/x-* attachments stripped (was: Re: gcc ML archive: text/x-patch attachments no longer shown inline (was:Re: Mailing list stripping off attachments))

[Martin Liška]

On 3/9/20 11:25 AM, Jakub Jelinek wrote:

On Mon, Mar 09, 2020 at 10:46:31AM +0100, Tobias Burnus wrote:

Hi Thomas, hi Overseers

I can confirm that those are stripped off!

I did sent an email with three attachments:
* test.txt (text/plain)
* test.diff (text/x-diff)
* the company's disclaimer

The attachment with 'text/x-diff' MIME was removed :-(
See: https://gcc.gnu.org/pipermail/fortran/current/054078.html


A different mail archiver is now used it seems.
For the mails before the sourceware move, one can access the old one too,
e.g.
https://gcc.gnu.org/legacy-ml/gcc-bugs/2020-03/
is the old one vs.
https://gcc.gnu.org/pipermail/gcc-bugs/2020-March/
I've been using the gcc-bugs mail archive all the time in the past, but I'm
afraid pipermail at least in current configuration is significant step back
and I'll likely just use my mailbox from now on.
Some reasons:
1) the by date monthly list of mails used to be ordered newest to oldest
mails first, now it is oldest to newest, so when dealing with new stuff one
has to always scroll down
2) the dates and times of mails used to be shown (date as a section in the
list, times in the left column), now there is nothing, so without clicking
something it is hard to guess how exactly old it is
3) the columns were nicer (date, subject left justified, email right
justified, now there are no columns)
4) some headers were shown, now there is nothing
5) emails used to be sanitized against harvesters, now they aren't
6) there used to be a Raw text URL to grab the raw email, now there is nothing

Jakub



Hello.

I agree with Jakub that the listed feature were nicer about the previous mail 
list
archiver. On the other hand, I agree that we want to use something more recent 
that
is support and under some development. That said, we did we decide to use 
mailman-2.1
which is a legacy release that can be shortly out of support? Have you consider
using version 3.3.0?

I'm willing to customize the mail archiver, it should be quite simple.
Would it be possible to apply local patches for a RHEL package that's install
on the system?

Thanks,
Martin

mailman customization

[Martin Liška]

Hello.

I believe we can quite easily customize mailman 2.1 to match our needs.
The biggest challenge I see is a proper testing as I don't see it easy
to set up a local mailman instance. I've got a patch that changes:

- by date sorting will be done in reverse order
- default link of e.g. https://gcc.gnu.org/pipermail/gcc-patches/2020-April/ 
will
  point to sorting by date
- email date is added to the listing

Further changes would be possible but I'll need a cooperation from oversees 
people.

Thanks,
Martin
diff --git a/Mailman/Archiver/HyperArch.py b/Mailman/Archiver/HyperArch.py
index 4469193..2e186ff 100644
--- a/Mailman/Archiver/HyperArch.py
+++ b/Mailman/Archiver/HyperArch.py
@@ -637,7 +637,7 @@ class HyperArchive(pipermail.T):
 FILEMODE = 0660
 
 VERBOSE = 0
-DEFAULTINDEX = 'thread'
+DEFAULTINDEX = 'date'
 ARCHIVE_PERIOD = 'month'
 
 THREADLAZY = 0
diff --git a/Mailman/Archiver/HyperDatabase.py b/Mailman/Archiver/HyperDatabase.py
index 2475d47..3566425 100644
--- a/Mailman/Archiver/HyperDatabase.py
+++ b/Mailman/Archiver/HyperDatabase.py
@@ -71,7 +71,7 @@ class DumbBTree:
 def __sort(self, dirty=None):
 if self.__dirty == 1 or dirty:
 self.sorted = self.dict.keys()
-self.sorted.sort()
+self.sorted.sort(reverse = self.path.endswith('date'))
 self.__dirty = 0
 
 def lock(self):
diff --git a/templates/en/archidxentry.html b/templates/en/archidxentry.html
index f9bb57a..365e836 100644
--- a/templates/en/archidxentry.html
+++ b/templates/en/archidxentry.html
@@ -1,3 +1,5 @@
+%(datestr)s
+
 %(subject)s
  
 %(author)s

Re: mailman customization

[Martin Liška]

On 4/3/20 5:54 PM, Frank Ch. Eigler wrote:

Hi -


I believe we can quite easily customize mailman 2.1 to match our needs.
The biggest challenge I see is a proper testing as I don't see it easy
to set up a local mailman instance. I've got a patch that changes:


I suppose we can do some local RPM respins - as long as these changes
are small and rare.


That would be great. Should I create a git repo where we'll stack these changes?


 Even with a deadish upstream, distro reporting
would be nice, at least at the centos/fedora point (?), as a reference
place to stash the patch and get us a bug#.


Can you please do it for me as I don't have any experience with Fedora
packaging?

Thank you,
Martin



- FChE

Re: subversion status on gcc.gnu.org

[Martin Liška]

On 4/3/20 10:19 PM, Frank Ch. Eigler wrote:

Hi -


https://gcc.gnu.org/r2000
maps to a search and you get thousands of hits for SVN: r2000**


The gcc git svn-rev alias handles that (and also handles release and other
branches) using approx.
git log --all --grep="From-SVN: r$rev\b" | head -n 1 | awk '{print $2}'
where $rev is 2000 in your case.


I'll look into fixing the forwarding regexp.


Btw. what about statically generated database (files) which can be
used for redirection?  I'm sending an example and I have a script
that can generate any file content.


I don't think we will require something that elaborate.


Well, I still believe we want to use the map file. Reason is that
the gitweb redirection can't deal with SVN revision that don't belong
to trunk.

Thanks,
Martin



- FChE

Re: subversion status on gcc.gnu.org

[Martin Liška]

On 4/6/20 1:57 AM, Frank Ch. Eigler wrote:

Hi -

Courtesy of a lovely httpd RewriteMap-basd hack courtesy of Martin, we
have all the svn r# redirects working, and faster than before.


Great. Thank you application of the RewriteMap!

Martin



- FChE

Re: subversion status on gcc.gnu.org

[Martin Liška]

On 4/6/20 10:37 AM, Jakub Jelinek wrote:

On Mon, Apr 06, 2020 at 10:09:24AM +0200, Martin Liška wrote:

On 4/6/20 1:57 AM, Frank Ch. Eigler wrote:

Hi -

Courtesy of a lovely httpd RewriteMap-basd hack courtesy of Martin, we
have all the svn r# redirects working, and faster than before.


Great. Thank you application of the RewriteMap!


E.g. https://gcc.gnu.org/r105377 or https://gcc.gnu.org/r12345
don't work.


These look not valid by svn-rev:

$ git svn-rev 105377 | wc -l
0
$ git svn-rev 12345 | wc -l
0

Martin



Jakub

Re: subversion status on gcc.gnu.org

[Martin Liška]

On 4/6/20 10:55 AM, Jakub Jelinek wrote:

On Mon, Apr 06, 2020 at 10:46:34AM +0200, Martin Liška wrote:

On 4/6/20 10:37 AM, Jakub Jelinek wrote:

On Mon, Apr 06, 2020 at 10:09:24AM +0200, Martin Liška wrote:

On 4/6/20 1:57 AM, Frank Ch. Eigler wrote:

Hi -

Courtesy of a lovely httpd RewriteMap-basd hack courtesy of Martin, we
have all the svn r# redirects working, and faster than before.


Great. Thank you application of the RewriteMap!


E.g. https://gcc.gnu.org/r105377 or https://gcc.gnu.org/r12345
don't work.


These look not valid by svn-rev:

$ git svn-rev 105377 | wc -l
0
$ git svn-rev 12345 | wc -l
0


Dunno about the latter, but the former then looks like a repo conversion
bug:
https://gcc.gnu.org/legacy-ml/gcc-cvs/2005-10/msg01053.html
(that is the first commit to svn after cvs conversion).


I hope we can live with that one missing :) The date of the commit seems 
suspicious:

commit d3e5a995ec5ceb2c474dd52b9d8558265187829a
Author: Daniel Berlin 
Date:   Fri Oct 28 13:21:13 2005 +

SVN was not moved to SVN :)

From-SVN: r105929


while:
r105377 - /trunk/gcc/DATESTAMP
Date: Thu Oct 27 14:59:55 2005

Based on the SVN revision it should live somewhere:

commit 6c06fbce5c77ffbb0be281d22780a919de4877fe
Author: Mark Mitchell 
Date:   Thu Oct 13 23:59:57 2005 +

re PR c++/20721 (crossing of a initialization left undetected on goto)

PR c++/20721

* cp-tree.h (DECL_NONTRIVIALLY_INITIALIZED_P): New macro.
* decl.c (duplicate_decls): Merge it into new declarations.
(decl_jump_unsafe): Use it, rather than DECL_INITIAL.
(cp_finish_decl): Set it, when appropriate.
PR c++/20721
* g++.dg/init/goto2.C: New test.

From-SVN: r105380


commit 02f3e085c7ba33279329aae728aaf42dc922add6
Author: Andrew Haley 
Date:   Thu Oct 13 17:36:07 2005 +

re PR java/24251 (BC-compiled interfaces in libgcj can't be called from 
non-BC code)

2005-10-12  Andrew Haley  

PR java/24251

* link.cc (ensure_method_table_complete): Install Miranda methods
for interfaces too.

From-SVN: r105375


Martin



Jakub

Re: subversion status on gcc.gnu.org

[Martin Liška]

On 4/6/20 5:04 PM, Joseph Myers wrote:

On Mon, 6 Apr 2020, Andrew Pinski via Gcc wrote:


That is r105377 till r105390 was only ever done on a test SVN repo and
r105927 (hooks) was the first commit to SVN after the conversion from


Actually r105926 (creating the hooks directory) was the first commit in
the real SVN conversion.  But in addition to SVN hooks deliberately not
being converted to git, commits that only create / remote empty
directories generally weren't converted, as git doesn't represent empty
directories so such commits would be empty, and thus not idiomatically
present at all, in git.

As for the question about r12345 (cvs2svn-generated creation of a tag that
was deleted shortly after the move from CVS to SVN), it's in git as commit
229098288e4883d3b78400650e0b4143e12d0a76.


Hi.

You are right, one can really find the git commit:
https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=229098288e4883d3b78400650e0b4143e12d0a76

?  Given a mirror of the full

repository, refs/deleted/r106023.256/tags/libc-960701 points to that
commit - it's not a very useful commit, but it's there if you have the
full repository.


Can you please help me how can one clone (or pull) complete content of git repo?


 Maybe the RewriteMap was generated based on the
fetched-by-default parts of the repository and should be regenerated based
on a mirror clone to get everything converted from an SVN commit to a git
commit?


Yes, it was based on normal git repo one can get from 
git://gcc.gnu.org/git/gcc.git.

Thanks,
Martin

Re: subversion status on gcc.gnu.org

[Martin Liška]

On 4/7/20 9:22 AM, Andreas Schwab wrote:

On Apr 07 2020, Martin Liška wrote:


Can you please help me how can one clone (or pull) complete content of git repo?


Easiest way is to do a mirror clone.


All right, --mirror is a new option to me.

Doing that.
Thanks,
Martin



Andreas.

Re: subversion status on gcc.gnu.org

[Martin Liška]

On 4/7/20 9:29 AM, Martin Liška wrote:

On 4/7/20 9:22 AM, Andreas Schwab wrote:

On Apr 07 2020, Martin Liška wrote:


Can you please help me how can one clone (or pull) complete content of git repo?


Easiest way is to do a mirror clone.


All right, --mirror is a new option to me.


There's update SVN to GIT map file:
https://drive.google.com/file/d/1DMuFDu476stLdMxKSaDzv4c81rhMAGEI/view?usp=sharing

It increased number of SVN revisions from 227979 to 275662.

@Frank: Can you please update the MapFile with this one?
Thanks,
Martin



Doing that.
Thanks,
Martin



Andreas.

Re: Questions about code instrumentation in FDO

[Martin Liška]

On 4/7/20 2:01 PM, Erick Ochoa wrote:

Hello,

Can someone help me understand better GCC's profile driven instrumentation? I 
know this can be a long topic, but I am not looking for a long discussion. I am 
just trying to orient myself regarding GCC's FDO implementation.


Hello.

Sure.



1. I know that gcc uses an instrumentation based profiler. This instrumentation based profiler instruments binaries when the flag `-fprofile-use` is used. 


Yes, one get instrumented binary with -fprofile-generate.

I also know that `gprof` is also a gnu profiler that can also generate 
profiling information that can be consumed by gcc.

No, it's designed for a third-party consumers, where intrumented binary calls 
mcount function at the
very beginning of each function. Also linux kernel uses that for various 
reasons.


Does the profiler used when the flag `-fprofile-use` share some code with 
`gprof`? Are they the same?


No.



2. I know that the profiler used with `-fprofile-use` can generate basic block 
frequencies and branch probabilities. Does it also profile the call graph?


Yes, function entry block execution count is equal to number of calls of the 
function. Similarly for calls, we
known how many times each cgraph edge is executed.


I am specifically interested in callsite edge counters with a context 
sensitivity of k = 1. This is to say that if I have the following method:

```
int guard(void* (func_ptr)(void))
{
   if (func_ptr == some_function)
   {
     func_ptr(); // call site 1
   } else {
     func_ptr(); // call site 2
   }
}
```

profiling information should be enough to say the target of the call site 1 is 
always some_function and the target of the call site 2 can be any function 
which was stored in func_ptr that is not some_function.


Yes, that's exactly what speculative devirtualization for C++ does. How do you 
want to leverage
information that func_ptr != some_function?




3. I know that there are other profilers that are able to generate data that can be consumed by GCC. In particular there is AutoFDO. 


Yes, but the infrastructure is basically broken right now. It was developed by 
Google people who moved to Clang right now.

According to the options available on GCC, this is still an option available. 
Are there any other profilers that you know about that can generate profile 
data used by GCC?

No.

Martin

Re: subversion status on gcc.gnu.org

[Martin Liška]

On 4/7/20 4:09 PM, Frank Ch. Eigler wrote:

Hi -


There's update SVN to GIT map file:
https://drive.google.com/file/d/1DMuFDu476stLdMxKSaDzv4c81rhMAGEI/view?usp=sharing


Updated.


Thanks. I can form https://gcc.gnu.org/r12345 works right now.

Martin



- FChE

Re: Martin Liska appointed GCC IPA Reviewer

[Martin Liška]

On 4/13/20 9:54 PM, David Edelsohn wrote:

I am pleased to announce that the GCC Steering Committee has
appointed Martin Liska as IPA Reviewer.

Please join me in congratulating Martin on his new role.
Martin, please update your listing in the MAINTAINERS file.

Happy hacking!
David



Hello

Thank you for the trust!
I've just updated MAINTAINERS files accordingly.

Martin

Re: GCC optimizations with O3

[Martin Liška]

On 4/22/20 4:01 PM, Erick Ochoa wrote:

Hello,

Does anyone know if the following text from the GCC internals [0] is outdated?

-O3

     Optimize yet more. -O3 turns on all optimizations specified by -O2 and 
also turns on the following optimization flags:

     -fgcse-after-reload
     -fipa-cp-clone
     -floop-interchange
     -floop-unroll-and-jam
     -fpeel-loops
     -fpredictive-commoning
     -fsplit-paths
     -ftree-loop-distribute-patterns
     -ftree-loop-distribution
     -ftree-loop-vectorize
     -ftree-partial-pre
     -ftree-slp-vectorize
     -funswitch-loops
     -fvect-cost-model
     -fversion-loops-for-strides

I ask because I have an optimization pass that interacts poorly with -O3, but 
compiles and runs correctly with -O2.
I'm trying to find which individual (or combination of optimizations) is 
triggering the runtime error I'm debugging.
I tried running -O2 and manually enable all the optimizations turned on by -O3 
(but without specifying -O3) and my pass runs successfully.


Note that using -O2 and enabling all options individually is not equal to -O3. 
There are some global decisions
done based on optimization level.

For a difference, you can do:

$ gcc --help=optimize -Q -O2 > /tmp/O2
$ gcc --help=optimize -Q -O3 > /tmp/O3

$ diff -U 0 /tmp/O2 /tmp/O3
--- /tmp/O2 2020-04-22 16:19:54.123495072 +0200
+++ /tmp/O3 2020-04-22 16:19:56.903467468 +0200
@@ -54 +54 @@
-  -fgcse-after-reload  [disabled]
+  -fgcse-after-reload  [enabled]
@@ -73 +73 @@
-  -fipa-cp-clone   [disabled]
+  -fipa-cp-clone   [enabled]
@@ -103 +103 @@
-  -floop-interchange   [disabled]
+  -floop-interchange   [enabled]
@@ -106 +106 @@
-  -floop-unroll-and-jam[disabled]
+  -floop-unroll-and-jam[enabled]
@@ -122 +122 @@
-  -fpeel-loops [disabled]
+  -fpeel-loops [enabled]
@@ -126 +126 @@
-  -fpredictive-commoning   [disabled]
+  -fpredictive-commoning   [enabled]
@@ -178,2 +178,2 @@
-  -fsplit-loops[disabled]
-  -fsplit-paths[disabled]
+  -fsplit-loops[enabled]
+  -fsplit-paths[enabled]
@@ -215 +215 @@
-  -ftree-loop-distribution [disabled]
+  -ftree-loop-distribution [enabled]
@@ -220 +220 @@
-  -ftree-loop-vectorize[disabled]
+  -ftree-loop-vectorize[enabled]
@@ -223 +223 @@
-  -ftree-partial-pre   [disabled]
+  -ftree-partial-pre   [enabled]
@@ -230 +230 @@
-  -ftree-slp-vectorize [disabled]
+  -ftree-slp-vectorize [enabled]
@@ -242 +242 @@
-  -funswitch-loops [disabled]
+  -funswitch-loops [enabled]
@@ -249,2 +249,2 @@
-  -fvect-cost-model=[unlimited|dynamic|cheap]  cheap
-  -fversion-loops-for-strides  [disabled]
+  -fvect-cost-model=[unlimited|dynamic|cheap]  dynamic
+  -fversion-loops-for-strides  [enabled]

Martin



So, either:
* -O3 does more than just enable these optimizations, or
* This documentation is outdated and there's something missing.

Does someone have an answer? (And if possible point to some code locations so 
that I can learn where it is.) Thanks!

[0] https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html

C-Vise project introduction

[Martin Liška]

Hello.

I would like to announce a new Python fork of the C-Reduce tool:
https://github.com/marxin/cvise

I'm going to start with a bit of history and motivation behind the project.
All started 6 years ago when I joined SUSE as a toolchain developer and I 
quickly
became a friend with C-Reduce that I've been using for reduction of the GCC 
compiler
test-cases. No having enough numbers, but I claim I've used the tool probably
one thousand times and I must be one the biggest users of the tool. I wanted
to contribute to the project since then but I'm not much familiar with Perl 
language.
There's a pending attempt of Moritz who rewrote the tool to Python language 4 
year ago
and he hasn't received a proper review. I feel the original authors don't like 
the language
and when he announced a final comparable port (2 years ago) there was still no 
feedback.
I took his work and now I'm offering the port to the community with the 
following improvements:

1) C-Vise is a super-parallel implementation where all passes run their 
transform and test script
   phase in parallel; that resolves #160; I've seen significant speed up on a 
16-core Zen CPU
2) the project contains native BinaryState object which can help with fast 
traversal of a state space
3) Python is hopefully more popular language than Perl

Minor improvements:
- --log-level option - one can print different level of messages
- --remove-pass - one can skip a pass in pass manager (#200, #202)
- --list-passes - listing of existing passes (#150)
- project contains unit tests for some passes
- LLVM 9 and 10 support (#193, #204)
- --clang-delta-std option - workaround for #210
- unifdef and delta subprojects are removed from the repository
- --timing-since-start - print timestamps since the start of a reduction

Known limitations:
- configure & make build support is dropped
- FreeBSD and Windows port is not tested
- dependency on python-Pebble package
- a small issue observed in Pebble (noxdafox/pebble#58)

I'm planning to create a proper Github release soon and I've done an openSUSE 
package:
https://build.opensuse.org/package/show/home:marxin:devel:tools/cvise

I would like to thank the original authors and Moritz for the port.
I'm planning to share the codebase of clang_delta (and clex) which is an 
essential
part of each C/C++ test-case reduction process.

Feel free to test the project and come up with a feedback.

Martin

Automatically generated ChangeLog files - script

[Martin Liška]

Hello.

Based on the initial discussion with Richi and Jakub, we would like to remove
need to update ChangeLog files with each git commit. Instead, a script will be
used to extract ChangeLog entries from a git commit messages.

I've started with extraction of ~2K commits since we switched to git (via git 
format-patch).
The script [1] parses majority of the git commits and provides error message 
when a format
is not supported. We currently support rich variety of formats and I tried to 
document
that here: [2]. Analysis of the existing commits can be seen here: [3] and the 
script
can now parse 1471/2032 of git commits.

The script can also generate a final version of ChangeLog entries, e.g.:

$ ./changelog.py patches/1957-c-generic-lambda-forwarding-function-PR94546.patch
OK
-- gcc/cp/ChangeLog --
2020-04-22  Jason Merrill  

PR c++/94546
* pt.c (register_parameter_specializations): If the instantiation is
still a parameter pack, don't wrap it in a NONTYPE_ARGUMENT_PACK.
(tsubst_pack_expansion, tsubst_expr): Adjust.
-- gcc/testsuite/ChangeLog --
2020-04-22  Jason Merrill  

gcc/testsuite/g++.dg/cpp2a/lambda-generic-variadic20.C: New file.

Currently supported features can be seen here: [4]. Feel free to test the script
and suggest what can we improve. Our integration plan is to come up with
a new git hook that will first work only in a reporting mode (no commits 
refusal).
Later on, we can add the automatic generation of ChangeLog entries.

Martin

[1] https://github.com/marxin/gcc-changelog
[2] https://github.com/marxin/gcc-changelog#supported-changelog-format
[3] https://raw.githubusercontent.com/marxin/gcc-changelog/master/report.txt
[4] https://github.com/marxin/gcc-changelog/issues/1#issuecomment-620162080

Re: Automatically generated ChangeLog files - script

[Martin Liška]

On 4/30/20 3:45 PM, Jakub Jelinek wrote:

If this is what is really created, then for the new file, missing * space,
gcc/testsuite/ that shouldn't be there and missing PR c++/94546 line above
it.


I've just fixed all these, thanks!

About the 'PR c++/94546': it's mentioned in context of 'gcc/cp/ChangeLog',
so I would not add it.

Martin

Re: GCC 10.1 Release Candidate available from gcc.gnu.org

[Martin Liška]

On 4/30/20 11:21 PM, Jakub Jelinek via Gcc wrote:

The first release candidate for GCC 10.1 is available from

  https://gcc.gnu.org/pub/gcc/snapshots/10.1.0-RC-20200430/
  ftp://gcc.gnu.org/pub/gcc/snapshots/10.1.0-RC-20200430


What about naming these 10.1.0-rc1?



and shortly its mirrors.  It has been generated from git revision
r10-8080-g591d857164c37cd0bb96da2a293148e01f280e0f.


Can we also use proper git tags for it please?

Thanks,
Martin

Re: GCC 10.1 Release Candidate available from gcc.gnu.org

[Martin Liška]

On 5/1/20 10:04 AM, Jakub Jelinek wrote:

On Fri, May 01, 2020 at 09:23:33AM +0200, Martin Liška wrote:

On 4/30/20 11:21 PM, Jakub Jelinek via Gcc wrote:

The first release candidate for GCC 10.1 is available from

   https://gcc.gnu.org/pub/gcc/snapshots/10.1.0-RC-20200430/
   ftp://gcc.gnu.org/pub/gcc/snapshots/10.1.0-RC-20200430


What about naming these 10.1.0-rc1?


This is naming that has been used for year, I just keep forgetting what
exact -r argument to pass for the RCs, and the RC and date part come
from the script.  Due to the versioning conventions,
BASE-VER can't be 10.1.0 yet, because there is no release, but the filenames
in this case contain RC.


I see.




and shortly its mirrors.  It has been generated from git revision
r10-8080-g591d857164c37cd0bb96da2a293148e01f280e0f.


Can we also use proper git tags for it please?


Is it worth it?


Probably not.


RCs are something of interest just for the week at most and
then forgotten, the releases is what matters and that is properly tagged
(including signing).


Works for me,
Martin



Jakub

Re: Automatically generated ChangeLog files - script

[Martin Liška]

On 4/30/20 5:29 PM, Jakub Jelinek wrote:

In this last entry, as there were more than one ChangeLog snippets and
they had different PR lines, I wouldn't put anything into the automatically
added gcc/testsuite/ChangeLog entry.

Does this make sense?  Basically, try to do what is most likely the user
wanted.


Works for me, it's implemented right now.

Martin

Re: Automatically generated ChangeLog files - script

[Martin Liška]

Once it is enforced we'd probably disallow touching ChangeLog files in a
commit with other changed files and during the 
maintainer-scripts/update_version_git
script ought to check in in addition to the DATESTAMP updates in the same
commit also updates to all the ChangeLog files since the last DATESTAMP
commit.  That way, if there is some ChangeLog entry screw-up that gets
through the script, one could still wait until DATESTAMP update and
afterwards in a separate commit adjust the ChangeLog files.

Jakub



Hello.

I've made a refactoring of the script I tend to divide it into 3 components:
- abstract GitCommit class that can check commit message and format and
  can generate final ChangeLog entries
- integration part into the git hooks ([1]): it should get git message and 
changed
  files and process _only_ format validation step; note that our hooks use 
Python2
  and so we should get message and modified files via a subprocess execution:
  $ git log -n1 ac6eaa55a5199196ea0a25763114ce05333a14d3  --format=%b
When I implemented C++20 parenthesized initialization of aggregates
I introduced this bogus cp_unevaluated_operand check, thus disabling
...
  $ git diff 
ac6eaa55a5199196ea0a25763114ce05333a14d3~..ac6eaa55a5199196ea0a25763114ce05333a14d3
 --name-status
  M   gcc/cp/ChangeLog
  M   gcc/cp/call.c
  M   gcc/testsuite/ChangeLog
  A   gcc/testsuite/g++.dg/cpp2a/paren-init21.C

- a DATESTAMP update script - it can be a Python3 script that can utilize a 
nice git abstraction
  library: [2]; the script was added and can be tested with:
$ ./git_repo.py ~/Programming/gcc 6607bdd4c834f92fce924abdaea3405f62dc
-- gcc/ChangeLog --
2020-05-02  H.J. Lu  

PR target/93492
* cfgexpand.c (pass_expand::execute): Set crtl->patch_area_size
and crtl->patch_area_entry.
* emit-rtl.h (rtl_data): Add patch_area_size and patch_area_entry.
* opts.c (common_handle_option): Limit
function_entry_patch_area_size and function_entry_patch_area_start
to USHRT_MAX.  Fix a typo in error message.
* varasm.c (assemble_start_function): Use crtl->patch_area_size
and crtl->patch_area_entry.
* doc/invoke.texi: Document the maximum value for
-fpatchable-function-entry.
-- gcc/c-family/ChangeLog --
2020-05-02  H.J. Lu  

PR target/93492
* c-attribs.c (handle_patchable_function_entry_attribute): Limit
value to USHRT_MAX (65535).
-- gcc/testsuite/ChangeLog --
2020-05-02  H.J. Lu  

PR target/93492
* c-c++-common/patchable_function_entry-error-1.c: New test.
* c-c++-common/patchable_function_entry-error-2.c: Likewise.
* c-c++-common/patchable_function_entry-error-3.c: Likewise.

What's missing right now is how will we declare a Backport format.
Can we just use something like: 'Backport from 
6607bdd4c834f92fce924abdaea3405f62dc'?
It can rapidly simplify parsing of git messages and the git_repo.py can generate
corresponding ChangeLog entries.

Martin

[1] https://github.com/AdaCore/git-hooks
[2] https://gitpython.readthedocs.io/en/stable/tutorial.html

Re: gcc 10.0.1 20200506 build fails to compile linux kernel

[Martin Liška]

On 5/6/20 6:44 AM, Tetsuji Rai via Gcc wrote:

I wonder how Fedora project built its own kernel.  I can't build custom
kernel with it.

What's wrong with 10.1-RC or how can I report my problem?


Hi.

Is it possible that you reached 
https://lore.kernel.org/lkml/20200417190607.GY2424@tucnak/T/ ?

I can try to reproduce that but please file a bug at or bugzilla and provide
steps how to reproduce the issue. Can you also reproduce it within a qemu VM?

Thanks,
Martin

Re: gcc 10.0.1 20200506 build fails to compile linux kernel

[Martin Liška]

On 5/6/20 2:01 PM, Tetsuji Rai wrote:

Hi Martin,

Thank you for your reply!

Spot on!

It was my kernel config problem associated with stronger stack
protection of gcc-10, not a gcc problem.  But I can't find this in
kernel.org bugzilla or bugzilla.redhat.com (searched with "gcc 10" and
"gcc-10".)


It was probably not reported to Linux bugzilla.



It happens not in qemu or any VM, but on a native machine.  As in
https://bugzilla.redhat.com/show_bug.cgi?id=1796780 (this is the first
url in the page you mentioned), turning CONFIG_STACKPROTECTOR_STRONG
off, lo and behold, the kernel works perfectly!! And as a matter of
course, it works for the latest kernel 5.6.11 also.

My problem was that I brought my custom kernel config file from Fedora
31 with gcc-9.   I should have started from the prototype config file
/boot/config-5.6.8-300.fc32.x86_64, where CONFIG_STACKPROTECTOR_STRONG
is off for its native gcc.

Thank you very much!!


Great it's a known issue.

Martin



-Tetsuji

On 5/6/20 4:19 PM, Martin Liška wrote:

On 5/6/20 6:44 AM, Tetsuji Rai via Gcc wrote:

I wonder how Fedora project built its own kernel.  I can't build custom
kernel with it.

What's wrong with 10.1-RC or how can I report my problem?


Hi.

Is it possible that you reached
https://lore.kernel.org/lkml/20200417190607.GY2424@tucnak/T/ ?

I can try to reproduce that but please file a bug at or bugzilla and
provide
steps how to reproduce the issue. Can you also reproduce it within a
qemu VM?

Thanks,
Martin

Automatically generated ChangeLog files - PHASE 1

[Martin Liška]

Hi.

Thanks to Jakub, we finally set up an experimental environment:
gcc.gnu.org/home/gccadmin/gcc-reposurgeon-8.git

The repository now contains a new pre-commit hook that validates
the git commit format ([1]) and provides a reasonable error message
when violated. The hook is based on [2] and the page also contains
a fuzzy definition of what is supported. Cloning [2], one can also
check what will be added to ChangeLog entries by:

$ ./git_changelog.py /home/marxin/Programming/gcc-reposurgeon-8 
8a37df5e5cb2de8302f9412173103593ec53961e
-- gcc/ChangeLog --
2020-01-13  Martin Jambor  

PR ipa/93223
* ipa-cp.c (devirtualization_time_bonus): Check whether isummary is
NULL.
-- gcc/testsuite/ChangeLog --
2020-01-13  Martin Jambor  

PR ipa/93223
testsuite/
* g++.dg/ipa/pr93223.C: New test.

(one needs [3] Python package for that)

We encourage people to test both the hook and the script. We hope we'll cover
majority of the used formats. I also support _not_ using DATESTAMP and committer
name, these can be automatically deduced from a commit. That will simplify 
workflow
as people won't have to adjust a message before pushing.

Unresolved questions:
- format of reverted patches
- what to do with backports

Here I suggest to use native 'git revert XYZ' and 'git cherry-pick -x XYZ'.
Doing that the commit messages will provide link to original commit and the 
script
can later append corresponding 'Backported ..' or 'Reverted' line.

For the possible issues or questions, please open a Github issue at [4].

Thoughts?
Martin

[1] https://github.com/marxin/git-hooks/tree/gcc-changelog
[2] https://github.com/marxin/gcc-changelog
[3] https://gitpython.readthedocs.io/en/stable/intro.html
[4] https://github.com/marxin/gcc-changelog/issues

 

Re: Automatically generated ChangeLog files - PHASE 1

[Martin Liška]

I'm also CCing gcc-patches and fortran ML.

Martin

On 5/12/20 11:05 AM, Martin Liška wrote:

Hi.

Thanks to Jakub, we finally set up an experimental environment:
gcc.gnu.org/home/gccadmin/gcc-reposurgeon-8.git

The repository now contains a new pre-commit hook that validates
the git commit format ([1]) and provides a reasonable error message
when violated. The hook is based on [2] and the page also contains
a fuzzy definition of what is supported. Cloning [2], one can also
check what will be added to ChangeLog entries by:

$ ./git_changelog.py /home/marxin/Programming/gcc-reposurgeon-8 
8a37df5e5cb2de8302f9412173103593ec53961e
-- gcc/ChangeLog --
2020-01-13  Martin Jambor  

 PR ipa/93223
 * ipa-cp.c (devirtualization_time_bonus): Check whether isummary is
 NULL.
-- gcc/testsuite/ChangeLog --
2020-01-13  Martin Jambor  

 PR ipa/93223
 testsuite/
 * g++.dg/ipa/pr93223.C: New test.

(one needs [3] Python package for that)

We encourage people to test both the hook and the script. We hope we'll cover
majority of the used formats. I also support _not_ using DATESTAMP and committer
name, these can be automatically deduced from a commit. That will simplify 
workflow
as people won't have to adjust a message before pushing.

Unresolved questions:
- format of reverted patches
- what to do with backports

Here I suggest to use native 'git revert XYZ' and 'git cherry-pick -x XYZ'.
Doing that the commit messages will provide link to original commit and the 
script
can later append corresponding 'Backported ..' or 'Reverted' line.

For the possible issues or questions, please open a Github issue at [4].

Thoughts?
Martin

[1] https://github.com/marxin/git-hooks/tree/gcc-changelog
[2] https://github.com/marxin/gcc-changelog
[3] https://gitpython.readthedocs.io/en/stable/intro.html
[4] https://github.com/marxin/gcc-changelog/issues

Re: Automatically generated ChangeLog files - PHASE 1

[Martin Liška]

On 5/13/20 10:16 AM, Richard Sandiford wrote:

As far as this particular example goes, shouldn't the "testsuite/" line
be dropped from the above?


Good point. Fixes now with:

$ ./git_email.py 
patches/0020-IPA-Avoid-segfault-in-devirtualization_time_bonus-PR.patch
Errors:
first line should start with a tab, asterisk and space:"   testsuite/"

Martin

Re: Automatically generated ChangeLog files - PHASE 1

[Martin Liška]

On 5/13/20 11:29 AM, Richard Biener wrote:

Hmm, it's OK in the commit but it should be omitted in the
ChangeLog files.


No, ChangeLog file identification should not be prepended
with a tab.
"testsuite/" will work.

Martin

Re: Automatically generated ChangeLog files - PHASE 1

[Martin Liška]

On 5/13/20 11:50 AM, Jozef Lawrynowicz wrote:

"git_changelog.py" is nowhere to be found in the gcc-changelog or
gcc-reposurgeon-8 repos. Is it equivalent to any of the other scripts in
gcc-changelog?


Yes, I made some refactoring and I'm going to send a proper gcc-patches 
submission.
The scripts will be located in contrib folder.

Martin

ChangeLog files - server and client scripts

[Martin Liška]

Hi.

I'm sending the gcc-changelog relates scripts which should be added to contrib
folder. The patch contains:
- git_check_commit.py - checking script that verifies git message format
- git_update_version.py - a replacement of 
maintainer-scripts/update_version_git which
bumps DATESTAMP and generates ChangeLog entries (for now into ChangeLog.test 
files)
- git_commit.py, git_email.py and git_repository.py - helper classes

I also added a new git.config alias: 'gcc-verify' which can be used in the 
following
way:

$ git gcc-verify HEAD~2..HEAD -p -n
Checking 0e4009e9d523270e26856d2441c1be3d8119a477
OK
@@CL contrib
2020-05-13  Martin Liska  

* gcc-changelog/git_check_commit.py: New file.
* gcc-changelog/git_commit.py: New file.
* gcc-changelog/git_email.py: New file.
* gcc-changelog/git_repository.py: New file.
* gcc-changelog/git_update_version.py: New file.
* gcc-git-customization.sh: Add gcc-verify alias.
@@CL
Checking 18edc195442291525e04f0fa4d5ef972155117da
OK
@@CL gcc
2020-05-13  Jakub Jelinek  

PR debug/95080
* cfgrtl.c (purge_dead_edges): Skip over debug and note insns even
if the last insn is a note.
@@CL gcc/testsuite
2020-05-13  Jakub Jelinek  

PR debug/95080
* g++.dg/opt/pr95080.C: New test.
@@CL

Note the -n option which disables _strict mode_ (modification of both ChangeLog
and another files).

The second part is git hook that will reject all commits for release and master 
branches.
that violate ChangeLog format. Right now, strict mode is disabled in the hooks.

What's still missing to be done is format of Revert and Backport commits.
I suggest to use native 'git revert XYZ' and 'git cherry-pick -x XYZ'.
Doing that the commit messages will provide link to original commit and the 
script
can later append corresponding 'Backported ..' or 'Reverted' line.

Thoughts?
Martin
>From 0e4009e9d523270e26856d2441c1be3d8119a477 Mon Sep 17 00:00:00 2001
From: Martin Liska 
Date: Wed, 13 May 2020 12:22:39 +0200
Subject: [PATCH] Add gcc-changelog related scripts.

contrib/ChangeLog:

2020-05-13  Martin Liska  

	* gcc-changelog/git_check_commit.py: New file.
	* gcc-changelog/git_commit.py: New file.
	* gcc-changelog/git_email.py: New file.
	* gcc-changelog/git_repository.py: New file.
	* gcc-changelog/git_update_version.py: New file.
	* gcc-git-customization.sh: Add gcc-verify alias.
---
 contrib/gcc-changelog/git_check_commit.py   |  49 ++
 contrib/gcc-changelog/git_commit.py | 536 
 contrib/gcc-changelog/git_email.py  |  92 
 contrib/gcc-changelog/git_repository.py |  60 +++
 contrib/gcc-changelog/git_update_version.py | 105 
 contrib/gcc-git-customization.sh|   2 +
 6 files changed, 844 insertions(+)
 create mode 100755 contrib/gcc-changelog/git_check_commit.py
 create mode 100755 contrib/gcc-changelog/git_commit.py
 create mode 100755 contrib/gcc-changelog/git_email.py
 create mode 100755 contrib/gcc-changelog/git_repository.py
 create mode 100755 contrib/gcc-changelog/git_update_version.py

diff --git a/contrib/gcc-changelog/git_check_commit.py b/contrib/gcc-changelog/git_check_commit.py
new file mode 100755
index 000..b2d1d08a242
--- /dev/null
+++ b/contrib/gcc-changelog/git_check_commit.py
@@ -0,0 +1,49 @@
+#!/usr/bin/env python3
+#
+# This file is part of GCC.
+#
+# GCC is free software; you can redistribute it and/or modify it under
+# the terms of the GNU General Public License as published by the Free
+# Software Foundation; either version 3, or (at your option) any later
+# version.
+#
+# GCC is distributed in the hope that it will be useful, but WITHOUT ANY
+# WARRANTY; without even the implied warranty of MERCHANTABILITY or
+# FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+# for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GCC; see the file COPYING3.  If not see
+# .  */
+
+import argparse
+
+from git_repository import parse_git_revisions
+
+parser = argparse.ArgumentParser(description='Check git ChangeLog format '
+ 'of a commit')
+parser.add_argument('revisions',
+help='Git revisions (e.g. hash~5..hash or just hash)')
+parser.add_argument('-g', '--git-path', default='.',
+help='Path to git repository')
+parser.add_argument('-p', '--print-changelog', action='store_true',
+help='Print final changelog entires')
+parser.add_argument('-n', '--allow-non-strict-mode', action='store_true',
+help='Allow non-strict mode (change in both ChangeLog and '
+'other files.')
+args = parser.parse_args()
+
+retval = 0
+for git_commit in parse_git_revisions(args.git_path, args.revisions,
+  not args.allow_non_strict_mode):
+print('Checking %s' % git_commit.hexsha)
+if git_co

ChangeLog files - server and client scripts (git cherry-pick)

[Martin Liška]

On 5/13/20 1:05 PM, Martin Liška wrote:

I suggest to use native 'git revert XYZ' and 'git cherry-pick -x XYZ'.


I've prepared a working version of Revert format:
https://github.com/marxin/gcc-changelog/tree/cherry-pick

So using git cherry-pick -x HASH one gets something like:

$ cat patches-artificial/0001-Test-tree.h.patch
From a71eeba28ffa2427d24d5b2654e93b261980b9e3 Mon Sep 17 00:00:00 2001
From: Martin Liska 
Date: Wed, 13 May 2020 13:19:22 +0200
Subject: [PATCH] Test tree.h.

gcc/ChangeLog:

2020-01-03  Martin Liska  

PR ipa/12345
* tree.h: Just test it.

(cherry picked from commit a2bdf56b15b51c3a7bd988943bdbc42aa156f133)
---
 gcc/tree.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/gcc/tree.h b/gcc/tree.h
index 9ca9ab58ec0..99a9e1a73d9 100644
--- a/gcc/tree.h
+++ b/gcc/tree.h
@@ -1,6 +1,8 @@
 /* Definitions for the ubiquitous 'tree' type for GNU compilers.
Copyright (C) 1989-2020 Free Software Foundation, Inc.
 
+

+
 This file is part of GCC.
 
 GCC is free software; you can redistribute it and/or modify it under

--
2.26.2

and the script generates:

$ ./git_email.py patches-artificial/0001-Test-tree.h.patch
OK
@@CL gcc
2020-05-13  Martin Liska  

Backport from master:
2020-01-03  Martin Liska  

PR ipa/12345
* tree.h: Just test it.
@@CL

So the datestamp and the author is taken from commit and original authors
are added after 'Backport from master' line. The script scans for the
'(cherry picked from commit' line in the message.

Benefit of the approach is that one can adjust the commit message (which 
influences
ChangeLog output).

Martin

Re: ChangeLog files - server and client scripts

[Martin Liška]

The scripts were just installed to master except the git alias.
I'm sending that in a separate patch. Now the alias can be used
from any subfolder in a gcc git repository.

Martin
>From eb47191e8d8cbbda285c4df7eb2d1e98091edab9 Mon Sep 17 00:00:00 2001
From: Martin Liska 
Date: Wed, 13 May 2020 14:32:50 +0200
Subject: [PATCH] Add gcc-verify alias.

contrib/ChangeLog:

2020-05-13  Martin Liska  

	* gcc-git-customization.sh: Add gcc-verify alias
	that uses contrib/gcc-changelog/git_check_commit.py.
---
 contrib/gcc-git-customization.sh | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/contrib/gcc-git-customization.sh b/contrib/gcc-git-customization.sh
index a932bf8c06a..ce293d1fe42 100755
--- a/contrib/gcc-git-customization.sh
+++ b/contrib/gcc-git-customization.sh
@@ -25,6 +25,8 @@ git config alias.svn-rev '!f() { rev=$1; shift; git log --all --grep="^From-SVN:
 git config alias.gcc-descr \!"f() { if test \${1:-no} = --full; then c=\${2:-master}; r=\$(git describe --all --abbrev=40 --match 'basepoints/gcc-[0-9]*' \$c | sed -n 's,^\\(tags/\\)\\?basepoints/gcc-,r,p'); expr match \${r:-no} '^r[0-9]\\+\$' >/dev/null && r=\${r}-0-g\$(git rev-parse \${2:-master}); else c=\${1:-master}; r=\$(git describe --all --match 'basepoints/gcc-[0-9]*' \$c | sed -n 's,^\\(tags/\\)\\?basepoints/gcc-\\([0-9]\\+\\)-\\([0-9]\\+\\)-g[0-9a-f]*\$,r\\2-\\3,p;s,^\\(tags/\\)\\?basepoints/gcc-\\([0-9]\\+\\)\$,r\\2-0,p'); fi; if test -n \$r; then o=\$(git config --get gcc-config.upstream); rr=\$(echo \$r | sed -n 's,^r\\([0-9]\\+\\)-[0-9]\\+\\(-g[0-9a-f]\\+\\)\\?\$,\\1,p'); if git rev-parse --verify --quiet \${o:-origin}/releases/gcc-\$rr >/dev/null; then m=releases/gcc-\$rr; else m=master; fi; git merge-base --is-ancestor \$c \${o:-origin}/\$m && \echo \${r}; fi; }; f"
 git config alias.gcc-undescr \!"f() { o=\$(git config --get gcc-config.upstream); r=\$(echo \$1 | sed -n 's,^r\\([0-9]\\+\\)-[0-9]\\+\$,\\1,p'); n=\$(echo \$1 | sed -n 's,^r[0-9]\\+-\\([0-9]\\+\\)\$,\\1,p'); test -z \$r && echo Invalid id \$1 && exit 1; h=\$(git rev-parse --verify --quiet \${o:-origin}/releases/gcc-\$r); test -z \$h && h=\$(git rev-parse --verify --quiet \${o:-origin}/master); p=\$(git describe --all --match 'basepoints/gcc-'\$r \$h | sed -n 's,^\\(tags/\\)\\?basepoints/gcc-[0-9]\\+-\\([0-9]\\+\\)-g[0-9a-f]*\$,\\2,p;s,^\\(tags/\\)\\?basepoints/gcc-[0-9]\\+\$,0,p'); git rev-parse --verify \$h~\$(expr \$p - \$n); }; f"
 
+git config alias.gcc-verify '!f() { "`git rev-parse --show-toplevel`/contrib/gcc-changelog/git_check_commit.py" $@; } ; f'
+
 # Make diff on MD files use "(define" as a function marker.
 # Use this in conjunction with a .gitattributes file containing
 # *.mddiff=md
-- 
2.26.2

Re: ChangeLog files - server and client scripts

[Martin Liška]

On 5/13/20 3:24 PM, Richard Earnshaw wrote:

I've just realized this doesn't give us an easy way to mark changes for
the root-level ChangeLog file, unless, perhaps "@@ CL ." works?


This works fine:
'ChangeLog:'

as seen for instance here:

commit 9ad3c1d81c129fc76594b9df5b798c380cbf03ee
Author: Stefan Schulze Frielinghaus 
Date:   Wed Apr 22 09:20:08 2020 +0200

MAINTAINERS: add myself for write after approval

ChangeLog:

2020-04-22  Stefan Schulze Frielinghaus  

* MAINTAINERS (Write After Approval): add myself


Martin

C-Vise: speed?

[Martin Liška]

Hello.

I've made some measurements for GCC PRs that I've reduced recently:

- PR92516 - C++ - 6.5MB
  C-Reduce: 77 minutes
  C-Vise: 35 minutes

- PR94523 - C++ - 2.1MB
  C-Reduce: 33 minutes
  C-Vise: 15 minutes

- PR94632 - C++ - 3.3MB
  C-Reduce: 28 minutes
  C-Vise: 20 minutes

- PR94937 - C++ - 8.5MB
  C-Reduce: 303 minutes
  C-Vise: 242 minutes

Note that I also added cvise-delta, which simulates popular delta tool,
but runs in a parallel mode.

Martin

Re: ChangeLog files - server and client scripts (git cherry-pick)

[Martin Liška]

Hello.

I'm sending patch candidate that adds 2 new git aliases:
- gcc-backport - simple alias to 'git cherry-pick -x'
- gcc-revert - it similarly appends '(this reverts commit 
365e3cde4978c6a7dbfa50865720226254c016be)'
to a reverted commit message

The script normally parses content of a git message and adds corresponding 
'Revert:' or
'Backport from master:' lines. Right now, there's missing date of the original 
commit and
author. I hope it's acceptable.

Thoughts?
Martin
>From fd916394f66831ebe8f5cadb455d559aa3917fc3 Mon Sep 17 00:00:00 2001
From: Martin Liska 
Date: Thu, 14 May 2020 14:34:18 +0200
Subject: [PATCH] gcc-changelog: introduce gcc-revert and gcc-backport.

contrib/ChangeLog:

2020-05-14  Martin Liska  

	* gcc-changelog/git_commit.py: Add support
	for CHERRY_PICK_PREFIX and REVERT_PREFIX.
	* gcc-changelog/test_email.py: Add 2 new tests.
	* gcc-changelog/test_patches.txt: Add 2 patches.
	* gcc-git-customization.sh: Add gcc-backport and
	gcc-revert aliases.
---
 contrib/gcc-changelog/git_commit.py| 46 ++
 contrib/gcc-changelog/test_email.py| 14 
 contrib/gcc-changelog/test_patches.txt | 88 ++
 contrib/gcc-git-customization.sh   |  3 +
 4 files changed, 140 insertions(+), 11 deletions(-)

diff --git a/contrib/gcc-changelog/git_commit.py b/contrib/gcc-changelog/git_commit.py
index 5214cc36538..bf82f6206b6 100755
--- a/contrib/gcc-changelog/git_commit.py
+++ b/contrib/gcc-changelog/git_commit.py
@@ -150,6 +150,8 @@ star_prefix_regex = re.compile(r'\t\*(?P\ *)(?P.*)')
 LINE_LIMIT = 100
 TAB_WIDTH = 8
 CO_AUTHORED_BY_PREFIX = 'co-authored-by: '
+CHERRY_PICK_PREFIX = '(cherry picked from commit '
+REVERT_PREFIX = '(this reverts commit '
 
 
 class Error:
@@ -221,6 +223,8 @@ class GitCommit:
 self.top_level_authors = []
 self.co_authors = []
 self.top_level_prs = []
+self.cherry_pick = False
+self.revert = False
 
 project_files = [f for f in self.modified_files
  if self.is_changelog_filename(f[0])
@@ -372,7 +376,11 @@ class GitCommit:
 last_entry.author_lines.append(author_tuple)
 continue
 
-if not line.startswith('\t'):
+if line.startswith(CHERRY_PICK_PREFIX):
+self.cherry_pick = True
+elif line.startswith(REVERT_PREFIX):
+self.revert = True
+elif not line.startswith('\t'):
 err = Error('line should start with a tab', line)
 self.errors.append(err)
 elif pr_line:
@@ -500,24 +508,40 @@ class GitCommit:
 err = Error(msg % (entry.folder, changelog_location), file)
 self.errors.append(err)
 
+@classmethod
+def format_authors_in_changelog(cls, authors, timestamp, prefix=''):
+output = ''
+for i, author in enumerate(authors):
+if i == 0:
+output += '%s%s  %s\n' % (prefix, timestamp, author)
+else:
+output += '%s\t%s\n' % (prefix, author)
+output += '\n'
+return output
+
 def to_changelog_entries(self):
+current_timestamp = self.date.strftime('%Y-%m-%d')
 for entry in self.changelog_entries:
 output = ''
 timestamp = entry.datetime
 if not timestamp:
 timestamp = self.date.strftime('%Y-%m-%d')
 authors = entry.authors if entry.authors else [self.author]
-# add Co-Authored-By authors to all ChangeLog entries
-for author in self.co_authors:
-if author not in authors:
-authors.append(author)
-
-for i, author in enumerate(authors):
-if i == 0:
-output += '%s  %s\n' % (timestamp, author)
+if self.cherry_pick or self.revert:
+output += self.format_authors_in_changelog(authors,
+   current_timestamp)
+if self.cherry_pick:
+header = 'Backport from master'
 else:
-output += '\t%s\n' % author
-output += '\n'
+header = 'Revert'
+output += '\t%s:\n' % header
+else:
+# add Co-Authored-By authors to all ChangeLog entries
+for author in self.co_authors:
+if author not in authors:
+authors.append(author)
+
+output += self.format_authors_in_changelog(authors, timestamp)
 for pr in entry.prs:
 output += '\t%s\n' % pr
 for line in entry.lines:
diff --git a/contrib/gcc-changelog/test_email.py b/contrib/gcc-changelog/test_email.py
index 03abc763212..e1d955a9c21 100755
--- a/contrib/gcc-changelog/test_email.py
+++ b/contrib/gcc-chan

Re: ChangeLog files - server and client scripts

[Martin Liška]

On 5/13/20 7:53 PM, Joseph Myers wrote:

On Wed, 13 May 2020, Martin Liška wrote:


I'm sending the gcc-changelog relates scripts which should be added to contrib
folder. The patch contains:
- git_check_commit.py - checking script that verifies git message format


We need a documentation patch to contribute.html or gitwrite.html that
describes the exact commit message format being used.


Sure, I'm sending patch for that.




- git_update_version.py - a replacement of
maintainer-scripts/update_version_git which
bumps DATESTAMP and generates ChangeLog entries (for now into ChangeLog.test
files)


Where does this check things out?  (The existing ~gccadmin/gcc-checkout
isn't suitable for that, it needs to stay on master to have the correct
version of maintainer-scripts rather than being switched to other
branches, though I suppose a second long-lived checkout that gets updated
automatically could be used.  If you check things out somewhere else
temporarily, it's important to be sure the checkout gets deleted
afterwards rather than having multiple checkouts accumulating.  That's
especially the case if you use a checkout in /tmp as a single GCC
repository clone / checkout uses a significant proportion of the free
space on the root filesystem; /sourceware/snapshot-tmp/gcc has more free
space for large temporary directories.)


Well, we can make a proper git clone of the original repository that will
be used for the daily bumps. I bet we'll have a place for one more clone?




The second part is git hook that will reject all commits for release and
master branches.
that violate ChangeLog format. Right now, strict mode is disabled in the
hooks.


Note that the present state of having GCC-specific patches to the git
hooks is supposed to be a temporary one; we want to move to all relevant
GCC-specific configuration being in refs/meta/config rather than custom
code, so GCC and sourceware can share a single instance of the hooks which
in turn can use the same code as in the upstream AdaCore repository, so
that future updates of the hooks from upstream are easier.  See the issues
I filed at https://github.com/AdaCore/git-hooks/issues for the existing
custom GCC changes and the pull request
https://github.com/AdaCore/git-hooks/pull/12 to bring in implementations
of many of those features (not sure if it covers everything or not).  So
it's important to consider how these checks could be implemented without
needing GCC-specific code directly in these hooks (for example, using the
new hooks.update-hook mechanism added by one of the commits in that pull
request, or getting extra features added to the upstream hooks in a
generic form if necessary).



I welcome the attempt to unify the hooks with AdaCore upstream. I believe this
should not block gcc-changelog attempt now. Later on, we can add it to
hooks.update-hook mechanism.

Thanks,
Martin
diff --git a/htdocs/codingconventions.html b/htdocs/codingconventions.html
index f4732ef6..ffa6db32 100644
--- a/htdocs/codingconventions.html
+++ b/htdocs/codingconventions.html
@@ -112,9 +112,14 @@ maintained and kept up to date.  In particular:
 
 ChangeLogs
 
-GCC requires ChangeLog entries for documentation changes; for the web
-pages (apart from java/ and libstdc++/) the CVS
-commit logs are sufficient.
+
+ChangeLog entries are part of git commit messages and are automatically put
+into a corresponding ChangeLog file.  A ChangeLog template can be easily generated
+with ./contrib/mklog script.  GCC offers a checking script that
+verifies a proper ChangeLog formatting (see git gcc-verify git alias).
+for a particular git commit.  The checking script covers most commonly used ChangeLog
+formats and the following paragraphs explain what it supports.
+
 
 See also what the http://www.gnu.org/prep/standards_toc.html";>GNU Coding
@@ -124,19 +129,95 @@ in comments rather than the ChangeLog, though a single line overall
 description of the changes may be useful above the ChangeLog entry for
 a large batch of changes.
 
-For changes that are ported from another branch, we recommend to
-use a single entry whose body contains a verbatim copy of the original
-entries describing the changes on that branch, possibly preceded by a
-single-line overall description of the changes.
+Components
+
+
+git_description - a leading text with git commit description
+author_timestamp - line with timestamp and an author name and email (2 spaces before and after name) 
+example: 2020-04-23␣␣Martin Liska␣␣<mli...@suse.cz>
+additional_author - line with additional commit author name and email (starting with a tabular and 4 spaces) 
+example: \tMartin Liska␣␣<mli...@suse.cz>
+changelog_location - a location to a ChangeLog file 
+supported formats: a/b/c/ChangeLog, a/b/c/ChangeLog:, a/b/c/ (where ChangeLog file lives in the folder), \ta/b/c/ and a/b/c
+pr_entry - bug report reference 
+example: \tP

New mklog script

[Martin Liška]

Hi.

Since we moved to git world and we're in the preparation for ChangeLog messages
being in git commit messages, I think it's the right time to also simplify mklog
script.

I'm sending a new version (which should eventually replace contrib/mklog and 
contrib/mklog.pl).
Changes made in the version:

- the script uses unifdiff - it rapidly simplifies parsing of the '+-!' lines 
that is done
  in contrib/mklog
- no author nor date stamp is used - that all can be get from git
- --inline option is not supported - I don't see a use-case for it now
- the new script has a unit tests (just few of them for now)

I compares results in between the old Python script for last 80 commits and 
it's very close,
in some cases it does even better.

I'm planning to maintain and improve the script for the future.

Thoughts?
Martin
>From 9fa5d13856f0f5ba153801baf57d4a732829f609 Mon Sep 17 00:00:00 2001
From: Martin Liska 
Date: Fri, 15 May 2020 00:44:07 +0200
Subject: [PATCH] Add mklog-ng.py and gcc-mklog git alias.

contrib/ChangeLog:

	* gcc-git-customization.sh: Add gcc-mklog alias.
	* mklog_ng.py: New file.
	* test_mklog_ng.py: New file.
---
 contrib/gcc-git-customization.sh |   2 +
 contrib/mklog_ng.py  | 192 +++
 contrib/test_mklog_ng.py | 158 +
 3 files changed, 352 insertions(+)
 create mode 100755 contrib/mklog_ng.py
 create mode 100755 contrib/test_mklog_ng.py

diff --git a/contrib/gcc-git-customization.sh b/contrib/gcc-git-customization.sh
index a932bf8c06a..b7b97327be3 100755
--- a/contrib/gcc-git-customization.sh
+++ b/contrib/gcc-git-customization.sh
@@ -25,6 +25,8 @@ git config alias.svn-rev '!f() { rev=$1; shift; git log --all --grep="^From-SVN:
 git config alias.gcc-descr \!"f() { if test \${1:-no} = --full; then c=\${2:-master}; r=\$(git describe --all --abbrev=40 --match 'basepoints/gcc-[0-9]*' \$c | sed -n 's,^\\(tags/\\)\\?basepoints/gcc-,r,p'); expr match \${r:-no} '^r[0-9]\\+\$' >/dev/null && r=\${r}-0-g\$(git rev-parse \${2:-master}); else c=\${1:-master}; r=\$(git describe --all --match 'basepoints/gcc-[0-9]*' \$c | sed -n 's,^\\(tags/\\)\\?basepoints/gcc-\\([0-9]\\+\\)-\\([0-9]\\+\\)-g[0-9a-f]*\$,r\\2-\\3,p;s,^\\(tags/\\)\\?basepoints/gcc-\\([0-9]\\+\\)\$,r\\2-0,p'); fi; if test -n \$r; then o=\$(git config --get gcc-config.upstream); rr=\$(echo \$r | sed -n 's,^r\\([0-9]\\+\\)-[0-9]\\+\\(-g[0-9a-f]\\+\\)\\?\$,\\1,p'); if git rev-parse --verify --quiet \${o:-origin}/releases/gcc-\$rr >/dev/null; then m=releases/gcc-\$rr; else m=master; fi; git merge-base --is-ancestor \$c \${o:-origin}/\$m && \echo \${r}; fi; }; f"
 git config alias.gcc-undescr \!"f() { o=\$(git config --get gcc-config.upstream); r=\$(echo \$1 | sed -n 's,^r\\([0-9]\\+\\)-[0-9]\\+\$,\\1,p'); n=\$(echo \$1 | sed -n 's,^r[0-9]\\+-\\([0-9]\\+\\)\$,\\1,p'); test -z \$r && echo Invalid id \$1 && exit 1; h=\$(git rev-parse --verify --quiet \${o:-origin}/releases/gcc-\$r); test -z \$h && h=\$(git rev-parse --verify --quiet \${o:-origin}/master); p=\$(git describe --all --match 'basepoints/gcc-'\$r \$h | sed -n 's,^\\(tags/\\)\\?basepoints/gcc-[0-9]\\+-\\([0-9]\\+\\)-g[0-9a-f]*\$,\\2,p;s,^\\(tags/\\)\\?basepoints/gcc-[0-9]\\+\$,0,p'); git rev-parse --verify \$h~\$(expr \$p - \$n); }; f"
 
+git config alias.gcc-mklog '!f() { "`git rev-parse --show-toplevel`/contrib/mklog_ng.py" $@; } ; f'
+
 # Make diff on MD files use "(define" as a function marker.
 # Use this in conjunction with a .gitattributes file containing
 # *.mddiff=md
diff --git a/contrib/mklog_ng.py b/contrib/mklog_ng.py
new file mode 100755
index 000..a67fc007759
--- /dev/null
+++ b/contrib/mklog_ng.py
@@ -0,0 +1,192 @@
+#!/usr/bin/env python3
+
+# Copyright (C) 2020 Free Software Foundation, Inc.
+#
+# This file is part of GCC.
+#
+# GCC is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3, or (at your option)
+# any later version.
+#
+# GCC is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GCC; see the file COPYING.  If not, write to
+# the Free Software Foundation, 51 Franklin Street, Fifth Floor,
+# Boston, MA 02110-1301, USA.
+
+# This script parses a .diff file generated with 'diff -up' or 'diff -cp'
+# and adds a skeleton ChangeLog file to the file. It does not try to be
+# too smart when parsing function names, but it produces a reasonable
+# approximation.
+#
+# Author: Martin Liska 
+
+import argparse
+import os
+import re
+import sys
+
+from unidiff import PatchSet
+
+pr_regex = re.compile(r'(\/(\/|\*)|[Cc*!])\s+(?PPR [a-z+-]+\/[0-9]+)')
+identifier_regex = re.compile(r'^([a-zA-Z0-9_#].*)')
+comment_re

Re: ChangeLog files - server and client scripts

[Martin Liška]

On 5/14/20 6:47 PM, Joseph Myers wrote:

On Thu, 14 May 2020, Martin Liška wrote:


On 5/13/20 7:53 PM, Joseph Myers wrote:

On Wed, 13 May 2020, Martin Liška wrote:


I'm sending the gcc-changelog relates scripts which should be added to
contrib
folder. The patch contains:
- git_check_commit.py - checking script that verifies git message format


We need a documentation patch to contribute.html or gitwrite.html that
describes the exact commit message format being used.


Sure, I'm sending patch for that.


Thanks.  There are references to author timestamps there.  The date in a
ChangeLog entry should always be a commit timestamp, not an author one, so
author timestamps present either in commit messages or in the git commit
metadata should be ignored, with only the committer timestamps from the
git commit metadata being used when generating ChangeLog files.


You are fully right, a committer date is what should be used.
Fixed in the documentation, note that the scripts use committed date.

Martin

diff --git a/htdocs/codingconventions.html b/htdocs/codingconventions.html
index f4732ef6..d2e73962 100644
--- a/htdocs/codingconventions.html
+++ b/htdocs/codingconventions.html
@@ -112,9 +112,14 @@ maintained and kept up to date.  In particular:
 
 ChangeLogs
 
-GCC requires ChangeLog entries for documentation changes; for the web
-pages (apart from java/ and libstdc++/) the CVS
-commit logs are sufficient.
+
+ChangeLog entries are part of git commit messages and are automatically put
+into a corresponding ChangeLog file.  A ChangeLog template can be easily generated
+with ./contrib/mklog script.  GCC offers a checking script that
+verifies a proper ChangeLog formatting (see git gcc-verify git alias).
+for a particular git commit.  The checking script covers most commonly used ChangeLog
+formats and the following paragraphs explain what it supports.
+
 
 See also what the http://www.gnu.org/prep/standards_toc.html";>GNU Coding
@@ -124,19 +129,95 @@ in comments rather than the ChangeLog, though a single line overall
 description of the changes may be useful above the ChangeLog entry for
 a large batch of changes.
 
-For changes that are ported from another branch, we recommend to
-use a single entry whose body contains a verbatim copy of the original
-entries describing the changes on that branch, possibly preceded by a
-single-line overall description of the changes.
+Components
+
+
+git_description - a leading text with git commit description
+committer_timestamp - line with timestamp and an author name and email (2 spaces before and after name) 
+example: 2020-04-23␣␣Martin Liska␣␣<mli...@suse.cz>
+additional_author - line with additional commit author name and email (starting with a tabular and 4 spaces) 
+example: \tMartin Liska␣␣<mli...@suse.cz>
+changelog_location - a location to a ChangeLog file 
+supported formats: a/b/c/ChangeLog, a/b/c/ChangeLog:, a/b/c/ (where ChangeLog file lives in the folder), \ta/b/c/ and a/b/c
+pr_entry - bug report reference 
+example: \tPR component/12345
+changelog_file - a modified file mentined in a ChangeLog:
+supported formats: \t* a/b/c/file.c:, \t* a/b/c/file.c (function):, \t* a/b/c/file1.c, a/b/c/file2.c:
+changelog_file_comment - line that follows a changelog_file with description of changes in the file;
+must start with \t
+co_authored_by - https://help.github.com/en/github/committing-changes-to-your-project/creating-a-commit-with-multiple-authors";>GitHub format for a Co-Authored-By
+
+
+Format rules
+
+
+git_description - optional; ends right before one of the other compoments is found
+committer_timestamp - optional; when found before a changelog_file, then it is added
+to each changelog entry
+additional_author - optional
+changelog_location - optional; parser attempts to identify ChangeLog file based
+on modified files; $changelog_location belonging to a different ChangeLog must
+be separated with an empty line
+pr_entry - optional; can contain any number of PR entries
+changelog_file - each changelog_location must contain at least one file
+changelog_file_comment - optional
+co_authored_by - optional, can contain more than one
+
+
+Documented behaviour
+
+
+a missing changelog_location file location can be deduced based on group of changelog_files
+script automatically generates missing "New file." entries for files that are added in a commit
+changed files that are not mentioned in a ChangeLog file generate an error
+similarly for unchanged files that are mentioned in a ChangeLog file
+a commit author and committer date stamp can be automatically deduced from a git commit - we recommend to use it
+co_authored_by is added to each ChangeLog entry
+a PR component is checked against list of valid components
+ChangeLog files, DATESTAMP, BASE-VER and DEV-

Re: New mklog script

[Martin Liška]

On 5/15/20 12:58 PM, David Malcolm wrote:

On Fri, 2020-05-15 at 10:59 +0200, Martin Liška wrote:

Hi.

Since we moved to git world and we're in the preparation for
ChangeLog messages
being in git commit messages, I think it's the right time to also
simplify mklog
script.

I'm sending a new version (which should eventually replace
contrib/mklog and contrib/mklog.pl).
Changes made in the version:

- the script uses unifdiff - it rapidly simplifies parsing of the '+-
!' lines that is done
in contrib/mklog
- no author nor date stamp is used - that all can be get from git
- --inline option is not supported - I don't see a use-case for it
now
- the new script has a unit tests (just few of them for now)

I compares results in between the old Python script for last 80
commits and it's very close,
in some cases it does even better.

I'm planning to maintain and improve the script for the future.

Thoughts?
Martin



+class TestMklog(unittest.TestCase):
+def test_macro_definition(self):
+changelog = generate_changelog(PATCH1)
+assert changelog == EXPECTED1
+
+def test_changed_argument(self):
+changelog = generate_changelog(PATCH2)
+assert changelog == EXPECTED2
+
+def test_enum_and_struct(self):
+changelog = generate_changelog(PATCH3)
+assert changelog == EXPECTED3
+
+def test_no_function(self):
+changelog = generate_changelog(PATCH3, True)
+assert changelog == EXPECTED3B


Thank you David for review.

However I see the same output for both operator== and assertEqual. Probably
because of usage of pytest version 4?

assertEqual:

$ pytest contrib/test_mklog_ng.py
Test session starts (platform: linux, Python 3.8.2, pytest 4.6.9, pytest-sugar 
0.9.3)
benchmark: 3.2.3 (defaults: timer=time.perf_counter disable_gc=False 
min_rounds=5 min_time=0.05 max_time=1.0 calibration_precision=10 
warmup=False warmup_iterations=10)
rootdir: /home/marxin/Programming/gcc
plugins: xdist-1.32.0, sugar-0.9.3, forked-1.1.3, benchmark-3.2.3, 
aspectlib-1.5.0, cov-2.8.1, flake8-1.0.5
collecting ...
 contrib/test_mklog_ng.py ✓ 


  25% ██▌


 TestMklog.test_enum_and_struct 


self = 

def test_enum_and_struct(self):
changelog = generate_changelog(PATCH3)

  self.assertEqual(changelog, EXPECTED3)

E   AssertionError: 'libc[23 chars]clude/cpplib.h (enum c_lang):\n\t(struct 
cpp_options):\n\n' != 'libc[23 chars]clude/cppli22b.h (enum c_lang):\n\t(struct 
cpp_optio44ns):\n\n'
E libcpp/ChangeLog:
E
E   -   * include/cpplib.h (enum c_lang):
E   +   * include/cppli22b.h (enum c_lang):
E   ?  ++
E   -   (struct cpp_options):
E   +   (struct cpp_optio44ns):
E   ?++

contrib/test_mklog_ng.py:154: AssertionError

operator==:

pytest contrib/test_mklog_ng.py
Test session starts (platform: linux, Python 3.8.2, pytest 4.6.9, pytest-sugar 
0.9.3)
benchmark: 3.2.3 (defaults: timer=time.perf_counter disable_gc=False 
min_rounds=5 min_time=0.05 max_time=1.0 calibration_precision=10 
warmup=False warmup_iterations=10)
rootdir: /home/marxin/Programming/gcc
plugins: xdist-1.32.0, sugar-0.9.3, forked-1.1.3, benchmark-3.2.3, 
aspectlib-1.5.0, cov-2.8.1, flake8-1.0.5
collecting ...
 contrib/test_mklog_ng.py ✓ 


  25% ██▌


 TestMklog.test_enum_and_struct 


self = 

def test_enum_and_struct(self):
changelog = generate_changelog(PATCH3)

  assert changelog == EXPECTED3

E   AssertionError: assert 'libcpp/Chang...options):\n\n' == 
'libcpp/Change...tio44ns):\n\n'
E   libcpp/ChangeLog:
E
E - * include/cpplib.h (enum c_lang):
E + * include/cppli22b.h (enum c_lang):
E ?++
E - (struct cpp_options):
E + (struct cpp_optio44ns):...
E
E ...Full output truncated (3 lines hidden), use '-vv' to show

Martin



Use self.assertEqual(a, b) rather than a

Re: New mklog script

[Martin Liška]

On 5/15/20 2:42 PM, Marek Polacek wrote:

I actually use mklog -i all the time.  But I can work around it if it
disappears.


Ah, I can see a consumer.
There's an updated version that supports that.

For the future, will you still use the option? Wouldn't be better
to put the ChangeLog content directly to commit message? Note
that you won't have to copy the entries to a particular ChangeLog file.

Martin
>From d7d5e3aa7450449a8b0cb30d6bf485538990ea3f Mon Sep 17 00:00:00 2001
From: Martin Liska 
Date: Fri, 15 May 2020 00:44:07 +0200
Subject: [PATCH] Add mklog-ng.py and gcc-mklog git alias.

contrib/ChangeLog:

	* gcc-git-customization.sh: Add gcc-mklog alias.
	* mklog_ng.py: New file.
	* test_mklog_ng.py: New file.
---
 contrib/gcc-git-customization.sh |   2 +
 contrib/mklog_ng.py  | 209 +++
 contrib/test_mklog_ng.py | 158 +++
 3 files changed, 369 insertions(+)
 create mode 100755 contrib/mklog_ng.py
 create mode 100755 contrib/test_mklog_ng.py

diff --git a/contrib/gcc-git-customization.sh b/contrib/gcc-git-customization.sh
index a932bf8c06a..b7b97327be3 100755
--- a/contrib/gcc-git-customization.sh
+++ b/contrib/gcc-git-customization.sh
@@ -25,6 +25,8 @@ git config alias.svn-rev '!f() { rev=$1; shift; git log --all --grep="^From-SVN:
 git config alias.gcc-descr \!"f() { if test \${1:-no} = --full; then c=\${2:-master}; r=\$(git describe --all --abbrev=40 --match 'basepoints/gcc-[0-9]*' \$c | sed -n 's,^\\(tags/\\)\\?basepoints/gcc-,r,p'); expr match \${r:-no} '^r[0-9]\\+\$' >/dev/null && r=\${r}-0-g\$(git rev-parse \${2:-master}); else c=\${1:-master}; r=\$(git describe --all --match 'basepoints/gcc-[0-9]*' \$c | sed -n 's,^\\(tags/\\)\\?basepoints/gcc-\\([0-9]\\+\\)-\\([0-9]\\+\\)-g[0-9a-f]*\$,r\\2-\\3,p;s,^\\(tags/\\)\\?basepoints/gcc-\\([0-9]\\+\\)\$,r\\2-0,p'); fi; if test -n \$r; then o=\$(git config --get gcc-config.upstream); rr=\$(echo \$r | sed -n 's,^r\\([0-9]\\+\\)-[0-9]\\+\\(-g[0-9a-f]\\+\\)\\?\$,\\1,p'); if git rev-parse --verify --quiet \${o:-origin}/releases/gcc-\$rr >/dev/null; then m=releases/gcc-\$rr; else m=master; fi; git merge-base --is-ancestor \$c \${o:-origin}/\$m && \echo \${r}; fi; }; f"
 git config alias.gcc-undescr \!"f() { o=\$(git config --get gcc-config.upstream); r=\$(echo \$1 | sed -n 's,^r\\([0-9]\\+\\)-[0-9]\\+\$,\\1,p'); n=\$(echo \$1 | sed -n 's,^r[0-9]\\+-\\([0-9]\\+\\)\$,\\1,p'); test -z \$r && echo Invalid id \$1 && exit 1; h=\$(git rev-parse --verify --quiet \${o:-origin}/releases/gcc-\$r); test -z \$h && h=\$(git rev-parse --verify --quiet \${o:-origin}/master); p=\$(git describe --all --match 'basepoints/gcc-'\$r \$h | sed -n 's,^\\(tags/\\)\\?basepoints/gcc-[0-9]\\+-\\([0-9]\\+\\)-g[0-9a-f]*\$,\\2,p;s,^\\(tags/\\)\\?basepoints/gcc-[0-9]\\+\$,0,p'); git rev-parse --verify \$h~\$(expr \$p - \$n); }; f"
 
+git config alias.gcc-mklog '!f() { "`git rev-parse --show-toplevel`/contrib/mklog_ng.py" $@; } ; f'
+
 # Make diff on MD files use "(define" as a function marker.
 # Use this in conjunction with a .gitattributes file containing
 # *.mddiff=md
diff --git a/contrib/mklog_ng.py b/contrib/mklog_ng.py
new file mode 100755
index 000..8dca6dbeef0
--- /dev/null
+++ b/contrib/mklog_ng.py
@@ -0,0 +1,209 @@
+#!/usr/bin/env python3
+
+# Copyright (C) 2020 Free Software Foundation, Inc.
+#
+# This file is part of GCC.
+#
+# GCC is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3, or (at your option)
+# any later version.
+#
+# GCC is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GCC; see the file COPYING.  If not, write to
+# the Free Software Foundation, 51 Franklin Street, Fifth Floor,
+# Boston, MA 02110-1301, USA.
+
+# This script parses a .diff file generated with 'diff -up' or 'diff -cp'
+# and adds a skeleton ChangeLog file to the file. It does not try to be
+# too smart when parsing function names, but it produces a reasonable
+# approximation.
+#
+# Author: Martin Liska 
+
+import argparse
+import os
+import re
+import sys
+import tempfile
+
+from unidiff import PatchSet
+
+pr_regex = re.compile(r'(\/(\/|\*)|[Cc*!])\s+(?PPR [a-z+-]+\/[0-9]+)')
+identifier_regex = re.compile(r'^([a-zA-Z0-9_#].*)')
+comment_regex = re.compile(r'^\/\*')
+struct_regex = re.compile(r'^((class|struct|union|enum)\s+[a-zA-Z0-9_]+)')
+macro_regex = re.compile(r'#\s*(define|undef)\s+([a-zA-Z0-9_]+)')
+super_macro_regex = re.compile(r'^DEF[A-Z0-9_]+\s*\(([a-zA-Z0-9_]+)')
+fn_regex = re.compile(r'([a-zA-Z_][^()\s]*)\s*\([^*]')
+template_and_param_regex = re.compile(r'<[^<>]*>')
+
+function_extensions = set(['.c', '.cpp', '.C', '.

Re: New mklog script

[Martin Liška]

On 5/15/20 3:22 PM, Marek Polacek wrote:

On Fri, May 15, 2020 at 03:12:27PM +0200, Martin Liška wrote:

On 5/15/20 2:42 PM, Marek Polacek wrote:

I actually use mklog -i all the time.  But I can work around it if it
disappears.


Ah, I can see a consumer.
There's an updated version that supports that.

For the future, will you still use the option? Wouldn't be better
to put the ChangeLog content directly to commit message? Note
that you won't have to copy the entries to a particular ChangeLog file.


The way I do it is to generate a patch using format-patch, use mklog -i
on it, then add the ChangeLog entry to the commit message via commit --amend.


Hmm, you can do much better with:

$ git diff | ./contrib/mklog > changelog && git commit -a -t changelog

Or for an already created commit you can do:

$ git diff HEAD~ | ./contrib/mklog > changelog && git commit -a --amend -e -F 
changelog

That said, I believe usage of -i is legacy.

Martin



Anything that has to do with ChangeLogs is pointless make-work, so the less
I have to do, the better.  ;-)

Marek

Re: New mklog script

[Martin Liška]

On 5/15/20 5:06 PM, Martin Sebor wrote:

On 5/15/20 2:59 AM, Martin Liška wrote:

Hi.

Since we moved to git world and we're in the preparation for ChangeLog messages
being in git commit messages, I think it's the right time to also simplify mklog
script.

I'm sending a new version (which should eventually replace contrib/mklog and 
contrib/mklog.pl).
Changes made in the version:

- the script uses unifdiff - it rapidly simplifies parsing of the '+-!' lines 
that is done
   in contrib/mklog
- no author nor date stamp is used - that all can be get from git
- --inline option is not supported - I don't see a use-case for it now
- the new script has a unit tests (just few of them for now)

I compares results in between the old Python script for last 80 commits and 
it's very close,
in some cases it does even better.

I'm planning to maintain and improve the script for the future.

Thoughts?


Hello Martin.

I welcome the feedback. Leitmotif of the gcc-changelog changes is to simplify
scripts used by individual contributors. I must confess I have a special script
that takes content of changelog entries, applies them to individual files, or
appends Backported header to them. All that should be gone and only git messages
will be used.



It's pretty nice.  I have a script of my own that does the same thing
in a slightly different way.  Here's an example of its output:
https://gcc.gnu.org/pipermail/gcc-patches/attachments/20200323/5437db5a/attachment-0001.bin

I find this format more helpful for the reasons below so unless your
script can be tweaked to do something similar I'd like to be able to
continue to use mine going forward with the new infrastructure.


Let's extend the contrib script.



As for my comments on mklog_ng.py: In the one test I did the script
produced a single long ChangeLog entry with all the files in the diff
I gave it, tests and all, in alphabetical order.  The script fills in
"New test." for new tests.  The rest has to be edited as one would
expect.

I would find the output easier to work with if it a) grouped files by
"subsystem" corresponding to each ChangeLog directory (and if it also


The script does that, let's consider a patch 'p':

./contrib/mklog_ng.py p
gcc/ChangeLog:

* ipa-icf.c:
* varasm.c:

gcc/testsuite/ChangeLog:

* gcc.dg/pr40209.c:

The idea is to put changes in the order in which it appears in a patch.
Because normal work flow is to follow a patch and fulfill ChangeLog entries.
As seen in the patch, changes in varasm.c follow the change in
gcc/testsuite/, which can't be handled as varasm.c needs to be documented
in gcc/ChangeLog.


identified each subsystem), b) put the testsuite section last, and
(as a bonus)


Good idea, I'll do it!


c) grouped all new files in each section together.


Likewise here. And what about deleted files at the very end of a section?



First, I find this logical grouping helpful in thinking about how
the changes are structured (e.g., would it make sense to restructure
them or break things up to reduce coupling and make review easier),
and whom they need to be reviewed by.


Fully yes, please see the updated patch and tell me what's missing.



Second, this is the grouping I'm already used to from my own script
(so YMMV here of course).

Finally, my script also looks up bugs in Bugzilla and adds a line with
each bug number and its Summary at the top of the patch.  This helps me
double-check the spelling of the bug id(s) in case I transpose digits
etc.


Can you please share how do you do it? It would be easy to add it.



Martin

PS My script modifies the patch file in place: it adds the ChangeLog
section if it doesn't exist yet, but it doesn't do anything it does.
I'd love for it to check the existing ChangeLog if it exists and
update it when it finds differences between it and the latest patch
that aren't reflected there.


Well, what about using more git approach? I mean putting all you need
into a commit message.



Without this, each time a patch changes I have to review the entry
and update it as necessary.  That makes it too easy to miss things.


What you can use for a git commit is 'git gcc-verify' which will
inform you about missing (undocumented) Changes.

Martin
diff --git a/gcc/ipa-icf.c b/gcc/ipa-icf.c
index aab79492357..f0df1002488 100644
--- a/gcc/ipa-icf.c
+++ b/gcc/ipa-icf.c
@@ -1,5 +1,7 @@
 
 
+
+
 /* Interprocedural Identical Code Folding pass
Copyright (C) 2014-2020 Free Software Foundation, Inc.
 
diff --git a/gcc/testsuite/gcc.dg/pr40209.c b/gcc/testsuite/gcc.dg/pr40209.c
index 4e77df5c2e6..c23d69d1f1b 100644
--- a/gcc/testsuite/gcc.dg/pr40209.c
+++ b/gcc/testsuite/gcc.dg/pr40209.c
@@ -1,6 +1,8 @@
 /* { dg-do compile } */
 /* { dg-options "-O2 -fprofile-use -fopt-info -Wno-missing-profile" } */
 
+
+
 void process(const char *s

Re: New mklog script

[Martin Liška]

On 5/19/20 10:11 AM, Martin Liška wrote:

Can you please share how do you do it? It would be easy to add it.


I added the feature via --fill-up-bug-titles option. It uses common
request and beatifulsoup packages.

Martin
>From 5450c99b54131d1942ece3ffb6bbe415b1c85151 Mon Sep 17 00:00:00 2001
From: Martin Liska 
Date: Fri, 15 May 2020 00:44:07 +0200
Subject: [PATCH] New mklog script.

contrib/ChangeLog:

2020-05-15  Martin Liska  

	* gcc-git-customization.sh: Add
	alias.gcc-mklog new hook.
	* mklog_ng.py: New file.
	* test_mklog_ng.py: New file.
---
 contrib/gcc-git-customization.sh |   2 +
 contrib/mklog_ng.py  | 223 
 contrib/test_mklog_ng.py | 345 +++
 gcc/ipa-icf.c|   2 +
 4 files changed, 572 insertions(+)
 create mode 100755 contrib/mklog_ng.py
 create mode 100755 contrib/test_mklog_ng.py

diff --git a/contrib/gcc-git-customization.sh b/contrib/gcc-git-customization.sh
index a932bf8c06a..b7b97327be3 100755
--- a/contrib/gcc-git-customization.sh
+++ b/contrib/gcc-git-customization.sh
@@ -25,6 +25,8 @@ git config alias.svn-rev '!f() { rev=$1; shift; git log --all --grep="^From-SVN:
 git config alias.gcc-descr \!"f() { if test \${1:-no} = --full; then c=\${2:-master}; r=\$(git describe --all --abbrev=40 --match 'basepoints/gcc-[0-9]*' \$c | sed -n 's,^\\(tags/\\)\\?basepoints/gcc-,r,p'); expr match \${r:-no} '^r[0-9]\\+\$' >/dev/null && r=\${r}-0-g\$(git rev-parse \${2:-master}); else c=\${1:-master}; r=\$(git describe --all --match 'basepoints/gcc-[0-9]*' \$c | sed -n 's,^\\(tags/\\)\\?basepoints/gcc-\\([0-9]\\+\\)-\\([0-9]\\+\\)-g[0-9a-f]*\$,r\\2-\\3,p;s,^\\(tags/\\)\\?basepoints/gcc-\\([0-9]\\+\\)\$,r\\2-0,p'); fi; if test -n \$r; then o=\$(git config --get gcc-config.upstream); rr=\$(echo \$r | sed -n 's,^r\\([0-9]\\+\\)-[0-9]\\+\\(-g[0-9a-f]\\+\\)\\?\$,\\1,p'); if git rev-parse --verify --quiet \${o:-origin}/releases/gcc-\$rr >/dev/null; then m=releases/gcc-\$rr; else m=master; fi; git merge-base --is-ancestor \$c \${o:-origin}/\$m && \echo \${r}; fi; }; f"
 git config alias.gcc-undescr \!"f() { o=\$(git config --get gcc-config.upstream); r=\$(echo \$1 | sed -n 's,^r\\([0-9]\\+\\)-[0-9]\\+\$,\\1,p'); n=\$(echo \$1 | sed -n 's,^r[0-9]\\+-\\([0-9]\\+\\)\$,\\1,p'); test -z \$r && echo Invalid id \$1 && exit 1; h=\$(git rev-parse --verify --quiet \${o:-origin}/releases/gcc-\$r); test -z \$h && h=\$(git rev-parse --verify --quiet \${o:-origin}/master); p=\$(git describe --all --match 'basepoints/gcc-'\$r \$h | sed -n 's,^\\(tags/\\)\\?basepoints/gcc-[0-9]\\+-\\([0-9]\\+\\)-g[0-9a-f]*\$,\\2,p;s,^\\(tags/\\)\\?basepoints/gcc-[0-9]\\+\$,0,p'); git rev-parse --verify \$h~\$(expr \$p - \$n); }; f"
 
+git config alias.gcc-mklog '!f() { "`git rev-parse --show-toplevel`/contrib/mklog_ng.py" $@; } ; f'
+
 # Make diff on MD files use "(define" as a function marker.
 # Use this in conjunction with a .gitattributes file containing
 # *.mddiff=md
diff --git a/contrib/mklog_ng.py b/contrib/mklog_ng.py
new file mode 100755
index 000..cc3f937c253
--- /dev/null
+++ b/contrib/mklog_ng.py
@@ -0,0 +1,223 @@
+#!/usr/bin/env python3
+
+# Copyright (C) 2020 Free Software Foundation, Inc.
+#
+# This file is part of GCC.
+#
+# GCC is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3, or (at your option)
+# any later version.
+#
+# GCC is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GCC; see the file COPYING.  If not, write to
+# the Free Software Foundation, 51 Franklin Street, Fifth Floor,
+# Boston, MA 02110-1301, USA.
+
+# This script parses a .diff file generated with 'diff -up' or 'diff -cp'
+# and adds a skeleton ChangeLog file to the file. It does not try to be
+# too smart when parsing function names, but it produces a reasonable
+# approximation.
+#
+# Author: Martin Liska 
+
+import argparse
+import bs4
+import os
+import re
+import requests
+import sys
+
+from unidiff import PatchSet
+
+pr_regex = re.compile(r'(\/(\/|\*)|[Cc*!])\s+(?PPR [a-z+-]+\/[0-9]+)')
+identifier_regex = re.compile(r'^([a-zA-Z0-9_#].*)')
+comment_regex = re.compile(r'^\/\*')
+struct_regex = re.compile(r'^((class|struct|union|enum)\s+[a-zA-Z0-9_]+)')
+macro_regex = re.compile(r'#\s*(define|undef)\s+([a-zA-Z0-9_]+)')
+super_macro_regex = re.compile(r'^DEF[A-Z0-9_]+\s*\(([a-zA-Z0

Re: New mklog script

[Martin Liška]

On 5/19/20 10:23 AM, Jakub Jelinek wrote:

On Tue, May 19, 2020 at 10:11:28AM +0200, Martin Liška wrote:

I find this format more helpful for the reasons below so unless your
script can be tweaked to do something similar I'd like to be able to
continue to use mine going forward with the new infrastructure.


Let's extend the contrib script.


BTW, concerning mklog, the very common problem is that it doesn't do the
right thing because the patch doesn't contain enough context to figure out
what exactly has changed.  If the script would be used together with git
rather than just on a patch file, perhaps it could handle more, like
ask git for a patch with unlimited context (like -U1 on patch does).


Good idea but the regex parsing takes some time.


The common problems I remember is that e.g. when changing a function comment
above some function, it is attributed to the previous function rather than
following, labels in function confusing it:
  void
  foo ()
  {
...
  label:
...
-  ...
+  ...
  }


I've just tested that and it will take function for patch context 
(sem_variable::equals):
@@ -1875,6 +1875,7 @@ sem_variable::equals (tree t1, tree t2)
 default:
   return return_false_with_msg ("Unknown TREE code reached");
 }
+
 }


will result in (label), GTY markers confusing it
  struct GTY foobar {
...
-  ...
+  ...
  };
resulting in (struct GTY)


Yes, I know about these and I'll improve stripping of GTY markers.


or so, another common problem is too large
function names (or more often *.md define_* names); here I'm afraid
diff doesn't have an argument to not truncate it, or sometimes e.g. changes
to #define being attributed to something else.
I know some of the issues can be pretty hard to deal with.


;)

Martin



Jakub

Re: ChangeLog files - server and client scripts

[Martin Liška]

Hello.

We've just installed server git hooks that verify git messages
for a correct ChangeLog format. For a limited time period, please
still apply ChangeLog changes to the corresponding ChangeLog files.
We'll use it for comparison of auto-generated CangeLog entries.

The format is documented here:
https://gcc.gnu.org/codingconventions.html#ChangeLogs

And I would recommend to install the new 'git gcc-verify' hook from:
contrib/gcc-git-customization.sh

Feel free to contact me about future troubles you'll see.

Thanks,
Martin

Re: New mklog script

[Martin Liška]

On 5/19/20 10:53 AM, Martin Liška wrote:

On 5/19/20 10:11 AM, Martin Liška wrote:

Can you please share how do you do it? It would be easy to add it.


I added the feature via --fill-up-bug-titles option. It uses common
request and beatifulsoup packages.

Martin


Ok, I'm going to install the following 2 patches that put 2 legacy scripts
into contrib/legacy folder. And a new 'git gcc-mklog' alias is added for the
new one.

Hope other are fine with the change. I'm planning to work on the new script
and fix future limitations.

Martin
>From 577083b84f6851eee0b2d36e26aef42f69676942 Mon Sep 17 00:00:00 2001
From: Martin Liska 
Date: Fri, 15 May 2020 00:44:07 +0200
Subject: [PATCH 2/2] New mklog script.

contrib/ChangeLog:

2020-05-15  Martin Liska  

	* gcc-git-customization.sh: Add
	alias.gcc-mklog new hook.
	* mklog.py: New file.
	* test_mklog.py: New file.
---
 contrib/gcc-git-customization.sh |   2 +
 contrib/mklog.py | 223 
 contrib/test_mklog.py| 345 +++
 3 files changed, 570 insertions(+)
 create mode 100755 contrib/mklog.py
 create mode 100755 contrib/test_mklog.py

diff --git a/contrib/gcc-git-customization.sh b/contrib/gcc-git-customization.sh
index ce293d1fe42..91d378ba32a 100755
--- a/contrib/gcc-git-customization.sh
+++ b/contrib/gcc-git-customization.sh
@@ -27,6 +27,8 @@ git config alias.gcc-undescr \!"f() { o=\$(git config --get gcc-config.upstream)
 
 git config alias.gcc-verify '!f() { "`git rev-parse --show-toplevel`/contrib/gcc-changelog/git_check_commit.py" $@; } ; f'
 
+git config alias.gcc-mklog '!f() { "`git rev-parse --show-toplevel`/contrib/mklog.py" $@; } ; f'
+
 # Make diff on MD files use "(define" as a function marker.
 # Use this in conjunction with a .gitattributes file containing
 # *.mddiff=md
diff --git a/contrib/mklog.py b/contrib/mklog.py
new file mode 100755
index 000..cc3f937c253
--- /dev/null
+++ b/contrib/mklog.py
@@ -0,0 +1,223 @@
+#!/usr/bin/env python3
+
+# Copyright (C) 2020 Free Software Foundation, Inc.
+#
+# This file is part of GCC.
+#
+# GCC is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3, or (at your option)
+# any later version.
+#
+# GCC is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GCC; see the file COPYING.  If not, write to
+# the Free Software Foundation, 51 Franklin Street, Fifth Floor,
+# Boston, MA 02110-1301, USA.
+
+# This script parses a .diff file generated with 'diff -up' or 'diff -cp'
+# and adds a skeleton ChangeLog file to the file. It does not try to be
+# too smart when parsing function names, but it produces a reasonable
+# approximation.
+#
+# Author: Martin Liska 
+
+import argparse
+import bs4
+import os
+import re
+import requests
+import sys
+
+from unidiff import PatchSet
+
+pr_regex = re.compile(r'(\/(\/|\*)|[Cc*!])\s+(?PPR [a-z+-]+\/[0-9]+)')
+identifier_regex = re.compile(r'^([a-zA-Z0-9_#].*)')
+comment_regex = re.compile(r'^\/\*')
+struct_regex = re.compile(r'^((class|struct|union|enum)\s+[a-zA-Z0-9_]+)')
+macro_regex = re.compile(r'#\s*(define|undef)\s+([a-zA-Z0-9_]+)')
+super_macro_regex = re.compile(r'^DEF[A-Z0-9_]+\s*\(([a-zA-Z0-9_]+)')
+fn_regex = re.compile(r'([a-zA-Z_][^()\s]*)\s*\([^*]')
+template_and_param_regex = re.compile(r'<[^<>]*>')
+
+function_extensions = set(['.c', '.cpp', '.C', '.cc', '.h', '.inc', '.def'])
+
+help_message = """\
+Generate ChangeLog template for PATCH.
+PATCH must be generated using diff(1)'s -up or -cp options
+(or their equivalent in git).
+"""
+
+script_folder = os.path.realpath(__file__)
+gcc_root = os.path.dirname(os.path.dirname(script_folder))
+
+
+def find_changelog(path):
+folder = os.path.split(path)[0]
+while True:
+if os.path.exists(os.path.join(gcc_root, folder, 'ChangeLog')):
+return folder
+folder = os.path.dirname(folder)
+if folder == '':
+return folder
+raise AssertionError()
+
+
+def extract_function_name(line):
+if comment_regex.match(line):
+return None
+m = struct_regex.search(line)
+if m:
+# Struct declaration
+return m.group(1)
+m = macro_regex.search(line)
+if m:
+# Macro definition
+return m.group(2)
+m = super_macro_regex.search(line)
+if m:
+

Re: New mklog script

[Martin Liška]

On 5/19/20 5:53 PM, Joseph Myers wrote:

On Tue, 19 May 2020, Martin Liška wrote:


On 5/19/20 10:11 AM, Martin Liška wrote:

Can you please share how do you do it? It would be easy to add it.


I added the feature via --fill-up-bug-titles option. It uses common
request and beatifulsoup packages.


The REST interface is much better to use for extracting bug data than
screen scraping of HTML output.  Fetch e.g.
https://gcc.gnu.org/bugzilla/rest.cgi/bug?id=12345&include_fields=summary
to get JSON bug data (change or omit include_fields if you want more than
just the summary).



You are right, there's a patch I'm going to install.

Martin

>From b5a89069a074aff0ae94176c676eda069ff0a1c3 Mon Sep 17 00:00:00 2001
From: Martin Liska 
Date: Tue, 19 May 2020 21:14:36 +0200
Subject: [PATCH] Use REST API for bug titles in mklog.

contrib/ChangeLog:

2020-05-19  Martin Liska  

	* mklog.py: Use REST API for bug title downloading.
---
 contrib/mklog.py | 20 +---
 1 file changed, 9 insertions(+), 11 deletions(-)

diff --git a/contrib/mklog.py b/contrib/mklog.py
index 45559afbe6b..b27fad0ca2e 100755
--- a/contrib/mklog.py
+++ b/contrib/mklog.py
@@ -31,8 +31,6 @@ import os
 import re
 import sys
 
-import bs4
-
 import requests
 
 from unidiff import PatchSet
@@ -46,6 +44,8 @@ macro_regex = re.compile(r'#\s*(define|undef)\s+([a-zA-Z0-9_]+)')
 super_macro_regex = re.compile(r'^DEF[A-Z0-9_]+\s*\(([a-zA-Z0-9_]+)')
 fn_regex = re.compile(r'([a-zA-Z_][^()\s]*)\s*\([^*]')
 template_and_param_regex = re.compile(r'<[^<>]*>')
+bugzilla_url = 'https://gcc.gnu.org/bugzilla/rest.cgi/bug?id=%s&;' \
+   'include_fields=summary'
 
 function_extensions = set(['.c', '.cpp', '.C', '.cc', '.h', '.inc', '.def'])
 
@@ -106,18 +106,16 @@ def sort_changelog_files(changed_file):
 
 
 def get_pr_titles(prs):
-if not prs:
-return ''
-
 output = ''
 for pr in prs:
 id = pr.split('/')[-1]
-r = requests.get('https://gcc.gnu.org/PR%s' % id)
-html = bs4.BeautifulSoup(r.text, features='lxml')
-title = html.title.text
-title = title[title.find('–') + 1:].strip()
-output += '%s - %s\n' % (pr, title)
-output += '\n'
+r = requests.get(bugzilla_url % id)
+bugs = r.json()['bugs']
+if len(bugs) == 1:
+output += '%s - %s\n' % (pr, bugs[0]['summary'])
+print(output)
+if output:
+output += '\n'
 return output
 
 
-- 
2.26.2

Re: ERR: file not changed in a patch:"gcc/cp/cp-tree.c"

[Martin Liška]

On 5/19/20 9:19 PM, Martin Sebor via Gcc wrote:

Yep, that was it.  It's one of those things that you can stare at
for minutes before you notice it.


Thank you for the report, it's 1:0 for the hook ;)
Next time, please CC me to a similar issue with the hook.

Martin

Re: ChangeLog files - server and client scripts (git cherry-pick)

[Martin Liška]

On 5/14/20 2:42 PM, Martin Liška wrote:

Hello.

I'm sending patch candidate that adds 2 new git aliases:
- gcc-backport - simple alias to 'git cherry-pick -x'
- gcc-revert - it similarly appends '(this reverts commit 
365e3cde4978c6a7dbfa50865720226254c016be)'
to a reverted commit message

The script normally parses content of a git message and adds corresponding 
'Revert:' or
'Backport from master:' lines. Right now, there's missing date of the original 
commit and
author. I hope it's acceptable.

Thoughts?
Martin


Hello.

I'm going to install the following patch that will allow '(cherry picked from 
commit hash)' line.
Generated ChangeLog entry will look the same as the original one (No Backported 
from leading lines).
Apart from that, I'm adding 'git gcc-backport' which is simple alias for 
'cherry-pick -x'.

Martin

>From 5394cd8d0ec4aa774228bff1687cdace5cdc7552 Mon Sep 17 00:00:00 2001
From: Martin Liska 
Date: Wed, 20 May 2020 09:49:48 +0200
Subject: [PATCH] Add gcc-backport and support git cherry pick.

Unknown ChangeLog:

2020-05-20  Martin Liska  

	* contrib/gcc-changelog/git_commit.py: Support cherry pick
	prefix.
	* contrib/gcc-changelog/test_email.py: Test it.
	* contrib/gcc-changelog/test_patches.txt: Add new patch.
	* contrib/gcc-git-customization.sh: Add gcc-backport.
---
 contrib/gcc-changelog/git_commit.py|  3 +++
 contrib/gcc-changelog/test_email.py|  4 
 contrib/gcc-changelog/test_patches.txt | 29 ++
 contrib/gcc-git-customization.sh   |  1 +
 4 files changed, 37 insertions(+)

diff --git a/contrib/gcc-changelog/git_commit.py b/contrib/gcc-changelog/git_commit.py
index f6b9c5b1586..5cc8c4f5935 100755
--- a/contrib/gcc-changelog/git_commit.py
+++ b/contrib/gcc-changelog/git_commit.py
@@ -150,6 +150,7 @@ star_prefix_regex = re.compile(r'\t\*(?P\ *)(?P.*)')
 LINE_LIMIT = 100
 TAB_WIDTH = 8
 CO_AUTHORED_BY_PREFIX = 'co-authored-by: '
+CHERRY_PICK_PREFIX = '(cherry picked from commit '
 
 
 class Error:
@@ -349,6 +350,8 @@ class GitCommit:
 author = self.format_git_author(name)
 self.co_authors.append(author)
 continue
+elif line.startswith(CHERRY_PICK_PREFIX):
+continue
 
 # ChangeLog name will be deduced later
 if not last_entry:
diff --git a/contrib/gcc-changelog/test_email.py b/contrib/gcc-changelog/test_email.py
index 03abc763212..5e99d3240e8 100755
--- a/contrib/gcc-changelog/test_email.py
+++ b/contrib/gcc-changelog/test_email.py
@@ -258,3 +258,7 @@ class TestGccChangelog(unittest.TestCase):
 email = self.from_patch_glob('0020-IPA-Avoid')
 assert (email.errors[0].message
 == 'first line should start with a tab, asterisk and space')
+
+def test_cherry_pick_format(self):
+email = self.from_patch_glob('0001-c-Alias.patch')
+assert not email.errors
diff --git a/contrib/gcc-changelog/test_patches.txt b/contrib/gcc-changelog/test_patches.txt
index 39e4753c0ab..ec667be9a92 100644
--- a/contrib/gcc-changelog/test_patches.txt
+++ b/contrib/gcc-changelog/test_patches.txt
@@ -2382,3 +2382,32 @@ index 000..66c87d48694
 -- 
 2.26.1
 
+=== 0001-c-Alias.patch ===
+From 3f1a149fc35cdba988464562e2fb824b10652d6b Mon Sep 17 00:00:00 2001
+From: Nathan Sidwell 
+Date: Tue, 19 May 2020 13:29:19 -0700
+Subject: [PATCH] c++: Alias template instantiation template info
+
+I discovered that the alias instantiation machinery would setup
+template_info, and then sometime later overwrite that with equivalent
+info.  This broke modules, because the template info, once set, is
+logically immutable.  Let's just not do that.
+
+	* pt.c (lookup_template_class_1): Do not reinit template_info of an
+	alias here.
+
+(cherry picked from commit 74744bb1f2847b5b9ce3e97e0fec9c23bb0e499f)
+---
+ gcc/cp/pt.c | 17 +++--
+ 1 file changed, 15 insertions(+), 2 deletions(-)
+
+diff --git a/gcc/cp/pt.c b/gcc/cp/pt.c
+index b8f03d18541..7230ac724ba 100644
+--- a/gcc/cp/pt.c
 b/gcc/cp/pt.c
+@@ -1 +1,2 @@
+
++
+-- 
+2.26.2
+
diff --git a/contrib/gcc-git-customization.sh b/contrib/gcc-git-customization.sh
index 91d378ba32a..7a950ae5f38 100755
--- a/contrib/gcc-git-customization.sh
+++ b/contrib/gcc-git-customization.sh
@@ -26,6 +26,7 @@ git config alias.gcc-descr \!"f() { if test \${1:-no} = --full; then c=\${2:-mas
 git config alias.gcc-undescr \!"f() { o=\$(git config --get gcc-config.upstream); r=\$(echo \$1 | sed -n 's,^r\\([0-9]\\+\\)-[0-9]\\+\$,\\1,p'); n=\$(echo \$1 | sed -n 's,^r[0-9]\\+-\\([0-9]\\+\\)\$,\\1,p'); test -z \$r && echo Invalid id \$1 && exit 1; h=\$(git rev-parse --verify --quiet \${o:-origin}/releases/gcc-\$r); test -z \$h && h=\$(git rev-parse --verify --

Re: ERR: file not changed in a patch:"gcc/cp/cp-tree.c"

[Martin Liška]

On 5/19/20 9:31 PM, Martin Sebor wrote:

On 5/19/20 1:26 PM, Martin Liška wrote:

On 5/19/20 9:19 PM, Martin Sebor via Gcc wrote:

Yep, that was it.  It's one of those things that you can stare at
for minutes before you notice it.


Thank you for the report, it's 1:0 for the hook ;)
Next time, please CC me to a similar issue with the hook.


I didn't know we were keeping score, otherwise I would have kept
this silly blunder to myself ;-)


Heh, that was just a joke.



By the way, thanks for working on the automation!  I can't wait
for this ChangeLog hassle (at least some of it) to finally go
away.


Thanks. Me to, that's why I invested some time in that. It's waste
of time to manipulate the ChangeLog entries.


This commit took me four attempts: the first two failed
due to ChangeLog conflicts, and the last because of this mistake.

Martin

Re: ChangeLog files - server and client scripts

[Martin Liška]

On 5/20/20 12:20 AM, Jonathan Wakely wrote:

Or this one that actually adds the closing parenthesis


Thank you, it's definitely an improvement.

There's final version of the patch I've just applied.

Martin
>From a55c1018c9d7c53b643203e7f71b06953fae86a1 Mon Sep 17 00:00:00 2001
From: Jonathan Wakely 
Date: Wed, 20 May 2020 10:03:51 +0200
Subject: [PATCH] git_check_commit: shorted option name

contrib/ChangeLog:

2020-05-20  Martin Liska  

	* gcc-changelog/git_check_commit.py: Change
	--allow-non-strict-mode to --non-strict-mode.
---
 contrib/gcc-changelog/git_check_commit.py | 8 
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/contrib/gcc-changelog/git_check_commit.py b/contrib/gcc-changelog/git_check_commit.py
index 8553c90a96f..2601ae4f613 100755
--- a/contrib/gcc-changelog/git_check_commit.py
+++ b/contrib/gcc-changelog/git_check_commit.py
@@ -28,14 +28,14 @@ parser.add_argument('-g', '--git-path', default='.',
 help='Path to git repository')
 parser.add_argument('-p', '--print-changelog', action='store_true',
 help='Print final changelog entires')
-parser.add_argument('-n', '--allow-non-strict-mode', action='store_true',
-help='Allow non-strict mode (change in both ChangeLog and '
-'other files.')
+parser.add_argument('-n', '--non-strict-mode', action='store_true',
+help='Use non-strict mode (allow changes in ChangeLog and '
+'other automatically updated files).')
 args = parser.parse_args()
 
 retval = 0
 for git_commit in parse_git_revisions(args.git_path, args.revisions,
-  not args.allow_non_strict_mode):
+  not args.non_strict_mode):
 res = 'OK' if git_commit.success else 'FAILED'
 print('Checking %s: %s' % (git_commit.hexsha, res))
 if git_commit.success:
-- 
2.26.2

Re: Commit hook for cherry-pick commit

[Martin Liška]

On 5/20/20 12:59 AM, H.J. Lu wrote:

I got:

$ git push origin releases/gcc-10
Enumerating objects: 15, done.
Counting objects: 100% (15/15), done.
Delta compression using up to 8 threads
Compressing objects: 100% (6/6), done.
Writing objects: 100% (8/8), 1.15 KiB | 1.15 MiB/s, done.
Total 8 (delta 7), reused 2 (delta 2), pack-reused 0
remote: *** ChangeLog format failed:
remote: ERR: line should start with a tab:"(cherry picked from commit
453954451be68d22462442268a29f54809182d2b)"
remote: ERR: could not deduce ChangeLog file
remote:
remote: Please see: https://gcc.gnu.org/codingconventions.html#ChangeLogs
remote:
remote: error: hook declined to update refs/heads/releases/gcc-10
To ssh://gcc.gnu.org/git/gcc.git
  ! [remote rejected] releases/gcc-10 -> releases/gcc-10 (hook declined)
error: failed to push some refs to 'ssh://h...@gcc.gnu.org/git/gcc.git'

My cherry-pick commit message has

(cherry picked from commit 453954451be68d22462442268a29f54809182d2b)

Shouldn't it be allowed?



Hello.

It is fixed now, please try to push the commit.

Martin

Re: ChangeLog files - server and client scripts (git cherry-pick)

[Martin Liška]

On 5/20/20 11:19 AM, Thomas Koenig wrote:

Hm, one question: I find the r11-1234 type commit to be much more
readable, in ChangeLog files and everywhere else.

Would it be possible to have that format instead of
"cherry picked from commit $HEX_SOUP" ?


I'm not aware of how to do it.
Please let's keep it simple as possible (cherry-pick -x). You can
always adjust the commit message and mention a r11-1234 revision
there.

Martin

Re: Commit hook for cherry-pick commit

[Martin Liška]

On 5/20/20 1:31 PM, H.J. Lu wrote:

On Wed, May 20, 2020 at 1:40 AM Martin Liška  wrote:


On 5/20/20 12:59 AM, H.J. Lu wrote:

I got:

$ git push origin releases/gcc-10
Enumerating objects: 15, done.
Counting objects: 100% (15/15), done.
Delta compression using up to 8 threads
Compressing objects: 100% (6/6), done.
Writing objects: 100% (8/8), 1.15 KiB | 1.15 MiB/s, done.
Total 8 (delta 7), reused 2 (delta 2), pack-reused 0
remote: *** ChangeLog format failed:
remote: ERR: line should start with a tab:"(cherry picked from commit
453954451be68d22462442268a29f54809182d2b)"
remote: ERR: could not deduce ChangeLog file
remote:
remote: Please see: https://gcc.gnu.org/codingconventions.html#ChangeLogs
remote:
remote: error: hook declined to update refs/heads/releases/gcc-10
To ssh://gcc.gnu.org/git/gcc.git
   ! [remote rejected] releases/gcc-10 -> releases/gcc-10 (hook 
declined)
error: failed to push some refs to 'ssh://h...@gcc.gnu.org/git/gcc.git'

My cherry-pick commit message has

(cherry picked from commit 453954451be68d22462442268a29f54809182d2b)

Shouldn't it be allowed?



Hello.

It is fixed now, please try to push the commit.

Martin


I got

remote: *** ChangeLog format failed:
remote: ERR: line should start with a tab:"(cherry picked from commit
1e46a443f25d26816536c0c480211714b123a1d5)"
remote: ERR: could not deduce ChangeLog file



Can you please send me the patch via email (git format-patch)?

Thanks,
Martin

Re: Commit hook for cherry-pick commit

[Martin Liška]

On 5/20/20 3:48 PM, H.J. Lu wrote:

On Wed, May 20, 2020 at 6:47 AM Martin Liška  wrote:


On 5/20/20 1:31 PM, H.J. Lu wrote:

On Wed, May 20, 2020 at 1:40 AM Martin Liška  wrote:


On 5/20/20 12:59 AM, H.J. Lu wrote:

I got:

$ git push origin releases/gcc-10
Enumerating objects: 15, done.
Counting objects: 100% (15/15), done.
Delta compression using up to 8 threads
Compressing objects: 100% (6/6), done.
Writing objects: 100% (8/8), 1.15 KiB | 1.15 MiB/s, done.
Total 8 (delta 7), reused 2 (delta 2), pack-reused 0
remote: *** ChangeLog format failed:
remote: ERR: line should start with a tab:"(cherry picked from commit
453954451be68d22462442268a29f54809182d2b)"
remote: ERR: could not deduce ChangeLog file
remote:
remote: Please see: https://gcc.gnu.org/codingconventions.html#ChangeLogs
remote:
remote: error: hook declined to update refs/heads/releases/gcc-10
To ssh://gcc.gnu.org/git/gcc.git
! [remote rejected] releases/gcc-10 -> releases/gcc-10 (hook 
declined)
error: failed to push some refs to 'ssh://h...@gcc.gnu.org/git/gcc.git'

My cherry-pick commit message has

(cherry picked from commit 453954451be68d22462442268a29f54809182d2b)

Shouldn't it be allowed?



Hello.

It is fixed now, please try to push the commit.

Martin


I got

remote: *** ChangeLog format failed:
remote: ERR: line should start with a tab:"(cherry picked from commit
1e46a443f25d26816536c0c480211714b123a1d5)"
remote: ERR: could not deduce ChangeLog file



Can you please send me the patch via email (git format-patch)?



Here.



Which branch do you want to push it to?

Martin

Re: Commit hook for cherry-pick commit

[Martin Liška]

On 5/20/20 3:57 PM, H.J. Lu wrote:

On Wed, May 20, 2020 at 6:55 AM Martin Liška  wrote:


On 5/20/20 3:48 PM, H.J. Lu wrote:

On Wed, May 20, 2020 at 6:47 AM Martin Liška  wrote:


On 5/20/20 1:31 PM, H.J. Lu wrote:

On Wed, May 20, 2020 at 1:40 AM Martin Liška  wrote:


On 5/20/20 12:59 AM, H.J. Lu wrote:

I got:

$ git push origin releases/gcc-10
Enumerating objects: 15, done.
Counting objects: 100% (15/15), done.
Delta compression using up to 8 threads
Compressing objects: 100% (6/6), done.
Writing objects: 100% (8/8), 1.15 KiB | 1.15 MiB/s, done.
Total 8 (delta 7), reused 2 (delta 2), pack-reused 0
remote: *** ChangeLog format failed:
remote: ERR: line should start with a tab:"(cherry picked from commit
453954451be68d22462442268a29f54809182d2b)"
remote: ERR: could not deduce ChangeLog file
remote:
remote: Please see: https://gcc.gnu.org/codingconventions.html#ChangeLogs
remote:
remote: error: hook declined to update refs/heads/releases/gcc-10
To ssh://gcc.gnu.org/git/gcc.git
 ! [remote rejected] releases/gcc-10 -> releases/gcc-10 (hook 
declined)
error: failed to push some refs to 'ssh://h...@gcc.gnu.org/git/gcc.git'

My cherry-pick commit message has

(cherry picked from commit 453954451be68d22462442268a29f54809182d2b)

Shouldn't it be allowed?



Hello.

It is fixed now, please try to push the commit.

Martin


I got

remote: *** ChangeLog format failed:
remote: ERR: line should start with a tab:"(cherry picked from commit
1e46a443f25d26816536c0c480211714b123a1d5)"
remote: ERR: could not deduce ChangeLog file



Can you please send me the patch via email (git format-patch)?



Here.



Which branch do you want to push it to?

Martin


git push origin releases/gcc-10
Enumerating objects: 13, done.
Counting objects: 100% (13/13), done.
Delta compression using up to 8 threads
Compressing objects: 100% (7/7), done.
Writing objects: 100% (7/7), 844 bytes | 844.00 KiB/s, done.
Total 7 (delta 6), reused 0 (delta 0), pack-reused 0
remote: *** ChangeLog format failed:
remote: ERR: line should start with a tab:"(cherry picked from commit
1e46a443f25d26816536c0c480211714b123a1d5)"
remote: ERR: could not deduce ChangeLog file
remote:
remote: Please see: https://gcc.gnu.org/codingconventions.html#ChangeLogs
remote:
remote: error: hook declined to update refs/heads/releases/gcc-10
To ssh://gcc.gnu.org/git/gcc.git
  ! [remote rejected] releases/gcc-10 -> releases/gcc-10 (hook declined)
error: failed to push some refs to 'ssh://h...@gcc.gnu.org/git/gcc.git'




Thank you, working on that as current git_commit.py script works fine with that.
Note that we sync the file to GIT hooks I think we're using an older version.

I'll inform you,
Martin

Re: Is commit hook broken?

[Martin Liška]

On 5/21/20 2:52 AM, H.J. Lu wrote:

On Wed, May 20, 2020 at 7:23 AM Martin Liška  wrote:


On 5/20/20 3:57 PM, H.J. Lu wrote:

On Wed, May 20, 2020 at 6:55 AM Martin Liška  wrote:


On 5/20/20 3:48 PM, H.J. Lu wrote:

On Wed, May 20, 2020 at 6:47 AM Martin Liška  wrote:


On 5/20/20 1:31 PM, H.J. Lu wrote:

On Wed, May 20, 2020 at 1:40 AM Martin Liška  wrote:


On 5/20/20 12:59 AM, H.J. Lu wrote:

I got:

$ git push origin releases/gcc-10
Enumerating objects: 15, done.
Counting objects: 100% (15/15), done.
Delta compression using up to 8 threads
Compressing objects: 100% (6/6), done.
Writing objects: 100% (8/8), 1.15 KiB | 1.15 MiB/s, done.
Total 8 (delta 7), reused 2 (delta 2), pack-reused 0
remote: *** ChangeLog format failed:
remote: ERR: line should start with a tab:"(cherry picked from commit
453954451be68d22462442268a29f54809182d2b)"
remote: ERR: could not deduce ChangeLog file
remote:
remote: Please see: https://gcc.gnu.org/codingconventions.html#ChangeLogs
remote:
remote: error: hook declined to update refs/heads/releases/gcc-10
To ssh://gcc.gnu.org/git/gcc.git
  ! [remote rejected] releases/gcc-10 -> releases/gcc-10 (hook 
declined)
error: failed to push some refs to 'ssh://h...@gcc.gnu.org/git/gcc.git'

My cherry-pick commit message has

(cherry picked from commit 453954451be68d22462442268a29f54809182d2b)

Shouldn't it be allowed?



Hello.

It is fixed now, please try to push the commit.

Martin


I got

remote: *** ChangeLog format failed:
remote: ERR: line should start with a tab:"(cherry picked from commit
1e46a443f25d26816536c0c480211714b123a1d5)"
remote: ERR: could not deduce ChangeLog file



Can you please send me the patch via email (git format-patch)?



Here.



Which branch do you want to push it to?

Martin


git push origin releases/gcc-10
Enumerating objects: 13, done.
Counting objects: 100% (13/13), done.
Delta compression using up to 8 threads
Compressing objects: 100% (7/7), done.
Writing objects: 100% (7/7), 844 bytes | 844.00 KiB/s, done.
Total 7 (delta 6), reused 0 (delta 0), pack-reused 0
remote: *** ChangeLog format failed:
remote: ERR: line should start with a tab:"(cherry picked from commit
1e46a443f25d26816536c0c480211714b123a1d5)"
remote: ERR: could not deduce ChangeLog file
remote:
remote: Please see: https://gcc.gnu.org/codingconventions.html#ChangeLogs
remote:
remote: error: hook declined to update refs/heads/releases/gcc-10
To ssh://gcc.gnu.org/git/gcc.git
   ! [remote rejected] releases/gcc-10 -> releases/gcc-10 (hook 
declined)
error: failed to push some refs to 'ssh://h...@gcc.gnu.org/git/gcc.git'




Thank you, working on that as current git_commit.py script works fine with that.
Note that we sync the file to GIT hooks I think we're using an older version.

I'll inform you,
Martin


Now I got
  git push  origin releases/gcc-10
Enumerating objects: 13, done.
Counting objects: 100% (13/13), done.
Delta compression using up to 8 threads
Compressing objects: 100% (7/7), done.
Writing objects: 100% (7/7), 846 bytes | 846.00 KiB/s, done.
Total 7 (delta 6), reused 0 (delta 0), pack-reused 0
remote: Traceback (most recent call last):
remote:   File "hooks/update.py", line 13, in 
remote: from updates.factory import new_update
remote:   File "/sourceware1/home/gccadmin/git-hooks/hooks/updates/__init__.py",
line 8, in 
remote: from pre_commit_checks import (check_revision_history,
style_check_commit,
remote:   File 
"/sourceware1/home/gccadmin/git-hooks/hooks/pre_commit_checks.py",
line 6, in 
remote: from git_commit import GitCommit
remote: ImportError: No module named git_commit
remote: error: hook declined to update refs/heads/releases/gcc-10
To ssh://gcc.gnu.org/git/gcc.git
  ! [remote rejected] releases/gcc-10 -> releases/gcc-10 (hook declined)
error: failed to push some refs to 'ssh://h...@gcc.gnu.org/git/gcc.git'



It's fixed now and I pushed your commit:
https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=efcf41fcfa7351f2e941075d2f1fd73cc282de5b

We had a broken file permissions on the server. Jonathan fixed that.

Martin

Re: New mklog script

[Martin Liška]

Hello Martin.

Can you please compare the current mklog.py. Is there anything
you miss compared to your current script?

Thanks,
Martin

Re: ChangeLog files - server and client scripts

[Martin Liška]

On 5/21/20 5:14 PM, Rainer Orth wrote:

Hi Martin,


We've just installed server git hooks that verify git messages
for a correct ChangeLog format. For a limited time period, please
still apply ChangeLog changes to the corresponding ChangeLog files.
We'll use it for comparison of auto-generated CangeLog entries.

The format is documented here:
https://gcc.gnu.org/codingconventions.html#ChangeLogs


two comments:

* Can you please avoid the use grey highlighting in that section?  Black
   script on a grey background is already hard to read for someone with
   reasonable vision.  I suspect it will be much harder for
   vision-impaired people.


You are right, I fixed that.



* In changelog_location, you allow only (among others) "a/b/c/" and
   "\ta/b/c/".  Please also accept the "a/b/c:" and "\ta/b/c:" forms
   here: especially the second seems quite common.


Sure, can you please link some git revisions that use the format?

Thanks,
Martin



Thanks.
 Rainer

Re: ChangeLog files - server and client scripts

[Martin Liška]

On 5/21/20 8:52 PM, Jason Merrill wrote:

Was there a decision somewhere to require ChangeLog entries for all testcase 
changes now, as the hook is enforcing?  They were optional before.


Right now we ignore newly added test-case, these don't have to be mentioned.
Can you please attach the patch (git format-patch)?

Are you talking about modified or delete test-cases? If so, we can definitely
relax the rules..

Martin

Re: ChangeLog files - server and client scripts

[Martin Liška]

On 5/21/20 9:51 PM, Jason Merrill wrote:

Modified.  Adjustments to expected errors in testcases don't seem to me worth 
documenting in a ChangeLog.


I see. As Jakub mentioned, I would keep the hook stricter for now.

Martin

Re: ChangeLog files - server and client scripts

[Martin Liška]

On 5/21/20 11:01 PM, Jason Merrill wrote:

Why?  What is the use of requiring ChangeLog entries at all for these changes?


I must confirm a common test-suite ChangeLog entry is something like:

$ grep ':' gcc/testsuite/ChangeLog | sed 's/.*://' | sort | uniq -c | sort -n | 
tac | head -n 15
   6309  Likewise.
   1306  New test.
231  New.
 68  New testcase.
 55
 54  Ditto.
 47  New file.
 33  Same.
 27  Move into ...
 24  New tests.
 21  Add typedef for int32_t.
 19  This.  Clean up
 17  Rename to...
 16  This.
 12  ...this.

$ grep ':' gcc/testsuite/ChangeLog-2019 | sed 's/.*://' | sort | uniq -c | sort 
-n | tac | head -n 15
   2981  Likewise.
   2287  New test.
399  New testcase.
263  Same.
263  New.
217  Ditto.
111  Adjust.
 88  New file.
 62
 32  Require exceptions.
 30  Remove.
 25  New
 20  Update test.
 17  New tests.
 15  Add

$ grep ':' gcc/testsuite/ChangeLog-2018 | sed 's/.*://' | sort | uniq -c | sort 
-n | tac | head -n 15
   5071  Likewise.
   2071  New test.
443  New.
413  New testcase.
395  Remove.
256  Ditto.
224  Same.
108  New file.
 66  Likwise.
 65  Adjust.
 53
 52  Dito.
 39  Add -flinker-output=nolto-rel.
 32  New test case.
 26  Delete testcase.

So I'm open for relaxation of the rule.
What about the others?

Martin

Re: ChangeLog files - server and client scripts

[Martin Liška]

On 5/22/20 6:57 AM, Jakub Jelinek wrote:

so perhaps it just misses gcc/testsuite/go.test/test ?


Hello.

I've just added the location to ignored locations.


Or what exact files you've changed in your script?


@Ian: Please send us patch with git format-patch.
@Jakub: Can you please sync up the script to the server hooks? I'll
be AFK till Monday.

Martin

Re: ChangeLog files - server and client scripts

[Martin Liška]

On 5/23/20 12:14 AM, Ian Lance Taylor wrote:

Sure, I can wait.  Thanks for looking at it.


Hello.

Thank you for patience. There's a patch that fixes that,
I'm going to install it.

Martin
>From 76e18b91250f265a37d85063860fb38aa8f6aac3 Mon Sep 17 00:00:00 2001
From: Martin Liska 
Date: Mon, 25 May 2020 09:40:50 +0200
Subject: [PATCH] Allow only ignored files in ChangeLog entries.

contrib/ChangeLog:

2020-05-25  Martin Liska  

	* gcc-changelog/git_commit.py: Add trailing '/'
	for libdruntime.  Allow empty changelog for
	only ignored files.
	* gcc-changelog/test_email.py: New test for go
	patch in ignored location.
	* gcc-changelog/test_patches.txt: Add test.
---
 contrib/gcc-changelog/git_commit.py|  5 +--
 contrib/gcc-changelog/test_email.py|  4 +++
 contrib/gcc-changelog/test_patches.txt | 43 ++
 3 files changed, 50 insertions(+), 2 deletions(-)

diff --git a/contrib/gcc-changelog/git_commit.py b/contrib/gcc-changelog/git_commit.py
index 8c5fa2c0fc9..2cfdbc83d09 100755
--- a/contrib/gcc-changelog/git_commit.py
+++ b/contrib/gcc-changelog/git_commit.py
@@ -130,7 +130,7 @@ ignored_prefixes = [
 'gcc/go/gofrontend/',
 'gcc/testsuite/go.test/test/',
 'libgo/',
-'libphobos/libdruntime',
+'libphobos/libdruntime/',
 'libphobos/src/',
 'libsanitizer/',
 ]
@@ -233,7 +233,8 @@ class GitCommit:
 
 project_files = [f for f in self.modified_files
  if self.is_changelog_filename(f[0])
- or f[0] in misc_files]
+ or f[0] in misc_files
+ or self.in_ignored_location(f[0])]
 if len(project_files) == len(self.modified_files):
 # All modified files are only MISC files
 return
diff --git a/contrib/gcc-changelog/test_email.py b/contrib/gcc-changelog/test_email.py
index d522e6ef7e3..aa516c6e6d1 100755
--- a/contrib/gcc-changelog/test_email.py
+++ b/contrib/gcc-changelog/test_email.py
@@ -276,3 +276,7 @@ class TestGccChangelog(unittest.TestCase):
 def test_dr_entry(self):
 email = self.from_patch_glob('0001-c-C-20-DR-2237.patch')
 assert email.changelog_entries[0].prs == ['DR 2237']
+
+def test_changes_only_in_ignored_location(self):
+email = self.from_patch_glob('0001-go-in-ignored-location.patch')
+assert not email.errors
diff --git a/contrib/gcc-changelog/test_patches.txt b/contrib/gcc-changelog/test_patches.txt
index 3445c3d9f11..58fd81c85c9 100644
--- a/contrib/gcc-changelog/test_patches.txt
+++ b/contrib/gcc-changelog/test_patches.txt
@@ -2568,3 +2568,46 @@ index a6a5d975af3..a8082d39aca 100644
 @@ -1 +1,2 @@
 
 +
+
+=== 0001-go-in-ignored-location.patch ===
+From 81994eab700da7fea6644541c163aa0f0f3b8cf1 Mon Sep 17 00:00:00 2001
+From: =?UTF-8?q?Cl=C3=A9ment=20Chigot?= 
+Date: Tue, 19 May 2020 16:03:54 +0200
+Subject: libgo: update x/sys/cpu after gccgo support added
+
+Reviewed-on: https://go-review.googlesource.com/c/gofrontend/+/234597
+---
+ gcc/go/gofrontend/MERGE   |  2 +-
+ .../sys/cpu/{cpu_aix_ppc64.go => cpu_aix.go}  |  2 +-
+ .../golang.org/x/sys/cpu/syscall_aix_gccgo.go | 27 +++
+ 3 files changed, 29 insertions(+), 2 deletions(-)
+ rename libgo/go/golang.org/x/sys/cpu/{cpu_aix_ppc64.go => cpu_aix.go} (96%)
+ create mode 100644 libgo/go/golang.org/x/sys/cpu/syscall_aix_gccgo.go
+
+diff --git a/gcc/go/gofrontend/MERGE b/gcc/go/gofrontend/MERGE
+index bc9c1f07eda..284374820b0 100644
+--- a/gcc/go/gofrontend/MERGE
 b/gcc/go/gofrontend/MERGE
+@@ -1 +1,2 @@
+
++
+diff --git a/libgo/go/golang.org/x/sys/cpu/cpu_aix_ppc64.go b/libgo/go/golang.org/x/sys/cpu/cpu_aix.go
+similarity index 96%
+rename from libgo/go/golang.org/x/sys/cpu/cpu_aix_ppc64.go
+rename to libgo/go/golang.org/x/sys/cpu/cpu_aix.go
+index b0ede112d4e..02d03129e50 100644
+--- a/libgo/go/golang.org/x/sys/cpu/cpu_aix_ppc64.go
 b/libgo/go/golang.org/x/sys/cpu/cpu_aix.go
+@@ -1 +1,2 @@
+
++
+diff --git a/libgo/go/golang.org/x/sys/cpu/syscall_aix_gccgo.go b/libgo/go/golang.org/x/sys/cpu/syscall_aix_gccgo.go
+new file mode 100644
+index 000..2609cc49ae7
+--- /dev/null
 b/libgo/go/golang.org/x/sys/cpu/syscall_aix_gccgo.go
+@@ -0,0 +1 @@
++
+
+-- 
+2.27.0.rc0.183.gde8f92d652-goog
-- 
2.26.2

Re: New mklog script

[Martin Liška]

On 5/22/20 6:43 PM, Martin Sebor wrote:

On 5/21/20 2:16 AM, Martin Liška wrote:

Hello Martin.

Can you please compare the current mklog.py. Is there anything
you miss compared to your current script?


Nope, it matches the format I get with my script and even works
better and runs faster.  Very nice!  I'll be happy to switch to
using it instead.


Great, good to hear!



Thanks!
Martin

PS A couple of ideas for future enhancements are to have the script
print "New function." or "New type." for newly added functions and
types, and to print "Adjust comments." for changes to comments alone.


Feel free to send patches for the new script ;)

Martin

Re: ChangeLog files - server and client scripts

[Martin Liška]

On 5/21/20 5:14 PM, Rainer Orth wrote:

* In changelog_location, you allow only (among others) "a/b/c/" and
   "\ta/b/c/".  Please also accept the "a/b/c:" and "\ta/b/c:" forms
   here: especially the second seems quite common.


Ok, I believe these formats are supported as well. Feel free to mention
some git revisions that are not recognized.

Thanks,
Martin

Re: New mklog script

[Martin Liška]

On 5/22/20 11:01 PM, Jason Merrill wrote:

On Thu, May 21, 2020 at 6:03 PM Jason Merrill  wrote:


On Fri, May 15, 2020 at 11:39 AM Martin Liška  wrote:


On 5/15/20 3:22 PM, Marek Polacek wrote:

On Fri, May 15, 2020 at 03:12:27PM +0200, Martin Liška wrote:

On 5/15/20 2:42 PM, Marek Polacek wrote:

I actually use mklog -i all the time.  But I can work around it if it
disappears.


Ah, I can see a consumer.
There's an updated version that supports that.

For the future, will you still use the option? Wouldn't be better
to put the ChangeLog content directly to commit message? Note
that you won't have to copy the entries to a particular ChangeLog file.


The way I do it is to generate a patch using format-patch, use mklog -i
on it, then add the ChangeLog entry to the commit message via commit --amend.


Hmm, you can do much better with:

$ git diff | ./contrib/mklog > changelog && git commit -a -t changelog

Or for an already created commit you can do:

$ git diff HEAD~ | ./contrib/mklog > changelog && git commit -a --amend -e -F 
changelog


With these git aliases:

 mklog-editor = "!f() { git show | git gcc-mklog >> $1; }; f"
 addlog = "!f() { GIT_EDITOR='git mklog-editor' git commit --amend; }; 
f"

I can 'git addlog' to append the output of mklog to the current
commit.  Probably better would be to do something with
prepare-commit-msg.


This is pretty rudimentary, but good enough as a start:


I like the idea of usage of the prepare commit hook.



#!/bin/sh

#COMMIT_MSG_FILE=$1
#COMMIT_SOURCE=$2
#SHA1=$3


It's better to use the named arguments.



if ! [ -f "$1" ]; then exit 0; fi

#echo "# $0 $1 $2 $3" >> $1

if fgrep 'ChangeLog:' $1 > /dev/null 2>&1; then exit 0; fi

if [ -z "$2" ]; then
 cmd="diff --cached"
elif [ $2 == commit ]; then
 cmd="show $3"
else
 exit 0
fi

git $cmd | git gcc-mklog >> $1



Well, that will generate changelog entry for each commit.
For a user branch development, it's not desirable.

What about more explicit approach:

1) making an alias for: git diff | git gcc-mklog > commit.msg
2) hook:

if test -f commit.msg; then
  cat commit.msg >> "$COMMIT_MSG_FILE"
  rm commit.msg
fi

So the changelog is created explicitly and included implicitly.

Martin

Re: [IMPORTANT] ChangeLog related changes

[Martin Liška]

On 5/26/20 7:22 AM, Hongtao Liu via Gcc wrote:

i commit a separate patch alone only for ChangeLog files, should i revert it?


Hello.

I've just done it.

Martin

Re: ChangeLog files - server and client scripts

[Martin Liška]

On 5/26/20 7:31 AM, Alexandre Oliva wrote:

On May 25, 2020, Martin Liška  wrote:


On 5/21/20 5:14 PM, Rainer Orth wrote:

* In changelog_location, you allow only (among others) "a/b/c/" and
"\ta/b/c/".  Please also accept the "a/b/c:" and "\ta/b/c:" forms
here: especially the second seems quite common.



Ok, I believe these formats are supported as well. Feel free to mention
some git revisions that are not recognized.


Hello.



I've long used the following syntax to start ChangeLog entries:

for  /ChangeLog


Ah, it's new for me.



It was introduced over 20 years ago, with the (so far never formally
released) GNU CVS-Utilities.  Among other goodies, there were scripts to
turn diffs for ChangeLog files into the above format, and vice-versa,
that I've used to this day.  It went through cvs, svn and git.  It would
be quite nice if I could keep on using it with GCC.


Sure. Starting from now, you don't need to put ChangeLog entries to their
corresponding files, it will be done automatically.



The patch below seems to be enough to pass gcc-verify, and to recognize
and print the expected ChangeLog files.  I suppose I'll have to adjust
the formatting to be able to push it, but, aside from that, is it ok to
install?


I'm fine with the patch. Alternative approach is to start using
./contrib/mklog.py (a.k.a. git mklog).



Do any hooks need to be adjusted to match?


Yes, we sync the script from the GCC repository.




I'm also a little concerned about '*/ChangeLog.*' files.  Are we no
longer supposed to introduce them, or new ChangeLog entries to them?  Or
should the scripts be extended to cover them?


Right now we cover only ChangeLog files, so e.g. ChangeLog.dataflow is not 
affected
(and checked). For newly added ChangeLog files, we can add them, but they must 
first
appear in the git_commit.py script where we list all allowed locations.

Martin




for  contrib/ChangeLog

* gcc-changelog/git_commit.py (changelog_regex): Accept optional
'for' prefix.

diff --git a/contrib/gcc-changelog/git_commit.py 
b/contrib/gcc-changelog/git_commit.py
index 2cfdbc8..b8362c1 100755
--- a/contrib/gcc-changelog/git_commit.py
+++ b/contrib/gcc-changelog/git_commit.py
@@ -144,7 +144,7 @@ misc_files = [
  author_line_regex = \
  re.compile(r'^(?P\d{4}-\d{2}-\d{2})\ {2}(?P.*  <.*>)')
  additional_author_regex = re.compile(r'^\t(?P\ *)?(?P.*  <.*>)')
-changelog_regex = re.compile(r'^([a-z0-9+-/]*)/ChangeLog:?')
+changelog_regex = re.compile(r'^(?:[fF]or +)([a-z0-9+-/]*)/ChangeLog:?')
  pr_regex = re.compile(r'\tPR (?P[a-z+-]+\/)?([0-9]+)$')
  dr_regex = re.compile(r'\tDR ([0-9]+)$')
  star_prefix_regex = re.compile(r'\t\*(?P\ *)(?P.*)')

Re: New mklog script

[Martin Liška]

On 5/26/20 12:23 PM, Richard Earnshaw wrote:

I thought we had a convention that aliases we added were prefixed with
'gcc-'?  This seems to go against that.


You are right, but this one is so handy ;)
What name do you suggest?

Martin

Re: Automatically generated ChangeLog files - script

[Martin Liška]

On 5/26/20 12:15 PM, Pierre-Marie de Rodat wrote:

Hello Martin,

First, thank you for your work on this new ChangeLog workflow. :-)


Hello.

Thank you.



I’d like to report a “regression”: I can’t push the attached patch:

remote: *** ChangeLog format failed:
remote: ERR: changed file not mentioned in a ChangeLog:"gcc/ada/sem_ch4.adb"
remote: ERR: changed file not mentioned in a ChangeLog:"gcc/ada/sem_ch7.adb"
remote: ERR: changed file not mentioned in a ChangeLog:"gcc/ada/sem_ch8.adb"
remote: ERR: changed file not mentioned in a ChangeLog:"gcc/ada/sem_elab.adb"
remote: ERR: changed file not mentioned in a ChangeLog:"gcc/ada/sem_type.adb"
remote: ERR: changed file not mentioned in a ChangeLog:"gcc/ada/sem_util.adb"
remote: remote: Please see: 
https://gcc.gnu.org/codingconventions.html#ChangeLogs
remote: remote: error: hook declined to update refs/heads/master


It looks like the hook does not accept multi-line ChangeLog entries affecting 
multiple files:

    * contracts.adb, einfo.adb, exp_ch9.adb, sem_ch12.adb,
    sem_ch4.adb, sem_ch7.adb, sem_ch8.adb, sem_elab.adb,
    sem_type.adb, sem_util.adb: Reuse Is_Package_Or_Generic_Package
    where possible (similarly, reuse Is_Concurrent_Type if it was
    possible in the same expressions).


Would it be possible to enhance the hook to support that?


It's not supported right now and it will make the filename parsing much more 
complicated.
I would recommend using ./contrib/mklog and using:

* a.adb: Foo bar.
* b.adb: Likewise.
...

Thanks,
Martin



Thanks!

Re: [IMPORTANT] ChangeLog related changes

[Martin Liška]

On 5/26/20 1:34 PM, Jakub Jelinek via Gcc wrote:

On Tue, May 26, 2020 at 12:27:59PM +0100, Richard Earnshaw wrote:

I haven't investigated in detail, but could we use a merge strategy with
the cherry-pick to drop ChangeLog entries?


If that works, sure.
Note, when cherry-picking commits from before conversion to git or whenever
people started to write usable ChangeLog entries in the commit messages,
one will also need to reconstruct the commit message additions from the
ChangeLog files.

Jakub



Hi.

There's a script candidate that does git cherry-pick and drops
and ChangeLog changes from both index and conflicting files.

Thoughts?
Martin
#!/usr/bin/env python3

# Copyright (C) 2020 Free Software Foundation, Inc.
#
# This file is part of GCC.
#
# GCC is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 3, or (at your option)
# any later version.
#
# GCC is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with GCC; see the file COPYING.  If not, write to
# the Free Software Foundation, 51 Franklin Street, Fifth Floor,
# Boston, MA 02110-1301, USA.

import argparse
import subprocess

if __name__ == '__main__':
parser = argparse.ArgumentParser(description='Backport a git revision and '
 'stash all ChangeLog files.')
parser.add_argument('revision', help='Revision')
args = parser.parse_args()

r = subprocess.run('git cherry-pick -x %s' % args.revision, shell=True)
if r.returncode == 0:
subprocess.check_output('git show --name-only --pretty="" | '
'grep ChangeLog | '
'xargs git checkout HEAD~', shell=True)
subprocess.check_output('git commit --amend --no-edit', shell=True)
else:
# 1) remove all ChangeLog files from conflicts
out = subprocess.check_output('git diff --name-only --diff-filter=U',
  shell=True,
  encoding='utf8')
conflicts = out.strip().split('\n')
changelogs = [c for c in conflicts if c.endswith('ChangeLog')]
if changelogs:
cmd = 'git checkout --theirs %s' % '\n'.join(changelogs)
subprocess.check_output(cmd, shell=True)
# 2) remove all ChangeLog files from index
out = subprocess.check_output('git diff --name-only --diff-filter=M HEAD',
  shell=True,
  encoding='utf8')
out = out.strip().split('\n')
modified = [c for c in out if c.endswith('ChangeLog')]
for m in modified:
subprocess.check_output('git reset %s' % m, shell=True)
subprocess.check_output('git checkout %s' % m, shell=True)
 
# try to continue
if len(conflicts) == len(changelogs):
subprocess.check_output('git cherry-pick --continue', shell=True)
else:
print('Please resolve all remaining file conflicts.')

Re: Automatically generated ChangeLog files - script

[Martin Liška]

On 5/26/20 2:35 PM, Rainer Orth wrote:

Hi Martin,


It looks like the hook does not accept multi-line ChangeLog entries
affecting multiple files:

     * contracts.adb, einfo.adb, exp_ch9.adb, sem_ch12.adb,
     sem_ch4.adb, sem_ch7.adb, sem_ch8.adb, sem_elab.adb,
     sem_type.adb, sem_util.adb: Reuse Is_Package_Or_Generic_Package
     where possible (similarly, reuse Is_Concurrent_Type if it was
     possible in the same expressions).


Would it be possible to enhance the hook to support that?


It's not supported right now and it will make the filename parsing much
more complicated.


however, that's a format Emacs' ChangeLog mode uses and supports, and
it's way less chatty than the one-file-per-line one, both for writers
and readers.


I see, but as mentioned it makes the parsing of the list files much more
complicated. Feel free to provide a patch that will support multi-line
entries.

Martin



Rainer

Re: New mklog script

[Martin Liška]

On 5/26/20 1:18 PM, Richard Earnshaw wrote:

On 26/05/2020 12:14, Martin Liška wrote:

On 5/26/20 12:23 PM, Richard Earnshaw wrote:

I thought we had a convention that aliases we added were prefixed with
'gcc-'?  This seems to go against that.


You are right, but this one is so handy ;)
What name do you suggest?

Martin


gcc-ci?


What the abbreviation stands for?

Martin



R.

Re: New mklog script

[Martin Liška]

On 5/26/20 3:11 PM, Richard Earnshaw wrote:

On 26/05/2020 14:09, Martin Liška wrote:

On 5/26/20 1:18 PM, Richard Earnshaw wrote:

On 26/05/2020 12:14, Martin Liška wrote:

On 5/26/20 12:23 PM, Richard Earnshaw wrote:

I thought we had a convention that aliases we added were prefixed with
'gcc-'?  This seems to go against that.


You are right, but this one is so handy ;)
What name do you suggest?

Martin


gcc-ci?


What the abbreviation stands for?

Martin



R.





CheckIn

For those who come from the SVN days where ci was the standard
abbreviation for committing :-)


Ah, I see. Anyway, I prefer the original name even though it violates
the naming policy.

Let other express their preferences (Jason?).

Martin



R.