cortexa57_extra_costs' alu.shift_reg

[Andrew Pinski]

Hi,
  I was looking into reassoc (for PR 93131) and I noticed that the
alu.shift_reg is set to COSTS_N_INSNS (1).  This prevents an
optimization where we combine some if statements into shifts.  I
looked into the Corext A57 software optimization guide[1] and saw that
shift with a register has a latency of 1 which is the same as a shift
with an immediate.  I looked in the history but it was there to begin
with. Is there a reason for this and would it be acceptable if we
change a57's costs to be correct?

I looked into A65, A72, A75, A76, A77 and N1 software optimization
guides too and they have a similar thing of a latency of one cycle and
those are the ones which use the cortexa57_extra_costs variable too.
Note I could not find the A73 guide though.

Thanks,
Andrew Pinski


[1] 
http://infocenter.arm.com/help/topic/com.arm.doc.uan0015b/Cortex_A57_Software_Optimization_Guide_external.pdf

Re: GIT: Monotonically increasing trunk and release branch ids

[Jakub Jelinek]

On Fri, Jan 10, 2020 at 07:50:02PM +0100, Jakub Jelinek wrote:
> The changes I was asking for is, for gcc master and releases/gcc-* branch
> commits to have the monotonically increasing short ids (printed by git descr
>  with the git aliases I've posted) both somewhere early
> in the subject before the one-line description, and somewhere in the body
> too.  For release branch commits, perhaps the

Here it is in patch form (against Joseph's git-hooks copy), but untested
(both because the repo isn't converted yet, basepoints don't exist there
either and because not sure how exactly it can be tested in a dry-run, say
sending me mail instead of to the public list etc.).
So, all I did test is some simple thing in python3 interactively because
it is close to 20 years since I used python regularly.

diff --git a/hooks/updates/__init__.py b/hooks/updates/__init__.py
index c5d8181..4ddc566 100644
--- a/hooks/updates/__init__.py
+++ b/hooks/updates/__init__.py
@@ -299,11 +299,31 @@ class AbstractUpdate(object):
 # name in full to label the branch name.
 branch = '(%s)' % self.ref_name
 
-subject = '[%(repo)s%(branch)s] %(subject)s' % {
-'repo': self.email_info.project_name,
-'branch': branch,
-'subject': commit.subject[:SUBJECT_MAX_SUBJECT_CHARS],
-}
+# GCC monotonically increasing commit ids.
+rev_id = ''
+rev_id_short = ''
+if self.email_info.project_name == 'gcc'
+   and (self.short_ref_name == 'master'
+or self.short_ref_name.startswith('releases/gcc-')):
+rev_id = git.describe(commit.rev, all=True,
+  match='basepoints/gcc-[0-9]*')
+if rev_id.starts_with('tags/basepoints/gcc-'):
+rev_id = 'r' + rev_id[len('tags/basepoints/gcc-'):]
+rev_id_short = rev_id[:rev_id.find('-g')]
+else:
+rev_id = ''
+
+if not rev_id:
+subject = '[%(repo)s%(branch)s] %(subject)s' % {
+'repo': self.email_info.project_name,
+'branch': branch,
+'subject': commit.subject[:SUBJECT_MAX_SUBJECT_CHARS],
+}
+else:
+subject = '%(rev_id)s %(subject)s' % {
+'rev_id': rev_id_short,
+'subject': commit.subject[:SUBJECT_MAX_SUBJECT_CHARS],
+}
 
 # Generate the body of the email in two pieces:
 #   1. The commit description without the patch;
@@ -322,6 +342,10 @@ class AbstractUpdate(object):
 # by stripping it from the output.
 
 body = git.log(commit.rev, max_count="1") + '\n'
+
+if rev_id:
+body = re.sub(r'^commit ' + commit.rev', 'commit ' + rev_id,
+  body, 1, re.M)
 if git_config('hooks.commit-url') is not None:
 url_info = {'rev': commit.rev,
 'ref_name': self.ref_name}


Jakub

Re: cortexa57_extra_costs' alu.shift_reg

[Andrew Pinski]

On Sat, Jan 11, 2020 at 2:02 AM Andrew Pinski  wrote:
>
> Hi,
>   I was looking into reassoc (for PR 93131) and I noticed that the
> alu.shift_reg is set to COSTS_N_INSNS (1).  This prevents an
> optimization where we combine some if statements into shifts.  I
> looked into the Corext A57 software optimization guide[1] and saw that
> shift with a register has a latency of 1 which is the same as a shift
> with an immediate.  I looked in the history but it was there to begin
> with. Is there a reason for this and would it be acceptable if we
> change a57's costs to be correct?
>
> I looked into A65, A72, A75, A76, A77 and N1 software optimization
> guides too and they have a similar thing of a latency of one cycle and
> those are the ones which use the cortexa57_extra_costs variable too.
> Note I could not find the A73 guide though.

I forgot to mention this causes lshift_cheap_p to return false which
is not true for these cores.

Thanks,
Andrew Pinski

>
> Thanks,
> Andrew Pinski
>
>
> [1] 
> http://infocenter.arm.com/help/topic/com.arm.doc.uan0015b/Cortex_A57_Software_Optimization_Guide_external.pdf

Re: GIT: Monotonically increasing trunk and release branch ids

[Jakub Jelinek]

On Fri, Jan 10, 2020 at 10:47:47PM +0100, Jakub Jelinek wrote:
> Ah, you suggested g: rather than just g.
> We could then support
> rN (1-6 decimal digits) - the svn revs, either for old repo, or 
> transformed
> g:X (X is any [0-9a-zA-Z_-], something else?, 1 or more chars) - gitweb 
> with X
> rNN-M (1-2 decimal digits, 1-6 decimal digits) - git undescr to hash
> rNN-M-gX (perhaps limit X to hashes here, i.e. 7-40 hex digits) - 
> gitweb with X
> both in bugzilla and in the redirectors.

For the redirectors, it could be something like following patch, except the
last new redirect would need also a yet to be written cgi script that would
git undescr the argument and print
Location: https://gcc.gnu.org/git/gitweb.cgi?p=gcc.git;h=...

--- /etc/httpd/conf.d/sourceware-vhost-gcc.include  2019-11-21 
19:25:03.172399228 +
+++ /etc/httpd/conf.d/sourceware-vhost-gcc.include  2020-01-11 
11:22:00.752261787 +
@@ -59,6 +59,11 @@
   # Support short URLs for referring to SVN revisions.
   RedirectMatch ^/r([0-9]+)$   
https://gcc.gnu.org/viewcvs/gcc?view=revision\&revision=$1
 
+  # Similarly for GIT commits.
+  RedirectMatch ^/g:([0-9a-zA-Z._{}~^-]+)$ 
https://gcc.gnu.org/git/gitweb.cgi?p=gcc.git;h=$1
+  RedirectMatch ^/r[0-9]+-[0-9]+-g([0-9a-f]+)$ 
https://gcc.gnu.org/git/gitweb.cgi?p=gcc.git;h=$1
+  RedirectMatch ^/(r[0-9]+-[0-9]+)$
https://gcc.gnu.org/cgi-bin/gcc-gitref.cgi?r=$1
+
 # ViewCVS setup
   
   RewriteRule ^/cgi-bin/cvsweb.cgi https://gcc.gnu.org/viewvc/gcc [L]


Jakub

Re: Proposal for the transition timetable for the move to GIT

[Segher Boessenkool]

On Thu, Jan 09, 2020 at 12:12:49PM +, Richard Earnshaw (lists) wrote:
> On 09/01/2020 02:38, Segher Boessenkool wrote:
> >Where and when and by who was it decided to use this conversion?
> >
> >Will it at least be *tested* first?
> 
> Tested for what?

Acceptance test, of course, the only test that matters.  I.e. the GCC
community gets to decide if this conversion is acceptable to them, instead
of being confronted with it as a fait accompli.

> I want to also take this opportunity to thank Maxim for the work he has 
> done.  Having that fallback option has meant that we could press harder 
> for a timely solution and has also driven several significant 
> improvements to the overall result.  I do not think we would have 
> achieved as good a result overall if he hadn't developed his scripts.

And my thanks go to you and everyone else who tried to make this result
in a git conversion that is the most useful for us, the GCC developers
(and other consumers of our repo)!


Segher

Re: Proposal for the transition timetable for the move to GIT

[Segher Boessenkool]

On Fri, Jan 10, 2020 at 09:49:41AM +, Richard Earnshaw (lists) wrote:
> On 10/01/2020 07:33, Maxim Kuvyrkov wrote:
> >>On Jan 9, 2020, at 5:38 AM, Segher Boessenkool 
> >> wrote:
> >>Where and when and by who was it decided to use this conversion?
> >
> >Joseph, please point to message on gcc@ mailing list that expresses 
> >consensus of GCC community to use reposurgeon conversion.  Otherwise, it 
> >is not appropriate to substitute one's opinion for community consensus.
> 
> I've gone back through this thread (if I've missed, or misrepresented, 
> anybody who's expressed an opinion I apologize now).
> 
> Segher Boessenkool 
> "If Joseph and Richard agree a candidate is good, then I will agree as
> well.  All that can be left is nit-picking, and that is not worth it
> anyway:"

That is not saying I agree the reposurgeon conversion is best if you two
agree.  It says that if you think that is a good conversion, then I agree.
However I do still think it is the worst of the three options, in some
regards.

> So I don't see any clear dissent and most folks just want to get this 
> done.

Yes.  After the GCC community took over five years to decide to switch
to git, and then we were delayed by another almost five years because it
just *had* to be done using reposurgeon, we just want it *done*, and even
the reposurgeon option is acceptable, in my book.

I don't look at old commit messages *at all* (*).  Mangled patch authors
can be harder, but I do have old trees as well, worst case.  We'll
survive, the info in the changelogs is still there.  And hopefully new
patches will eventually have good author info and commit messages.

To a gitty future, onwards and upwards, etc.,


Segher


(*) That's a lie: I look at it a lot, but only to extract the SVN
revision number from it!

Re: Proposal for the transition timetable for the move to GIT

[Segher Boessenkool]

On Fri, Jan 10, 2020 at 12:38:10PM +0100, Richard Biener wrote:
> Just to chime in I also just want to get it done (well, I can handle
> SVN as well :P).

I will never have to learn it!  I'm so happy!

> I trust Joseph, too, but then from my POV anything not worse than the current
> mirror works for me.  Thanks to Maxim anyway for all the work - without that
> we'd not switch in 10 other years...

Absolutely agreed!  Thank you, Maxim.


Segher

Re: [RFC] builtin functions and `-ffreestanding -nostartfies` with static binaries

[Segher Boessenkool]

Hi!

On Fri, Jan 10, 2020 at 09:50:48PM +0530, Siddhesh Poyarekar wrote:
> Statically built independent programs that implement their own program
> entry points (i.e. -ffreestanding -nostartfiles) and call __builtin_*
> functions break when the builtin function in question is implemented as
> an IFUNC in glibc and the builtin results in a glibc call instead of
> some inline code.
> 
> This happens because the startup code where ifuncs are resolved never
> gets executed (since glibc's startup code is never executed) and hence
> the PLT jumps fail.  The bug report talks about this as an aarch64
> problem but I've been able to reproduce the problem on x86_64 as well.
> One just needs to make sure that the __builtin_foo call results in a
> glibc call.

-ffreestanding means you might not have any of the C standard library,
and -nostartfiles means you do not do any of the standard initialisation.
Why then would you expect any ifunc to work?

> I spent some time thinking about this and while it's trivial to fix by
> disabling ifuncs for static glibc, I wanted a solution that wasn't such
> a big hammer.  The other alternative I could think of is to have an
> exported alias (called __builtin_strlen for example instead of strlen)
> of a default implementation of the builtin function in glibc that gcc
> generates a call to if freestanding && nostartfiles && static.
> 
> Any thoughts or other ideas on how this could be implemented?

Why do you not want the startfiles, but do want their effects?


Segher

Re: git conversion in progress

[Richard Earnshaw]

On 11/01/2020 01:18, Joseph Myers wrote:
> The GCC SVN repository is now read-only for the move to git, as is the old 
> git-svn mirror; the cron job updating that mirror has been disabled, as 
> have gccadmin's cron jobs updating DATESTAMP, generating snapshots and 
> updating online documentation, until those are updated to work from git.
> 
> Assuming validation passes, I expect to make the converted repository 
> available in place of the git-svn mirror (git://gcc.gnu.org/git/gcc.git 
> and git+ssh://gcc.gnu.org/git/gcc.git) some time on Saturday.  That will 
> be initially read-only to allow for people to review it in case any 
> significant conversion issues are noticed late that would justify 
> rerunning the conversion.  I propose to make it writable on Monday, at 
> which point we'll be in development stage 4 for GCC 10 (and to leave 
> gcc-reposurgeon-8.git around for a while in case people wish to use that, 
> rather than the main repository, for any further testing, but to delete 
> the older test conversions 1 through 7).  Some cron jobs may be re-enabled 
> before then, subject to testing (I have git changes to gcc_release ready, 
> for example, for testing snapshot generation), but the DATESTAMP updates 
> won't be enabled until the repository is writable.
> 
> I encourage people to continue to work on improving the documentation for 
> using git with GCC 
> ( and 
>  list some of 
> the things that it seems it might be useful to document).  Once the 
> conversion is done we'll need to update various places referring to SVN on 
> the website and in the manuals.
> 
> The old git-svn mirror will remain available read-only as 
> git://gcc.gnu.org/git/gcc-old.git (which is already available), in 
> addition to the refs from that mirror being in the new repository as 
> refs/git-old/heads/*, refs/git-old/tags/* and refs/git-svn-old/heads/* 
> (thus, gitweb links to commits in that repository will continue to work, 
> as long as those commits were reachable as of when git mirror went 
> read-only rather than having been on an old version of a branch that was 
> since deleted or rebased).
> 
> Where people have active branches in that mirror, I hope they will rebase 
> those branches onto the new version of the history, putting the results in 
> refs/heads/devel/* or refs/users/*/heads/* according to whether they are 
> considered generally shared development branches (can't have 
> non-fast-forward pushes, should be documented on the GCC website) or user 
> branches (can have non-fast-forward pushes, need not be documented).
> 

Joseph is unlikely to upload his conversion to gcc.gnu.org until all the
tests have been completed.  However, I've done a parallel conversion
using the same scripts and the same version of reposurgeon.

For those that are desperate for an early look, I've uploaded a copy of
this to gitlab: https://gitlab.com/rearnsha/gcc.

Beware that testing is *still* in progress, but hopefully this will be
identical to the version that Joseph finally uploads.

Please let us know, ASAP, if you see any *critical* issues.

R.

Re: [RFC] builtin functions and `-ffreestanding -nostartfies` with static binaries

[Siddhesh Poyarekar]

On 11/01/20 6:55 pm, Segher Boessenkool wrote:
> -ffreestanding means you might not have any of the C standard library,
> and -nostartfiles means you do not do any of the standard initialisation.
> Why then would you expect any ifunc to work?
> 

Agreed, based on Alexander's feedback I have suggested changing the
tests instead.

Siddhesh

Re: GIT: Monotonically increasing trunk and release branch ids

[Segher Boessenkool]

On Fri, Jan 10, 2020 at 07:50:02PM +0100, Jakub Jelinek wrote:
> Maybe even add https://gcc.gnu.org/gNNN for 7-40 hexadecimal digits
> redirect to https://gcc.gnu.org/git/gitweb.cgi?p=gcc.git;h=NNN
> and then just put as URL those
> https://gcc.gnu.org/g3b2b821c3f1bac2ac6dd2481f461ec33a13eac9b
> URLs instead to make it somewhat shorter, and again teach bugzilla to
> URLize g3b2b821c3f1bac2ac6dd2481f461ec33a13eac9b with 7-40 hexadecimal
> digits. 

Please never post links with a short hash.  Those *will* collide after a
while, making your link go stale.  Use at least ten chars, or twelve to
be save (and it is a nice round number).

(It is of course fine for the redirect itself to allow short names).


Segher

Re: GIT: Monotonically increasing trunk and release branch ids

[Jakub Jelinek]

On Sat, Jan 11, 2020 at 07:43:17AM -0600, Segher Boessenkool wrote:
> On Fri, Jan 10, 2020 at 07:50:02PM +0100, Jakub Jelinek wrote:
> > Maybe even add https://gcc.gnu.org/gNNN for 7-40 hexadecimal digits
> > redirect to https://gcc.gnu.org/git/gitweb.cgi?p=gcc.git;h=NNN
> > and then just put as URL those
> > https://gcc.gnu.org/g3b2b821c3f1bac2ac6dd2481f461ec33a13eac9b
> > URLs instead to make it somewhat shorter, and again teach bugzilla to
> > URLize g3b2b821c3f1bac2ac6dd2481f461ec33a13eac9b with 7-40 hexadecimal
> > digits. 
> 
> Please never post links with a short hash.  Those *will* collide after a
> while, making your link go stale.  Use at least ten chars, or twelve to
> be save (and it is a nice round number).

Yeah, the 7 was just that it is what git describe prints by default,
in the git-hooks tentative patch I've posted today it would use the
r10-1234 style ids in the subject (those will never collide) and in
the body the
r10-1234-g3b2b821c3f1bac2ac6dd2481f461ec33a13eac9b
style one.

Jakub

Re: GIT: Monotonically increasing trunk and release branch ids

[Jason Merrill]

On Sat, Jan 11, 2020 at 6:28 AM Jakub Jelinek  wrote:

> On Fri, Jan 10, 2020 at 10:47:47PM +0100, Jakub Jelinek wrote:
> > Ah, you suggested g: rather than just g.
> > We could then support
> > rN (1-6 decimal digits) - the svn revs, either for old repo, or
> transformed
> > g:X (X is any [0-9a-zA-Z_-], something else?, 1 or more chars) -
> gitweb with X
>

I was thinking that anything at all after g: should go to gitweb.


> > rNN-M (1-2 decimal digits, 1-6 decimal digits) - git undescr to hash
> > rNN-M-gX (perhaps limit X to hashes here, i.e. 7-40 hex digits)
> - gitweb with X
>

Why complicate this with the added 'g'?

Jason

Re: GIT: Monotonically increasing trunk and release branch ids

[Jakub Jelinek]

On Sat, Jan 11, 2020 at 09:15:11AM -0500, Jason Merrill wrote:
> > On Fri, Jan 10, 2020 at 10:47:47PM +0100, Jakub Jelinek wrote:
> > > Ah, you suggested g: rather than just g.
> > > We could then support
> > > rN (1-6 decimal digits) - the svn revs, either for old repo, or
> > transformed
> > > g:X (X is any [0-9a-zA-Z_-], something else?, 1 or more chars) -
> > gitweb with X
> >
> 
> I was thinking that anything at all after g: should go to gitweb.

For the redirect perhaps.  For bugzilla comments, it needs to know where it
does stop and shouldn't match unrelated text, plus as I wrote in the
bugzilla tweaks PR, not sure how smart are e.g. g:master or g:master^^ or
g:releases/gcc-9@{yesterday} references (and, there we at least shouldn't
match whitespace).

> > > rNN-M (1-2 decimal digits, 1-6 decimal digits) - git undescr to hash
> > > rNN-M-gX (perhaps limit X to hashes here, i.e. 7-40 hex digits)
> > - gitweb with X
> >
> 
> Why complicate this with the added 'g'?

The advantage of the g in there is that git then can directly understand it:
git rev-parse r9-7160-ga5bc16f3199
a5bc16f3199c9aa43aec5af2c2839a1cd4bfce1e
while if the g is removed, it doesn't.

Jakub

Re: git conversion in progress

[Joseph Myers]

On Sat, 11 Jan 2020, Richard Earnshaw wrote:

> On 11/01/2020 01:18, Joseph Myers wrote:
> > The GCC SVN repository is now read-only for the move to git, as is the old 
> > git-svn mirror; the cron job updating that mirror has been disabled, as 
> > have gccadmin's cron jobs updating DATESTAMP, generating snapshots and 
> > updating online documentation, until those are updated to work from git.
> > 
> > Assuming validation passes, I expect to make the converted repository 
> > available in place of the git-svn mirror (git://gcc.gnu.org/git/gcc.git 
> > and git+ssh://gcc.gnu.org/git/gcc.git) some time on Saturday.  That will 
> > be initially read-only to allow for people to review it in case any 
> > significant conversion issues are noticed late that would justify 
> > rerunning the conversion.  I propose to make it writable on Monday, at 
> > which point we'll be in development stage 4 for GCC 10 (and to leave 

This conversion is now in place, read-only for checking purposes.  I've 
done all the usual validation, including in particular checking branch 
tips and tags against SVN.

> For those that are desperate for an early look, I've uploaded a copy of
> this to gitlab: https://gitlab.com/rearnsha/gcc.
> 
> Beware that testing is *still* in progress, but hopefully this will be
> identical to the version that Joseph finally uploads.

Richard's conversion was using older Bugzilla data (I updated the data 
from Bugzilla right before running my final conversion, Richard didn't), 
so they don't have identical commit summaries everywhere.

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: git conversion in progress

[Jakub Jelinek]

On Sat, Jan 11, 2020 at 02:39:43PM +, Joseph Myers wrote:
> > On 11/01/2020 01:18, Joseph Myers wrote:
> > > The GCC SVN repository is now read-only for the move to git, as is the 
> > > old 
> > > git-svn mirror; the cron job updating that mirror has been disabled, as 
> > > have gccadmin's cron jobs updating DATESTAMP, generating snapshots and 
> > > updating online documentation, until those are updated to work from git.
> > > 
> > > Assuming validation passes, I expect to make the converted repository 
> > > available in place of the git-svn mirror (git://gcc.gnu.org/git/gcc.git 
> > > and git+ssh://gcc.gnu.org/git/gcc.git) some time on Saturday.  That will 
> > > be initially read-only to allow for people to review it in case any 
> > > significant conversion issues are noticed late that would justify 
> > > rerunning the conversion.  I propose to make it writable on Monday, at 
> > > which point we'll be in development stage 4 for GCC 10 (and to leave 
> 
> This conversion is now in place, read-only for checking purposes.  I've 
> done all the usual validation, including in particular checking branch 
> tips and tags against SVN.

Thanks.
Is there or will there be any way using gitweb to browse the vendor and user
branches (and also even query which vendors/users and their branches there
are)?
I've tried
https://gcc.gnu.org/git/?p=gcc.git;a=shortlog;h=refs/heads/vendors/
https://gcc.gnu.org/git/?p=gcc.git;a=shortlog;h=refs/heads/vendors/redhat/
https://gcc.gnu.org/git/?p=gcc.git;a=shortlog;h=refs/heads/vendors/redhat/gcc-9
https://gcc.gnu.org/git/?p=gcc.git;a=shortlog;h=refs/heads/vendors/redhat/gcc-9-branch/
and none of that worked.  They don't show up in the heads list
https://gcc.gnu.org/git/?p=gcc.git;a=heads

Jakub

Re: git conversion in progress

[Richard Earnshaw]

On 11/01/2020 14:48, Jakub Jelinek wrote:
> On Sat, Jan 11, 2020 at 02:39:43PM +, Joseph Myers wrote:
>>> On 11/01/2020 01:18, Joseph Myers wrote:
 The GCC SVN repository is now read-only for the move to git, as is the old 
 git-svn mirror; the cron job updating that mirror has been disabled, as 
 have gccadmin's cron jobs updating DATESTAMP, generating snapshots and 
 updating online documentation, until those are updated to work from git.

 Assuming validation passes, I expect to make the converted repository 
 available in place of the git-svn mirror (git://gcc.gnu.org/git/gcc.git 
 and git+ssh://gcc.gnu.org/git/gcc.git) some time on Saturday.  That will 
 be initially read-only to allow for people to review it in case any 
 significant conversion issues are noticed late that would justify 
 rerunning the conversion.  I propose to make it writable on Monday, at 
 which point we'll be in development stage 4 for GCC 10 (and to leave 
>>
>> This conversion is now in place, read-only for checking purposes.  I've 
>> done all the usual validation, including in particular checking branch 
>> tips and tags against SVN.
> 
> Thanks.
> Is there or will there be any way using gitweb to browse the vendor and user
> branches (and also even query which vendors/users and their branches there
> are)?
> I've tried
> https://gcc.gnu.org/git/?p=gcc.git;a=shortlog;h=refs/heads/vendors/
> https://gcc.gnu.org/git/?p=gcc.git;a=shortlog;h=refs/heads/vendors/redhat/
> https://gcc.gnu.org/git/?p=gcc.git;a=shortlog;h=refs/heads/vendors/redhat/gcc-9
> https://gcc.gnu.org/git/?p=gcc.git;a=shortlog;h=refs/heads/vendors/redhat/gcc-9-branch/
> and none of that worked.  They don't show up in the heads list
> https://gcc.gnu.org/git/?p=gcc.git;a=heads
> 
>   Jakub
> 

Try refs/vendors/redhat/heads

R.

Re: git conversion in progress

[Richard Earnshaw]

On 11/01/2020 14:49, Richard Earnshaw wrote:
> On 11/01/2020 14:48, Jakub Jelinek wrote:
>> On Sat, Jan 11, 2020 at 02:39:43PM +, Joseph Myers wrote:
 On 11/01/2020 01:18, Joseph Myers wrote:
> The GCC SVN repository is now read-only for the move to git, as is the 
> old 
> git-svn mirror; the cron job updating that mirror has been disabled, as 
> have gccadmin's cron jobs updating DATESTAMP, generating snapshots and 
> updating online documentation, until those are updated to work from git.
>
> Assuming validation passes, I expect to make the converted repository 
> available in place of the git-svn mirror (git://gcc.gnu.org/git/gcc.git 
> and git+ssh://gcc.gnu.org/git/gcc.git) some time on Saturday.  That will 
> be initially read-only to allow for people to review it in case any 
> significant conversion issues are noticed late that would justify 
> rerunning the conversion.  I propose to make it writable on Monday, at 
> which point we'll be in development stage 4 for GCC 10 (and to leave 
>>>
>>> This conversion is now in place, read-only for checking purposes.  I've 
>>> done all the usual validation, including in particular checking branch 
>>> tips and tags against SVN.
>>
>> Thanks.
>> Is there or will there be any way using gitweb to browse the vendor and user
>> branches (and also even query which vendors/users and their branches there
>> are)?
>> I've tried
>> https://gcc.gnu.org/git/?p=gcc.git;a=shortlog;h=refs/heads/vendors/
>> https://gcc.gnu.org/git/?p=gcc.git;a=shortlog;h=refs/heads/vendors/redhat/
>> https://gcc.gnu.org/git/?p=gcc.git;a=shortlog;h=refs/heads/vendors/redhat/gcc-9
>> https://gcc.gnu.org/git/?p=gcc.git;a=shortlog;h=refs/heads/vendors/redhat/gcc-9-branch/
>> and none of that worked.  They don't show up in the heads list
>> https://gcc.gnu.org/git/?p=gcc.git;a=heads
>>
>>  Jakub
>>
> 
> Try refs/vendors/redhat/heads
> 
> R.
> 

https://gcc.gnu.org/git/?p=gcc.git;a=shortlog;h=refs/vendors/redhat/heads/gcc-9-branch

Works

Or for tags s/heads/tags/

R.

Re: git conversion in progress

[Richard Earnshaw]

On 11/01/2020 14:51, Richard Earnshaw wrote:
> On 11/01/2020 14:49, Richard Earnshaw wrote:
>> On 11/01/2020 14:48, Jakub Jelinek wrote:
>>> On Sat, Jan 11, 2020 at 02:39:43PM +, Joseph Myers wrote:
> On 11/01/2020 01:18, Joseph Myers wrote:
>> The GCC SVN repository is now read-only for the move to git, as is the 
>> old 
>> git-svn mirror; the cron job updating that mirror has been disabled, as 
>> have gccadmin's cron jobs updating DATESTAMP, generating snapshots and 
>> updating online documentation, until those are updated to work from git.
>>
>> Assuming validation passes, I expect to make the converted repository 
>> available in place of the git-svn mirror (git://gcc.gnu.org/git/gcc.git 
>> and git+ssh://gcc.gnu.org/git/gcc.git) some time on Saturday.  That will 
>> be initially read-only to allow for people to review it in case any 
>> significant conversion issues are noticed late that would justify 
>> rerunning the conversion.  I propose to make it writable on Monday, at 
>> which point we'll be in development stage 4 for GCC 10 (and to leave 

 This conversion is now in place, read-only for checking purposes.  I've 
 done all the usual validation, including in particular checking branch 
 tips and tags against SVN.
>>>
>>> Thanks.
>>> Is there or will there be any way using gitweb to browse the vendor and user
>>> branches (and also even query which vendors/users and their branches there
>>> are)?

Once you have a checkout, "git ls-remote" will show all the refs on the
server.  I don't know if you can extract the list via the web UI, though.

>>> I've tried
>>> https://gcc.gnu.org/git/?p=gcc.git;a=shortlog;h=refs/heads/vendors/
>>> https://gcc.gnu.org/git/?p=gcc.git;a=shortlog;h=refs/heads/vendors/redhat/
>>> https://gcc.gnu.org/git/?p=gcc.git;a=shortlog;h=refs/heads/vendors/redhat/gcc-9
>>> https://gcc.gnu.org/git/?p=gcc.git;a=shortlog;h=refs/heads/vendors/redhat/gcc-9-branch/
>>> and none of that worked.  They don't show up in the heads list
>>> https://gcc.gnu.org/git/?p=gcc.git;a=heads
>>>
>>> Jakub
>>>
>>
>> Try refs/vendors/redhat/heads
>>
>> R.
>>
> 
> https://gcc.gnu.org/git/?p=gcc.git;a=shortlog;h=refs/vendors/redhat/heads/gcc-9-branch
> 
> Works
> 
> Or for tags s/heads/tags/
> 
> R.
> 

Re: git conversion in progress

[Jakub Jelinek]

On Sat, Jan 11, 2020 at 02:51:21PM +, Richard Earnshaw wrote:
> https://gcc.gnu.org/git/?p=gcc.git;a=shortlog;h=refs/vendors/redhat/heads/gcc-9-branch
> 
> Works
> 
> Or for tags s/heads/tags/

Indeed, this works, but if one doesn't know what branches there are for
particular vendor or what vendors there are (or what users), I can't find
how to query that.
Can one query it even in git, or just has to know the right spelling of a
particular vendor and then do git-fetch-vendor.sh that_vendor to see it?

Jakub

Re: git conversion in progress

[Richard Earnshaw]

On 11/01/2020 14:58, Jakub Jelinek wrote:
> On Sat, Jan 11, 2020 at 02:51:21PM +, Richard Earnshaw wrote:
>> https://gcc.gnu.org/git/?p=gcc.git;a=shortlog;h=refs/vendors/redhat/heads/gcc-9-branch
>>
>> Works
>>
>> Or for tags s/heads/tags/
> 
> Indeed, this works, but if one doesn't know what branches there are for
> particular vendor or what vendors there are (or what users), I can't find
> how to query that.
> Can one query it even in git, or just has to know the right spelling of a
> particular vendor and then do git-fetch-vendor.sh that_vendor to see it?
> 

git ls-remote | grep vendors

will list all the vendor branches and tags

Filter that as appropriate.

R.

>   Jakub
> 

Re: git conversion in progress

[Richard Earnshaw]

On 11/01/2020 15:01, Richard Earnshaw wrote:
> On 11/01/2020 14:58, Jakub Jelinek wrote:
>> On Sat, Jan 11, 2020 at 02:51:21PM +, Richard Earnshaw wrote:
>>> https://gcc.gnu.org/git/?p=gcc.git;a=shortlog;h=refs/vendors/redhat/heads/gcc-9-branch
>>>
>>> Works
>>>
>>> Or for tags s/heads/tags/
>>
>> Indeed, this works, but if one doesn't know what branches there are for
>> particular vendor or what vendors there are (or what users), I can't find
>> how to query that.
>> Can one query it even in git, or just has to know the right spelling of a
>> particular vendor and then do git-fetch-vendor.sh that_vendor to see it?
>>
> 
> git ls-remote | grep vendors
> 
> will list all the vendor branches and tags
> 
> Filter that as appropriate.
> 
> R.
> 
>>  Jakub
>>
> 


$ git ls-remote|grep vendors|sed -r "s|.*vendors/([^/]+).*|\1|"|sort|uniq
apple
ARM
csl
google
ibm
ix86
linaro
microblaze
redhat
st
suse
ubuntu

We could add this to git-fetch-vendor if invoked without an option.

R.

Re: git conversion in progress

[Joseph Myers]

On Sat, 11 Jan 2020, Richard Earnshaw wrote:

> On 11/01/2020 14:58, Jakub Jelinek wrote:
> > On Sat, Jan 11, 2020 at 02:51:21PM +, Richard Earnshaw wrote:
> >> https://gcc.gnu.org/git/?p=gcc.git;a=shortlog;h=refs/vendors/redhat/heads/gcc-9-branch
> >>
> >> Works
> >>
> >> Or for tags s/heads/tags/
> > 
> > Indeed, this works, but if one doesn't know what branches there are for
> > particular vendor or what vendors there are (or what users), I can't find
> > how to query that.
> > Can one query it even in git, or just has to know the right spelling of a
> > particular vendor and then do git-fetch-vendor.sh that_vendor to see it?
> > 
> 
> git ls-remote | grep vendors

We should probably put something about git ls-remote in git.html.  (I put 
"git branch -a" and "git tag -l" in there as the usual ways to list 
branches / tags once you have a checkout, but ls-remote seems relevant 
both for listing branches and tags without a checkout, and for listing 
those that aren't fetched by default.)

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: GIT: Monotonically increasing trunk and release branch ids

[Jakub Jelinek]

On Sat, Jan 11, 2020 at 12:28:01PM +0100, Jakub Jelinek wrote:
> For the redirectors, it could be something like following patch, except the
> last new redirect would need also a yet to be written cgi script that would
> git undescr the argument and print
> Location: https://gcc.gnu.org/git/gitweb.cgi?p=gcc.git;h=...
> 
> --- /etc/httpd/conf.d/sourceware-vhost-gcc.include2019-11-21 
> 19:25:03.172399228 +
> +++ /etc/httpd/conf.d/sourceware-vhost-gcc.include2020-01-11 
> 11:22:00.752261787 +
> @@ -59,6 +59,11 @@
># Support short URLs for referring to SVN revisions.
>RedirectMatch ^/r([0-9]+)$ 
> https://gcc.gnu.org/viewcvs/gcc?view=revision\&revision=$1
>  
> +  # Similarly for GIT commits.
> +  RedirectMatch ^/g:([0-9a-zA-Z._{}~^-]+)$   
> https://gcc.gnu.org/git/gitweb.cgi?p=gcc.git;h=$1
> +  RedirectMatch ^/r[0-9]+-[0-9]+-g([0-9a-f]+)$   
> https://gcc.gnu.org/git/gitweb.cgi?p=gcc.git;h=$1
> +  RedirectMatch ^/(r[0-9]+-[0-9]+)$  
> https://gcc.gnu.org/cgi-bin/gcc-gitref.cgi?r=$1
> +
>  # ViewCVS setup
>
>RewriteRule ^/cgi-bin/cvsweb.cgi   https://gcc.gnu.org/viewvc/gcc [L]
> 

And the cgi script could be something like (after the basepoints tags would
be pushed to upstream) attached, tested by hand against my copy of the
converted repo with:
$ git tag -l | grep basepoints
basepoints/gcc-0
basepoints/gcc-10
basepoints/gcc-5
basepoints/gcc-6
basepoints/gcc-7
basepoints/gcc-8
basepoints/gcc-9
$ for i in 0 `seq 5 10`; do echo -n basepoints/gcc-$i" "; git log 
basepoints/gcc-$i --max-count=1 | head -n 1; done
basepoints/gcc-0 commit 6f39d4ff6715973bbdf6510a69fccc46f9a746b9
basepoints/gcc-5 commit 89eb97de15f1618bd63b6138e8a5536b935861e8
basepoints/gcc-6 commit 1a46d358050cf6964df0d8ceaffafd0cc88539b2
basepoints/gcc-7 commit 70f4fdeb67fe065041a52304bb20f6ce701b1cf7
basepoints/gcc-8 commit aa90195aedfa348284c6e4e8ce59dc1b734f1d71
basepoints/gcc-9 commit 3bedd3463137d0712b72d582836a0b14c74ef620
basepoints/gcc-10 commit 68ec60c4a377b532ec2d265ea542107c36b1d15c

Jakub
#!/bin/sh
rel=`echo "$@" | sed -n 's/^r=r\([0-9]\+\)-[0-9]\+$/\1/p'`
cnt=`echo "$@" | sed -n 's/^r=r[0-9]\+-\([0-9]\+\)$/\1/p'`
repo=/sourceware/git/gcc.git

ret=
if [ -n "$rel" ]; then
  if [ -f /opt/rh/git19/enable ]; then
. /opt/rh/git19/enable
  fi

  sha=`git --git-dir=$repo rev-parse --verify --quiet releases/gcc-$rel`
  if [ -z "$sha" ]; then
sha=`git --git-dir=$repo rev-parse --verify --quiet master`
  fi
  if [ -n "$sha" ]; then
num=`git --git-dir=$repo describe --all --match basepoints/gcc-$rel $sha 
2>/dev/null \
 | sed -n 
's,^tags/basepoints/gcc-[0-9]\+-\([0-9]\+\)-g[0-9a-f]*$,\1,p;s,^tags/basepoints/gcc-[0-9]\+$,0,p'`
if [ -n "$num" ]; then
  num=`expr $num - $cnt`
  ret=`git --git-dir=$repo rev-parse --verify $sha~$num`
fi
  fi
fi
if expr match "$ret" "[0-9a-f]\{7,40\}" > /dev/null; then
  echo 'Content-type: text/html'
  echo
  echo ''
  echo 'https://gcc.gnu.org/git/gitweb.cgi?p=gcc.git;h='$ret'">'
  echo ''
else
  echo 'Status: 400 Bad Request'
  echo 'Content-type: text/html'
  echo
  echo ''
  echo '400 Bad Request'
  echo ''
  echo 'Error'
  echo 'Invalid argument or could not determine git revision.'
  echo ''
  echo ''
fi

Re: git conversion in progress

[Thomas Koenig]

Am 11.01.20 um 15:39 schrieb Joseph Myers:

This conversion is now in place, read-only for checking purposes.  I've
done all the usual validation, including in particular checking branch
tips and tags against SVN.


Is checkout via git+ssh supposed to work for this?  It does not for me
(something about public key rejected), but then I am a complete novice
at using git, so I am more or less doing vodoo git here.

Regards

Thomas

Re: git conversion in progress

[Andreas Schwab]

On Jan 11 2020, Richard Earnshaw wrote:

> Once you have a checkout, "git ls-remote" will show all the refs on the
> server.

You don't need a checkout, git ls-remote works on a remote URL.

Andreas.

-- 
Andreas Schwab, sch...@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: git conversion in progress

[Richard Earnshaw]

On 11/01/2020 16:33, Thomas Koenig wrote:
> Am 11.01.20 um 15:39 schrieb Joseph Myers:
>> This conversion is now in place, read-only for checking purposes.  I've
>> done all the usual validation, including in particular checking branch
>> tips and tags against SVN.
> 
> Is checkout via git+ssh supposed to work for this?  It does not for me
> (something about public key rejected), but then I am a complete novice
> at using git, so I am more or less doing vodoo git here.
> 
> Regards
> 
> Thomas

The same key as you use for checking out the svn should work.

It seems that the gcc server is heavily loaded at present - many folk
checking out the new repo.  So it might just be a load issue

Have patience...

R.

Re: git conversion in progress

[Andreas Schwab]

On Jan 11 2020, Richard Earnshaw wrote:

> $ git ls-remote|grep vendors|sed -r "s|.*vendors/([^/]+).*|\1|"|sort|uniq

git ls-remote URL "*/vendors/*" | ...

Andreas.

-- 
Andreas Schwab, sch...@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: GIT: Monotonically increasing trunk and release branch ids

[Andreas Schwab]

On Jan 11 2020, Jakub Jelinek wrote:

> g:releases/gcc-9@{yesterday} references (and, there we at least shouldn't

@{yesterday} operates on reflogs, a local concept that is useless
outside your own repository.

Andreas.

-- 
Andreas Schwab, sch...@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: git conversion in progress

[Martin Liška]

On 1/11/20 5:33 PM, Thomas Koenig wrote:

Am 11.01.20 um 15:39 schrieb Joseph Myers:

This conversion is now in place, read-only for checking purposes.  I've
done all the usual validation, including in particular checking branch
tips and tags against SVN.


Is checkout via git+ssh supposed to work for this?


It works for me:

$ git remote --verbose
origin  git+ssh://gcc.gnu.org/git/gcc.git (fetch)
origin  git+ssh://gcc.gnu.org/git/gcc.git (push)


  It does not for me
(something about public key rejected), but then I am a complete novice
at using git, so I am more or less doing vodoo git here.


Please paste the error message you face?
Martin



Regards

 Thomas

Re: git conversion in progress

[Joseph Myers]

On Sat, 11 Jan 2020, Thomas Koenig wrote:

> Am 11.01.20 um 15:39 schrieb Joseph Myers:
> > This conversion is now in place, read-only for checking purposes.  I've
> > done all the usual validation, including in particular checking branch
> > tips and tags against SVN.
> 
> Is checkout via git+ssh supposed to work for this?  It does not for me
> (something about public key rejected), but then I am a complete novice
> at using git, so I am more or less doing vodoo git here.

If your sourceware username is different from your local username, you 
either need to put username@ in the URL (as described in gitwrite.html), 
or put something like

Host gcc.gnu.org
User 

in your .ssh/config (I prefer the .ssh/config approach, then you never 
need to put the username in the URL).

One thing to note that we need to add to gitwrite.html is to set up git's 
notion of your name and email address,

git config --global user.name "Your Name"
git config --global user.email "Your Email Address"

if you don't already have that in ~/.gitconfig.  If the name or email 
address you want to use for contributing to GCC are different from those 
in ~/.gitconfig - if you want to use different details in different git 
repositories - then do such configuration without --global in your GCC 
checkout to set the name/email address to use when working in that clone.

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: git conversion in progress

[Thomas Koenig]

Hi Martin,

  It does not for me
(something about public key rejected), but then I am a complete novice
at using git, so I am more or less doing vodoo git here.


Please paste the error message you face?


Currently, gcc.gnu.org is totally under water, even accessing the
wiki leads to a timeout, so I can't read the wiki on what to do.

I will try again in a few hours.

Hm... I just hope this is a one-time effect, and isn't an indication
that git uses much more resources, server-side, so the current
infrastructure is not up to the task.  Is git that much more
resource hungry than svn? Or is this unrelated?

Regards

Thomas

Re: git conversion in progress

[Eric S. Raymond]

Thomas Koenig :
> Hm... I just hope this is a one-time effect, and isn't an indication
> that git uses much more resources, server-side, so the current
> infrastructure is not up to the task.  Is git that much more
> resource hungry than svn? Or is this unrelated?

Almost certanly unrelated. In normal use git is *spectacularly* faster than
SVN on equivalent operations. 
-- 
http://www.catb.org/~esr/";>Eric S. Raymond

Re: git conversion in progress

[Joseph Myers]

On Sat, 11 Jan 2020, Thomas Koenig wrote:

> Hm... I just hope this is a one-time effect, and isn't an indication
> that git uses much more resources, server-side, so the current
> infrastructure is not up to the task.  Is git that much more
> resource hungry than svn? Or is this unrelated?

I think fetching everything from scratch (which everyone is now doing) is 
much more resource hungry than updating an existing git clone.

The GitHub mirror has successfully updated: 
https://github.com/gcc-mirror/gcc (that has everything, including the 
not-fetched by default refs, plus GitHub-specific refs/pull/* refs for 
pull requests people created on GitHub in the past).

I've also pushed the converted repository to GitLab: 
https://gitlab.com/jsm28/gcc (that has everything as it came out of the 
conversion process, but not the hook configuration in refs/meta/config 
that I created after the conversion was done).

So you can clone either of those and then update the settings in 
.git/config to point to the repository on gcc.gnu.org instead for future 
fetches and commits.

The GitHub mirror should keep automatically updating in future, but I 
won't be updating the GitLab repository.

It's possible that doing a first clone with --mirror and then other clones 
using --reference to use the mirror clone for the objects will also reduce 
resource usage (it will certainly save on disk space locally if you have 
separate checkouts of different branches), but note the caveats about 
--reference in the git-clone manpage.

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: GIT: Monotonically increasing trunk and release branch ids

[Julien "FrnchFrgg" Rivaud]

Le 11/01/2020 à 14:49, Jakub Jelinek a écrit :

On Sat, Jan 11, 2020 at 07:43:17AM -0600, Segher Boessenkool wrote:

On Fri, Jan 10, 2020 at 07:50:02PM +0100, Jakub Jelinek wrote:

Maybe even add https://gcc.gnu.org/gNNN for 7-40 hexadecimal digits
redirect to https://gcc.gnu.org/git/gitweb.cgi?p=gcc.git;h=NNN
and then just put as URL those
https://gcc.gnu.org/g3b2b821c3f1bac2ac6dd2481f461ec33a13eac9b
URLs instead to make it somewhat shorter, and again teach bugzilla to
URLize g3b2b821c3f1bac2ac6dd2481f461ec33a13eac9b with 7-40 hexadecimal
digits.


Please never post links with a short hash.  Those *will* collide after a
while, making your link go stale.  Use at least ten chars, or twelve to
be save (and it is a nice round number).


Yeah, the 7 was just that it is what git describe prints by default,


IIRC git-describe will automatically increase the number of chars if 
there are collisions in the repository at a given number. But this of 
course doesn't help short hashes already published in an email, so I 
agree with Segher.



in the git-hooks tentative patch I've posted today it would use the
r10-1234 style ids in the subject (those will never collide) and in
the body the
r10-1234-g3b2b821c3f1bac2ac6dd2481f461ec33a13eac9b
style one.

Jakub

gcc-9-20200111 is now available

[gccadmin]

Snapshot gcc-9-20200111 is now available on
  https://gcc.gnu.org/pub/gcc/snapshots/9-20200111/
and on various mirrors, see http://gcc.gnu.org/mirrors.html for details.

This snapshot has been generated from the GCC 9 git branch
with the following options: git://gcc.gnu.org/git/gcc.git branch releases/gcc-9 
revision 4b9ea6845f2629c6a8a9c597789ae6c14bc7ed7d

You'll find:

 gcc-9-20200111.tar.xzComplete GCC

  SHA256=f3963e5f92b5f79cd8316dd8c6c334e3992bbda0ee545914bd7305c3137a29f5
  SHA1=41ec73a719ed9d7098b3d17c802df948eaee9059

Diffs from 9-20200104 are available in the diffs/ subdirectory.

When a particular snapshot is ready for public consumption the LATEST-9
link is updated and a message is sent to the gcc list.  Please do not use
a snapshot before it has been announced that way.

Re: git conversion in progress

[Joseph Myers]

On Sat, 11 Jan 2020, Joseph Myers wrote:

> the older test conversions 1 through 7).  Some cron jobs may be re-enabled 
> before then, subject to testing (I have git changes to gcc_release ready, 
> for example, for testing snapshot generation), but the DATESTAMP updates 
> won't be enabled until the repository is writable.

Snapshot generation is working (not yet enabled from cron).  I also have 
DATESTAMP updates working (tested with gcc-reposurgeon-8.git).  However, 
they seem to be much slower than expected.  The slow step is pushing a 
commit made locally to the gcc-reposurgeon-8.git repository; I see 
processes

gccadmin 121505  0.0  0.0 106080  1420 pts/2S+   22:35   0:00  \_ 
/bin/sh ./update_version_git
gccadmin 121834  0.0  0.0 214308 52808 pts/2Sl+  22:35   0:00  \_ 
git push origin master
gccadmin 121835  0.0  0.0 215396 54300 pts/2Sl+  22:35   0:00  
\_ git-receive-pack /home/gccadmin/gcc-reposurgeon-8.git
gccadmin 121839 99.3  0.8 1597152 613180 pts/2  R+   22:35   1:58  
\_ git pack-objects --all-progress-implied --revs --stdout --thin 
--delta-base-offset --progress

and there should be no need for a time-consuming pack-objects step, the 
DATESTAMP update commit should never add more than four small new objects 
(one commit, two trees, one blob) and such a push when done from a git+ssh 
checkout runs very quickly.  Any advice on why pushing a trival commit 
locally (but not remotely) should be so slow?

-- 
Joseph S. Myers
jos...@codesourcery.com