Re: Multi-Threading GCC Continuation
Hi Nick, On Sun, Dec 08, 2019 at 03:03:56PM -0500, Nicholas Krause wrote: > The first questions are: > 1. What current heuristics do we have as it seems none for figuring out > what state is shared > as it seems none? If I correct the first thing to do is discuss what > bits/bitmasks we want > for figuring out shared state or other ways. Shared between what and what? > 2. MD files seem to be a major source of shared state or reading them. > Is it possible > to read from them async? Doesn't seem to be a problem but the current > docs don't > mention it nor does it seem easy to do. MD files are not read *at all* by the compiler itself; they aren't installed, even. They are read by the gen* programs when the compiler itself is built, to create the insn-*.c files and the like. > 3. There are two ways to write this for RTL either one class for all the > state or a core > class will each major part being a subclass like delayed branch > scheduling e.t.c.Not sure > which is better so thought I would ask. RTL as it is is pretty efficient. Please keep it that way. It also is a dumb (and very "open") data structure, by design. See how "XEXP" and similar work. That could be changed of course, for non-trivial cost, but what for? Segher
Re: How to modify Bugzilla tracker
Hi! I'm not an admin there; CCing the Overseers. On 2019-12-12T01:58:46+, Feng Xue OS wrote: > I have a sourceware.org account, but can not correlate the account with > Bugzilla login. Yes: 'gcc.gnu.org' is a separate Bugzilla instance. > Just want to assign a tracker to someone. To be able to do that, as far as I know, you'll need to use (set up a new?) account associated with your 'f...@gcc.gnu.org' email address. Addresses '@gcc.gnu.org' automatically get special permissions. I see you do have an account 'f...@os.amperecomputing.com'. Maybe it's possible to add another email address to that one (or replace it?), or if not, I remember that I once (manually?) got two accounts merged into 'tschwi...@gcc.gnu.org', so that's another option for you, assuming you'd like just one account. Grüße Thomas > From: Jonathan Wakely > Sent: Wednesday, December 11, 2019 7:07 PM > To: Feng Xue OS > Cc: gcc@gcc.gnu.org > Subject: Re: How to modify Bugzilla tracker > > On Wed, 11 Dec 2019 at 02:25, Feng Xue OS wrote: >> >> Hi, >> >> I want to alter some information of an existing Bugzilla tracker, such as >> assignee, but found there is no entrance in page of the Bugzilla tracker to >> do this. How can I get it? > > Usually that's restricted to GCC developers, using their u...@gcc.gnu.org > login. > > What do you want to change? signature.asc Description: PGP signature
Re: How to modify Bugzilla tracker
On Thu, 12 Dec 2019 at 09:19, Thomas Schwinge wrote: > > Hi! > > I'm not an admin there; CCing the Overseers. > > On 2019-12-12T01:58:46+, Feng Xue OS wrote: > > I have a sourceware.org account, but can not correlate the account with > > Bugzilla login. > > Yes: 'gcc.gnu.org' is a separate Bugzilla instance. > > > Just want to assign a tracker to someone. > > To be able to do that, as far as I know, you'll need to use (set up a > new?) account associated with your 'f...@gcc.gnu.org' email address. > Addresses '@gcc.gnu.org' automatically get special permissions. > > I see you do have an account 'f...@os.amperecomputing.com'. Maybe it's > possible to add another email address to that one (or replace it?), or if > not, I remember that I once (manually?) got two accounts merged into > 'tschwi...@gcc.gnu.org', so that's another option for you, assuming you'd > like just one account. @sourceware.org and @gcc.gnu.org accounts are also separate, and I don't see your name in the GCC MAINTAINERS file (apologies if I missed it).
Re: How to modify Bugzilla tracker
On Dez 12 2019, Jonathan Wakely wrote: > On Thu, 12 Dec 2019 at 09:19, Thomas Schwinge wrote: >> >> Hi! >> >> I'm not an admin there; CCing the Overseers. >> >> On 2019-12-12T01:58:46+, Feng Xue OS wrote: >> > I have a sourceware.org account, but can not correlate the account with >> > Bugzilla login. >> >> Yes: 'gcc.gnu.org' is a separate Bugzilla instance. >> >> > Just want to assign a tracker to someone. >> >> To be able to do that, as far as I know, you'll need to use (set up a >> new?) account associated with your 'f...@gcc.gnu.org' email address. >> Addresses '@gcc.gnu.org' automatically get special permissions. >> >> I see you do have an account 'f...@os.amperecomputing.com'. Maybe it's >> possible to add another email address to that one (or replace it?), or if >> not, I remember that I once (manually?) got two accounts merged into >> 'tschwi...@gcc.gnu.org', so that's another option for you, assuming you'd >> like just one account. > > @sourceware.org and @gcc.gnu.org accounts are also separate, and I > don't see your name in the GCC MAINTAINERS file (apologies if I missed > it). The f...@gcc.gnu.org account already exists, he just needs to use it. Andreas. -- Andreas Schwab, SUSE Labs, sch...@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different."
Re: How to modify Bugzilla tracker
On Thu, 12 Dec 2019 at 09:29, Andreas Schwab wrote: > > On Dez 12 2019, Jonathan Wakely wrote: > > > On Thu, 12 Dec 2019 at 09:19, Thomas Schwinge wrote: > >> > >> Hi! > >> > >> I'm not an admin there; CCing the Overseers. > >> > >> On 2019-12-12T01:58:46+, Feng Xue OS > >> wrote: > >> > I have a sourceware.org account, but can not correlate the account with > >> > Bugzilla login. > >> > >> Yes: 'gcc.gnu.org' is a separate Bugzilla instance. > >> > >> > Just want to assign a tracker to someone. > >> > >> To be able to do that, as far as I know, you'll need to use (set up a > >> new?) account associated with your 'f...@gcc.gnu.org' email address. > >> Addresses '@gcc.gnu.org' automatically get special permissions. > >> > >> I see you do have an account 'f...@os.amperecomputing.com'. Maybe it's > >> possible to add another email address to that one (or replace it?), or if > >> not, I remember that I once (manually?) got two accounts merged into > >> 'tschwi...@gcc.gnu.org', so that's another option for you, assuming you'd > >> like just one account. > > > > @sourceware.org and @gcc.gnu.org accounts are also separate, and I > > don't see your name in the GCC MAINTAINERS file (apologies if I missed > > it). > > The f...@gcc.gnu.org account already exists, he just needs to use it. Ah OK, so just use the "reset password" link to get access to that account then.
Re: Multi-Threading GCC Continuation
On 12/12/19 4:11 AM, Segher Boessenkool wrote: Hi Nick, On Sun, Dec 08, 2019 at 03:03:56PM -0500, Nicholas Krause wrote: The first questions are: 1. What current heuristics do we have as it seems none for figuring out what state is shared as it seems none? If I correct the first thing to do is discuss what bits/bitmasks we want for figuring out shared state or other ways. Shared between what and what? Between the passes in gcc. If we can launch certain passes in gcc on another thread and join up to other passes that depend on the state used by a earlier pass. For example if a loop pass does not touch outside a function in GIMPLE or RTL we should launch it on another thread. Then join up to the next pass that requires the state. Seems not all passes touch everything or only parts of either GIMPLE or RTL so this may be worth considering. The question was what internal compiler data can be currently use for finding when this should be the case . I don't see anything so figuring out how to detect this is going to be part of the challenge. I'm going to write up a wiki article on the GCC wiki explaining it better but that's a very brief idea alongside some other ideas like figuring out if dominator trees should or can be lockless or very close to in nature for insertion/deletion. 2. MD files seem to be a major source of shared state or reading them. Is it possible to read from them async? Doesn't seem to be a problem but the current docs don't mention it nor does it seem easy to do. MD files are not read *at all* by the compiler itself; they aren't installed, even. They are read by the gen* programs when the compiler itself is built, to create the insn-*.c files and the like. 3. There are two ways to write this for RTL either one class for all the state or a core class will each major part being a subclass like delayed branch scheduling e.t.c.Not sure which is better so thought I would ask. RTL as it is is pretty efficient. Please keep it that way. It also is a dumb (and very "open") data structure, by design. See how "XEXP" and similar work. That could be changed of course, for non-trivial cost, but what for? I'm not talking about changing RTL itself it terms of its optimizations but rewriting it for reading work queues in parallel on non shared state between the current running pass and joining it back up to the next pass requiring it. For example why not run parts of the register allocation on separate work queues if possible? I was asking Peter at Cauldron about the register part and he seems to like doing something like this for cost of allocating registers if I recall correctly. Hopefully that explains it a little better, Nick Segher
Re: GCC static analysis branch now available on Compiler Explorer
On 12/10/19 1:42 PM, Jeff Law wrote: On Tue, 2019-12-10 at 12:51 -0700, Martin Sebor wrote: On 12/10/19 10:56 AM, David Malcolm wrote: On Tue, 2019-12-10 at 09:36 -0700, Martin Sebor wrote: On 12/10/19 8:46 AM, David Malcolm wrote: For the adventurous/curious, my static analyzer branch of GCC [1] is now available on Compiler Explorer (aka godbolt.org) so you can try it out without building it yourself. [Thanks to Matt Godbolt, Patrick Quist and others at the Compiler Explorer project] On https://godbolt.org/ within the C and C++ languages, select "x86-64 gcc (static analysis)" as the compiler (though strictly speaking only C is in-scope right now). It's configured to automatically inject -fanalyzer (just on this site; it isn't the default in the branch). Some precanned examples: * various malloc issues: https://godbolt.org/z/tnx65n * stdio issues: https://godbolt.org/z/4BP-Tj * fprintf in signal handler: https://godbolt.org/z/ew7mW6 * tainted data affecting control flow: https://godbolt.org/z/3v8vSj * password-leakage: https://godbolt.org/z/pRPYiv (the non-malloc examples are much more in "proof-of-concept" territory) Would it make sense to add an "analyzer" component to our bugzilla, even though this is still on a branch? (with me as default assignee) Yes, that would make sense to me for bugs/enhancements specific to just the analyzer. I think it's important to have a shared understanding how requests for new warnings (either completely new features or enhancements to existing ones) should be triaged in Bugzilla. For instance, pr82520 is a request to enhance -Wreturn-local-addr to detect other forms of escaping dangling pointers besides by returning them from functions. I think the request (and a number of others) would be a good fit for the analyzer as well, but assigning it to the analyzer component would suggest that it's suitable only for it. How do we track requests that we'd like handled in both? It feels to me that the question of BZ component is a proxy for a deeper scope question about what we want to handle in each warning. If it's clear that we want an issue handled by both the core code *and* by the analyzer, then we should have two PRs. Perhaps discuss in the PRs which things we expect to be handled by which warning (e.g. some cases might be capable of being caught by -Wreturn-local-addr, others might require the deeper analysis of -fanalyzer). You're quite right that the two questions are related, and it is the latter that prompted my comment above. My concern is that I'm not sure we have a good rule of thumb to decide whether a given warning is better suited to just the analyzer or just the middle-end, or that it should be implemented in both. Or even a good basis for making such decisions. We have not discussed yet if we should even worry about overlap (duplicate warnings issued by both the analyzer and the middle-end). I asked a few questions to help get more clarity here in my comments on the initial patch but only heard from Jeff. So here are some of my thoughts. I'd be very interested in yours (or anyone else's who cares about this subject). I can think of no warning that's currently implemented in the middle- end (or that could be implemented there) that would not also be appropriate for the analyzer. IIUC, it has the potential for finding more problems but may be more noisy than a middle-end implementation, and take longer to analyze code. Are there any kinds of problems that you think are not appropriate for it? Any inherent limitations? At the same time I also think there's room for and value in providing at least some support in the middle-end for pretty much every warning that the analyzer already exposes options for. They may not find as many problems as the analyzer eventually will but they don't have as much overhead (either to implement, or in terms of compile time -- the analysis often benefits optimizations as well). In addition, by being tied to the optimizers the detection provides an opportunity to also prevent the bugs (by folding invalid code away, or inserting traps or branches around it, perhaps under the control of users as we have discussed). In my mind the only reason to have to choose between one and other is the duplication of effort and resource constraints: we don't have enough contributors into either area. There isn't a lot we can about the latter but we should make an effort to minimize the duplication (starting with bug triage). But maybe I'm overanalyzing it and playing it by ear will work just fine. I think one of the key components will be whether or not one needs intra-procedural state tracking to do a reasonable job. THe double- free checking is a great example. You can build one with what's in the tree today, but the result isn't really useful except for toy examples. To do a reasonable job you've got to explode the graphs and track state intra-procedurally. Thus it's a good
Re: Multi-Threading GCC Continuation
On Thu, Dec 12, 2019 at 10:21:03AM -0500, Nicholas Krause wrote: > On 12/12/19 4:11 AM, Segher Boessenkool wrote: > >On Sun, Dec 08, 2019 at 03:03:56PM -0500, Nicholas Krause wrote: > >>The first questions are: > >>1. What current heuristics do we have as it seems none for figuring out > >>what state is shared > >>as it seems none? If I correct the first thing to do is discuss what > >>bits/bitmasks we want > >>for figuring out shared state or other ways. > >Shared between what and what? > Between the passes in gcc. If we can launch certain passes in gcc on > another > thread and join up to other passes that depend on the state used by a > earlier > pass. For example if a loop pass does not touch outside a function in GIMPLE > or RTL we should launch it on another thread. Then join up to the next pass > that requires the state. Seems not all passes touch everything or only parts > of either GIMPLE or RTL so this may be worth considering. The question > was what internal compiler data can be currently use for finding when this > should be the case . I don't see anything so figuring out how to detect > this > is going to be part of the challenge. Every pass depends on the previous pass. You cannot do later passes before earlier ones (for the same function). Each separate pass can do its own thing in whatever way it wants of course. Nothing else will access the insn stream while a pass is running. > >>3. There are two ways to write this for RTL either one class for all the > >>state or a core > >>class will each major part being a subclass like delayed branch > >>scheduling e.t.c.Not sure > >>which is better so thought I would ask. > >RTL as it is is pretty efficient. Please keep it that way. It also is > >a dumb (and very "open") data structure, by design. See how "XEXP" and > >similar work. > > > >That could be changed of course, for non-trivial cost, but what for? > I'm not talking about changing RTL itself it terms of its optimizations > but rewriting it for reading work queues in parallel on non shared > state between the current running pass and joining it back up > to the next pass requiring it. > > For example why not run parts of the register allocation on separate > work queues if possible? I was asking Peter at Cauldron about the > register part and he seems to like doing something like this for > cost of allocating registers if I recall correctly. > > Hopefully that explains it a little better, No, I still do not understand what you mean at all :-/ Segher
Re: Multi-Threading GCC Continuation
On 12/12/19 11:38 AM, Segher Boessenkool wrote: On Thu, Dec 12, 2019 at 10:21:03AM -0500, Nicholas Krause wrote: On 12/12/19 4:11 AM, Segher Boessenkool wrote: On Sun, Dec 08, 2019 at 03:03:56PM -0500, Nicholas Krause wrote: The first questions are: 1. What current heuristics do we have as it seems none for figuring out what state is shared as it seems none? If I correct the first thing to do is discuss what bits/bitmasks we want for figuring out shared state or other ways. Shared between what and what? Between the passes in gcc. If we can launch certain passes in gcc on another thread and join up to other passes that depend on the state used by a earlier pass. For example if a loop pass does not touch outside a function in GIMPLE or RTL we should launch it on another thread. Then join up to the next pass that requires the state. Seems not all passes touch everything or only parts of either GIMPLE or RTL so this may be worth considering. The question was what internal compiler data can be currently use for finding when this should be the case . I don't see anything so figuring out how to detect this is going to be part of the challenge. Every pass depends on the previous pass. You cannot do later passes before earlier ones (for the same function). Each separate pass can do its own thing in whatever way it wants of course. Nothing else will access the insn stream while a pass is running. My idea was to use something similar to: https://linux-kernel-labs.github.io/master/labs/deferred_work.html It's under the section workqueues. It would allow us to defer non shared work between the passes. For example if a loop pass a has state that c depends on but not b, why not launch it on a thread and join back up to c. Here's it sorta draw out: a-> has state that c depends on so launch a workqueue -> b no shared state with a so continue this in parallel -> c waits for a to join back up with shared or dependency state Of course if its the same function this would not work but the question is and it seems to be a lot of passes touch only certain parts of GIMPLE or RTL. Therefore doing in a workqueue would be ideal if the state is not shared. The real question is how to detect when to launch a workqueue based on how shared state is passed to dependent passes on the earlier passes transformations of GIMPLE or RTL. 3. There are two ways to write this for RTL either one class for all the state or a core class will each major part being a subclass like delayed branch scheduling e.t.c.Not sure which is better so thought I would ask. RTL as it is is pretty efficient. Please keep it that way. It also is a dumb (and very "open") data structure, by design. See how "XEXP" and similar work. That could be changed of course, for non-trivial cost, but what for? I'm not talking about changing RTL itself it terms of its optimizations but rewriting it for reading work queues in parallel on non shared state between the current running pass and joining it back up to the next pass requiring it. For example why not run parts of the register allocation on separate work queues if possible? I was asking Peter at Cauldron about the register part and he seems to like doing something like this for cost of allocating registers if I recall correctly. Hopefully that explains it a little better, No, I still do not understand what you mean at all :-/ I'm going to invest a little further into how to do this on the register side but hopefully my diagram above makes sense. Nick Sorry for the confusion, Nick Segher
Re: How to modify Bugzilla tracker
> On Thu, 12 Dec 2019 at 09:29, Andreas Schwab wrote: >> >> On Dez 12 2019, Jonathan Wakely wrote: >> >> > On Thu, 12 Dec 2019 at 09:19, Thomas Schwinge wrote: >> >> >> >> Hi! >> >> >> >> I'm not an admin there; CCing the Overseers. >> >> >> >> On 2019-12-12T01:58:46+, Feng Xue OS >> >> wrote: >> >> > I have a sourceware.org account, but can not correlate the account with >> >> > Bugzilla login. >> >> >> >> Yes: 'gcc.gnu.org' is a separate Bugzilla instance. >> >> >> >> > Just want to assign a tracker to someone. >> >> >> >> To be able to do that, as far as I know, you'll need to use (set up a >> >> new?) account associated with your 'f...@gcc.gnu.org' email address. >> >> Addresses '@gcc.gnu.org' automatically get special permissions. >> >> >> >> I see you do have an account 'f...@os.amperecomputing.com'. Maybe it's >> >> possible to add another email address to that one (or replace it?), or if >> >> not, I remember that I once (manually?) got two accounts merged into >> >> 'tschwi...@gcc.gnu.org', so that's another option for you, assuming you'd >> >> like just one account. >> > >> > @sourceware.org and @gcc.gnu.org accounts are also separate, and I >> > don't see your name in the GCC MAINTAINERS file (apologies if I missed >> > it). >> >> The f...@gcc.gnu.org account already exists, he just needs to use it. > Ah OK, so just use the "reset password" link to get access to that account > then. But I did not know I own f...@gcc.gnu.org, and do not know how to access the email, to which "reset password" instructions will be sent. Actually, I tried to ask for oversees to create a gcc.gnu.org account, but was told it is not needed for I already have one @sourceware.org. BTW: I'm not a maintainer. Thanks, Feng
Re: How to modify Bugzilla tracker
It works. Thanks, Feng From: gcc-ow...@gcc.gnu.org on behalf of Feng Xue OS Sent: Friday, December 13, 2019 10:13 AM To: Jonathan Wakely; Andreas Schwab Cc: Thomas Schwinge; gcc@gcc.gnu.org; overse...@sourceware.org Subject: Re: How to modify Bugzilla tracker > On Thu, 12 Dec 2019 at 09:29, Andreas Schwab wrote: >> >> On Dez 12 2019, Jonathan Wakely wrote: >> >> > On Thu, 12 Dec 2019 at 09:19, Thomas Schwinge wrote: >> >> >> >> Hi! >> >> >> >> I'm not an admin there; CCing the Overseers. >> >> >> >> On 2019-12-12T01:58:46+, Feng Xue OS >> >> wrote: >> >> > I have a sourceware.org account, but can not correlate the account with >> >> > Bugzilla login. >> >> >> >> Yes: 'gcc.gnu.org' is a separate Bugzilla instance. >> >> >> >> > Just want to assign a tracker to someone. >> >> >> >> To be able to do that, as far as I know, you'll need to use (set up a >> >> new?) account associated with your 'f...@gcc.gnu.org' email address. >> >> Addresses '@gcc.gnu.org' automatically get special permissions. >> >> >> >> I see you do have an account 'f...@os.amperecomputing.com'. Maybe it's >> >> possible to add another email address to that one (or replace it?), or if >> >> not, I remember that I once (manually?) got two accounts merged into >> >> 'tschwi...@gcc.gnu.org', so that's another option for you, assuming you'd >> >> like just one account. >> > >> > @sourceware.org and @gcc.gnu.org accounts are also separate, and I >> > don't see your name in the GCC MAINTAINERS file (apologies if I missed >> > it). >> >> The f...@gcc.gnu.org account already exists, he just needs to use it. > Ah OK, so just use the "reset password" link to get access to that account > then. But I did not know I own f...@gcc.gnu.org, and do not know how to access the email, to which "reset password" instructions will be sent. Actually, I tried to ask for oversees to create a gcc.gnu.org account, but was told it is not needed for I already have one @sourceware.org. BTW: I'm not a maintainer. Thanks, Feng
