Re: Doubts regarding the _Dependent_ptr keyword
On Wed, Jul 3, 2019 at 6:33 PM Paul E. McKenney wrote:
>
> On Wed, Jul 03, 2019 at 05:47:56PM +0200, Richard Biener wrote:
> > On July 3, 2019 5:14:58 PM GMT+02:00, "Paul E. McKenney"
> > wrote:
> > >On Wed, Jul 03, 2019 at 12:39:41AM +0530, Akshat Garg wrote:
> > >> On Tue, Jul 2, 2019 at 8:40 PM Paul E. McKenney
> > >
> > >> wrote:
> > >>
> > >> > On Tue, Jul 02, 2019 at 02:15:55PM +0100, Ramana Radhakrishnan
> > >wrote:
> > >> > > On Tue, Jul 2, 2019 at 1:38 PM Paul E. McKenney
> > >
> > >> > wrote:
> > >> > >
> > >> > > >
> > >> > > > Once a user-created non-dependent pointer is assigned to, it is
> > >OK to
> > >> > > > break the dependency.
> > >> > >
> > >> > > Ok, that's good.
> > >> > > >
> > >> > > > Or am I missing the point here?
> > >> > >
> > >> > > I was just trying to make sure we were on the same page. I wonder
> > >if
> > >> > > marking this volatile would be sufficient for prototyping. I
> > >suspect
> > >> > > we would need another flag somewhere which someone with gimple
> > >> > > knowledge might be able to help us with.
> > >> >
> > >> > I expect that marking it as volatile would do the trick. ;-)
> > >> >
> > >> > Thanx, Paul
> > >> >
> > >> So, marking this pointer as volatile will not allow the compiler to
> > >> modify/optimize the statements, the pointer is appearing in. And we
> > >don't
> > >> need to push any other code inside any of the passes. Due to this, we
> > >want
> > >> to automatically say those dependent pointers are volatile and
> > >introduce a
> > >> new flag for this. Am I getting you guys correctly? Kindly, let me
> > >know?
> > >
> > >While I suspect that this might work, it would suppress way more
> > >optimizations than would be good. For but one example, consider:
> > >
> > > _Dependent_ptr int *p;
> > >
> > > p = atomic_load_explicit(gp, memory_order_consume);
> > > a = p->a;
> > > b = p->b;
> > >
> > >If "p" is volatile, then the compiler will be prevented from keeping
> > >it in a register, which would not make people coding fastpaths at
> > >all happy. ;-)
> > >
> > >Still, use of volatile might be a good technique for prototyping and
> > >analysis of _Dependent_ptr.
> >
> > With this example can you quickly summarize what kind of guarantees
> > _Dependent_ptr gives and how a compiler
> > Could possibly break those?
>
> First I suppose I should fix the bug in the above code. Or one of the
> bugs, at least. :-/
>
> struct foo {
> int a;
> int b;
> };
>
> _Dependent_ptr struct foo *p;
>
> p = atomic_load_explicit(gp, memory_order_consume);
> a = p->a;
> b = p->b;
>
> And then let me tweak the example a bit. For the first tweak:
>
> struct foo {
> int a;
> int b;
> };
>
> struct foo default_foo = { .a = 42, .b = 43 };
> int *gp = &default_foo;
>
> ...
>
> _Dependent_ptr int *p;
>
> p = atomic_load_explicit(gp, memory_order_consume);
> a = p->a;
> b = p->b;
>
> Suppose that the compiler used feedback-driven optimization, and noticed
> that the value of gp was almost always &default_foo. The compiler might
> decide to transform the last three lines as follows:
>
> p = atomic_load_explicit(gp, memory_order_consume);
> if (p == &default_foo) {
> a = default_foo.a;
> b = default_foo.b;
> } else {
> a = p->a;
> b = p->b;
> }
>
> Now, as long as the value of gp had remained &default_foo for the full
> duration of execution, no harm done. But suppose the following code
> was executing concurrently with the above transformed code:
>
> struct foo *q;
>
> q = malloc(sizeof(*q));
> assert(q);
> q->a = 1729;
> q->b = 1730;
> atomic_store_explicit(gp, q, memory_order_release);
> do_something();
> default_foo.a = 1;
> default_foo.b = 2;
> atomic_store_explicit(gp, &default_foo, memory_order_release);
>
> In this case, if the memory_order_consume() came just after the pointer
> was reset to &default_foo, it is possible that the transformed code
> would set "a" to 42 and "b" to 43, which might not be what the guy
> writing the code wanted to happen.
>
> One of the purposes of _Dependent_ptr is to prevent this transformation.
>
> This transformation can also happen if the developer's code contained a
> comparison to &default_foo -- an ARM or PowerPC compiler backend, upon
> seeing two pointers containing the same bits, would likely consider the
> two pointers as being interchangeable, and thus might do the dereferences
> using the copy that was not tagged with the hardware dependencies.
>
> There are quite a few other examples. The C++ standards committee
> working papers shown below go through a number of them, in case the
> above examp
LTO+profiled enabled builds
I'm running into some issues building LTO+profiled enabled configurations in some constrained build environment called buildds, having four cores and 16GB of RAM. configured for all frontends (maximum number of LTO links) and configured with --enable-bootstrap \ --with-build-config=bootstrap-lto-lean \ --enable-link-mutex and building the make profiledbootstrap-lean target. Most builds time out after 150 minutes. A typical LTO link runs for around one minute on this hardware, however a LTO link with -fprofile-use runs for up to three hours. So gcc/lock-and-run.sh runs the first lto-link, waits for all other 300 seconds, then removes the "stale" locks, and runs everything in parallel ... Which surprisingly goes well, because -flto=jobserver is in effect, so I don't see any memory constraints yet. The machine then starts building all front-ends, but apparently is not overloaded, as -flto=jobserver is in effect. However there is no output, and that triggers the timeout. Richi mentioned on IRC that the LTO links only have buffered output (unless you run in debug mode), and that is only emitted once the link finishes. However even with unbuffered output, there could be times when nothing is happening, no warnings? I'm currently experimenting with a modified lock-and-run.sh, which basically sets the delay for releasing the "stale" locks to 30min instead of 5 min, runs the LTO link in the background and checks for the status of the background job, emitting some "running ..." messages while not finished. Still adjusting some parameters, but at least that succeeds on some of my configurations. The locking mechanism was introduced in 2013, https://gcc.gnu.org/ml/gcc-patches/2013-05/msg1.html lock-and-run.sh should probably modified not to release the "stale" locks based on a fixed timeout value. How? While the "no-output" problem can be fixed in the lock script as well (attached), this doesn't apply to third party apps. Having unbuffered output and/or an option to print progress would be beneficial. Matthias lock-and-run.sh Description: application/shellscript
Implicit function declarations and GCC 10
Implicit function declarations were removed from C99, more than twenty years ago. So far, GCC only warns about them because there were too many old configure scripts where an error would lead to incorrect configure check failures. I can try to fix the remaining configure scripts in Fedora and submit the required changes during this summer and fall. I would appreciate if GCC 10 refused to declare functions implicitly by default. According to my observations, lack of an error diagnostic has turned into a major usability issue. For bugs related to pointer truncation, we could perhaps change the C front end to produce a hard error if an int value returned from an implicitly declared function is converted to a pointer. But the other case involves functions defined as returning _Bool, and the result is used in a boolean context. The x86-64 ABI only requires that the lowest 8 bits of the return value are defined, so an implicit int results in int values which incorrectly compare as inqueal to zero. Given that the pointer truncation issue is only slightly more common, than the _Bool issue, I don't think the diagnostic improvement for pointers would be very helpful, and we should just transition to errors. Implicit int we should remove as well. Checking configure scripts for both issues at the same time would not be much more work. Thanks, Florian
The 8th HelloLLVM / HelloGCC social in China: Hangzhou, July 20, 2019
Hi all, The 8th HelloLLVM / HelloGCC social in China will happen on July 20, 2019. The location is at Hangzhou. Everyone interested in LLVM/GCC Toolchain related projects is invited to join. Event details is at https://mp.weixin.qq.com/s/qK9V62atBpXyTKNP_J7A8Q Presentations are welcome :-) Current Topics: 1. TVM Stack Introduction. 2. New GCC Tutorial Plan from HelloGCC community. 3. The State of MLIR, Polyhedral, etc. 4. Lighting talks. Looking forward to meet you! -- Best wishes, Wei Wu (吴伟)
Re: Doubts regarding the _Dependent_ptr keyword
On Thu, Jul 04, 2019 at 01:00:18PM +0200, Richard Biener wrote:
> On Wed, Jul 3, 2019 at 6:33 PM Paul E. McKenney wrote:
> >
> > On Wed, Jul 03, 2019 at 05:47:56PM +0200, Richard Biener wrote:
> > > On July 3, 2019 5:14:58 PM GMT+02:00, "Paul E. McKenney"
> > > wrote:
> > > >On Wed, Jul 03, 2019 at 12:39:41AM +0530, Akshat Garg wrote:
> > > >> On Tue, Jul 2, 2019 at 8:40 PM Paul E. McKenney
> > > >
> > > >> wrote:
> > > >>
> > > >> > On Tue, Jul 02, 2019 at 02:15:55PM +0100, Ramana Radhakrishnan
> > > >wrote:
> > > >> > > On Tue, Jul 2, 2019 at 1:38 PM Paul E. McKenney
> > > >
> > > >> > wrote:
> > > >> > >
> > > >> > > >
> > > >> > > > Once a user-created non-dependent pointer is assigned to, it is
> > > >OK to
> > > >> > > > break the dependency.
> > > >> > >
> > > >> > > Ok, that's good.
> > > >> > > >
> > > >> > > > Or am I missing the point here?
> > > >> > >
> > > >> > > I was just trying to make sure we were on the same page. I wonder
> > > >if
> > > >> > > marking this volatile would be sufficient for prototyping. I
> > > >suspect
> > > >> > > we would need another flag somewhere which someone with gimple
> > > >> > > knowledge might be able to help us with.
> > > >> >
> > > >> > I expect that marking it as volatile would do the trick. ;-)
> > > >> >
> > > >> > Thanx, Paul
> > > >> >
> > > >> So, marking this pointer as volatile will not allow the compiler to
> > > >> modify/optimize the statements, the pointer is appearing in. And we
> > > >don't
> > > >> need to push any other code inside any of the passes. Due to this, we
> > > >want
> > > >> to automatically say those dependent pointers are volatile and
> > > >introduce a
> > > >> new flag for this. Am I getting you guys correctly? Kindly, let me
> > > >know?
> > > >
> > > >While I suspect that this might work, it would suppress way more
> > > >optimizations than would be good. For but one example, consider:
> > > >
> > > > _Dependent_ptr int *p;
> > > >
> > > > p = atomic_load_explicit(gp, memory_order_consume);
> > > > a = p->a;
> > > > b = p->b;
> > > >
> > > >If "p" is volatile, then the compiler will be prevented from keeping
> > > >it in a register, which would not make people coding fastpaths at
> > > >all happy. ;-)
> > > >
> > > >Still, use of volatile might be a good technique for prototyping and
> > > >analysis of _Dependent_ptr.
> > >
> > > With this example can you quickly summarize what kind of guarantees
> > > _Dependent_ptr gives and how a compiler
> > > Could possibly break those?
> >
> > First I suppose I should fix the bug in the above code. Or one of the
> > bugs, at least. :-/
> >
> > struct foo {
> > int a;
> > int b;
> > };
> >
> > _Dependent_ptr struct foo *p;
> >
> > p = atomic_load_explicit(gp, memory_order_consume);
> > a = p->a;
> > b = p->b;
> >
> > And then let me tweak the example a bit. For the first tweak:
> >
> > struct foo {
> > int a;
> > int b;
> > };
> >
> > struct foo default_foo = { .a = 42, .b = 43 };
> > int *gp = &default_foo;
> >
> > ...
> >
> > _Dependent_ptr int *p;
> >
> > p = atomic_load_explicit(gp, memory_order_consume);
> > a = p->a;
> > b = p->b;
> >
> > Suppose that the compiler used feedback-driven optimization, and noticed
> > that the value of gp was almost always &default_foo. The compiler might
> > decide to transform the last three lines as follows:
> >
> > p = atomic_load_explicit(gp, memory_order_consume);
> > if (p == &default_foo) {
> > a = default_foo.a;
> > b = default_foo.b;
> > } else {
> > a = p->a;
> > b = p->b;
> > }
> >
> > Now, as long as the value of gp had remained &default_foo for the full
> > duration of execution, no harm done. But suppose the following code
> > was executing concurrently with the above transformed code:
> >
> > struct foo *q;
> >
> > q = malloc(sizeof(*q));
> > assert(q);
> > q->a = 1729;
> > q->b = 1730;
> > atomic_store_explicit(gp, q, memory_order_release);
> > do_something();
> > default_foo.a = 1;
> > default_foo.b = 2;
> > atomic_store_explicit(gp, &default_foo, memory_order_release);
> >
> > In this case, if the memory_order_consume() came just after the pointer
> > was reset to &default_foo, it is possible that the transformed code
> > would set "a" to 42 and "b" to 43, which might not be what the guy
> > writing the code wanted to happen.
> >
> > One of the purposes of _Dependent_ptr is to prevent this transformation.
> >
> > This transformation can also happen if the developer's code contained a
> > comparison to &default_foo -- an ARM or PowerPC compiler backend, upon
> > seeing two pointers containing th
Re: Doubts regarding the _Dependent_ptr keyword
On Thu, Jul 04, 2019 at 10:40:15AM -0700, Paul E. McKenney wrote: > > I think fully guaranteeing this is hard (besides when you use > > volatile), we have the very same issue when dealing with > > pointer provenance rules, known for years and not fixed > > (and I don't see a good way to fix these issues without > > sacrifying performance everywhere). > > > > Good luck ;) > > Thank you, we will need it! ;-) Well, luck probably isn't all you need, you'll need some way to represent those dependencies in the IL (both GIMPLE and RTL) that would ensure that they aren't optimized away, because just hoping that optimization don't mess that up or just patching a few optimizations you discover first isn't going to be very reliable. Say don't rewrite those into SSA form and represent them close to how volatile ptrs are handled at least for the beginning, and if there are safe optimizations special case those, rather than allowing all optimizations on those and hope it will work out. Jakub
Re: Implicit function declarations and GCC 10
On Thu, Jul 04, 2019 at 01:27:27PM +0200, Florian Weimer wrote: > Implicit function declarations were removed from C99, more than twenty > years ago. So far, GCC only warns about them because there were too > many old configure scripts where an error would lead to incorrect > configure check failures. > > I can try to fix the remaining configure scripts in Fedora and submit > the required changes during this summer and fall. > > I would appreciate if GCC 10 refused to declare functions implicitly by > default. [ The warning for it is enabled by default with -std=c99 or later, and with -Wall as well. And people still ignore that? Wow. ] We already have an option for that (-Werror=implicit-function-declaration), and it is an error by default with -pedantic-errors already. If you are asking to make it an error by default, I second that; there needs to be a wat to turn it off though. Maybe it should be an error for c99 and later only, anyway? > According to my observations, lack of an error diagnostic has turned > into a major usability issue. For bugs related to pointer truncation, > we could perhaps change the C front end to produce a hard error if an > int value returned from an implicitly declared function is converted to > a pointer. No, if we allow implicit declarations at all, your proposal would error on valid (but improbable :-) ) code. We should only ever give hard errors if we *know* something is wrong. > Implicit int we should remove as well. Checking configure scripts for > both issues at the same time would not be much more work. We could enable -Wimplicit-int by default for c99 and later, maybe even its -Werror- version. This conflicts with at least -fms-extensions it, seems, dunno what to do there. Segher
gcc-7-20190704 is now available
Snapshot gcc-7-20190704 is now available on ftp://gcc.gnu.org/pub/gcc/snapshots/7-20190704/ and on various mirrors, see http://gcc.gnu.org/mirrors.html for details. This snapshot has been generated from the GCC 7 SVN branch with the following options: svn://gcc.gnu.org/svn/gcc/branches/gcc-7-branch revision 273096 You'll find: gcc-7-20190704.tar.xzComplete GCC SHA256=4e23db70d2de15c10622909de8ba97400937c00028a098ba248d22d0115271d2 SHA1=fa9a25ab311e82d3ae917704a7b005be7607ed46 Diffs from 7-20190627 are available in the diffs/ subdirectory. When a particular snapshot is ready for public consumption the LATEST-7 link is updated and a message is sent to the gcc list. Please do not use a snapshot before it has been announced that way.
Re: Doubts regarding the _Dependent_ptr keyword
On Thu, Jul 4, 2019 at 11:39 PM Jakub Jelinek wrote: > On Thu, Jul 04, 2019 at 10:40:15AM -0700, Paul E. McKenney wrote: > > > I think fully guaranteeing this is hard (besides when you use > > > volatile), we have the very same issue when dealing with > > > pointer provenance rules, known for years and not fixed > > > (and I don't see a good way to fix these issues without > > > sacrifying performance everywhere). > > > > > > Good luck ;) > > > > Thank you, we will need it! ;-) > > Well, luck probably isn't all you need, you'll need some way to represent > those dependencies in the IL (both GIMPLE and RTL) that would ensure that > they aren't optimized away, because just hoping that optimization don't > mess > that up or just patching a few optimizations you discover first isn't going > to be very reliable. Say don't rewrite those into SSA form and represent > them close to how volatile ptrs are handled at least for the beginning, and > if there are safe optimizations special case those, rather than allowing > all > optimizations on those and hope it will work out. > > Jakub > I don't understand this statement completely "Say don't rewrite those into SSA form and represent them close to how volatile ptrs are handled at least for the beginning, and if there are safe optimizations special case those, rather than allowing all optimizations on those and hope it will work out.". Are you saying that we should try to represent dependent ptrs as volatile ptrs for some places? We should figure out all the places where a check for volatile ptrs happens and we check for dependent ptrs there also and see if we can allow the optimization or not. Am I getting you correctly, please tell? -Akshat
Re: Implicit function declarations and GCC 10
* Segher Boessenkool: > On Thu, Jul 04, 2019 at 01:27:27PM +0200, Florian Weimer wrote: >> Implicit function declarations were removed from C99, more than twenty >> years ago. So far, GCC only warns about them because there were too >> many old configure scripts where an error would lead to incorrect >> configure check failures. >> >> I can try to fix the remaining configure scripts in Fedora and submit >> the required changes during this summer and fall. >> >> I would appreciate if GCC 10 refused to declare functions implicitly by >> default. > > [ The warning for it is enabled by default with -std=c99 or later, and > with -Wall as well. And people still ignore that? Wow. ] Yes, we just got a glibc bug report about it. Even seasoned programmers miss it when it's buried in build logs. > We already have an option for that (-Werror=implicit-function-declaration), > and it is an error by default with -pedantic-errors already. If you are > asking to make it an error by default, I second that; there needs to be a > wat to turn it off though. Maybe it should be an error for c99 and later > only, anyway? Yes, it should be an error by default, without any flags. Which is gnu11 mode by now, I think. So it's not sufficient to do this for c99/c11 mode. >> According to my observations, lack of an error diagnostic has turned >> into a major usability issue. For bugs related to pointer truncation, >> we could perhaps change the C front end to produce a hard error if an >> int value returned from an implicitly declared function is converted to >> a pointer. > > No, if we allow implicit declarations at all, your proposal would error > on valid (but improbable :-) ) code. We should only ever give hard > errors if we *know* something is wrong. Yes, it should be valid code on 32-bit targets, which is why it's so common. >> Implicit int we should remove as well. Checking configure scripts for >> both issues at the same time would not be much more work. > > We could enable -Wimplicit-int by default for c99 and later, maybe even > its -Werror- version. This conflicts with at least -fms-extensions it, > seems, dunno what to do there. We can keep this a separate discussion if it helps. Thanks, Florian
Re: Implicit function declarations and GCC 10
Hi Florian, On Fri, Jul 05, 2019 at 07:49:21AM +0200, Florian Weimer wrote: > > We already have an option for that (-Werror=implicit-function-declaration), > > and it is an error by default with -pedantic-errors already. If you are > > asking to make it an error by default, I second that; there needs to be a > > wat to turn it off though. Maybe it should be an error for c99 and later > > only, anyway? > > Yes, it should be an error by default, without any flags. Which is > gnu11 mode by now, I think. So it's not sufficient to do this for > c99/c11 mode. When I say "c99 and later", of course that includes gnu99 and gnu11. My point is that we probably shouldn't by default error on this in c90 mode, since it is a valid construct there, and not extraordinarily harmful. > >> According to my observations, lack of an error diagnostic has turned > >> into a major usability issue. For bugs related to pointer truncation, > >> we could perhaps change the C front end to produce a hard error if an > >> int value returned from an implicitly declared function is converted to > >> a pointer. > > > > No, if we allow implicit declarations at all, your proposal would error > > on valid (but improbable :-) ) code. We should only ever give hard > > errors if we *know* something is wrong. > > Yes, it should be valid code on 32-bit targets, which is why it's so > common. It is valid for *all* targets. C11 6.3.2.3/5: An integer may be converted to any pointer type. Except as previously specified, the result is implementation-defined, might not be correctly aligned, might not point to an entity of the referenced type, and might be a trap representation. and GCC documents that implementation-defined behaviour as A cast from integer to pointer discards most-significant bits if the pointer representation is smaller than the integer type, extends according to the signedness of the integer type if the pointer representation is larger than the integer type, otherwise the bits are unchanged. > >> Implicit int we should remove as well. Checking configure scripts for > >> both issues at the same time would not be much more work. > > > > We could enable -Wimplicit-int by default for c99 and later, maybe even > > its -Werror- version. This conflicts with at least -fms-extensions it, > > seems, dunno what to do there. > > We can keep this a separate discussion if it helps. Yes please. Can you make separate PRs? There is essentially no overlap. Segher
