Re: Hundreds of gcc.dg/guality failures on both 14 and 13.1 branches
Hi, > Which is why people should just compare testsuite results from earlier run > on the same configuration to watch for regressions, especially in the > guality testsuite. All this gives the idea of a test framework that is too rigid, or tests that are too fragile. I mean, The accumulation of noise just decreases the value of the the test results. FX
Re: Hundreds of gcc.dg/guality failures on both 14 and 13.1 branches
Martin Jambor writes: > Hi, > > On Sat, Jul 15 2023, FX Coudert via Gcc wrote: >> Hi, >> >> I am finding it very hard to reliably compare test results and regressions >> with the very large number of gcc.dg/guality test failures that are >> apparently the new norm on x86_64-linux: more than one hundred on 13.1, and >> several hundreds on 14. Is there any on-going discussion about this? >> >> I mean, from an almost-external point of view, these tests should probably >> be xfail'ed and a PR opened against them to reenable them. >> > > As far as I understand it, the main problem is that it is not really > possible to XFAIL a test for one combination of options (say "-O2 > -flto") and not others. FWIW, we do do that for aarch64*. The principle is that if anyone wants to work on improving debug quality, the XFAILs in the testsuite are just as good a reference as bugzilla (unless the bugzilla PR has specific analysis). But it is still dependent on the default configuration and on the version of GDB used. It gives zero failures on my set-up, and seems to do the same for some other gcc-testresults@ emails I've seen. But it probably won't for everyone. I tend to update the XFAIL lists in stage 3/4, when no major changes are expected. Having the XFAILs doesn't really change the workflow. It's still necessary to do a before-after comparison. But having fewer XPASSes & FAILs makes it less likely that something gets missed. It might even encourage people to take guality regressions more seriously :) Before, there were so many XPASSes and FAILs in the "before" log that it was easy think "well, what's one more?" or "this thing looks generally broken, so I'll just ignore it". Richard
Re: Hundreds of gcc.dg/guality failures on both 14 and 13.1 branches
Hi, On Sun, Jul 16, 2023 at 09:50:24AM +0200, FX Coudert via Gcc wrote: > > Which is why people should just compare testsuite results from earlier run > > on the same configuration to watch for regressions, especially in the > > guality testsuite. > > All this gives the idea of a test framework that is too rigid, or > tests that are too fragile. I mean, The accumulation of noise just > decreases the value of the the test results. Note that one way to analyze whether tests are too fragile is to look at the collected bunsen results: https://builder.sourceware.org/testruns/?has_keyvalue_k=testrun.git_describe&has_keyvalue_op=glob&has_keyvalue_v=*gcc* As far as I can see you are right that there are ~100 FAILs (out of ~4700) in guality.exp on all arches/configs. (CC Frank, the fedrawhide builder doesn't seem to include guality.exp, do you know why?) Cheers, Mark
Re: Hundreds of gcc.dg/guality failures on both 14 and 13.1 branches
Hi - Hi - > (CC Frank, the fedrawhide builder doesn't seem to include guality.exp, > do you know why?) Good question. The gcc.log testsuite file appears to be truncated as it arrived into bunsen. The test does appear to be running on the bot. Maybe the gcc's own dg* log processing scripts are messing something up, before bunsen ever gets a chance to grab them. :( - FChE
gcc-14-20230716 is now available
Snapshot gcc-14-20230716 is now available on https://gcc.gnu.org/pub/gcc/snapshots/14-20230716/ and on various mirrors, see http://gcc.gnu.org/mirrors.html for details. This snapshot has been generated from the GCC 14 git branch with the following options: git://gcc.gnu.org/git/gcc.git branch master revision 061f74c06735e1fa35b910ae0bcf01b61a74ec23 You'll find: gcc-14-20230716.tar.xz Complete GCC SHA256=8be111a65a068a151f59098277dde6676358ff6bb032854d72cc66d92a19876e SHA1=6734736abe8db8202514ab3a0e33e283dc05e042 Diffs from 14-20230709 are available in the diffs/ subdirectory. When a particular snapshot is ready for public consumption the LATEST-14 link is updated and a message is sent to the gcc list. Please do not use a snapshot before it has been announced that way.
