Re: [TCWG CI] 471.omnetpp slowed down by 8% after gcc: Avoid invalid loop transformations in jump threading registry.
> On 27 Sep 2021, at 16:52, Aldy Hernandez wrote: > > [CCing Jeff and list for broader audience] > > On 9/27/21 2:53 PM, Maxim Kuvyrkov wrote: >> Hi Aldy, >> Your patch seems to slow down 471.omnetpp by 8% at -O3. Could you please >> take a look if this is something that could be easily fixed? > > First of all, thanks for chasing this down. It's incredibly useful to have > these types of bug reports. Thanks, Aldy, this is music to my ears :-). We have built this automated benchmarking CI that bisects code-speed and code-size regressions down to a single commit. It is still work-in-progress, and I’m forwarding these reports to patch authors, whose patches caused regressions. If GCC community finds these useful, we can also setup posting to one of GCC’s mailing lists. > > Jeff and I have been discussing the repercussions of adjusting the loop > crossing restrictions in the various threaders. He's seen some regressions > in embedded targets when disallowing certain corner cases of loop crossing > threads causes all sorts of grief. > > Out of curiosity, does the attached (untested) patch fix the regression? I’ll test the patch and will follow up. Regards, -- Maxim Kuvyrkov https://www.linaro.org > > Aldy > >> Regards, >> -- >> Maxim Kuvyrkov >> https://www.linaro.org >>> On 27 Sep 2021, at 02:52, ci_not...@linaro.org wrote: >>> >>> After gcc commit 4a960d548b7d7d942f316c5295f6d849b74214f5 >>> Author: Aldy Hernandez >>> >>>Avoid invalid loop transformations in jump threading registry. >>> >>> the following benchmarks slowed down by more than 2%: >>> - 471.omnetpp slowed down by 8% from 6348 to 6828 perf samples >>> >>> Below reproducer instructions can be used to re-build both "first_bad" and >>> "last_good" cross-toolchains used in this bisection. Naturally, the >>> scripts will fail when triggerring benchmarking jobs if you don't have >>> access to Linaro TCWG CI. >>> >>> For your convenience, we have uploaded tarballs with pre-processed source >>> and assembly files at: >>> - First_bad save-temps: >>> https://ci.linaro.org/job/tcwg_bmk_ci_gnu-bisect-tcwg_bmk_tk1-gnu-master-arm-spec2k6-O3/40/artifact/artifacts/build-4a960d548b7d7d942f316c5295f6d849b74214f5/save-temps/ >>> - Last_good save-temps: >>> https://ci.linaro.org/job/tcwg_bmk_ci_gnu-bisect-tcwg_bmk_tk1-gnu-master-arm-spec2k6-O3/40/artifact/artifacts/build-29c92857039d0a105281be61c10c9e851aaeea4a/save-temps/ >>> - Baseline save-temps: >>> https://ci.linaro.org/job/tcwg_bmk_ci_gnu-bisect-tcwg_bmk_tk1-gnu-master-arm-spec2k6-O3/40/artifact/artifacts/build-baseline/save-temps/ >>> >>> Configuration: >>> - Benchmark: SPEC CPU2006 >>> - Toolchain: GCC + Glibc + GNU Linker >>> - Version: all components were built from their tip of trunk >>> - Target: arm-linux-gnueabihf >>> - Compiler flags: -O3 -marm >>> - Hardware: NVidia TK1 4x Cortex-A15 >>> >>> This benchmarking CI is work-in-progress, and we welcome feedback and >>> suggestions at linaro-toolch...@lists.linaro.org . In our improvement >>> plans is to add support for SPEC CPU2017 benchmarks and provide "perf >>> report/annotate" data behind these reports. >>> >>> THIS IS THE END OF INTERESTING STUFF. BELOW ARE LINKS TO BUILDS, >>> REPRODUCTION INSTRUCTIONS, AND THE RAW COMMIT. >>> >>> This commit has regressed these CI configurations: >>> - tcwg_bmk_gnu_tk1/gnu-master-arm-spec2k6-O3 >>> >>> First_bad build: >>> https://ci.linaro.org/job/tcwg_bmk_ci_gnu-bisect-tcwg_bmk_tk1-gnu-master-arm-spec2k6-O3/40/artifact/artifacts/build-4a960d548b7d7d942f316c5295f6d849b74214f5/ >>> Last_good build: >>> https://ci.linaro.org/job/tcwg_bmk_ci_gnu-bisect-tcwg_bmk_tk1-gnu-master-arm-spec2k6-O3/40/artifact/artifacts/build-29c92857039d0a105281be61c10c9e851aaeea4a/ >>> Baseline build: >>> https://ci.linaro.org/job/tcwg_bmk_ci_gnu-bisect-tcwg_bmk_tk1-gnu-master-arm-spec2k6-O3/40/artifact/artifacts/build-baseline/ >>> Even more details: >>> https://ci.linaro.org/job/tcwg_bmk_ci_gnu-bisect-tcwg_bmk_tk1-gnu-master-arm-spec2k6-O3/40/artifact/artifacts/ >>> >>> Reproduce builds: >>> >>> mkdir investigate-gcc-4a960d548b7d7d942f316c5295f6d849b74214f5 >>> cd investigate-gcc-4a960d548b7d7d942f316c5295f6d849b74214f5 >>> >>> # Fetch scripts >>> git clone https://git.linaro.org/toolchain/jenkins-scripts >>> >>> # Fetch manifests and test.sh script >>> mkdir -p artifacts/manifests >>> curl -o artifacts/manifests/build-baseline.sh >>> https://ci.linaro.org/job/tcwg_bmk_ci_gnu-bisect-tcwg_bmk_tk1-gnu-master-arm-spec2k6-O3/40/artifact/artifacts/manifests/build-baseline.sh >>> --fail >>> curl -o artifacts/manifests/build-parameters.sh >>> https://ci.linaro.org/job/tcwg_bmk_ci_gnu-bisect-tcwg_bmk_tk1-gnu-master-arm-spec2k6-O3/40/artifact/artifacts/manifests/build-parameters.sh >>> --fail >>> curl -o artifacts/test.sh >>> https://ci.linaro.org/job/tcwg_bmk_ci_gnu-bisect-tcwg_bmk_tk1-gnu-master-arm-spec2k6-O3/40/artifact/artifacts/test.sh >>> --fail >>> chmod +x artifacts/
Re: [TCWG CI] 471.omnetpp slowed down by 8% after gcc: Avoid invalid loop transformations in jump threading registry.
> On 27 Sep 2021, at 19:02, Andrew MacLeod wrote:
>
> On 9/27/21 11:39 AM, Maxim Kuvyrkov via Gcc wrote:
>>> On 27 Sep 2021, at 16:52, Aldy Hernandez wrote:
>>>
>>> [CCing Jeff and list for broader audience]
>>>
>>> On 9/27/21 2:53 PM, Maxim Kuvyrkov wrote:
>>>> Hi Aldy,
>>>> Your patch seems to slow down 471.omnetpp by 8% at -O3. Could you please
>>>> take a look if this is something that could be easily fixed?
>>> First of all, thanks for chasing this down. It's incredibly useful to have
>>> these types of bug reports.
>> Thanks, Aldy, this is music to my ears :-).
>>
>> We have built this automated benchmarking CI that bisects code-speed and
>> code-size regressions down to a single commit. It is still
>> work-in-progress, and I’m forwarding these reports to patch authors, whose
>> patches caused regressions. If GCC community finds these useful, we can
>> also setup posting to one of GCC’s mailing lists.
>
> I second that this sort of thing is incredibly useful. I don't suppose its
> easy to do the reverse?... let patch authors know when they've caused a
> significant improvement? :-) That would be much less common I suspect, so
> perhaps not worth it :-)
We do this occasionally, when identifying a regression in a patch revert commit
:-). Seriously, though, it’s an easy enough code-change to the metric, but we
are maxing out our benchmarking capacity with current configuration matrix.
>
> Its certainly very useful when we are making a wholesale change to a pass
> which we think is beneficial, but aren't sure.
>
> And a followup question... Sometimes we have no good way of determining the
> widespread run-time effects of a change. You seem to be running SPEC/other
> things continuously then?
We continuously run SPEC CPU2006 on {arm,aarch64}-{-Os/-O2/-O3}-{no LTO/LTO}
matrix for GNU and LLVM toolchains.
In the GNU toolchain we track master branches and latest-release branches of
Binutils, GCC and Glibc — and detect code-speed and code-size regressions
across all toolchain components.
> Does it run like once a day/some-time-period, and if you note a regression,
> narrow it down?
Configurations that track master branches have 3-day intervals. Configurations
that track release branches — 6 days. If a regression is detected it is
narrowed down to component first — binutils, gcc or glibc — and then the commit
range of the component is bisected down to a specific commit. All. Done.
Automatically.
I will make a presentation on this CI at the next GNU Tools Cauldron.
> Regardless, I think it could be very useful to be able to see the results of
> anything you do run at whatever frequency it happens.
Thanks!
--
Maxim Kuvyrkov
https://www.linaro.org
Re: [TCWG CI] 471.omnetpp slowed down by 8% after gcc: Avoid invalid loop transformations in jump threading registry.
> On 29 Sep 2021, at 21:21, Andrew MacLeod wrote: > > On 9/29/21 7:59 AM, Maxim Kuvyrkov wrote: >> >>> Does it run like once a day/some-time-period, and if you note a >>> regression, narrow it down? >> Configurations that track master branches have 3-day intervals. >> Configurations that track release branches — 6 days. If a regression is >> detected it is narrowed down to component first — binutils, gcc or glibc — >> and then the commit range of the component is bisected down to a specific >> commit. All. Done. Automatically. >> >> I will make a presentation on this CI at the next GNU Tools Cauldron. >> >>> Regardless, I think it could be very useful to be able to see the results >>> of anything you do run at whatever frequency it happens. >> Thanks! >> >> -- > > One more follow on question.. is this information/summary of the results > every 3rd day interval of master published anywhere? ie, to a web page or > posted somewhere?that seems like it could useful, especially with a +/- > differential from the previous run (which you obviously calculate to > determine if there is a regression). It’s our next big improvement — to provide a dashboard with current performance numbers and historical stats. Performance summary information is publicly available as artifacts in jenkins jobs (e.g., [1]), but one needs to know exactly where to look. We plan to implement the dashboard before the end of the year. We also have raw perf.data files and benchmark executables stashed for detailed inspection. I /think/, we can publish these for SPEC CPU2xxx benchmarks — they are all based on open-source software. For other benchmarks (EEMBC, CoreMark Pro) we can’t publish much beyond time/size metrics. [1] https://ci.linaro.org/view/tcwg_bmk_ci_gnu/job/tcwg_bmk_ci_gnu-build-tcwg_bmk_tx1-gnu-master-aarch64-spec2k6-O2/237/artifact/artifacts/11-check_regression/results.csv/*view*/ Regards, -- Maxim Kuvyrkov https://www.linaro.org
Re: [TCWG CI] 471.omnetpp slowed down by 8% after gcc: Avoid invalid loop transformations in jump threading registry.
> On 8 Oct 2021, at 13:22, Martin Jambor wrote: > > Hi, > > On Fri, Oct 01 2021, Gerald Pfeifer wrote: >> On Wed, 29 Sep 2021, Maxim Kuvyrkov via Gcc wrote: >>> Configurations that track master branches have 3-day intervals. >>> Configurations that track release branches — 6 days. If a regression is >>> detected it is narrowed down to component first — binutils, gcc or glibc >>> — and then the commit range of the component is bisected down to a >>> specific commit. All. Done. Automatically. >>> >>> I will make a presentation on this CI at the next GNU Tools Cauldron. >> >> Yes, please! :-) >> >> On Fri, 1 Oct 2021, Maxim Kuvyrkov via Gcc wrote: >>> It’s our next big improvement — to provide a dashboard with current >>> performance numbers and historical stats. >> >> Awesome. And then we can even link from gcc.gnu.org. >> > > You all are aware of the openSUSE LNT periodic SPEC benchmarker, right? > Martin may explain better how to move around it, but the two most > interesting result pages are: > > - https://lnt.opensuse.org/db_default/v4/SPEC/latest_runs_report and > - https://lnt.opensuse.org/db_default/v4/SPEC/spec_report/branch > Hi Martin, The novel part of TCWG CI is that it bisects “regressions” down to a single commit, thus pin-pointing the interesting commit, and can send out notifications to patch authors. We do generate a fair number of benchmarking data for AArch64 and AArch32, and I want to have them plotted somewhere. I have started to put together an LNT instance to do that, but after a couple of days I couldn't figure out the setup. Could you share the configuration of your LNT instance? Or, perhaps, make it open to the community so that others can upload the results? Thanks, -- Maxim Kuvyrkov https://www.linaro.org
Re: [PATCH v6 0/4] P1689R5 support
> On Jun 7, 2023, at 00:50, Ben Boeckel via Gcc-patches > wrote: > > Hi, > > This patch series adds initial support for ISO C++'s [P1689R5][], a > format for describing C++ module requirements and provisions based on > the source code. This is required because compiling C++ with modules is > not embarrassingly parallel and need to be ordered to ensure that > `import some_module;` can be satisfied in time by making sure that any > TU with `export import some_module;` is compiled first. Hi Ben, This patch series causes ICEs on arm-linux-gnueabihf. Would you please investigate? Please let me know if you need any in reproducing these. === g++ tests === Running g++:g++.dg/modules/modules.exp ... FAIL: g++.dg/modules/ben-1_a.C -std=c++17 (internal compiler error: Segmentation fault) FAIL: g++.dg/modules/ben-1_a.C -std=c++17 (test for excess errors) FAIL: g++.dg/modules/ben-1_a.C -std=c++2a (internal compiler error: Segmentation fault) FAIL: g++.dg/modules/ben-1_a.C -std=c++2a (test for excess errors) FAIL: g++.dg/modules/ben-1_a.C -std=c++2b (internal compiler error: Segmentation fault) FAIL: g++.dg/modules/ben-1_a.C -std=c++2b (test for excess errors) FAIL: g++.dg/modules/ben-1_a.C module-cmi =partitions/module-import.mod (partitions/module-import.mod) FAIL: g++.dg/modules/ben-1_b.C -std=c++17 (internal compiler error: Segmentation fault) FAIL: g++.dg/modules/ben-1_b.C -std=c++17 (test for excess errors) FAIL: g++.dg/modules/ben-1_b.C -std=c++2a (internal compiler error: Segmentation fault) FAIL: g++.dg/modules/ben-1_b.C -std=c++2a (test for excess errors) FAIL: g++.dg/modules/ben-1_b.C -std=c++2b (internal compiler error: Segmentation fault) FAIL: g++.dg/modules/ben-1_b.C -std=c++2b (test for excess errors) FAIL: g++.dg/modules/ben-1_b.C module-cmi =module.mod (module.mod) FAIL: g++.dg/modules/gc-2_a.C -std=c++17 (internal compiler error: Segmentation fault) FAIL: g++.dg/modules/gc-2_a.C -std=c++17 (test for excess errors) FAIL: g++.dg/modules/gc-2_a.C -std=c++2a (internal compiler error: Segmentation fault) FAIL: g++.dg/modules/gc-2_a.C -std=c++2a (test for excess errors) FAIL: g++.dg/modules/gc-2_a.C -std=c++2b (internal compiler error: Segmentation fault) FAIL: g++.dg/modules/gc-2_a.C -std=c++2b (test for excess errors) FAIL: g++.dg/modules/gc-2_a.C module-cmi =map-1_a.nms (map-1_a.nms) UNRESOLVED: g++.dg/modules/map-1 -std=c++17 execute UNRESOLVED: g++.dg/modules/map-1 -std=c++17 link UNRESOLVED: g++.dg/modules/map-1 -std=c++2a execute UNRESOLVED: g++.dg/modules/map-1 -std=c++2a link UNRESOLVED: g++.dg/modules/map-1 -std=c++2b execute UNRESOLVED: g++.dg/modules/map-1 -std=c++2b link FAIL: g++.dg/modules/map-1_a.C -std=c++17 (internal compiler error: Segmentation fault) FAIL: g++.dg/modules/map-1_a.C -std=c++17 (test for excess errors) FAIL: g++.dg/modules/map-1_a.C -std=c++2a (internal compiler error: Segmentation fault) FAIL: g++.dg/modules/map-1_a.C -std=c++2a (test for excess errors) FAIL: g++.dg/modules/map-1_a.C -std=c++2b (internal compiler error: Segmentation fault) FAIL: g++.dg/modules/map-1_a.C -std=c++2b (test for excess errors) FAIL: g++.dg/modules/map-1_a.C module-cmi =map-1_a.nms (map-1_a.nms) FAIL: g++.dg/modules/map-1_b.C -std=c++17 (internal compiler error: Segmentation fault) FAIL: g++.dg/modules/map-1_b.C -std=c++17 (test for excess errors) FAIL: g++.dg/modules/map-1_b.C -std=c++2a (internal compiler error: Segmentation fault) FAIL: g++.dg/modules/map-1_b.C -std=c++2a (test for excess errors) FAIL: g++.dg/modules/map-1_b.C -std=c++2b (internal compiler error: Segmentation fault) FAIL: g++.dg/modules/map-1_b.C -std=c++2b (test for excess errors) FAIL: g++.dg/modules/map-2.C -std=c++17 at line 8 (test for errors, line 7) FAIL: g++.dg/modules/map-2.C -std=c++17 at line 9 (test for errors, line ) FAIL: g++.dg/modules/map-2.C -std=c++17 (internal compiler error: Segmentation fault) FAIL: g++.dg/modules/map-2.C -std=c++17 (test for excess errors) FAIL: g++.dg/modules/map-2.C -std=c++2a at line 8 (test for errors, line 7) FAIL: g++.dg/modules/map-2.C -std=c++2a at line 9 (test for errors, line ) FAIL: g++.dg/modules/map-2.C -std=c++2a (internal compiler error: Segmentation fault) FAIL: g++.dg/modules/map-2.C -std=c++2a (test for excess errors) FAIL: g++.dg/modules/map-2.C -std=c++2b at line 8 (test for errors, line 7) FAIL: g++.dg/modules/map-2.C -std=c++2b at line 9 (test for errors, line ) FAIL: g++.dg/modules/map-2.C -std=c++2b (internal compiler error: Segmentation fault) FAIL: g++.dg/modules/map-2.C -std=c++2b (test for excess errors) === Thanks, -- Maxim Kuvyrkov https://www.linaro.org > > [P1689R5]: https://isocpp.org/files/papers/P1689R5.html > > I've also added patches to include imported module CMI files and the > module mapper file as dependencies of the compilation. I briefly looked > into adding dependencies on response files as well, but that appeared to > need some code contortions to have a `class mkdeps` available before > parsing the comma
Re: Checks that autotools generated files were re-generated correctly
> On Nov 6, 2023, at 21:19, Christophe Lyon wrote: > > Hi! > > On Mon, 6 Nov 2023 at 18:05, Martin Jambor wrote: >> >> Hello, >> >> I have inherited Martin Liška's buildbot script that checks that all >> sorts of autotools generated files, mainly configure scripts, were >> re-generated correctly when appropriate. While the checks are hopefully >> useful, they report issues surprisingly often and reporting them feels >> especially unproductive. >> >> Could such checks be added to our server side push hooks so that commits >> introducing these breakages would get refused automatically. While the >> check might be a bit expensive, it only needs to be run on files >> touching the generated files and/or the files these are generated from. >> >> Alternatively, Maxim, you seem to have an infrastructure that is capable >> of sending email. Would you consider adding the check to your buildbot >> instance and report issues automatically? The level of totally > > After the discussions we had during Cauldron, I actually thought we > should add such a bot. > > Initially I was thinking about adding this as a "precommit" check, to > make sure the autogenerated files were submitted correctly, but I > realized that the policy is actually not to send autogenerated files > as part of the patch (thus making pre-commit check impracticable in > such cases, unless we autogenerate those files after applying the > patch) > > I understand you mean to run this as a post-commit bot, meaning we > would continue to "accept" broken commits, but now automatically send > a notification, asking for a prompt fix? > > We can probably implement that, indeed. Is that the general agreement? [CC: Siddhesh, Carlos] Hi Martin, I agree with Christophe, and we can add various source-level checks and wrap them up as a post-commit job. The job will then send out email reports to developers whose patches failed it. Where the current script is located? These checks would be useful for all GNU Toolchain projects -- binutils/GDB, GCC, Glibc and, maybe, Newlib -- so it would be useful to put it in a separate "gnutools" repo. I think Siddhesh and Carlos are looking into creating such a repo on gitlab? Thanks, -- Maxim Kuvyrkov https://www.linaro.org
