Re: [TCWG CI] 433.milc slowed down by 4% after llvm: Add missing header
Hi David, This is a false positive, sorry for the noise. Our CI bisects performance regressions down to a single commit, and notifies patch authors about significant regressions. Since this is benchmarking CI, some noise is expected. Apparently, 433.milc started to show bi-modal performance, and bisection [mistakenly] converged on this patch. We work to reduce benchmarking noise and otherwise improve benchmarking CI. Regards! -- Maxim Kuvyrkov https://www.linaro.org > On 7 Dec 2021, at 06:12, David Blaikie wrote: > > Seems... unlikely this change had a performance impact. > > Also is this email meant to be sent to public contributors like myself, or > only intended for some Linaro folks? > > On Sun, Dec 5, 2021 at 6:18 AM wrote: > >> After llvm commit bd4c6a476fd037fb07a1c484f75d93ee40713d3d >> Author: David Blaikie >> >>Add missing header >> >> the following benchmarks slowed down by more than 2%: >> - 433.milc slowed down by 4% from 12427 to 12916 perf samples >> >> Below reproducer instructions can be used to re-build both "first_bad" and >> "last_good" cross-toolchains used in this bisection. Naturally, the >> scripts will fail when triggerring benchmarking jobs if you don't have >> access to Linaro TCWG CI. >> >> For your convenience, we have uploaded tarballs with pre-processed source >> and assembly files at: >> - First_bad save-temps: >> https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O2_LTO/34/artifact/artifacts/build-bd4c6a476fd037fb07a1c484f75d93ee40713d3d/save-temps/ >> - Last_good save-temps: >> https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O2_LTO/34/artifact/artifacts/build-7d4da4e1ab7f79e51db0d5c2a0f5ef1711122dd7/save-temps/ >> - Baseline save-temps: >> https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O2_LTO/34/artifact/artifacts/build-baseline/save-temps/ >> >> Configuration: >> - Benchmark: SPEC CPU2006 >> - Toolchain: Clang + Glibc + LLVM Linker >> - Version: all components were built from their tip of trunk >> - Target: aarch64-linux-gnu >> - Compiler flags: -O2 -flto >> - Hardware: NVidia TX1 4x Cortex-A57 >> >> This benchmarking CI is work-in-progress, and we welcome feedback and >> suggestions at linaro-toolchain@lists.linaro.org . In our improvement >> plans is to add support for SPEC CPU2017 benchmarks and provide "perf >> report/annotate" data behind these reports. >> >> THIS IS THE END OF INTERESTING STUFF. BELOW ARE LINKS TO BUILDS, >> REPRODUCTION INSTRUCTIONS, AND THE RAW COMMIT. >> >> This commit has regressed these CI configurations: >> - tcwg_bmk_llvm_tx1/llvm-master-aarch64-spec2k6-O2_LTO >> >> First_bad build: >> https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O2_LTO/34/artifact/artifacts/build-bd4c6a476fd037fb07a1c484f75d93ee40713d3d/ >> Last_good build: >> https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O2_LTO/34/artifact/artifacts/build-7d4da4e1ab7f79e51db0d5c2a0f5ef1711122dd7/ >> Baseline build: >> https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O2_LTO/34/artifact/artifacts/build-baseline/ >> Even more details: >> https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O2_LTO/34/artifact/artifacts/ >> >> Reproduce builds: >> >> mkdir investigate-llvm-bd4c6a476fd037fb07a1c484f75d93ee40713d3d >> cd investigate-llvm-bd4c6a476fd037fb07a1c484f75d93ee40713d3d >> >> # Fetch scripts >> git clone https://git.linaro.org/toolchain/jenkins-scripts >> >> # Fetch manifests and test.sh script >> mkdir -p artifacts/manifests >> curl -o artifacts/manifests/build-baseline.sh >> https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O2_LTO/34/artifact/artifacts/manifests/build-baseline.sh >> --fail >> curl -o artifacts/manifests/build-parameters.sh >> https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O2_LTO/34/artifact/artifacts/manifests/build-parameters.sh >> --fail >> curl -o artifacts/test.sh >> https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O2_LTO/34/artifact/artifacts/test.sh >> --fail >> chmod +x artifacts/test.sh >> >> # Reproduce the baseline build (build all pre-requisites) >> ./jenkins-scripts/tcwg_bmk-build.sh @@ >> artifacts/manifests/build-baseline.sh >> >> # Save baseline build state (which is then restored in artifacts/test.sh) >> mkdir -p ./bisect >> rsync -a --del --delete-excluded --exclude /bisect/ --exclude /artifacts/ >> --exclude /llvm/ ./ ./bisect/baseline/ >> >> cd llvm >> >> # Reproduce first_bad build >> git checkout --detach bd4c6a476fd037fb07a1c484f75d93ee40713d3d >> ../artifacts/test.sh >> >> # Reproduce last_good build >> git checkout --detach 7d4da4e1ab7f79e51db0d5c2a0f5ef1711122dd7 >> ../artifacts/test.s
Re: [TCWG CI] 453.povray failed to build after llvm: [SLP]Fix reused extracts cost.
Hi Alexey, After your patch Clang crashes while building 453.povray for aarch64-linux-gnu. Apparently, this happens only with LTO enabled at -O2 and -O3. Did you get any bug reports against this patch already? Thanks, -- Maxim Kuvyrkov https://www.linaro.org > On 5 Dec 2021, at 02:55, ci_not...@linaro.org wrote: > > After llvm commit ba74bb3a226e1b4660537f274627285b1bf41ee1 > Author: Alexey Bataev > >[SLP]Fix reused extracts cost. > > the following benchmarks slowed down by more than 2%: > - 453.povray failed to build > > Below reproducer instructions can be used to re-build both "first_bad" and > "last_good" cross-toolchains used in this bisection. Naturally, the scripts > will fail when triggerring benchmarking jobs if you don't have access to > Linaro TCWG CI. > > For your convenience, we have uploaded tarballs with pre-processed source and > assembly files at: > - First_bad save-temps: > https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O3_LTO/39/artifact/artifacts/build-ba74bb3a226e1b4660537f274627285b1bf41ee1/save-temps/ > - Last_good save-temps: > https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O3_LTO/39/artifact/artifacts/build-78cc133c63173a4b5b7a43750cc507d4cff683cf/save-temps/ > - Baseline save-temps: > https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O3_LTO/39/artifact/artifacts/build-baseline/save-temps/ > > Configuration: > - Benchmark: SPEC CPU2006 > - Toolchain: Clang + Glibc + LLVM Linker > - Version: all components were built from their tip of trunk > - Target: aarch64-linux-gnu > - Compiler flags: -O3 -flto > - Hardware: NVidia TX1 4x Cortex-A57 > > This benchmarking CI is work-in-progress, and we welcome feedback and > suggestions at linaro-toolchain@lists.linaro.org . In our improvement plans > is to add support for SPEC CPU2017 benchmarks and provide "perf > report/annotate" data behind these reports. > > THIS IS THE END OF INTERESTING STUFF. BELOW ARE LINKS TO BUILDS, > REPRODUCTION INSTRUCTIONS, AND THE RAW COMMIT. > > This commit has regressed these CI configurations: > - tcwg_bmk_llvm_tx1/llvm-master-aarch64-spec2k6-O3_LTO > > First_bad build: > https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O3_LTO/39/artifact/artifacts/build-ba74bb3a226e1b4660537f274627285b1bf41ee1/ > Last_good build: > https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O3_LTO/39/artifact/artifacts/build-78cc133c63173a4b5b7a43750cc507d4cff683cf/ > Baseline build: > https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O3_LTO/39/artifact/artifacts/build-baseline/ > Even more details: > https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O3_LTO/39/artifact/artifacts/ > > Reproduce builds: > > mkdir investigate-llvm-ba74bb3a226e1b4660537f274627285b1bf41ee1 > cd investigate-llvm-ba74bb3a226e1b4660537f274627285b1bf41ee1 > > # Fetch scripts > git clone https://git.linaro.org/toolchain/jenkins-scripts > > # Fetch manifests and test.sh script > mkdir -p artifacts/manifests > curl -o artifacts/manifests/build-baseline.sh > https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O3_LTO/39/artifact/artifacts/manifests/build-baseline.sh > --fail > curl -o artifacts/manifests/build-parameters.sh > https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O3_LTO/39/artifact/artifacts/manifests/build-parameters.sh > --fail > curl -o artifacts/test.sh > https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O3_LTO/39/artifact/artifacts/test.sh > --fail > chmod +x artifacts/test.sh > > # Reproduce the baseline build (build all pre-requisites) > ./jenkins-scripts/tcwg_bmk-build.sh @@ artifacts/manifests/build-baseline.sh > > # Save baseline build state (which is then restored in artifacts/test.sh) > mkdir -p ./bisect > rsync -a --del --delete-excluded --exclude /bisect/ --exclude /artifacts/ > --exclude /llvm/ ./ ./bisect/baseline/ > > cd llvm > > # Reproduce first_bad build > git checkout --detach ba74bb3a226e1b4660537f274627285b1bf41ee1 > ../artifacts/test.sh > > # Reproduce last_good build > git checkout --detach 78cc133c63173a4b5b7a43750cc507d4cff683cf > ../artifacts/test.sh > > cd .. > > > Full commit (up to 1000 lines): > > commit ba74bb3a226e1b4660537f274627285b1bf41ee1 > Author: Alexey Bataev > Date: Thu Dec 2 04:22:55 2021 -0800 > >[SLP]Fix reused extracts cost. > >If the extractelement instruction is used multiple times in the >different tree entries (either vectorized, or gathered), need to >compensate the scalar cost of such instructions. They are completely >removed if all users are part of the tree but we need to compensate the >cost only once for
Re: [TCWG CI] 453.povray failed to build after llvm: [SLP]Fix reused extracts cost.
I committed a fix yesterday, should be fixed. Another one planning to commit later today or tomorrow. Best regards, Alexey Bataev > 7 дек. 2021 г., в 07:08, Maxim Kuvyrkov > написал(а): > > Hi Alexey, > > After your patch Clang crashes while building 453.povray for > aarch64-linux-gnu. Apparently, this happens only with LTO enabled at -O2 and > -O3. > > Did you get any bug reports against this patch already? > > Thanks, > > -- > Maxim Kuvyrkov > https://www.linaro.org > >> On 5 Dec 2021, at 02:55, ci_not...@linaro.org wrote: >> >> After llvm commit ba74bb3a226e1b4660537f274627285b1bf41ee1 >> Author: Alexey Bataev >> >> [SLP]Fix reused extracts cost. >> >> the following benchmarks slowed down by more than 2%: >> - 453.povray failed to build >> >> Below reproducer instructions can be used to re-build both "first_bad" and >> "last_good" cross-toolchains used in this bisection. Naturally, the scripts >> will fail when triggerring benchmarking jobs if you don't have access to >> Linaro TCWG CI. >> >> For your convenience, we have uploaded tarballs with pre-processed source >> and assembly files at: >> - First_bad save-temps: >> https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O3_LTO/39/artifact/artifacts/build-ba74bb3a226e1b4660537f274627285b1bf41ee1/save-temps/ >> - Last_good save-temps: >> https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O3_LTO/39/artifact/artifacts/build-78cc133c63173a4b5b7a43750cc507d4cff683cf/save-temps/ >> - Baseline save-temps: >> https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O3_LTO/39/artifact/artifacts/build-baseline/save-temps/ >> >> Configuration: >> - Benchmark: SPEC CPU2006 >> - Toolchain: Clang + Glibc + LLVM Linker >> - Version: all components were built from their tip of trunk >> - Target: aarch64-linux-gnu >> - Compiler flags: -O3 -flto >> - Hardware: NVidia TX1 4x Cortex-A57 >> >> This benchmarking CI is work-in-progress, and we welcome feedback and >> suggestions at linaro-toolchain@lists.linaro.org . In our improvement plans >> is to add support for SPEC CPU2017 benchmarks and provide "perf >> report/annotate" data behind these reports. >> >> THIS IS THE END OF INTERESTING STUFF. BELOW ARE LINKS TO BUILDS, >> REPRODUCTION INSTRUCTIONS, AND THE RAW COMMIT. >> >> This commit has regressed these CI configurations: >> - tcwg_bmk_llvm_tx1/llvm-master-aarch64-spec2k6-O3_LTO >> >> First_bad build: >> https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O3_LTO/39/artifact/artifacts/build-ba74bb3a226e1b4660537f274627285b1bf41ee1/ >> Last_good build: >> https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O3_LTO/39/artifact/artifacts/build-78cc133c63173a4b5b7a43750cc507d4cff683cf/ >> Baseline build: >> https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O3_LTO/39/artifact/artifacts/build-baseline/ >> Even more details: >> https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O3_LTO/39/artifact/artifacts/ >> >> Reproduce builds: >> >> mkdir investigate-llvm-ba74bb3a226e1b4660537f274627285b1bf41ee1 >> cd investigate-llvm-ba74bb3a226e1b4660537f274627285b1bf41ee1 >> >> # Fetch scripts >> git clone https://git.linaro.org/toolchain/jenkins-scripts >> >> # Fetch manifests and test.sh script >> mkdir -p artifacts/manifests >> curl -o artifacts/manifests/build-baseline.sh >> https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O3_LTO/39/artifact/artifacts/manifests/build-baseline.sh >> --fail >> curl -o artifacts/manifests/build-parameters.sh >> https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O3_LTO/39/artifact/artifacts/manifests/build-parameters.sh >> --fail >> curl -o artifacts/test.sh >> https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O3_LTO/39/artifact/artifacts/test.sh >> --fail >> chmod +x artifacts/test.sh >> >> # Reproduce the baseline build (build all pre-requisites) >> ./jenkins-scripts/tcwg_bmk-build.sh @@ artifacts/manifests/build-baseline.sh >> >> # Save baseline build state (which is then restored in artifacts/test.sh) >> mkdir -p ./bisect >> rsync -a --del --delete-excluded --exclude /bisect/ --exclude /artifacts/ >> --exclude /llvm/ ./ ./bisect/baseline/ >> >> cd llvm >> >> # Reproduce first_bad build >> git checkout --detach ba74bb3a226e1b4660537f274627285b1bf41ee1 >> ../artifacts/test.sh >> >> # Reproduce last_good build >> git checkout --detach 78cc133c63173a4b5b7a43750cc507d4cff683cf >> ../artifacts/test.sh >> >> cd .. >> >> >> Full commit (up to 1000 lines): >> >> commit ba74bb3a226e1b4660537f274627285b1bf41ee1 >> Author: Alexey Bataev >> Date: Thu Dec 2 04:22:55 2021 -0800 >> >> [SLP]Fix reused extract
Re: [TCWG CI] 453.povray failed to build after llvm: [SLP]Fix reused extracts cost.
Cool, thanks! -- Maxim Kuvyrkov https://www.linaro.org > On 7 Dec 2021, at 15:10, Alexey Bataev wrote: > > I committed a fix yesterday, should be fixed. Another one planning to commit > later today or tomorrow. > > Best regards, > Alexey Bataev > >> 7 дек. 2021 г., в 07:08, Maxim Kuvyrkov >> написал(а): >> >> Hi Alexey, >> >> After your patch Clang crashes while building 453.povray for >> aarch64-linux-gnu. Apparently, this happens only with LTO enabled at -O2 >> and -O3. >> >> Did you get any bug reports against this patch already? >> >> Thanks, >> >> -- >> Maxim Kuvyrkov >> https://www.linaro.org >> >>> On 5 Dec 2021, at 02:55, ci_not...@linaro.org wrote: >>> >>> After llvm commit ba74bb3a226e1b4660537f274627285b1bf41ee1 >>> Author: Alexey Bataev >>> >>> [SLP]Fix reused extracts cost. >>> >>> the following benchmarks slowed down by more than 2%: >>> - 453.povray failed to build >>> >>> Below reproducer instructions can be used to re-build both "first_bad" and >>> "last_good" cross-toolchains used in this bisection. Naturally, the >>> scripts will fail when triggerring benchmarking jobs if you don't have >>> access to Linaro TCWG CI. >>> >>> For your convenience, we have uploaded tarballs with pre-processed source >>> and assembly files at: >>> - First_bad save-temps: >>> https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O3_LTO/39/artifact/artifacts/build-ba74bb3a226e1b4660537f274627285b1bf41ee1/save-temps/ >>> - Last_good save-temps: >>> https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O3_LTO/39/artifact/artifacts/build-78cc133c63173a4b5b7a43750cc507d4cff683cf/save-temps/ >>> - Baseline save-temps: >>> https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O3_LTO/39/artifact/artifacts/build-baseline/save-temps/ >>> >>> Configuration: >>> - Benchmark: SPEC CPU2006 >>> - Toolchain: Clang + Glibc + LLVM Linker >>> - Version: all components were built from their tip of trunk >>> - Target: aarch64-linux-gnu >>> - Compiler flags: -O3 -flto >>> - Hardware: NVidia TX1 4x Cortex-A57 >>> >>> This benchmarking CI is work-in-progress, and we welcome feedback and >>> suggestions at linaro-toolchain@lists.linaro.org . In our improvement >>> plans is to add support for SPEC CPU2017 benchmarks and provide "perf >>> report/annotate" data behind these reports. >>> >>> THIS IS THE END OF INTERESTING STUFF. BELOW ARE LINKS TO BUILDS, >>> REPRODUCTION INSTRUCTIONS, AND THE RAW COMMIT. >>> >>> This commit has regressed these CI configurations: >>> - tcwg_bmk_llvm_tx1/llvm-master-aarch64-spec2k6-O3_LTO >>> >>> First_bad build: >>> https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O3_LTO/39/artifact/artifacts/build-ba74bb3a226e1b4660537f274627285b1bf41ee1/ >>> Last_good build: >>> https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O3_LTO/39/artifact/artifacts/build-78cc133c63173a4b5b7a43750cc507d4cff683cf/ >>> Baseline build: >>> https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O3_LTO/39/artifact/artifacts/build-baseline/ >>> Even more details: >>> https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O3_LTO/39/artifact/artifacts/ >>> >>> Reproduce builds: >>> >>> mkdir investigate-llvm-ba74bb3a226e1b4660537f274627285b1bf41ee1 >>> cd investigate-llvm-ba74bb3a226e1b4660537f274627285b1bf41ee1 >>> >>> # Fetch scripts >>> git clone https://git.linaro.org/toolchain/jenkins-scripts >>> >>> # Fetch manifests and test.sh script >>> mkdir -p artifacts/manifests >>> curl -o artifacts/manifests/build-baseline.sh >>> https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O3_LTO/39/artifact/artifacts/manifests/build-baseline.sh >>> --fail >>> curl -o artifacts/manifests/build-parameters.sh >>> https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O3_LTO/39/artifact/artifacts/manifests/build-parameters.sh >>> --fail >>> curl -o artifacts/test.sh >>> https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O3_LTO/39/artifact/artifacts/test.sh >>> --fail >>> chmod +x artifacts/test.sh >>> >>> # Reproduce the baseline build (build all pre-requisites) >>> ./jenkins-scripts/tcwg_bmk-build.sh @@ artifacts/manifests/build-baseline.sh >>> >>> # Save baseline build state (which is then restored in artifacts/test.sh) >>> mkdir -p ./bisect >>> rsync -a --del --delete-excluded --exclude /bisect/ --exclude /artifacts/ >>> --exclude /llvm/ ./ ./bisect/baseline/ >>> >>> cd llvm >>> >>> # Reproduce first_bad build >>> git checkout --detach ba74bb3a226e1b4660537f274627285b1bf41ee1 >>> ../artifacts/test.sh >>> >>> # Reproduce last_good build >>> git checkout --detach 78cc133c63173a4b5b7a43750cc507d4cff683cf >>> ../ar
Re: [TCWG CI] 433.milc slowed down by 4% after llvm: Add missing header
OK , thanks - appreciate all the hard work of making performance regression analysis & continued improvement to reduce false positives! On Tue, Dec 7, 2021 at 4:03 AM Maxim Kuvyrkov wrote: > Hi David, > > This is a false positive, sorry for the noise. Our CI bisects performance > regressions down to a single commit, and notifies patch authors about > significant regressions. > > Since this is benchmarking CI, some noise is expected. Apparently, > 433.milc started to show bi-modal performance, and bisection [mistakenly] > converged on this patch. We work to reduce benchmarking noise and > otherwise improve benchmarking CI. > > Regards! > > -- > Maxim Kuvyrkov > https://www.linaro.org > > > On 7 Dec 2021, at 06:12, David Blaikie wrote: > > > > Seems... unlikely this change had a performance impact. > > > > Also is this email meant to be sent to public contributors like myself, > or > > only intended for some Linaro folks? > > > > On Sun, Dec 5, 2021 at 6:18 AM wrote: > > > >> After llvm commit bd4c6a476fd037fb07a1c484f75d93ee40713d3d > >> Author: David Blaikie > >> > >>Add missing header > >> > >> the following benchmarks slowed down by more than 2%: > >> - 433.milc slowed down by 4% from 12427 to 12916 perf samples > >> > >> Below reproducer instructions can be used to re-build both "first_bad" > and > >> "last_good" cross-toolchains used in this bisection. Naturally, the > >> scripts will fail when triggerring benchmarking jobs if you don't have > >> access to Linaro TCWG CI. > >> > >> For your convenience, we have uploaded tarballs with pre-processed > source > >> and assembly files at: > >> - First_bad save-temps: > >> > https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O2_LTO/34/artifact/artifacts/build-bd4c6a476fd037fb07a1c484f75d93ee40713d3d/save-temps/ > >> - Last_good save-temps: > >> > https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O2_LTO/34/artifact/artifacts/build-7d4da4e1ab7f79e51db0d5c2a0f5ef1711122dd7/save-temps/ > >> - Baseline save-temps: > >> > https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O2_LTO/34/artifact/artifacts/build-baseline/save-temps/ > >> > >> Configuration: > >> - Benchmark: SPEC CPU2006 > >> - Toolchain: Clang + Glibc + LLVM Linker > >> - Version: all components were built from their tip of trunk > >> - Target: aarch64-linux-gnu > >> - Compiler flags: -O2 -flto > >> - Hardware: NVidia TX1 4x Cortex-A57 > >> > >> This benchmarking CI is work-in-progress, and we welcome feedback and > >> suggestions at linaro-toolchain@lists.linaro.org . In our improvement > >> plans is to add support for SPEC CPU2017 benchmarks and provide "perf > >> report/annotate" data behind these reports. > >> > >> THIS IS THE END OF INTERESTING STUFF. BELOW ARE LINKS TO BUILDS, > >> REPRODUCTION INSTRUCTIONS, AND THE RAW COMMIT. > >> > >> This commit has regressed these CI configurations: > >> - tcwg_bmk_llvm_tx1/llvm-master-aarch64-spec2k6-O2_LTO > >> > >> First_bad build: > >> > https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O2_LTO/34/artifact/artifacts/build-bd4c6a476fd037fb07a1c484f75d93ee40713d3d/ > >> Last_good build: > >> > https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O2_LTO/34/artifact/artifacts/build-7d4da4e1ab7f79e51db0d5c2a0f5ef1711122dd7/ > >> Baseline build: > >> > https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O2_LTO/34/artifact/artifacts/build-baseline/ > >> Even more details: > >> > https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O2_LTO/34/artifact/artifacts/ > >> > >> Reproduce builds: > >> > >> mkdir investigate-llvm-bd4c6a476fd037fb07a1c484f75d93ee40713d3d > >> cd investigate-llvm-bd4c6a476fd037fb07a1c484f75d93ee40713d3d > >> > >> # Fetch scripts > >> git clone https://git.linaro.org/toolchain/jenkins-scripts > >> > >> # Fetch manifests and test.sh script > >> mkdir -p artifacts/manifests > >> curl -o artifacts/manifests/build-baseline.sh > >> > https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O2_LTO/34/artifact/artifacts/manifests/build-baseline.sh > >> --fail > >> curl -o artifacts/manifests/build-parameters.sh > >> > https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O2_LTO/34/artifact/artifacts/manifests/build-parameters.sh > >> --fail > >> curl -o artifacts/test.sh > >> > https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O2_LTO/34/artifact/artifacts/test.sh > >> --fail > >> chmod +x artifacts/test.sh > >> > >> # Reproduce the baseline build (build all pre-requisites) > >> ./jenkins-scripts/tcwg_bmk-build.sh @@ > >> artifacts/manifests/build-baseline.sh > >> > >> # Save baseline build state (which is then restored in > artifacts/test.sh) > >> mk
