I've looked at the build status reporter, which I think is https://github.com/llvm/llvm-zorg/blob/ec94a29bfb0c247f0123c4efaf080a9b215d4433/zorg/buildbot/reporters/utils.py#L206. I don't see anything wrong with that, and nothing that it removed from Buildbot's version of that function looks like it would cause this problem.
If the previous result was somehow still the last green build, this could happen. However I don't see how that could be the case as there is a finished build on the same worker prior to this one. So even if every other worker was still busy, there's at least one previously failed result. My guess from the Buildbot code is that prev_build["result"] will be None if the build has not finished, so even in this case it would return False and not send a message (though I suspect in progress builds aren't part of these checks or perhaps it waits for the previous one to finish, not sure). I do see that the build prior to yours finished later, and some of the builds immediately after the green build have some unusual timings where earlier builds finish after later builds. This is expected with the 3 workers we have, but perhaps could have affected when the results get written to the database? Perhaps every build between the last green and yours was notified and either no one said anything, or only builds with 1 commit were notified and didn't say anything. The other 1 commit build is https://lab.llvm.org/buildbot/#/builders/4/builds/940, I will ask the author if they received anything. Though even if they did I'm not sure what we could conclude from that. On Wed, 7 Aug 2024 at 02:12, David Blaikie <dblai...@gmail.com> wrote: > Any update on this? > > On Fri, Jul 26, 2024 at 1:28 AM David Spickett <david.spick...@linaro.org> > wrote: > >> I don't see any special configuration for this builder that would make it >> do this. >> ( >> https://github.com/llvm/llvm-zorg/blob/9496234c3e76d33aaae3988903c641fd406e1098/buildbot/osuosl/master/config/builders.py#L527 >> ) >> >> It's unlikely that this build finished before previous jobs because it >> only takes an hour and we have several builds in the hours before it. I >> thought it may also be the first build on that worker that failed, but this >> is also not the case (and if it were, this would be a buildbot bug since >> it's supposed to look at the builder as a whole, not specific workers). >> >> It is a single commit blame list, and I know we started posting results >> to GitHub in these cases. Presumably those have to also be green -> red >> transitions, but perhaps something slipped through here and because there >> was no PR to associate this change with, it sent an email instead? >> >> Thanks, >> David Spickett. >> >> On Fri, 26 Jul 2024 at 03:03, David Blaikie <dblai...@gmail.com> wrote: >> >>> ---------- Forwarded message --------- >>> From: David Blaikie <dblai...@gmail.com> >>> Date: Thu, Jul 25, 2024 at 6:43 PM >>> Subject: Re: ☠ Buildbot (LLVM Buildbot): clang-aarch64-sve-vls-2stage - >>> failed test (failure) (main) >>> To: <llvm.buildmas...@lab.llvm.org> >>> Cc: <llvm.buildmas...@lab.llvm.org> >>> >>> >>> This bot seems to be sending fail-mail when it's already red. My >>> understanding was that fail mail only went out on green->red transitions. >>> Is that understanding incorrect? Should this be reconfigured to match >>> that >>> understanding/not send fail mail when it's already red? >>> >>> On Thu, Jul 25, 2024 at 4:10 PM <llvm.buildmas...@lab.llvm.org> wrote: >>> >>> > The Buildbot has detected a failed build on builder >>> > clang-aarch64-sve-vls-2stage while building clang. >>> > >>> > Full details are available at: >>> > https://lab.llvm.org/buildbot/#/builders/4/builds/943 >>> > >>> > Worker for this Build: linaro-g3-02 >>> > Blamelist: >>> > David Blaikie <dblai...@gmail.com> >>> > >>> > BUILD FAILED: failed test (failure) >>> > >>> > Step 18 (test-suite) failure: test (failure) >>> > ... >>> > [ 24%] Built target >>> > gfortran-regression-execute-regression__internal_readwrite_1_f90 >>> > cd >>> > >>> /home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Fortran/gfortran/regression >>> > && >>> > >>> /home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/stage2.install/bin/llvm-size >>> > --format=sysv >>> > >>> /home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Fortran/gfortran/regression/gfortran-regression-execute-regression__internal_readwrite_4_f90 >>> > > >>> > >>> /home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Fortran/gfortran/regression/gfortran-regression-execute-regression__internal_readwrite_4_f90.size >>> > make[2]: Leaving directory >>> > >>> '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build' >>> > [ 24%] Built target >>> > gfortran-regression-execute-regression__internal_readwrite_4_f90 >>> > make[1]: Leaving directory >>> > >>> '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build' >>> > make: *** [Makefile:136: all] Error 2 >>> > 2024-07-25 23:09:27 INFO: Testing... >>> > 2024-07-25 23:09:27 INFO: Execute: >>> > >>> /home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/stage1/bin/llvm-lit >>> > -v -j 32 >>> > >>> /home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build >>> > -o >>> > >>> /home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/outputqee5waec.json >>> > -- Testing: 9770 tests, 32 workers -- >>> > NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_1026.test >>> (1 of >>> > 9770) >>> > ******************** TEST 'test-suite :: >>> > Bitcode/simd_ops/simd_ops_test_op_abs_1026.test' FAILED >>> ******************** >>> > Executable >>> > >>> '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_1026' >>> > is missing >>> > ******************** >>> > NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_1028.test >>> (2 of >>> > 9770) >>> > ******************** TEST 'test-suite :: >>> > Bitcode/simd_ops/simd_ops_test_op_abs_1028.test' FAILED >>> ******************** >>> > Executable >>> > >>> '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_1028' >>> > is missing >>> > ******************** >>> > NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_367.test (3 >>> of >>> > 9770) >>> > ******************** TEST 'test-suite :: >>> > Bitcode/simd_ops/simd_ops_test_op_abs_367.test' FAILED >>> ******************** >>> > Executable >>> > >>> '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_367' >>> > is missing >>> > ******************** >>> > NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_40.test (4 >>> of >>> > 9770) >>> > ******************** TEST 'test-suite :: >>> > Bitcode/simd_ops/simd_ops_test_op_abs_40.test' FAILED >>> ******************** >>> > Executable >>> > >>> '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_40' >>> > is missing >>> > ******************** >>> > NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_366.test (5 >>> of >>> > 9770) >>> > ******************** TEST 'test-suite :: >>> > Bitcode/simd_ops/simd_ops_test_op_abs_366.test' FAILED >>> ******************** >>> > Executable >>> > >>> '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_366' >>> > is missing >>> > ******************** >>> > NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_add_1030.test >>> (6 of >>> > 9770) >>> > ******************** TEST 'test-suite :: >>> > Bitcode/simd_ops/simd_ops_test_op_add_1030.test' FAILED >>> ******************** >>> > Executable >>> > >>> '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_add_1030' >>> > is missing >>> > ******************** >>> > NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_699.test (7 >>> of >>> > 9770) >>> > ******************** TEST 'test-suite :: >>> > Bitcode/simd_ops/simd_ops_test_op_abs_699.test' FAILED >>> ******************** >>> > Executable >>> > >>> '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_699' >>> > is missing >>> > ******************** >>> > NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_368.test (8 >>> of >>> > 9770) >>> > ******************** TEST 'test-suite :: >>> > Bitcode/simd_ops/simd_ops_test_op_abs_368.test' FAILED >>> ******************** >>> > Executable >>> > >>> '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_368' >>> > is missing >>> > ******************** >>> > NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_add_1031.test >>> (9 of >>> > 9770) >>> > ******************** TEST 'test-suite :: >>> > Bitcode/simd_ops/simd_ops_test_op_add_1031.test' FAILED >>> ******************** >>> > Executable >>> > >>> '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_add_1031' >>> > is missing >>> > ******************** >>> > NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_39.test (10 >>> of >>> > 9770) >>> > ******************** TEST 'test-suite :: >>> > Bitcode/simd_ops/simd_ops_test_op_abs_39.test' FAILED >>> ******************** >>> > Executable >>> > >>> '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_39' >>> > is missing >>> > ******************** >>> > >>> > Sincerely, >>> > LLVM Buildbot >>> > >>> > >>> _______________________________________________ >>> linaro-toolchain mailing list -- linaro-toolchain@lists.linaro.org >>> To unsubscribe send an email to linaro-toolchain-le...@lists.linaro.org >>> >> _______________________________________________ linaro-toolchain mailing list -- linaro-toolchain@lists.linaro.org To unsubscribe send an email to linaro-toolchain-le...@lists.linaro.org
