Fair - any chance we can put some logging/more verbose output into the buildbot config so we might know more about how this happens next time?
On Fri, Aug 9, 2024 at 1:31 AM David Spickett <david.spick...@linaro.org> wrote: > > The other 1 commit build is > https://lab.llvm.org/buildbot/#/builders/4/builds/940, I will ask the > author if they received anything. Though even if they did I'm not sure what > we could conclude from that. > > They did not receive any notifications. And I'm out of ideas at this > point, maybe someone else can spot something. There is > https://github.com/llvm/llvm-zorg/issues/219 which is along the same > lines but no solid evidence that it's the same cause. > > David thanks for reporting this, even if we don't manage to figure it out > it's still better to know about these than not. > > On Thu, 8 Aug 2024 at 16:36, David Spickett <david.spick...@linaro.org> > wrote: > >> I've looked at the build status reporter, which I think is >> https://github.com/llvm/llvm-zorg/blob/ec94a29bfb0c247f0123c4efaf080a9b215d4433/zorg/buildbot/reporters/utils.py#L206. >> I don't see anything wrong with that, and nothing that it removed from >> Buildbot's version of that function looks like it would cause this problem. >> >> If the previous result was somehow still the last green build, this could >> happen. However I don't see how that could be the case as there is a >> finished build on the same worker prior to this one. So even if every other >> worker was still busy, there's at least one previously failed result. My >> guess from the Buildbot code is that prev_build["result"] will be None if >> the build has not finished, so even in this case it would return False and >> not send a message (though I suspect in progress builds aren't part of >> these checks or perhaps it waits for the previous one to finish, not sure). >> >> I do see that the build prior to yours finished later, and some of the >> builds immediately after the green build have some unusual timings where >> earlier builds finish after later builds. This is expected with the 3 >> workers we have, but perhaps could have affected when the results get >> written to the database? Perhaps every build between the last green and >> yours was notified and either no one said anything, or only builds with 1 >> commit were notified and didn't say anything. >> >> The other 1 commit build is >> https://lab.llvm.org/buildbot/#/builders/4/builds/940, I will ask the >> author if they received anything. Though even if they did I'm not sure what >> we could conclude from that. >> >> On Wed, 7 Aug 2024 at 02:12, David Blaikie <dblai...@gmail.com> wrote: >> >>> Any update on this? >>> >>> On Fri, Jul 26, 2024 at 1:28 AM David Spickett < >>> david.spick...@linaro.org> wrote: >>> >>>> I don't see any special configuration for this builder that would make >>>> it do this. >>>> ( >>>> https://github.com/llvm/llvm-zorg/blob/9496234c3e76d33aaae3988903c641fd406e1098/buildbot/osuosl/master/config/builders.py#L527 >>>> ) >>>> >>>> It's unlikely that this build finished before previous jobs because it >>>> only takes an hour and we have several builds in the hours before it. I >>>> thought it may also be the first build on that worker that failed, but this >>>> is also not the case (and if it were, this would be a buildbot bug since >>>> it's supposed to look at the builder as a whole, not specific workers). >>>> >>>> It is a single commit blame list, and I know we started posting results >>>> to GitHub in these cases. Presumably those have to also be green -> red >>>> transitions, but perhaps something slipped through here and because there >>>> was no PR to associate this change with, it sent an email instead? >>>> >>>> Thanks, >>>> David Spickett. >>>> >>>> On Fri, 26 Jul 2024 at 03:03, David Blaikie <dblai...@gmail.com> wrote: >>>> >>>>> ---------- Forwarded message --------- >>>>> From: David Blaikie <dblai...@gmail.com> >>>>> Date: Thu, Jul 25, 2024 at 6:43 PM >>>>> Subject: Re: ☠ Buildbot (LLVM Buildbot): clang-aarch64-sve-vls-2stage - >>>>> failed test (failure) (main) >>>>> To: <llvm.buildmas...@lab.llvm.org> >>>>> Cc: <llvm.buildmas...@lab.llvm.org> >>>>> >>>>> >>>>> This bot seems to be sending fail-mail when it's already red. My >>>>> understanding was that fail mail only went out on green->red >>>>> transitions. >>>>> Is that understanding incorrect? Should this be reconfigured to match >>>>> that >>>>> understanding/not send fail mail when it's already red? >>>>> >>>>> On Thu, Jul 25, 2024 at 4:10 PM <llvm.buildmas...@lab.llvm.org> wrote: >>>>> >>>>> > The Buildbot has detected a failed build on builder >>>>> > clang-aarch64-sve-vls-2stage while building clang. >>>>> > >>>>> > Full details are available at: >>>>> > https://lab.llvm.org/buildbot/#/builders/4/builds/943 >>>>> > >>>>> > Worker for this Build: linaro-g3-02 >>>>> > Blamelist: >>>>> > David Blaikie <dblai...@gmail.com> >>>>> > >>>>> > BUILD FAILED: failed test (failure) >>>>> > >>>>> > Step 18 (test-suite) failure: test (failure) >>>>> > ... >>>>> > [ 24%] Built target >>>>> > gfortran-regression-execute-regression__internal_readwrite_1_f90 >>>>> > cd >>>>> > >>>>> /home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Fortran/gfortran/regression >>>>> > && >>>>> > >>>>> /home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/stage2.install/bin/llvm-size >>>>> > --format=sysv >>>>> > >>>>> /home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Fortran/gfortran/regression/gfortran-regression-execute-regression__internal_readwrite_4_f90 >>>>> > > >>>>> > >>>>> /home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Fortran/gfortran/regression/gfortran-regression-execute-regression__internal_readwrite_4_f90.size >>>>> > make[2]: Leaving directory >>>>> > >>>>> '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build' >>>>> > [ 24%] Built target >>>>> > gfortran-regression-execute-regression__internal_readwrite_4_f90 >>>>> > make[1]: Leaving directory >>>>> > >>>>> '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build' >>>>> > make: *** [Makefile:136: all] Error 2 >>>>> > 2024-07-25 23:09:27 INFO: Testing... >>>>> > 2024-07-25 23:09:27 INFO: Execute: >>>>> > >>>>> /home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/stage1/bin/llvm-lit >>>>> > -v -j 32 >>>>> > >>>>> /home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build >>>>> > -o >>>>> > >>>>> /home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/outputqee5waec.json >>>>> > -- Testing: 9770 tests, 32 workers -- >>>>> > NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_1026.test >>>>> (1 of >>>>> > 9770) >>>>> > ******************** TEST 'test-suite :: >>>>> > Bitcode/simd_ops/simd_ops_test_op_abs_1026.test' FAILED >>>>> ******************** >>>>> > Executable >>>>> > >>>>> '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_1026' >>>>> > is missing >>>>> > ******************** >>>>> > NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_1028.test >>>>> (2 of >>>>> > 9770) >>>>> > ******************** TEST 'test-suite :: >>>>> > Bitcode/simd_ops/simd_ops_test_op_abs_1028.test' FAILED >>>>> ******************** >>>>> > Executable >>>>> > >>>>> '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_1028' >>>>> > is missing >>>>> > ******************** >>>>> > NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_367.test >>>>> (3 of >>>>> > 9770) >>>>> > ******************** TEST 'test-suite :: >>>>> > Bitcode/simd_ops/simd_ops_test_op_abs_367.test' FAILED >>>>> ******************** >>>>> > Executable >>>>> > >>>>> '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_367' >>>>> > is missing >>>>> > ******************** >>>>> > NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_40.test >>>>> (4 of >>>>> > 9770) >>>>> > ******************** TEST 'test-suite :: >>>>> > Bitcode/simd_ops/simd_ops_test_op_abs_40.test' FAILED >>>>> ******************** >>>>> > Executable >>>>> > >>>>> '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_40' >>>>> > is missing >>>>> > ******************** >>>>> > NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_366.test >>>>> (5 of >>>>> > 9770) >>>>> > ******************** TEST 'test-suite :: >>>>> > Bitcode/simd_ops/simd_ops_test_op_abs_366.test' FAILED >>>>> ******************** >>>>> > Executable >>>>> > >>>>> '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_366' >>>>> > is missing >>>>> > ******************** >>>>> > NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_add_1030.test >>>>> (6 of >>>>> > 9770) >>>>> > ******************** TEST 'test-suite :: >>>>> > Bitcode/simd_ops/simd_ops_test_op_add_1030.test' FAILED >>>>> ******************** >>>>> > Executable >>>>> > >>>>> '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_add_1030' >>>>> > is missing >>>>> > ******************** >>>>> > NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_699.test >>>>> (7 of >>>>> > 9770) >>>>> > ******************** TEST 'test-suite :: >>>>> > Bitcode/simd_ops/simd_ops_test_op_abs_699.test' FAILED >>>>> ******************** >>>>> > Executable >>>>> > >>>>> '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_699' >>>>> > is missing >>>>> > ******************** >>>>> > NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_368.test >>>>> (8 of >>>>> > 9770) >>>>> > ******************** TEST 'test-suite :: >>>>> > Bitcode/simd_ops/simd_ops_test_op_abs_368.test' FAILED >>>>> ******************** >>>>> > Executable >>>>> > >>>>> '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_368' >>>>> > is missing >>>>> > ******************** >>>>> > NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_add_1031.test >>>>> (9 of >>>>> > 9770) >>>>> > ******************** TEST 'test-suite :: >>>>> > Bitcode/simd_ops/simd_ops_test_op_add_1031.test' FAILED >>>>> ******************** >>>>> > Executable >>>>> > >>>>> '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_add_1031' >>>>> > is missing >>>>> > ******************** >>>>> > NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_39.test >>>>> (10 of >>>>> > 9770) >>>>> > ******************** TEST 'test-suite :: >>>>> > Bitcode/simd_ops/simd_ops_test_op_abs_39.test' FAILED >>>>> ******************** >>>>> > Executable >>>>> > >>>>> '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_39' >>>>> > is missing >>>>> > ******************** >>>>> > >>>>> > Sincerely, >>>>> > LLVM Buildbot >>>>> > >>>>> > >>>>> _______________________________________________ >>>>> linaro-toolchain mailing list -- linaro-toolchain@lists.linaro.org >>>>> To unsubscribe send an email to >>>>> linaro-toolchain-le...@lists.linaro.org >>>>> >>>> _______________________________________________ linaro-toolchain mailing list -- linaro-toolchain@lists.linaro.org To unsubscribe send an email to linaro-toolchain-le...@lists.linaro.org
