There are some `log.msg(` calls in the code already, those would show up on the master's side. I don't know whether they're being saved or not. Galina do you know about that?
On Fri, 9 Aug 2024 at 18:13, David Blaikie <dblai...@gmail.com> wrote: > Fair - any chance we can put some logging/more verbose output into the > buildbot config so we might know more about how this happens next time? > > On Fri, Aug 9, 2024 at 1:31 AM David Spickett <david.spick...@linaro.org> > wrote: > >> > The other 1 commit build is >> https://lab.llvm.org/buildbot/#/builders/4/builds/940, I will ask the >> author if they received anything. Though even if they did I'm not sure what >> we could conclude from that. >> >> They did not receive any notifications. And I'm out of ideas at this >> point, maybe someone else can spot something. There is >> https://github.com/llvm/llvm-zorg/issues/219 which is along the same >> lines but no solid evidence that it's the same cause. >> >> David thanks for reporting this, even if we don't manage to figure it out >> it's still better to know about these than not. >> >> On Thu, 8 Aug 2024 at 16:36, David Spickett <david.spick...@linaro.org> >> wrote: >> >>> I've looked at the build status reporter, which I think is >>> https://github.com/llvm/llvm-zorg/blob/ec94a29bfb0c247f0123c4efaf080a9b215d4433/zorg/buildbot/reporters/utils.py#L206. >>> I don't see anything wrong with that, and nothing that it removed from >>> Buildbot's version of that function looks like it would cause this problem. >>> >>> If the previous result was somehow still the last green build, this >>> could happen. However I don't see how that could be the case as there is a >>> finished build on the same worker prior to this one. So even if every other >>> worker was still busy, there's at least one previously failed result. My >>> guess from the Buildbot code is that prev_build["result"] will be None if >>> the build has not finished, so even in this case it would return False and >>> not send a message (though I suspect in progress builds aren't part of >>> these checks or perhaps it waits for the previous one to finish, not sure). >>> >>> I do see that the build prior to yours finished later, and some of the >>> builds immediately after the green build have some unusual timings where >>> earlier builds finish after later builds. This is expected with the 3 >>> workers we have, but perhaps could have affected when the results get >>> written to the database? Perhaps every build between the last green and >>> yours was notified and either no one said anything, or only builds with 1 >>> commit were notified and didn't say anything. >>> >>> The other 1 commit build is >>> https://lab.llvm.org/buildbot/#/builders/4/builds/940, I will ask the >>> author if they received anything. Though even if they did I'm not sure what >>> we could conclude from that. >>> >>> On Wed, 7 Aug 2024 at 02:12, David Blaikie <dblai...@gmail.com> wrote: >>> >>>> Any update on this? >>>> >>>> On Fri, Jul 26, 2024 at 1:28 AM David Spickett < >>>> david.spick...@linaro.org> wrote: >>>> >>>>> I don't see any special configuration for this builder that would make >>>>> it do this. >>>>> ( >>>>> https://github.com/llvm/llvm-zorg/blob/9496234c3e76d33aaae3988903c641fd406e1098/buildbot/osuosl/master/config/builders.py#L527 >>>>> ) >>>>> >>>>> It's unlikely that this build finished before previous jobs because it >>>>> only takes an hour and we have several builds in the hours before it. I >>>>> thought it may also be the first build on that worker that failed, but >>>>> this >>>>> is also not the case (and if it were, this would be a buildbot bug since >>>>> it's supposed to look at the builder as a whole, not specific workers). >>>>> >>>>> It is a single commit blame list, and I know we started posting >>>>> results to GitHub in these cases. Presumably those have to also be green >>>>> -> >>>>> red transitions, but perhaps something slipped through here and >>>>> because there was no PR to associate this change with, it sent an email >>>>> instead? >>>>> >>>>> Thanks, >>>>> David Spickett. >>>>> >>>>> On Fri, 26 Jul 2024 at 03:03, David Blaikie <dblai...@gmail.com> >>>>> wrote: >>>>> >>>>>> ---------- Forwarded message --------- >>>>>> From: David Blaikie <dblai...@gmail.com> >>>>>> Date: Thu, Jul 25, 2024 at 6:43 PM >>>>>> Subject: Re: ☠ Buildbot (LLVM Buildbot): clang-aarch64-sve-vls-2stage >>>>>> - >>>>>> failed test (failure) (main) >>>>>> To: <llvm.buildmas...@lab.llvm.org> >>>>>> Cc: <llvm.buildmas...@lab.llvm.org> >>>>>> >>>>>> >>>>>> This bot seems to be sending fail-mail when it's already red. My >>>>>> understanding was that fail mail only went out on green->red >>>>>> transitions. >>>>>> Is that understanding incorrect? Should this be reconfigured to match >>>>>> that >>>>>> understanding/not send fail mail when it's already red? >>>>>> >>>>>> On Thu, Jul 25, 2024 at 4:10 PM <llvm.buildmas...@lab.llvm.org> >>>>>> wrote: >>>>>> >>>>>> > The Buildbot has detected a failed build on builder >>>>>> > clang-aarch64-sve-vls-2stage while building clang. >>>>>> > >>>>>> > Full details are available at: >>>>>> > https://lab.llvm.org/buildbot/#/builders/4/builds/943 >>>>>> > >>>>>> > Worker for this Build: linaro-g3-02 >>>>>> > Blamelist: >>>>>> > David Blaikie <dblai...@gmail.com> >>>>>> > >>>>>> > BUILD FAILED: failed test (failure) >>>>>> > >>>>>> > Step 18 (test-suite) failure: test (failure) >>>>>> > ... >>>>>> > [ 24%] Built target >>>>>> > gfortran-regression-execute-regression__internal_readwrite_1_f90 >>>>>> > cd >>>>>> > >>>>>> /home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Fortran/gfortran/regression >>>>>> > && >>>>>> > >>>>>> /home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/stage2.install/bin/llvm-size >>>>>> > --format=sysv >>>>>> > >>>>>> /home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Fortran/gfortran/regression/gfortran-regression-execute-regression__internal_readwrite_4_f90 >>>>>> > > >>>>>> > >>>>>> /home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Fortran/gfortran/regression/gfortran-regression-execute-regression__internal_readwrite_4_f90.size >>>>>> > make[2]: Leaving directory >>>>>> > >>>>>> '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build' >>>>>> > [ 24%] Built target >>>>>> > gfortran-regression-execute-regression__internal_readwrite_4_f90 >>>>>> > make[1]: Leaving directory >>>>>> > >>>>>> '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build' >>>>>> > make: *** [Makefile:136: all] Error 2 >>>>>> > 2024-07-25 23:09:27 INFO: Testing... >>>>>> > 2024-07-25 23:09:27 INFO: Execute: >>>>>> > >>>>>> /home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/stage1/bin/llvm-lit >>>>>> > -v -j 32 >>>>>> > >>>>>> /home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build >>>>>> > -o >>>>>> > >>>>>> /home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/outputqee5waec.json >>>>>> > -- Testing: 9770 tests, 32 workers -- >>>>>> > NOEXE: test-suite :: >>>>>> Bitcode/simd_ops/simd_ops_test_op_abs_1026.test (1 of >>>>>> > 9770) >>>>>> > ******************** TEST 'test-suite :: >>>>>> > Bitcode/simd_ops/simd_ops_test_op_abs_1026.test' FAILED >>>>>> ******************** >>>>>> > Executable >>>>>> > >>>>>> '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_1026' >>>>>> > is missing >>>>>> > ******************** >>>>>> > NOEXE: test-suite :: >>>>>> Bitcode/simd_ops/simd_ops_test_op_abs_1028.test (2 of >>>>>> > 9770) >>>>>> > ******************** TEST 'test-suite :: >>>>>> > Bitcode/simd_ops/simd_ops_test_op_abs_1028.test' FAILED >>>>>> ******************** >>>>>> > Executable >>>>>> > >>>>>> '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_1028' >>>>>> > is missing >>>>>> > ******************** >>>>>> > NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_367.test >>>>>> (3 of >>>>>> > 9770) >>>>>> > ******************** TEST 'test-suite :: >>>>>> > Bitcode/simd_ops/simd_ops_test_op_abs_367.test' FAILED >>>>>> ******************** >>>>>> > Executable >>>>>> > >>>>>> '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_367' >>>>>> > is missing >>>>>> > ******************** >>>>>> > NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_40.test >>>>>> (4 of >>>>>> > 9770) >>>>>> > ******************** TEST 'test-suite :: >>>>>> > Bitcode/simd_ops/simd_ops_test_op_abs_40.test' FAILED >>>>>> ******************** >>>>>> > Executable >>>>>> > >>>>>> '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_40' >>>>>> > is missing >>>>>> > ******************** >>>>>> > NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_366.test >>>>>> (5 of >>>>>> > 9770) >>>>>> > ******************** TEST 'test-suite :: >>>>>> > Bitcode/simd_ops/simd_ops_test_op_abs_366.test' FAILED >>>>>> ******************** >>>>>> > Executable >>>>>> > >>>>>> '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_366' >>>>>> > is missing >>>>>> > ******************** >>>>>> > NOEXE: test-suite :: >>>>>> Bitcode/simd_ops/simd_ops_test_op_add_1030.test (6 of >>>>>> > 9770) >>>>>> > ******************** TEST 'test-suite :: >>>>>> > Bitcode/simd_ops/simd_ops_test_op_add_1030.test' FAILED >>>>>> ******************** >>>>>> > Executable >>>>>> > >>>>>> '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_add_1030' >>>>>> > is missing >>>>>> > ******************** >>>>>> > NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_699.test >>>>>> (7 of >>>>>> > 9770) >>>>>> > ******************** TEST 'test-suite :: >>>>>> > Bitcode/simd_ops/simd_ops_test_op_abs_699.test' FAILED >>>>>> ******************** >>>>>> > Executable >>>>>> > >>>>>> '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_699' >>>>>> > is missing >>>>>> > ******************** >>>>>> > NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_368.test >>>>>> (8 of >>>>>> > 9770) >>>>>> > ******************** TEST 'test-suite :: >>>>>> > Bitcode/simd_ops/simd_ops_test_op_abs_368.test' FAILED >>>>>> ******************** >>>>>> > Executable >>>>>> > >>>>>> '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_368' >>>>>> > is missing >>>>>> > ******************** >>>>>> > NOEXE: test-suite :: >>>>>> Bitcode/simd_ops/simd_ops_test_op_add_1031.test (9 of >>>>>> > 9770) >>>>>> > ******************** TEST 'test-suite :: >>>>>> > Bitcode/simd_ops/simd_ops_test_op_add_1031.test' FAILED >>>>>> ******************** >>>>>> > Executable >>>>>> > >>>>>> '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_add_1031' >>>>>> > is missing >>>>>> > ******************** >>>>>> > NOEXE: test-suite :: Bitcode/simd_ops/simd_ops_test_op_abs_39.test >>>>>> (10 of >>>>>> > 9770) >>>>>> > ******************** TEST 'test-suite :: >>>>>> > Bitcode/simd_ops/simd_ops_test_op_abs_39.test' FAILED >>>>>> ******************** >>>>>> > Executable >>>>>> > >>>>>> '/home/tcwg-buildbot/worker/clang-aarch64-sve-vls-2stage/test/sandbox/build/Bitcode/simd_ops/simd_ops_test_op_abs_39' >>>>>> > is missing >>>>>> > ******************** >>>>>> > >>>>>> > Sincerely, >>>>>> > LLVM Buildbot >>>>>> > >>>>>> > >>>>>> _______________________________________________ >>>>>> linaro-toolchain mailing list -- linaro-toolchain@lists.linaro.org >>>>>> To unsubscribe send an email to >>>>>> linaro-toolchain-le...@lists.linaro.org >>>>>> >>>>> _______________________________________________ linaro-toolchain mailing list -- linaro-toolchain@lists.linaro.org To unsubscribe send an email to linaro-toolchain-le...@lists.linaro.org
