Yes, it happens every time for android builder. On Tue, Dec 15, 2015 at 1:45 PM, Todd Fiala <todd.fi...@gmail.com> wrote:
> Hmm, yeah it looks like it did the rerun and then after finishing the > rerun, it's just hanging. > > Let's have a look right after r255676 goes through this builder. I hit a > hang in the curses output display due to the recursive taking of a lock on > a lock that was not recursive-enabled. While I would have expected to see > that with the basic results output that this builder here is using when I > was testing earlier, it's possible somehow that we're hitting a path here > that is attempting to recursively take a lock. > > Do you know if it is happening every single time a rerun occurs? > (Hopefully yes?) > > -Todd > > On Tue, Dec 15, 2015 at 1:38 PM, Todd Fiala <todd.fi...@gmail.com> wrote: > >> Yep, I'll have a look! >> >> On Tue, Dec 15, 2015 at 12:43 PM, Ying Chen <chy...@google.com> wrote: >> >>> Hi Todd, >>> >>> It is noticed on lldb android builders that the test_runner didn't exit >>> after rerun, which caused buildbot timeout since the process was hanging >>> for over 20 minutes. >>> Could you please take a look if that's related to your change? >>> >>> Please see the following builds. >>> >>> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4305/steps/test3/logs/stdio >>> >>> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4305/steps/test7/logs/stdio >>> >>> Thanks, >>> Ying >>> >>> On Mon, Dec 14, 2015 at 4:52 PM, Todd Fiala via lldb-dev < >>> lldb-dev@lists.llvm.org> wrote: >>> >>>> And, btw, this shows the rerun logic working (via the >>>> --rerun-all-issues flag): >>>> >>>> time test/dotest.py --executable `pwd`/build/Debug/lldb --threads 24 >>>> --rerun-all-issues >>>> Testing: 416 test suites, 24 threads >>>> 377 out of 416 test suites processed - TestSBTypeTypeClass.py >>>> >>>> Session logs for test failures/errors/unexpected successes will go into >>>> directory '2015-12-14-16_44_28' >>>> Command invoked: test/dotest.py --executable >>>> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24 >>>> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior >>>> -p TestMultithreaded.py >>>> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test >>>> --event-add-entries worker_index=3:int >>>> >>>> Configuration: arch=x86_64 compiler=clang >>>> ---------------------------------------------------------------------- >>>> Collected 8 tests >>>> >>>> lldb_codesign: no identity found >>>> lldb_codesign: no identity found >>>> lldb_codesign: no identity found >>>> lldb_codesign: no identity found >>>> lldb_codesign: no identity found >>>> lldb_codesign: no identity found >>>> lldb_codesign: no identity found >>>> >>>> [TestMultithreaded.py FAILED] >>>> Command invoked: /usr/bin/python test/dotest.py --executable >>>> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24 >>>> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior >>>> -p TestMultithreaded.py >>>> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test >>>> --event-add-entries worker_index=3:int >>>> 396 out of 416 test suites processed - TestMiBreak.py >>>> >>>> Session logs for test failures/errors/unexpected successes will go into >>>> directory '2015-12-14-16_44_28' >>>> Command invoked: test/dotest.py --executable >>>> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24 >>>> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior >>>> -p TestDataFormatterObjC.py >>>> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test >>>> --event-add-entries worker_index=12:int >>>> >>>> Configuration: arch=x86_64 compiler=clang >>>> ---------------------------------------------------------------------- >>>> Collected 26 tests >>>> >>>> >>>> [TestDataFormatterObjC.py FAILED] >>>> Command invoked: /usr/bin/python test/dotest.py --executable >>>> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24 >>>> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior >>>> -p TestDataFormatterObjC.py >>>> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test >>>> --event-add-entries worker_index=12:int >>>> 416 out of 416 test suites processed - TestLldbGdbServer.py >>>> 2 test files marked for rerun >>>> >>>> >>>> Rerunning the following files: >>>> >>>> functionalities/data-formatter/data-formatter-objc/TestDataFormatterObjC.py >>>> api/multithreaded/TestMultithreaded.py >>>> Testing: 2 test suites, 1 thread >>>> 2 out of 2 test suites processed - TestMultithreaded.py >>>> Test rerun complete >>>> >>>> >>>> ============= >>>> Issue Details >>>> ============= >>>> UNEXPECTED SUCCESS: test_symbol_name_dsym >>>> (functionalities/completion/TestCompletion.py) >>>> UNEXPECTED SUCCESS: test_symbol_name_dwarf >>>> (functionalities/completion/TestCompletion.py) >>>> >>>> =================== >>>> Test Result Summary >>>> =================== >>>> Test Methods: 1695 >>>> Reruns: 30 >>>> Success: 1367 >>>> Expected Failure: 90 >>>> Failure: 0 >>>> Error: 0 >>>> Exceptional Exit: 0 >>>> Unexpected Success: 2 >>>> Skip: 236 >>>> Timeout: 0 >>>> Expected Timeout: 0 >>>> >>>> On Mon, Dec 14, 2015 at 4:51 PM, Todd Fiala <todd.fi...@gmail.com> >>>> wrote: >>>> >>>>> And that fixed the rest as well. Thanks, Siva! >>>>> >>>>> -Todd >>>>> >>>>> On Mon, Dec 14, 2015 at 4:44 PM, Todd Fiala <todd.fi...@gmail.com> >>>>> wrote: >>>>> >>>>>> Heh you were skinning the same cat :-) >>>>>> >>>>>> That fixed the one I was just looking at, running the others now. >>>>>> >>>>>> On Mon, Dec 14, 2015 at 4:42 PM, Todd Fiala <todd.fi...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Yep, will try now... (I was just looking at the condition testing >>>>>>> logic since it looks like something isn't quite right there). >>>>>>> >>>>>>> On Mon, Dec 14, 2015 at 4:39 PM, Siva Chandra < >>>>>>> sivachan...@google.com> wrote: >>>>>>> >>>>>>>> Can you try again after taking my change at r255584? >>>>>>>> >>>>>>>> On Mon, Dec 14, 2015 at 4:31 PM, Todd Fiala via lldb-dev >>>>>>>> <lldb-dev@lists.llvm.org> wrote: >>>>>>>> > I'm having some of these blow up. >>>>>>>> > >>>>>>>> > In the case of test/lang/c/typedef/Testtypedef.py, it looks like >>>>>>>> some of the >>>>>>>> > @expected decorators were changed a bit, and perhaps they are not >>>>>>>> pound for >>>>>>>> > pound the same. For example, this test used to really be marked >>>>>>>> XFAIL (via >>>>>>>> > an expectedFailureClang directive), but it looks like the current >>>>>>>> marking of >>>>>>>> > compiler="clang" is either not right or not working, since the >>>>>>>> test is run >>>>>>>> > on OS X and is treated like it is expected to pass. >>>>>>>> > >>>>>>>> > I'm drilling into that a bit more, that's just the first of >>>>>>>> several that >>>>>>>> > fail with these changes on OS X. >>>>>>>> > >>>>>>>> > On Mon, Dec 14, 2015 at 3:03 PM, Zachary Turner < >>>>>>>> ztur...@google.com> wrote: >>>>>>>> >> >>>>>>>> >> I've checked in r255567 which fixes a problem pointed out by >>>>>>>> Siva. It >>>>>>>> >> doesn't sound related to in 255542, but looking at those logs I >>>>>>>> can't really >>>>>>>> >> tell how my CL would be related. If r255567 doesn't fix the >>>>>>>> bots, would >>>>>>>> >> someone mind helping me briefly? r255542 seems pretty >>>>>>>> straightforward, so I >>>>>>>> >> don't see why it would have an effect here. >>>>>>>> >> >>>>>>>> >> On Mon, Dec 14, 2015 at 2:35 PM Todd Fiala <todd.fi...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>> >>>>>>>> >>> Ah yes I see. Thanks, Ying (and Siva! Saw your comments too). >>>>>>>> >>> >>>>>>>> >>> On Mon, Dec 14, 2015 at 2:34 PM, Ying Chen <chy...@google.com> >>>>>>>> wrote: >>>>>>>> >>>> >>>>>>>> >>>> Seems this is the first build that fails, and it only has one >>>>>>>> CL 255542. >>>>>>>> >>>> >>>>>>>> >>>> >>>>>>>> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-cmake/builds/9446 >>>>>>>> >>>> I believe Zachary is looking at that problem. >>>>>>>> >>>> >>>>>>>> >>>> On Mon, Dec 14, 2015 at 2:18 PM, Todd Fiala < >>>>>>>> todd.fi...@gmail.com> >>>>>>>> >>>> wrote: >>>>>>>> >>>>> >>>>>>>> >>>>> I am seeing several failures on the Ubuntu 14.04 testbot, but >>>>>>>> >>>>> unfortunately there are a number of changes that went in at >>>>>>>> the same time on >>>>>>>> >>>>> that build. The failures I'm seeing are not appearing at all >>>>>>>> related to the >>>>>>>> >>>>> test running infrastructure. >>>>>>>> >>>>> >>>>>>>> >>>>> Anybody with a fast Linux system able to take a look to see >>>>>>>> what >>>>>>>> >>>>> exactly is failing there? >>>>>>>> >>>>> >>>>>>>> >>>>> -Todd >>>>>>>> >>>>> >>>>>>>> >>>>> On Mon, Dec 14, 2015 at 1:39 PM, Todd Fiala < >>>>>>>> todd.fi...@gmail.com> >>>>>>>> >>>>> wrote: >>>>>>>> >>>>>> >>>>>>>> >>>>>> Hi all, >>>>>>>> >>>>>> >>>>>>>> >>>>>> I just put in the single-worker, low-load, follow-up test >>>>>>>> run pass in >>>>>>>> >>>>>> r255543. Most of the work for it went in late last week, >>>>>>>> this just mostly >>>>>>>> >>>>>> flips it on. >>>>>>>> >>>>>> >>>>>>>> >>>>>> The feature works like this: >>>>>>>> >>>>>> >>>>>>>> >>>>>> * First test phase works as before: run all tests using >>>>>>>> whatever level >>>>>>>> >>>>>> of concurrency is normally used. (e.g. 8 works on an >>>>>>>> 8-logical-core box). >>>>>>>> >>>>>> >>>>>>>> >>>>>> * Any timeouts, failures, errors, or anything else that >>>>>>>> would have >>>>>>>> >>>>>> caused a test failure is eligible for rerun if either (1) it >>>>>>>> was marked as a >>>>>>>> >>>>>> flakey test via the flakey decorator, or (2) if the >>>>>>>> --rerun-all-issues >>>>>>>> >>>>>> command line flag is provided. >>>>>>>> >>>>>> >>>>>>>> >>>>>> * After the first test phase, if there are any tests that >>>>>>>> met rerun >>>>>>>> >>>>>> eligibility that would have caused a test failure, those get >>>>>>>> run using a >>>>>>>> >>>>>> serial test phase. Their results will overwrite (i.e. >>>>>>>> replace) the previous >>>>>>>> >>>>>> result for the given test method. >>>>>>>> >>>>>> >>>>>>>> >>>>>> The net result should be that tests that were load sensitive >>>>>>>> and >>>>>>>> >>>>>> intermittently fail during the first higher-concurrency test >>>>>>>> phase should >>>>>>>> >>>>>> (in theory) pass in the second, single worker test phase >>>>>>>> when the test suite >>>>>>>> >>>>>> is only using a single worker. This should make the test >>>>>>>> suite generate >>>>>>>> >>>>>> fewer false positives on test failure notification, which >>>>>>>> should make >>>>>>>> >>>>>> continuous integration servers (testbots) much more useful >>>>>>>> in terms of >>>>>>>> >>>>>> generating actionable signals caused by version control >>>>>>>> changes to the lldb >>>>>>>> >>>>>> or related sources. >>>>>>>> >>>>>> >>>>>>>> >>>>>> Please let me know if you see any issues with this when >>>>>>>> running the >>>>>>>> >>>>>> test suite using the default output. I'd like to fix this >>>>>>>> up ASAP. And for >>>>>>>> >>>>>> those interested in the implementation, I'm happy to do >>>>>>>> post-commit >>>>>>>> >>>>>> review/changes as needed to get it in good shape. >>>>>>>> >>>>>> >>>>>>>> >>>>>> I'll be watching the builders now and will address any >>>>>>>> issues as I >>>>>>>> >>>>>> see them. >>>>>>>> >>>>>> >>>>>>>> >>>>>> Thanks! >>>>>>>> >>>>>> -- >>>>>>>> >>>>>> -Todd >>>>>>>> >>>>> >>>>>>>> >>>>> >>>>>>>> >>>>> >>>>>>>> >>>>> >>>>>>>> >>>>> -- >>>>>>>> >>>>> -Todd >>>>>>>> >>>> >>>>>>>> >>>> >>>>>>>> >>> >>>>>>>> >>> >>>>>>>> >>> >>>>>>>> >>> -- >>>>>>>> >>> -Todd >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > -- >>>>>>>> > -Todd >>>>>>>> > >>>>>>>> > _______________________________________________ >>>>>>>> > lldb-dev mailing list >>>>>>>> > lldb-dev@lists.llvm.org >>>>>>>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev >>>>>>>> > >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> -Todd >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> -Todd >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> -Todd >>>>> >>>> >>>> >>>> >>>> -- >>>> -Todd >>>> >>>> _______________________________________________ >>>> lldb-dev mailing list >>>> lldb-dev@lists.llvm.org >>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev >>>> >>>> >>> >> >> >> -- >> -Todd >> > > > > -- > -Todd >
_______________________________________________ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
