[lldb-dev] expected timeouts and reruns

[Todd Fiala via lldb-dev]

Hi all,

If you happen to use --rerun-all-issues to turn on test rerunning (via
single worker thread) for any failed issue, one thing to be aware of is
that expected timeouts that do time out will not be rerun.  They are not
eligible for rerun (at least as of r255641) since they wouldn't cause the
test run to fail.  (I just hit this and thought I'd note it since it might
not be intuitive).  We only rerun things that would cause a test run to
otherwise fail.

One way to deal with this is to not have these tests marked as expected
timeout, but that'll only be useful if the test would be eligible for
rerun.  Right now rerun eligibility is controlled by either directly
marking a test as flakey or using the --rerun-all-issues command line
argument.

I expect we'll likely discuss and finesse this as we go forward.

-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] test rerun phase is in

[Ying Chen via lldb-dev]

Hi Todd,

It is noticed on lldb android builders that the test_runner didn't exit
after rerun, which caused buildbot timeout since the process was hanging
for over 20 minutes.
Could you please take a look if that's related to your change?

Please see the following builds.
http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4305/steps/test3/logs/stdio
http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4305/steps/test7/logs/stdio

Thanks,
Ying

On Mon, Dec 14, 2015 at 4:52 PM, Todd Fiala via lldb-dev <
lldb-dev@lists.llvm.org> wrote:

> And, btw, this shows the rerun logic working (via the --rerun-all-issues
> flag):
>
> time test/dotest.py --executable `pwd`/build/Debug/lldb --threads 24
> --rerun-all-issues
> Testing: 416 test suites, 24 threads
> 377 out of 416 test suites processed - TestSBTypeTypeClass.py
>
> Session logs for test failures/errors/unexpected successes will go into
> directory '2015-12-14-16_44_28'
> Command invoked: test/dotest.py --executable
> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
> -p TestMultithreaded.py
> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
> --event-add-entries worker_index=3:int
>
> Configuration: arch=x86_64 compiler=clang
> --
> Collected 8 tests
>
> lldb_codesign: no identity found
> lldb_codesign: no identity found
> lldb_codesign: no identity found
> lldb_codesign: no identity found
> lldb_codesign: no identity found
> lldb_codesign: no identity found
> lldb_codesign: no identity found
>
> [TestMultithreaded.py FAILED]
> Command invoked: /usr/bin/python test/dotest.py --executable
> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
> -p TestMultithreaded.py
> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
> --event-add-entries worker_index=3:int
> 396 out of 416 test suites processed - TestMiBreak.py
>
> Session logs for test failures/errors/unexpected successes will go into
> directory '2015-12-14-16_44_28'
> Command invoked: test/dotest.py --executable
> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
> -p TestDataFormatterObjC.py
> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
> --event-add-entries worker_index=12:int
>
> Configuration: arch=x86_64 compiler=clang
> --
> Collected 26 tests
>
>
> [TestDataFormatterObjC.py FAILED]
> Command invoked: /usr/bin/python test/dotest.py --executable
> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
> -p TestDataFormatterObjC.py
> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
> --event-add-entries worker_index=12:int
> 416 out of 416 test suites processed - TestLldbGdbServer.py
> 2 test files marked for rerun
>
>
> Rerunning the following files:
>
> functionalities/data-formatter/data-formatter-objc/TestDataFormatterObjC.py
>   api/multithreaded/TestMultithreaded.py
> Testing: 2 test suites, 1 thread
> 2 out of 2 test suites processed - TestMultithreaded.py
> Test rerun complete
>
>
> =
> Issue Details
> =
> UNEXPECTED SUCCESS: test_symbol_name_dsym
> (functionalities/completion/TestCompletion.py)
> UNEXPECTED SUCCESS: test_symbol_name_dwarf
> (functionalities/completion/TestCompletion.py)
>
> ===
> Test Result Summary
> ===
> Test Methods:   1695
> Reruns:   30
> Success:1367
> Expected Failure: 90
> Failure:   0
> Error: 0
> Exceptional Exit:  0
> Unexpected Success:2
> Skip:236
> Timeout:   0
> Expected Timeout:  0
>
> On Mon, Dec 14, 2015 at 4:51 PM, Todd Fiala  wrote:
>
>> And that fixed the rest as well.  Thanks, Siva!
>>
>> -Todd
>>
>> On Mon, Dec 14, 2015 at 4:44 PM, Todd Fiala  wrote:
>>
>>> Heh you were skinning the same cat :-)
>>>
>>> That fixed the one I was just looking at, running the others now.
>>>
>>> On Mon, Dec 14, 2015 at 4:42 PM, Todd Fiala 
>>> wrote:
>>>
 Yep, will try now...  (I was just looking at the condition testing
 logic since it looks like something isn't quite right there).

 On Mon, Dec 14, 2015 at 4:39 PM, Siva Chandra 
 wrote:

> Can you try again after taking my change at r255584?
>
> On Mon, Dec 14, 2015 at 4:31 PM, Todd Fiala via lldb-dev
>  wrote:
> > I'm having some of these blow up.
> >
> > In the case of test/lang/c/typedef/Testtypedef.py, it looks like
> some of the
> > @expected decorators were changed a bit, and perhaps they are not
> pound for
> > pound 

Re: [lldb-dev] test rerun phase is in

[Todd Fiala via lldb-dev]

Yep, I'll have a look!

On Tue, Dec 15, 2015 at 12:43 PM, Ying Chen  wrote:

> Hi Todd,
>
> It is noticed on lldb android builders that the test_runner didn't exit
> after rerun, which caused buildbot timeout since the process was hanging
> for over 20 minutes.
> Could you please take a look if that's related to your change?
>
> Please see the following builds.
>
> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4305/steps/test3/logs/stdio
>
> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4305/steps/test7/logs/stdio
>
> Thanks,
> Ying
>
> On Mon, Dec 14, 2015 at 4:52 PM, Todd Fiala via lldb-dev <
> lldb-dev@lists.llvm.org> wrote:
>
>> And, btw, this shows the rerun logic working (via the --rerun-all-issues
>> flag):
>>
>> time test/dotest.py --executable `pwd`/build/Debug/lldb --threads 24
>> --rerun-all-issues
>> Testing: 416 test suites, 24 threads
>> 377 out of 416 test suites processed - TestSBTypeTypeClass.py
>>
>> Session logs for test failures/errors/unexpected successes will go into
>> directory '2015-12-14-16_44_28'
>> Command invoked: test/dotest.py --executable
>> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
>> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
>> -p TestMultithreaded.py
>> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
>> --event-add-entries worker_index=3:int
>>
>> Configuration: arch=x86_64 compiler=clang
>> --
>> Collected 8 tests
>>
>> lldb_codesign: no identity found
>> lldb_codesign: no identity found
>> lldb_codesign: no identity found
>> lldb_codesign: no identity found
>> lldb_codesign: no identity found
>> lldb_codesign: no identity found
>> lldb_codesign: no identity found
>>
>> [TestMultithreaded.py FAILED]
>> Command invoked: /usr/bin/python test/dotest.py --executable
>> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
>> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
>> -p TestMultithreaded.py
>> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
>> --event-add-entries worker_index=3:int
>> 396 out of 416 test suites processed - TestMiBreak.py
>>
>> Session logs for test failures/errors/unexpected successes will go into
>> directory '2015-12-14-16_44_28'
>> Command invoked: test/dotest.py --executable
>> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
>> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
>> -p TestDataFormatterObjC.py
>> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
>> --event-add-entries worker_index=12:int
>>
>> Configuration: arch=x86_64 compiler=clang
>> --
>> Collected 26 tests
>>
>>
>> [TestDataFormatterObjC.py FAILED]
>> Command invoked: /usr/bin/python test/dotest.py --executable
>> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
>> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
>> -p TestDataFormatterObjC.py
>> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
>> --event-add-entries worker_index=12:int
>> 416 out of 416 test suites processed - TestLldbGdbServer.py
>> 2 test files marked for rerun
>>
>>
>> Rerunning the following files:
>>
>> functionalities/data-formatter/data-formatter-objc/TestDataFormatterObjC.py
>>   api/multithreaded/TestMultithreaded.py
>> Testing: 2 test suites, 1 thread
>> 2 out of 2 test suites processed - TestMultithreaded.py
>> Test rerun complete
>>
>>
>> =
>> Issue Details
>> =
>> UNEXPECTED SUCCESS: test_symbol_name_dsym
>> (functionalities/completion/TestCompletion.py)
>> UNEXPECTED SUCCESS: test_symbol_name_dwarf
>> (functionalities/completion/TestCompletion.py)
>>
>> ===
>> Test Result Summary
>> ===
>> Test Methods:   1695
>> Reruns:   30
>> Success:1367
>> Expected Failure: 90
>> Failure:   0
>> Error: 0
>> Exceptional Exit:  0
>> Unexpected Success:2
>> Skip:236
>> Timeout:   0
>> Expected Timeout:  0
>>
>> On Mon, Dec 14, 2015 at 4:51 PM, Todd Fiala  wrote:
>>
>>> And that fixed the rest as well.  Thanks, Siva!
>>>
>>> -Todd
>>>
>>> On Mon, Dec 14, 2015 at 4:44 PM, Todd Fiala 
>>> wrote:
>>>
 Heh you were skinning the same cat :-)

 That fixed the one I was just looking at, running the others now.

 On Mon, Dec 14, 2015 at 4:42 PM, Todd Fiala 
 wrote:

> Yep, will try now...  (I was just looking at the condition testing
> logic since it looks like something isn't quite right there).
>
> On Mon, Dec 14, 2015 at 4:39 PM, Siva Chandra 
> wrote:
>
>> Can you try again after taking my change at r255584?
>>
>> On Mon, Dec 14, 2015 at 4:31 PM, Todd Fiala via lldb-dev
>>  wrote:
>> > I'

Re: [lldb-dev] test rerun phase is in

[Todd Fiala via lldb-dev]

Hmm, yeah it looks like it did the rerun and then after finishing the
rerun, it's just hanging.

Let's have a look right after r255676 goes through this builder.  I hit a
hang in the curses output display due to the recursive taking of a lock on
a lock that was not recursive-enabled.  While I would have expected to see
that with the basic results output that this builder here is using when I
was testing earlier, it's possible somehow that we're hitting a path here
that is attempting to recursively take a lock.

Do you know if it is happening every single time a rerun occurs?
 (Hopefully yes?)

-Todd

On Tue, Dec 15, 2015 at 1:38 PM, Todd Fiala  wrote:

> Yep, I'll have a look!
>
> On Tue, Dec 15, 2015 at 12:43 PM, Ying Chen  wrote:
>
>> Hi Todd,
>>
>> It is noticed on lldb android builders that the test_runner didn't exit
>> after rerun, which caused buildbot timeout since the process was hanging
>> for over 20 minutes.
>> Could you please take a look if that's related to your change?
>>
>> Please see the following builds.
>>
>> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4305/steps/test3/logs/stdio
>>
>> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4305/steps/test7/logs/stdio
>>
>> Thanks,
>> Ying
>>
>> On Mon, Dec 14, 2015 at 4:52 PM, Todd Fiala via lldb-dev <
>> lldb-dev@lists.llvm.org> wrote:
>>
>>> And, btw, this shows the rerun logic working (via the --rerun-all-issues
>>> flag):
>>>
>>> time test/dotest.py --executable `pwd`/build/Debug/lldb --threads 24
>>> --rerun-all-issues
>>> Testing: 416 test suites, 24 threads
>>> 377 out of 416 test suites processed - TestSBTypeTypeClass.py
>>>
>>> Session logs for test failures/errors/unexpected successes will go into
>>> directory '2015-12-14-16_44_28'
>>> Command invoked: test/dotest.py --executable
>>> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
>>> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
>>> -p TestMultithreaded.py
>>> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
>>> --event-add-entries worker_index=3:int
>>>
>>> Configuration: arch=x86_64 compiler=clang
>>> --
>>> Collected 8 tests
>>>
>>> lldb_codesign: no identity found
>>> lldb_codesign: no identity found
>>> lldb_codesign: no identity found
>>> lldb_codesign: no identity found
>>> lldb_codesign: no identity found
>>> lldb_codesign: no identity found
>>> lldb_codesign: no identity found
>>>
>>> [TestMultithreaded.py FAILED]
>>> Command invoked: /usr/bin/python test/dotest.py --executable
>>> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
>>> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
>>> -p TestMultithreaded.py
>>> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
>>> --event-add-entries worker_index=3:int
>>> 396 out of 416 test suites processed - TestMiBreak.py
>>>
>>> Session logs for test failures/errors/unexpected successes will go into
>>> directory '2015-12-14-16_44_28'
>>> Command invoked: test/dotest.py --executable
>>> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
>>> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
>>> -p TestDataFormatterObjC.py
>>> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
>>> --event-add-entries worker_index=12:int
>>>
>>> Configuration: arch=x86_64 compiler=clang
>>> --
>>> Collected 26 tests
>>>
>>>
>>> [TestDataFormatterObjC.py FAILED]
>>> Command invoked: /usr/bin/python test/dotest.py --executable
>>> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
>>> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
>>> -p TestDataFormatterObjC.py
>>> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
>>> --event-add-entries worker_index=12:int
>>> 416 out of 416 test suites processed - TestLldbGdbServer.py
>>> 2 test files marked for rerun
>>>
>>>
>>> Rerunning the following files:
>>>
>>> functionalities/data-formatter/data-formatter-objc/TestDataFormatterObjC.py
>>>   api/multithreaded/TestMultithreaded.py
>>> Testing: 2 test suites, 1 thread
>>> 2 out of 2 test suites processed - TestMultithreaded.py
>>> Test rerun complete
>>>
>>>
>>> =
>>> Issue Details
>>> =
>>> UNEXPECTED SUCCESS: test_symbol_name_dsym
>>> (functionalities/completion/TestCompletion.py)
>>> UNEXPECTED SUCCESS: test_symbol_name_dwarf
>>> (functionalities/completion/TestCompletion.py)
>>>
>>> ===
>>> Test Result Summary
>>> ===
>>> Test Methods:   1695
>>> Reruns:   30
>>> Success:1367
>>> Expected Failure: 90
>>> Failure:   0
>>> Error: 0
>>> Exceptional Exit:  0
>>> Unexpected Success:2
>>> Skip:236
>>> Timeout:   0
>>> Expected

Re: [lldb-dev] test rerun phase is in

[Ying Chen via lldb-dev]

Yes, it happens every time for android builder.

On Tue, Dec 15, 2015 at 1:45 PM, Todd Fiala  wrote:

> Hmm, yeah it looks like it did the rerun and then after finishing the
> rerun, it's just hanging.
>
> Let's have a look right after r255676 goes through this builder.  I hit a
> hang in the curses output display due to the recursive taking of a lock on
> a lock that was not recursive-enabled.  While I would have expected to see
> that with the basic results output that this builder here is using when I
> was testing earlier, it's possible somehow that we're hitting a path here
> that is attempting to recursively take a lock.
>
> Do you know if it is happening every single time a rerun occurs?
>  (Hopefully yes?)
>
> -Todd
>
> On Tue, Dec 15, 2015 at 1:38 PM, Todd Fiala  wrote:
>
>> Yep, I'll have a look!
>>
>> On Tue, Dec 15, 2015 at 12:43 PM, Ying Chen  wrote:
>>
>>> Hi Todd,
>>>
>>> It is noticed on lldb android builders that the test_runner didn't exit
>>> after rerun, which caused buildbot timeout since the process was hanging
>>> for over 20 minutes.
>>> Could you please take a look if that's related to your change?
>>>
>>> Please see the following builds.
>>>
>>> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4305/steps/test3/logs/stdio
>>>
>>> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4305/steps/test7/logs/stdio
>>>
>>> Thanks,
>>> Ying
>>>
>>> On Mon, Dec 14, 2015 at 4:52 PM, Todd Fiala via lldb-dev <
>>> lldb-dev@lists.llvm.org> wrote:
>>>
 And, btw, this shows the rerun logic working (via the
 --rerun-all-issues flag):

 time test/dotest.py --executable `pwd`/build/Debug/lldb --threads 24
 --rerun-all-issues
 Testing: 416 test suites, 24 threads
 377 out of 416 test suites processed - TestSBTypeTypeClass.py

 Session logs for test failures/errors/unexpected successes will go into
 directory '2015-12-14-16_44_28'
 Command invoked: test/dotest.py --executable
 /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
 --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
 -p TestMultithreaded.py
 /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
 --event-add-entries worker_index=3:int

 Configuration: arch=x86_64 compiler=clang
 --
 Collected 8 tests

 lldb_codesign: no identity found
 lldb_codesign: no identity found
 lldb_codesign: no identity found
 lldb_codesign: no identity found
 lldb_codesign: no identity found
 lldb_codesign: no identity found
 lldb_codesign: no identity found

 [TestMultithreaded.py FAILED]
 Command invoked: /usr/bin/python test/dotest.py --executable
 /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
 --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
 -p TestMultithreaded.py
 /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
 --event-add-entries worker_index=3:int
 396 out of 416 test suites processed - TestMiBreak.py

 Session logs for test failures/errors/unexpected successes will go into
 directory '2015-12-14-16_44_28'
 Command invoked: test/dotest.py --executable
 /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
 --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
 -p TestDataFormatterObjC.py
 /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
 --event-add-entries worker_index=12:int

 Configuration: arch=x86_64 compiler=clang
 --
 Collected 26 tests


 [TestDataFormatterObjC.py FAILED]
 Command invoked: /usr/bin/python test/dotest.py --executable
 /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
 --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
 -p TestDataFormatterObjC.py
 /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
 --event-add-entries worker_index=12:int
 416 out of 416 test suites processed - TestLldbGdbServer.py
 2 test files marked for rerun


 Rerunning the following files:

 functionalities/data-formatter/data-formatter-objc/TestDataFormatterObjC.py
   api/multithreaded/TestMultithreaded.py
 Testing: 2 test suites, 1 thread
 2 out of 2 test suites processed - TestMultithreaded.py
 Test rerun complete


 =
 Issue Details
 =
 UNEXPECTED SUCCESS: test_symbol_name_dsym
 (functionalities/completion/TestCompletion.py)
 UNEXPECTED SUCCESS: test_symbol_name_dwarf
 (functionalities/completion/TestCompletion.py)

 ===
 Test Result Summary
 ===
 Test Methods:   1695
 Reruns:   30
 

Re: [lldb-dev] test rerun phase is in

[Todd Fiala via lldb-dev]

Hey Ying,

Do you mind if we clear the android builder queue to get a build with
r255676 in it?  There are what looks like at least 3 or 4 builds between
now and then, and with timeouts it may take several hours.

-Todd

On Tue, Dec 15, 2015 at 1:50 PM, Ying Chen  wrote:

> Yes, it happens every time for android builder.
>
> On Tue, Dec 15, 2015 at 1:45 PM, Todd Fiala  wrote:
>
>> Hmm, yeah it looks like it did the rerun and then after finishing the
>> rerun, it's just hanging.
>>
>> Let's have a look right after r255676 goes through this builder.  I hit a
>> hang in the curses output display due to the recursive taking of a lock on
>> a lock that was not recursive-enabled.  While I would have expected to see
>> that with the basic results output that this builder here is using when I
>> was testing earlier, it's possible somehow that we're hitting a path here
>> that is attempting to recursively take a lock.
>>
>> Do you know if it is happening every single time a rerun occurs?
>>  (Hopefully yes?)
>>
>> -Todd
>>
>> On Tue, Dec 15, 2015 at 1:38 PM, Todd Fiala  wrote:
>>
>>> Yep, I'll have a look!
>>>
>>> On Tue, Dec 15, 2015 at 12:43 PM, Ying Chen  wrote:
>>>
 Hi Todd,

 It is noticed on lldb android builders that the test_runner didn't exit
 after rerun, which caused buildbot timeout since the process was hanging
 for over 20 minutes.
 Could you please take a look if that's related to your change?

 Please see the following builds.

 http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4305/steps/test3/logs/stdio

 http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4305/steps/test7/logs/stdio

 Thanks,
 Ying

 On Mon, Dec 14, 2015 at 4:52 PM, Todd Fiala via lldb-dev <
 lldb-dev@lists.llvm.org> wrote:

> And, btw, this shows the rerun logic working (via the
> --rerun-all-issues flag):
>
> time test/dotest.py --executable `pwd`/build/Debug/lldb --threads 24
> --rerun-all-issues
> Testing: 416 test suites, 24 threads
> 377 out of 416 test suites processed - TestSBTypeTypeClass.py
>
> Session logs for test failures/errors/unexpected successes will go
> into directory '2015-12-14-16_44_28'
> Command invoked: test/dotest.py --executable
> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
> -p TestMultithreaded.py
> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
> --event-add-entries worker_index=3:int
>
> Configuration: arch=x86_64 compiler=clang
> --
> Collected 8 tests
>
> lldb_codesign: no identity found
> lldb_codesign: no identity found
> lldb_codesign: no identity found
> lldb_codesign: no identity found
> lldb_codesign: no identity found
> lldb_codesign: no identity found
> lldb_codesign: no identity found
>
> [TestMultithreaded.py FAILED]
> Command invoked: /usr/bin/python test/dotest.py --executable
> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
> -p TestMultithreaded.py
> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
> --event-add-entries worker_index=3:int
> 396 out of 416 test suites processed - TestMiBreak.py
>
> Session logs for test failures/errors/unexpected successes will go
> into directory '2015-12-14-16_44_28'
> Command invoked: test/dotest.py --executable
> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
> -p TestDataFormatterObjC.py
> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
> --event-add-entries worker_index=12:int
>
> Configuration: arch=x86_64 compiler=clang
> --
> Collected 26 tests
>
>
> [TestDataFormatterObjC.py FAILED]
> Command invoked: /usr/bin/python test/dotest.py --executable
> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
> -p TestDataFormatterObjC.py
> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
> --event-add-entries worker_index=12:int
> 416 out of 416 test suites processed - TestLldbGdbServer.py
> 2 test files marked for rerun
>
>
> Rerunning the following files:
>
> functionalities/data-formatter/data-formatter-objc/TestDataFormatterObjC.py
>   api/multithreaded/TestMultithreaded.py
> Testing: 2 test suites, 1 thread
> 2 out of 2 test suites processed - TestMultithreaded.py
> Test rerun complete
>
>
> 

Re: [lldb-dev] test rerun phase is in

[Ying Chen via lldb-dev]

Sure. Please go ahead to do that.
BTW, the pending builds should be merged into one build once current build
is done.

On Tue, Dec 15, 2015 at 2:12 PM, Todd Fiala  wrote:

> Hey Ying,
>
> Do you mind if we clear the android builder queue to get a build with
> r255676 in it?  There are what looks like at least 3 or 4 builds between
> now and then, and with timeouts it may take several hours.
>
> -Todd
>
> On Tue, Dec 15, 2015 at 1:50 PM, Ying Chen  wrote:
>
>> Yes, it happens every time for android builder.
>>
>> On Tue, Dec 15, 2015 at 1:45 PM, Todd Fiala  wrote:
>>
>>> Hmm, yeah it looks like it did the rerun and then after finishing the
>>> rerun, it's just hanging.
>>>
>>> Let's have a look right after r255676 goes through this builder.  I hit
>>> a hang in the curses output display due to the recursive taking of a lock
>>> on a lock that was not recursive-enabled.  While I would have expected to
>>> see that with the basic results output that this builder here is using when
>>> I was testing earlier, it's possible somehow that we're hitting a path here
>>> that is attempting to recursively take a lock.
>>>
>>> Do you know if it is happening every single time a rerun occurs?
>>>  (Hopefully yes?)
>>>
>>> -Todd
>>>
>>> On Tue, Dec 15, 2015 at 1:38 PM, Todd Fiala 
>>> wrote:
>>>
 Yep, I'll have a look!

 On Tue, Dec 15, 2015 at 12:43 PM, Ying Chen  wrote:

> Hi Todd,
>
> It is noticed on lldb android builders that the test_runner didn't
> exit after rerun, which caused buildbot timeout since the process was
> hanging for over 20 minutes.
> Could you please take a look if that's related to your change?
>
> Please see the following builds.
>
> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4305/steps/test3/logs/stdio
>
> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4305/steps/test7/logs/stdio
>
> Thanks,
> Ying
>
> On Mon, Dec 14, 2015 at 4:52 PM, Todd Fiala via lldb-dev <
> lldb-dev@lists.llvm.org> wrote:
>
>> And, btw, this shows the rerun logic working (via the
>> --rerun-all-issues flag):
>>
>> time test/dotest.py --executable `pwd`/build/Debug/lldb --threads 24
>> --rerun-all-issues
>> Testing: 416 test suites, 24 threads
>> 377 out of 416 test suites processed - TestSBTypeTypeClass.py
>>
>> Session logs for test failures/errors/unexpected successes will go
>> into directory '2015-12-14-16_44_28'
>> Command invoked: test/dotest.py --executable
>> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
>> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
>> -p TestMultithreaded.py
>> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
>> --event-add-entries worker_index=3:int
>>
>> Configuration: arch=x86_64 compiler=clang
>> --
>> Collected 8 tests
>>
>> lldb_codesign: no identity found
>> lldb_codesign: no identity found
>> lldb_codesign: no identity found
>> lldb_codesign: no identity found
>> lldb_codesign: no identity found
>> lldb_codesign: no identity found
>> lldb_codesign: no identity found
>>
>> [TestMultithreaded.py FAILED]
>> Command invoked: /usr/bin/python test/dotest.py --executable
>> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
>> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
>> -p TestMultithreaded.py
>> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
>> --event-add-entries worker_index=3:int
>> 396 out of 416 test suites processed - TestMiBreak.py
>>
>> Session logs for test failures/errors/unexpected successes will go
>> into directory '2015-12-14-16_44_28'
>> Command invoked: test/dotest.py --executable
>> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
>> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
>> -p TestDataFormatterObjC.py
>> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
>> --event-add-entries worker_index=12:int
>>
>> Configuration: arch=x86_64 compiler=clang
>> --
>> Collected 26 tests
>>
>>
>> [TestDataFormatterObjC.py FAILED]
>> Command invoked: /usr/bin/python test/dotest.py --executable
>> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
>> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
>> -p TestDataFormatterObjC.py
>> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
>> --event-add-entries worker_index=12:int
>> 416 out of 416 test suites processed - TestLldbGdbServer.py
>> 2 test files marked for rerun
>>
>>
>> Rerunning the f

Re: [lldb-dev] test rerun phase is in

[Todd Fiala via lldb-dev]

Okay cool.  Will do.

On Tue, Dec 15, 2015 at 2:22 PM, Ying Chen  wrote:

> Sure. Please go ahead to do that.
> BTW, the pending builds should be merged into one build once current build
> is done.
>
> On Tue, Dec 15, 2015 at 2:12 PM, Todd Fiala  wrote:
>
>> Hey Ying,
>>
>> Do you mind if we clear the android builder queue to get a build with
>> r255676 in it?  There are what looks like at least 3 or 4 builds between
>> now and then, and with timeouts it may take several hours.
>>
>> -Todd
>>
>> On Tue, Dec 15, 2015 at 1:50 PM, Ying Chen  wrote:
>>
>>> Yes, it happens every time for android builder.
>>>
>>> On Tue, Dec 15, 2015 at 1:45 PM, Todd Fiala 
>>> wrote:
>>>
 Hmm, yeah it looks like it did the rerun and then after finishing the
 rerun, it's just hanging.

 Let's have a look right after r255676 goes through this builder.  I hit
 a hang in the curses output display due to the recursive taking of a lock
 on a lock that was not recursive-enabled.  While I would have expected to
 see that with the basic results output that this builder here is using when
 I was testing earlier, it's possible somehow that we're hitting a path here
 that is attempting to recursively take a lock.

 Do you know if it is happening every single time a rerun occurs?
  (Hopefully yes?)

 -Todd

 On Tue, Dec 15, 2015 at 1:38 PM, Todd Fiala 
 wrote:

> Yep, I'll have a look!
>
> On Tue, Dec 15, 2015 at 12:43 PM, Ying Chen  wrote:
>
>> Hi Todd,
>>
>> It is noticed on lldb android builders that the test_runner didn't
>> exit after rerun, which caused buildbot timeout since the process was
>> hanging for over 20 minutes.
>> Could you please take a look if that's related to your change?
>>
>> Please see the following builds.
>>
>> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4305/steps/test3/logs/stdio
>>
>> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4305/steps/test7/logs/stdio
>>
>> Thanks,
>> Ying
>>
>> On Mon, Dec 14, 2015 at 4:52 PM, Todd Fiala via lldb-dev <
>> lldb-dev@lists.llvm.org> wrote:
>>
>>> And, btw, this shows the rerun logic working (via the
>>> --rerun-all-issues flag):
>>>
>>> time test/dotest.py --executable `pwd`/build/Debug/lldb --threads 24
>>> --rerun-all-issues
>>> Testing: 416 test suites, 24 threads
>>> 377 out of 416 test suites processed - TestSBTypeTypeClass.py
>>>
>>> Session logs for test failures/errors/unexpected successes will go
>>> into directory '2015-12-14-16_44_28'
>>> Command invoked: test/dotest.py --executable
>>> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
>>> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 
>>> --inferior
>>> -p TestMultithreaded.py
>>> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
>>> --event-add-entries worker_index=3:int
>>>
>>> Configuration: arch=x86_64 compiler=clang
>>>
>>> --
>>> Collected 8 tests
>>>
>>> lldb_codesign: no identity found
>>> lldb_codesign: no identity found
>>> lldb_codesign: no identity found
>>> lldb_codesign: no identity found
>>> lldb_codesign: no identity found
>>> lldb_codesign: no identity found
>>> lldb_codesign: no identity found
>>>
>>> [TestMultithreaded.py FAILED]
>>> Command invoked: /usr/bin/python test/dotest.py --executable
>>> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
>>> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 
>>> --inferior
>>> -p TestMultithreaded.py
>>> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
>>> --event-add-entries worker_index=3:int
>>> 396 out of 416 test suites processed - TestMiBreak.py
>>>
>>> Session logs for test failures/errors/unexpected successes will go
>>> into directory '2015-12-14-16_44_28'
>>> Command invoked: test/dotest.py --executable
>>> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
>>> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 
>>> --inferior
>>> -p TestDataFormatterObjC.py
>>> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
>>> --event-add-entries worker_index=12:int
>>>
>>> Configuration: arch=x86_64 compiler=clang
>>>
>>> --
>>> Collected 26 tests
>>>
>>>
>>> [TestDataFormatterObjC.py FAILED]
>>> Command invoked: /usr/bin/python test/dotest.py --executable
>>> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
>>> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 
>>> --inferior
>>> -p TestDataFormatterObjC.py
>

Re: [lldb-dev] test rerun phase is in

[Todd Fiala via lldb-dev]

Build >= #4310 is what I'll be watching.


On Tue, Dec 15, 2015 at 2:30 PM, Todd Fiala  wrote:

> Okay cool.  Will do.
>
> On Tue, Dec 15, 2015 at 2:22 PM, Ying Chen  wrote:
>
>> Sure. Please go ahead to do that.
>> BTW, the pending builds should be merged into one build once current
>> build is done.
>>
>> On Tue, Dec 15, 2015 at 2:12 PM, Todd Fiala  wrote:
>>
>>> Hey Ying,
>>>
>>> Do you mind if we clear the android builder queue to get a build with
>>> r255676 in it?  There are what looks like at least 3 or 4 builds between
>>> now and then, and with timeouts it may take several hours.
>>>
>>> -Todd
>>>
>>> On Tue, Dec 15, 2015 at 1:50 PM, Ying Chen  wrote:
>>>
 Yes, it happens every time for android builder.

 On Tue, Dec 15, 2015 at 1:45 PM, Todd Fiala 
 wrote:

> Hmm, yeah it looks like it did the rerun and then after finishing the
> rerun, it's just hanging.
>
> Let's have a look right after r255676 goes through this builder.  I
> hit a hang in the curses output display due to the recursive taking of a
> lock on a lock that was not recursive-enabled.  While I would have 
> expected
> to see that with the basic results output that this builder here is using
> when I was testing earlier, it's possible somehow that we're hitting a 
> path
> here that is attempting to recursively take a lock.
>
> Do you know if it is happening every single time a rerun occurs?
>  (Hopefully yes?)
>
> -Todd
>
> On Tue, Dec 15, 2015 at 1:38 PM, Todd Fiala 
> wrote:
>
>> Yep, I'll have a look!
>>
>> On Tue, Dec 15, 2015 at 12:43 PM, Ying Chen 
>> wrote:
>>
>>> Hi Todd,
>>>
>>> It is noticed on lldb android builders that the test_runner didn't
>>> exit after rerun, which caused buildbot timeout since the process was
>>> hanging for over 20 minutes.
>>> Could you please take a look if that's related to your change?
>>>
>>> Please see the following builds.
>>>
>>> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4305/steps/test3/logs/stdio
>>>
>>> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4305/steps/test7/logs/stdio
>>>
>>> Thanks,
>>> Ying
>>>
>>> On Mon, Dec 14, 2015 at 4:52 PM, Todd Fiala via lldb-dev <
>>> lldb-dev@lists.llvm.org> wrote:
>>>
 And, btw, this shows the rerun logic working (via the
 --rerun-all-issues flag):

 time test/dotest.py --executable `pwd`/build/Debug/lldb --threads
 24 --rerun-all-issues
 Testing: 416 test suites, 24 threads
 377 out of 416 test suites processed - TestSBTypeTypeClass.py

 Session logs for test failures/errors/unexpected successes will go
 into directory '2015-12-14-16_44_28'
 Command invoked: test/dotest.py --executable
 /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
 --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 
 --inferior
 -p TestMultithreaded.py
 /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
 --event-add-entries worker_index=3:int

 Configuration: arch=x86_64 compiler=clang

 --
 Collected 8 tests

 lldb_codesign: no identity found
 lldb_codesign: no identity found
 lldb_codesign: no identity found
 lldb_codesign: no identity found
 lldb_codesign: no identity found
 lldb_codesign: no identity found
 lldb_codesign: no identity found

 [TestMultithreaded.py FAILED]
 Command invoked: /usr/bin/python test/dotest.py --executable
 /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
 --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 
 --inferior
 -p TestMultithreaded.py
 /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
 --event-add-entries worker_index=3:int
 396 out of 416 test suites processed - TestMiBreak.py

 Session logs for test failures/errors/unexpected successes will go
 into directory '2015-12-14-16_44_28'
 Command invoked: test/dotest.py --executable
 /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
 --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 
 --inferior
 -p TestDataFormatterObjC.py
 /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
 --event-add-entries worker_index=12:int

 Configuration: arch=x86_64 compiler=clang

 --
 Collected 26 tests


 [TestDataFormatterObjC.py FAILED]
 Command invoked: /u

Re: [lldb-dev] test rerun phase is in

[Todd Fiala via lldb-dev]

#4310 failed for some other reason.

#4311 looks like it might be stuck in the test3 phase but it is showing
less output than it had before (maybe because it hasn't timed out yet).

I'm usually running with --rerun-all-issues, but I can force similar
failures to what this bot is seeing when I crank up the load over there on
an OS X box.  I'm doing that now and I'm omitting the --rerun-all-issues
flag, which should be close to how the android testbot is running.
Hopefully I can force it to fail here.

If not, I'll temporarily disable the rerun unless --rerun-all-issues until
we can figure out what's causing the stall.

BTW - how many cores are present on that box?  That will help me figure out
which runner is being used for the main phase.

Thanks!

-Todd

On Tue, Dec 15, 2015 at 2:34 PM, Todd Fiala  wrote:

> Build >= #4310 is what I'll be watching.
>
>
> On Tue, Dec 15, 2015 at 2:30 PM, Todd Fiala  wrote:
>
>> Okay cool.  Will do.
>>
>> On Tue, Dec 15, 2015 at 2:22 PM, Ying Chen  wrote:
>>
>>> Sure. Please go ahead to do that.
>>> BTW, the pending builds should be merged into one build once current
>>> build is done.
>>>
>>> On Tue, Dec 15, 2015 at 2:12 PM, Todd Fiala 
>>> wrote:
>>>
 Hey Ying,

 Do you mind if we clear the android builder queue to get a build with
 r255676 in it?  There are what looks like at least 3 or 4 builds between
 now and then, and with timeouts it may take several hours.

 -Todd

 On Tue, Dec 15, 2015 at 1:50 PM, Ying Chen  wrote:

> Yes, it happens every time for android builder.
>
> On Tue, Dec 15, 2015 at 1:45 PM, Todd Fiala 
> wrote:
>
>> Hmm, yeah it looks like it did the rerun and then after finishing the
>> rerun, it's just hanging.
>>
>> Let's have a look right after r255676 goes through this builder.  I
>> hit a hang in the curses output display due to the recursive taking of a
>> lock on a lock that was not recursive-enabled.  While I would have 
>> expected
>> to see that with the basic results output that this builder here is using
>> when I was testing earlier, it's possible somehow that we're hitting a 
>> path
>> here that is attempting to recursively take a lock.
>>
>> Do you know if it is happening every single time a rerun occurs?
>>  (Hopefully yes?)
>>
>> -Todd
>>
>> On Tue, Dec 15, 2015 at 1:38 PM, Todd Fiala 
>> wrote:
>>
>>> Yep, I'll have a look!
>>>
>>> On Tue, Dec 15, 2015 at 12:43 PM, Ying Chen 
>>> wrote:
>>>
 Hi Todd,

 It is noticed on lldb android builders that the test_runner didn't
 exit after rerun, which caused buildbot timeout since the process was
 hanging for over 20 minutes.
 Could you please take a look if that's related to your change?

 Please see the following builds.

 http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4305/steps/test3/logs/stdio

 http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4305/steps/test7/logs/stdio

 Thanks,
 Ying

 On Mon, Dec 14, 2015 at 4:52 PM, Todd Fiala via lldb-dev <
 lldb-dev@lists.llvm.org> wrote:

> And, btw, this shows the rerun logic working (via the
> --rerun-all-issues flag):
>
> time test/dotest.py --executable `pwd`/build/Debug/lldb --threads
> 24 --rerun-all-issues
> Testing: 416 test suites, 24 threads
> 377 out of 416 test suites processed - TestSBTypeTypeClass.py
>
> Session logs for test failures/errors/unexpected successes will go
> into directory '2015-12-14-16_44_28'
> Command invoked: test/dotest.py --executable
> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 
> --inferior
> -p TestMultithreaded.py
> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
> --event-add-entries worker_index=3:int
>
> Configuration: arch=x86_64 compiler=clang
>
> --
> Collected 8 tests
>
> lldb_codesign: no identity found
> lldb_codesign: no identity found
> lldb_codesign: no identity found
> lldb_codesign: no identity found
> lldb_codesign: no identity found
> lldb_codesign: no identity found
> lldb_codesign: no identity found
>
> [TestMultithreaded.py FAILED]
> Command invoked: /usr/bin/python test/dotest.py --executable
> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 
> --inferior
> -p TestMultithreaded.py
> /Users/tfi

Re: [lldb-dev] test rerun phase is in

[Todd Fiala via lldb-dev]

Hey Ying,

I'm going to check in something that stops the rerun logic when both (1) -A
aarch64 is specified and (2) --rerun-all-issues is not specified.

That'll give me some time to drill into what's getting stuck on the android
buildbot.

-Todd

On Tue, Dec 15, 2015 at 3:36 PM, Todd Fiala  wrote:

> #4310 failed for some other reason.
>
> #4311 looks like it might be stuck in the test3 phase but it is showing
> less output than it had before (maybe because it hasn't timed out yet).
>
> I'm usually running with --rerun-all-issues, but I can force similar
> failures to what this bot is seeing when I crank up the load over there on
> an OS X box.  I'm doing that now and I'm omitting the --rerun-all-issues
> flag, which should be close to how the android testbot is running.
> Hopefully I can force it to fail here.
>
> If not, I'll temporarily disable the rerun unless --rerun-all-issues until
> we can figure out what's causing the stall.
>
> BTW - how many cores are present on that box?  That will help me figure
> out which runner is being used for the main phase.
>
> Thanks!
>
> -Todd
>
> On Tue, Dec 15, 2015 at 2:34 PM, Todd Fiala  wrote:
>
>> Build >= #4310 is what I'll be watching.
>>
>>
>> On Tue, Dec 15, 2015 at 2:30 PM, Todd Fiala  wrote:
>>
>>> Okay cool.  Will do.
>>>
>>> On Tue, Dec 15, 2015 at 2:22 PM, Ying Chen  wrote:
>>>
 Sure. Please go ahead to do that.
 BTW, the pending builds should be merged into one build once current
 build is done.

 On Tue, Dec 15, 2015 at 2:12 PM, Todd Fiala 
 wrote:

> Hey Ying,
>
> Do you mind if we clear the android builder queue to get a build with
> r255676 in it?  There are what looks like at least 3 or 4 builds between
> now and then, and with timeouts it may take several hours.
>
> -Todd
>
> On Tue, Dec 15, 2015 at 1:50 PM, Ying Chen  wrote:
>
>> Yes, it happens every time for android builder.
>>
>> On Tue, Dec 15, 2015 at 1:45 PM, Todd Fiala 
>> wrote:
>>
>>> Hmm, yeah it looks like it did the rerun and then after finishing
>>> the rerun, it's just hanging.
>>>
>>> Let's have a look right after r255676 goes through this builder.  I
>>> hit a hang in the curses output display due to the recursive taking of a
>>> lock on a lock that was not recursive-enabled.  While I would have 
>>> expected
>>> to see that with the basic results output that this builder here is 
>>> using
>>> when I was testing earlier, it's possible somehow that we're hitting a 
>>> path
>>> here that is attempting to recursively take a lock.
>>>
>>> Do you know if it is happening every single time a rerun occurs?
>>>  (Hopefully yes?)
>>>
>>> -Todd
>>>
>>> On Tue, Dec 15, 2015 at 1:38 PM, Todd Fiala 
>>> wrote:
>>>
 Yep, I'll have a look!

 On Tue, Dec 15, 2015 at 12:43 PM, Ying Chen 
 wrote:

> Hi Todd,
>
> It is noticed on lldb android builders that the test_runner didn't
> exit after rerun, which caused buildbot timeout since the process was
> hanging for over 20 minutes.
> Could you please take a look if that's related to your change?
>
> Please see the following builds.
>
> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4305/steps/test3/logs/stdio
>
> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4305/steps/test7/logs/stdio
>
> Thanks,
> Ying
>
> On Mon, Dec 14, 2015 at 4:52 PM, Todd Fiala via lldb-dev <
> lldb-dev@lists.llvm.org> wrote:
>
>> And, btw, this shows the rerun logic working (via the
>> --rerun-all-issues flag):
>>
>> time test/dotest.py --executable `pwd`/build/Debug/lldb --threads
>> 24 --rerun-all-issues
>> Testing: 416 test suites, 24 threads
>> 377 out of 416 test suites processed - TestSBTypeTypeClass.py
>>
>> Session logs for test failures/errors/unexpected successes will
>> go into directory '2015-12-14-16_44_28'
>> Command invoked: test/dotest.py --executable
>> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
>> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 
>> --inferior
>> -p TestMultithreaded.py
>> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
>> --event-add-entries worker_index=3:int
>>
>> Configuration: arch=x86_64 compiler=clang
>>
>> --
>> Collected 8 tests
>>
>> lldb_codesign: no identity found
>> lldb_codesign: no identity found
>> lldb_codesign: no identity found
>> lldb_codesign: no identity found
>> lldb_codes

Re: [lldb-dev] Problem with dotest_channels.py

[Adrian McCarthy via lldb-dev]

With Todd's change, I was getting a Ninja crash.  Zach and I replaced the
returns Todd added with raises, in order to propagate the exception up the
stack, and that avoids the Ninja crash, so I'll check that in in a moment.

In the mean time, here's the error message we got out of it.

155 out of 416 test suites processed - TestBacktraceAll.py
INFO: received socket error when reading data from test inferior:
[Errno 10054] An existing connection was forcibly closed by the remote host
error: uncaptured python exception, closing channel
 (:[Errno 10054] An existing
connection was forcibly closed by the remote host
[D:\Python_for_lldb\x86\lib\asyncore.py|read|83]
[D:\Python_for_lldb\x86\lib\asyncore.py|handle_read_event|449]
[D:\src\llvm\llvm\tools\lldb\packages\Python\lldbsuite\test\dotest_channels.py|handle_read|137]
[D:\Python_for_lldb\x86\lib\asyncore.py|recv|387])
175 out of 416 test suites processed - TestNoSuchArch.py

On Mon, Dec 14, 2015 at 3:58 PM, Todd Fiala via lldb-dev <
lldb-dev@lists.llvm.org> wrote:

> Hey Zachary,
>
> I just put in:
> r255581
>
> which should hopefully:
> (1) catch the exception you see there,
> (2) handle it gracefully in the common and to-be-expected case of the test
> inferior going down hard, and
> (3) print out an error if anything else unexpected is happening here.
>
> Let me know if you get any more info with it.  Thanks!
>
> -Todd
>
> On Mon, Dec 14, 2015 at 2:16 PM, Todd Fiala  wrote:
>
>> Yeah that's a messed up exception scenario that is hard to read.  I'll
>> figure something out when I repro it.  One side is closing before the other
>> is expecting it, but likely in a way we need to expect.
>>
>> I think it is ugly-ified because it is coming from some kind of worker
>> thread within async-core.
>>
>> I will get something in to help it today.  The first bit may be just
>> catching the exception as you mentioned.
>>
>> On Mon, Dec 14, 2015 at 2:05 PM, Zachary Turner 
>> wrote:
>>
>>> If nothing else, maybe we can print out a more useful exception
>>> backtrace.  What kind of exception, what line, and what was the message?
>>> That might help give us a better idea of what's causing it.
>>>
>>> On Mon, Dec 14, 2015 at 2:03 PM Todd Fiala  wrote:
>>>
 Hi Zachary!





 On Mon, Dec 14, 2015 at 1:28 PM, Zachary Turner via lldb-dev <
 lldb-dev@lists.llvm.org> wrote:

> Hi Todd, lately I've been seeing this sporadically when running the
> test suite.
>
> [TestNamespaceLookup.py FAILED]
> Command invoked: C:\Python27_LLDB\x86\python_d.exe
> D:\src\llvm\tools\lldb\test\dotest.pyc -q --arch=i686 --executable
> D:/src/llvmbuild/ninja/bin/lldb.exe -s
> D:/src/llvmbuild/ninja/lldb-test-traces -u CXXFLAGS -u CFLAGS
> --enable-crash-dialog -C d:\src\llvmbuild\ninja_release\bin\clang.exe
> --results-port 55886 --inferior -p TestNamespaceLookup.py
> D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test --event-add-entries
> worker_index=10:int
> 416 out of 416 test suites processed - TestAddDsymCommand.py
> error: uncaptured python exception, closing channel
>  connected
> 127.0.0.1:56008 at 0x2bdd578> (:[Errno 10054]
> An existing connection was forcibly closed by the remote host
> [C:\Python27_LLDB\x86\lib\asyncore.py|read|83]
> [C:\Python27_LLDB\x86\lib\asyncore.py|handle_read_event|449]
> [D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test\dotest_channels.py|handle_read|133]
> [C:\Python27_LLDB\x86\lib\asyncore.py|recv|387])
>
> It seems to happen randomly and not always on the same test.
> Sometimes it doesn't happen at all.  I wonder if this could be related to
> some of the work that's been going on recently.  Are you seeing this?  Any
> idea how to diagnose?
>

 Eww.

 That *looks* like one side of the connection between the inferior and
 the test runner process choked on reading content from the test event
 socket when the other end went down.  Reading it a bit more carefully, it
 looks like it is the event collector (which would be the parallel test
 runner side) that was receiving when the socket went down.

 I think this means I just need to put a try block around the receiver
 and just bail out gracefully (possibly with a message) when that happens at
 an unexpected time.  Since test inferiors can die at any time, possibly due
 to a timeout where they are forcibly killed, we do need to handle that
 gracefully.'

 I'll see if I can force it, replicate it, and fix it.  I'll look at
 that now (pending watching the buildbots for the other change I just put
 in).

 And yes, this would be a code path that we use heavily with the xUnit
 reporter, but only started getting used by you more recently when I turned
 on the newer summary results by default.  (The newer summary results use
 the test event system, which means test inferior

Re: [lldb-dev] test rerun phase is in

[Ying Chen via lldb-dev]

On Tue, Dec 15, 2015 at 3:36 PM, Todd Fiala  wrote:

> #4310 failed for some other reason.
>

Sorry for the confusion, the builds on android builders should be triggered
by android buildserver
,
since that's where lldb-server was built. It failed because buildserver
hasn't completed that specific revision.


> #4311 looks like it might be stuck in the test3 phase but it is showing
> less output than it had before (maybe because it hasn't timed out yet).
>

#4311 contains 255676.


> I'm usually running with --rerun-all-issues, but I can force similar
> failures to what this bot is seeing when I crank up the load over there on
> an OS X box.  I'm doing that now and I'm omitting the --rerun-all-issues
> flag, which should be close to how the android testbot is running.
> Hopefully I can force it to fail here.
> If not, I'll temporarily disable the rerun unless --rerun-all-issues until
> we can figure out what's causing the stall.
>
Sounds a good plan.


> BTW - how many cores are present on that box?  That will help me figure
> out which runner is being used for the main phase.
>
There're 40 cores on that builder. But dotest.py was running with 8 threads
for andorid targets.
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

[lldb-dev] [Bug 25844] test rerun logic is hanging on android cross compiler buildbot

[via lldb-dev]

https://llvm.org/bugs/show_bug.cgi?id=25844

Todd Fiala  changed:

   What|Removed |Added

 Status|NEW |ASSIGNED
   Assignee|lldb-dev@lists.llvm.org |todd.fi...@gmail.com

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

[lldb-dev] [Bug 25844] New: test rerun logic is hanging on android cross compiler buildbot

[via lldb-dev]

https://llvm.org/bugs/show_bug.cgi?id=25844

Bug ID: 25844
   Summary: test rerun logic is hanging on android cross compiler
buildbot
   Product: lldb
   Version: unspecified
  Hardware: PC
OS: All
Status: NEW
  Severity: normal
  Priority: P
 Component: All Bugs
  Assignee: lldb-dev@lists.llvm.org
  Reporter: todd.fi...@gmail.com
CC: llvm-b...@lists.llvm.org
Classification: Unclassified

See builds like:
http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4311

Change coming in a moment to workaround by disabling rerun logic for eligible
tests when aarch64 is in the architecture list.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Problem with dotest_channels.py

[Zachary Turner via lldb-dev]

I wonder if you need a flush somewhere before you invoke the cleanup func?
Would that do it?  It looks like the sending side of the connection is
closing before the receiving side has received all its data.

On Tue, Dec 15, 2015 at 3:49 PM Adrian McCarthy  wrote:

> With Todd's change, I was getting a Ninja crash.  Zach and I replaced the
> returns Todd added with raises, in order to propagate the exception up the
> stack, and that avoids the Ninja crash, so I'll check that in in a moment.
>
> In the mean time, here's the error message we got out of it.
>
> 155 out of 416 test suites processed - TestBacktraceAll.py
> INFO: received socket error when reading data from test inferior:
> [Errno 10054] An existing connection was forcibly closed by the remote host
> error: uncaptured python exception, closing channel
>  127.
> 0.0.1:58961 at 0x2bb8878> (:[Errno 10054] An
> existing connection was forcibly closed by the remote host
> [D:\Python_for_lldb\x86\lib\asyncore.py|read|83]
> [D:\Python_for_lldb\x86\lib\asyncore.py|handle_read_event|449]
> [D:\src\llvm\llvm\tools\lldb\packages\Python\lldbsuite\test\dotest_channels.py|handle_read|137]
> [D:\Python_for_lldb\x86\lib\asyncore.py|recv|387])
> 175 out of 416 test suites processed - TestNoSuchArch.py
>
> On Mon, Dec 14, 2015 at 3:58 PM, Todd Fiala via lldb-dev <
> lldb-dev@lists.llvm.org> wrote:
>
>> Hey Zachary,
>>
>> I just put in:
>> r255581
>>
>> which should hopefully:
>> (1) catch the exception you see there,
>> (2) handle it gracefully in the common and to-be-expected case of the
>> test inferior going down hard, and
>> (3) print out an error if anything else unexpected is happening here.
>>
>> Let me know if you get any more info with it.  Thanks!
>>
>> -Todd
>>
>> On Mon, Dec 14, 2015 at 2:16 PM, Todd Fiala  wrote:
>>
>>> Yeah that's a messed up exception scenario that is hard to read.  I'll
>>> figure something out when I repro it.  One side is closing before the other
>>> is expecting it, but likely in a way we need to expect.
>>>
>>> I think it is ugly-ified because it is coming from some kind of worker
>>> thread within async-core.
>>>
>>> I will get something in to help it today.  The first bit may be just
>>> catching the exception as you mentioned.
>>>
>>> On Mon, Dec 14, 2015 at 2:05 PM, Zachary Turner 
>>> wrote:
>>>
 If nothing else, maybe we can print out a more useful exception
 backtrace.  What kind of exception, what line, and what was the message?
 That might help give us a better idea of what's causing it.

 On Mon, Dec 14, 2015 at 2:03 PM Todd Fiala 
 wrote:

> Hi Zachary!
>
>
>
>
>
> On Mon, Dec 14, 2015 at 1:28 PM, Zachary Turner via lldb-dev <
> lldb-dev@lists.llvm.org> wrote:
>
>> Hi Todd, lately I've been seeing this sporadically when running the
>> test suite.
>>
>> [TestNamespaceLookup.py FAILED]
>> Command invoked: C:\Python27_LLDB\x86\python_d.exe
>> D:\src\llvm\tools\lldb\test\dotest.pyc -q --arch=i686 --executable
>> D:/src/llvmbuild/ninja/bin/lldb.exe -s
>> D:/src/llvmbuild/ninja/lldb-test-traces -u CXXFLAGS -u CFLAGS
>> --enable-crash-dialog -C d:\src\llvmbuild\ninja_release\bin\clang.exe
>> --results-port 55886 --inferior -p TestNamespaceLookup.py
>> D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test --event-add-entries
>> worker_index=10:int
>> 416 out of 416 test suites processed - TestAddDsymCommand.py
>>   error: uncaptured python exception, closing channel
>> > connected
>> 127.0.0.1:56008 at 0x2bdd578> (:[Errno 10054]
>> An existing connection was forcibly closed by the remote host
>> [C:\Python27_LLDB\x86\lib\asyncore.py|read|83]
>> [C:\Python27_LLDB\x86\lib\asyncore.py|handle_read_event|449]
>> [D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test\dotest_channels.py|handle_read|133]
>> [C:\Python27_LLDB\x86\lib\asyncore.py|recv|387])
>>
>> It seems to happen randomly and not always on the same test.
>> Sometimes it doesn't happen at all.  I wonder if this could be related to
>> some of the work that's been going on recently.  Are you seeing this?  
>> Any
>> idea how to diagnose?
>>
>
> Eww.
>
> That *looks* like one side of the connection between the inferior and
> the test runner process choked on reading content from the test event
> socket when the other end went down.  Reading it a bit more carefully, it
> looks like it is the event collector (which would be the parallel test
> runner side) that was receiving when the socket went down.
>
> I think this means I just need to put a try block around the receiver
> and just bail out gracefully (possibly with a message) when that happens 
> at
> an unexpected time.  Since test inferiors can die at any time, possibly 
> due
> to a timeout where they are forcibly killed, we do need to handle that
> gracefully.'
>

Re: [lldb-dev] test rerun phase is in

[Ying Chen via lldb-dev]

It also happened for -A arm.
http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4307/steps/test7/logs/stdio

On Tue, Dec 15, 2015 at 3:46 PM, Todd Fiala  wrote:

> Hey Ying,
>
> I'm going to check in something that stops the rerun logic when both (1)
> -A aarch64 is specified and (2) --rerun-all-issues is not specified.
>
> That'll give me some time to drill into what's getting stuck on the
> android buildbot.
>
> -Todd
>
> On Tue, Dec 15, 2015 at 3:36 PM, Todd Fiala  wrote:
>
>> #4310 failed for some other reason.
>>
>> #4311 looks like it might be stuck in the test3 phase but it is showing
>> less output than it had before (maybe because it hasn't timed out yet).
>>
>> I'm usually running with --rerun-all-issues, but I can force similar
>> failures to what this bot is seeing when I crank up the load over there on
>> an OS X box.  I'm doing that now and I'm omitting the --rerun-all-issues
>> flag, which should be close to how the android testbot is running.
>> Hopefully I can force it to fail here.
>>
>> If not, I'll temporarily disable the rerun unless --rerun-all-issues
>> until we can figure out what's causing the stall.
>>
>> BTW - how many cores are present on that box?  That will help me figure
>> out which runner is being used for the main phase.
>>
>> Thanks!
>>
>> -Todd
>>
>> On Tue, Dec 15, 2015 at 2:34 PM, Todd Fiala  wrote:
>>
>>> Build >= #4310 is what I'll be watching.
>>>
>>>
>>> On Tue, Dec 15, 2015 at 2:30 PM, Todd Fiala 
>>> wrote:
>>>
 Okay cool.  Will do.

 On Tue, Dec 15, 2015 at 2:22 PM, Ying Chen  wrote:

> Sure. Please go ahead to do that.
> BTW, the pending builds should be merged into one build once current
> build is done.
>
> On Tue, Dec 15, 2015 at 2:12 PM, Todd Fiala 
> wrote:
>
>> Hey Ying,
>>
>> Do you mind if we clear the android builder queue to get a build with
>> r255676 in it?  There are what looks like at least 3 or 4 builds between
>> now and then, and with timeouts it may take several hours.
>>
>> -Todd
>>
>> On Tue, Dec 15, 2015 at 1:50 PM, Ying Chen  wrote:
>>
>>> Yes, it happens every time for android builder.
>>>
>>> On Tue, Dec 15, 2015 at 1:45 PM, Todd Fiala 
>>> wrote:
>>>
 Hmm, yeah it looks like it did the rerun and then after finishing
 the rerun, it's just hanging.

 Let's have a look right after r255676 goes through this builder.  I
 hit a hang in the curses output display due to the recursive taking of 
 a
 lock on a lock that was not recursive-enabled.  While I would have 
 expected
 to see that with the basic results output that this builder here is 
 using
 when I was testing earlier, it's possible somehow that we're hitting a 
 path
 here that is attempting to recursively take a lock.

 Do you know if it is happening every single time a rerun occurs?
  (Hopefully yes?)

 -Todd

 On Tue, Dec 15, 2015 at 1:38 PM, Todd Fiala 
 wrote:

> Yep, I'll have a look!
>
> On Tue, Dec 15, 2015 at 12:43 PM, Ying Chen 
> wrote:
>
>> Hi Todd,
>>
>> It is noticed on lldb android builders that the test_runner
>> didn't exit after rerun, which caused buildbot timeout since the 
>> process
>> was hanging for over 20 minutes.
>> Could you please take a look if that's related to your change?
>>
>> Please see the following builds.
>>
>> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4305/steps/test3/logs/stdio
>>
>> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4305/steps/test7/logs/stdio
>>
>> Thanks,
>> Ying
>>
>> On Mon, Dec 14, 2015 at 4:52 PM, Todd Fiala via lldb-dev <
>> lldb-dev@lists.llvm.org> wrote:
>>
>>> And, btw, this shows the rerun logic working (via the
>>> --rerun-all-issues flag):
>>>
>>> time test/dotest.py --executable `pwd`/build/Debug/lldb
>>> --threads 24 --rerun-all-issues
>>> Testing: 416 test suites, 24 threads
>>> 377 out of 416 test suites processed - TestSBTypeTypeClass.py
>>>
>>> Session logs for test failures/errors/unexpected successes will
>>> go into directory '2015-12-14-16_44_28'
>>> Command invoked: test/dotest.py --executable
>>> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
>>> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 
>>> --inferior
>>> -p TestMultithreaded.py
>>> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
>>> --event-add-entries worker_index=3:int
>>>
>>> Configuration: arch=x86_

Re: [lldb-dev] [3.8 Release] Schedule and call for testers

[Brian Cain via lldb-dev]

I can do SLES.  I also have an arm7l ubuntu 15.x machine if no one else is
already covering that target.

On Mon, Dec 14, 2015 at 3:26 PM, Nikola Smiljanic 
wrote:

> I'll do Fedora and openSUSE.
>
> On Mon, Dec 14, 2015 at 9:08 PM, Daniel Sanders  > wrote:
>
>> Sounds good to me. I'll do the usual mips packages.
>>
>> > -Original Message-
>> > From: hwennb...@google.com [mailto:hwennb...@google.com] On Behalf
>> > Of Hans Wennborg
>> > Sent: 11 December 2015 23:15
>> > To: llvm-dev; cfe-dev; lldb-dev@lists.llvm.org;
>> openmp-...@lists.llvm.org
>> > Cc: Dimitry Andric; Sebastian Dreßler; Renato Golin; Pavel Labath;
>> Sylvestre
>> > Ledru; Ed Maste; Ben Pope; Daniel Sanders; Nikola Smiljanić; Brian
>> Cain; Tom
>> > Stellard
>> > Subject: [3.8 Release] Schedule and call for testers
>> >
>> > Dear everyone,
>> >
>> > It's not quite time to start the 3.8 release process, but it's time to
>> > start planning.
>> >
>> > Please let me know if you want to help with testing and building
>> > release binaries for your favourite platform. (If you were a tester on
>> > the previous release, you're cc'd on this email.)
>> >
>> > I propose the following schedule for the 3.8 release:
>> >
>> > - 13 January: Create 3.8 branch. Testing Phase 1: RC1 binaries built
>> > and tested, bugs fixed. Any almost-complete features need to be
>> > wrapped up or disabled on the branch ASAP, and definitely before this
>> > phase ends.
>> >
>> > - 27 January: Testing Phase 2: RC2 binaries built and tested. Only
>> > critical bug fixes from now on. Further RCs published as we approach..
>> >
>> > - 18 February: Cut the final release, build binaries, ship when ready.
>> >
>> > Unless there are any objections, I'll post this on the web page.
>> >
>> > Cheers,
>> > Hans
>>
>
>


-- 
-Brian
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] test rerun phase is in

[Todd Fiala via lldb-dev]

Arg okay.

I restarted the aarch64 builder so that it will pick up my suppression
there.

I'll adjust it to suppress for arm as well, I'll have to hit that in about
an hour or so but I will do it tonight.

-Todd

On Tue, Dec 15, 2015 at 4:00 PM, Ying Chen  wrote:

> It also happened for -A arm.
>
> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4307/steps/test7/logs/stdio
>
> On Tue, Dec 15, 2015 at 3:46 PM, Todd Fiala  wrote:
>
>> Hey Ying,
>>
>> I'm going to check in something that stops the rerun logic when both (1)
>> -A aarch64 is specified and (2) --rerun-all-issues is not specified.
>>
>> That'll give me some time to drill into what's getting stuck on the
>> android buildbot.
>>
>> -Todd
>>
>> On Tue, Dec 15, 2015 at 3:36 PM, Todd Fiala  wrote:
>>
>>> #4310 failed for some other reason.
>>>
>>> #4311 looks like it might be stuck in the test3 phase but it is showing
>>> less output than it had before (maybe because it hasn't timed out yet).
>>>
>>> I'm usually running with --rerun-all-issues, but I can force similar
>>> failures to what this bot is seeing when I crank up the load over there on
>>> an OS X box.  I'm doing that now and I'm omitting the --rerun-all-issues
>>> flag, which should be close to how the android testbot is running.
>>> Hopefully I can force it to fail here.
>>>
>>> If not, I'll temporarily disable the rerun unless --rerun-all-issues
>>> until we can figure out what's causing the stall.
>>>
>>> BTW - how many cores are present on that box?  That will help me figure
>>> out which runner is being used for the main phase.
>>>
>>> Thanks!
>>>
>>> -Todd
>>>
>>> On Tue, Dec 15, 2015 at 2:34 PM, Todd Fiala 
>>> wrote:
>>>
 Build >= #4310 is what I'll be watching.


 On Tue, Dec 15, 2015 at 2:30 PM, Todd Fiala 
 wrote:

> Okay cool.  Will do.
>
> On Tue, Dec 15, 2015 at 2:22 PM, Ying Chen  wrote:
>
>> Sure. Please go ahead to do that.
>> BTW, the pending builds should be merged into one build once current
>> build is done.
>>
>> On Tue, Dec 15, 2015 at 2:12 PM, Todd Fiala 
>> wrote:
>>
>>> Hey Ying,
>>>
>>> Do you mind if we clear the android builder queue to get a build
>>> with r255676 in it?  There are what looks like at least 3 or 4 builds
>>> between now and then, and with timeouts it may take several hours.
>>>
>>> -Todd
>>>
>>> On Tue, Dec 15, 2015 at 1:50 PM, Ying Chen 
>>> wrote:
>>>
 Yes, it happens every time for android builder.

 On Tue, Dec 15, 2015 at 1:45 PM, Todd Fiala 
 wrote:

> Hmm, yeah it looks like it did the rerun and then after finishing
> the rerun, it's just hanging.
>
> Let's have a look right after r255676 goes through this builder.
> I hit a hang in the curses output display due to the recursive taking 
> of a
> lock on a lock that was not recursive-enabled.  While I would have 
> expected
> to see that with the basic results output that this builder here is 
> using
> when I was testing earlier, it's possible somehow that we're hitting 
> a path
> here that is attempting to recursively take a lock.
>
> Do you know if it is happening every single time a rerun occurs?
>  (Hopefully yes?)
>
> -Todd
>
> On Tue, Dec 15, 2015 at 1:38 PM, Todd Fiala 
> wrote:
>
>> Yep, I'll have a look!
>>
>> On Tue, Dec 15, 2015 at 12:43 PM, Ying Chen 
>> wrote:
>>
>>> Hi Todd,
>>>
>>> It is noticed on lldb android builders that the test_runner
>>> didn't exit after rerun, which caused buildbot timeout since the 
>>> process
>>> was hanging for over 20 minutes.
>>> Could you please take a look if that's related to your change?
>>>
>>> Please see the following builds.
>>>
>>> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4305/steps/test3/logs/stdio
>>>
>>> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4305/steps/test7/logs/stdio
>>>
>>> Thanks,
>>> Ying
>>>
>>> On Mon, Dec 14, 2015 at 4:52 PM, Todd Fiala via lldb-dev <
>>> lldb-dev@lists.llvm.org> wrote:
>>>
 And, btw, this shows the rerun logic working (via the
 --rerun-all-issues flag):

 time test/dotest.py --executable `pwd`/build/Debug/lldb
 --threads 24 --rerun-all-issues
 Testing: 416 test suites, 24 threads
 377 out of 416 test suites processed - TestSBTypeTypeClass.py

 Session logs for test failures/errors/unexpected successes will
 go into directory '2015-12-14-16_44_28'
 Command invoked: 

Re: [lldb-dev] Problem with dotest_channels.py

[Todd Fiala via lldb-dev]

Yeah I'll have a look at what it's doing.

I wouldn't expect a return to crash there, just not receive the data.  I'm
guessing other parts of asyncore code might be doing invalid things with
the socket at that point.  We do need to be able to handle this case,
though, on timeouts that kill the sending/inferior side.

On Tue, Dec 15, 2015 at 3:56 PM, Zachary Turner  wrote:

> I wonder if you need a flush somewhere before you invoke the cleanup
> func?  Would that do it?  It looks like the sending side of the connection
> is closing before the receiving side has received all its data.
>
> On Tue, Dec 15, 2015 at 3:49 PM Adrian McCarthy 
> wrote:
>
>> With Todd's change, I was getting a Ninja crash.  Zach and I replaced the
>> returns Todd added with raises, in order to propagate the exception up the
>> stack, and that avoids the Ninja crash, so I'll check that in in a moment.
>>
>> In the mean time, here's the error message we got out of it.
>>
>> 155 out of 416 test suites processed - TestBacktraceAll.py
>> INFO: received socket error when reading data from test inferior:
>> [Errno 10054] An existing connection was forcibly closed by the remote
>> host
>> error: uncaptured python exception, closing channel
>> > 127.
>> 0.0.1:58961 at 0x2bb8878> (:[Errno 10054] An
>> existing connection was forcibly closed by the remote host
>> [D:\Python_for_lldb\x86\lib\asyncore.py|read|83]
>> [D:\Python_for_lldb\x86\lib\asyncore.py|handle_read_event|449]
>> [D:\src\llvm\llvm\tools\lldb\packages\Python\lldbsuite\test\dotest_channels.py|handle_read|137]
>> [D:\Python_for_lldb\x86\lib\asyncore.py|recv|387])
>> 175 out of 416 test suites processed - TestNoSuchArch.py
>>
>> On Mon, Dec 14, 2015 at 3:58 PM, Todd Fiala via lldb-dev <
>> lldb-dev@lists.llvm.org> wrote:
>>
>>> Hey Zachary,
>>>
>>> I just put in:
>>> r255581
>>>
>>> which should hopefully:
>>> (1) catch the exception you see there,
>>> (2) handle it gracefully in the common and to-be-expected case of the
>>> test inferior going down hard, and
>>> (3) print out an error if anything else unexpected is happening here.
>>>
>>> Let me know if you get any more info with it.  Thanks!
>>>
>>> -Todd
>>>
>>> On Mon, Dec 14, 2015 at 2:16 PM, Todd Fiala 
>>> wrote:
>>>
 Yeah that's a messed up exception scenario that is hard to read.  I'll
 figure something out when I repro it.  One side is closing before the other
 is expecting it, but likely in a way we need to expect.

 I think it is ugly-ified because it is coming from some kind of worker
 thread within async-core.

 I will get something in to help it today.  The first bit may be just
 catching the exception as you mentioned.

 On Mon, Dec 14, 2015 at 2:05 PM, Zachary Turner 
 wrote:

> If nothing else, maybe we can print out a more useful exception
> backtrace.  What kind of exception, what line, and what was the message?
> That might help give us a better idea of what's causing it.
>
> On Mon, Dec 14, 2015 at 2:03 PM Todd Fiala 
> wrote:
>
>> Hi Zachary!
>>
>>
>>
>>
>>
>> On Mon, Dec 14, 2015 at 1:28 PM, Zachary Turner via lldb-dev <
>> lldb-dev@lists.llvm.org> wrote:
>>
>>> Hi Todd, lately I've been seeing this sporadically when running the
>>> test suite.
>>>
>>> [TestNamespaceLookup.py FAILED]
>>> Command invoked: C:\Python27_LLDB\x86\python_d.exe
>>> D:\src\llvm\tools\lldb\test\dotest.pyc -q --arch=i686 --executable
>>> D:/src/llvmbuild/ninja/bin/lldb.exe -s
>>> D:/src/llvmbuild/ninja/lldb-test-traces -u CXXFLAGS -u CFLAGS
>>> --enable-crash-dialog -C d:\src\llvmbuild\ninja_release\bin\clang.exe
>>> --results-port 55886 --inferior -p TestNamespaceLookup.py
>>> D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test 
>>> --event-add-entries
>>> worker_index=10:int
>>> 416 out of 416 test suites processed - TestAddDsymCommand.py
>>>   error: uncaptured python exception, closing channel
>>> >> connected
>>> 127.0.0.1:56008 at 0x2bdd578> (:[Errno 10054]
>>> An existing connection was forcibly closed by the remote host
>>> [C:\Python27_LLDB\x86\lib\asyncore.py|read|83]
>>> [C:\Python27_LLDB\x86\lib\asyncore.py|handle_read_event|449]
>>> [D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test\dotest_channels.py|handle_read|133]
>>> [C:\Python27_LLDB\x86\lib\asyncore.py|recv|387])
>>>
>>> It seems to happen randomly and not always on the same test.
>>> Sometimes it doesn't happen at all.  I wonder if this could be related 
>>> to
>>> some of the work that's been going on recently.  Are you seeing this?  
>>> Any
>>> idea how to diagnose?
>>>
>>
>> Eww.
>>
>> That *looks* like one side of the connection between the inferior and
>> the test runner process choked on reading content from the test event
>> socket when the other end went down.  Reading it a bit more

Re: [lldb-dev] test rerun phase is in

[Todd Fiala via lldb-dev]

I see that build #4314 no longer hanged on aarch64.  (It failed, but for
what looks like legitimate reasons).

I'll make the change for arm as well.

-Todd

On Tue, Dec 15, 2015 at 5:14 PM, Todd Fiala  wrote:

> Arg okay.
>
> I restarted the aarch64 builder so that it will pick up my suppression
> there.
>
> I'll adjust it to suppress for arm as well, I'll have to hit that in about
> an hour or so but I will do it tonight.
>
> -Todd
>
> On Tue, Dec 15, 2015 at 4:00 PM, Ying Chen  wrote:
>
>> It also happened for -A arm.
>>
>> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4307/steps/test7/logs/stdio
>>
>> On Tue, Dec 15, 2015 at 3:46 PM, Todd Fiala  wrote:
>>
>>> Hey Ying,
>>>
>>> I'm going to check in something that stops the rerun logic when both (1)
>>> -A aarch64 is specified and (2) --rerun-all-issues is not specified.
>>>
>>> That'll give me some time to drill into what's getting stuck on the
>>> android buildbot.
>>>
>>> -Todd
>>>
>>> On Tue, Dec 15, 2015 at 3:36 PM, Todd Fiala 
>>> wrote:
>>>
 #4310 failed for some other reason.

 #4311 looks like it might be stuck in the test3 phase but it is showing
 less output than it had before (maybe because it hasn't timed out yet).

 I'm usually running with --rerun-all-issues, but I can force similar
 failures to what this bot is seeing when I crank up the load over there on
 an OS X box.  I'm doing that now and I'm omitting the --rerun-all-issues
 flag, which should be close to how the android testbot is running.
 Hopefully I can force it to fail here.

 If not, I'll temporarily disable the rerun unless --rerun-all-issues
 until we can figure out what's causing the stall.

 BTW - how many cores are present on that box?  That will help me figure
 out which runner is being used for the main phase.

 Thanks!

 -Todd

 On Tue, Dec 15, 2015 at 2:34 PM, Todd Fiala 
 wrote:

> Build >= #4310 is what I'll be watching.
>
>
> On Tue, Dec 15, 2015 at 2:30 PM, Todd Fiala 
> wrote:
>
>> Okay cool.  Will do.
>>
>> On Tue, Dec 15, 2015 at 2:22 PM, Ying Chen  wrote:
>>
>>> Sure. Please go ahead to do that.
>>> BTW, the pending builds should be merged into one build once current
>>> build is done.
>>>
>>> On Tue, Dec 15, 2015 at 2:12 PM, Todd Fiala 
>>> wrote:
>>>
 Hey Ying,

 Do you mind if we clear the android builder queue to get a build
 with r255676 in it?  There are what looks like at least 3 or 4 builds
 between now and then, and with timeouts it may take several hours.

 -Todd

 On Tue, Dec 15, 2015 at 1:50 PM, Ying Chen 
 wrote:

> Yes, it happens every time for android builder.
>
> On Tue, Dec 15, 2015 at 1:45 PM, Todd Fiala 
> wrote:
>
>> Hmm, yeah it looks like it did the rerun and then after finishing
>> the rerun, it's just hanging.
>>
>> Let's have a look right after r255676 goes through this builder.
>> I hit a hang in the curses output display due to the recursive 
>> taking of a
>> lock on a lock that was not recursive-enabled.  While I would have 
>> expected
>> to see that with the basic results output that this builder here is 
>> using
>> when I was testing earlier, it's possible somehow that we're hitting 
>> a path
>> here that is attempting to recursively take a lock.
>>
>> Do you know if it is happening every single time a rerun occurs?
>>  (Hopefully yes?)
>>
>> -Todd
>>
>> On Tue, Dec 15, 2015 at 1:38 PM, Todd Fiala > > wrote:
>>
>>> Yep, I'll have a look!
>>>
>>> On Tue, Dec 15, 2015 at 12:43 PM, Ying Chen 
>>> wrote:
>>>
 Hi Todd,

 It is noticed on lldb android builders that the test_runner
 didn't exit after rerun, which caused buildbot timeout since the 
 process
 was hanging for over 20 minutes.
 Could you please take a look if that's related to your change?

 Please see the following builds.

 http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4305/steps/test3/logs/stdio

 http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4305/steps/test7/logs/stdio

 Thanks,
 Ying

 On Mon, Dec 14, 2015 at 4:52 PM, Todd Fiala via lldb-dev <
 lldb-dev@lists.llvm.org> wrote:

> And, btw, this shows the rerun logic working (via the
> --rerun-all-issues flag):
>
> time test/dotest.py --executable `pwd`/b

[lldb-dev] [Bug 25847] New: "f s <#>" fails as short form for "frame select <#>"

[via lldb-dev]

https://llvm.org/bugs/show_bug.cgi?id=25847

Bug ID: 25847
   Summary: "f s <#>" fails as short form for "frame select <#>"
   Product: lldb
   Version: unspecified
  Hardware: PC
OS: FreeBSD
Status: NEW
  Severity: normal
  Priority: P
 Component: All Bugs
  Assignee: lldb-dev@lists.llvm.org
  Reporter: ema...@freebsd.org
CC: llvm-b...@lists.llvm.org
Classification: Unclassified

"f s 2" reports "error: invalid arguments." and "error: Frame index
(4294967295) out of range." while "frame select 2" is successful.

Session transcript:

(lldb) target create "/bin/sleep"
Current executable set to '/bin/sleep' (x86_64).
(lldb) settings set -- target.run-args  "3600"
(lldb) run
Process 24755 launching
Process 24755 launched: '/bin/sleep' (x86_64)
Process 24755 stopped
* thread #1: tid = 100103, 0x000800b0925a libc.so.7`__sys_nanosleep + 10 at
_nanosleep.S:3, stop reason = signal SIGSTOP
frame #0: 0x000800b0925a libc.so.7`__sys_nanosleep + 10 at
_nanosleep.S:3
   1#include "compat.h"
   2#include "SYS.h"
-> 3PSEUDO(nanosleep)
   4.section .note.GNU-stack,"",%progbits
(lldb) bt
* thread #1: tid = 100103, 0x000800b0925a libc.so.7`__sys_nanosleep + 10 at
_nanosleep.S:3, stop reason = signal SIGSTOP
  * frame #0: 0x000800b0925a libc.so.7`__sys_nanosleep + 10 at
_nanosleep.S:3
frame #1: 0x00400ae6 sleep`main(argc=,
argv=) + 150 at sleep.c:91 [opt]
frame #2: 0x0040091f sleep`_start(ap=,
cleanup=) + 367 at crt1.c:78 [opt]
(lldb) f s 2
error: invalid arguments.


Command Options Usage:
  frame select [-r ] []

   -r  ( --relative  )
A relative frame index offset from the current frame index.
error: Frame index (4294967295) out of range.
(lldb) frame select 2
sleep was compiled with optimization - stepping may behave oddly; variables may
not be available.
frame #2: 0x0040091f sleep`_start(ap=,
cleanup=) + 367 at crt1.c:78 [opt]
   75   #endif
   76  
   77   handle_static_init(argc, argv, env);
-> 78   exit(main(argc, argv, env));
   79   }
(lldb)

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev