Re: [lldb-dev] BasicResultsFormatter - new test results summary
HI Todd, You changed the way the test failure list is printed in a way that now we only print the name of the test function failing with the name of the test file in parenthesis. Can we add back the name of the test class to this list? There are 2 reason I am asking for it: * To run only a specific test we have to specify the "-f" option to dotest.py and it takes the fully qualified function name as an argument. Before your change it was displayed in the test output (in a bit uncomfortable way) but after your change we have to open the test file and copy the class name from there to run only a single test suit. * With the new output format the output of the buildbot only displays the list of the failing test function names what isn't too specific in a lot of case (e.g. we have several test method called test_dwarf). This point is less important as the file name can be added to the output from the buildbot perspective. Thanks, Tamas On Wed, Dec 9, 2015 at 7:57 PM Ying Chen wrote: > I submitted this patch to include "ERROR" lines in buildbot step results. > http://reviews.llvm.org/rL255145 > > Error results will be displayed in step result like this after the patch, > "ERROR: 9 (SIGKILL) test_buildbot_catches_exceptional_exit_dwarf" > > Thanks, > Ying > > On Wed, Dec 9, 2015 at 10:45 AM, Todd Fiala wrote: > >> Great, thanks Tamas! >> >> I left the default turned on, and just essentially removed the issues by >> parking them as .py.parked files. That way we can flip them on in the >> future if we want to verify a testbot's detection of these. >> >> I will be going back to the xUnit Results formatter and making sure it >> maps timeouts and exceptional errors to the xUnit error type with details. >> >> On Wed, Dec 9, 2015 at 10:30 AM, Tamas Berghammer > > wrote: >> >>> Thank you for making the experiment. It looks reasonable. For the ERROR >>> the buildbot detected it and it will fail the build but it isn't listed in >>> the list of failing tests what should be fixed. After this experiment I >>> think it is fine to change the default output formatter from our side. >>> >>> Tamas >>> >>> On Wed, Dec 9, 2015 at 6:26 PM Todd Fiala wrote: >>> The reports look good at the test level: http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-cmake/builds/9294 I'd say the buildbot reflection script missed the ERROR, so that is something maybe Ying can look at (the summary line in the build run), but that is unrelated AFAICT. I'm going to move aside the failures. On Wed, Dec 9, 2015 at 10:13 AM, Todd Fiala wrote: > I am going to stop the current build on that builder. There was one > change in it, and it will be another 20 minutes before it completes. I > don't want the repo in a known broken state that long. > > On Wed, Dec 9, 2015 at 10:07 AM, Todd Fiala > wrote: > >> I forced a build on the ubuntu 14.04 cmake builder. The build >> _after_ 9292 will contain the two changes (and we will expect failures on >> it). >> >> On Wed, Dec 9, 2015 at 10:05 AM, Todd Fiala >> wrote: >> >>> These went in as: >>> >>> r255130 - turn it on by default >>> r255131 - create known issues. This one is to be reverted if all 3 >>> types show up properly. >>> >>> On Wed, Dec 9, 2015 at 9:41 AM, Todd Fiala >>> wrote: >>> It is a small change. I almost have all the trial tests ready, so I'll just commit both changes at the same time (the flip on, and the trial balloon issues). If all goes well and the three types of issue show up, then the last of the two will get reverted (the one with the failures). If none (or only some) of the issues show up, they'll both get reverted. -Todd On Wed, Dec 9, 2015 at 9:39 AM, Pavel Labath wrote: > If it's not too much work, I think the extra bit of noise will not > be > a problem. But I don't think it is really necessary either. > > I assume the actual flip will be a small change that we can back > out > easily if we notice troubles... After a sufficient grace period we > can > remove the old formatter altogether and hopefully simplify the code > somewhat. > > pl > > On 9 December 2015 at 17:08, Todd Fiala > wrote: > > Here's what I can do. > > > > Put in the change (setting the default to use the new format). > > > > Separately, put in a trial balloon commit with one failing test, > one > > exceptional exit test, and one timeout test, and watch the > ubuntu 14.04 > > buildbot catch it and fail. Then reverse this out. That should > show beyond > > a reasonable doubt whether the buil
Re: [lldb-dev] BasicResultsFormatter - new test results summary
Hi Tamas, On Thu, Dec 10, 2015 at 2:52 AM, Tamas Berghammer wrote: > HI Todd, > > You changed the way the test failure list is printed in a way that now we > only print the name of the test function failing with the name of the test > file in parenthesis. Can we add back the name of the test class to this > list? > Sure. I originally planned to have that in there but there was some discussion about it being too much info. I'm happy to add that back. > > There are 2 reason I am asking for it: > * To run only a specific test we have to specify the "-f" option to > dotest.py and it takes the fully qualified function name as an argument. > Before your change it was displayed in the test output (in a bit > uncomfortable way) but after your change we have to open the test file and > copy the class name from there to run only a single test suit. > * With the new output format the output of the buildbot only displays the > list of the failing test function names what isn't too specific in a lot of > case (e.g. we have several test method called test_dwarf). This point is > less important as the file name can be added to the output from the > buildbot perspective. > Okay, not sure exactly what is doing that printing, but the file name is available from the details of the summary. Sounds like it is a text parsing change needed based on the changes I made to the format. > > Thanks, > Sure thing! -Todd > Tamas > > On Wed, Dec 9, 2015 at 7:57 PM Ying Chen wrote: > >> I submitted this patch to include "ERROR" lines in buildbot step results. >> http://reviews.llvm.org/rL255145 >> >> Error results will be displayed in step result like this after the patch, >> "ERROR: 9 (SIGKILL) test_buildbot_catches_exceptional_exit_dwarf" >> >> Thanks, >> Ying >> >> On Wed, Dec 9, 2015 at 10:45 AM, Todd Fiala wrote: >> >>> Great, thanks Tamas! >>> >>> I left the default turned on, and just essentially removed the issues by >>> parking them as .py.parked files. That way we can flip them on in the >>> future if we want to verify a testbot's detection of these. >>> >>> I will be going back to the xUnit Results formatter and making sure it >>> maps timeouts and exceptional errors to the xUnit error type with details. >>> >>> On Wed, Dec 9, 2015 at 10:30 AM, Tamas Berghammer < >>> tbergham...@google.com> wrote: >>> Thank you for making the experiment. It looks reasonable. For the ERROR the buildbot detected it and it will fail the build but it isn't listed in the list of failing tests what should be fixed. After this experiment I think it is fine to change the default output formatter from our side. Tamas On Wed, Dec 9, 2015 at 6:26 PM Todd Fiala wrote: > The reports look good at the test level: > > > http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-cmake/builds/9294 > > I'd say the buildbot reflection script missed the ERROR, so that is > something maybe Ying can look at (the summary line in the build run), but > that is unrelated AFAICT. > > I'm going to move aside the failures. > > On Wed, Dec 9, 2015 at 10:13 AM, Todd Fiala > wrote: > >> I am going to stop the current build on that builder. There was one >> change in it, and it will be another 20 minutes before it completes. I >> don't want the repo in a known broken state that long. >> >> On Wed, Dec 9, 2015 at 10:07 AM, Todd Fiala >> wrote: >> >>> I forced a build on the ubuntu 14.04 cmake builder. The build >>> _after_ 9292 will contain the two changes (and we will expect failures >>> on >>> it). >>> >>> On Wed, Dec 9, 2015 at 10:05 AM, Todd Fiala >>> wrote: >>> These went in as: r255130 - turn it on by default r255131 - create known issues. This one is to be reverted if all 3 types show up properly. On Wed, Dec 9, 2015 at 9:41 AM, Todd Fiala wrote: > It is a small change. > > I almost have all the trial tests ready, so I'll just commit both > changes at the same time (the flip on, and the trial balloon issues). > > If all goes well and the three types of issue show up, then the > last of the two will get reverted (the one with the failures). > > If none (or only some) of the issues show up, they'll both get > reverted. > > -Todd > > On Wed, Dec 9, 2015 at 9:39 AM, Pavel Labath > wrote: > >> If it's not too much work, I think the extra bit of noise will >> not be >> a problem. But I don't think it is really necessary either. >> >> I assume the actual flip will be a small change that we can back >> out >> easily if we notice troubles... After a sufficient grace period >> we can >> remove the old formatter altog
Re: [lldb-dev] BasicResultsFormatter - new test results summary
On Thu, Dec 10, 2015 at 12:54 PM Todd Fiala wrote: > Hi Tamas, > > > > On Thu, Dec 10, 2015 at 2:52 AM, Tamas Berghammer > wrote: > >> HI Todd, >> >> You changed the way the test failure list is printed in a way that now we >> only print the name of the test function failing with the name of the test >> file in parenthesis. Can we add back the name of the test class to this >> list? >> > > Sure. I originally planned to have that in there but there was some > discussion about it being too much info. I'm happy to add that back. > Can we have it tied to verbosity level? We have -t and -v, maybe one of those could trigger more detail in the summary view. ___ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
[lldb-dev] [Bug 25801] New: Arch Linux fails to build python support
https://llvm.org/bugs/show_bug.cgi?id=25801 Bug ID: 25801 Summary: Arch Linux fails to build python support Product: lldb Version: unspecified Hardware: PC OS: Linux Status: NEW Severity: normal Priority: P Component: All Bugs Assignee: lldb-dev@lists.llvm.org Reporter: todd.fi...@gmail.com CC: llvm-b...@lists.llvm.org Classification: Unclassified There are possibly two issues here: * python 2.x and 3.x, where we want to prefer python 2.x * symbol definitions around Py_InitModule4, where ArchLinux x86_64 is needing it to be defined Py_InitModule4_64 whereas Ubuntu x86_64 doesn't require that. Breaking this one out separately from a Gentoo one since the solutions for both are not necessarily one and the same. -- You are receiving this mail because: You are the assignee for the bug. ___ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] BasicResultsFormatter - new test results summary
Sure, I can do that. Tamas, okay to give more detail on -v? I'll give it a shot to see what else comes out if we do that. -Todd On Thu, Dec 10, 2015 at 12:58 PM, Zachary Turner wrote: > > > On Thu, Dec 10, 2015 at 12:54 PM Todd Fiala wrote: > >> Hi Tamas, >> >> >> >> On Thu, Dec 10, 2015 at 2:52 AM, Tamas Berghammer > > wrote: >> >>> HI Todd, >>> >>> You changed the way the test failure list is printed in a way that now >>> we only print the name of the test function failing with the name of the >>> test file in parenthesis. Can we add back the name of the test class to >>> this list? >>> >> >> Sure. I originally planned to have that in there but there was some >> discussion about it being too much info. I'm happy to add that back. >> > Can we have it tied to verbosity level? We have -t and -v, maybe one of > those could trigger more detail in the summary view. > -- -Todd ___ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] BasicResultsFormatter - new test results summary
Checked this in as r255310. Let me know if you find any issues with that, Tamas. You will need '-v' to enable it. Otherwise, it will just print the method name. -Todd On Thu, Dec 10, 2015 at 2:39 PM, Todd Fiala wrote: > Sure, I can do that. > > Tamas, okay to give more detail on -v? I'll give it a shot to see what > else comes out if we do that. > > -Todd > > On Thu, Dec 10, 2015 at 12:58 PM, Zachary Turner > wrote: > >> >> >> On Thu, Dec 10, 2015 at 12:54 PM Todd Fiala wrote: >> >>> Hi Tamas, >>> >>> >>> >>> On Thu, Dec 10, 2015 at 2:52 AM, Tamas Berghammer < >>> tbergham...@google.com> wrote: >>> HI Todd, You changed the way the test failure list is printed in a way that now we only print the name of the test function failing with the name of the test file in parenthesis. Can we add back the name of the test class to this list? >>> >>> Sure. I originally planned to have that in there but there was some >>> discussion about it being too much info. I'm happy to add that back. >>> >> Can we have it tied to verbosity level? We have -t and -v, maybe one of >> those could trigger more detail in the summary view. >> > > > > -- > -Todd > -- -Todd ___ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
[lldb-dev] Separating test runner and tests
Currently our folder structure looks like this: lldbsuite |-- test |-- dotest.py |-- dosep.py |-- lldbtest.py |-- ... |-- functionalities |-- lang |-- expression_command |-- ... etc I've been thinking about organizing it like this instead: lldbsuite |-- test |-- functionalities |-- lang |-- expression_command |-- ... |-- engine |-- dotest.py |-- dosep.py |-- lldbtest.py |-- ... Anybody have any thoughts on this? Good idea or bad idea? The main reason I want to do this is because as we start breaking up some of the code, it makes sense to start having some subpackages under the `engine` folder (or the `test` folder in our current world). For example, Todd and I have discussed the idea of putting formatter related stuff under a `formatters` subpackage. In the current world, there's no way to differentiate between folders which contain tests and folders which contain test infrastructure, so when we walk the directory tree looking for tests we end up walking a bunch of directories that are used for test infrastructure code and not actual tests. So I like the logical separation this provides -- having the tests themselves all under a single subpackage. Thoughts? ___ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
