Hi Tamas,
On Thu, Dec 10, 2015 at 2:52 AM, Tamas Berghammer <tbergham...@google.com> wrote: > HI Todd, > > You changed the way the test failure list is printed in a way that now we > only print the name of the test function failing with the name of the test > file in parenthesis. Can we add back the name of the test class to this > list? > Sure. I originally planned to have that in there but there was some discussion about it being too much info. I'm happy to add that back. > > There are 2 reason I am asking for it: > * To run only a specific test we have to specify the "-f" option to > dotest.py and it takes the fully qualified function name as an argument. > Before your change it was displayed in the test output (in a bit > uncomfortable way) but after your change we have to open the test file and > copy the class name from there to run only a single test suit. > * With the new output format the output of the buildbot only displays the > list of the failing test function names what isn't too specific in a lot of > case (e.g. we have several test method called test_dwarf). This point is > less important as the file name can be added to the output from the > buildbot perspective. > Okay, not sure exactly what is doing that printing, but the file name is available from the details of the summary. Sounds like it is a text parsing change needed based on the changes I made to the format. > > Thanks, > Sure thing! -Todd > Tamas > > On Wed, Dec 9, 2015 at 7:57 PM Ying Chen <chy...@google.com> wrote: > >> I submitted this patch to include "ERROR" lines in buildbot step results. >> http://reviews.llvm.org/rL255145 >> >> Error results will be displayed in step result like this after the patch, >> "ERROR: 9 (SIGKILL) test_buildbot_catches_exceptional_exit_dwarf" >> >> Thanks, >> Ying >> >> On Wed, Dec 9, 2015 at 10:45 AM, Todd Fiala <todd.fi...@gmail.com> wrote: >> >>> Great, thanks Tamas! >>> >>> I left the default turned on, and just essentially removed the issues by >>> parking them as .py.parked files. That way we can flip them on in the >>> future if we want to verify a testbot's detection of these. >>> >>> I will be going back to the xUnit Results formatter and making sure it >>> maps timeouts and exceptional errors to the xUnit error type with details. >>> >>> On Wed, Dec 9, 2015 at 10:30 AM, Tamas Berghammer < >>> tbergham...@google.com> wrote: >>> >>>> Thank you for making the experiment. It looks reasonable. For the ERROR >>>> the buildbot detected it and it will fail the build but it isn't listed in >>>> the list of failing tests what should be fixed. After this experiment I >>>> think it is fine to change the default output formatter from our side. >>>> >>>> Tamas >>>> >>>> On Wed, Dec 9, 2015 at 6:26 PM Todd Fiala <todd.fi...@gmail.com> wrote: >>>> >>>>> The reports look good at the test level: >>>>> >>>>> >>>>> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-cmake/builds/9294 >>>>> >>>>> I'd say the buildbot reflection script missed the ERROR, so that is >>>>> something maybe Ying can look at (the summary line in the build run), but >>>>> that is unrelated AFAICT. >>>>> >>>>> I'm going to move aside the failures. >>>>> >>>>> On Wed, Dec 9, 2015 at 10:13 AM, Todd Fiala <todd.fi...@gmail.com> >>>>> wrote: >>>>> >>>>>> I am going to stop the current build on that builder. There was one >>>>>> change in it, and it will be another 20 minutes before it completes. I >>>>>> don't want the repo in a known broken state that long. >>>>>> >>>>>> On Wed, Dec 9, 2015 at 10:07 AM, Todd Fiala <todd.fi...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> I forced a build on the ubuntu 14.04 cmake builder. The build >>>>>>> _after_ 9292 will contain the two changes (and we will expect failures >>>>>>> on >>>>>>> it). >>>>>>> >>>>>>> On Wed, Dec 9, 2015 at 10:05 AM, Todd Fiala <todd.fi...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> These went in as: >>>>>>>> >>>>>>>> r255130 - turn it on by default >>>>>>>> r255131 - create known issues. This one is to be reverted if all 3 >>>>>>>> types show up properly. >>>>>>>> >>>>>>>> On Wed, Dec 9, 2015 at 9:41 AM, Todd Fiala <todd.fi...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> It is a small change. >>>>>>>>> >>>>>>>>> I almost have all the trial tests ready, so I'll just commit both >>>>>>>>> changes at the same time (the flip on, and the trial balloon issues). >>>>>>>>> >>>>>>>>> If all goes well and the three types of issue show up, then the >>>>>>>>> last of the two will get reverted (the one with the failures). >>>>>>>>> >>>>>>>>> If none (or only some) of the issues show up, they'll both get >>>>>>>>> reverted. >>>>>>>>> >>>>>>>>> -Todd >>>>>>>>> >>>>>>>>> On Wed, Dec 9, 2015 at 9:39 AM, Pavel Labath <lab...@google.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> If it's not too much work, I think the extra bit of noise will >>>>>>>>>> not be >>>>>>>>>> a problem. But I don't think it is really necessary either. >>>>>>>>>> >>>>>>>>>> I assume the actual flip will be a small change that we can back >>>>>>>>>> out >>>>>>>>>> easily if we notice troubles... After a sufficient grace period >>>>>>>>>> we can >>>>>>>>>> remove the old formatter altogether and hopefully simplify the >>>>>>>>>> code >>>>>>>>>> somewhat. >>>>>>>>>> >>>>>>>>>> pl >>>>>>>>>> >>>>>>>>>> On 9 December 2015 at 17:08, Todd Fiala <todd.fi...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> > Here's what I can do. >>>>>>>>>> > >>>>>>>>>> > Put in the change (setting the default to use the new format). >>>>>>>>>> > >>>>>>>>>> > Separately, put in a trial balloon commit with one failing >>>>>>>>>> test, one >>>>>>>>>> > exceptional exit test, and one timeout test, and watch the >>>>>>>>>> ubuntu 14.04 >>>>>>>>>> > buildbot catch it and fail. Then reverse this out. That >>>>>>>>>> should show beyond >>>>>>>>>> > a reasonable doubt whether the buildbot catches new failures >>>>>>>>>> and errors. (I >>>>>>>>>> > think this is a noisy way to accomplish this, but it certainly >>>>>>>>>> would >>>>>>>>>> > validate if its working). >>>>>>>>>> > >>>>>>>>>> > -Todd >>>>>>>>>> > >>>>>>>>>> > On Wed, Dec 9, 2015 at 8:06 AM, Todd Fiala < >>>>>>>>>> todd.fi...@gmail.com> wrote: >>>>>>>>>> >> >>>>>>>>>> >> Specifically, the markers for issue details are: >>>>>>>>>> >> >>>>>>>>>> >> FAIL >>>>>>>>>> >> ERROR >>>>>>>>>> >> UNEXPECTED SUCCESS >>>>>>>>>> >> TIMEOUT >>>>>>>>>> >> >>>>>>>>>> >> (These are the fourth field in the array entries (lines 275 - >>>>>>>>>> 290) of >>>>>>>>>> >> packages/Python/lldbsuite/test/basic_results_formatter.py). >>>>>>>>>> >> >>>>>>>>>> >> -Todd >>>>>>>>>> >> >>>>>>>>>> >> On Wed, Dec 9, 2015 at 8:04 AM, Todd Fiala < >>>>>>>>>> todd.fi...@gmail.com> wrote: >>>>>>>>>> >>> >>>>>>>>>> >>> That's a good point, Tamas. >>>>>>>>>> >>> >>>>>>>>>> >>> I use (so I claim) the same all upper-case markers for the >>>>>>>>>> test result >>>>>>>>>> >>> details. Including, not using XPASS but rather UNEXPECTED >>>>>>>>>> SUCCESS for >>>>>>>>>> >>> unexpected successes. (The former would trigger the lit >>>>>>>>>> script IIRC to >>>>>>>>>> >>> parse that as a failing-style result). >>>>>>>>>> >>> >>>>>>>>>> >>> The intent is this is a no-op on the test runner. >>>>>>>>>> >>> >>>>>>>>>> >>> On Wed, Dec 9, 2015 at 8:02 AM, Tamas Berghammer < >>>>>>>>>> tbergham...@google.com> >>>>>>>>>> >>> wrote: >>>>>>>>>> >>>> >>>>>>>>>> >>>> +Ying Chen >>>>>>>>>> >>>> >>>>>>>>>> >>>> Ying, what do we have to do on the build bot side to support >>>>>>>>>> a change in >>>>>>>>>> >>>> the default test result summary formatter? >>>>>>>>>> >>>> >>>>>>>>>> >>>> On Wed, Dec 9, 2015 at 4:00 PM Todd Fiala via lldb-dev >>>>>>>>>> >>>> <lldb-dev@lists.llvm.org> wrote: >>>>>>>>>> >>>>> >>>>>>>>>> >>>>> Hi all, >>>>>>>>>> >>>>> >>>>>>>>>> >>>>> Per a previous thread on this, I've made all the changes I >>>>>>>>>> intended to >>>>>>>>>> >>>>> make last night to get the intended replacement of test run >>>>>>>>>> results meet or >>>>>>>>>> >>>>> exceed current requirements. >>>>>>>>>> >>>>> >>>>>>>>>> >>>>> I'd like to switch over to that by default. I'm depending >>>>>>>>>> on the test >>>>>>>>>> >>>>> event system to be able to handle test method reruns in >>>>>>>>>> test results >>>>>>>>>> >>>>> accounting. >>>>>>>>>> >>>>> >>>>>>>>>> >>>>> The primary thing missing before was that timeouts were not >>>>>>>>>> routed >>>>>>>>>> >>>>> through the test events system, nor were exception process >>>>>>>>>> exits (i.e. test >>>>>>>>>> >>>>> inferiors exiting with a signal on POSIX systems). Those >>>>>>>>>> were added last >>>>>>>>>> >>>>> night so that test events are generated for those, and the >>>>>>>>>> >>>>> BasicResultsFormatter presents that information properly. >>>>>>>>>> >>>>> >>>>>>>>>> >>>>> I will switch it over to being the default output in a bit >>>>>>>>>> here. >>>>>>>>>> >>>>> Please let me know if you have any concerns once I flip it >>>>>>>>>> on by default. >>>>>>>>>> >>>>> >>>>>>>>>> >>>>> Thanks! >>>>>>>>>> >>>>> -- >>>>>>>>>> >>>>> -Todd >>>>>>>>>> >>>>> _______________________________________________ >>>>>>>>>> >>>>> lldb-dev mailing list >>>>>>>>>> >>>>> lldb-dev@lists.llvm.org >>>>>>>>>> >>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev >>>>>>>>>> >>> >>>>>>>>>> >>> >>>>>>>>>> >>> >>>>>>>>>> >>> >>>>>>>>>> >>> -- >>>>>>>>>> >>> -Todd >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> -- >>>>>>>>>> >> -Todd >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > -- >>>>>>>>>> > -Todd >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> -Todd >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> -Todd >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> -Todd >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> -Todd >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> -Todd >>>>> >>>> >>> >>> >>> -- >>> -Todd >>> >> >> -- -Todd
_______________________________________________ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
