On 14 December 2015 at 16:19, Todd Fiala wrote:
>> We would lose the ability to individually expect "failures" and
>> "timeouts", but I don't think that is really necessary, and I think it
>> will be worth the extra maintainability we get from the fact of having
>> fewer test decorators.
>>
>
> OT
On Mon, Dec 14, 2015 at 6:58 AM, Pavel Labath wrote:
> Hi,
>
> thanks a lot for fixing the timeout issue on such a short notice.
Sure thing. I didn't mind at all once I understood the context in which
they were being used.
> I
> didn't think I'd find myself defending them, as I remember bein
Hi,
thanks a lot for fixing the timeout issue on such a short notice. I
didn't think I'd find myself defending them, as I remember being quite
upset when they went in, but they have proven useful in stabilising
the build bots, and I think it's likely you may need them as well.
I'll try to now add
> (btw, I haven't checked, is it possible to XFAIL crashes now?
This currently doesn't work. We'd need a smarter rerun mechanism
(something I do intend to do at some point), where we (1) know all the
tests that should run from a given test file before any are run, and (2)
when a timeout or exce
Merging threads.
> The concept is not there to protect against timeouts, which are caused
by processes being too slow, for these we have been increasing
timeouts where necessary.
Okay, I see. If that's the intent, then expected timeout sounds
reasonable. (My abhorrence was against the idea of u
On Fri, Dec 11, 2015 at 3:26 AM, Pavel Labath wrote:
> Todd, I've had to disable the new result formatter as it was not
> working with the expected timeout logic we have for the old one. The
> old XTIMEOUT code is a massive hack and I will be extremely glad when
> we get rid of it, but we can't k
Todd, I've had to disable the new result formatter as it was not
working with the expected timeout logic we have for the old one. The
old XTIMEOUT code is a massive hack and I will be extremely glad when
we get rid of it, but we can't keep our buildbot red until then, so
I've switched it off.
I am
Checked this in as r255310. Let me know if you find any issues with that,
Tamas.
You will need '-v' to enable it. Otherwise, it will just print the method
name.
-Todd
On Thu, Dec 10, 2015 at 2:39 PM, Todd Fiala wrote:
> Sure, I can do that.
>
> Tamas, okay to give more detail on -v? I'll gi
Sure, I can do that.
Tamas, okay to give more detail on -v? I'll give it a shot to see what
else comes out if we do that.
-Todd
On Thu, Dec 10, 2015 at 12:58 PM, Zachary Turner wrote:
>
>
> On Thu, Dec 10, 2015 at 12:54 PM Todd Fiala wrote:
>
>> Hi Tamas,
>>
>>
>>
>> On Thu, Dec 10, 2015 at
On Thu, Dec 10, 2015 at 12:54 PM Todd Fiala wrote:
> Hi Tamas,
>
>
>
> On Thu, Dec 10, 2015 at 2:52 AM, Tamas Berghammer
> wrote:
>
>> HI Todd,
>>
>> You changed the way the test failure list is printed in a way that now we
>> only print the name of the test function failing with the name of the
Hi Tamas,
On Thu, Dec 10, 2015 at 2:52 AM, Tamas Berghammer
wrote:
> HI Todd,
>
> You changed the way the test failure list is printed in a way that now we
> only print the name of the test function failing with the name of the test
> file in parenthesis. Can we add back the name of the test c
HI Todd,
You changed the way the test failure list is printed in a way that now we
only print the name of the test function failing with the name of the test
file in parenthesis. Can we add back the name of the test class to this
list?
There are 2 reason I am asking for it:
* To run only a specif
I submitted this patch to include "ERROR" lines in buildbot step results.
http://reviews.llvm.org/rL255145
Error results will be displayed in step result like this after the patch,
"ERROR: 9 (SIGKILL) test_buildbot_catches_exceptional_exit_dwarf"
Thanks,
Ying
On Wed, Dec 9, 2015 at 10:45 AM, Tod
Great, thanks Tamas!
I left the default turned on, and just essentially removed the issues by
parking them as .py.parked files. That way we can flip them on in the
future if we want to verify a testbot's detection of these.
I will be going back to the xUnit Results formatter and making sure it m
Thank you for making the experiment. It looks reasonable. For the ERROR the
buildbot detected it and it will fail the build but it isn't listed in the
list of failing tests what should be fixed. After this experiment I think
it is fine to change the default output formatter from our side.
Tamas
O
Verification tests parked (i.e. disabled) here:
r255134
I decided to leave them in the repo so it is faster/easier to do this in
the future.
-Todd
On Wed, Dec 9, 2015 at 10:26 AM, Todd Fiala wrote:
> The reports look good at the test level:
>
>
> http://lab.llvm.org:8011/builders/lldb-x86_64-u
The reports look good at the test level:
http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-cmake/builds/9294
I'd say the buildbot reflection script missed the ERROR, so that is
something maybe Ying can look at (the summary line in the build run), but
that is unrelated AFAICT.
I'm going
I am going to stop the current build on that builder. There was one change
in it, and it will be another 20 minutes before it completes. I don't want
the repo in a known broken state that long.
On Wed, Dec 9, 2015 at 10:07 AM, Todd Fiala wrote:
> I forced a build on the ubuntu 14.04 cmake buil
I forced a build on the ubuntu 14.04 cmake builder. The build _after_ 9292
will contain the two changes (and we will expect failures on it).
On Wed, Dec 9, 2015 at 10:05 AM, Todd Fiala wrote:
> These went in as:
>
> r255130 - turn it on by default
> r255131 - create known issues. This one is t
These went in as:
r255130 - turn it on by default
r255131 - create known issues. This one is to be reverted if all 3 types
show up properly.
On Wed, Dec 9, 2015 at 9:41 AM, Todd Fiala wrote:
> It is a small change.
>
> I almost have all the trial tests ready, so I'll just commit both changes
>
If it's not too much work, I think the extra bit of noise will not be
a problem. But I don't think it is really necessary either.
I assume the actual flip will be a small change that we can back out
easily if we notice troubles... After a sufficient grace period we can
remove the old formatter alt
It is a small change.
I almost have all the trial tests ready, so I'll just commit both changes
at the same time (the flip on, and the trial balloon issues).
If all goes well and the three types of issue show up, then the last of the
two will get reverted (the one with the failures).
If none (or
Here's what I can do.
Put in the change (setting the default to use the new format).
Separately, put in a trial balloon commit with one failing test, one
exceptional exit test, and one timeout test, and watch the ubuntu 14.04
buildbot catch it and fail. Then reverse this out. That should show
b
+Ying Chen
Ying, what do we have to do on the build bot side to support a change in
the default test result summary formatter?
On Wed, Dec 9, 2015 at 4:00 PM Todd Fiala via lldb-dev <
lldb-dev@lists.llvm.org> wrote:
> Hi all,
>
> Per a previous thread on this, I've made all the changes I intend
Specifically, the markers for issue details are:
FAIL
ERROR
UNEXPECTED SUCCESS
TIMEOUT
(These are the fourth field in the array entries (lines 275 - 290) of
packages/Python/lldbsuite/test/basic_results_formatter.py).
-Todd
On Wed, Dec 9, 2015 at 8:04 AM, Todd Fiala wrote:
> That's a good poin
That's a good point, Tamas.
I use (so I claim) the same all upper-case markers for the test result
details. Including, not using XPASS but rather UNEXPECTED SUCCESS for
unexpected successes. (The former would trigger the lit script IIRC to
parse that as a failing-style result).
The intent is th
Hi all,
Per a previous thread on this, I've made all the changes I intended to make
last night to get the intended replacement of test run results meet or
exceed current requirements.
I'd like to switch over to that by default. I'm depending on the test
event system to be able to handle test met
27 matches
Mail list logo
