[lldb-dev] TestRaise.py test_restart_bug flakey stats
Hi Tamas, I think you grabbed me stats on failing tests in the past. Can you dig up the failure rate for TestRaise.py's test_restart_bug() variants on Ubuntu 14.04 x86_64? I'd like to mark it as flaky on Linux, since it is passing most of the time over here. But I want to see if that's valid across all Ubuntu 14.04 x86_64. (If it is passing some of the time, I'd prefer marking it flakey so that we don't see unexpected successes). Thanks! -- -Todd ___ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] TestRaise.py test_restart_bug flakey stats
Hmm, the flakey behavior may be specific to dwo. Testing it locally as unconditionally flaky on Linux is failing on dwarf. All the ones I see succeed are dwo. I wouldn't expect a diff there but that seems to be the case. So, the request still stands but I won't be surprised if we find that dwo sometimes passes while dwarf doesn't (or at least not enough to get through the flakey setting). On Sat, Oct 17, 2015 at 4:57 PM, Todd Fiala wrote: > Hi Tamas, > > I think you grabbed me stats on failing tests in the past. Can you dig up > the failure rate for TestRaise.py's test_restart_bug() variants on Ubuntu > 14.04 x86_64? I'd like to mark it as flaky on Linux, since it is passing > most of the time over here. But I want to see if that's valid across all > Ubuntu 14.04 x86_64. (If it is passing some of the time, I'd prefer > marking it flakey so that we don't see unexpected successes). > > Thanks! > > -- > -Todd > -- -Todd ___ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] TestRaise.py test_restart_bug flakey stats
Nope, no good either when I limit the flakey to DWO. So perhaps I don't understand how the flakey marking works. I thought it meant: * run the test. * If it passes, it goes as a successful test. Then we're done. * run the test again. * If it passes, then we're done and mark it a successful test. If it fails, then mark it an expected failure. But that's definitely not the behavior I'm seeing, as a flakey marking in the above scheme should never produce a failing test. I'll have to revisit the flakey test marking to see what it's really doing since my understanding is clearly flawed! On Sat, Oct 17, 2015 at 5:57 PM, Todd Fiala wrote: > Hmm, the flakey behavior may be specific to dwo. Testing it locally as > unconditionally flaky on Linux is failing on dwarf. All the ones I see > succeed are dwo. I wouldn't expect a diff there but that seems to be the > case. > > So, the request still stands but I won't be surprised if we find that dwo > sometimes passes while dwarf doesn't (or at least not enough to get through > the flakey setting). > > On Sat, Oct 17, 2015 at 4:57 PM, Todd Fiala wrote: > >> Hi Tamas, >> >> I think you grabbed me stats on failing tests in the past. Can you dig >> up the failure rate for TestRaise.py's test_restart_bug() variants on >> Ubuntu 14.04 x86_64? I'd like to mark it as flaky on Linux, since it is >> passing most of the time over here. But I want to see if that's valid >> across all Ubuntu 14.04 x86_64. (If it is passing some of the time, I'd >> prefer marking it flakey so that we don't see unexpected successes). >> >> Thanks! >> >> -- >> -Todd >> > > > > -- > -Todd > -- -Todd ___ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
