Re: [lldb-dev] Digging into Linux unexpected successes

2015-09-15 Thread Todd Fiala via lldb-dev
Change http://reviews.llvm.org/D12831 in review (waiting on Windows results for that) adds a test event stream that supports pluggable test event formatters. The first formatter I've added is JUnit/XUnit output. That's to support typical JUnit/XUnit output handling built into most commercial and

Re: [lldb-dev] Digging into Linux unexpected successes

2015-09-15 Thread Siva Chandra via lldb-dev
On Tue, Sep 15, 2015 at 9:59 AM, Tamas Berghammer wrote: > I think downloading data with the json API won't help because it will only > list the failures displayed on the Web UI what don't contain full test > names and don't contain info about the UnexpectedSuccess-es. If you want to > download i

Re: [lldb-dev] Digging into Linux unexpected successes

2015-09-15 Thread Tamas Berghammer via lldb-dev
Unfortunately the GCE logs aren't public at the moment and the amount of them isn't make it easy to make them accessible in any way (~30MB/build) and they aren't much more machine parsable then the stdout from the build. I think downloading data with the json API won't help because it will only li

Re: [lldb-dev] Digging into Linux unexpected successes

2015-09-15 Thread Siva Chandra via lldb-dev
IIRC, doing it from Python is straightforward and simple: json.load(urlparse.urlopen(<...>)) Could be a little more, but should not be much. On Tue, Sep 15, 2015 at 9:52 AM, Todd Fiala wrote: > Yep looks like there's a decent interface to it. Thanks, Siva! > > I see there's some docs here too

Re: [lldb-dev] Digging into Linux unexpected successes

2015-09-15 Thread Todd Fiala via lldb-dev
Yep looks like there's a decent interface to it. Thanks, Siva! I see there's some docs here too: http://docs.buildbot.net/current/index.html On Tue, Sep 15, 2015 at 9:42 AM, Siva Chandra wrote: > On Tue, Sep 15, 2015 at 9:25 AM, Todd Fiala via lldb-dev < > lldb-dev@lists.llvm.org> wrote: > >>

Re: [lldb-dev] Digging into Linux unexpected successes

2015-09-15 Thread Siva Chandra via lldb-dev
On Tue, Sep 15, 2015 at 9:25 AM, Todd Fiala via lldb-dev < lldb-dev@lists.llvm.org> wrote: > > The cmake builder runs in GCE and it uploads all test logs to Google > Cloud Storage (including full host logs and server logs). I used a python > script (running also in GCE) to download this data and t

Re: [lldb-dev] Digging into Linux unexpected successes

2015-09-15 Thread Todd Fiala via lldb-dev
> The cmake builder runs in GCE and it uploads all test logs to Google Cloud Storage (including full host logs and server logs). I used a python script (running also in GCE) to download this data and to parse the test output from the test traces. Are the GCE logs public? If not, do you know if ou

Re: [lldb-dev] Digging into Linux unexpected successes

2015-09-15 Thread Tamas Berghammer via lldb-dev
Yes, you are reading it correctly (for totclang we mean the totclang at the time when the test suit was run). The cmake builder runs in GCE and it uploads all test logs to Google Cloud Storage (including full host logs and server logs). I used a python script (running also in GCE) to download this

Re: [lldb-dev] Digging into Linux unexpected successes

2015-09-15 Thread Todd Fiala via lldb-dev
Just to make sure I'm reading these right: == Compiler: totclang Architecture: x86_64 == UnexpectedSuccess TestMiInterpreterExec.MiInterpreterExecTestCase.test_lldbmi_settings_set_target_run_args_before (250/250 100.00%) TestRaise.RaiseTestCase.test_restart_bug_with_dwarf (119

Re: [lldb-dev] Digging into Linux unexpected successes

2015-09-15 Thread Todd Fiala via lldb-dev
Wow Tamas, this is perfect. Thanks for pulling that together! Don't worry about the bigger file. Thanks much. -Todd On Tue, Sep 15, 2015 at 8:56 AM, Tamas Berghammer wrote: > I created a new statistic what separates the data based on compiler and > architecture and I also extended it to the

Re: [lldb-dev] Digging into Linux unexpected successes

2015-09-15 Thread Todd Fiala via lldb-dev
On Tue, Sep 15, 2015 at 2:57 AM, Tamas Berghammer wrote: > Hi Todd, > > I attached the statistic of the last 100 test run on the Linux x86_64 > builder (http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-cmake). > The data might be a little bit noisy because of the actual test failures >

Re: [lldb-dev] Digging into Linux unexpected successes

2015-09-15 Thread Tamas Berghammer via lldb-dev
Hi Todd, I attached the statistic of the last 100 test run on the Linux x86_64 builder (http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-cmake). The data might be a little bit noisy because of the actual test failures happening because of a temporary regression, but they should give you