On Saturday, September 1, 2012 10:08:53 AM UTC-4, Ehsan Akhgari wrote: > On 12-08-31 4:03 PM, Chris AtLee wrote: > > > On 31/08/12 03:59 PM, Ehsan Akhgari wrote: > > >> On 12-08-31 11:45 AM, Chris AtLee wrote: > > >>> On 31/08/12 11:32 AM, Ehsan Akhgari wrote:> There are extremely > > >>> non-stable Talos tests, and relatively stable ones. > > >>> > Let's focus on the relatively stable ones. There are extremely > > >>> hard > > >>> > to diagnose performance regressions, and extremely easy ones (i.e., > > >>> > let's not wait on this lock, do this I/O, run this exponential > > >>> > algorithm, load tons of XUL/XBL when a window opens, etc.) We have > > >>> many > > >>> > great tools for the job, so not all regressions need to be treated > > >>> the > > >>> > same. > > >>> > > >>> What value do the extremely non-stable Talos tests have? Shouldn't we > > >>> stop running them if they're not giving useful information? > > >> > > >> Either that, or find some way of making them more stable, such as not > > >> measuring the wall clock time. > > > > > > Sure, that sounds like a great project. Until that's finished, is there > > > any value to running these suites, or are they expensive random number > > > generators? > > > > I think that is something that needs to be evaluated on a per-test > > per-platform basis, hopefully by someone who knows a bit about > > statistics. :-) > > > > Cheers, > > Ehsan
We are detecting regressions with this despite the large levels of noise. So while it might appear to be a waste of machine resources to some, Talos serves a purpose. Having people look at the results more frequently will solve many of the problems. I would say a handful of tests/counters on certain platforms are not very useful in the current way we are reporting numbers. _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
