On 12-08-31 4:03 PM, Chris AtLee wrote:
On 31/08/12 03:59 PM, Ehsan Akhgari wrote:
On 12-08-31 11:45 AM, Chris AtLee wrote:
On 31/08/12 11:32 AM, Ehsan Akhgari wrote:> There are extremely
non-stable Talos tests, and relatively stable ones.
> Let's focus on the relatively stable ones. There are extremely
hard
> to diagnose performance regressions, and extremely easy ones (i.e.,
> let's not wait on this lock, do this I/O, run this exponential
> algorithm, load tons of XUL/XBL when a window opens, etc.) We have
many
> great tools for the job, so not all regressions need to be treated
the
> same.
What value do the extremely non-stable Talos tests have? Shouldn't we
stop running them if they're not giving useful information?
Either that, or find some way of making them more stable, such as not
measuring the wall clock time.
Sure, that sounds like a great project. Until that's finished, is there
any value to running these suites, or are they expensive random number
generators?
I think that is something that needs to be evaluated on a per-test
per-platform basis, hopefully by someone who knows a bit about
statistics. :-)
Cheers,
Ehsan
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
