Hi everyone,
The Talos regression detection emails caught a number of regressions
during the Monday uplift (see [1] for Aurora and [2] for Beta
regressions). To put things into perspective, I prepared a spreadsheet
of the most notable performance regressions [3] (and please do take a
look at the spreadsheet!).
The way the current situation happens is that many of the developers
ignore the Talos regression emails that go to dev-tree-management, and
in many cases regressions of a few percents slide in without being
tracked. This trend of relatively big performance regressions becomes
more evident every time we do an uplift, which means that 6 weeks worth
of development get compared to the previous version.
A few people (myself included) have tried to go through these emails and
notify the people responsible in the past. This process has proved to
be ineffective, because (1) the problem is not officially owned by
anyone (currently the only person going through those emails is
mbrubeck), and (2) because of problems such as the difficulty of
diagnosing and reproducing performance regressions, many people think
that their patches are unlikely to have caused a regression, and
therefore no investigation gets done.
Some people have noted in the past that some Talos measurements are not
representative of something that the users would see, the Talos numbers
are noisy, and we don't have good tools to deal with these types of
regressions. There might be some truth to all of these, but I believe
that the bigger problem is that nobody owns watching over these numbers,
and as a result as take regressions in some benchmarks which can
actually be representative of what our users experience.
I don't believe that the current situation is acceptable, especially
with the recent focus on performance (through the Snappy project), and I
would like to ask people if they have any ideas on what we can do to fix
this. The fix might be turning off some Talos tests if they're really
not useful, asking someone or a group of people to go over these test
results, get better tools with them, etc. But _something_ needs to
happen here.
Cheers,
Ehsan
[1]
<https://groups.google.com/forum/?fromgroups=#!topic/mozilla.dev.tree-management/QYvG8wIen6Y>
[2]
<https://groups.google.com/forum/?fromgroups=#!topic/mozilla.dev.tree-management/Sh62iFF4GtY>
[3]
<https://docs.google.com/spreadsheet/pub?key=0AuDE0NKbf0EOdF9uRi1qa1hwNFhneEFLcUt2TzI3WXc&single=true&gid=0&output=html>
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
