I concur with Ryan here, and I'd add that IME 90% if not more of these
timeouts (where they are really timeouts because the test is long,
rather than just brokenness in the test that leaves it hanging until the
timeout) happen on debug/asan builds, where "perf regressions" isn't
really a meaningful concept to regression-analyze for, compared to the
debug and asan overhead.
~ Gijs
On 09/02/2016 17:54, Ryan VanderMeulen wrote:
I'd have a much easier time accepting that argument if my experience
didn't tell me that nearly every single "Test took longer than expected"
or "Test timed out" intermittent ends with a RequestLongerTimeout as the
fix.
-Ryan
On 2/9/2016 12:50 PM, Haik Aftandilian wrote:
On Tue, Feb 9, 2016 at 2:47 AM, Marco Bonardo <mbona...@mozilla.com>
wrote:
Based on that, bumping the timeout may have 2 downsides, long term:
- slower tests for everyone
- sooner or later 90 seconds won't be enough again. Are we going to
bump to
180 then?
Essentially restating Marco's concern, increasing timeouts has the side
effect where performance regressions are not noticed. i.e., a new bug
that
causes a test to take longer, but still pass, is not detected. With the
original lower timeouts, the test would fail with a timeout. So a little
bit of the value of the tests is lost, and it's difficult to address
later.
Haik
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
