There are a few issues here which should be easy to address and a few other issues which are not so easy to address.
First off everybody who is interested in talos should read the wiki page: https://wiki.mozilla.org/Buildbot/Talos This explains where the code lives, what tests we run and on which branches and many other details on what is done to run talos. If you want to run talos locally, check out these instructions: https://wiki.mozilla.org/Buildbot/Talos#Running_locally_-_Source_Code Another concern I have read in this thread and have heard over the last few months is why are we even running these tests as they are old, irrelevant and nobody looks at them. A valid concern and something I have asked myself many times while working on Talos. I took it upon myself earlier this summer to find a developer who is a point of contact for each and every test we run. Then we figured out if the tests were relevant and testing things we care about. Many tests have been updated/added/disabled in the last couple months. A similar complaint is about the noise in the numbers and how we can realistically detect a regression or gain value. For minor regressions our current toolchain will not be very effective. A lot of work has been done to look into how we run tests, the tools we use and if we can apply different models to the numbers to gain more reliable data. Most of that work is documented in the Signal from Noise project: https://wiki.mozilla.org/Auto-tools/Projects/Signal_From_Noise. I encourage folks to join into the public meetings we have to learn more about how we are actually solving this problem. Back on subject, we want to detect regressions to the exact changeset as well as reducing our false positives that get mailed to dev.tree-management. There is probably no silver bullet or policy we can create today which will fix our problems. There is a big lag between a current patch's run of talos and when we get a notification in dev.tree-management. For large regressions this can be detected by visually looking at graph server (we have links to everything from tbpl), but for small regressions, you have to see this over time as a minor increase could look like the regular noise we have in our numbers. Coming from a talos tool maintainer perspective, I am committed to making talos easy to run and documented so we can all work on fixing regressions instead of offering sacrifices to the try server. When there are requests for features, fixes or test adjustments somebody on the A*Team usually will resolve it quickly. While this only solves some of the pain, it is a step in the right direction until Signal From Noise can come out and solve a large portion of the other problems. _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
