wouldn’t abandoning stochastic testing mean that execution paths would no longer be tested and therefore increase issues found in production?
On Sun, Jan 20, 2019 at 5:13 AM Marco de Abreu <[email protected]> wrote: > Hello everyone, > > a few months ago, I have enabled test coverage recording for the MXNet > repository [1]. The next step would be to report the coverage changes to > pull requests so we can make them part of the review. For this, I've > created a pull request here at [2]. > > Before this feature can be enabled though, we have to stabilize the test > coverage. Due to the extensive usage of random inputs in our tests, the > execution takes different paths and thus, the coverage results undergo a > heavy variance. If we would enable the coverage report for PRs, they would > be marked as increased/decreased coverage without actually doing so. To get > a few examples, visit [3]. > > I'd like to provide some detailed examples or screenshots, but with big > reports [4], CodeCov's webinterface tends to time out. I currently have an > open ticket with CodeCov that will hopefully improve the situation soon, > but we have to live with the timeouts until then. > > My proposal to improve the situation would be to divide the work across the > community in two stages: > 1. Replace random inputs in tests with deterministic ones if applicable. > 2. If randomness cannot be prevented or this 'flakiness' persists, we could > run the same test multiple times with coverage, look at the coverage diff > and then write targeted unittests for the inconsistent parts. > > If there is more interest towards test coverage, I'd be happy to write a > guide that explains how to measure test coverage and detect flaky coverage > areas so everybody can help contribute towards a stable test coverage. > Please let me know what you think. > > Best regards, > Marco > > [1]: https://codecov.io/gh/apache/incubator-mxnet > [2]: https://github.com/apache/incubator-mxnet/pull/12648 > [3]: https://codecov.io/gh/apache/incubator-mxnet/pulls > [4]: https://codecov.io/gh/apache/incubator-mxnet/pull/13849 >
