On Fri, Jun 6, 2014 at 2:29 AM, James Graham <ja...@hoppipolla.co.uk> wrote: > * All the todo stuff is mixing concerns. It forces you into a mode of > test writing where properties of a single implementation are hardcoded > into the testcases. This isn't a huge problem when there is only a > single relevant implementation, but we do a lot of work on standards > where there are multiple implementations.
I agree with this. Ideally you would separate the test from the "these are the parts of the test we expect to fail". In practice I think this is somewhat hard though. For simple tests it'd be possible to load separate data which indicates "we currently consistently fail assertion 5, 7 and 12, and randomly fail assertion 4". But tests that perform thousands of assertions due to testing lots of combinations of features, this quickly gets hard to maintain and to update. Even worse is tests that perform an inconsistent number of assertions due to timing issues (for example tests of progress events). If we make this part of writing tests, that significantly ups the cost of writing tests, which is something we all seem to agree is one of the most important thing to avoid. I'd love to find a solution to this. In the meantime flipping todo()s into is()s when we send tests to W3C, and flipping is()s into todo()s when we import tests from W3C seems like a low-cost solution. / Jonas _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
