Sean Whitton <spwhit...@spwhitton.name> writes: > I agree with the principle that test failures should be RC by default.
This is something which seems to have no disagreement here. My concern is just that I want to have a simple way to override this, to assign this to a different package etc. I want to have the same flexibility here as for bugs. > I think we need an additional field in d/tests/control to mark > individual tests as non-critical (this wouldn't really help Ole's 9000 > tests package though). While this is a really large test suite, it also wouldn't help for others, since there are many packages with > 100 tests: aplpy has ~250 tests, astroml has ~210 tests, gnudatalanguage has ~170 tests etc. Even when the package itself is rather "small", the test count is quite nice -- the github ecosystem with test coverage tools (and badges) helps here, as well as powerful Python test packages. Especially modern packages with a highly motivated upstream come with a large number of tests (and they care about failures). This makes autokgtest to a very valuable tool, but it shouldn't become a maintainer's nightmare. And if I would have to maintain test-specific severities, that would be a nightmare, not only for me but also for upstream, who would need to be involved as the knowledgable instance here. Best regards Ole
