#33264: DiscoverRunner doesn't counts unexpectedSuccess as an error
---------------------------------------------+------------------------
Reporter: Baptiste Mispelon | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: 3.2
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
---------------------------------------------+------------------------
I noticed today that my CI test suite was green even though its console
output printed:
{{{
FAILED (expected failures=13, unexpected successes=1)
}}}
It turns out that Django's default `DiscoverRunner` does not count
"unexpected successes" as errors when deciding the return code of its
`test` management command. [1]
So even though it prints "FAILED", the command returns with an exit code
of 0 which my CI (correctly) interprets as a success.
The fix seems quite simple:
{{{#!diff
diff --git a/django/test/runner.py b/django/test/runner.py
index 09ac4e142a..2e36514922 100644
--- a/django/test/runner.py
+++ b/django/test/runner.py
@@ -875,7 +875,7 @@ class DiscoverRunner:
teardown_test_environment()
def suite_result(self, suite, result, **kwargs):
- return len(result.failures) + len(result.errors)
+ return len(result.failures) + len(result.errors) +
len(result.unexpectedSuccesses)
def _get_databases(self, suite):
databases = {}
}}}
But I wonder if that would be considered backwards incompatible.
I also think this is a pretty serious issue and I wonder if backports
would be considered for this.
[1]
https://github.com/django/django/blob/60503cc747eeda7c61bab02b71f8f55a733a6eea/django/test/runner.py#L877-L878
--
Ticket URL: <https://code.djangoproject.com/ticket/33264>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/052.dcfb6b0cb8e2b25700bb5403e593e084%40djangoproject.com.
