request to reopen ticket #24522: --random test option
I'd like to ask folks if they would support reopening ticket #24522, which is to add a --random option to Django's test runner: https://code.djangoproject.com/ticket/24522 It was opened 6 years ago and closed as won't fix. I've been doing some simplifications of the test-suite related code in Django's DiscoverRunner, and I think a --random option could be added pretty simply. It seems like every project could benefit by running their tests in a random order. It would help to unearth hidden dependencies between tests. If I were to implement this, the --random option would accept an optional integer "seed" argument. If --random is passed with no seed argument, the seed would be generated randomly by choosing a random integer between 0 and some large integer. Whether or not an explicit seed value is passed, Django can log the seed value being used before running tests (if --random was passed), and DiscoverRunner can store the seed as an attribute like self.random_order_seed. That would give callers the option of doing something with the seed programmatically, if logging to stderr isn't enough. When --random is used, the order of classes can be randomized within their groups: https://docs.djangoproject.com/en/3.1/topics/testing/overview/#order-in-which-tests-are-executed and the order of methods within their class can also be randomized. The seed can be used by instantiating a Python random.Random() object with it: https://docs.python.org/3/library/random.html#random.Random and the random orders could be generated by calling methods of that now-deterministic object. --Chris -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAOTb1wd7YnjmLwS_Gvvj3iGOx_Oodp7Xv%2BwZb_tCJ2Qmy%2BHO5g%40mail.gmail.com.
Re: request to reopen ticket #24522: --random test option
On Friday, March 12, 2021 at 10:47:56 AM UTC+1 Florian Apolloner wrote: > Is the current order defined in any way -- if not one should consider it > randoms anyways and as such actually randomizing should not make much of a > difference. FWIW I am fine with making it the default and outputting the > seed at the end of the run so it can reproduced. I am curious if it finds > many remaining issues. > I would characterize it as "mostly" sorted -- I think. I believe unittest's test discovery is used by default, in conjunction with a couple large test groups that Django configures (`DiscoverRunner.reorder_by = (TestCase, SimpleTestCase)`). The unittest package's test discovery is documented as first sorting by file path: > Paths are sorted before being imported so that execution order is the same > even if the underlying file system’s ordering is not dependent on file name. https://docs.python.org/3/library/unittest.html#unittest.TestLoader.discover And within test cases, test methods are sorted by name. For example, see-- https://docs.python.org/3/library/unittest.html#unittest.TestLoader.getTestCaseNames and-- > Note: The order in which the various tests will be run is determined by > sorting the test method names with respect to the built-in ordering for > strings. https://docs.python.org/3/library/unittest.html#organizing-test-code I don't know about classes within modules. On Fri, Mar 12, 2021 at 1:53 AM Florian Apolloner wrote: > ... Having randomization in CI might be interesting, but then we probably > need a way to derive the salt from the PR number or so, because I think the > number of tests inside a PR is small enough that you probably do not see > isolation failures, but you really want to see progress in a PR without > random failures due to isolation. > The question of how to use seeds in conjunction with CI if randomization were the default is an interesting one (e.g. how to tell CI to repeat a particular seed). Certainly if randomization were the default, CI could still be configured to be deterministic by passing --no-random / --sorted or whatever the opt-out option would be called. But, regardless of the default, to get the full benefit I think you'd want at least one of the test runs in CI to be a random one. --Chris > -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAOTb1wfbbdFbauJXCtZwa_N7F-K5dMcB-Kw4EgLiwKgL_MRCsw%40mail.gmail.com.
request to reopen issue #23332 (using the dotted test name in test output)
Hi, I would like to reopen the following issue from two years ago to change the test name in Django's test output from unittest's default to the full "dotted name": https://code.djangoproject.com/ticket/23332 This would make rerunning failing tests easier because then the test name could simply be copy and pasted from the command-line as is. Tim told me that to reopen the issue, it needed to be discussed on this list first and consensus reached. See Tim's comment (as well as some of my reasoning for why I think the issue should be reopened) here: https://code.djangoproject.com/ticket/27255 I put together a patch here with tests so you can see the PR that I was planning to propose: https://github.com/cjerdonek/django/commits/dotted-name-test-output The approach is simply to subclass unittest.TextTestResult and override its getDescription() method to use the dotted name instead of unittest's current, less helpful format. Django already has an example of subclassing TextTestResult with its DebugSQLTextTestResult class. Thanks, --Chris -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/5ab3f23b-04c0-455e-8f33-7502e14f9fe9%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: request to reopen issue #23332 (using the dotted test name in test output)
On Tuesday, September 20, 2016 at 5:28:46 PM UTC-7, Tim Graham wrote: > > I updated the Python issue to ask whether or not there's consensus to make > the change there. Even if that issue proceeds, I guess it would be a nice > convenience for current versions of Python that won't receive the change. > http://bugs.python.org/issue22431 > Thanks for getting clarification on the Python issue, Tim. To expand on Tim's latter point, because changing this in Python wouldn't be considered a bug fix in Python, the earliest Python version that could have the change would be 3.7. If the time between 3.6 and 3.7 is the same as the expected time between 3.5 and 3.6 (see [1] and [2] for those schedules), then 3.7 would be released in March 2018. Django users using Python 2.x or 3.6 or lower would never see the change if it's changed in Python alone. --Chris [1]: https://www.python.org/dev/peps/pep-0478/ [2]: https://www.python.org/dev/peps/pep-0494/ > On Tuesday, September 20, 2016 at 8:16:27 PM UTC-4, Chris Jerdonek wrote: >> >> Hi, I would like to reopen the following issue from two years ago to >> change the test name in Django's test output from unittest's default to the >> full "dotted name": >> >> https://code.djangoproject.com/ticket/23332 >> >> This would make rerunning failing tests easier because then the test name >> could simply be copy and pasted from the command-line as is. >> >> Tim told me that to reopen the issue, it needed to be discussed on this >> list first and consensus reached. See Tim's comment (as well as some of my >> reasoning for why I think the issue should be reopened) here: >> >> https://code.djangoproject.com/ticket/27255 >> >> I put together a patch here with tests so you can see the PR that I was >> planning to propose: >> >> https://github.com/cjerdonek/django/commits/dotted-name-test-output >> >> The approach is simply to subclass unittest.TextTestResult and override >> its getDescription() method to use the dotted name instead of unittest's >> current, less helpful format. Django already has an example of subclassing >> TextTestResult with its DebugSQLTextTestResult class. >> >> Thanks, >> --Chris >> >> -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/77cfcd16-601a-461a-bc6f-dcae97c03caa%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
