Anything to make the test suite faster is a worthwhile effort. The speed up will be even better for those of us with slower dev systems. Getting the speed boost for in memory sqlite is a good start and Django is much more than an ORM. It'll take work to improve the database isolation for the test suite, but that is something that should probably happen regardless of parallelization.
I'm imagining database backends being able to control whether or not they support parallelization and if each process needs its own database. Regards, Michael Manfre On Fri, Feb 6, 2015 at 11:05 AM, Aymeric Augustin < aymeric.augus...@polytechnique.org> wrote: > Hello, > > As the test suite is growing, it’s getting slower. I’ve tried to make it > faster by running tests in parallel. > > The current state of my experiment is here: > https://github.com/django/django/pull/4063 > > I’m distributing the tests to workers with the multiprocessing module. > While the idea is simple, the unittest APIs make its implementation painful. > > ** Results ** > > Without the patch: > > Ran 9016 tests in 350.610s > ./runtests.py 355,86s user 20,48s system 92% cpu 6:48,23 total > > With the patch > > Ran 9016 tests in 125.778s > ./runtests.py --parallel 512,31s user 29,92s system 300% cpu 3:00,73 total > > Since it takes almost one minute to create databases, parallelization > makes the execution of tests go from 6 minutes to 2 minutes. > > This isn’t bad, but the x3 speedup is a bit disappointing given that I > have 4 physical / 8 logical cores. Perhaps the IPC is expensive. > > Does anyone have insights about scaling with multiprocessing? > > ** Limitations ** > > This technique works well with in-memory SQLite databases. Each process > gets its own copy of the database in its memory space. > > It fails with on-disk SQLite databases. SQLite can’t cope with this level > of concurrency. It timeouts while attempting to lock the database. > > It fails with PostgreSQL (and, I assume, other databases) because tests > collide, for instance when they attempt to load the same fixture. > > ** Next steps ** > > At this point, the patch improves the common use case of running > `./runtests.py` locally to check a database-independent change, and little > else. > > Do you think it would be useful to include it in Django anyway? Do you > have concerns about the implementation? Charitably, I’ll say that “it > works”… > > Releasing it separately as a custom test runner may be more appropriate. > > What do you think? > > -- > Aymeric. > > > > -- > You received this message because you are subscribed to the Google Groups > "Django developers (Contributions to Django itself)" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to django-developers+unsubscr...@googlegroups.com. > To post to this group, send email to django-developers@googlegroups.com. > Visit this group at http://groups.google.com/group/django-developers. > To view this discussion on the web visit > https://groups.google.com/d/msgid/django-developers/639C2955-7AAB-4BC6-940D-EA69F7F51280%40polytechnique.org > <https://groups.google.com/d/msgid/django-developers/639C2955-7AAB-4BC6-940D-EA69F7F51280%40polytechnique.org?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAGdCwBujE%2BJSs%2BiupVxWCcTKV5VcT7%3DmZw71DnaVODQDZ%3DqxuA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
