Yak shaving the test framework on the way to pluggable user models (#3011)
Hi all, So, I've been working on a Django branch [1] to implement the approach to pluggable user models that was decided upon earlier this year [2] [1] https://github.com/freakboy3742/django/tree/t3011 [2] https://code.djangoproject.com/wiki/ContribAuthImprovements The user-swapping code itself is coming together well. However, I've hit a snag during the process of writing tests that may require a little yak shaving. The problem is this: With pluggable auth models, you lose the certainty over exactly which User model is present, which makes it much harder to write some tests, especially ones that will continue to work in an unpredictable end-user testing environment. With regards to pluggable Users, there are 5 types of tests that can be run: 1) Contrib.auth tests that validate that the internals of contrib.auth work as expected 2) Contrib.auth tests that validate that the internals of contrib.auth work with a custom User model 3) Contrib.auth tests that validate that the currently specified user model meets the requirements of the User contract The problem is that because of the way syncdb works during testing some of these tests are effectively mutually exclusive. The test framework runs syncdb at the start of the test run, which sets up the models that will be available for the duration of testing -- which then constrains the tests that can actually be run. This doesn't affect the Django core tests so much; Django's tests will synchronise auth.User by default, which allows tests of type 1 to run. It can also provide a custom User model, and use @override_settings to swap in that model as required for tests of type 2. Tests of type 3 are effectively integration tests which will pass with *any* interface compliant User model. However, if I have my own project that includes contrib.auth in INSTALLED_APPS, ./manage.py test will attempt to run *all* the tests from contrib.auth. If I have a custom User model in play, that means that the tests of type 1 *can't* pass, because auth.User won't be synchronised to the database. I can't even use @override_settings to force auth.User into use -- the opportunity for syncdb to pick up auth.User has passed. We *could* just mark the affected tests that require auth.User as "skipUnless(user model == auth.User)", but that would mean some projects would run the tests, and some wouldn't. That seems like an odd inconsistency to me -- the tests either should be run, or they shouldn't. In thinking about this problem, it occurred to me that what is needed is for us to finally solve an old problem with Django's testing -- the fact that there is a difference between different types of tests. There are tests in auth.User that Django's core team needs to run before we cut a release, and there are integration tests that validate that when contrib.auth is deployed in your own project, that it will operate as designed. The internal tests need to run against a clean, known environment; integration tests must run against your project's native environment. Thinking more broadly, there may be other categories -- "smoke tests" for quick sanity check that a system is working; "Interaction tests" that run live browser tests; and so on. Python's unittest library contains the concept of Suites, which seems to me like a reasonable analog of what I'm talking about here. What is missing is a syntax for executing those suites, and maybe some helpers to make it easier to build those suites in the first place. I don't have a concrete proposal at this point (beyond the high level idea that suites seem like the way to handle this). This is an attempt to feel out community opinion about the problem as a whole. I know there are efforts underway to modify Django's test discovery mechanism (#17365), and there might be some overlap here. There are also a range of tickets relating to controlling the test execution process (#9156, #11593), so there has been plenty of thought put into this general problem in the past. If anyone has any opinions or alternate proposals, I'd like to hear them. Yours, Russ Magee %-) -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Yak shaving the test framework on the way to pluggable user models (#3011)
On 25 août 2012, at 10:15, Russell Keith-Magee wrote: > We *could* just mark the affected tests that require auth.User as > "skipUnless(user model == auth.User)", but that would mean some > projects would run the tests, and some wouldn't. That seems like an > odd inconsistency to me -- the tests either should be run, or they > shouldn't. FWIW it doesn't seem odd to me. If a project doesn't use Django's built-in User model then you don't need to test it in that project's test suite. Indeed, running django.contrib.* tests within a project can fail in a variety of interesting ways. Trying to isolate them sufficiently is an endless quest. Generally speaking, I like the idea to identify various types of test. The only downside I can imagine is the risk that some categories lose meaning over time, leading to situations similar to modeltests vs. regressiontests today. Best regards, -- Aymeric. -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Ticket #18855: persist socket in runserver
First time contributing to Django, but I've used it for a few years now. I'd like to enhance to the dev runserver to persist a socket through autoreloads. I've submitted a ticket and a pull request on Github is linked in the ticket. If someone wouldn't mind reviewing the code and giving me some feedback, I'd like to help in any way I can to get this into Django. With this change, the default behavior of runserver is changed, but the user is allowed to opt-out passing "--nopersistsocket" to runserver. Thanks for your time. - Dan LaMotte -- You received this message because you are subscribed to the Google Groups "Django developers" group. To view this discussion on the web visit https://groups.google.com/d/msg/django-developers/-/veb_dBZ_YNoJ. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Ticket #18855: persist socket in runserver
On 25 elo, 19:10, Dan LaMotte wrote: > First time contributing to Django, but I've used it for a few years now. > I'd like to enhance to the dev runserver to persist a socket through > autoreloads. I've submitted a ticket and a pull request on Github is > linked in the ticket. > > If someone wouldn't mind reviewing the code and giving me some feedback, > I'd like to help in any way I can to get this into Django. > > With this change, the default behavior of runserver is changed, but the > user is allowed to opt-out passing "--nopersistsocket" to runserver. I added some comments to the ticket. In short: yes to the idea, but with some concerns about how to implement this safely on all supported platforms, or adding new annoying reload problems. BTW there is no need to post to django-developers immediately after creating a ticket. If it seems the ticket didn't get any attention (give at least a couple of days for the devs to react), or you have reason to believe the ticket is Really Important (a release blocker just before release), then post here. Of course, first time participants to django-developers are always welcome :) Thanks for working on this, this should be a nice feature addition to runserver assuming the issues mentioned in the ticket can be solved. - Anssi -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
ORM refactoring, part 2
I have done some more ORM refactoring work. I thought it would be a good idea to post a summary of what is going on. First, I haven't committed the utils.tree refactoring patch I was planning to commit [https://github.com/akaariai/django/commits/ refactor_utils_tree]. The reason is that I now feel that add_filter() and deeper levels of the Query class need some refactorings first. There is some work going on in the refactoring of the deeper levels of the ORM. I committed join promotion improvement patch today: [https:// code.djangoproject.com/ticket/16715#comment:25]. I have more planned improvements to commit: * Remove "dupe avoidance" logic from the ORM (#18748). This is removing some code I strongly believe isn't needed any more. This is still pending Malcolm's review. * Fix join promotion in disjunctive filters (#18854). This one should make join promotion in disjunction cases cleaner and more efficient (as in less OUTER JOINS). In addition, this will make add_q/add_filter a bit easier to understand. The problem currently is that the first ORed filter is added to the query with connector=AND, the rest of the filters in the disjunction with connector=OR. This is done for join promotion reasons. However, this is very confusing when trying to work with add_q and add_filter, and doesn't actually work that well (see the added tests in the ticket's patch). * Remove "trim" argument from add_filter (#18816). The trim argument is only needed for split_exclude (that is, pushing negated m2m filters to subqueries). So, add_filter (and then also trim_joins) needs to handle split_exclude's special case. Handling this case inside split_exclude() turned out to be a little ugly, but IMO less ugly than the trim flag. This also allows further cleanups in the following item. * A biggie: complete refactoring of setup_joins (#10790, a separate ticket could be good for this). Best described in the last post in the ticket [https:// code.djangoproject.com/ticket/10790#comment:41]. The patch isn't completely ready (there are still stale comments floating around, the commit history is ugly). Applying the above patches should make the ORM code easier to follow. Further features like custom lookups should be easier to add. The produced queries should be of higher quality (less joins, less left joins). And, it should be easier to do cleanups in the ORM. A note about extra_filters: The setup_joins() refactoring removes the "extra_filters" system used for generic relations. The system adds new filters to the query for any joins generated by generic relations. However, these are pushed into the WHERE clause, and this doesn't work nicely with LOUTER JOINS. The refactoring adds the extra condition directly into the join's ON clause, thus producing correct left outer join conditions. The extra_filters is (as far as I know) private API, but I'd like to know if this is heavily used by the community. If so, it is easy enough to leave this API in place (while still fixing the generic relations stuff). I hope I can get reviews for the above tickets. Getting reviews from people who do not know the ORM is valuable, too, as one of the goals is to make the ORM easier to understand. As the author I can't easily see if my attempts to make the code easier to follow actually improve anything. Even if I do not get any reviews, I think it is a good idea to push these patches in. Thus far it has been hard to get reviews for larger ORM patches, and I am afraid that the refactoring work will stall if I have to wait for a full review for every patch. If you want to review the patches, but don't have time just now, please make a note in the tickets about this. There is no hurry. If pushing these patches without reviews seems like a bad idea to you, then please say so (preferably before I commit anything)... I am sorry if I haven't worked on other patches I thought I had time to work on. The core ORM refactorings are IMO really important to work on, and thus they have bypassed some other items in my admittedly too long TODO list. - Anssi -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: ORM refactoring, part 2
On Sat, Aug 25, 2012 at 12:35 PM, Anssi Kääriäinen wrote: > I have done some more ORM refactoring work. I thought it would be a > good idea to post a summary of what is going on. > > First, I haven't committed the utils.tree refactoring patch I was > planning to commit [https://github.com/akaariai/django/commits/ > refactor_utils_tree]. The reason is that I now feel that add_filter() > and deeper levels of the Query class need some refactorings first. > > There is some work going on in the refactoring of the deeper levels of > the ORM. I committed join promotion improvement patch today: [https:// > code.djangoproject.com/ticket/16715#comment:25]. I have more planned > improvements to commit: > > > * Remove "dupe avoidance" logic from the ORM (#18748). > This is removing some code I strongly believe isn't needed any more. > This is still pending Malcolm's review. > > * Fix join promotion in disjunctive filters (#18854). > This one should make join promotion in disjunction cases cleaner and > more efficient (as in less OUTER JOINS). In addition, this will make > add_q/add_filter a bit easier to understand. The problem currently is > that the first ORed filter is added to the query with connector=AND, > the rest of the filters in the disjunction with connector=OR. This is > done for join promotion reasons. However, this is very confusing when > trying to work with add_q and add_filter, and doesn't actually work > that well (see the added tests in the ticket's patch). > > * Remove "trim" argument from add_filter (#18816). > The trim argument is only needed for split_exclude (that is, pushing > negated m2m filters to subqueries). So, add_filter (and then also > trim_joins) needs to handle split_exclude's special case. Handling > this case inside split_exclude() turned out to be a little ugly, but > IMO less ugly than the trim flag. This also allows further cleanups in > the following item. > > * A biggie: complete refactoring of setup_joins (#10790, a separate > ticket could be good for this). > Best described in the last post in the ticket [https:// > code.djangoproject.com/ticket/10790#comment:41]. The patch isn't > completely ready (there are still stale comments floating around, the > commit history is ugly). > > > Applying the above patches should make the ORM code easier to follow. > Further features like custom lookups should be easier to add. The > produced queries should be of higher quality (less joins, less left > joins). And, it should be easier to do cleanups in the ORM. > > A note about extra_filters: The setup_joins() refactoring removes the > "extra_filters" system used for generic relations. The system adds new > filters to the query for any joins generated by generic relations. > However, these are pushed into the WHERE clause, and this doesn't work > nicely with LOUTER JOINS. The refactoring adds the extra condition > directly into the join's ON clause, thus producing correct left outer > join conditions. > > The extra_filters is (as far as I know) private API, but I'd like to > know if this is heavily used by the community. If so, it is easy > enough to leave this API in place (while still fixing the generic > relations stuff). > > I hope I can get reviews for the above tickets. Getting reviews from > people who do not know the ORM is valuable, too, as one of the goals > is to make the ORM easier to understand. As the author I can't easily > see if my attempts to make the code easier to follow actually improve > anything. > > Even if I do not get any reviews, I think it is a good idea to push > these patches in. Thus far it has been hard to get reviews for larger > ORM patches, and I am afraid that the refactoring work will stall if I > have to wait for a full review for every patch. If you want to review > the patches, but don't have time just now, please make a note in the > tickets about this. There is no hurry. > > If pushing these patches without reviews seems like a bad idea to you, > then please say so (preferably before I commit anything)... > > I am sorry if I haven't worked on other patches I thought I had time > to work on. The core ORM refactorings are IMO really important to work > on, and thus they have bypassed some other items in my admittedly too > long TODO list. > > - Anssi > > -- > You received this message because you are subscribed to the Google Groups > "Django developers" group. > To post to this group, send email to django-developers@googlegroups.com. > To unsubscribe from this group, send email to > django-developers+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/django-developers?hl=en. > > Anssi, Thanks for all your awesome work on this, of the patches I've reviewed so far they've been really good. I don't think I'll have time to review until DjangoCon, so feel free to either commit now or wait, whichever you'd prefer! Alex -- "I disapprove of what you say, but I will defend to the death your right to
Re: Streaming HttpResponse revisted. Any core devs, please take a look if you can :)
Hi Tai, Thanks for your work and sorry for the late answer. I just had the time to review the discussion. In my opinion, option 2 is reasonable. I'm -0 on a stream_content keyword argument for HttpResponse. I +1 on adding a separate HttpStreamingResponse class, because it better reflects the difference in behavior, and because it makes it possible to define a different API from the regular HttpResponse (for instance, we can chose not to define response.content). Then we can use a deprecation path to remove support for streaming responses in the regular HttpResponse class. Once the deprecation path completes, I expect that: - HttpResponse will be simpler and more predictable (as it'll no longer contain code to support streaming) - both HttpResponse and HttpStreamingResponse will have intuitive and well-behaved APIs (which isn't true right now) - the implementations will be easier to understand and to maintain — HttpResponse was one of the hardest classes to port to Python 3, so much that I created a ticket saying it needs a refactoring. At a high level, I think this is a good compromise. It is fully backwards compatible for people that don't use streaming — the vast majority of Django users. It provides a robust foundation for the minority who does. It takes a bit more work than your current patch :) In addition, I'm -1 on encoding include or exclude filters in the MIDDLEWARE_CLASSES setting. Chains of includes and excludes are never a good idea — anyone who's struggled with a complex Apache configuration knows what I'm talking about! Best regards, -- Aymeric. -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Streaming HttpResponse revisted. Any core devs, please take a look if you can :)
On 25 elo, 23:22, Aymeric Augustin wrote: > In my opinion, option 2 is reasonable. I'm -0 on a stream_content keyword > argument for HttpResponse. I +1 on adding a separate HttpStreamingResponse > class, because it better reflects the difference in behavior, and because it > makes it possible to define a different API from the regular HttpResponse > (for instance, we can chose not to define response.content). Then we can use > a deprecation path to remove support for streaming responses in the regular > HttpResponse class. I have been advocating for the stream_content approach simply because of possible problems in using HttpResponse subclasses in streaming situations. This is not a problem for me, as I usually have only HttpResponseForbidden and HttpResponseReverse in use, and those doesn't seem too useful with streaming content... However, if somebody has a heavily customized HttpResponse subclass in use, and needs to use it both in streaming and normal content situations, then that user might have some problems. Are there such uses for HttpResponse? There are more votes for the subclass approach. So, if no further opinions are raised lets go with that one. - Anssi -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Yak shaving the test framework on the way to pluggable user models (#3011)
On Sat, Aug 25, 2012 at 4:24 PM, Aymeric Augustin wrote: > On 25 août 2012, at 10:15, Russell Keith-Magee wrote: > >> We *could* just mark the affected tests that require auth.User as >> "skipUnless(user model == auth.User)", but that would mean some >> projects would run the tests, and some wouldn't. That seems like an >> odd inconsistency to me -- the tests either should be run, or they >> shouldn't. > > FWIW it doesn't seem odd to me. If a project doesn't use Django's > built-in User model then you don't need to test it in that project's > test suite. A possible miscommunication here -- I don't think it's odd that the tests wouldn't be run; I only think it would be odd to have those tests report as "skipped". It feels to me like they should be either run or not run, not reported as skipped. > Indeed, running django.contrib.* tests within a project can fail in > a variety of interesting ways. Trying to isolate them sufficiently > is an endless quest. Generally speaking, I like the idea to identify > various types of test. The only downside I can imagine is the risk > that some categories lose meaning over time, leading to situations > similar to modeltests vs. regressiontests today. Completely agreed. The bug tracker is filled with dozens of bugs (both open and closed) that follow the pattern of "this test can't pass if X is/isn't in settings". I'm not so concerned about the proliferation/confusion of categories, however. To my mind, there's only 2 types of tests that really matter. * Internal checks of system functionality. These need to run against a clean set of known settings. There might be multiple tests for different configurations (e.g., run this test when a cache backend is defined; now run it when a cache backend is *not* defined); but ultimately this is just "run with known settings", for different values of "known". * Integration tests. These need to run against against the currently active project settings. Others may want to add other categories (like the smoke and interaction tests that I mentioned), but we can accommodate that by providing flexibility -- we don't have to use them ourselves. Yours, Russ Magee %-) -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Yak shaving the test framework on the way to pluggable user models (#3011)
On Aug 25, 2012, at 1:15 AM, Russell Keith-Magee wrote: > Hi all, > > So, I've been working on a Django branch [1] to implement the approach > to pluggable user models that was decided upon earlier this year [2] > > [1] https://github.com/freakboy3742/django/tree/t3011 > [2] https://code.djangoproject.com/wiki/ContribAuthImprovements > > The user-swapping code itself is coming together well. However, I've > hit a snag during the process of writing tests that may require a > little yak shaving. > > The problem is this: With pluggable auth models, you lose the > certainty over exactly which User model is present, which makes it > much harder to write some tests, especially ones that will continue to > work in an unpredictable end-user testing environment. > > With regards to pluggable Users, there are 5 types of tests that can be run: > > 1) Contrib.auth tests that validate that the internals of > contrib.auth work as expected > > 2) Contrib.auth tests that validate that the internals of > contrib.auth work with a custom User model > > 3) Contrib.auth tests that validate that the currently specified user > model meets the requirements of the User contract > > The problem is that because of the way syncdb works during testing > some of these tests are effectively mutually exclusive. The test > framework runs syncdb at the start of the test run, which sets up the > models that will be available for the duration of testing -- which > then constrains the tests that can actually be run. > > This doesn't affect the Django core tests so much; Django's tests will > synchronise auth.User by default, which allows tests of type 1 to run. > It can also provide a custom User model, and use @override_settings to > swap in that model as required for tests of type 2. Tests of type 3 > are effectively integration tests which will pass with *any* interface > compliant User model. > > However, if I have my own project that includes contrib.auth in > INSTALLED_APPS, ./manage.py test will attempt to run *all* the tests > from contrib.auth. If I have a custom User model in play, that means > that the tests of type 1 *can't* pass, because auth.User won't be > synchronised to the database. I can't even use @override_settings to > force auth.User into use -- the opportunity for syncdb to pick up > auth.User has passed. > > We *could* just mark the affected tests that require auth.User as > "skipUnless(user model == auth.User)", but that would mean some > projects would run the tests, and some wouldn't. That seems like an > odd inconsistency to me -- the tests either should be run, or they > shouldn't. > > In thinking about this problem, it occurred to me that what is needed > is for us to finally solve an old problem with Django's testing -- the > fact that there is a difference between different types of tests. > There are tests in auth.User that Django's core team needs to run > before we cut a release, and there are integration tests that validate > that when contrib.auth is deployed in your own project, that it will > operate as designed. The internal tests need to run against a clean, > known environment; integration tests must run against your project's > native environment. > > Thinking more broadly, there may be other categories -- "smoke tests" > for quick sanity check that a system is working; "Interaction tests" > that run live browser tests; and so on. > > Python's unittest library contains the concept of Suites, which seems > to me like a reasonable analog of what I'm talking about here. What is > missing is a syntax for executing those suites, and maybe some helpers > to make it easier to build those suites in the first place. > > I don't have a concrete proposal at this point (beyond the high level > idea that suites seem like the way to handle this). This is an attempt > to feel out community opinion about the problem as a whole. I know > there are efforts underway to modify Django's test discovery mechanism > (#17365), and there might be some overlap here. There are also a range > of tickets relating to controlling the test execution process (#9156, > #11593), so there has been plenty of thought put into this general > problem in the past. If anyone has any opinions or alternate > proposals, I'd like to hear them. Perhaps some inspiration could be found in other testing frameworks. In particular, py.test has an interesting 'marking' feature [1] and I believe that Nose also has something similar. This allows to limit the execution of tests that have been marked with a particular tag, or in other terms, to dynamically specify which sub-suite to execute. The challenge is that everyone in the industry seems to have different definitions for what constitutes a smoke, unit, functional, integration or system test. So it's probably best to follow a flexible, non-authoritative approach as the meaning for those types of tests is so subjective. Therefore my suggestion would be to: - add a feature t
Re: Yak shaving the test framework on the way to pluggable user models (#3011)
Once #17365 (as per https://github.com/jezdez/django-discover-runner) is implemented it will be a lot cleaner, since that splits out 1 & 2 from 3. Lets call them Django, Django api compatible, and Django project api Three things hammered home for me that Carl's approach is the correct way to do tests: 1.) Deploying to Heroku which encourages a small virtualenv size to as small as possible for deployment. Including test code in deployment is daft. 2) The usefulness of 3rd party application test suites can be limited especially when developing against trunk or varying versions. I don't want to fork just to make their test suite pass if there's nothing wrong with the code base itself - just the tests. 3.) My project provided a custom middleware and auth backend which extended the auth functionality by adding sites to users, and authenticating against that (along with email usernames). It broke the django tests in all sorts of places. That's when I discovered Carl's talk and was an instant convert. If I'm developing django itself or to be compatible with the django api then I need to run the django test suite and ensure it passes with my custom app, but there are any number of ways that a project or even other applications can break the test suite so project tests should never be run against the django test suite. On Saturday, 25 August 2012 18:15:07 UTC+10, Russell Keith-Magee wrote: > > Hi all, > > So, I've been working on a Django branch [1] to implement the approach > to pluggable user models that was decided upon earlier this year [2] > > [1] https://github.com/freakboy3742/django/tree/t3011 > [2] https://code.djangoproject.com/wiki/ContribAuthImprovements > > The user-swapping code itself is coming together well. However, I've > hit a snag during the process of writing tests that may require a > little yak shaving. > > The problem is this: With pluggable auth models, you lose the > certainty over exactly which User model is present, which makes it > much harder to write some tests, especially ones that will continue to > work in an unpredictable end-user testing environment. > > With regards to pluggable Users, there are 5 types of tests that can be > run: > > 1) Contrib.auth tests that validate that the internals of > contrib.auth work as expected > > 2) Contrib.auth tests that validate that the internals of > contrib.auth work with a custom User model > > 3) Contrib.auth tests that validate that the currently specified user > model meets the requirements of the User contract > > The problem is that because of the way syncdb works during testing > some of these tests are effectively mutually exclusive. The test > framework runs syncdb at the start of the test run, which sets up the > models that will be available for the duration of testing -- which > then constrains the tests that can actually be run. > > This doesn't affect the Django core tests so much; Django's tests will > synchronise auth.User by default, which allows tests of type 1 to run. > It can also provide a custom User model, and use @override_settings to > swap in that model as required for tests of type 2. Tests of type 3 > are effectively integration tests which will pass with *any* interface > compliant User model. > > However, if I have my own project that includes contrib.auth in > INSTALLED_APPS, ./manage.py test will attempt to run *all* the tests > from contrib.auth. If I have a custom User model in play, that means > that the tests of type 1 *can't* pass, because auth.User won't be > synchronised to the database. I can't even use @override_settings to > force auth.User into use -- the opportunity for syncdb to pick up > auth.User has passed. > > We *could* just mark the affected tests that require auth.User as > "skipUnless(user model == auth.User)", but that would mean some > projects would run the tests, and some wouldn't. That seems like an > odd inconsistency to me -- the tests either should be run, or they > shouldn't. > > In thinking about this problem, it occurred to me that what is needed > is for us to finally solve an old problem with Django's testing -- the > fact that there is a difference between different types of tests. > There are tests in auth.User that Django's core team needs to run > before we cut a release, and there are integration tests that validate > that when contrib.auth is deployed in your own project, that it will > operate as designed. The internal tests need to run against a clean, > known environment; integration tests must run against your project's > native environment. > > Thinking more broadly, there may be other categories -- "smoke tests" > for quick sanity check that a system is working; "Interaction tests" > that run live browser tests; and so on. > > Python's unittest library contains the concept of Suites, which seems > to me like a reasonable analog of what I'm talking about here. What is > missing is a s