Yak shaving the test framework on the way to pluggable user models (#3011)

2012-08-25 Thread Russell Keith-Magee
Hi all,

So, I've been working on a Django branch [1] to implement the approach
to pluggable user models that was decided upon earlier this year [2]

[1] https://github.com/freakboy3742/django/tree/t3011
[2] https://code.djangoproject.com/wiki/ContribAuthImprovements

The user-swapping code itself is coming together well. However, I've
hit a snag during the process of writing tests that may require a
little yak shaving.

The problem is this: With pluggable auth models, you lose the
certainty over exactly which User model is present, which makes it
much harder to write some tests, especially ones that will continue to
work in an unpredictable end-user testing environment.

With regards to pluggable Users, there are 5 types of tests that can be run:

 1) Contrib.auth tests that validate that the internals of
contrib.auth work as expected

 2) Contrib.auth tests that validate that the internals of
contrib.auth work with a custom User model

 3) Contrib.auth tests that validate that the currently specified user
model meets the requirements of the User contract

The problem is that because of the way syncdb works during testing
some of these tests are effectively mutually exclusive. The test
framework runs syncdb at the start of the test run, which sets up the
models that will be available for the duration of testing -- which
then constrains the tests that can actually be run.

This doesn't affect the Django core tests so much; Django's tests will
synchronise auth.User by default, which allows tests of type 1 to run.
It can also provide a custom User model, and use @override_settings to
swap in that model as required for tests of type 2. Tests of type 3
are effectively integration tests which will pass with *any* interface
compliant User model.

However, if I have my own project that includes contrib.auth in
INSTALLED_APPS, ./manage.py test will attempt to run *all* the tests
from contrib.auth. If I have a custom User model in play, that means
that the tests of type 1 *can't* pass, because auth.User won't be
synchronised to the database. I can't even use @override_settings to
force auth.User into use -- the opportunity for syncdb to pick up
auth.User has passed.

We *could* just mark the affected tests that require auth.User as
"skipUnless(user model == auth.User)", but that would mean some
projects would run the tests, and some wouldn't. That seems like an
odd inconsistency to me -- the tests either should be run, or they
shouldn't.

In thinking about this problem, it occurred to me that what is needed
is for us to finally solve an old problem with Django's testing -- the
fact that there is a difference between different types of tests.
There are tests in auth.User that Django's core team needs to run
before we cut a release, and there are integration tests that validate
that when contrib.auth is deployed in your own project, that it will
operate as designed. The internal tests need to run against a clean,
known environment; integration tests must run against your project's
native environment.

Thinking more broadly, there may be other categories -- "smoke tests"
for  quick sanity check that a system is working; "Interaction tests"
that run live browser tests; and so on.

Python's unittest library contains the concept of Suites, which seems
to me like a reasonable analog of what I'm talking about here. What is
missing is a syntax for executing those suites, and maybe some helpers
to make it easier to build those suites in the first place.

I don't have a concrete proposal at this point (beyond the high level
idea that suites seem like the way to handle this). This is an attempt
to feel out community opinion about the problem as a whole. I know
there are efforts underway to modify Django's test discovery mechanism
(#17365), and there might be some overlap here. There are also a range
of tickets relating to controlling the test execution process (#9156,
#11593), so there has been plenty of thought put into this general
problem in the past. If anyone has any opinions or alternate
proposals, I'd like to hear them.

Yours,
Russ Magee %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.



Re: Yak shaving the test framework on the way to pluggable user models (#3011)

2012-08-25 Thread Aymeric Augustin
On 25 août 2012, at 10:15, Russell Keith-Magee wrote:

> We *could* just mark the affected tests that require auth.User as
> "skipUnless(user model == auth.User)", but that would mean some
> projects would run the tests, and some wouldn't. That seems like an
> odd inconsistency to me -- the tests either should be run, or they
> shouldn't.

FWIW it doesn't seem odd to me. If a project doesn't use Django's
built-in User model then you don't need to test it in that project's
test suite.

Indeed, running django.contrib.* tests within a project can fail in
a variety of interesting ways. Trying to isolate them sufficiently
is an endless quest. Generally speaking, I like the idea to identify
various types of test. The only downside I can imagine is the risk
that some categories lose meaning over time, leading to situations
similar to modeltests vs. regressiontests today.

Best regards,

-- 
Aymeric.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.



Ticket #18855: persist socket in runserver

2012-08-25 Thread Dan LaMotte
First time contributing to Django, but I've used it for a few years now. 
 I'd like to enhance to the dev runserver to persist a socket through 
autoreloads.  I've submitted a ticket and a pull request on Github is 
linked in the ticket.

If someone wouldn't mind reviewing the code and giving me some feedback, 
I'd like to help in any way I can to get this into Django.

With this change, the default behavior of runserver is changed, but the 
user is allowed to opt-out passing "--nopersistsocket" to runserver.

Thanks for your time.
- Dan LaMotte

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/veb_dBZ_YNoJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.



Re: Ticket #18855: persist socket in runserver

2012-08-25 Thread Anssi Kääriäinen
On 25 elo, 19:10, Dan LaMotte  wrote:
> First time contributing to Django, but I've used it for a few years now.
>  I'd like to enhance to the dev runserver to persist a socket through
> autoreloads.  I've submitted a ticket and a pull request on Github is
> linked in the ticket.
>
> If someone wouldn't mind reviewing the code and giving me some feedback,
> I'd like to help in any way I can to get this into Django.
>
> With this change, the default behavior of runserver is changed, but the
> user is allowed to opt-out passing "--nopersistsocket" to runserver.

I added some comments to the ticket. In short: yes to the idea, but
with some concerns about how to implement this safely on all supported
platforms, or adding new annoying reload problems.

BTW there is no need to post to django-developers immediately after
creating a ticket. If it seems the ticket didn't get any attention
(give at least a couple of days for the devs to react), or you have
reason to believe the ticket is Really Important (a release blocker
just before release), then post here.

Of course, first time participants to django-developers are always
welcome :)

Thanks for working on this, this should be a nice feature addition to
runserver assuming the issues mentioned in the ticket can be solved.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.



ORM refactoring, part 2

2012-08-25 Thread Anssi Kääriäinen
I have done some more ORM refactoring work. I thought it would be a
good idea to post a summary of what is going on.

First, I haven't committed the utils.tree refactoring patch I was
planning to commit [https://github.com/akaariai/django/commits/
refactor_utils_tree]. The reason is that I now feel that add_filter()
and deeper levels of the Query class need some refactorings first.

There is some work going on in the refactoring of the deeper levels of
the ORM. I committed join promotion improvement patch today: [https://
code.djangoproject.com/ticket/16715#comment:25]. I have more planned
improvements to commit:


* Remove "dupe avoidance" logic from the ORM (#18748).
This is removing some code I strongly believe isn't needed any more.
This is still pending Malcolm's review.

* Fix join promotion in disjunctive filters (#18854).
This one should make join promotion in disjunction cases cleaner and
more efficient (as in less OUTER JOINS). In addition, this will make
add_q/add_filter a bit easier to understand. The problem currently is
that the first ORed filter is added to the query with connector=AND,
the rest of the filters in the disjunction with connector=OR. This is
done for join promotion reasons. However, this is very confusing when
trying to work with add_q and add_filter, and doesn't actually work
that well (see the added tests in the ticket's patch).

* Remove "trim" argument from add_filter (#18816).
The trim argument is only needed for split_exclude (that is, pushing
negated m2m filters to subqueries). So, add_filter (and then also
trim_joins) needs to handle split_exclude's special case. Handling
this case inside split_exclude() turned out to be a little ugly, but
IMO less ugly than the trim flag. This also allows further cleanups in
the following item.

* A biggie: complete refactoring of setup_joins (#10790, a separate
ticket could be good for this).
Best described in the last post in the ticket [https://
code.djangoproject.com/ticket/10790#comment:41]. The patch isn't
completely ready (there are still stale comments floating around, the
commit history is ugly).


Applying the above patches should make the ORM code easier to follow.
Further features like custom lookups should be easier to add. The
produced queries should be of higher quality (less joins, less left
joins). And, it should be easier to do cleanups in the ORM.

A note about extra_filters: The setup_joins() refactoring removes the
"extra_filters" system used for generic relations. The system adds new
filters to the query for any joins generated by generic relations.
However, these are pushed into the WHERE clause, and this doesn't work
nicely with LOUTER JOINS. The refactoring adds the extra condition
directly into the join's ON clause, thus producing correct left outer
join conditions.

The extra_filters is (as far as I know) private API, but I'd like to
know if this is heavily used by the community. If so, it is easy
enough to leave this API in place (while still fixing the generic
relations stuff).

I hope I can get reviews for the above tickets. Getting reviews from
people who do not know the ORM is valuable, too, as one of the goals
is to make the ORM easier to understand. As the author I can't easily
see if my attempts to make the code easier to follow actually improve
anything.

Even if I do not get any reviews, I think it is a good idea to push
these patches in. Thus far it has been hard to get reviews for larger
ORM patches, and I am afraid that the refactoring work will stall if I
have to wait for a full review for every patch. If you want to review
the patches, but don't have time just now, please make a note in the
tickets about this. There is no hurry.

If pushing these patches without reviews seems like a bad idea to you,
then please say so (preferably before I commit anything)...

I am sorry if I haven't worked on other patches I thought I had time
to work on. The core ORM refactorings are IMO really important to work
on, and thus they have bypassed some other items in my admittedly too
long TODO list.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.



Re: ORM refactoring, part 2

2012-08-25 Thread Alex Gaynor
On Sat, Aug 25, 2012 at 12:35 PM, Anssi Kääriäinen
wrote:

> I have done some more ORM refactoring work. I thought it would be a
> good idea to post a summary of what is going on.
>
> First, I haven't committed the utils.tree refactoring patch I was
> planning to commit [https://github.com/akaariai/django/commits/
> refactor_utils_tree]. The reason is that I now feel that add_filter()
> and deeper levels of the Query class need some refactorings first.
>
> There is some work going on in the refactoring of the deeper levels of
> the ORM. I committed join promotion improvement patch today: [https://
> code.djangoproject.com/ticket/16715#comment:25]. I have more planned
> improvements to commit:
>
>
> * Remove "dupe avoidance" logic from the ORM (#18748).
> This is removing some code I strongly believe isn't needed any more.
> This is still pending Malcolm's review.
>
> * Fix join promotion in disjunctive filters (#18854).
> This one should make join promotion in disjunction cases cleaner and
> more efficient (as in less OUTER JOINS). In addition, this will make
> add_q/add_filter a bit easier to understand. The problem currently is
> that the first ORed filter is added to the query with connector=AND,
> the rest of the filters in the disjunction with connector=OR. This is
> done for join promotion reasons. However, this is very confusing when
> trying to work with add_q and add_filter, and doesn't actually work
> that well (see the added tests in the ticket's patch).
>
> * Remove "trim" argument from add_filter (#18816).
> The trim argument is only needed for split_exclude (that is, pushing
> negated m2m filters to subqueries). So, add_filter (and then also
> trim_joins) needs to handle split_exclude's special case. Handling
> this case inside split_exclude() turned out to be a little ugly, but
> IMO less ugly than the trim flag. This also allows further cleanups in
> the following item.
>
> * A biggie: complete refactoring of setup_joins (#10790, a separate
> ticket could be good for this).
> Best described in the last post in the ticket [https://
> code.djangoproject.com/ticket/10790#comment:41]. The patch isn't
> completely ready (there are still stale comments floating around, the
> commit history is ugly).
>
>
> Applying the above patches should make the ORM code easier to follow.
> Further features like custom lookups should be easier to add. The
> produced queries should be of higher quality (less joins, less left
> joins). And, it should be easier to do cleanups in the ORM.
>
> A note about extra_filters: The setup_joins() refactoring removes the
> "extra_filters" system used for generic relations. The system adds new
> filters to the query for any joins generated by generic relations.
> However, these are pushed into the WHERE clause, and this doesn't work
> nicely with LOUTER JOINS. The refactoring adds the extra condition
> directly into the join's ON clause, thus producing correct left outer
> join conditions.
>
> The extra_filters is (as far as I know) private API, but I'd like to
> know if this is heavily used by the community. If so, it is easy
> enough to leave this API in place (while still fixing the generic
> relations stuff).
>
> I hope I can get reviews for the above tickets. Getting reviews from
> people who do not know the ORM is valuable, too, as one of the goals
> is to make the ORM easier to understand. As the author I can't easily
> see if my attempts to make the code easier to follow actually improve
> anything.
>
> Even if I do not get any reviews, I think it is a good idea to push
> these patches in. Thus far it has been hard to get reviews for larger
> ORM patches, and I am afraid that the refactoring work will stall if I
> have to wait for a full review for every patch. If you want to review
> the patches, but don't have time just now, please make a note in the
> tickets about this. There is no hurry.
>
> If pushing these patches without reviews seems like a bad idea to you,
> then please say so (preferably before I commit anything)...
>
> I am sorry if I haven't worked on other patches I thought I had time
> to work on. The core ORM refactorings are IMO really important to work
> on, and thus they have bypassed some other items in my admittedly too
> long TODO list.
>
>  - Anssi
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To post to this group, send email to django-developers@googlegroups.com.
> To unsubscribe from this group, send email to
> django-developers+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-developers?hl=en.
>
> Anssi,

Thanks for all your awesome work on this, of the patches I've reviewed so
far they've been really good. I don't think I'll have time to review until
DjangoCon, so feel free to either commit now or wait, whichever you'd
prefer!

Alex


-- 
"I disapprove of what you say, but I will defend to the death your right to

Re: Streaming HttpResponse revisted. Any core devs, please take a look if you can :)

2012-08-25 Thread Aymeric Augustin
Hi Tai,

Thanks for your work and sorry for the late answer. I just had the time to 
review the discussion.


In my opinion, option 2 is reasonable. I'm -0 on a stream_content keyword 
argument for HttpResponse. I +1 on adding a separate HttpStreamingResponse 
class, because it better reflects the difference in behavior, and because it 
makes it possible to define a different API from the regular HttpResponse (for 
instance, we can chose not to define response.content).  Then we can use a 
deprecation path to remove support for streaming responses in the regular 
HttpResponse class.

Once the deprecation path completes, I expect that:
- HttpResponse will be simpler and more predictable (as it'll no longer contain 
code to support streaming)
- both HttpResponse and HttpStreamingResponse will have intuitive and 
well-behaved APIs (which isn't true right now)
- the implementations will be easier to understand and to maintain — 
HttpResponse was one of the hardest classes to port to Python 3, so much that I 
created a ticket saying it needs a refactoring.

At a high level, I think this is a good compromise. It is fully backwards 
compatible for people that don't use streaming — the vast majority of Django 
users. It provides a robust foundation for the minority who does. It takes a 
bit more work than your current patch :)


In addition, I'm -1 on encoding include or exclude filters in the 
MIDDLEWARE_CLASSES setting. Chains of includes and excludes are never a good 
idea — anyone who's struggled with a complex Apache configuration knows what 
I'm talking about!


Best regards,

-- 
Aymeric.


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.



Re: Streaming HttpResponse revisted. Any core devs, please take a look if you can :)

2012-08-25 Thread Anssi Kääriäinen
On 25 elo, 23:22, Aymeric Augustin
 wrote:
> In my opinion, option 2 is reasonable. I'm -0 on a stream_content keyword 
> argument for HttpResponse. I +1 on adding a separate HttpStreamingResponse 
> class, because it better reflects the difference in behavior, and because it 
> makes it possible to define a different API from the regular HttpResponse 
> (for instance, we can chose not to define response.content).  Then we can use 
> a deprecation path to remove support for streaming responses in the regular 
> HttpResponse class.

I have been advocating for the stream_content approach simply because
of possible problems in using HttpResponse subclasses in streaming
situations. This is not a problem for me, as I usually have only
HttpResponseForbidden and HttpResponseReverse in use, and those
doesn't seem too useful with streaming content...  However, if
somebody has a heavily customized HttpResponse subclass in use, and
needs to use it both in streaming and normal content situations, then
that user might have some problems. Are there such uses for
HttpResponse?

There are more votes for the subclass approach. So, if no further
opinions are raised lets go with that one.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.



Re: Yak shaving the test framework on the way to pluggable user models (#3011)

2012-08-25 Thread Russell Keith-Magee
On Sat, Aug 25, 2012 at 4:24 PM, Aymeric Augustin
 wrote:
> On 25 août 2012, at 10:15, Russell Keith-Magee wrote:
>
>> We *could* just mark the affected tests that require auth.User as
>> "skipUnless(user model == auth.User)", but that would mean some
>> projects would run the tests, and some wouldn't. That seems like an
>> odd inconsistency to me -- the tests either should be run, or they
>> shouldn't.
>
> FWIW it doesn't seem odd to me. If a project doesn't use Django's
> built-in User model then you don't need to test it in that project's
> test suite.

A possible miscommunication here -- I don't think it's odd that the
tests wouldn't be run; I only think it would be odd to have those
tests report as "skipped". It feels to me like they should be either
run or not run, not reported as skipped.

> Indeed, running django.contrib.* tests within a project can fail in
> a variety of interesting ways. Trying to isolate them sufficiently
> is an endless quest. Generally speaking, I like the idea to identify
> various types of test. The only downside I can imagine is the risk
> that some categories lose meaning over time, leading to situations
> similar to modeltests vs. regressiontests today.

Completely agreed. The bug tracker is filled with dozens of bugs (both
open and closed) that follow the pattern of "this test can't pass if X
is/isn't in settings".

I'm not so concerned about the proliferation/confusion of categories,
however. To my mind, there's only 2 types of tests that really matter.

 * Internal checks of system functionality. These need to run against
a clean set of known settings. There might be multiple tests for
different configurations (e.g., run this test when a cache backend is
defined; now run it when a cache backend is *not* defined); but
ultimately this is just "run with known settings", for different
values of "known".

 * Integration tests. These need to run against against the currently
active project settings.

Others may want to add other categories (like the smoke and
interaction tests that I mentioned), but we can accommodate that by
providing flexibility -- we don't have to use them ourselves.

Yours,
Russ Magee %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.



Re: Yak shaving the test framework on the way to pluggable user models (#3011)

2012-08-25 Thread Julien Phalip

On Aug 25, 2012, at 1:15 AM, Russell Keith-Magee wrote:

> Hi all,
> 
> So, I've been working on a Django branch [1] to implement the approach
> to pluggable user models that was decided upon earlier this year [2]
> 
> [1] https://github.com/freakboy3742/django/tree/t3011
> [2] https://code.djangoproject.com/wiki/ContribAuthImprovements
> 
> The user-swapping code itself is coming together well. However, I've
> hit a snag during the process of writing tests that may require a
> little yak shaving.
> 
> The problem is this: With pluggable auth models, you lose the
> certainty over exactly which User model is present, which makes it
> much harder to write some tests, especially ones that will continue to
> work in an unpredictable end-user testing environment.
> 
> With regards to pluggable Users, there are 5 types of tests that can be run:
> 
> 1) Contrib.auth tests that validate that the internals of
> contrib.auth work as expected
> 
> 2) Contrib.auth tests that validate that the internals of
> contrib.auth work with a custom User model
> 
> 3) Contrib.auth tests that validate that the currently specified user
> model meets the requirements of the User contract
> 
> The problem is that because of the way syncdb works during testing
> some of these tests are effectively mutually exclusive. The test
> framework runs syncdb at the start of the test run, which sets up the
> models that will be available for the duration of testing -- which
> then constrains the tests that can actually be run.
> 
> This doesn't affect the Django core tests so much; Django's tests will
> synchronise auth.User by default, which allows tests of type 1 to run.
> It can also provide a custom User model, and use @override_settings to
> swap in that model as required for tests of type 2. Tests of type 3
> are effectively integration tests which will pass with *any* interface
> compliant User model.
> 
> However, if I have my own project that includes contrib.auth in
> INSTALLED_APPS, ./manage.py test will attempt to run *all* the tests
> from contrib.auth. If I have a custom User model in play, that means
> that the tests of type 1 *can't* pass, because auth.User won't be
> synchronised to the database. I can't even use @override_settings to
> force auth.User into use -- the opportunity for syncdb to pick up
> auth.User has passed.
> 
> We *could* just mark the affected tests that require auth.User as
> "skipUnless(user model == auth.User)", but that would mean some
> projects would run the tests, and some wouldn't. That seems like an
> odd inconsistency to me -- the tests either should be run, or they
> shouldn't.
> 
> In thinking about this problem, it occurred to me that what is needed
> is for us to finally solve an old problem with Django's testing -- the
> fact that there is a difference between different types of tests.
> There are tests in auth.User that Django's core team needs to run
> before we cut a release, and there are integration tests that validate
> that when contrib.auth is deployed in your own project, that it will
> operate as designed. The internal tests need to run against a clean,
> known environment; integration tests must run against your project's
> native environment.
> 
> Thinking more broadly, there may be other categories -- "smoke tests"
> for  quick sanity check that a system is working; "Interaction tests"
> that run live browser tests; and so on.
> 
> Python's unittest library contains the concept of Suites, which seems
> to me like a reasonable analog of what I'm talking about here. What is
> missing is a syntax for executing those suites, and maybe some helpers
> to make it easier to build those suites in the first place.
> 
> I don't have a concrete proposal at this point (beyond the high level
> idea that suites seem like the way to handle this). This is an attempt
> to feel out community opinion about the problem as a whole. I know
> there are efforts underway to modify Django's test discovery mechanism
> (#17365), and there might be some overlap here. There are also a range
> of tickets relating to controlling the test execution process (#9156,
> #11593), so there has been plenty of thought put into this general
> problem in the past. If anyone has any opinions or alternate
> proposals, I'd like to hear them.

Perhaps some inspiration could be found in other testing frameworks. In 
particular, py.test has an interesting 'marking' feature [1] and I believe that 
Nose also has something similar. This allows to limit the execution of tests 
that have been marked with a particular tag, or in other terms, to dynamically 
specify which sub-suite to execute.

The challenge is that everyone in the industry seems to have different 
definitions for what constitutes a smoke, unit, functional, integration or 
system test. So it's probably best to follow a flexible, non-authoritative 
approach as the meaning for those types of tests is so subjective. Therefore my 
suggestion would be to:
- add a feature t

Re: Yak shaving the test framework on the way to pluggable user models (#3011)

2012-08-25 Thread Brett H
Once #17365 (as per https://github.com/jezdez/django-discover-runner) is 
implemented it will be a lot cleaner, since that splits out 1 & 2 from 3. 
Lets call them Django, Django api compatible, and Django project api

Three things hammered home for me that Carl's approach is the correct way 
to do tests:

1.) Deploying to Heroku which encourages a small virtualenv size to as 
small as possible for deployment. Including test code in deployment is daft.

2)  The usefulness of 3rd party application test suites can be limited 
especially when developing against trunk or varying versions. I don't want 
to fork just to make their test suite pass if there's nothing wrong with 
the code base itself - just the tests.

3.) My project provided a custom middleware and auth backend which extended 
the auth functionality by adding sites to users, and authenticating against 
that (along with email usernames). It broke the django tests in all sorts 
of places.

That's when I discovered Carl's talk and was an instant convert.

If I'm developing django itself or to be compatible with the django api 
then I need to run the django test suite and ensure it passes with my 
custom app, but there are any number of ways that a project or even other 
applications can break the test suite so project tests should never be run 
against the django test suite.


On Saturday, 25 August 2012 18:15:07 UTC+10, Russell Keith-Magee wrote:
>
> Hi all, 
>
> So, I've been working on a Django branch [1] to implement the approach 
> to pluggable user models that was decided upon earlier this year [2] 
>
> [1] https://github.com/freakboy3742/django/tree/t3011 
> [2] https://code.djangoproject.com/wiki/ContribAuthImprovements 
>
> The user-swapping code itself is coming together well. However, I've 
> hit a snag during the process of writing tests that may require a 
> little yak shaving. 
>
> The problem is this: With pluggable auth models, you lose the 
> certainty over exactly which User model is present, which makes it 
> much harder to write some tests, especially ones that will continue to 
> work in an unpredictable end-user testing environment. 
>
> With regards to pluggable Users, there are 5 types of tests that can be 
> run: 
>
>  1) Contrib.auth tests that validate that the internals of 
> contrib.auth work as expected 
>
>  2) Contrib.auth tests that validate that the internals of 
> contrib.auth work with a custom User model 
>
>  3) Contrib.auth tests that validate that the currently specified user 
> model meets the requirements of the User contract 
>
> The problem is that because of the way syncdb works during testing 
> some of these tests are effectively mutually exclusive. The test 
> framework runs syncdb at the start of the test run, which sets up the 
> models that will be available for the duration of testing -- which 
> then constrains the tests that can actually be run. 
>
> This doesn't affect the Django core tests so much; Django's tests will 
> synchronise auth.User by default, which allows tests of type 1 to run. 
> It can also provide a custom User model, and use @override_settings to 
> swap in that model as required for tests of type 2. Tests of type 3 
> are effectively integration tests which will pass with *any* interface 
> compliant User model. 
>
> However, if I have my own project that includes contrib.auth in 
> INSTALLED_APPS, ./manage.py test will attempt to run *all* the tests 
> from contrib.auth. If I have a custom User model in play, that means 
> that the tests of type 1 *can't* pass, because auth.User won't be 
> synchronised to the database. I can't even use @override_settings to 
> force auth.User into use -- the opportunity for syncdb to pick up 
> auth.User has passed. 
>
> We *could* just mark the affected tests that require auth.User as 
> "skipUnless(user model == auth.User)", but that would mean some 
> projects would run the tests, and some wouldn't. That seems like an 
> odd inconsistency to me -- the tests either should be run, or they 
> shouldn't. 
>
> In thinking about this problem, it occurred to me that what is needed 
> is for us to finally solve an old problem with Django's testing -- the 
> fact that there is a difference between different types of tests. 
> There are tests in auth.User that Django's core team needs to run 
> before we cut a release, and there are integration tests that validate 
> that when contrib.auth is deployed in your own project, that it will 
> operate as designed. The internal tests need to run against a clean, 
> known environment; integration tests must run against your project's 
> native environment. 
>
> Thinking more broadly, there may be other categories -- "smoke tests" 
> for  quick sanity check that a system is working; "Interaction tests" 
> that run live browser tests; and so on. 
>
> Python's unittest library contains the concept of Suites, which seems 
> to me like a reasonable analog of what I'm talking about here. What is 
> missing is a s