[GSoC 2012] Schema Alteration API proposal
Abstract
--
A database migration helper has been one of the most long standing feature
requests in Django. Though Django has an excellent database creation helper,
when faced with schema design changes, developers have to resort to either
writing raw SQL and manually performing the migrations, or using third party
apps like South[1] and Nashvegas[2].
Clearly Django will benefit from having a database migration helper as an
integral part of its codebase.
>From [3], the consensus seems to be on building a Ruby on Rails ActiveRecord
Migrations[4] like framework, which will essentially emit python code after
inspecting user models and current state of the database. The python code
generated will then be fed to a 'migrations API' that will actually handle
the
task of migration. This is the approach followed by South (as opposed to
Nashvegas's approach of generating raw SQL migration files). This ensures
modularity, one of the trademarks of Django. Third party developers can
create
their own inspection and ORM versioning tools, provided the inspection tool
emits python code conforming to our new migrations API.
To sum up, the complete migrations framework will need, at the highest
level:
1. A migrations API that accepts python code and actually performs the
migrations.
2. An inspection tool that generates the appropriate python code after
inspecting models and current state of database.
3. A versioning tool to keep track of migrations. This will allow 'backward'
migrations.
4. Glue code to tie the above three together.
Implementation plan
--
Before discussing the implementation plan for the migrations framework, I
would like to digress for a moment and discuss the final state of the
migrations framework when it will be implemented.
For the user, syncing and migrating databases will consist of issuing the
commands syncdb and a new 'migrate' command.
syncdb will be have to be rewritten and a new migrate command will be
written.
South's syncdb:
class Command(NoArgsCommand):
def handle_noargs(self, migrate_all=False, **options):
...
apps_needing_sync = []
apps_migrated = []
for app in models.get_apps():
app_label = get_app_label(app)
if migrate_all:
apps_needing_sync.append(app_label)
else:
try:
migrations = migration.Migrations(app_label)
except NoMigrations:
# It needs syncing
apps_needing_sync.append(app_label)
else:
# This is a migrated app, leave it
apps_migrated.append(app_label)
verbosity = int(options.get('verbosity', 0))
# Run the original syncdb procedure for apps_needing_sync
# If migrate is passed as a parameter, run migrate command for rest
The above code is from South's override of syncdb command. It basically
divides INSTALLED_APPS into apps that have a migration history and will be
handled by the migrations framework and those that do not have a migrations
history and will be handled by Django's syncdb. South expects users to
manually run a 'schemamigration --initial' command for every app they want
to
be handled by South's migration framework.
If migrations become a core part of Django, every user app will have a
migration folder(module) under it, created at the time of issuing
django-admin.py startapp. Thus by modifying the startapp command to create a
migrations module for every app it creates, we will be able to use South's
syncdb code as is and will also save the user from issuing
schemamigration --initial for all his/her apps.
Now that we have a guaranteed migrations history for every user app, migrate
command will also be more or less a copy of South's migrate command.
Coming back to the migrations API,
There are three fundamental operations that can be performed during a
migration:
1. Creation of a new model.
2. Alteration in an existing model.
3. Deletion of an existing model.
As much as I would have liked to use Django creation API's code for creating
and destroying models, we cannot. The reason for this is Django's creation
API
uses its inspection tools to generate *SQL* which is then directly fed to
cursor.execute. What we need is a migrations API which gobbles up *python*
code generated by the inspection tool. Moreover deprecating/removing
Django's
creation API to use the new migrations API everywhere will give rise to
performance issues since time will be wasted in generating python code and
then
converting python to SQL for Django's core apps which will never have
migrations anyways.
The creation API and code that depends on it (syncdb, sql,
django.test.simple
and django.contrib.gis.db.backends) will be left as is.
Therefore much of the code for our new migra
Re: auth.User refactor: reboot
A feature I would love to see is the ability to support multiple forms of
authentication for a single user account. (One account to many credentials.)
On Sat, Mar 17, 2012 at 11:19 PM, Ian Lewis wrote:
> Hi,
>
> On Sun, Mar 18, 2012 at 9:41 AM, Russell Keith-Magee
> wrote:
> >> 1. Django shouldn't decide what fields go on the user model. The app
> >> provides an abstract base class which developers subclass to add the
> >> appropriate fields they need.
> >
> > +1
>
> THX
>
> >> 2. Django shouldn't decide the type of the primary key. The app only
> >> relies on the fact that the object has a pk. The id field can be named
> >> anything you wish and can be any type (integer, char, uuid, etc.).
> >
> > +1
>
> THX again
>
> >> 3. Third party apps don't rely on the user having any fields but
> >> rather the base user class defines methods that are implemented by
> >> subclasses. Methods like get_display_name() which provides a way for
> >> third party apps to get something to display.
> >> 4. Rather than provide mixins or something, we should have conventions
> >> for the field names like 'email' and third party apps should check if
> >> the user has one using duck typing e.g. hasattr(user, 'email'). An
> >> alternative could be to provide some kind of API for commonly used
> >> actions like emailing users.
> >
> > This is essentially all I was proposing when I spoke of an "admin User
> contract"; that we define some basic "identity" functions that every User
> object is expected to provide -- short name, long name, and so on.
> >
> > The admin case is a little more complicated because there is also a
> required API for permissions and groups, but to my mind, these are
> different contracts, and should be documented as such.
>
> My solution is simply authentication, authorization would need to be
> added on or in a separate app built on top of newauth.
>
> >> 5. Basic username (or email)/password authentication can be provided.
> >> The app has a base user class from which a basic abstract user with
> >> username/password is defined. This can implement setting passwords
> >> properly and provide forms etc.
> >> 6. Multiple user models can be defined (Like say for normal users and
> >> affiliate users or admin users). If one wants to create a project
> >> currently with a separate user model, none of the machinery in the
> >> auth app can be used.
> >
> > Sure you can -- you have a base User, and then subclasses to get
> AdminUser and NormalUser -- both of which are effectively just another type
> of UserProfile.
>
> I meant one that was a completely separate concrete base model. The
> current auth forces you to take along with you all the fields on the
> User model.
>
> >> You create users by creating your own app in your project and creating
> >> a User there:
> >>
> >> account/models.py
> >>
> >> from django.db import models
> >>
> >> from newauth.models import UserBase
> >>
> >> class User(BaseUser):
> >>full_name = models.CharField(u"Full Name", max_length=255)
> >>email = models.EmailField('Email Address')
> >>profile = models.TextField('Profile Bio', blank=True, null=True)
> >>avatar = models.ImageField('Avatar', upload_to='profileimg/',
> >> blank=True, null=True)
> >>
> >>def get_display_name(self):
> >>return self.full_name
> >>
> >>class Meta:
> >>db_table = 'my_user_table'
> >>verbose_name = u"Djangonaut"
> >>verbose_name_plural = u"Djangonaut"
> >>
> >> There are even docs and tests.
> >
> > How does this address the issue of reusable apps referencing User? Let's
> say I write a comments app, and want an Author field. I need a ForeignKey
> to "User". But I can't have a foreign key to BaseUser, because it's an
> abstract class. How do I define my Comment model in such a way that it can
> reference a generic "User"?
> >
> > It seems to me that the solution you're proposing requires the LazyFK
> and app-refactor infrastructure I've described in order to be useful in the
> general case (not that I'm complaining, mind -- just pointing out that our
> two proposals are complementary :-).
>
> This is a bad example for showing how that works. I just wanted to
> illustrate how you would make your own User model. In the case where
> you want a foreign key to User you can import the default user model
> from newauth.models as User much like you do with the current django
> auth app.
>
> See: http://ianlewis.bitbucket.org/django-newauth/third_party.html
>
> >> This is going to be the biggest problem with my solution. There would
> >> probably have to be some kind of compatibility layer added to make
> >> existing apps work or to provide a simpler migration path.
> >
> > Isn't the compatibility layer just an implementation of the existing
> auth.User class that extends from BaseUser? We're going to have to ship
> this user class anyway, so that everything works out of the box; then if
> anyone wants to define their own User class, they can.
>
> Perhaps.
Re: Ticket 16317 https://code.djangoproject.com/ticket/16317
On Friday, March 16, 2012 7:00:32 AM UTC-7, Marcob wrote: > > The ticket 16317 has a very nice 4 months old patch (it's has a two > lines fix, remainder are just test fixes). > It was marked for 1.3 version but now it's better to change it to 1.4 > version. > As I really hate to patch django I think my only solution is "to > lobby" this ticket here :-) > Is there anything else I can do? > This is where the community can help the core about by: - applying the patch and verifying that it applies cleanly to current trunk - see if it fixes the issue as you understand it or are experiencing it - do your best to review the tests and determine if docs would be needed If all looks OK, comment that you have done the above and mark "Ready for checkin" I think too few people realize that this last step is something they can do. It is not a guarantee of a prompt checkin - but it certainly increases the chance it will be seen. I'd say only after it has sat in that state for a bit, does it make sense to post a nudge here. -Preston Ciao. > Marco. > P.S. Someone said the putting your email in cc in a ticket isn't the > way to go. > -- You received this message because you are subscribed to the Google Groups "Django developers" group. To view this discussion on the web visit https://groups.google.com/d/msg/django-developers/-/3Delk-NooHsJ. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Ticket 16317 https://code.djangoproject.com/ticket/16317
On Sun, Mar 18, 2012 at 17:19, ptone wrote: > If all looks OK, comment that you have done the above and mark "Ready for > checkin" Thanks Preston, I've update the patch a the current revision (the content is not changed) I'think is ready for checkin, test are included on fixtures testcases https://code.djangoproject.com/ticket/16317 ./runtests.py --settings=test_sqlite fixtures Creating test database for alias 'default'... Creating test database for alias 'other'... -- Ran 16 tests in 0.681s OK Destroying test database for alias 'default'... Destroying test database for alias 'other'... S -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: [GSoC 2012] Schema Alteration API proposal
On 18/03/2012, at 7:38 PM, Kushagra Sinha wrote: > Abstract > -- > A database migration helper has been one of the most long standing feature > requests in Django. Though Django has an excellent database creation helper, > when faced with schema design changes, developers have to resort to either > writing raw SQL and manually performing the migrations, or using third party > apps like South[1] and Nashvegas[2]. > > Clearly Django will benefit from having a database migration helper as an > integral part of its codebase. > > From [3], the consensus seems to be on building a Ruby on Rails ActiveRecord > Migrations[4] like framework, which will essentially emit python code after > inspecting user models and current state of the database. Check the edit dates on that wiki -- most of the content on that page is historical, reflecting discussions that were happening over 3 years ago. There have been many more recent discussions. The "current consensus" (at least, the consensus of what the core team is likely to accept) is better reflected by the GSoC project that was accepted, but not completed last year. I posted to Django-developers about this a week or so ago [1]; there were some follow up conversations in that thread, too [2]. [1] http://groups.google.com/group/django-developers/msg/cf379a4f353a37f8 [2] http://groups.google.com/group/django-developers/msg/2f287e5e3dc9f459 > The python code > generated will then be fed to a 'migrations API' that will actually handle the > task of migration. This is the approach followed by South (as opposed to > Nashvegas's approach of generating raw SQL migration files). This ensures > modularity, one of the trademarks of Django. I don't think you're going to be able to ignore raw SQL migrations quite that easily. Just like the ORM isn't able to express every query, there will be migrations that you can't express in any schema migration abstraction. Raw SQL migrations will always need to be an option (even if they're feature limited). > Third party developers can create > their own inspection and ORM versioning tools, provided the inspection tool > emits python code conforming to our new migrations API. > > To sum up, the complete migrations framework will need, at the highest level: > 1. A migrations API that accepts python code and actually performs the >migrations. This is certainly needed. I'm a little concerned by your phrasing of an "API that accepts python code", though. An API is something that Python code can invoke, not the other way around. We're looking for django.db.backends.migration as an analog of django.db.backends.creation, not a code consuming utility library. > 2. An inspection tool that generates the appropriate python code after >inspecting models and current state of database. The current consensus is that this shouldn't be Django's domain -- at least, not in the first instance. It might be appropriate to expose an API to extract the current model state in a Pythonic form, but a fully-fledged, user accessible "tool". > 3. A versioning tool to keep track of migrations. This will allow 'backward' >migrations. If backward migrations is the only reason to have a versioning tool, then I'd argue you don't need versioning. However, that's not the only reason to have versioning, is it :-) > South's syncdb: > class Command(NoArgsCommand): > def handle_noargs(self, migrate_all=False, **options): As a guide for the future -- large wads of code like this aren't very compelling as part of a proposal unless you're trying to demonstrate something specific. In this case, you're just duplicating some of South's internals -- "I'm going to take South's lead" is all you really needed to say. > If migrations become a core part of Django, every user app will have a > migration folder(module) under it, created at the time of issuing > django-admin.py startapp. Thus by modifying the startapp command to create a > migrations module for every app it creates, we will be able to use South's > syncdb code as is and will also save the user from issuing > schemamigration --initial for all his/her apps. > > Now that we have a guaranteed migrations history for every user app, migrate > command will also be more or less a copy of South's migrate command. What does this "history" look like? Are migrations named? Are they dated? Numbered? How do you handle dependencies? Ordering? Collisions between parallel development? *This* is the sort of thing a proposal should be elaborating. > > As much as I would have liked to use Django creation API's code for creating > and destroying models, we cannot. The reason for this is Django's creation API > uses its inspection tools to generate *SQL* which is then directly fed to > cursor.execute. What we need is a migrations API which gobbles up *python* > code generated by the inspection tool. Moreover deprecating
Re: GSoc Improved Error Reporting Project
On 18/03/2012, at 5:44 AM, melanie.s@googlemail.com wrote: > Hi, > > My name is Melanie Rao, I am a student at the University of Edinburgh. > In the final year of my Bachelors in Artificial Intelligence and > Software Engineering, I was hoping to work on an open source project > this summer, and would be very keep to join Django through the Google > Summer of Code scheme. > > In particular, I am interested in the Error Report Project. I have > checkout the code from the SVN, and will try to get familiar with it > as much as possible in the next few days. Could you please tell me > what would be good ideas to familiarise myself with the system? Hi Melanie, If you're new to Django, the best approach would be to work through the tutorial. Then, set yourself the task of building a simple web app of some kind; As you go, take note of every error message you get that is confusing or misleading. Then, start pulling on the threads. Go back and try and find out *why* each error message is generated -- not just what it means, but what sequence of code has caused it to be invoked. Try and work out how the error could be more useful e.g., is it just that the text is wrong? Is the text correct in some circumstances, but not others? Or is it a case of something where Django is doing something badly, like swallowing or wrapping an exception? If you're an old hand at Django, then skip the first step, and go straight to the thread-pulling :-) As some high-level project selection guidance, the error reporting project will be better suited to someone who has been using Django for a while, and is familiar with the ways Django reports errors -- and any patterns that emerge in errors. An existing familiarity with Django's internals wouldn't go astray, either. If you're newer to Django (or to Django's internals), might I suggest that the Validation project [1] might be a better project. The Validation problem has a much clearer definition, and there's a much smaller footprint of code that needs to be understood. Validation is a case of something where Django has an existing feature, but isn't making good use of that feature internally, and isn't exposing an API so that others can hook onto the feature. As such, it's a lot easier to establish exactly what needs to be done, and how hard it's likely to be, which means it's much more likely that your project will be accepted. [1] https://code.djangoproject.com/wiki/SummerOfCode2012#Validationfunctionalityrevamping Yours, Russ Magee %-) -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: auth.User refactor: reboot
Hi, On Mon, Mar 19, 2012 at 1:00 AM, Joe Tennies wrote: > A feature I would love to see is the ability to support multiple forms of > authentication for a single user account. (One account to many credentials.) You can do this already with Django auth by specifying multiple backends to the AUTHENTICATION_BACKENDS setting. Each one can take different credentials but potentially return the same User. See: https://docs.djangoproject.com/en/1.3/ref/settings/#std:setting-AUTHENTICATION_BACKENDS -- Ian http://www.ianlewis.org/ -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Login on the fly.
Sorry! I forgot to change the "GET" back to "POST" and uncomment the first two lines inside process_request. André Caldas. -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Login on the fly.
Hello! I have dedicated this weekend to learn Python and Django! I am really, really impressed! Everything is so well written and so well documented! When I get to know a knew language or a knew framework, I always end up with the impression that I could have done much better. But with Django I know that I could NEVER do better then you did! Congratulations! Well, studying the django.contrib.auth, I realized that the middleware class RemoteUserMiddleware is capable of authenticating the user at any time, on any page. On the other hand, the AuthenticationMiddleware does not even authenticate. It simply (if I got it right) checks the user_cache and sets request.user accordingly. I don't know if I am doing anything really useful, but I have written a middleware to handle a "Authentication On The Fly", just like RemoteUserMiddleware does. I do not know the proper way to sets a message for failed attempts, so I put a "assert False". I also do not know what is the best way to handle the validated (and failed) form to the view/template, so I simply put a variable on the request object. Please, tell me what you think about it. :-) Cheers, André Caldas. -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en. from django.contrib import auth from django.core.exceptions import ImproperlyConfigured from django.contrib.auth.forms import AuthenticationForm class LoginOnTheFly ( object ): ''' Login on any page if username/password was submited The django.contrib.auth.middleware.RemoteUserMiddleware does authentication on the fly. You don't need to go to a "/process_login/" URL in order to authenticate. But the django.contrib.auth.middleware.AuthenticationMiddleware does not log the user in! It just reads the cache and sets request.user in case the user is ALREADY logged. Here, we imitate RemoteUserMiddleware to make the login on the fly. ''' def process_request ( self, request ): #if 'POST' != request.method: #return # AuthenticationMiddleware is required so that request.user exists. if not hasattr(request, 'user'): raise ImproperlyConfigured( "The LoginOnTheFly auth middleware requires the" " authentication middleware to be installed. Edit your" " MIDDLEWARE_CLASSES setting to insert" " 'django.contrib.auth.middleware.AuthenticationMiddleware'" " before the LoginOnTheFly class.") # TODO: Use a "global" form. form = AuthenticationForm(data=request.GET) request.on_the_fly_auth_form = form username = form['username'].value() password = form['password'].value() # This is not an authentication attempt. if not username and not password: return # If the user is already authenticated and that user is the user we are # getting passed in the headers, then the correct user is already # persisted in the session and we don't need to continue. if request.user.is_authenticated(): if request.user.username == username: return # We are seeing this user for the first time in this session, attempt # to authenticate the user. if not form.is_valid(): # TODO: Put a message to be displayed... assert False return user = form.get_user() # User is valid. Set request.user and persist user in the session # by logging the user in. request.user = user# <-- Is this needed? auth.login(request, user) def clean_username(self, username, request): return username
Re: Login on the fly.
Hi Andre, In order to contribute to Django, you may want read the guides first: https://docs.djangoproject.com/en/dev/internals/contributing/ Regards, -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
commit_on_success leaves incorrect PostgreSQL isolation mode?
While exploring the Django transaction stuff (in 1.4rc1), I ran across the
following behavior. I use commit_on_success as the example here, but the other
transaction decorators/context managers have the same issue.
It seems to me to be a bug, but I wanted to confirm this before I opened an
issue.
The configuration is running Django using the psycopg2 backend, with 'OPTIONS':
{ 'autocommit': True, }
Consider the following code:
from django.db import transaction, DEFAULT_DB_ALIAS, connections
from myapp.mymodels import X
x = X.objects.get(id=1)
print connections[DEFAULT_DB_ALIAS].isolation_level # As expected, it's 0
here.
x.myfield = 'Foo'
with commit_on_success():
x.save()
print connections[DEFAULT_DB_ALIAS].isolation_level # As expected, it's
1 here.
print connections[DEFAULT_DB_ALIAS].isolation_level # It's now 1 here, but
shouldn't it be back to 0?
The bug seems to be that the isolation level does not get reset back to 0, even
when leaving connection management. This means that any further operations on
the database will open a new transaction (since psycopg2 will automatically
open), but this transaction won't be managed in any way.
The bug appears to be in
django.db.backends.BaseDatabaseWrapper.leave_transaction_management; it calls
the _leave_transaction_management hook first thing, but this means that
is_managed() will return true (since the decorators call managed(True)), which
means that _leave_transaction_management in the psycopg2 backend will not reset
the transaction isolation level; the code in the psycopg2 backend seems to
assume that it will be run in the new transaction context, not the previous one.
Or am I missing something?
--
-- Christophe Pettus
x...@thebuild.com
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
