Re: [AUDIT] Enable Django Quick Start / Database Evolution Support
and i would be said SoC applicant. :) Kenneth Gonsalves wrote: > On Thursday 25 May 2006 10:43 pm, Joseph Kocherhans wrote: >> Schema evolution support has already been discussed at length. >> See: http://code.djangoproject.com/wiki/SchemaEvolution but it's >> not implemented yet. > > there is a django soc project for this > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers -~--~~~~--~~--~--~---
Re: Schema evolution
jeroen, brant, ilias, etc: schema evolution was an idea suggested and specifically granted for the SoC project. i don't know who here was involved with ranking the proposals, but for better or worse mine was accepted. to a certain extent i expect this to come with a effort not to trivialize my existence here before i can even get off the ground. more to the point: if you have and ideas/recommendations, i'd love to hear them. if you have questions regarding my abilities/qualifications, i'll answer those too. but in terms of going forward with your own implementations, i ask that you put them on hold for just a little while. if i fail miserably you won't be but a month or two behind schedule. surely, until then, there are other areas in which you could contribute without so sharply undercutting my chance for success. thank you, -- derek Jeroen Ruigrok van der Werven wrote: > Brant, > > On 4/24/06, Brant Harris <[EMAIL PROTECTED]> wrote: >> I've created a proposal for working all of the Schema Evolution tidbits out: >> http://code.djangoproject.com/wiki/SchemaEvolutionProposal >> >> If you get a chance to read and comment, I'd appreciate. > > I noticed in the code that you have auto_add_now, whereas it should be > auto_now_add. At least according to > http://www.djangoproject.com/documentation/model_api/ > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers -~--~~~~--~~--~--~---
Re: Schema evolution
proposal is here: https://kered.org/blog/2006/05/24/summer-of-code/ it's more of a bit on a previous/related effort i did than a specific plan for django. i'm new to this project, and just starting my way through your all's modeling framework. more will be coming. jeroen: not miffed. just marking a little tree in a big forest. :) Todd O'Bryan wrote: > On May 29, 2006, at 11:09 AM, Derek Anderson wrote: > >> schema evolution was an idea suggested and specifically granted for >> the >> SoC project. i don't know who here was involved with ranking the >> proposals, but for better or worse mine was accepted. > > And I for one am very excited. > > Do you have an outline somewhere, or is there a copy of your proposal > around that we can comment on? > > Todd > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers -~--~~~~--~~--~--~---
Re: the so-called [AUDIT]
and now: http://en.wikipedia.org/wiki/Ilias_Lazaridis Carlo C8E Miron wrote: > On 6/2/06, Julian 'Julik' Tarkhanov <[EMAIL PROTECTED]> wrote: >> I would advise all respected Django contributors to follow the path >> mentioned here: >> >> http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/138966 > > and maybe also > > http://www.encyclopediadramatica.com/index.php/Ilias > > ;) > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers -~--~~~~--~~--~--~---
Re: Proposal: default escaping
the problem is that there are multiple types of escaping. sql? html?
javascript? new-web-tech-of-the-day? do you escape them all, or just some?
personally, i don't like my framework to auto-munge my data behind my
back. esp. in ways that are not clearly defined and could change on a
whim. too many potential secondary effects. plus it stinks to me of a
false sense of security while implicitly OKing people to ignore security.
but if it is going to be done, i'd suggest a flag on the field in the
model. ("automunge-html":="true"?) with perhaps a model default.
SmileyChris wrote:
> Here's how I see it:
> - 99% of the time, templates are HTML
> - most template variables should be escaped
> - developers are human and will miss variables that need escaping
>
> My proposal is that all templates variables are escaped by default.
>
>
> Think about it for a bit before you throw the idea away. Then reply
> with your thoughts.
>
>
> Of course we need an easy method to NOT auto-escape variables. Perhaps
> something like raw_variable ?
>
> There is also the issue of MASSIVE backwards incompatibility. The two
> options I see ane:
> 1. A new variable type is created for auto-escaping instead
> 2. Provide a setting which turns this new functionality on but is off
> by default
>
>
> >
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers
-~--~~~~--~~--~--~---
Re: Proposal: default escaping
the idea of it being in the model was more along the lines of validating
incoming data than it was munging outgoing. html is almost always
either acceptable or it's not in a given field. (per your example: who
want's arbitrary HTML allowed in a plain text email and not in a web
page?)
but i still argue that no implicit magic munging happen anywhere. it's
not that hard to get into safe-from-XSS coding styles. we did it for
sql injection, didn't we? :)
however, i would much rather have a flag/tag at the top of my template
than a global default based on template file type.
Simon Willison wrote:
> On 14 Jun 2006, at 14:44, Derek Anderson wrote:
>
>> the problem is that there are multiple types of escaping. sql? html?
>> javascript? new-web-tech-of-the-day? do you escape them all, or
>> just some?
>>
>> personally, i don't like my framework to auto-munge my data behind my
>> back. esp. in ways that are not clearly defined and could change on a
>> whim. too many potential secondary effects. plus it stinks to me
>> of a
>> false sense of security while implicitly OKing people to ignore
>> security.
>>
>> but if it is going to be done, i'd suggest a flag on the field in the
>> model. ("automunge-html":="true"?) with perhaps a model default.
>
> The model is definitely the wrong place for this - after all, a model
> field might be output in a plain text email where escaping isn't
> appropriate.
>
> The problem here is very simple: XSS is the most common vulnerability
> on the Web. It's unbelievably easy for an XSS vulnerability to sneak
> in to an application - even experienced programmers who completely
> understand the security implications are likely to forget to add a |
> escape filter once in a while.
>
> Obviously we DO need to be able to turn auto escaping off - there are
> plenty of cases where it isn't appropriate. A classic example from
> Django at the moment would be:
>
> {% value|markdown %}
>
> We should also be able to turn it off for people who don't like it,
> like yourself!
>
> BUT... we can't have it as a global setting. magic quotes in PHP has
> taught us that much - global settings relating to auto filtering of
> data lead to insanity when you start wanting to create reusable
> applications.
>
> That's why I'm keen on having escaping set at the template level. I'm
> actually starting to feel that using the template extension might not
> be a bad idea here. "index.html" has auto escaping, "index.txt"
> doesn't. That way templates don't have to include an ugly extra tag
> at the top of the code.
>
> Cheers,
>
> Simon
>
> >
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers
-~--~~~~--~~--~--~---
evolution test case solicitation
subject line says it all - i want your models! :) if it's obvious what the change is with a comment, that's fine. if not, before and after versions would be appreciated. i'd like everything from the mundane to the esoteric. to avoid flooding the list, you can send them directly to [EMAIL PROTECTED] unless you think the list needs a discussion regarding how an evolution should be made, of course. thanks! -- derek --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers -~--~~~~--~~--~--~---
Re: schema-evolution: status?
hey matt, would you be willing to send me a copy of your app, your original schema, and whatever load script failed? derek Matthew Flanagan wrote: > I had some to make some changes to some models today that would result > in schema changes so I decided to give the schema-evolution branch a > go. > > I didn't get far as the script I use to load sample data into my > applications failed [1] and trying to manually add data via my public > CRUD views failed because 'User' object has no attribute > 'is_authenticated'. So it seems the branch is quite far behind the > trunk. > > Any chance of merging the latest trunk changes in the schema-evolution branch? > > > [1] http://paste.e-scribe.com/1384/ > > regards > > matthew > > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers -~--~~~~--~~--~--~---
Re: schema-evolution: status?
i merged with head before i checked it in. unless i really frelled up w/ svn, it should only be ~2 weeks behind. could you send just your models? (+ load script, original schema) Matthew Flanagan wrote: > Derek, > > On 31/08/06, Derek Anderson <[EMAIL PROTECTED]> wrote: >> hey matt, >> >> would you be willing to send me a copy of your app, your original >> schema, and whatever load script failed? >> >> derek > > Unfortunately I can't send you a copy of the app. However looking at > your branch it seems it may benefit from a merge from a more recent > revision of the trunk or atleast with the 0.95 release (See my message > below re: the User.is_authenticated() missing from your branch) > > Alternatively I'm on #django if we can manage to hook up there. > > matthew > >> >> Matthew Flanagan wrote: >>> I had some to make some changes to some models today that would result >>> in schema changes so I decided to give the schema-evolution branch a >>> go. >>> >>> I didn't get far as the script I use to load sample data into my >>> applications failed [1] and trying to manually add data via my public >>> CRUD views failed because 'User' object has no attribute >>> 'is_authenticated'. So it seems the branch is quite far behind the >>> trunk. >>> >>> Any chance of merging the latest trunk changes in the schema-evolution >>> branch? >>> >>> >>> [1] http://paste.e-scribe.com/1384/ >>> >>> regards >>> >>> matthew >>> >> > > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers -~--~~~~--~~--~--~---
Re: schema-evolution: status?
which backend are you using?
Matthew Flanagan wrote:
> Derek,
>
> I have manually merged the trunk into my local working copy of the
> schema-evolution branch and started playing with it. I wanted to
> question the SQL "sqlevolve" is outputting. I have this model in an
> application called "asset":
>
> class Interface(models.Model):
> name = models.CharField(maxlength=64, core=True, db_index=True,
> help_text='The name of the interface as given by the asset.')
> interfacetype = models.ForeignKey(InterfaceType)
> ipaddress = models.ForeignKey(IPAddress, verbose_name='IP Address',
> raw_id_admin=True)
> # allow for EUI-48 and EUI-64 addresses
> mac_address = models.CharField(maxlength=24, blank=True,
> help_text='The EUI-48 or EUI-64 physical address of the interface.')
> domain = models.CharField(maxlength=255, blank=True,
> help_text='The DNS domain this host resides in.')
> asset = models.ForeignKey(Asset, edit_inline=models.TABULAR,
> num_in_admin=10, num_extra_on_change=5)
> objects = InterfaceManager()
>
> def _get_meta(self):
> return self._meta
> meta = property(_get_meta)
>
> def __str__(self):
> return "%s:%s" % (self.asset, self.name)
>
> def get_absolute_url(self):
> return self.asset.get_absolute_url()
>
> class Meta:
> ordering = ['name']
> unique_together = (('asset', 'name'),)
>
> class Admin:
> pass
>
> and the schema from "./manage.py sql asset":
>
> CREATE TABLE "asset_interface" (
> "id" serial NOT NULL PRIMARY KEY,
> "name" varchar(64) NOT NULL,
> "interfacetype_id" integer NOT NULL,
> "ipaddress_id" integer NOT NULL REFERENCES "ip_ipaddress" ("id"),
> "mac_address" varchar(24) NOT NULL,
> "domain" varchar(255) NOT NULL,
> "asset_id" integer NOT NULL REFERENCES "asset_asset" ("id"),
> UNIQUE ("asset_id", "name")
> );
>
>
> when I run "./manage.py sqlevolve asset" with absolutely no changes to
> my models it outputs:
>
> BEGIN;
> ALTER TABLE "asset_interface" ADD COLUMN "name_tmp" varchar(64);
> UPDATE "asset_interface" SET "name_tmp" = "name";
> ALTER TABLE "asset_interface" DROP COLUMN "name";
> ALTER TABLE "asset_interface" RENAME COLUMN "name_tmp" TO "name";
> ALTER TABLE "asset_interface" ALTER COLUMN "name" SET NOT NULL;
> COMMIT;
>
>
> Any ideas why it is doing this?
>
> regards
>
> matthew
>
> >
>
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers
-~--~~~~--~~--~--~---
Re: schema-evolution: status?
i only ever made two commits into django's repo:
changeset 3646 - the meat and potatoes
changeset 3647 - test code
can someone who knows svn better than i help me out as to how i would
refresh the branch with the latest from the trunk, and then remerge
these changes back into it?
thanks,
derek
Victor Ng wrote:
> Any luck with schema evolution lately?
>
> I'm starting to look at seeing if i can merge this back to trunk as
> well, but wanted to know if you've made any progress.
>
> I'm using the postgresql_psycopg2 backend.
>
> vic
>
> On 9/14/06, Matthew Flanagan <[EMAIL PROTECTED]> wrote:
>> postgresql
>>
>> On 14/09/06, Derek Anderson <[EMAIL PROTECTED]> wrote:
>>> which backend are you using?
>>>
>>> Matthew Flanagan wrote:
>>>> Derek,
>>>>
>>>> I have manually merged the trunk into my local working copy of the
>>>> schema-evolution branch and started playing with it. I wanted to
>>>> question the SQL "sqlevolve" is outputting. I have this model in an
>>>> application called "asset":
>>>>
>>>> class Interface(models.Model):
>>>> name = models.CharField(maxlength=64, core=True, db_index=True,
>>>> help_text='The name of the interface as given by the asset.')
>>>> interfacetype = models.ForeignKey(InterfaceType)
>>>> ipaddress = models.ForeignKey(IPAddress, verbose_name='IP Address',
>>>> raw_id_admin=True)
>>>> # allow for EUI-48 and EUI-64 addresses
>>>> mac_address = models.CharField(maxlength=24, blank=True,
>>>> help_text='The EUI-48 or EUI-64 physical address of the
>>>> interface.')
>>>> domain = models.CharField(maxlength=255, blank=True,
>>>> help_text='The DNS domain this host resides in.')
>>>> asset = models.ForeignKey(Asset, edit_inline=models.TABULAR,
>>>> num_in_admin=10, num_extra_on_change=5)
>>>> objects = InterfaceManager()
>>>>
>>>> def _get_meta(self):
>>>> return self._meta
>>>> meta = property(_get_meta)
>>>>
>>>> def __str__(self):
>>>> return "%s:%s" % (self.asset, self.name)
>>>>
>>>> def get_absolute_url(self):
>>>> return self.asset.get_absolute_url()
>>>>
>>>> class Meta:
>>>> ordering = ['name']
>>>> unique_together = (('asset', 'name'),)
>>>>
>>>> class Admin:
>>>> pass
>>>>
>>>> and the schema from "./manage.py sql asset":
>>>>
>>>> CREATE TABLE "asset_interface" (
>>>> "id" serial NOT NULL PRIMARY KEY,
>>>> "name" varchar(64) NOT NULL,
>>>> "interfacetype_id" integer NOT NULL,
>>>> "ipaddress_id" integer NOT NULL REFERENCES "ip_ipaddress" ("id"),
>>>> "mac_address" varchar(24) NOT NULL,
>>>> "domain" varchar(255) NOT NULL,
>>>> "asset_id" integer NOT NULL REFERENCES "asset_asset" ("id"),
>>>> UNIQUE ("asset_id", "name")
>>>> );
>>>>
>>>>
>>>> when I run "./manage.py sqlevolve asset" with absolutely no changes to
>>>> my models it outputs:
>>>>
>>>> BEGIN;
>>>> ALTER TABLE "asset_interface" ADD COLUMN "name_tmp" varchar(64);
>>>> UPDATE "asset_interface" SET "name_tmp" = "name";
>>>> ALTER TABLE "asset_interface" DROP COLUMN "name";
>>>> ALTER TABLE "asset_interface" RENAME COLUMN "name_tmp" TO "name";
>>>> ALTER TABLE "asset_interface" ALTER COLUMN "name" SET NOT NULL;
>>>> COMMIT;
>>>>
>>>>
>>>> Any ideas why it is doing this?
>>>>
>>>> regards
>>>>
>>>> matthew
>>>>
>>>
>
>
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers
-~--~~~~--~~--~--~---
Re: schema-evolution: status?
"svn merge http://code.djangoproject.com/svn/django/trunk"; gives me the "svn help merge" message. here's what i did: $ svn co http://code.djangoproject.com/svn/django/branches/schema-evolution/ django_tmp_src [normal checkout crap] $ cd django_tmp_src/ $ svn merge http://code.djangoproject.com/svn/django/trunk [help message] god i miss cvs... Kenneth Gonsalves wrote: > > On 26-Oct-06, at 12:40 PM, Derek Anderson wrote: > >> can someone who knows svn better than i help me out as to how i would >> refresh the branch with the latest from the trunk, and then remerge >> these changes back into it? > > in your working copy root do: > > svn merge http://code.djangoproject.com/svn/django/trunk > > and then, after resolving conflicts do > > svn commit > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers -~--~~~~--~~--~--~---
Re: schema-evolution: status?
thanks for your help. the merge went smoothly, and has been committed as revision 3937. Matthew Flanagan wrote: > Hi Derek, > > To do the merge you need to look at the last revision that you merged > from trunk or in your case when your branch was created. In this case > it is revision 3332. So to merge the latest trunk you need to: > > cd django_tmp_src > svn merge -r 3332:HEAD http://code.djangoproject.com/svn/django/trunk > > [resolve and conflicts and test your code] > svn co -m 'merged latest changes from trunk' > > To keep up to date with the trunk you just need to run an 'svn log' in > your working copy and look at the last revision that you merged from > the trunk, rev xyz say, and use that like this: > > svn merge -r xyz:HEAD http://code.djangoproject.com/svn/django/trunk > etc > > regards > > matthew > > On 26/10/06, Derek Anderson <[EMAIL PROTECTED]> wrote: >> "svn merge http://code.djangoproject.com/svn/django/trunk"; gives me the >> "svn help merge" message. >> >> here's what i did: >> >> $ svn co >> http://code.djangoproject.com/svn/django/branches/schema-evolution/ >> django_tmp_src >> [normal checkout crap] >> >> $ cd django_tmp_src/ >> >> $ svn merge http://code.djangoproject.com/svn/django/trunk >> [help message] >> >> >> god i miss cvs... >> >> >> >> Kenneth Gonsalves wrote: >>> On 26-Oct-06, at 12:40 PM, Derek Anderson wrote: >>> >>>> can someone who knows svn better than i help me out as to how i would >>>> refresh the branch with the latest from the trunk, and then remerge >>>> these changes back into it? >>> in your working copy root do: >>> >>> svn merge http://code.djangoproject.com/svn/django/trunk >>> >>> and then, after resolving conflicts do >>> >>> svn commit >>> >> > > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers -~--~~~~--~~--~--~---
Re: running the unit tests in the schema evolution branch
Victor Ng wrote: > The tests don't look like they're integrated with runtests.py at all. no, they're standalone. > 3) Run reset_all_to_pre (note that this doesn't have a shebang at the top) > 5) Run reset_all_to_post (note that this script also doesn't have a > shebang at the top) we're all running bash, aren't we? ;p > So do basically something like this: > > cd branches/schema-evolution/tests/evolvedbtests > ./rest_all_to_pre > python manage.py syncdb > ./reset_all_to_post > python manage.py sqlevolve case01_add_field > python manage.py sqlevolve case02_rename_field > python manage.py sqlevolve case03_rename_model > python manage.py sqlevolve case04_change_flag > python manage.py sqlevolve case05_remove_field yep, except you should add: ./rest_all_to_pre python manage.py sqlevolve case01_add_field python manage.py sqlevolve case02_rename_field python manage.py sqlevolve case03_rename_model python manage.py sqlevolve case04_change_flag python manage.py sqlevolve case05_remove_field to prove it all works full circle. > There doesn't seem to be any automation around what the expected SQL > output is, and as far as I can tell, this only works on the mysql > backend. it should work on all three, with the exception of remove_field under sqlite (for obvious reasons) > The emitted output on my machine is always empty. I always get this: > > ~/svk-ws/remote/mb-django/branches/schema-evolution/tests/evolvedbtests> > python manage.py sqlevolve case01_add_field > BEGIN; > COMMIT; > Which is obviously not what I really want. > Any thoughts? check to make sure you've switched your symlinks from pre to post (or back again) since your last sync, and that the models actually are different than the tables in your db. (i can't tell you how many times this summer i was asking myself "wtf" for this exact reason) -- derek --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
schema evolution
Hey all, I've ported my schema evolution work from my SoC project last summer to Django v0.96. To use it, download the patch below, and run the following: $ cd //site-packages/django/ $ patch -p1 < ~//django_schema_evolution-v096patch.txt It should output the following: patching file core/management.py patching file db/backends/mysql/base.py patching file db/backends/mysql/introspection.py patching file db/backends/postgresql/base.py patching file db/backends/postgresql/introspection.py patching file db/backends/sqlite3/base.py patching file db/backends/sqlite3/introspection.py patching file db/models/fields/__init__.py patching file db/models/options.py To use it: $ cd // $ ./manage.py sqlevolve It should output something like this: BEGIN; ALTER TABLE `main_query` CHANGE COLUMN `accuracy` `accuracynew` numeric(10, 6) NULL; ALTER TABLE `main_query` ADD COLUMN `price` varchar(256) NULL; COMMIT; Assuming you have a model such as this: class Query(models.Model): query = models.CharField(maxlength=256, blank=False) accuracynew = models.FloatField(max_digits=10, decimal_places=6, null=True, blank=True, aka='accuracy') price = models.CharField(maxlength=256, null=True, blank=True) # new column Note the aka field where I changed the name of "accuracy" to "accuracynew". Let me know if you find any bugs. http://kered.org/blog/2007-07-19/django-schema-evolution/ http://kered.org/blog/wp-content/uploads/2007/07/django_schema_evolution-v096patch.txt -- Derek --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: schema evolution
Hey all, Sorry for the double-post, but I've written up some examples / documentation: http://kered.org/blog/wp-content/uploads/2007/07/django_schema_evolution_documentation.html Also, I've ported the changes to SVN. I would like to solicit testers, for potential inclusion in django-proper. Thanks, Derek Derek Anderson wrote: > Hey all, > > I've ported my schema evolution work from my SoC project last summer to > Django v0.96. To use it, download the patch below, and run the following: > > $ cd //site-packages/django/ > $ patch -p1 < ~//django_schema_evolution-v096patch.txt > > It should output the following: > > patching file core/management.py > patching file db/backends/mysql/base.py > patching file db/backends/mysql/introspection.py > patching file db/backends/postgresql/base.py > patching file db/backends/postgresql/introspection.py > patching file db/backends/sqlite3/base.py > patching file db/backends/sqlite3/introspection.py > patching file db/models/fields/__init__.py > patching file db/models/options.py > > To use it: > > $ cd // > $ ./manage.py sqlevolve > > It should output something like this: > > BEGIN; > ALTER TABLE `main_query` CHANGE COLUMN `accuracy` `accuracynew` > numeric(10, 6) NULL; > ALTER TABLE `main_query` ADD COLUMN `price` varchar(256) NULL; > COMMIT; > > Assuming you have a model such as this: > > class Query(models.Model): > query = models.CharField(maxlength=256, blank=False) > accuracynew = models.FloatField(max_digits=10, decimal_places=6, > null=True, blank=True, aka='accuracy') > price = models.CharField(maxlength=256, null=True, blank=True) # > new column > > Note the aka field where I changed the name of "accuracy" to "accuracynew". > > Let me know if you find any bugs. > > http://kered.org/blog/2007-07-19/django-schema-evolution/ > http://kered.org/blog/wp-content/uploads/2007/07/django_schema_evolution-v096patch.txt > > -- Derek > > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: schema evolution
heh, i read this and was like "there are other backends?!?" ;p nope, sqlite/mysql/postgres is all she wrote right now, as that was all that existed last summer when i did the bulk of the work on this. but i'll add oracle and mssql, if someone can provide me access to a test db. (well, maybe...i don't have any windows boxes around - can you access mssql from a linux machine?) pk11 wrote: > Hi all, > > Derek, nice job! does your solution support all of the db backends or > just sqlite/mysql/postgres? > > (we use a hacked version of dbmigration (http://www.aswmc.com/ > dbmigration/) at work - > works like a charm.) > > Thanks, > Peter > > On Jul 20, 11:22 am, Sebastian Macias <[EMAIL PROTECTED]> > wrote: >> Thanks for the hard work. I have been looking for this forever. It >> should definitely be merged to trunk as Django proper. >> >> Sebastian Macias >> >> On Jul 19, 5:33 pm, Derek Anderson <[EMAIL PROTECTED]> wrote:> Hey all, >> >>> Sorry for the double-post, but I've written up some examples / >>> documentation: >>> http://kered.org/blog/wp-content/uploads/2007/07/django_schema_evolut... >>> Also, I've ported the changes to SVN. I would like to solicit testers, >>> for potential inclusion in django-proper. >>> Thanks, >>> Derek >>> Derek Anderson wrote: >>>> Hey all, >>>> I've ported my schema evolution work from my SoC project last summer to >>>> Django v0.96. To use it, download the patch below, and run the following: >>>> $ cd //site-packages/django/ >>>> $ patch -p1 < ~//django_schema_evolution-v096patch.txt >>>> It should output the following: >>>> patching file core/management.py >>>> patching file db/backends/mysql/base.py >>>> patching file db/backends/mysql/introspection.py >>>> patching file db/backends/postgresql/base.py >>>> patching file db/backends/postgresql/introspection.py >>>> patching file db/backends/sqlite3/base.py >>>> patching file db/backends/sqlite3/introspection.py >>>> patching file db/models/fields/__init__.py >>>> patching file db/models/options.py >>>> To use it: >>>> $ cd // >>>> $ ./manage.py sqlevolve >>>> It should output something like this: >>>> BEGIN; >>>> ALTER TABLE `main_query` CHANGE COLUMN `accuracy` `accuracynew` >>>> numeric(10, 6) NULL; >>>> ALTER TABLE `main_query` ADD COLUMN `price` varchar(256) NULL; >>>> COMMIT; >>>> Assuming you have a model such as this: >>>> class Query(models.Model): >>>> query = models.CharField(maxlength=256, blank=False) >>>> accuracynew = models.FloatField(max_digits=10, decimal_places=6, >>>> null=True, blank=True, aka='accuracy') >>>> price = models.CharField(maxlength=256, null=True, blank=True) # >>>> new column >>>> Note the aka field where I changed the name of "accuracy" to "accuracynew". >>>> Let me know if you find any bugs. >>>> http://kered.org/blog/2007-07-19/django-schema-evolution/ >>>> http://kered.org/blog/wp-content/uploads/2007/07/django_schema_evolut... >>>> -- Derek > > > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: schema evolution
yep. i just updated the schema-evolution branch to match the trunk. or you can apply this patch instead: http://kered.org/blog/wp-content/uploads/2007/07/django_schema_evolution-svn20070719patch.txt Sebastian Macias wrote: > Does it work with the trunk? > > On Jul 19, 5:33 pm, Derek Anderson <[EMAIL PROTECTED]> wrote: >> Hey all, >> >> Sorry for the double-post, but I've written up some examples / >> documentation: >> >> http://kered.org/blog/wp-content/uploads/2007/07/django_schema_evolut... >> >> Also, I've ported the changes to SVN. I would like to solicit testers, >> for potential inclusion in django-proper. >> >> Thanks, >> Derek >> >> Derek Anderson wrote: >>> Hey all, >>> I've ported my schema evolution work from my SoC project last summer to >>> Django v0.96. To use it, download the patch below, and run the following: >>> $ cd //site-packages/django/ >>> $ patch -p1 < ~//django_schema_evolution-v096patch.txt >>> It should output the following: >>> patching file core/management.py >>> patching file db/backends/mysql/base.py >>> patching file db/backends/mysql/introspection.py >>> patching file db/backends/postgresql/base.py >>> patching file db/backends/postgresql/introspection.py >>> patching file db/backends/sqlite3/base.py >>> patching file db/backends/sqlite3/introspection.py >>> patching file db/models/fields/__init__.py >>> patching file db/models/options.py >>> To use it: >>> $ cd // >>> $ ./manage.py sqlevolve >>> It should output something like this: >>> BEGIN; >>> ALTER TABLE `main_query` CHANGE COLUMN `accuracy` `accuracynew` >>> numeric(10, 6) NULL; >>> ALTER TABLE `main_query` ADD COLUMN `price` varchar(256) NULL; >>> COMMIT; >>> Assuming you have a model such as this: >>> class Query(models.Model): >>> query = models.CharField(maxlength=256, blank=False) >>> accuracynew = models.FloatField(max_digits=10, decimal_places=6, >>> null=True, blank=True, aka='accuracy') >>> price = models.CharField(maxlength=256, null=True, blank=True) # >>> new column >>> Note the aka field where I changed the name of "accuracy" to "accuracynew". >>> Let me know if you find any bugs. >>> http://kered.org/blog/2007-07-19/django-schema-evolution/ >>> http://kered.org/blog/wp-content/uploads/2007/07/django_schema_evolut... >>> -- Derek > > > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: schema evolution
actually, "model changes, db needs update" is exactly what i think about
when i think of schema evolution. ;) but seriously...
> * You have to change all your code at once to match the new name
>(evolution is a slow process in nature ;-)
excellent point. for large apps, optionally/temporarily maintaining api
compatibility through a special field type (that maps to the new field)
is a great idea. but i see that as augmenting the "aka" attribute, not
replacing it. you will (hopefully) eventually remove the old field from
the class interface, likely long before you lose interest in being able
to migrate older schemas. plus, i think the "aka" syntax is cleaner in
the probably much more common case where you don't care about keeping
the old api intact.
> * The old name cannot change the type of the field
> * The change-info can only contain a name, nothing more
>(goes hand in hand with the last point)
yes, these two are the same - how to keep the previous field metadata
when updating, without mucking up your model code with lots of
"old_maxlength=256" style attributes or duplicating the entire field
definition every change. but beautifully, you don't have to keep any of
it (sans the name changes), because the rest can be pulled from the
existing schema. don't duplicate information, i say. granted, my code
doesn't support all type changes yet, but this isn't a limitation of the
method. think 9-digit numeric SSN field changing to a 12-char (to
support the new 10-digit SSNs plus dashes - yes, it's coming folks. run
in fear) or even splitting/combining certain field types - the two new
fields both aka the original. all possible.
now as far as changing the inter-model relationship fields...let me
break that into another email later.
derek
David Danier wrote:
> I did some thinking about the "aka"-parameter you use to name the old
> fieldname/modelname and some things that would be nice to have when
> schema _evolution_ is supported by django. I highlighted evolution,
> because that means more than just "model changes, db needs update" I think.
>
> I think the aka-method has some shortcomings:
> * You have to change all your code at once to match the new name
>(evolution is a slow process in nature ;-)
> * The old name cannot change the type of the field
>(for example ForeignKey -> ManyToMany)
> * The change-info can only contain a name, nothing more
>(goes hand in hand with the last point)
>
> So after thinking some time about this I came up with this idea:
> Old:
> class SomeModel(models.Model):
> new_fieldname = models.FooField(..., aka='old_fieldname')
> New:
> class SomeModel(models.Model):
> new_fieldname = models.FooField(...)
> # Not sure if new_fieldname as string or the
> # new_fieldname-object ist better here.
> # But I think the string might be easier.
> old_fieldname = evolution.RenamedField('new_fieldname')
> For Models:
> class NewModel(models.Model):
> pass
> # May use models from other applications.
> OldModel = evolution.MovedModel(NewModel)
>
> In the following text I will only use fields as examples, but I think
> models will be similar.
>
> Because the old name is still an attribute in the model it can change
> the new attribute/return the new attribute if it is returned. So all
> code will work without changes. Now - because this is nonsense in the
> long term - it will write logs/print warning/... (you name it). This can
> be extended by using parameters:
> evolution.RenamedField(..., obsolete=True) -> writes warning
> evolution.RenamedField(..., deprecated=True) -> raises exception
> (Perhaps I mixed the meaning of "deprecated" and "obsolete" here...not
> sure about them)
> ...you could even give it a date where the old name gets rejected. I
> think that would give users a great advantage other the current solution.
> So the new use-case:
> Users can change the model, change every code they know of directly, but
> get warnings when they missed something (but code works). Now after some
> time they can mark the old name to be deprecated, so access to it will
> be rejected after they think they got all pieces. If the code works the
> old name can be removed from the code (and should be removed...think
> that goes for "aka" as well), if not the exception tells them what is
> missing.
>
> The next thing you could do now would be things like "Replace a
> ForeignKey with ManyToMany" (which might a common use-case for
> db-evolution I think), example:
> class SomeModel(models.Model):
> some_others = models.ManyToManyField(...)
> some_other = evolution.ForeignKeyToManyToMany('some_others')
> This now can first do some nice translation to fill the m2m-tables with
> the old data. "some_other" might just return the first object in the
> m2m-relation or None if the relation is empty. If this behavior doesn't
> fit the user-idea of doing this deprecated/obsolete could be used to
schema evolution [testers wanted]
hey all,
i have finished writing unit tests covering the mysql, postgresql and
sqlite3 backends, as well as notably expanding what
introspections/migrations the sqlite3 backend can handle. also, full
documentation (including installation and usage instructions) is
available both in the branch and here:
http://code.djangoproject.com/wiki/SchemaEvolutionDocumentation
additionally, the latest changes from the trunk (r5787) have been merged
into the schema-evolution branch, so you won't lose any recent updates
by giving it a shot.
obviously, this is an area with a lot of corner cases. if you can,
please give it a shot ("./manage sqlevolve app_name" after you've made a
model change) and send me whatever bugs you find [EMAIL PROTECTED]
please include if you can:
* which database you're using
* your models.py file
* your previous models.py and/or your existing database schema
thanks!
derek anderson
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---
Re: schema evolution [testers wanted]
Russell Keith-Magee wrote: > - The 'aka' approach has some serious flaws. for the scenario you give the aka approach (or any formatting of model+existing_schema+rename_metadata) works just fine. i detail how (with working examples and sql output) here: http://kered.org/blog/2007-08-03/schema-evolution-confusion-example-case/ i'm not saying we don't need additional metadata, but i haven't seen or thought up a use case for it yet. > Your tests evaluate whether the right SQL for any given task, but > doesn't check if the correct tasks are identified. [...] The fact that > something is difficult to test doesn't mean it shouldn't be tested. i don't see how this criticism makes sense. both *are* tested. if the evolution code doesn't identify the correct task, the wrong migration sql is assured, therefore the test fails. give it a scenario, and see if introspection + sql generation works. if you have a different suggestion for testing methodology i'm all ears, but this comment seems to be based on only a superficial look at the unit test code. btw, i'm not suggesting that the unit tests are comprehensive. we need a LOT more cases. (and i plan to add them one by one as either anyone thinks of them, or bug reports come in) but i certainly hope i haven't suggested i was *unwilling* to test something. > - On a housekeeping issue, I'm really not happy with all the migration > code sitting in management.py. management.py is already getting pretty > bloated; if we're adding a huge feature like evolution, I'd rather see > it abstracted into a module of its own, with only the invocation hooks > in management.py. sure thing...done. > In short, IMHO this is not yet ready for general user testing i hope you don't mind if i solicit it anyway. ;) but seriously, i don't get this. i think it's of great value to seek testing early and often. if you don't want to, that's fine, but don't discourage others. > That said, there is some good code in the patch - in particular, it > looks like you've done some good work on the database introspection > code. I just think we need to have a bit more of a think about how the > good code gets used. > > I've got a few ideas on what could be done; I'll try to put my > thoughts together coherently and write up these ideas for comment. thanks. the introspection is where prob. 1/2 of the work is. (the other 1/2 being the sql generation for discrete db modification types, esp. in sqlite) the user-facing-interface is a minor part... but again, i am anxious to see recommendations on how it could be better. thanks, derek --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
runtests.py
hey all, is there a way to get runtests.py to run the tests for multiple backends at a time? right now i'm just editing my settings.py file manually each time, and wanted to check before i wrote my own script. :) thanks, derek --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: schema evolution [testers wanted]
Brantley Harris wrote: > Your > philosophy seems to be, "Those problems are spiky edge cases", but I > think the hope was they would be solved as well, or at least put on > track for a solution later on. more like "low-hanging-most-useful-fruit first" :) everyone agrees that there will never be a 100% introspection solution, so why not release what is currently useful, and then grow the scope as we go? (as long as it doesn't break anything, of course...and i'm not there yet for the record) > For instance, if a user wanted to take a field > or group of fields from a Model and place them into a separate, > related Model. Correct me if I'm wrong, but I don't even think your > solution supports any sort of data processing like this. You would > have to manually save the data, migrate, and then import the altered > data into the new schema, which all but precludes a distributed > solution. no, it doesn't support moving fields from one model to another, but that's also on the todo list. (just as key support is still non-existent), but as long as the proper relation exists between the models...syntax-wise: class Article(models.Model): name = models.CharField(maxlength=40, default="woot") #tag = models.CharField(maxlength=40) class Tag(models.Model): article = models.ForeignKey(Article) tag_text = models.CharField(maxlength=40, aka='Article.tag') the following sql can be auto-generated: 1) add table tag (w/ fk) 2) create one tag record for each article using the old tag data 3) drop the tag column in article or if Tag already existed, it would simply add the column and update all Tag records with their respective values from article. some data migration is already supported btw...look at the workarounds i had to do for sqlite. proof-of-concept-worthy, at least. > But the "applied migration code" side of it is severely > lacking, and that's exactly what everyone wants to see. really? but that's the easy part of the two. here's the key: there is no need to store schema versioning information in the db. first, for reliability, you can't trust any version in the db, so you have to scan the entire schema regardless. and if you're doing that, just dump it to a string and run it through a hash algorithm. this hash becomes the schema fingerprint. then store your migration scripts mapped from pairs of hashes (from_hash, to_hash) and your hashes in a list of all known hash values. then on startup (or manually triggered) a tool scans the existing schema, uses the fingerprint/hash to figure out where it is, then applies any scripts found matching it's from value. loop(scan, apply). when you run out of scripts, you're at current. or, give the tool a fingerprint to stop at. works wonders. i wrote it a few years ago in java to manage the 200+ table schema the air force uses to track all their students. (they have their own university system) great thing is that it works both waysdowngrade scripts are stored right next to upgrade scripts. but anyway... i agree they are two different problems; one that feeds into the other. it just seemed natural to write the introspection first, and it is useful by itself. (i would argue the more useful) > ps. Did you ever see my long-ago proposal? > (http://code.djangoproject.com/wiki/SchemaEvolutionProposal) yes... i used your process description. :) derek --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: schema evolution (new and improved)
Russell Keith-Magee wrote: > Your approach is predicated on the assumption that migrating > v1->v2->v3 is equivalent to v1->v3. This is true if you only look at > renaming, deleting or adding the fields themselves. so i think i have a cross-over solution. :) what it boils down to is there are four basic categories of developers: 1. users who trust introspection and never want to touch/see SQL 2. users who mostly trust introspection but want the option of auto-applied upgrades for specific situations (Wash) 3. users who use introspection-generated SQL, but don't trust it (they want it generated at development and stored for use in production) 4. users who hate introspection and just want auto-application of their own scripts who wish to perform different combinations of the two basic subtasks of schema evolution: 1. generation of SQL via magical introspection 2. storage and auto-application of upgrade SQL the first i've detailed before, so i won't cover here. it works the same way. the second works through auto-identification of known schema types, and the application of pre-rendered scripts. now this is very similar what other code out there does (for instance mikeh's quite capable dbmigration project [http://www.aswmc.com/dbmigration/]), but i feel in a simultaneously simpler and more flexible way. specifically, the following i consider key goals: * there is no need for schema identification metadata to be stored in the database itself * you should be able to swap between introspection and custom scripts freely within the same project, using as much or at little of each as you want (with little or no transition costs) * full migration graphs should be supported, not just linear steppings * it should be butt easy to use (because if it's not simpler than keeping your own text file, noone is going to use it) note: all this is covered in greater depth here: http://code.djangoproject.com/wiki/SchemaEvolutionDocumentation for the developers work, all they have to do is define a new module named 'schema_evolution' in their app directory, next to 'models'. this contains two data structures: * a list of known schema fingerprints * a dict mapping (from_fprint,to_fprint) pairs to scripts (lists of sql statements and/or python function calls, to be run in order...think pre_script/sql/post_script) the fingerprints act as both version numbers and ids, and are automatically generated. (either by you, running './manage sqlfingerprint my_app' or by sqlevolve or syncdb when they're looking for what scripts are available and comparing them to the existing schema) when schema evolution is applied, specified scripts ALWAYS take priority over introspection generated sql...so if you never want to see an aka field or generated sql, you never have to. different users i imagine will use it in different ways, but i imagine that most people will use introspection the majority of the time, ignoring fingerprints until they hit a bump that introspection can't handle. they'll simply code that one transition and continue on their way storing only that one tricky transition. a different subset of developers i imagine will always use the stored scripts during deployment, but will use introspection to generate the majority of them. but they will store all transitions, having a full version history of every state of their db stored in svn. anyway, an example of the evolution file is available in the documentation. let me know what you think. :) oh yeah, it's mysql only at this pointi want to get some feedback on the design before i write the others. thanks! derek p.s. mike i like your stuff but i feel that the whole-hog approach requiring developers to either start their db with your scripts or face a complicated setup is overly burdensome. plus users are lazyi don't want their hand-written scripts to be the only check for whether or not it's safe to apply a particular script. but maybe there are parts where we can merge or projects together? pm me. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: schema evolution (new and improved)
Jacob Kaplan-Moss wrote:
> On 8/7/07, Russell Keith-Magee <[EMAIL PROTECTED]> wrote:
>> Again - IMHO, the 'aka' syntax is a non-starter (for all the reasons I
>
> Russ is doing a great job speaking for me here, too.
>
> We're currently trying to move non-required metadata *out* of the
> model (c.f. the moving of the admin declarations in newforms-admin),
> and that process is the right one. Django's ORM is currently too
> coupled to supposedly-not-required pieces, and adding core code to
> deal with extra field attributes ("aka" in this case) is a
> non-starter.
as i said before i'm not wed to it. (actually i feel it's a rather
minor point) propose a different location. i just want it somewhere
simple and obvious. (or if you two just hate the keyword, what would
you like to call it?)
the only thing i'm against is completely duplicating the entire model
structure somewhere else, just to store one tiny extra bit of metadata.
that seems to be an unreasonable extra burden for the developer. i'm
a big proponent of DRY, as i thought i've read others here are too.
(one of the reasons i like django so much better than all the java wed
and ORM frameworks)
> Oh, and one other thing:
>
> Derek, I really appreciate your hard work on this. You're getting an
> large amount of pushback, but that's because this is a critical
> feature and we want to get it right.
>
> So please don't confuse criticism with a lack of gratitude!
i'm a yankee...don't worry so much about hurting my feelings! :-P
(but thanks for the thought)
derek
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---
Re: schema evolution (new and improved)
Russell Keith-Magee wrote: >> * full migration graphs should be supported, not just linear steppings >> * a list of known schema fingerprints >> * a dict mapping (from_fprint,to_fprint) pairs to scripts (lists of >> sql statements and/or python function calls, to be run in order...think >> pre_script/sql/post_script) > > This is an interesting idea. Some comments/questions: > > 1) Isn't the list of known schema fingerprints just the list of keys > in the migration graph/dictionary? no. it adds ordering. (the keys are a set remember) we wouldn't need it if python had something similar to java's LinkedHashMap, but alas... (besides, it's easier to read, don't you think?) > 2) From your code, a schema fingerprint is a hash of the output of > SHOW TABLES; this is an interesting approch, but there will be a major > problem going cross-platform. If I remove a field from a model, the > rough description is platform independent, but the SQL required isn't. > As a result, I would need to manage multiple sets of migrations - one > for each database I have to support. Wouldn't the model itself be a > better source for the fingerprint (or, at least, and abstracted > representation that has been derived from the database)? well, show tables + show index. and yes, it will be different for different backends. but as you said, the SQL is different too. if you're writing migration scripts by hand (or even generating them via introspection), you have to store one copy for each backend you support anyway... so you want the fingerprints to be different. > 3) If you are using fingerprints to identify nodes in the graph, how > do you handle cyclic states (e.g., v1, I add a field, v2 delete it ,v3 > add it again)? As I read your proposal, the fingerprint for v1 and v3 > should be the same - which will cause some major problems determining > resolution order for applying migrations. no problem. migrations to later versions take priority, which following the django "do the right thing" philosophy. on the rare time you want to upgrade to v2, comment out the to_v3 migration temporarily. (or more likely, the only time you would want to upgrade to v2 would be when you checked out an older version, which wouldn't have the v3 script anyway) alternatively, a "migrate-to" argument to the sqlevolve action can be set. this is how i intend to support downgrades anyway. > 4) If you are using model fingerprints to identify changes, how do you > propose to keep parallel changes synchronized? For example, Model 1 > goes from v3->v4, but that change relies upon Model 2 going from > v6->v7. If the pair of changes share a common label, the label can be > used as a point of synchronization; however, the fingerprint (by > itself) doesn't provide such a reference point. the fingerprint is for the whole app schema, not individual models. this scenario couldn't exist. > 5) What is the role of the post-state here? Is it your intention to > use the post-state for validation that a migration has been > successful? yes, post-validation i believe is important. plus, it allows us to pick the right script when you have a cyclical schema change. plus, it lets you store reversal scripts, so you can have automatic downgrades as well as upgrades. plus, it lets you store arbitrary graphs of upgrades, so you can write a v1->v3 script along with v1->v2 and v2->v3. (important for the larger databases) > If so, how do I populate the post-state value before I > have applied the migration for the first time? you don't have to...just stick a dummy value in there (or leave it blank) and run your script. (either manually or w/ the tool) then call fingerprint, and copy-paste the new value over your dummy value. derek --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: schema evolution (new and improved)
i have had this exact situation happen to me before. (rolling out a change, then another change reversing it) actually several times. i have NEVER had a scenario where client Y says "can you please make sure i lose my data?" your schema integrity in assured. the data integrity is also assured (meaning no invalid data). but whether every value in every field is exactly the same between two clients? that's a whole other ballgame. but really, when have two clients EVER had exactly the same data? Mike H wrote: > Russell Keith-Magee wrote: >> 3) If you are using fingerprints to identify nodes in the graph, how >> do you handle cyclic states (e.g., v1, I add a field, v2 delete it ,v3 >> add it again)? As I read your proposal, the fingerprint for v1 and v3 >> should be the same - which will cause some major problems determining >> resolution order for applying migrations. >> >> > > Situations like this are the reason I have generated linear steppings > which are sequenced and run in order, in bulk. > > I may have misunderstood how this is proposed to work, so please correct > me if I have. > > Deleting the column and adding it again has the effect of resetting the > data to the default value for the column. > > Say I have two clients, X and Y. > > I roll out revision 1 to both clients. > > I roll out revision 2 to client X. This deletes a column. > > I roll out revision 3 to client X. This adds the column again. > > I roll out revision 3 to client Y. This does nothing to the column as > the fingerprints are the same. > > Now I have the same application, at the same revision, rolled out to two > clients, but they have not treated the data in the same way. This is not > acceptable - making sure you have the correct schema structure at the > end of a migration is vital, but data integrity must not be compromised > in any way. > > If I've misunderstood, please tell me how this situation would be dealt > with :) > > Cheers, > > Mike > > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: schema evolution (new and improved)
Mike H wrote: > Sorry, no. In my simple example, the data is not in the state that the > developer expected. He asked for the column to be deleted, then > recreated. this was brought up when i first designed this for the af. my answer was: it's pretty obvious when you're returning to an identical copy of a previous schema. if the developer _really_ needs to force the user to go through the drop/add, all he needs to do is make any trivial change ot the schema elsewhere. for instance, adding an empty table titled "PLEASE_DELETE_ME_AFTER_V2". ugly, yes, but it calmed the "what if" concerns. (and we never had to) but wearing my developer hat, i would have seriously reprimanded one of my guys had written a feature that would fail on "i wasn't expecting data here". if the schema can't verify it (such as is the case when you're changing how a number is calculated), the app needs to. > As I think I've said before (possibly not on this list) my ideal way of > solving this would be to run : > > ./manage.py generate-evolutions app_name > > which would generate [sql / python code representing the changes] and > put it into a migration file. I could then review this to put in any > data manipulation I needed, if I needed to. For 95% of changes, I > probably wouldn't need to. Then I'd apply it by running > > ./manage.py syncdb > > That way everything gets run in sequence and there is no possibility for > it surprising the developers. Yes, you might get a few more migrations yep. that's exactly the scenario i described for the "kaylee" developer here: http://code.djangoproject.com/wiki/SchemaEvolutionDocumentation and probably the ideal for large/serious development work. but i don't think we need to force everyone to do it this way. > Don't get me wrong, I've very excited about the prospect of > schema-evolution getting merged :) danke :) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: schema evolution (new and improved)
Yuri Baburov wrote: > 1) How one writes operation to reset all field values to default? Is > it a migration at all? just stick an "update mytable set col=value;" in your upgrade script. (but if you're doing this outside of any schema changes, yes i would argue that this is not a migration issue) > 2) How about python-written migrations? in the list of sql commands, pass in a function reference instead. (not implemented yet, but that's how i plan on it working) i originally thought of just adding "pre/post" function support, but then thought "why limit it to that?" what if you wanted sql/python/sql/python? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: schema evolution (new and improved)
Yuri Baburov wrote: \> 1) Would the app_name/schema_evolution.py be automatically generated/updated? > I don't want to be a monkey copying output of sqlevolve and > sqlfingerprint to that file. I better remove all parts of file that I > won't use. no. i think that those who don't trust introspection (who are most likely to be the heavy users of schema_evolution.py) aren't going to trust the introspection code to auto-place itself in their deployment scripts. (they might not notice an introspected change, and it get accidentally leaked to development wrecking the lives of millions) a copy-paste requirement is your protection from my introspected evil. :) > 2) There are a lot of cases of information loss with your proposed solution. > Automatic v1 => v3 wouldn't work exactly as v1 => v2 => v3 if you add > column in v2 and change default value of this column from null to same > value in v3. > My position (and not only mine): Updates shall be consistent! then don't use the full graph support. only write/approve v1=>v2, v2=>v3, v3=>v4 scripts and you always have a guaranteed always-equal migration for your apps. it's only there if you want it. > But I don't want them to be SQL, because they would > be db specific and you lost information again! > ... keep all history of my schema changes ... but not in models or > backend-specific sql scripts. "db-specific == losing information" i don't understand...but i've no interested in writing a new database access language. we've already got a ton (including the ORM already built into django), and you can use them in the function call handling i've already proposed. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: schema evolution (new and improved)
Russell Keith-Magee wrote:
> I thought you were proposing moving away form ordered sequences? Besides:
i am moving towards graphs of schema revisions. but while lines are
always graphs (and graphs can have order), only the simplest graph can
be represented as a line. and graph paths are always represented as
pairs of nodes. there is no way of representing a directed graph
without specifying the start node and end node of each path:
http://en.wikipedia.org/wiki/Directed_graph#Directed_graph
the post validation is just free icing.
> migrations = {
>'f1': stuff1,
>'f2': stuff2,
>'f3: stuff3,
> }
ie, this^^^ will never support anything but linear, one-way migrations.
which brings us to the greater point. no graph implementation can
guarantee you can't have multiple paths. that's up to the developer if
it's a constraint s/he wishes to impose. and multiple paths means you
can't have guaranteed data equivalence, as you pointed out. (even
though you can still have data validity, and schema consistency, which i
argue is good enough) but multiple paths is a feature, not a bug, of my
design. i'm not going to "fix" my implementation by removing support
for it.
yet you can easily still create single-path migrations. just define
your graph that way.
but on a larger "forest for the tree's" note russ, i'm hearing that:
1) no schema migration that supports multiple paths will ever be allowed
into django-proper. (not atomic, as you put it, even though i think
that's not what atomic means)
2) no introspection that works using my inadequate metadata syntax (or
any syntax similar to it, inside or outside models.py) will ever be
allowed in.
which means you're looking for something designed substantially
different from what i originally proposed last summer, what i've
developed, and what i have any interest in developing. should i go back
to just maintaining this for my own personal development use? can i
recommend dbmigration, which seems much more up your alley design-wise?
or does anyone have an interest in continued public development of
automated introspection + migration w/ multi-path support? (i promise
FK/M2M introspection is coming soon if there's a yes vote out
therebut it's mysql only right now and i won't bother finishing
postgres support unless i'm releasing publicly)
derek
p.s. not taking anything personal here, before anyone gets worried
about that again...just i think we might have irreconcilable differences
on the basic design elements, and with your "strong -3" on design i'm
not holding much hope that i'm doing more than pissing in the wind. :)
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---
Re: schema evolution (new and improved)
Brantley Harris wrote: > Er, could you re-factoritize your system into a stand-alone app? Then > it could be added by anyone. That seems like win/win to me. i considered this. but i have my doubts that i could get the necessary hooks pushed through to allow this to function as an external app. if i have even a minor interest base however i'm willing to continue publicly maintaining a patch against HEAD and/or the latest release. i just want to know... --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: schema evolution (new and improved)
Jacob Kaplan-Moss wrote: > So what hooks do you think you'd need? Would the sql-level hooks being > discussed in other threads be sufficient? would you allow a hook for still annotating in models.py? derek --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Django 100% threadsafe with DB?
> In short, Django was not *designed* to be threadsafe, but any obvious > problems that I'm aware of have been fixed. YMMV. that's scary. but then again, python itself isn't multi-threaded. (all threading is faked - google "global interpreter lock". lazy s.o.b. python devs) so all your really hairy "c=c+1" type issues are already nixed. so not so scary. derek Joseph Kocherhans wrote: > On 9/25/07, Graham Dumpleton <[EMAIL PROTECTED]> wrote: >> Can we please somehow settle this issue once and for all. I have tried >> to get discussions going on this issue in the past but have got >> minimal feedback. I thought that too a degree it had been determined >> that multithreaded servers were okay, although users should though >> ensure there own code is multithread safe, but now again someone is >> saying that Django itself is not multithread safe. :-( > > I talked with Jacob about this quite a while ago and he told me that > Django was not originally written to be threadsafe. The only threading > problems I remember hearing about were with the database connections, > and those issues were fixed in #1442 [1]. To my knowledge, there has > never been any review of the code to check for other possible sticky > spots. I used to deploy Django on Windows and never had any threading > problems, but the sites were mostly low traffic, internal, and > probably not good candidates for exposing problems. > > In short, Django was not *designed* to be threadsafe, but any obvious > problems that I'm aware of have been fixed. YMMV. > > Joseph > > [1] http://code.djangoproject.com/ticket/1442 > > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Django 100% threadsafe with DB?
you mean to say cooperative multithreading, right? if so, heck yeah. dear lord in heaven yeah. Nicola Larosa wrote: > Derek Anderson wrote: >> but then again, python itself isn't multi-threaded. (all threading is >> faked - google "global interpreter lock". lazy s.o.b. python devs) so >> all your really hairy "c=c+1" type issues are already nixed. >> >> so not so scary. > > Right. What *is* is scary is how much people cling to the horrible hack > that preemptive multithreading is. > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Django 100% threadsafe with DB?
well, that was sarcasm with the "lazy" comment...come on, i love python. why else would i be here? :) but it is faked. two python threads can't run concurrently on a two processor machine. unless, as you pointed out, you're calling a few hand-picked C operations (mostly i/o). they're like green threads from java 1.0. "faked" is about as good of an adjective as you get. still love python tho. derek Fredrik Lundh wrote: > Derek Anderson wrote: > >> but then again, python itself isn't multi-threaded. (all threading is >> faked - google "global interpreter lock". lazy s.o.b. python devs) > > given that a stock CPython interpreter releases the lock in a few > hundred places, primarily around potentially long-running or blocking C > operations, claiming that "all threading is faked" is a bit misleading. > maybe you should do a bit more research before you start calling > people names? > > > > > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Schema Evolution status
SmileyChris wrote: > I like Derek's take on model comparison using introspection, but > really think that it should be based on a migration system. I know > several Django developers agree with me on this (including Russell). i see them as two distinct, non-exclusive tasks, with automatic introspection during development *optionally* feeding into svn-ed migration scripts for large or widespread production deployments. derek --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Schema Evolution status (official)
ok, for the official status: as mentioned earlier, yuri and i are joining forces on introspection-driven schema evolution. and since automatic introspection is so controversial (and obviously a poison pill for acceptance into django-proper), we've rewriting it as an external library. the project page is here: http://code.google.com/p/deseb/ i expect a release in the next week or two. the dreaded "aka" syntax will remain, but will require one additional import statement to the top of your models.py. and management integration will continue through the external command support recently added into the trunk. i agree with yuri that requiring users to write migrations in some new django-specific SQL abstraction language is dumb, not DRYish and an all-around pita. if you want that that's ok, but it's not the itch we're scratching here. (*) as soon as we have a working implementation, i will be stopping support for the schema-evolution branch. thanks, derek (*) however that's not to say wide-scale-production-deployment management as a whole isn't necessary - we just see it as a separate issue. and one compatible with this project. (output from sqlevolve would be saved into whatever system you wish to use) Yuri Baburov wrote: >> I'm actually waiting for some code to be put up to >> http://code.google.com/p/django-evolution/ - Russell (I'm pretty sure >> it was him... we discussed it at the end of the sprint) has someone >> who has been working on some code and I really want to see it before I >> do any more work. > You better name the project django-migration then and left > django-evolution name for us ;) > >> I like Derek's take on model comparison using introspection, but >> really think that it should be based on a migration system. I know >> several Django developers agree with me on this (including Russell). > Maybe you are not against introspection but against lack of migration system? > I'm just don't want to make writing migrations by hands a necessary > operation. For small projects it's a pita. It's equally dumb to > manually fixing tables with sql. It's not DRYish. > But I'm vote with strong +1 for possibility to write them with hands > and correct them with hands. > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Schema Evolution status (official)
Russell Keith-Magee wrote: > I'm not sure where you got the idea that automatic introspection is > the issue - the proposal I put forward included automatic > introspection. The issue has always been the aka syntax, and the > consequences of that syntax on the overall design. because several of the complaints you've and others have listed would apply to any automatic introspection implementation. (most memorably the 'round-trip-change' data issue) but if i'm mistaken i apologize. btw, i must have missed your proposal. (unless you're referring to your 8/4/2007 email?) link? > Derek - on a housekeeping issue - are you happy for us to write-lock > the schema-evolution branches and close the outstanding tickets in > Django's ticket database relating to that branch? if you don't mind, i'd prefer you keep them open, lest as you suggest we get sudden fame and fortune and merging re-enters the realm of distant feasibility. :) (maintaining should be pretty easy anyway, once the majority of it is separated. i'll make that call once we release, i'm just not ready to commit to it yet) derek --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Schema Evolution status (official)
russell, i've re-read your linked email (from 8/4/07), and i'm still
totally lost as to what you're proposing. i'm reading of application
and verification of an SQL-abstraction syntax:
mutations = [
AddColumn('Author', 'dateofbirth', models.DateField,
initial_value=None),
DeleteColumn('Author', 'age')
]
combined with a signature list embedded in the Meta class in model.
(btw, you said earlier that "in_the_model==bad" ???)
which is ok and all, but there is nothing on how to actually generate
these mutations, except for the suggestion of calling it "syncdb --hint"
(instead of "sqlevolve" i presume). is it up to the user to write them?
where is the actual introspection/generation bit?
derek
Russell Keith-Magee wrote:
> On 9/27/07, Derek Anderson <[EMAIL PROTECTED]> wrote:
>> Russell Keith-Magee wrote:
>>> I'm not sure where you got the idea that automatic introspection is
>>> the issue - the proposal I put forward included automatic
>>> introspection. The issue has always been the aka syntax, and the
>>> consequences of that syntax on the overall design.
>> because several of the complaints you've and others have listed would
>> apply to any automatic introspection implementation. (most memorably
>> the 'round-trip-change' data issue) but if i'm mistaken i apologize.
>
> My objection was to the equivalence between A->B->C and A->C under
> your proposal. This is mostly a consequence of the limitations of the
> aka-syntax as applied to introspection. I have no problem with the
> concept of introspection itself, and the proposal I floated
> specifically mentions introspection as a tactic that can be used, both
> for validation and for identifying required migrations.
>
>> btw, i must have missed your proposal. (unless you're referring to your
>> 8/4/2007 email?) link?
>
> http://groups.google.com/group/django-developers/browse_thread/thread/da7831d08081d7b7/49347e67d22fa3cc?rnum=1#49347e67d22fa3cc
>
>>> Derek - on a housekeeping issue - are you happy for us to write-lock
>>> the schema-evolution branches and close the outstanding tickets in
>>> Django's ticket database relating to that branch?
>> if you don't mind, i'd prefer you keep them open, lest as you suggest we
>> get sudden fame and fortune and merging re-enters the realm of distant
>> feasibility. :) (maintaining should be pretty easy anyway, once the
>> majority of it is separated. i'll make that call once we release, i'm
>> just not ready to commit to it yet)
>
> No problems.
>
> Yours,
> Russ Magee %-)
>
> >
>
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---
Re: Schema Evolution status (official)
ok, yes, introspection we all agree has to be used for validation (1)
and signature creation (2). we're all on the same page as that being a
good thing.
but what SoC2006, my subsequent work and what i think most people here
are talking about when we say "evolution through introspection" is about
using introspection to determine *what needs* to change, not verifying
what has already changed. in your proposal, how do you write your
"mutations" list? the developer? or and introspection-based program?
if the former, just say so.
if the latter, how?
derek
Russell Keith-Magee wrote:
> On 9/27/07, Derek Anderson <[EMAIL PROTECTED]> wrote:
>> russell, i've re-read your linked email (from 8/4/07), and i'm still
>> totally lost as to what you're proposing. i'm reading of application
>> and verification of an SQL-abstraction syntax:
>>
>> mutations = [
>>AddColumn('Author', 'dateofbirth', models.DateField,
>> initial_value=None),
>>DeleteColumn('Author', 'age')
>> ]
>>
>> combined with a signature list embedded in the Meta class in model.
>> (btw, you said earlier that "in_the_model==bad" ???)
>
> It needs to be _somewhere_. Meta is one option, but there are better
> options. I was suggesting Meta as a better location than where you
> were/are putting the settings (i.e., on the attributes themselves).
> However, out of the model entirely would be preferable (and is
> entirely possible).
>
>> which is ok and all, but there is nothing on how to actually generate
>> these mutations, except for the suggestion of calling it "syncdb --hint"
>> (instead of "sqlevolve" i presume). is it up to the user to write them?
>> where is the actual introspection/generation bit?
>
> Introspection isn't an essential part of what I was proposing, but it
> certainly could be used in two places:
>
> 1) Validation of final state - that the final state of the table after
> migration matches the model definition. The AddColumn/DeleteColumn
> mutations know what modifications they will make to the model
> signatures, and this can be used as a form of soft validation.
> However, the ultimate validation is that the tables in the database
> match expectation at the end of a migration process. This can't be
> done without introspection.
>
> 2) Reconciliation of modifications that have been made outside the
> Django framework - i.e., if you tweak the tables outside of Django,
> the signature in the Evolution table won't match the actual database
> state. As a result, there will be a need for an evolution step to
> 'update the Evolution table'. This absolutely requires introspection,
> as the state of a model definition is irrelevant to the problem.
>
> Introspection could also be used as a replacement for the tag-tracking
> function of the Evolution table. However, the Evolution table is a
> much easier development task, and I think there is some value in
> keeping the migration process documented in the database.
>
> Yours,
> Russ Magee %-)
>
> >
>
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---
Re: Schema Evolution status (official)
Russell Keith-Magee wrote: > The latter, by comparing the signature of the models.py that you have > with the signature in the Evolution table. The evolution table > contains the signature of the last model that was sync'd; if this > doesn't correspond to the current model, you need migrations to update > the database. ok, i get that. i didn't realize you meant storing the entire model structure in the db. (might want to avoid calling this a signature for clarity, but that's semantics) so now you have two models to diff. how do you detect renames? derek --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Schema Evolution status (official)
Russell Keith-Magee wrote:
> It's not the _model_ per se - just a rendition of the significant data
> in the model.
a rose by any other name (but yes, i assumed you meant not the
actual textual rendition, but a data structure containing all the
database-relevant attributes of the model)
>> so now you have two models to diff. how do you detect renames?
>
> AFAICT, you _can't_ detect a rename with with 100% reliability.
> Whatever scheme you propose will be subject to either:
> [...]
> Either outcome is undesirable for a real database. This is why the
> 'hint and tweak' approach is so essential.
agreed 100% that fully that detecting renames without hinting is
impossible. in fact i argue that fully detecting renames is impossible
under any circumstances. (see corollary A)
==
so we have two remaining issues here:
1) how to store all database-relevant attributes of the
model in the database itself
2) how to let the user 'hint and tweak' at renames
==
it would seem everything else we're agreed on. lets see what we can do
to come together.
== issue #1 ==
originally i had planned something virtually identical to what you're
proposing now. push all relevant information into a data structure, and
store that in the database. but the key insight was as follows:
"all relevant information" by definition IS the schema (x)
and according to DRY (not to mention just general programming common
sense), if you have the information, but just in an inconvenient form,
transform it (from the most authoritative source). _don't_ re-store it.
because syncing issues are hard (which you're already acknowledged in
your "people may muck with the schema behind django's back" issue).
(note corollary B)
Q1: do you agree that IF claim(x) is true, we should use the schema?
Q2: do you agree with claim(x)?
== issue #2 ==
when do you ask for the hints? during development, in the code? or
during deployment, via user interaction with syncdb?
i argue that "via user interaction with syncdb" is bad for the following
reasons:
a) i don't want to have to leave notes for my deployment monkeys
saying "make sure you match column A with column C". this is
just asking for deployment problems.
b) i want that information stored in CVS (or SVN, etc)
so that leaves storing it in the codebase _somewhere_ as the better
option, IMHO.
Q3: do you agree with this? ^^^
::: corollary A :::
for the proof that no renaming scheme is perfect... (ie, no
false-positives & no false-negatives):
consider FRANK deletes a field, and re-ads the same field before running
"sqlevolve" or "syncdb -- hint" or whatever. he commits both changes to
his VCS as v1 and v2, so it's real. because there are no globally
unique field ids, no system can detect and handle this change
consistently without peeking into his VCS. by manual db management, all
the data in the field is lost. via all automatic evolution systems, the
data is not. you are guaranteed data-inconsistency.
(however schema-consistency is attainable)
Q4: do you agree with this? ^^^
::: corollary B :::
note that i'm not against storing historical model or schema
representations to the DB, just that for the current representation, we
should use the schema over using our own stored "we think the DB looks
like this" data structure, as the schema is the most authoritative source.
derek
p.s. long emails on complicated topicsdetails get murky. so i've
labeled the four major questions {Q1-4} so we can further isolate where
we do and do not see eye2eye. :)
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---
Re: Schema Evolution status
Paul Davis wrote: > > As near as I can tell these are the main issues that don't seem to be resolved: > > 1. Balancing ease of use with power of use (Ie, Alice vs. Carol) > 2. Level of versioning: Model vs. Application vs. Entire database > 3. Application of versions: v1 -> v2 -> v3 =? v1 -> v3 > 4. How do we represent a migration from state 1 to state 2 > #1 is a nobrainer. We want both. And we won't settle for less. agreed. this is why fingerprinting, versioning and controlled deployment scripts are all optional in my implementation. (as i feel they must be to support alice) > #2 There is a lot of talking around this issue but I haven't seen > anyone take it full on. > We have to version the entire schema as a single object. Anyone in > doubt, I refer to SVN's repository wide versioning vs CVS's file > specific versioning. i agree that at _least_ applications need to be versioned together. i lean towards agreeing that we should version entire schemas, but w/ django not having any existing inter-app dependency mechanism (other than they just won't work), i was reluctant to mandate it. mostly agree, but i think the ramifications need to be better thought out before we push it on anyone. > #3 There are those that think v1->v2->v3 == v1->v3. I am not one of > those people. > A database is defined (in my mind) by its schema and the data it > contains. i think it depends on who you are. for alice at least, can you agree it will be true? for the majority of apps/developers too, i'd argue. but for disburse deployments, i've conceded before that support needs to be there for v1->v2->v3. (and it is supported via my controlled deployment scripts) > Migrations are non-linear. thank $DEITY someone else said this. :) imho any controlled deployment system that _only_ supports lists of sequential versions is DOA. > #4 This is minor, but I felt it necessary to say that we're obviously > going to need python scripts to go from state to state. And custom sql > would be easy to integrate. agreed. > So those are the main points near as I can tell. Now I'll go over some > of the popular topics I saw: > > 1. The use of introspection: > Will introspection be used? Of course. Will it be *all* thats used? Of > course not. agreed, but i think the controversy is over using introspection to _generate_ our guesses. (heck even syncdb uses introspection - noone's arguing we shouldn't use any) > 2. The famous 'aka' syntax. For those of you out there saying that > this is absurd and doesn't provide any information, I'm thankful. The > syntax is horrible. Remember my quip about SVN vs. CVS versioning? > This is like some ultra brain dead system that trys to version > individual lines in a file. I mean, not even CVS is that bad. you're confused as to it's purpose. it is certainly *not* an attempt to version anything. it's there as a necessary hint to the guess_generation routine. i think everyone acknowledges we need some sore of a hinting syntax on top of the version tracking if we want to do any generation of change scripts... the question is where? the reasons i put it where i did are as follows: 1) easy to use / grok for everyone, even for alice. 2) all field referencing has some ambiguity, if you want to support fields that no longer exist.. best to have only one side ambiguous by attaching it directly to the other. i agree it would be a brain-dead VCS implementation, but it's not. so that being understood, is it more palatable now? derek --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Schema Evolution status
Paul Davis wrote: > > As near as I can tell these are the main issues that don't seem to be > resolved: > > 1. Balancing ease of use with power of use (Ie, Alice vs. Carol) > 2. Level of versioning: Model vs. Application vs. Entire database > 3. Application of versions: v1 -> v2 -> v3 =? v1 -> v3 > 4. How do we represent a migration from state 1 to state 2 > #1 is a nobrainer. We want both. And we won't settle for less. agreed. this is why fingerprinting, versioning and controlled deployment scripts are all optional in my implementation. (as i feel they must be to support alice) > #2 There is a lot of talking around this issue but I haven't seen > anyone take it full on. > We have to version the entire schema as a single object. Anyone in > doubt, I refer to SVN's repository wide versioning vs CVS's file > specific versioning. i agree that at _least_ applications need to be versioned together. i lean towards agreeing that we should version entire schemas, but w/ django not having any existing inter-app dependency mechanism (other than they just won't work), i was reluctant to mandate it. mostly agree, but i think the ramifications need to be better thought out before we push it on anyone. > #3 There are those that think v1->v2->v3 == v1->v3. I am not one of > those people. > A database is defined (in my mind) by its schema and the data it > contains. i think it depends on who you are. for alice at least, can you agree it will be true? for the majority of apps/developers too, i'd argue. but for disburse deployments, i've conceded before that support needs to be there for v1->v2->v3. (and it is supported via my controlled deployment scripts) > Migrations are non-linear. thank $DEITY someone else said this. :) imho any controlled deployment system that _only_ supports lists of sequential versions is DOA. > #4 This is minor, but I felt it necessary to say that we're obviously > going to need python scripts to go from state to state. And custom sql > would be easy to integrate. agreed. > So those are the main points near as I can tell. Now I'll go over some > of the popular topics I saw: > > 1. The use of introspection: > Will introspection be used? Of course. Will it be *all* thats used? Of > course not. agreed, but i think the controversy is over using introspection to _generate_ our guesses. (heck even syncdb uses introspection - noone's arguing we shouldn't use any) > 2. The famous 'aka' syntax. For those of you out there saying that > this is absurd and doesn't provide any information, I'm thankful. The > syntax is horrible. Remember my quip about SVN vs. CVS versioning? > This is like some ultra brain dead system that trys to version > individual lines in a file. I mean, not even CVS is that bad. you're confused as to it's purpose. it is certainly *not* an attempt to version anything. it's there as a necessary hint to the guess_generation routine. i think everyone acknowledges we need some sore of a hinting syntax on top of the version tracking if we want to do any generation of change scripts... the question is where? the reasons i put it where i did are as follows: 1) easy to use / grok for everyone, even for alice. 2) all field referencing has some ambiguity, if you want to support fields that no longer exist.. best to have only one side ambiguous by attaching it directly to the other. i agree it would be a brain-dead VCS implementation, but it's not. so that being understood, is it more palatable now? derek --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Schema Evolution status (official)
well, this i just don't understand. plenty of programming topics
considerably more challenging than this are solved via listserv every
day in the open source world. i'd rather have a public discussion
incorporating everyone's needs, ideas and concerns, not just "you and
ben" deciding between yourselves what you think is best...and then some
sort of wild-west-style publicity shootout.
to the other people who have weighed into this discussion...am i alone
thinking along these lines?
derek
Russell Keith-Magee wrote:
> On 9/28/07, Derek Anderson <[EMAIL PROTECTED]> wrote:
>> p.s. long emails on complicated topicsdetails get murky. so i've
>> labeled the four major questions {Q1-4} so we can further isolate where
>> we do and do not see eye2eye. :)
>
> Rather that descend in to another few weeks of multi-page emails where
> we misunderstand each others interpretation of terms like 'signature'
> and 'introspection', I think the most productive thing I can do at
> this point is to concentrate on helping my friend Ben get a prototype
> out the door so that we can all have a discussion based upon actual
> code, rather than a quick-and-dirty mailing list specification.
>
> I've seen your code, and pointed out where I see problems. I hope to
> be able to return the favour very soon. Then it will be use cases at
> 10 paces. :-)
>
> Yours,
> Russ Magee %-)
>
> >
>
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---
Re: Schema Evolution status (official)
hey marty, i agree. see: http://code.djangoproject.com/wiki/SchemaEvolutionDocumentation maybe another day or two for the stand-alone version... derek Marty Alchin wrote: > I won't get into the discussion on features or implementation yet, but > I do have to agree that working code speaks volumes compared to > descriptions. I'll have need for schema evolution in a future project, > so I've been following these discussions, and I've completely lost > track of who's working on what, and how each offering is expected to > work. > > Personally, what I'd like to see is this: > > * Each view is presented with working code. It doesn't have to be > feature complete, but does some basic jobs, to illustrate how it would > work > * Along with the code, give a list of features it provides. Again, it > doesn't have to be all the features it will ever provide, just those > that are currently implemented. > * A brief (yes, brief) description of why you chose a particular > approach. This way, if two people come up with different approaches to > the same problem, it's easier to compare the reasoning behind them. > * If possible, some sort of feature matrix comparing the states of the > various implementations. So if it looks like one group is tackling > half of the problems, while another group is tackling the other half, > maybe merging them could be easy. > > It'd probably be easiest to do the descriptions on the Wiki, with each > proposal having its own article. The current SchemaEvolution article > tries to cover everybody at once, and it does a poor job of organizing > information. I would suggest that article be used for descriptions of > the problems that need to be solved by schema migration, the feature > matrix comparing the implementations, and links to individual articles > that provide more detail for each implementation. > > Each article would then describe the implementation's motivations and > tactics, example usage code, and links to relevant supporting > articles, previous discussions, and working code. There's merit in > documenting future plans in each individual article, but I would like > to see the main SchemaEvolution only cover those features that are > actually finished. > > I'd be willing to set this up if there's support for it, but if you > guys don't want to maintain it, I won't bother. It's just one > suggestion, but I think it would help. > > -Gul > > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Django External Schema Evolution Branch (deseb)
Django External Schema Evolution Branch === http://code.google.com/p/deseb/ I've released v0.2, which supports both MySQL and Postgresql, and works with both django-head and django-v0.96. This release contains the introspection/generation code only. The "schema signing/fingerprinting" and "managed schema updates" have yet to be ported. Installation is dead simple. Download and extract it into your site-packages dir, and add "import deseb" to settings.py. If you have an interest, please give it a whirl and send me feedback. (either privately or here, I don't care) But please, restrain from publicly flogging it/me until AFTER you've actually used it. =p Love ya all, Derek -- looking to buy or sell anything? try: http://allurstuff.com it's a classified ads service that shows on a map where the seller is (think craigslist + google maps) plus it's 100% free :) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Django External Schema Evolution Branch (deseb)
hey nis,
1) psycopg2 has been added as a backend
2-4) the other three bugs i've verified and fixed in svn. (plus i've
started a new "everything" test case to check all the field types from
now on)
> I think the introspection code for postgres can be tightened a lot -
> especially if we limit ourselves to versions >=7.4 - I am happy to help
> in this area.
i'll take any suggestions you have. :)
thanks,
derek
Nis Jørgensen wrote:
> Derek Anderson skrev:
>> Django External Schema Evolution Branch
>> ===
>> http://code.google.com/p/deseb/
>>
>> I've released v0.2, which supports both MySQL and Postgresql, and works
>> with both django-head and django-v0.96.
>>
>> This release contains the introspection/generation code only. The
>> "schema signing/fingerprinting" and "managed schema updates" have yet to
>> be ported.
>>
>> Installation is dead simple. Download and extract it into your
>> site-packages dir, and add "import deseb" to settings.py.
>>
>> If you have an interest, please give it a whirl and send me feedback.
>> (either privately or here, I don't care) But please, restrain from
>> publicly flogging it/me until AFTER you've actually used it. =p
>>
> I am now running into my fourth problem needing fixing before I can
> "actually use it". I have no more time for this now - but I am happy to
> help later.
>
> First problem:
>
> psycopg2 is not supported
>
> First fix:
>
> --- deseb/schema_evolution.py 2007-09-30 21:06:17.0 +
> +++ /usr/lib/python2.4/site-packages/deseb/schema_evolution.py
> 2007-10-02 08:08:00.0 +
> @@ -29,8 +29,10 @@
>
> backend_type = str(connection.__class__).split('.')[3]
> if backend_type=='mysql': import deseb.backends.mysql as backend
> -if backend_type=='postgresql': import deseb.backends.postgresql as
> backend
> -if backend_type=='sqlite3': import deseb.backends.sqlite3 as backend
> +elif backend_type in ['postgresql', 'postgresql_psycopg2']: import
> deseb.backends.postgresql as backend
> +elif backend_type=='sqlite3': import deseb.backends.sqlite3 as backend
> +else:
> + raise Exception('Backend not supported') # This should raise a
> more relevant exception
> ops = backend.DatabaseOperations(connection, style)
> introspection = backend.DatabaseIntrospection(connection)
> return ops, introspection
>
> Second problem:
> Unknown function ops.quote_name
>
> Second fix:
>
> (this may not be how it was intended to work)
>
> --- deseb/schema_evolution.py 2007-09-30 21:06:17.0 +
> +++ /usr/lib/python2.4/site-packages/deseb/schema_evolution.py
> 2007-10-02 08:08:00.0 +
> @@ -409,8 +411,8 @@
> if f.rel:
> if f.rel.to in known_models:
> field_output.append(style.SQL_KEYWORD('REFERENCES') + '
> ' + \
> -
> style.SQL_TABLE(ops.quote_name(f.rel.to._meta.db_table)) + ' (' + \
> -
> style.SQL_FIELD(ops.quote_name(f.rel.to._meta.get_field(f.rel.field_name).column))
> + ')' #+
> +
> style.SQL_TABLE(connection.ops.quote_name(f.rel.to._meta.db_table)) + '
> (' + \
> +
> style.SQL_FIELD(connection.ops.quote_name(f.rel.to._meta.get_field(f.rel.field_name).column))
> + ')' #+
> #backend.get_deferrable_sql()
> )
> else:
>
> Third problem:
>
> Default values are not always strings:
>
> File "/usr/lib/python2.4/site-packages/deseb/backends/postgresql.py",
> line 126, in get_known_column_flags
> dict['default'] = row[1][1:row[1].index("'",1)]
> ValueError: substring not found
>
> Fix:
>
> (more problems are likely to exist for other data types - but at least I
> get past the error)
>
> Only in /usr/lib/python2.4/site-packages/deseb/backends: __init__.pyc
> diff -u -r deseb/backends/postgresql.py
> /usr/lib/python2.4/site-packages/deseb/backends/postgresql.py
> --- deseb/backends/postgresql.py2007-09-30 02:28:43.0 +
> +++ /usr/lib/python2.4/site-packages/deseb/backends/postgresql.py
> 2007-10-02 08:28:06.0 +
> @@ -122,7 +122,10 @@
> for row in cursor.fetchall():
> if row[0] == column_name:
> if row[1][0:7] == 'nextval': continue
> -
Re: Django External Schema Evolution Branch (deseb)
yes, because that is what django does with it's psycopg2 backend. (or at least did, when i originally added psycopg2 to the branch) i'm not a big postgres user, i don't know what the differences between the two are, i didn't know why they had done so originally, so until i figured it out i thought i should accept they had a reason and follow by example. but my preference would certainly be not duping. you're right, i should have just added "from postgresql import *" for now until i came across an actual difference. and if someone can explain to me the difference between the two and we can, i would love to merge them alltogether. derek Nis Jørgensen wrote: > Derek Anderson skrev: >> hey nis, >> >> 1) psycopg2 has been added as a backend >> > Any reason for not using my code (importing postgresql.py for both > backends) rather than having two source files which are identical? > Shouldn't we at least replace postgresql_pcycopg2.py with > > from postgresql import * > > ? > > I cannot imagine any case where we would want to have different code > depending on the driver, since we are working solely in the SQL layer. > > Don't Repeat Yourself > > Nis > > > > -- looking to buy or sell anything? try: http://allurstuff.com it's a classified ads service that shows on a map where the seller is (think craigslist + google maps) plus it's 100% free :) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Django External Schema Evolution Branch (deseb)
cool. looks like nothing that'll effect us. i've switched back to one common postgresql backend. danke. :) Yuri Baburov wrote: > 2007/10/3, Derek Anderson <[EMAIL PROTECTED]>: >> yes, because that is what django does with it's psycopg2 backend. (or >> at least did, when i originally added psycopg2 to the branch) i'm not a >> big postgres user, i don't know what the differences between the two >> are, i didn't know why they had done so originally, so until i figured >> it out i thought i should accept they had a reason and follow by example. > > http://www.initd.org/tracker/psycopg/wiki/PsycopgTwo > > psycopg is a PostgreSQL database adapter for the Python programming > language. Its main advantages are that it supports the full Python > DBAPI 2.0 and it is thread safe at level 2. It was designed for > heavily multi-threaded applications that create and destroy lots of > cursors and make a conspicuous number of concurrent INSERTs or > UPDATEs. The psycopg distribution includes ZPsycopgDA, a Zope Database > Adapter. > > This page is about psycopg 2, an almost complete rewrite of the > psycopg 1.1.x branch. Psycopg 2 features complete libpq v3 protocol > COPY TO/COPY FROM and full object adaptation for all basic Python 2.3 > types: strings (including unicode), ints, longs, floats, buffers > (binary objects), booleans, mx.DateTime and builtin datetime types. It > also supports unicode queries and Python lists mapped to PostgreSQL > arrays. > > Psycopg 2 somewhat breaks the compatibility with psycopg 1. See the > Psycopg 1 to Psycopg 2 Migration for more information: > http://www.initd.org/tracker/psycopg/wiki/Migration > -- looking to buy or sell anything? try: http://allurstuff.com it's a classified ads service that shows on a map where the seller is (think craigslist + google maps) plus it's 100% free :) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
./manage.py sqlreset --apply
hey, what are thoughts on a new option to sqlreset (and others, as appropriate) to apply the change to the db? something like this: ./manage.py sqlreset --apply just as a time saver. i'd write the patch if no one's opposed. derek -- looking to buy or sell anything? try: http://allurstuff.com it's a classified ads service that shows on a map where the seller is (think craigslist + google maps) plus it's 100% free :) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: ./manage.py sqlreset --apply
i didn't notice the reset command when i asked the original question. i always pipe my sql. which is what i'm most comfortable with, but i understand other people like the convenience of the management commands running the sql directly. no interactivity beyond "here's what i want to run. is this ok by you?" but it's optional via the "--noinput" flag. (like syncdb) also has the added benefit of automating the post-managed-migration fingerprint check. (and in the future, will eliminate the copy-paste step if you want to put all your SQL into managed-migrations.) Russell Keith-Magee wrote: > On 10/19/07, Derek Anderson <[EMAIL PROTECTED]> wrote: >> Russell Keith-Magee wrote: >> as was somewhat the intent of the question. :) i was trying to figure >> out of there's an existing style i should model my evolve commands after. > > That wasn't the question you asked. You asked if sqlreset should have > a new option that would perform the role currently performed by the > reset command. To this, the answer is no - firstly because it would > duplicate the 'reset' command, and secondly because the reset command > needs more fundamental fixes first. > >> right now i have them split: "./manage.py sqlevolve" outputs sql. >> "./manage.py evolvedb" is interactive, with a view/confirm/apply model. >> >> i'm open to input however. > > The implementation of the 'outer' reset/flush just apply the SQL > generated by the 'inner' sqlreset/sqlflush. This works because there > isn't any user input required to generate the SQL - the only > interactivity is on the outer commands in the form of a safety to > prevent accidental data loss. > > If you are able to generate the SQL without interactivity, then the > reset/sqlreset approach may be a useful approach to emulate. > > However, if your SQL generation process requires interactivity, you're > in fairly new territory. The closest match to this approach would be > syncdb, in the way that syncdb is interactive regarding creating the > admin user. However, you will note that there is no inner 'sqlsyncdb'. > > Yours, > Russ Magee %-) > > > > -- looking to buy or sell anything? try: http://allurstuff.com it's a classified ads service that shows on a map where the seller is (think craigslist + google maps) plus it's 100% free :) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: ./manage.py sqlreset --apply
Russell Keith-Magee wrote: > (sql)reset is broken as currently implemented, and as far as I can > make out, it can't really be fixed without some major changes. > However, I'm not convinced it is worth spending the effort to fix the > problems that exist. The problem (sql)reset is intended to solve would > be better served by a good schema evolution framework. In the absence > of an officially adopted schema-evolution approach, there is still a > place for (sql)reset, even in a limited/partially broken state. as was somewhat the intent of the question. :) i was trying to figure out of there's an existing style i should model my evolve commands after. right now i have them split: "./manage.py sqlevolve" outputs sql. "./manage.py evolvedb" is interactive, with a view/confirm/apply model. i'm open to input however. -- looking to buy or sell anything? try: http://allurstuff.com it's a classified ads service that shows on a map where the seller is (think craigslist + google maps) plus it's 100% free :) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
defaults in sql (postgres)
are column defaults not supposed to be set by django under postgres? see: $ ./manage.py sql auth BEGIN; [...] CREATE TABLE "auth_user" ( "id" serial NOT NULL PRIMARY KEY, "username" varchar(30) NOT NULL UNIQUE, "first_name" varchar(30) NOT NULL, "last_name" varchar(30) NOT NULL, "email" varchar(75) NOT NULL, "password" varchar(128) NOT NULL, "is_staff" boolean NOT NULL, "is_active" boolean NOT NULL, "is_superuser" boolean NOT NULL, "last_login" timestamp with time zone NOT NULL, "date_joined" timestamp with time zone NOT NULL ); [...] COMMIT; where: is_staff = models.BooleanField(..., default=False, ...) is_active = models.BooleanField(..., default=True, ...) is_superuser = models.BooleanField(..., default=False, ...) they're also not set under my own models: class Poll(models.Model): question = models.CharField(maxlength=200, default='test') CREATE TABLE "case02_rename_field_poll" ( "question" varchar(200) NOT NULL, ); thanks, derek -- looking to buy or sell anything? try: http://allurstuff.com it's a classified ads service that shows on a map where the seller is (think craigslist + google maps) plus it's 100% free :) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: defaults in sql (postgres)
i had thought they werebut it appears i am wrong. may i ask the core devs: why? derek Yuri Baburov wrote: > 2007/10/24, Derek Anderson <[EMAIL PROTECTED]>: >> are column defaults not supposed to be set by django under postgres? > yes, correct > defaults are never set by django for postgres > are they set for other databases? > don't know why it's so. > >> see: >> >> $ ./manage.py sql auth >> BEGIN; >> [...] >> CREATE TABLE "auth_user" ( >> "id" serial NOT NULL PRIMARY KEY, >> "username" varchar(30) NOT NULL UNIQUE, >> "first_name" varchar(30) NOT NULL, >> "last_name" varchar(30) NOT NULL, >> "email" varchar(75) NOT NULL, >> "password" varchar(128) NOT NULL, >> "is_staff" boolean NOT NULL, >> "is_active" boolean NOT NULL, >> "is_superuser" boolean NOT NULL, >> "last_login" timestamp with time zone NOT NULL, >> "date_joined" timestamp with time zone NOT NULL >> ); >> [...] >> COMMIT; >> >> where: >> is_staff = models.BooleanField(..., default=False, ...) >> is_active = models.BooleanField(..., default=True, ...) >> is_superuser = models.BooleanField(..., default=False, ...) >> >> they're also not set under my own models: >> >> class Poll(models.Model): >> question = models.CharField(maxlength=200, default='test') >> >> CREATE TABLE "case02_rename_field_poll" ( >> "question" varchar(200) NOT NULL, >> ); >> >> thanks, >> derek > -- looking to buy or sell anything? try: http://allurstuff.com it's a classified ads service that shows on a map where the seller is (think craigslist + google maps) plus it's 100% free :) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: defaults in sql (postgres)
ah, ok. danke. Jacob Kaplan-Moss wrote: > On 10/23/07, Derek Anderson <[EMAIL PROTECTED]> wrote: >> are column defaults not supposed to be set by django under postgres? > > Django doesn't set defaults in the database at all. > > The basic reason is that too many defaults simply couldn't be > represented in a database -- especially those oriented around > callables. It's not worth doing if it only works some of the time. > > IIRC there was a pretty big argument about this some time back; you > may want to search the archives for that. > > Jacob > > > > -- looking to buy or sell anything? try: http://allurstuff.com it's a classified ads service that shows on a map where the seller is (think craigslist + google maps) plus it's 100% free :) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Django 1.0 features -- the definitive list
what, no schema evolution? =p (ducks for cover behind fire-retardant suit ;) Adrian Holovaty wrote: > (I've been saving this e-mail since the last sprint. Given that we're > sprinting again this weekend, I figured it was about time to get this > conversation started.) > > Let's get a definitive list of features we want in Django 1.0, and > let's release it. > > I'll start with a proposed list, which no doubt has omissions. But > first, here's a proposal for how to handle this: > > 1. We decide on the list of features/changes. > > 1.5. Once the list is final, we do NOT add to it except in case of > an Act of God. > > 2. We set a deadline. > > 3. We work -- *primarily* on the list of features/changes, but > allowing some time for squeezing in any other small fixes that we have > time for. > > 4. Any feature that's not implemented by the deadline does NOT > make it into version 1.0. But fear not, because version 1.0 is not the > end of Django -- it's only the beginning! > > 5. Release, rejoice. > > The first order of business is deciding the features/changes. I'll > kick it off with the list I've been keeping. > > This is just my own list, of course, and I'm sure other > committers/contributors have other stuff. Please contribute! Just one > important thing to note: This list is for Big Stuff only. Do not > suggest features that would be able to be added/changed *after* the > 1.0 release in a backwards-compatible way. The goal here is to have a > simple, concrete list of major things that need to be done to the > framework -- not a list of 4,000 tiny things. > > Without further ado, here's my list: > > * newforms-admin > * queryset-refactor > * django.newforms becomes django.forms > * Model-level validation > * Change django.templatetags not to use __path__ hacking > * #3591 -- Make INSTALLED_APPS an instance, and each app an instance > * #5361 -- File storage refactoring > * #285 -- WSGI SCRIPT_NAME and PATH_INFO stuff > > What am I forgetting? > > And, finally, a bit of a controversial statement, but... > > I think we ought to call the release 2.0. > > Jacob and I have talked in the past about how we should've called > magic-removal version 1.0. (This ended up being 0.95.) For all intents > and purposes, it *was* 1.0 in spirit -- it was the first major > refactoring of several parts of the framework, and it was a point for > me personally where I started to feel like an acceptable number of the > legacy warts from pre-open-sourcing had been removed. > > So, that's one reason: philosophically, conceptually, in our minds, in > our hearts, we're really dealing with a 2.0 product. We know Django > rocks (and is rock-solid), and we should give it an appropriate > number. > > My second reason for choosing 2.0 is, shall we say, less wholesome. > After having endured a 2.5+ year deluge of "When is 1.0 coming out?" > blog entries, comments, e-mails and in-person confrontations from > people all around the world, I would find it incredibly satisfying, in > a devilish way, to release this thing and slap a "2.0" on it. It would > underscore the project's stability while at the same time > demonstrating that version numbers are completely arbitrary. > > It'd be like Google's IPO price, which was set to the mathematical > constant "e" -- a "we're not playing by your rules" message to Wall > Street. > > Something to ponder! > > Adrian > -- looking to buy or sell anything? try: http://allurstuff.com it's a classified ads service that shows on a map where the seller is (think craigslist + google maps) plus it's 100% free :) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Django 1.0 features -- the definitive list
twas a joke. i don't think any of the authors of any of the evolution mechanisms believe their implementations ready to be included into v1.0. (my own included :) Adrian Holovaty wrote: > On Nov 30, 2007 11:56 AM, Derek Anderson <[EMAIL PROTECTED]> wrote: >> what, no schema evolution? =p > > Schema evolution falls squarely in the category of "inessential for > this version, but can always be added in a subsequent incremental > version without breaking backwards compatibility." > > Adrian > -- looking to buy or sell anything? try: http://allurstuff.com it's a classified ads service that shows on a map where the seller is (think craigslist + google maps) plus it's 100% free :) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: filtering a QuerySet after a slice
even if it was possible, i don't think it's what you would really want. logically, this would return an indefinite number of records, somewhere between 0 and 10, depending on what's in your table, not the first 10 records of people who's name starts w/ an 'a'. why are you not doing this? UTPerson.objects.filter(name__startswith='a')[:10] Gary Wilson wrote: > Is there any reason why the following is disallowed: > UTPerson.objects.all()[:10].filter(name__startswith='a') > Traceback (most recent call last): > ... > AssertionError: Cannot filter a query once a slice has been taken. > > I would think that it should be possible as long as the QuerySet hasn't been > evaluated yet. > > Gary > > > > -- looking to buy or sell anything? try: http://allurstuff.com it's a classified ads service that shows on a map where the seller is (think craigslist + google maps) plus it's 100% free :) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Easier URL patterns
i *would* like to champion this, as django's regex -> function mapping is its best single feature, IMO. to reuse a well-known axiom: "those who don't know regular expressions are doomed to reinvent them... poorly" derek Rob Hudson wrote: > Just a note: I'm not going to champion this one since I do know > regular expressions. I simply read elsewhere that it is a common > stumbling block for newbies and thought I'd bring it up. > > Actually, the regex newbies I know usually just need a few examples to > get them off the ground. E.g. use "[-\w]+" for slugs, use "\d+" for > digits or id numbers, etc. > > Adrian makes a good point about not wanting multiple ways to specify > URLs. That does seem a bit ugly. > > Thanks for the discussion. :) > > -Rob > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
