Proposal: Database Constraints

[schinckel]

Some years ago, I discussed adding database-level check constraints into 
django: https://groups.google.com/forum/#!topic/django-developers/OJN5kpcseAg

There is also an issue: https://code.djangoproject.com/ticket/11964

I'm thinking about revisiting this, and wanted to get some discussion going 
about if this is a viable thing to do, and what it might look like.


Django already uses check constraints with PositiveIntegerField and 
friends, and at first blush I thought we might be able to co-opt some of 
that (indeed, I've got an internal monkey-patch that does, with some level 
of success). However, other than the fact there is already a 
'sql_create_check' string, the actual python code that creates the check 
constraint probably isn't that usable in this context. Also, I like the 
idea of having more complex constraints (postgres has EXCLUDE constraints, 
but I don't know if there is an equivalent for other backends).


The approach that I am thinking of could see a syntax similar to:


class MyModel(models.Model):
start = models.DateField()
finish = models.DateField(constraints=[('check', '> start')])
user = models.ForeignKey('auth.User')

This maps directly to creating a check constraint on the table:

 ALTER TABLE "myapp_mymodel" ADD CONSTRAINT CHECK (finish > start)

And, on the same model, a more complex constraint could look like:


class Meta:
constraints = [
('exclude', [
('overlaps', ['start','finish']),
('equal', 'user')
])
   ]

I'm still unsure of the best way to describe this: it's supposed to mean:

   ALTER TABLE "myapp_mymodel" ADD EXCLUDE USING gist (daterange(start, 
finish) WITH &&, user_id WITH =)

(but the python syntax is obviously immaterial at this stage).


Obviously, we can't just rely on the database to do the validation for us: 
it will just raise DatabaseErrors when something fails to validate anyway, 
so we would want to handle this stuff in django's validation framework.

One possibility, at least with the field-based check constraint, would be 
to automatically add a field validator in the case of a CHECK constraint, 
however in the case of an EXCLUDE constraint, we can't really validate 
_without_ hitting the database. Does this mean we should treat EXCLUDE-type 
constraints as something that is beyond the scope of Django?



With the new migrations framework, it's actually trivial to write a 
migration to add this type of constraint (or the check constraint), and a 
pre_save signal handler in conjunction with that would get 90% of the way, 
but it's still going to be open to a race condition anyway - the only way 
to actually do that is to try to save the object and see what the database 
says.


So, I guess I'm asking: is it worth pursuing this further, either in part 
or full?

Matt.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/250ed272-198a-478f-b2ce-920afcf533a2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [GSOC] Introduction and task proposal

[Daniel Pyrathon]

Hi Josh,

Sorry for getting back now. I have just finished my exam session, so I will 
be trying out the djangocore-box VM.

Regards,
Daniel Pyrathon

On Thursday, March 27, 2014 5:45:36 AM UTC+1, Josh Smeaton wrote:
>
> If you haven't already got all your databases installed on your 
> development machine, I *highly* recommend checking out 
> https://github.com/jphalip/djangocore-box. It's a vagrant VM environment 
> that already has MySQL, PostgreSQL, and SQLite installed (plus most GIS 
> installations), along with various versions of python. Unfortunately it 
> doesn't have Oracle - but being able to run tests on 3/4 databases is 
> especially handy.
>
> Josh
>
> On Thursday, 27 March 2014 11:14:35 UTC+11, Russell Keith-Magee wrote:
>>
>>
>> Hi Daniel,
>>
>> Nope - other than "cross your fingers" that your proposal is accepted :-)
>>
>> But seriously…
>>
>> If you haven't already, I'd suggest reading Django's contribution docs, 
>> and getting your development environment set up. Make sure you can run 
>> Django's test suite. If you're particularly enthused, try your hand at 
>> working on a patch for an open ticket so you can get used to the toolchain 
>> and the review process. That way you'll be up to speed with Django's 
>> development process, so when the GSoC period starts, you'll be comfortable 
>> with our basic process, so you'll just have to deal with the actual GSoC 
>> work.
>>
>> Yours,
>> Russ Magee %-)
>>
>> On Wed, Mar 26, 2014 at 7:22 PM, Daniel Pyrathon wrote:
>>
>>> Hi all,
>>>
>>> It's been a while since I submitted my GSOC proposal. Although I am 
>>> currently under exams, is there anything you would recommend me to do at 
>>> this point (other than hope that my proposal is successful)?
>>>
>>> Thanks,
>>> Daniel Pyrathon
>>>
>>>
>>> On Thursday, March 20, 2014 6:05:40 AM UTC, Russell Keith-Magee wrote:
>>>

 For the benefit of those reading along at home; I gave some revised 
 notes and had a conversation with Daniel on IRC a couple of hours ago.

 Yours,
 Russ Magee %-)


 On Thu, Mar 20, 2014 at 4:10 AM, Daniel Pyrathon wrote:

> Hi!
>
> Thanks for all the comments yesterday. They really helped me make the 
> proposal stronger.
> I have changed to proposal to reflect the changes. Would anyone like 
> to have a look and, possibly, comment more?
>
> https://docs.google.com/document/d/1yp2_skqkxyrc0egdRv6ofnRGCI9nmvxDFB
> kCXgy0Jwo/edit#heading=h.wyxx4xxijubt
>
> Thanks,
> Dan
>
> On Wednesday, March 19, 2014 7:32:10 AM UTC, Aymeric Augustin wrote:
>
>> On 19 mars 2014, at 06:27, Zach Borboa  wrote: 
>>
>> > Curious, how do you get REPL shell access to the server with 
>> DEBUG=True with a vanilla Django deployment? 
>>
>> That part of the discussion was about adding the werkzeug interactive 
>> debugger to Django’s default error page. 
>>
>> -- 
>> Aymeric. 
>>
>>
>>
>>
>>  -- 
> You received this message because you are subscribed to the Google 
> Groups "Django developers" group.
> To unsubscribe from this group and stop receiving emails from it, send 
> an email to django-develop...@googlegroups.com.
> To post to this group, send email to django-d...@googlegroups.com.
>
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/16264cb7-3f44-434d-865a-
> 8c33baa4d921%40googlegroups.com
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

  -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Django developers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to django-develop...@googlegroups.com.
>>> To post to this group, send email to django-d...@googlegroups.com.
>>> Visit this group at http://groups.google.com/group/django-developers.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/django-developers/ad5d6016-2c6a-45b4-91c8-0e4500befc74%40googlegroups.com
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this disc

ORM and admin

[Pradip Caulagi]

It is rather late in the day to bring this up but ...

should we remove ORM and admin from Django core?

Thanks,
Pradip P Caulagi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAJ-amneVaN4rOBD19%3DngL5Xf59dBw6xbQkyY6wkTGt86GOwRog%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: ORM and admin

[Rafał Pitoń]

April fools eh?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/cb5d1f89-c452-40ec-8ccb-f62d2ebb73c9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Proposal: Database Constraints

[Andrew Godwin]

Hmm, I'm not sure about this. On the one hand, during the migrations work I
refactored check constraints for the existing stuff (e.g.
PostitiveIntegerField) out of the type string and into a separate check
constraints string, so some of the work is already there, especially around
making sure they're present on field alteration.

On the other hand, check constraints vary wildly from database to database
- MySQL doesn't have them, SQLite has a basic version in later versions
only and parses but ignores them in earlier versions, PostgreSQL has full
support and Oracle has them too but with a different set of functions to
PostgreSQL, so it's unlikely anything other than simple comparison would be
portable.

For that reason, I wouldn't be the fan of the exploded syntax you've given;
I think I'd rather just be able to supply a list of strings for my
constraints that are SQL themselves; there's no real benefit to abstraction
here, and removing it doubtless removes a lot of code and bugs. Otherwise,
I think I'd be +0.

Andrew


On Tue, Apr 1, 2014 at 4:07 AM, schinckel  wrote:

> Some years ago, I discussed adding database-level check constraints into
> django:
> https://groups.google.com/forum/#!topic/django-developers/OJN5kpcseAg
>
> There is also an issue: https://code.djangoproject.com/ticket/11964
>
> I'm thinking about revisiting this, and wanted to get some discussion
> going about if this is a viable thing to do, and what it might look like.
>
>
> Django already uses check constraints with PositiveIntegerField and
> friends, and at first blush I thought we might be able to co-opt some of
> that (indeed, I've got an internal monkey-patch that does, with some level
> of success). However, other than the fact there is already a
> 'sql_create_check' string, the actual python code that creates the check
> constraint probably isn't that usable in this context. Also, I like the
> idea of having more complex constraints (postgres has EXCLUDE constraints,
> but I don't know if there is an equivalent for other backends).
>
>
> The approach that I am thinking of could see a syntax similar to:
>
>
> class MyModel(models.Model):
> start = models.DateField()
> finish = models.DateField(constraints=[('check', '> start')])
> user = models.ForeignKey('auth.User')
>
> This maps directly to creating a check constraint on the table:
>
>  ALTER TABLE "myapp_mymodel" ADD CONSTRAINT CHECK (finish > start)
>
> And, on the same model, a more complex constraint could look like:
>
>
> class Meta:
> constraints = [
> ('exclude', [
> ('overlaps', ['start','finish']),
> ('equal', 'user')
> ])
>]
>
> I'm still unsure of the best way to describe this: it's supposed to mean:
>
>ALTER TABLE "myapp_mymodel" ADD EXCLUDE USING gist
> (daterange(start, finish) WITH &&, user_id WITH =)
>
> (but the python syntax is obviously immaterial at this stage).
>
>
> Obviously, we can't just rely on the database to do the validation for us:
> it will just raise DatabaseErrors when something fails to validate anyway,
> so we would want to handle this stuff in django's validation framework.
>
> One possibility, at least with the field-based check constraint, would be
> to automatically add a field validator in the case of a CHECK constraint,
> however in the case of an EXCLUDE constraint, we can't really validate
> _without_ hitting the database. Does this mean we should treat EXCLUDE-type
> constraints as something that is beyond the scope of Django?
>
>
>
> With the new migrations framework, it's actually trivial to write a
> migration to add this type of constraint (or the check constraint), and a
> pre_save signal handler in conjunction with that would get 90% of the way,
> but it's still going to be open to a race condition anyway - the only way
> to actually do that is to try to save the object and see what the database
> says.
>
>
> So, I guess I'm asking: is it worth pursuing this further, either in part
> or full?
>
> Matt.
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/250ed272-198a-478f-b2ce-920afcf533a2%40googlegroups.com
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group a

Deprecating Widget Media class

[Tim Graham]

There's been some discussion in ticket 
#22298about deprecating the widget 
Media class. If this would make you 
particularly happy or sad, please leave your feedback on this thread or on 
the ticket.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5b3d6d91-8508-473d-8d08-3465f33a54f8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Deprecating Widget Media class

[Mark Lavin]

In my experience using the Media class is really only helpful in one place: 
the admin. If you have a third-party widget which requires CSS or JS is a 
pain to include it without the use of Media. Users want to just include the 
widget in the form class and have it "just work". A majority of the 
issues/questions I deal with on django-selectable are about how to get it 
working in the admin. Telling people to override and include it in 
the admin/base_site.html only seems adds confusion. It is documented how to 
override the admin templates but it would seem no one actually reads or 
understands them. In some cases they are trying to use apps which are 
already overriding it (like django-admin-tools). Outside the admin the 
Media class isn't nearly as helpful. It's easy enough to document adding 
the CSS/JS references in your base.html and users seem to understand that.

The biggest shortcoming of the Media class is summarized 
in https://code.djangoproject.com/ticket/12265. The order in which it is 
added isn't helpful nor well documented. I wouldn't complain too loudly if 
it was removed. It would be nice if it was replaced with another hook 
(possibly only in the admin) to improve the story for a third-party widget 
which requires some CSS/JS.

Best,

Mark

On Tuesday, April 1, 2014 2:32:15 PM UTC-4, Tim Graham wrote:
>
> There's been some discussion in ticket 
> #22298about deprecating the 
> widget Media class. If this would make you 
> particularly happy or sad, please leave your feedback on this thread or on 
> the ticket.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/76e70be1-35fa-4a28-ae05-0529bce15685%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Proposal: Database Constraints

[Simon Blanchard]

Hi

Just FYI: back in 2007 GSOC there was a project to add constraints. The
syntax was as follows:

class Manufacturer(models.Model):
mfg_name = models.CharField(maxlength=50)
car_sale_start = models.DateField()
car_sale_end = models.DateField()
quantity_sold = models.IntegerField()
car_price = models.IntegerField()

class Meta:
constraints = (
("check_name", Check(mfg_name__like = 'Merc*')),
("check_date", Check(car_sale_start__between = [date(2007,1,1),
date(2008,1,1)])),
("check_end_date", Check(car_sale_end__gte = 'car_sale_start')),
("check_quantity", Check(quantity_sold__gte = 0)),
("check_price", Check(car_price__between = [0, 1])),
  )


So a list of constraint name and a Check() pairs. I think the idea was that
Check() could be ANDd or ORd together a bit like the Q() object.

It worked with Postgres AFAIR.

Simon



On Tue, Apr 1, 2014 at 7:07 PM, schinckel  wrote:

> Some years ago, I discussed adding database-level check constraints into
> django:
> https://groups.google.com/forum/#!topic/django-developers/OJN5kpcseAg
>
> There is also an issue: https://code.djangoproject.com/ticket/11964
>
> I'm thinking about revisiting this, and wanted to get some discussion
> going about if this is a viable thing to do, and what it might look like.
>
>
> Django already uses check constraints with PositiveIntegerField and
> friends, and at first blush I thought we might be able to co-opt some of
> that (indeed, I've got an internal monkey-patch that does, with some level
> of success). However, other than the fact there is already a
> 'sql_create_check' string, the actual python code that creates the check
> constraint probably isn't that usable in this context. Also, I like the
> idea of having more complex constraints (postgres has EXCLUDE constraints,
> but I don't know if there is an equivalent for other backends).
>
>
> The approach that I am thinking of could see a syntax similar to:
>
>
> class MyModel(models.Model):
> start = models.DateField()
> finish = models.DateField(constraints=[('check', '> start')])
> user = models.ForeignKey('auth.User')
>
> This maps directly to creating a check constraint on the table:
>
>  ALTER TABLE "myapp_mymodel" ADD CONSTRAINT CHECK (finish > start)
>
> And, on the same model, a more complex constraint could look like:
>
>
> class Meta:
> constraints = [
> ('exclude', [
> ('overlaps', ['start','finish']),
> ('equal', 'user')
> ])
>]
>
> I'm still unsure of the best way to describe this: it's supposed to mean:
>
>ALTER TABLE "myapp_mymodel" ADD EXCLUDE USING gist
> (daterange(start, finish) WITH &&, user_id WITH =)
>
> (but the python syntax is obviously immaterial at this stage).
>
>
> Obviously, we can't just rely on the database to do the validation for us:
> it will just raise DatabaseErrors when something fails to validate anyway,
> so we would want to handle this stuff in django's validation framework.
>
> One possibility, at least with the field-based check constraint, would be
> to automatically add a field validator in the case of a CHECK constraint,
> however in the case of an EXCLUDE constraint, we can't really validate
> _without_ hitting the database. Does this mean we should treat EXCLUDE-type
> constraints as something that is beyond the scope of Django?
>
>
>
> With the new migrations framework, it's actually trivial to write a
> migration to add this type of constraint (or the check constraint), and a
> pre_save signal handler in conjunction with that would get 90% of the way,
> but it's still going to be open to a race condition anyway - the only way
> to actually do that is to try to save the object and see what the database
> says.
>
>
> So, I guess I'm asking: is it worth pursuing this further, either in part
> or full?
>
> Matt.
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/250ed272-198a-478f-b2ce-920afcf533a2%40googlegroups.com
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group a

Re: Proposal: Database Constraints

[schinckel]

On Wednesday, April 2, 2014 3:35:24 AM UTC+10:30, Andrew Godwin wrote:
>
> Hmm, I'm not sure about this. On the one hand, during the migrations work 
> I refactored check constraints for the existing stuff (e.g. 
> PostitiveIntegerField) out of the type string and into a separate check 
> constraints string, so some of the work is already there, especially around 
> making sure they're present on field alteration.
>
> On the other hand, check constraints vary wildly from database to database 
> - MySQL doesn't have them, SQLite has a basic version in later versions 
> only and parses but ignores them in earlier versions, PostgreSQL has full 
> support and Oracle has them too but with a different set of functions to 
> PostgreSQL, so it's unlikely anything other than simple comparison would be 
> portable.
>

Thanks Andrew. I haven't touched other databases for a long time, so it's 
good to get that feedback.
 

> For that reason, I wouldn't be the fan of the exploded syntax you've 
> given; I think I'd rather just be able to supply a list of strings for my 
> constraints that are SQL themselves; there's no real benefit to abstraction 
> here, and removing it doubtless removes a lot of code and bugs. Otherwise, 
> I think I'd be +0.
>

Pushing on from there: it's not much work from doing "constraints = ('check 
start < finish',)" to just writing a RunSQL migration operation. The 
exploded syntax was more me working things through late at night anyway. 
And it seems like there's not really much else to do other than raw SQL for 
exclude constraints.

So, it's actually possible to do _most_ of what I want already - just using 
a migration. The per-field check constraint could/should also be 
implemented as a validator. Exclude constraints are another kettle of fish: 
as I mentioned before, it's not actually possible to validate these 
constraints without hitting the database, and even then that's not alway 
enough. You still need to handle IntegrityErrors if an update between when 
you checked and when you actually try to save causes a conflict.

Matt.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/378962de-7ae6-496e-8a85-88c964d43e78%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.