Re: How to disable system check framework?

[Tim Graham]

I created a ticket and pull request for the silencing of error/critical 
messages:
https://code.djangoproject.com/ticket/25318#ticket
https://github.com/django/django/pull/5197

On Tuesday, August 25, 2015 at 11:03:18 AM UTC-4, Carl Meyer wrote:
>
> On 08/25/2015 08:57 AM, Marcin Nowak wrote: 
> > Well, I'm not sure now, because Tim wrote that it was a design decision 
> > and it is well documented. 
>
> I still think we should change that design. It is simply wrong to have a 
> setting called `SILENCED_SYSTEM_CHECKS` that fails to actually _silence_ 
> the check IDs listed. 
>
> > Also please note that silencing specific errors is not the best 
> solution. 
> > In the first case E116 may be untrue and may be silenced, but for other 
> > models this error may be true and shouldn't be silenced at all. 
> > Silencing all E116 will give us messy information about state. 
>
> That's why I think in this case the best solution is to actually fix the 
> error, not silence it; Django should be able to find a find a field even 
> if it's monkeypatched-in, as long as the monkeypatch happens in time. 
>
> Carl 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/4bd1edea-ca58-44b2-bf75-b05443f982ea%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Adding more __repr__() methods

[Tim Graham]

I'd like to ask for other opinions on this pull request which adds more 
__repr__() methods: https://github.com/django/django/pull/5059

Some concerns:

* Adding things that aren't part of the class (e.g. settings) seems 
questionable to me.

* There is a chance to leak sensitive data in __repr__ as done in #22990 
 (doesn't seems to be a 
concern with the changes here, but is something to be mindful of).


Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ec1b1249-b712-4b03-a436-dcffb06e9922%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to disable system check framework?

[Marcin Nowak]

On Thursday, August 27, 2015 at 3:09:06 PM UTC+2, Tim Graham wrote:
>
> I created a ticket and pull request for the silencing of error/critical 
> messages:
> https://code.djangoproject.com/ticket/25318#ticket
> https://github.com/django/django/pull/5197
>
>>
>>
Thanks, Tim!

Marcin.
 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/9ecf68e4-35a7-483b-983e-c5ef9f9d3533%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Keeping apps without migrations?

[Tim Graham]

It would be great if we could decide this by next Friday (September 4) or 
so, so that we have some time to do the necessary implementations and 
testing before alpha. (For example, running manage.py test without 
migrations currently gives "no such table ...". It would be nice to give a 
friendlier message if we require migrations for running tests 
(https://github.com/django/django/pull/5100 does this to some extent, but 
could be improved I think).

I know at least Markus said he would like to present an argument for 
keeping support for apps without migrations.

On Thursday, August 13, 2015 at 1:56:05 PM UTC-4, Tim Graham wrote:
>
> It's a one line change in the code to keep the status quo from 1.8. The 
> main drawback I see is continued support tickets about dependencies between 
> apps with and without migrations. I already removed the documentation 
> describing the problem and "won't fixed" tickets that suggested doing more 
> to detect and throw errors about these cases.
>
> I guess I'd like here why migrations could be a hindrance "for quick 
> prototyping and initial development." I'd argue that starting with 
> migrations from the start (manage.py makemigrations && manage.py migrate) 
> will save the need to learn about --fake-initial if you decide to add 
> migrations at some point later (and of course avoid dependency issues). Is 
> "manage.py makemigrations && manage.py migrate" more difficult that the 
> proposed ""manage.py migrate --allow-apps-without-migrations"? Am I missing 
> a motive behind the request?
>
> diff --git a/django/core/management/commands/migrate.py 
> b/django/core/management
> index 2f04824..dec86e1 100644
> --- a/django/core/management/commands/migrate.py
> +++ b/django/core/management/commands/migrate.py
> @@ -133,7 +133,7 @@ class Command(BaseCommand):
>  targets = executor.loader.graph.leaf_nodes()
>  
>  plan = executor.migration_plan(targets)
> -run_syncdb = options.get('run_syncdb') and 
> executor.loader.unmigrated_a
> +run_syncdb = bool(executor.loader.unmigrated_apps)
>  
>  # Print some useful info
>  if self.verbosity >= 1:
>
> On Thursday, August 6, 2015 at 9:50:10 PM UTC-4, Markus Holtermann wrote:
>>
>> Instead of rushing to a decision if and of how we continue to support 
>> "apps w/o migrations" I would argue that, since Django's test suite does 
>> infact create tables for apps w/o migrations, we could add a commandline 
>> flag (e.g. --run-syncdb or whatever) to the migrate command and warn the 
>> user if there are apps w/o migrations an the user doesn't supplied: 
>>
>> https://github.com/django/django/blob/master/django/core/management/commands/migrate.py#L136
>>  
>>
>> /Markus 
>>
>> On Thu, Aug 06, 2015 at 05:26:50PM -0700, Andrew Godwin wrote: 
>> >Do we know exactly what "support for apps without migrations" would 
>> consist 
>> >of at this point? I have that half-done code for replicating syncdb with 
>> >the autodetector stuck onto the migration executor, but it's not 
>> especially 
>> >speedy and would need some work to make it a sensible speed before it 
>> goes 
>> >forward. 
>> > 
>> >Alternately, we could modify the autodetector to have a "create only" 
>> mode 
>> >that's more efficient or just write a new generator that is quick 
>> because 
>> >it knows it's only making nonmigrated apps' migrations. 
>> > 
>> >Andrew 
>> > 
>> >On Thu, Aug 6, 2015 at 4:56 PM, Tim Graham  wrote: 
>> > 
>> >> With the alpha for 1.9 coming up in 6 weeks, we need to decide whether 
>> or 
>> >> not to continue support for apps without migrations (currently in 
>> master no 
>> >> tables are created for such apps (as the deprecation timeline says), 
>> but it 
>> >> might be appropriate to add a warning or error message for this case 
>> if 
>> >> things don't change). 
>> >> 
>> >> Is anyone sufficiently interested in continuing support for apps 
>> without 
>> >> migrations to commit to completing it by the alpha? If not, do you 
>> consider 
>> >> it a release blocker that I should spend time on? 
>> >> 
>> >> Related tickets: 
>> >> 
>> >> https://code.djangoproject.com/ticket/25144 - No way to create tables 
>> for 
>> >> apps without migrations 
>> >> https://code.djangoproject.com/ticket/24919 - Add an option not to 
>> run 
>> >> migrations when running tests 
>> >> https://code.djangoproject.com/ticket/24481 - Improve sqlmigrate to 
>> be 
>> >> more flexible and allow bypassing migrations on disk 
>> >> https://code.djangoproject.com/ticket/24901 - makemigrations should 
>> >> create empty migrations dir for any installed app without it 
>> >> https://code.djangoproject.com/ticket/24588 - Improve handling apps 
>> >> without migrations while running migrate command. 
>> >> 
>> >> On Sunday, January 18, 2015 at 5:11:53 PM UTC-5, Markus Holtermann 
>> wrote: 
>> >>> 
>> >>> Creating in-memory migrations for all apps that don't have migration 
>> >>> files seems to be an option to solve the de

Re: Django Admin New Look

[Tim Graham]

Carl pinged me in #django-dev to say that the technical board says it's ok 
to go ahead with dropping IE8 support in admin for 1.9.

Three +1s and two +0s. The +0s were due to feeling that:
a) IE isn't as dead as in some places as it is in the Valley
b) the advantages gained by dropping support are weak.

But both the +0s nonetheless felt that this was consistent with our general 
support practices, and thus was fine to do now.

On Friday, August 21, 2015 at 7:26:16 AM UTC-4, Hugo Osvaldo Barrera wrote:
>
>  
> On Fri, Aug 21, 2015, at 08:06, elky wrote:
>
> Does 'title' attribute help in terms of accessibility? 
>
>  
> `title` adds a tooltip, so no, not really.
>  
> --
> Hugo Osvaldo Barrera
>  
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/503cfd0b-5416-4491-9657-aa549a4b68f2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Django SQL templates

[Alexey Zankevich]

Hi all,

After reading future of QuerySet.extra() 
 
discussion, 
I was thinking about the current custom ways to tweak SQL queries in 
Django. Firstly, it was the current .extra method, which everyone likely 
wants to get rid of. Secondly, performing raw SQL queries from queryset 
manager or even directly from cursor.
It's really a great approach to perform raw SQL and still receive objects 
of the given model at the end, I think. However, the main disadvantage of 
the current implementation - it's required to type in real table/field 
names, which destroys model's abstraction level (when you never reference 
tables/columns by their real names).
For such cases it's great to have DSL to build SQL queries, but it's a big 
question how low- or high-level language should be. Anyway, it ended up 
with the idea to have SQL templates, similar to current Django HTML 
templates. I even implemented a sample as proof of the concept:

https://github.com/Nepherhotep/django-orm-sugar/blob/exec_sql_template/tests/sql_template_test_app/app/tests.py
https://github.com/Nepherhotep/django-orm-sugar/blob/exec_sql_template/tests/sql_template_test_app/app/templates/select_user_profile.sql

How it works - there is a template "select_user_profile.sql" in a standard 
templates folder:

{% autoescape off %}
SELECT * FROM {{ model }} WHERE {{ model.name }} = {{ search_term }}
{% endautoescape %}

I had to put autoescape tags to prevent quotes in model name converted into 
HTML entities.
A function below should be called somewhere in views:

raw_queryset = exec_sql_template(UserProfile, "select_user_profile.sql", 
{'search_term': 'Bob'})

This function automatically creates a variable "model" (which is actually a 
special model wrapper, which prints table or column names) and passes it 
into template context. To prevent SQL injection, passed params converted 
into %(variable_name)s representation, and escaped automatically by driver. 
Thus the solution above doesn't work with SQLite (as it doesn't support 
named variables).
Also I wanted to automatically provide aliases for all existing models in 
form . (not implemented yet):

SELECT *
FROM
{{ model }}
LEFT OUTER JOIN
{{ myapp.ModelName }} 
ON
{{ myapp.ModelName.profile }} = {{ model.pk }}

Almost everything can work out of the box using standard Django templates, 
except quotes converted into entities, unless they don't disabled 
explicitly. Also, not sure if all the databases support 'tablename'.'name' 
syntax to safely define table/column, otherwise wrapper classes should be 
engine dependent.

I don't suggest a concrete implementation, just wanted to get responses of 
the concept.

Regards,
Alexey

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/da716f91-e5ab-466b-8d3e-475cd3738c12%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django SQL templates

[Marcin Nowak]

On Thursday, August 27, 2015 at 9:43:40 PM UTC+2, Alexey Zankevich wrote:
>
> For such cases it's great to have DSL to build SQL queries, but it's a big 
> question how low- or high-level language should be.
>

Take a look at SQLAlchemy`s expressions API. The ORM is built top of it.
I think that Django` developers will continue extending expressions API 
(F(), Q(), Func(), and so on) to make query language more flexible. 

Regards,
Marcin

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/f782f5ef-2c83-4d60-b4a9-51a407f195f7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Keeping apps without migrations?

[Markus Holtermann]

The general benefit of *not* having migrations is the incredible better 
performance to setup the tables. Especially for larger projects this is 
something we should at least keep in mind 
(see https://code.djangoproject.com/ticket/24743). I spent a fair bit of 
time on the issue and have not yet found a satisfying solution.

Supporting apps without migrations would essentially continue to support 
the old `syncdb` behavior, though not as a standalone command, but as a 
flag for `manage.py migrate`. It takes the models from those apps and calls 
`schema_editor.create_model()`, taking the deferred SQL and running it 
after all models of apps w/o migrations have been created (current behavior 
[in tests] I think). This would continue to raise issues if models in apps 
w/o migrations refer to models in apps w/ migrations (missing constraints), 
but that is a limitation we had since 1.7 (silent failure; 1.8 fails loud).

While migrations should provide *the* way to maintain your database I see 
the points that have been made in other recent discussions. Although Django 
doesn't force you to use migrations there are some things (e.g. not create 
the `django_migrations` table) we probably should not do unless you 
actually use migrations.

We might want to consider looking at the `MIGRATION_MODULES` settings 
variable and allowing to set values to `None` in order to disable 
migrations for that app, but that's a different story.

/Markus

On Friday, August 28, 2015 at 4:26:19 AM UTC+10, Tim Graham wrote:
>
> It would be great if we could decide this by next Friday (September 4) or 
> so, so that we have some time to do the necessary implementations and 
> testing before alpha. (For example, running manage.py test without 
> migrations currently gives "no such table ...". It would be nice to give a 
> friendlier message if we require migrations for running tests 
> (https://github.com/django/django/pull/5100 does this to some extent, but 
> could be improved I think).
>
> I know at least Markus said he would like to present an argument for 
> keeping support for apps without migrations.
>
> On Thursday, August 13, 2015 at 1:56:05 PM UTC-4, Tim Graham wrote:
>>
>> It's a one line change in the code to keep the status quo from 1.8. The 
>> main drawback I see is continued support tickets about dependencies between 
>> apps with and without migrations. I already removed the documentation 
>> describing the problem and "won't fixed" tickets that suggested doing more 
>> to detect and throw errors about these cases.
>>
>> I guess I'd like here why migrations could be a hindrance "for quick 
>> prototyping and initial development." I'd argue that starting with 
>> migrations from the start (manage.py makemigrations && manage.py migrate) 
>> will save the need to learn about --fake-initial if you decide to add 
>> migrations at some point later (and of course avoid dependency issues). Is 
>> "manage.py makemigrations && manage.py migrate" more difficult that the 
>> proposed ""manage.py migrate --allow-apps-without-migrations"? Am I missing 
>> a motive behind the request?
>>
>> diff --git a/django/core/management/commands/migrate.py 
>> b/django/core/management
>> index 2f04824..dec86e1 100644
>> --- a/django/core/management/commands/migrate.py
>> +++ b/django/core/management/commands/migrate.py
>> @@ -133,7 +133,7 @@ class Command(BaseCommand):
>>  targets = executor.loader.graph.leaf_nodes()
>>  
>>  plan = executor.migration_plan(targets)
>> -run_syncdb = options.get('run_syncdb') and 
>> executor.loader.unmigrated_a
>> +run_syncdb = bool(executor.loader.unmigrated_apps)
>>  
>>  # Print some useful info
>>  if self.verbosity >= 1:
>>
>> On Thursday, August 6, 2015 at 9:50:10 PM UTC-4, Markus Holtermann wrote:
>>>
>>> Instead of rushing to a decision if and of how we continue to support 
>>> "apps w/o migrations" I would argue that, since Django's test suite does 
>>> infact create tables for apps w/o migrations, we could add a commandline 
>>> flag (e.g. --run-syncdb or whatever) to the migrate command and warn the 
>>> user if there are apps w/o migrations an the user doesn't supplied: 
>>>
>>> https://github.com/django/django/blob/master/django/core/management/commands/migrate.py#L136
>>>  
>>>
>>> /Markus 
>>>
>>> On Thu, Aug 06, 2015 at 05:26:50PM -0700, Andrew Godwin wrote: 
>>> >Do we know exactly what "support for apps without migrations" would 
>>> consist 
>>> >of at this point? I have that half-done code for replicating syncdb 
>>> with 
>>> >the autodetector stuck onto the migration executor, but it's not 
>>> especially 
>>> >speedy and would need some work to make it a sensible speed before it 
>>> goes 
>>> >forward. 
>>> > 
>>> >Alternately, we could modify the autodetector to have a "create only" 
>>> mode 
>>> >that's more efficient or just write a new generator that is quick 
>>> because 
>>> >it knows it's only making nonmigrated apps' migration

Re: Keeping apps without migrations?

[Andrew Godwin]

I think I agree with Markus that we do not yet have a good enough
performance story to make migrations mandatory. If you want to ditch them
and do your own database management, we should let you - which we could do
with some kind of MIGRATIONS_MODULES opt-out that basically sets everything
to pseudo-unmanaged, but I see no reason not to have the syncdb-like
solution keep working as it at least runs off of SchemaEditor now.

Andrew

On Thu, Aug 27, 2015 at 5:14 PM, Markus Holtermann  wrote:

> The general benefit of *not* having migrations is the incredible better
> performance to setup the tables. Especially for larger projects this is
> something we should at least keep in mind (see
> https://code.djangoproject.com/ticket/24743). I spent a fair bit of time
> on the issue and have not yet found a satisfying solution.
>
> Supporting apps without migrations would essentially continue to support
> the old `syncdb` behavior, though not as a standalone command, but as a
> flag for `manage.py migrate`. It takes the models from those apps and calls
> `schema_editor.create_model()`, taking the deferred SQL and running it
> after all models of apps w/o migrations have been created (current behavior
> [in tests] I think). This would continue to raise issues if models in apps
> w/o migrations refer to models in apps w/ migrations (missing constraints),
> but that is a limitation we had since 1.7 (silent failure; 1.8 fails loud).
>
> While migrations should provide *the* way to maintain your database I see
> the points that have been made in other recent discussions. Although Django
> doesn't force you to use migrations there are some things (e.g. not create
> the `django_migrations` table) we probably should not do unless you
> actually use migrations.
>
> We might want to consider looking at the `MIGRATION_MODULES` settings
> variable and allowing to set values to `None` in order to disable
> migrations for that app, but that's a different story.
>
> /Markus
>
>
> On Friday, August 28, 2015 at 4:26:19 AM UTC+10, Tim Graham wrote:
>>
>> It would be great if we could decide this by next Friday (September 4) or
>> so, so that we have some time to do the necessary implementations and
>> testing before alpha. (For example, running manage.py test without
>> migrations currently gives "no such table ...". It would be nice to give a
>> friendlier message if we require migrations for running tests
>> (https://github.com/django/django/pull/5100 does this to some extent,
>> but could be improved I think).
>>
>> I know at least Markus said he would like to present an argument for
>> keeping support for apps without migrations.
>>
>> On Thursday, August 13, 2015 at 1:56:05 PM UTC-4, Tim Graham wrote:
>>>
>>> It's a one line change in the code to keep the status quo from 1.8. The
>>> main drawback I see is continued support tickets about dependencies between
>>> apps with and without migrations. I already removed the documentation
>>> describing the problem and "won't fixed" tickets that suggested doing more
>>> to detect and throw errors about these cases.
>>>
>>> I guess I'd like here why migrations could be a hindrance "for quick
>>> prototyping and initial development." I'd argue that starting with
>>> migrations from the start (manage.py makemigrations && manage.py migrate)
>>> will save the need to learn about --fake-initial if you decide to add
>>> migrations at some point later (and of course avoid dependency issues). Is
>>> "manage.py makemigrations && manage.py migrate" more difficult that the
>>> proposed ""manage.py migrate --allow-apps-without-migrations"? Am I missing
>>> a motive behind the request?
>>>
>>> diff --git a/django/core/management/commands/migrate.py
>>> b/django/core/management
>>> index 2f04824..dec86e1 100644
>>> --- a/django/core/management/commands/migrate.py
>>> +++ b/django/core/management/commands/migrate.py
>>> @@ -133,7 +133,7 @@ class Command(BaseCommand):
>>>  targets = executor.loader.graph.leaf_nodes()
>>>
>>>  plan = executor.migration_plan(targets)
>>> -run_syncdb = options.get('run_syncdb') and
>>> executor.loader.unmigrated_a
>>> +run_syncdb = bool(executor.loader.unmigrated_apps)
>>>
>>>  # Print some useful info
>>>  if self.verbosity >= 1:
>>>
>>> On Thursday, August 6, 2015 at 9:50:10 PM UTC-4, Markus Holtermann wrote:

 Instead of rushing to a decision if and of how we continue to support
 "apps w/o migrations" I would argue that, since Django's test suite
 does
 infact create tables for apps w/o migrations, we could add a
 commandline
 flag (e.g. --run-syncdb or whatever) to the migrate command and warn
 the
 user if there are apps w/o migrations an the user doesn't supplied:

 https://github.com/django/django/blob/master/django/core/management/commands/migrate.py#L136

 /Markus

 On Thu, Aug 06, 2015 at 05:26:50PM -0700, Andrew Godwin wrote:
 >Do we know exactly wh

Re: Adding more __repr__() methods

[Josh Smeaton]

Generally, I'm a fan of adding repr methods. I don't think we should run 
around finding all the places they might be useful and adding them, but if 
someone is to put forward patches where they've noticed a benefit, we 
should encourage that.

I'm not sure I fully understand your concerns though. It seems to me that 
the repr methods in that PR only include things relating to the class being 
observed. Did I miss something?

Cheers

On Friday, 28 August 2015 01:16:02 UTC+10, Tim Graham wrote:
>
> I'd like to ask for other opinions on this pull request which adds more 
> __repr__() methods: https://github.com/django/django/pull/5059
>
> Some concerns:
>
> * Adding things that aren't part of the class (e.g. settings) seems 
> questionable to me.
>
> * There is a chance to leak sensitive data in __repr__ as done in #22990 
>  (doesn't seems to be a 
> concern with the changes here, but is something to be mindful of).
>
>
> Thanks!
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/1ca277b1-96ed-4c78-86b2-631f309eb5c7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django SQL templates

[Josh Smeaton]

Marcin is correct. The plan is to extend the concept of expressions so that 
you have more control over crafting your own queries. 

What we have:

- select expressions
- aggregation expressions
- case/when
- order by
- where expressions (lookups/transforms)
- column expressions (representing model fields)

What we need:

- more control over joins
- more control over subqueries
- maybe analytical structures? row_number (over ..)

I'd be very interested in seeing proposals that aim to improve user control 
over joins and subqueries specifically. Once we've locked those down, we 
can remove .extra and probably .raw.

Personally, the SQL template isn't something I like or would use. You might 
have some success with users that need highly tailored queries until we 
implement the missing constructs above, but I would not consider it 
appropriate for django core. We're not trying to rid ourselves of work 
arounds (extra() and raw()) just to introduce newer work arounds.

I would encourage you to develop this as a 3rd party library if it's 
something you'd find useful. I'm certain there will be others that'll find 
it useful.

Regards,

On Friday, 28 August 2015 06:20:28 UTC+10, Marcin Nowak wrote:
>
>
>
> On Thursday, August 27, 2015 at 9:43:40 PM UTC+2, Alexey Zankevich wrote:
>>
>> For such cases it's great to have DSL to build SQL queries, but it's a 
>> big question how low- or high-level language should be.
>>
>
> Take a look at SQLAlchemy`s expressions API. The ORM is built top of it.
> I think that Django` developers will continue extending expressions API 
> (F(), Q(), Func(), and so on) to make query language more flexible. 
>
> Regards,
> Marcin
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/9a285b63-0539-4a35-878a-a25794a1f56c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

CHECK Constraints and migrations

[Gavin Wahl]

I'm interested in writing a third-party app that allows users to
declaratively write check constraints and have them automatically be added
and removed by migrations. I'm envisioning being able to attach a list of Q
objects to your model, something like:


class Meta:
checks = [
Q(end_date__isnull=True) | Q(end_date__gt=F('start_date'))
]

Now that we have migrations, formalized Meta, and custom lookups, is it
possible for a third-party app to manage this seamlessly?

I'm having trouble figuring out where it could hook in to migrations to
detect changes to the check constraints, and automatically generate the
migrations with operations to update the constraints. What code I should be
reading to figure out how to do it?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CACPudh1gRB_Jov_bH-maDGr4EBRugFuV3Y%2BBzYM9w0yyki7zkg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: CHECK Constraints and migrations

[Andrew Godwin]

Hi Gavin,

Hooking into the migration autodetector is currently not really possible -
it's quite inextensible, which is something I'd love to get ironed out in
the long term but isn't really possible right now to modify it from
third-party code.

You can, however, supply custom migration Operation classes that people
could put into their migrations manually, or potentially have some other
way of outputting those migration files with operations in them (a lot of
the code to write them out is re-useable). Hopefully someone else has a
better idea!

Andrew

On Thu, Aug 27, 2015 at 6:27 PM, Gavin Wahl  wrote:

> I'm interested in writing a third-party app that allows users to
> declaratively write check constraints and have them automatically be added
> and removed by migrations. I'm envisioning being able to attach a list of Q
> objects to your model, something like:
>
>
> class Meta:
> checks = [
> Q(end_date__isnull=True) | Q(end_date__gt=F('start_date'))
> ]
>
> Now that we have migrations, formalized Meta, and custom lookups, is it
> possible for a third-party app to manage this seamlessly?
>
> I'm having trouble figuring out where it could hook in to migrations to
> detect changes to the check constraints, and automatically generate the
> migrations with operations to update the constraints. What code I should be
> reading to figure out how to do it?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CACPudh1gRB_Jov_bH-maDGr4EBRugFuV3Y%2BBzYM9w0yyki7zkg%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1urshtoQ%2BY%3D%3DvTUr9nXmmdOAoA3ExuPtNAxBLw3_wkXY-g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: CHECK Constraints and migrations

[schinckel]

That's the approach I've used: either have a migration operation that you 
can run (included in a 3rd party or local app), or a management command 
that generates a migration file with the relevant custom operations.

Matt.

On Friday, August 28, 2015 at 11:37:55 AM UTC+9:30, Andrew Godwin wrote:
>
> Hi Gavin,
>
> Hooking into the migration autodetector is currently not really possible - 
> it's quite inextensible, which is something I'd love to get ironed out in 
> the long term but isn't really possible right now to modify it from 
> third-party code.
>
> You can, however, supply custom migration Operation classes that people 
> could put into their migrations manually, or potentially have some other 
> way of outputting those migration files with operations in them (a lot of 
> the code to write them out is re-useable). Hopefully someone else has a 
> better idea!
>
> Andrew
>
> On Thu, Aug 27, 2015 at 6:27 PM, Gavin Wahl  > wrote:
>
>> I'm interested in writing a third-party app that allows users to 
>> declaratively write check constraints and have them automatically be added 
>> and removed by migrations. I'm envisioning being able to attach a list of Q 
>> objects to your model, something like:
>>
>>
>> class Meta:
>> checks = [
>> Q(end_date__isnull=True) | Q(end_date__gt=F('start_date'))
>> ]
>>
>> Now that we have migrations, formalized Meta, and custom lookups, is it 
>> possible for a third-party app to manage this seamlessly?
>>
>> I'm having trouble figuring out where it could hook in to migrations to 
>> detect changes to the check constraints, and automatically generate the 
>> migrations with operations to update the constraints. What code I should be 
>> reading to figure out how to do it?
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-develop...@googlegroups.com .
>> To post to this group, send email to django-d...@googlegroups.com 
>> .
>> Visit this group at http://groups.google.com/group/django-developers.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/CACPudh1gRB_Jov_bH-maDGr4EBRugFuV3Y%2BBzYM9w0yyki7zkg%40mail.gmail.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/dedacf17-865e-4900-832c-2135039437ba%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: CHECK Constraints and migrations

[Gavin Wahl]

That's too bad. I don't think it's really worth it unless the experience 
works seamlessly, because if I'm going to write a migration manually I 
might as well just use a RunSQL. I did have another slightly hacky idea 
that might work. Can custom Fields specify the operations used to create 
and drop them? It might then be possible to write a pseudo-field called 
'Check':

class FooModel(models.Model):
# ...
check_ends_after_start = Check(Q(end_date__isnull=True) | 
Q(end_date__gt=F('start_date')))

That instead of adding a column called ends_after_start, it would add a 
constraint.

Do you think it's possible for anyone besides you to work on making the 
migration framework more extensible? I'm interested in contributing in this 
area.


On Thursday, August 27, 2015 at 8:07:55 PM UTC-6, Andrew Godwin wrote:
>
> Hi Gavin,
>
> Hooking into the migration autodetector is currently not really possible - 
> it's quite inextensible, which is something I'd love to get ironed out in 
> the long term but isn't really possible right now to modify it from 
> third-party code.
>
> You can, however, supply custom migration Operation classes that people 
> could put into their migrations manually, or potentially have some other 
> way of outputting those migration files with operations in them (a lot of 
> the code to write them out is re-useable). Hopefully someone else has a 
> better idea!
>
> Andrew
>
> On Thu, Aug 27, 2015 at 6:27 PM, Gavin Wahl  > wrote:
>
>> I'm interested in writing a third-party app that allows users to 
>> declaratively write check constraints and have them automatically be added 
>> and removed by migrations. I'm envisioning being able to attach a list of Q 
>> objects to your model, something like:
>>
>>
>> class Meta:
>> checks = [
>> Q(end_date__isnull=True) | Q(end_date__gt=F('start_date'))
>> ]
>>
>> Now that we have migrations, formalized Meta, and custom lookups, is it 
>> possible for a third-party app to manage this seamlessly?
>>
>> I'm having trouble figuring out where it could hook in to migrations to 
>> detect changes to the check constraints, and automatically generate the 
>> migrations with operations to update the constraints. What code I should be 
>> reading to figure out how to do it?
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-develop...@googlegroups.com .
>> To post to this group, send email to django-d...@googlegroups.com 
>> .
>> Visit this group at http://groups.google.com/group/django-developers.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/CACPudh1gRB_Jov_bH-maDGr4EBRugFuV3Y%2BBzYM9w0yyki7zkg%40mail.gmail.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/d8db8897-93f6-43e8-aa13-563fafa6a451%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: CHECK Constraints and migrations

[Andrew Godwin]

No, fields can't really specify any of the SQL used to make/update/delete
them (partially as the SQL is heavily dependent on the database backend,
and Django traditionally has the fields as the database-agnostic bit and
the backends as the specific bit). All they can do is supply a db_type,
which has to fit in the space a type would go for ADD COLUMN, etc. (that's
how contrib.postgres has its fields work, for example).

I know that Marc Tamlyn was looking into this for some of his PostgreSQL
stuff, and other people have floated some ideas around letting fields have
more control over migration stuff too - they may be able to chime in when
we get to a sensible time in Europe on this.

I'm definitely far from the only person who can work on fixing up the
migration stuff; in fact, I haven't done much to it at all recently and
it's been all other people, so if you think you want to take a stab at it,
you're more than welcome, and I'm more than happy to review patches against
migrations.

Andrew

On Thu, Aug 27, 2015 at 9:25 PM, Gavin Wahl  wrote:

> That's too bad. I don't think it's really worth it unless the experience
> works seamlessly, because if I'm going to write a migration manually I
> might as well just use a RunSQL. I did have another slightly hacky idea
> that might work. Can custom Fields specify the operations used to create
> and drop them? It might then be possible to write a pseudo-field called
> 'Check':
>
> class FooModel(models.Model):
> # ...
> check_ends_after_start = Check(Q(end_date__isnull=True) |
> Q(end_date__gt=F('start_date')))
>
> That instead of adding a column called ends_after_start, it would add a
> constraint.
>
> Do you think it's possible for anyone besides you to work on making the
> migration framework more extensible? I'm interested in contributing in this
> area.
>
>
> On Thursday, August 27, 2015 at 8:07:55 PM UTC-6, Andrew Godwin wrote:
>>
>> Hi Gavin,
>>
>> Hooking into the migration autodetector is currently not really possible
>> - it's quite inextensible, which is something I'd love to get ironed out in
>> the long term but isn't really possible right now to modify it from
>> third-party code.
>>
>> You can, however, supply custom migration Operation classes that people
>> could put into their migrations manually, or potentially have some other
>> way of outputting those migration files with operations in them (a lot of
>> the code to write them out is re-useable). Hopefully someone else has a
>> better idea!
>>
>> Andrew
>>
>> On Thu, Aug 27, 2015 at 6:27 PM, Gavin Wahl  wrote:
>>
>>> I'm interested in writing a third-party app that allows users to
>>> declaratively write check constraints and have them automatically be added
>>> and removed by migrations. I'm envisioning being able to attach a list of Q
>>> objects to your model, something like:
>>>
>>>
>>> class Meta:
>>> checks = [
>>> Q(end_date__isnull=True) | Q(end_date__gt=F('start_date'))
>>> ]
>>>
>>> Now that we have migrations, formalized Meta, and custom lookups, is it
>>> possible for a third-party app to manage this seamlessly?
>>>
>>> I'm having trouble figuring out where it could hook in to migrations to
>>> detect changes to the check constraints, and automatically generate the
>>> migrations with operations to update the constraints. What code I should be
>>> reading to figure out how to do it?
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to django-develop...@googlegroups.com.
>>> To post to this group, send email to django-d...@googlegroups.com.
>>> Visit this group at http://groups.google.com/group/django-developers.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/django-developers/CACPudh1gRB_Jov_bH-maDGr4EBRugFuV3Y%2BBzYM9w0yyki7zkg%40mail.gmail.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/d8db8897-93f6-43e8-aa13-563fafa6a451%40googlegroups.com
> 
> .
>
> For more 

Re: Keeping apps without migrations?

[Anssi Kääriäinen]

Could we allow applications to have fake migration files? I'm thinking
of something like operations.AllModelsUpdated(). After this operation
is ran, Django will use the present models for that app. You could
place this as your last migration, and run direct SQL in previous
migrations (or use some completely different mechanism) . Direct SQL
without model state changes will be fast. The last migration would
supposedly be fast, as it only needs to construct up to date images of
the current models.

The problem is that I'm not at all sure how this will mix with
migrations from other applications, so this proposal is perhaps
technically impossible to do. But if this is feasible, then it should
be possible to mix migrations and manual database changes pretty much
seamlessly. This one would work much better if we had project level
migrations - implementing a migration operation that tells Django that
all of the project's models are up to date is almost trivial.

I believe a fake migration would work pretty well for the use case
where you want to use manual database control for your project, but
use 3rd party applications with migrations.

Another idea I've been thinking is that maybe we could have snapshots
for migrations. The reason I believe this would work is that building
the model state is expensive, but in many cases it is completely
redundant to build the model state from scratch.

The snapshots would essentially do the following operation: after
migrations X, Y and Z are applied the model state will be as presented
in this migration. The snapshots would only contain model state, not
any operations. When you run manage.py migrate, the app state builder
will be able to skip to latest snapshot that has all dependencies
already applied, and then continue building model state from there.
When performance begins to degrade, you can create a new snapshot. The
snapshots would probably have to be project level to work properly.
This would seriously speed up migrations, except for the case where
you run the migrations against fresh database.

 - Anssi


On Fri, Aug 28, 2015 at 3:18 AM, Andrew Godwin  wrote:
> I think I agree with Markus that we do not yet have a good enough
> performance story to make migrations mandatory. If you want to ditch them
> and do your own database management, we should let you - which we could do
> with some kind of MIGRATIONS_MODULES opt-out that basically sets everything
> to pseudo-unmanaged, but I see no reason not to have the syncdb-like
> solution keep working as it at least runs off of SchemaEditor now.
>
> Andrew
>
> On Thu, Aug 27, 2015 at 5:14 PM, Markus Holtermann
>  wrote:
>>
>> The general benefit of *not* having migrations is the incredible better
>> performance to setup the tables. Especially for larger projects this is
>> something we should at least keep in mind (see
>> https://code.djangoproject.com/ticket/24743). I spent a fair bit of time on
>> the issue and have not yet found a satisfying solution.
>>
>> Supporting apps without migrations would essentially continue to support
>> the old `syncdb` behavior, though not as a standalone command, but as a flag
>> for `manage.py migrate`. It takes the models from those apps and calls
>> `schema_editor.create_model()`, taking the deferred SQL and running it after
>> all models of apps w/o migrations have been created (current behavior [in
>> tests] I think). This would continue to raise issues if models in apps w/o
>> migrations refer to models in apps w/ migrations (missing constraints), but
>> that is a limitation we had since 1.7 (silent failure; 1.8 fails loud).
>>
>> While migrations should provide *the* way to maintain your database I see
>> the points that have been made in other recent discussions. Although Django
>> doesn't force you to use migrations there are some things (e.g. not create
>> the `django_migrations` table) we probably should not do unless you actually
>> use migrations.
>>
>> We might want to consider looking at the `MIGRATION_MODULES` settings
>> variable and allowing to set values to `None` in order to disable migrations
>> for that app, but that's a different story.
>>
>> /Markus
>>
>>
>> On Friday, August 28, 2015 at 4:26:19 AM UTC+10, Tim Graham wrote:
>>>
>>> It would be great if we could decide this by next Friday (September 4) or
>>> so, so that we have some time to do the necessary implementations and
>>> testing before alpha. (For example, running manage.py test without
>>> migrations currently gives "no such table ...". It would be nice to give a
>>> friendlier message if we require migrations for running tests
>>> (https://github.com/django/django/pull/5100 does this to some extent, but
>>> could be improved I think).
>>>
>>> I know at least Markus said he would like to present an argument for
>>> keeping support for apps without migrations.
>>>
>>> On Thursday, August 13, 2015 at 1:56:05 PM UTC-4, Tim Graham wrote:

 It's a one line change in the code to keep the status quo fro

Re: Django SQL templates

[Anssi Kääriäinen]

The django-refsql package implements a limited subset of this. You can
reference to columns in an existing ORM query using {{field}} syntax.
For example:
User.objects.annotate(ref=RefSQL("case when {{best_friend__age}} >
{{age}} then 'older' else 'younger' end"))
This would also set up a join to User's best friend model.

See https://github.com/akaariai/django-refsql

I don't like the template idea too much. While it might work well for
some use cases, it seems that ORM + raw SQL is enough for now. For
those use cases where you want to use this style, it should be
possible to implement this in a third party project. If it turns out
we are missing some public APIs in Django, we can consider adding some
more hooks.

 - Anssi

On Thu, Aug 27, 2015 at 10:43 PM, Alexey Zankevich
 wrote:
> Hi all,
>
> After reading future of QuerySet.extra() discussion, I was thinking about
> the current custom ways to tweak SQL queries in Django. Firstly, it was the
> current .extra method, which everyone likely wants to get rid of. Secondly,
> performing raw SQL queries from queryset manager or even directly from
> cursor.
> It's really a great approach to perform raw SQL and still receive objects of
> the given model at the end, I think. However, the main disadvantage of the
> current implementation - it's required to type in real table/field names,
> which destroys model's abstraction level (when you never reference
> tables/columns by their real names).
> For such cases it's great to have DSL to build SQL queries, but it's a big
> question how low- or high-level language should be. Anyway, it ended up with
> the idea to have SQL templates, similar to current Django HTML templates. I
> even implemented a sample as proof of the concept:
>
> https://github.com/Nepherhotep/django-orm-sugar/blob/exec_sql_template/tests/sql_template_test_app/app/tests.py
> https://github.com/Nepherhotep/django-orm-sugar/blob/exec_sql_template/tests/sql_template_test_app/app/templates/select_user_profile.sql
>
> How it works - there is a template "select_user_profile.sql" in a standard
> templates folder:
>
> {% autoescape off %}
> SELECT * FROM {{ model }} WHERE {{ model.name }} = {{ search_term }}
> {% endautoescape %}
>
> I had to put autoescape tags to prevent quotes in model name converted into
> HTML entities.
> A function below should be called somewhere in views:
>
> raw_queryset = exec_sql_template(UserProfile, "select_user_profile.sql",
> {'search_term': 'Bob'})
>
> This function automatically creates a variable "model" (which is actually a
> special model wrapper, which prints table or column names) and passes it
> into template context. To prevent SQL injection, passed params converted
> into %(variable_name)s representation, and escaped automatically by driver.
> Thus the solution above doesn't work with SQLite (as it doesn't support
> named variables).
> Also I wanted to automatically provide aliases for all existing models in
> form . (not implemented yet):
>
> SELECT *
> FROM
> {{ model }}
> LEFT OUTER JOIN
> {{ myapp.ModelName }}
> ON
> {{ myapp.ModelName.profile }} = {{ model.pk }}
>
> Almost everything can work out of the box using standard Django templates,
> except quotes converted into entities, unless they don't disabled
> explicitly. Also, not sure if all the databases support 'tablename'.'name'
> syntax to safely define table/column, otherwise wrapper classes should be
> engine dependent.
>
> I don't suggest a concrete implementation, just wanted to get responses of
> the concept.
>
> Regards,
> Alexey
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/da716f91-e5ab-466b-8d3e-475cd3738c12%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1GcFYiRC%2BM6Zk9bDtaCp4Yk%3DnOvb_6%3DHOBgEeuCGS-hiQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: CHECK Constraints and migrations

[Anssi Kääriäinen]

I recall another idea that would work very well with model level
Q-object based checks: making lookups and transforms work on Python
side, too. Then, when you supply a check  "ends_after_starts", we
could both add it as a database level constraint, and also validate
the constraint on Python side when altering data.

I think it is a really good idea to add more control to Fields over
what migrations they generate, but unfortunately I don't have time nor
expertise to work on this.

 - Anssi

On Fri, Aug 28, 2015 at 7:30 AM, Andrew Godwin  wrote:
> No, fields can't really specify any of the SQL used to make/update/delete
> them (partially as the SQL is heavily dependent on the database backend, and
> Django traditionally has the fields as the database-agnostic bit and the
> backends as the specific bit). All they can do is supply a db_type, which
> has to fit in the space a type would go for ADD COLUMN, etc. (that's how
> contrib.postgres has its fields work, for example).
>
> I know that Marc Tamlyn was looking into this for some of his PostgreSQL
> stuff, and other people have floated some ideas around letting fields have
> more control over migration stuff too - they may be able to chime in when we
> get to a sensible time in Europe on this.
>
> I'm definitely far from the only person who can work on fixing up the
> migration stuff; in fact, I haven't done much to it at all recently and it's
> been all other people, so if you think you want to take a stab at it, you're
> more than welcome, and I'm more than happy to review patches against
> migrations.
>
> Andrew
>
> On Thu, Aug 27, 2015 at 9:25 PM, Gavin Wahl  wrote:
>>
>> That's too bad. I don't think it's really worth it unless the experience
>> works seamlessly, because if I'm going to write a migration manually I might
>> as well just use a RunSQL. I did have another slightly hacky idea that might
>> work. Can custom Fields specify the operations used to create and drop them?
>> It might then be possible to write a pseudo-field called 'Check':
>>
>> class FooModel(models.Model):
>> # ...
>> check_ends_after_start = Check(Q(end_date__isnull=True) |
>> Q(end_date__gt=F('start_date')))
>>
>> That instead of adding a column called ends_after_start, it would add a
>> constraint.
>>
>> Do you think it's possible for anyone besides you to work on making the
>> migration framework more extensible? I'm interested in contributing in this
>> area.
>>
>>
>> On Thursday, August 27, 2015 at 8:07:55 PM UTC-6, Andrew Godwin wrote:
>>>
>>> Hi Gavin,
>>>
>>> Hooking into the migration autodetector is currently not really possible
>>> - it's quite inextensible, which is something I'd love to get ironed out in
>>> the long term but isn't really possible right now to modify it from
>>> third-party code.
>>>
>>> You can, however, supply custom migration Operation classes that people
>>> could put into their migrations manually, or potentially have some other way
>>> of outputting those migration files with operations in them (a lot of the
>>> code to write them out is re-useable). Hopefully someone else has a better
>>> idea!
>>>
>>> Andrew
>>>
>>> On Thu, Aug 27, 2015 at 6:27 PM, Gavin Wahl  wrote:

 I'm interested in writing a third-party app that allows users to
 declaratively write check constraints and have them automatically be added
 and removed by migrations. I'm envisioning being able to attach a list of Q
 objects to your model, something like:


 class Meta:
 checks = [
 Q(end_date__isnull=True) | Q(end_date__gt=F('start_date'))
 ]

 Now that we have migrations, formalized Meta, and custom lookups, is it
 possible for a third-party app to manage this seamlessly?

 I'm having trouble figuring out where it could hook in to migrations to
 detect changes to the check constraints, and automatically generate the
 migrations with operations to update the constraints. What code I should be
 reading to figure out how to do it?

 --
 You received this message because you are subscribed to the Google
 Groups "Django developers (Contributions to Django itself)" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to django-develop...@googlegroups.com.
 To post to this group, send email to django-d...@googlegroups.com.
 Visit this group at http://groups.google.com/group/django-developers.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/django-developers/CACPudh1gRB_Jov_bH-maDGr4EBRugFuV3Y%2BBzYM9w0yyki7zkg%40mail.gmail.com.
 For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+un

Re: CHECK Constraints and migrations

[Gavin Wahl]

> making lookups and transforms work on Python
side, too. Then, when you supply a check  "ends_after_starts", we
could both add it as a database level constraint, and also validate
the constraint on Python side when altering data.

Yep, that's my plan. Just have to figure out the migration stuff first.

On Thu, Aug 27, 2015 at 11:47 PM, Anssi Kääriäinen 
wrote:

> I recall another idea that would work very well with model level
> Q-object based checks: making lookups and transforms work on Python
> side, too. Then, when you supply a check  "ends_after_starts", we
> could both add it as a database level constraint, and also validate
> the constraint on Python side when altering data.
>
> I think it is a really good idea to add more control to Fields over
> what migrations they generate, but unfortunately I don't have time nor
> expertise to work on this.
>
>  - Anssi
>
> On Fri, Aug 28, 2015 at 7:30 AM, Andrew Godwin 
> wrote:
> > No, fields can't really specify any of the SQL used to make/update/delete
> > them (partially as the SQL is heavily dependent on the database backend,
> and
> > Django traditionally has the fields as the database-agnostic bit and the
> > backends as the specific bit). All they can do is supply a db_type, which
> > has to fit in the space a type would go for ADD COLUMN, etc. (that's how
> > contrib.postgres has its fields work, for example).
> >
> > I know that Marc Tamlyn was looking into this for some of his PostgreSQL
> > stuff, and other people have floated some ideas around letting fields
> have
> > more control over migration stuff too - they may be able to chime in
> when we
> > get to a sensible time in Europe on this.
> >
> > I'm definitely far from the only person who can work on fixing up the
> > migration stuff; in fact, I haven't done much to it at all recently and
> it's
> > been all other people, so if you think you want to take a stab at it,
> you're
> > more than welcome, and I'm more than happy to review patches against
> > migrations.
> >
> > Andrew
> >
> > On Thu, Aug 27, 2015 at 9:25 PM, Gavin Wahl  wrote:
> >>
> >> That's too bad. I don't think it's really worth it unless the experience
> >> works seamlessly, because if I'm going to write a migration manually I
> might
> >> as well just use a RunSQL. I did have another slightly hacky idea that
> might
> >> work. Can custom Fields specify the operations used to create and drop
> them?
> >> It might then be possible to write a pseudo-field called 'Check':
> >>
> >> class FooModel(models.Model):
> >> # ...
> >> check_ends_after_start = Check(Q(end_date__isnull=True) |
> >> Q(end_date__gt=F('start_date')))
> >>
> >> That instead of adding a column called ends_after_start, it would add a
> >> constraint.
> >>
> >> Do you think it's possible for anyone besides you to work on making the
> >> migration framework more extensible? I'm interested in contributing in
> this
> >> area.
> >>
> >>
> >> On Thursday, August 27, 2015 at 8:07:55 PM UTC-6, Andrew Godwin wrote:
> >>>
> >>> Hi Gavin,
> >>>
> >>> Hooking into the migration autodetector is currently not really
> possible
> >>> - it's quite inextensible, which is something I'd love to get ironed
> out in
> >>> the long term but isn't really possible right now to modify it from
> >>> third-party code.
> >>>
> >>> You can, however, supply custom migration Operation classes that people
> >>> could put into their migrations manually, or potentially have some
> other way
> >>> of outputting those migration files with operations in them (a lot of
> the
> >>> code to write them out is re-useable). Hopefully someone else has a
> better
> >>> idea!
> >>>
> >>> Andrew
> >>>
> >>> On Thu, Aug 27, 2015 at 6:27 PM, Gavin Wahl  wrote:
> 
>  I'm interested in writing a third-party app that allows users to
>  declaratively write check constraints and have them automatically be
> added
>  and removed by migrations. I'm envisioning being able to attach a
> list of Q
>  objects to your model, something like:
> 
> 
>  class Meta:
>  checks = [
>  Q(end_date__isnull=True) | Q(end_date__gt=F('start_date'))
>  ]
> 
>  Now that we have migrations, formalized Meta, and custom lookups, is
> it
>  possible for a third-party app to manage this seamlessly?
> 
>  I'm having trouble figuring out where it could hook in to migrations
> to
>  detect changes to the check constraints, and automatically generate
> the
>  migrations with operations to update the constraints. What code I
> should be
>  reading to figure out how to do it?
> 
>  --
>  You received this message because you are subscribed to the Google
>  Groups "Django developers (Contributions to Django itself)" group.
>  To unsubscribe from this group and stop receiving emails from it, send
>  an email to django-develop...@googlegroups.com.
>  To post to this group, send email to django-d

Re: Keeping apps without migrations?

[Andrew Godwin]

On Thu, Aug 27, 2015 at 10:25 PM, Anssi Kääriäinen 
wrote:

> Could we allow applications to have fake migration files? I'm thinking
> of something like operations.AllModelsUpdated(). After this operation
> is ran, Django will use the present models for that app. You could
> place this as your last migration, and run direct SQL in previous
> migrations (or use some completely different mechanism) . Direct SQL
> without model state changes will be fast. The last migration would
> supposedly be fast, as it only needs to construct up to date images of
> the current models.
>
> The problem is that I'm not at all sure how this will mix with
> migrations from other applications, so this proposal is perhaps
> technically impossible to do. But if this is feasible, then it should
> be possible to mix migrations and manual database changes pretty much
> seamlessly. This one would work much better if we had project level
> migrations - implementing a migration operation that tells Django that
> all of the project's models are up to date is almost trivial.
>

The problem is mainly swappable models. We can only allow one direction of
"migrated depending on unmigrated" or "unmigrated depending on migrated",
so I chose "migrated depending on unmigrated". The problem is all about
when to run the "syncdb" phase; if you run it before migrations you get one
solution, and after the other.

Allowing apps to depend on these fake migrations would only work if
"executing" them actually performed the syncdb phase of the cycle, which
unfortunately is written in an entire-project-at-once way, so it doesn't
have to handle FK dependencies. Solving that problem would seem to be very
difficult to me, and probably impossible (you'd need to be able to syncdb
individual apps at once, and correctly detect dependencies to migrated apps
and other unmigrated apps, and so on, until you just have given them
migrations in-memory).


>
> I believe a fake migration would work pretty well for the use case
> where you want to use manual database control for your project, but
> use 3rd party applications with migrations.
>
> Another idea I've been thinking is that maybe we could have snapshots
> for migrations. The reason I believe this would work is that building
> the model state is expensive, but in many cases it is completely
> redundant to build the model state from scratch.
>
> The snapshots would essentially do the following operation: after
> migrations X, Y and Z are applied the model state will be as presented
> in this migration. The snapshots would only contain model state, not
> any operations. When you run manage.py migrate, the app state builder
> will be able to skip to latest snapshot that has all dependencies
> already applied, and then continue building model state from there.
> When performance begins to degrade, you can create a new snapshot. The
> snapshots would probably have to be project level to work properly.
> This would seriously speed up migrations, except for the case where
> you run the migrations against fresh database.
>

I think Markus was toying with this idea, but I'm not sure if it was
similar or not. It definitely sounds faster - this is how South did it, of
course, and given we now have all the serialisation/deserialisation code
for models it would seem to be not too hard to do.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uo9%2B%2B5Nh-u2SKwX%2BhKQYckN%2BcAtAiRHTj9ru_-SY%2B4DsQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.