django migrations on live database for large tables for postgresql

[pavel . tyslacki]

Hi,

I found that django migrations don't designed for live updates on large 
tables for postgres, eg. index creation doesn't use concurrently or alter 
table operation can take ACCESS EXCLUSIVE lock for long time.

Before I mostly wrote custom migrations, but after one of downtime I 
decided to avoid most common cases at all and wrote postgres backend that 
helps me in it: https://github.com/tbicr/django-pg-zero-downtime-migrations. 
When I wrote it I found that it's a pretty easy extend core postgres 
backend, but a few things required rewrite on my side:

- 
https://github.com/django/django/blob/master/django/db/backends/base/schema.py#L152
 
used both for table columns creation and altering table add column, but not 
null constraint and index constraints can't be changed without whole method 
rewriting
- for one core sql statement I found more "safe" equivalent with two sql 
statements, but it's a bit hard to execute it, for example more save create 
invalid constraint and than validate it or create concurrently index and 
than create unique constraint

also I found that not all constraint names quoted in generated sql, for 
example add_unique_together got `ALTER TABLE "table_name" ADD CONSTRAINT 
table_name_f1_f2_hash_uniq UNIQUE ("f1", "f2")`, but expected `ALTER TABLE 
"table_name" ADD CONSTRAINT "table_name_f1_f2_hash_uniq" UNIQUE ("f1", 
"f2")`

So I just curious is anybody interested to make django migrations more 
reliable for large tables in prod or interested in some improvements 
highlighted above, that can simplify writing backends for this purpose?

I also will be appreciated if somebody can share another solutions for this 
purpose.

Paveł

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/421f001b-ca11-42bf-8735-0c6947b41988%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Model creation in autocommit mode.

[Florian Apolloner]

Hi there,

a fun issue came up on IRC today: Even when in autocommit mode, Django 
starts a transaction when doing Model.objects.create 
(https://github.com/django/django/blob/fb2964a4106b1282c4179b6fbbd0374f5be1ccac/django/db/models/base.py#L752).
 
This makes some sense when there are parents to be saved also, but imo 
results in a rather high database roundtrip number for the simple case 
where there is a single model without inheritance. Does anyone know why we 
do this?

I've created a really simple patch 
https://github.com/django/django/pull/10448 which already (mostly) passes 
the testsuite.

Cheers,
Florian

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/9fe4e535-f19d-4a36-85ba-82db43363ebc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Model creation in autocommit mode.

[Tim Graham]

https://code.djangoproject.com/ticket/21171 looks like it should help.

On Wednesday, September 26, 2018 at 2:56:34 PM UTC-4, Florian Apolloner 
wrote:
>
> Hi there,
>
> a fun issue came up on IRC today: Even when in autocommit mode, Django 
> starts a transaction when doing Model.objects.create (
> https://github.com/django/django/blob/fb2964a4106b1282c4179b6fbbd0374f5be1ccac/django/db/models/base.py#L752).
>  
> This makes some sense when there are parents to be saved also, but imo 
> results in a rather high database roundtrip number for the simple case 
> where there is a single model without inheritance. Does anyone know why we 
> do this?
>
> I've created a really simple patch 
> https://github.com/django/django/pull/10448 which already (mostly) passes 
> the testsuite.
>
> Cheers,
> Florian
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5251ad5d-0384-4848-a933-ab7f6e3ab240%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Model creation in autocommit mode.

[Florian Apolloner]

Jupp, that ticket at least shows that I am not the first one to stumble 
onto this :) Baring no objections I will at least try to fix it for the 
"easy" cases; seems like an easy win.

On Wednesday, September 26, 2018 at 9:54:05 PM UTC+2, Tim Graham wrote:
>
> https://code.djangoproject.com/ticket/21171 looks like it should help.
>
> On Wednesday, September 26, 2018 at 2:56:34 PM UTC-4, Florian Apolloner 
> wrote:
>>
>> Hi there,
>>
>> a fun issue came up on IRC today: Even when in autocommit mode, Django 
>> starts a transaction when doing Model.objects.create (
>> https://github.com/django/django/blob/fb2964a4106b1282c4179b6fbbd0374f5be1ccac/django/db/models/base.py#L752).
>>  
>> This makes some sense when there are parents to be saved also, but imo 
>> results in a rather high database roundtrip number for the simple case 
>> where there is a single model without inheritance. Does anyone know why we 
>> do this?
>>
>> I've created a really simple patch 
>> https://github.com/django/django/pull/10448 which already (mostly) 
>> passes the testsuite.
>>
>> Cheers,
>> Florian
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/71d4497a-d749-4899-8f8a-55e9c78284a8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: 301 vs 308 in default redirect class for CommonMiddleware

[Adam Johnson]

I don't think it's worth the deprecation path to rename the old classes -
it doesn't seem like there's a substantial benefit to using the different
codes and getting everyone to change their imports is a lot of work to
impose on the world.

There must be some name for the 308 one that would work, but I can't think
of one off the top of my head.

The new class could also be mentioned in the 301/302 classes' docstrings.

On Sun, 23 Sep 2018 at 06:56, Matthieu Bonnefoy 
wrote:

> Hi Adam,
>
> Thanks for your reply.
>
> I agree, I think that would be a nice addition. The issue we’ll have is
> that the current naming of the HttpResponsePermanentRedirect (301 Moved
> Permanently) will be confusing.
>
> existing
> - HttpResponsePermanentRedirect (301 Moved Permanently) => should probably
> be HttpResponseMovedPermanently
> - HttpResponseRedirect (302 Found) => should probably be HttpResponseFound
>
> new additions
> - HttpResponseTemporaryRedirect (307 Temporary Redirect) => probably ok
> with the existing naming for 302
> - ? (308 Permanent Redirect) ==> naming conflict with the existing 301,
> HttpResponsePermanentRedirect308? HttpResponsePermanentRedirection?
> HttpResponsePermanentlyRedirected? … not super convinced by these
> suggestions. What do you think?
>
> I could add the new responses and mention them in the documentation here
> with the recommendation
>
> https://docs.djangoproject.com/en/2.1/ref/request-response/#httpresponse-subclasses
>
> We could also mention the 308 redirect at the end of the class docstring
> here
> https://docs.djangoproject.com/en/2.1/_modules/django/middleware/common/
>
> Cheers,
> Matthieu
>
>
>
> On Tuesday, 18 September 2018 23:55:17 UTC+12, Adam Johnson wrote:
>>
>> This is historical, 307 and 308 were added later, and I think we couldn't
>> change it without breaking backwards compatibility. That said, 307 and 308
>> classes could be added to django.http.response and the documentation could
>> recommend their use. Mozilla docs show that they're compatible with all
>> tracked web browsers so most Django sites could use them.
>>
>> On Tue, 18 Sep 2018 at 07:14, Matthieu Bonnefoy 
>> wrote:
>>
>>> Hi there,
>>>
>>> I am wondering why the default redirect class in the common middleware
>>> is a 301 Moved Permanently response.
>>>
>>> https://github.com/django/django/blob/master/django/middleware/common.py#L32
>>> https://github.com/django/django/blob/master/django/http/response.py#L476
>>>
>>> I just got the issue of a POST request being changed as a GET request by
>>> the redirect and found it quite confusing.
>>> It seems that a 308 Permanent Redirect (by the way the name of the
>>> redirect subclass is now a bit confusing) is now widely supported and would
>>> be a better option.
>>> https://tools.ietf.org/html/rfc7538
>>> https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/308
>>>
>>>+---+---+---+
>>>|   | Permanent | Temporary |
>>>+---+---+---+
>>>| Allows changing the request method from   | 301   | 302   |
>>>| POST to GET   |   |   |
>>>| Does not allow changing the request   | 308   | 307   |
>>>| method from POST to GET   |   |   |
>>>+---+---+---+
>>>
>>>
>>> What do you think?
>>>
>>> Thanks,
>>> Matthieu
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to django-develop...@googlegroups.com.
>>> To post to this group, send email to django-d...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/django-developers.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/django-developers/a9f807b8-d31c-4729-a9bc-ac80274d4590%40googlegroups.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>> --
>> Adam
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/e9400ed9-f79b-4202-9f15-fa2addb3cf92%40googlegroups.com
>