[Feature Request][Model, ORM] Disabling a field before removal to support continuous delivery

[Matthieu Rudelle]

Hi there, 

I can't find any previous ticket proposing a solution to this problem so 
here are my findings: 

**Use case**:
When using continuous delivery several versions of the code can be running 
in parallel on se same DB. Say for instance that release 2.42 is in 
production, 2.43 is about to be rolled out and in this release one field 
(say ''MyModel.my_unused_field'') is not used anymore and was removed. 
Before rolling out 2.43 the DB is migrated and column ''my_unused_field'' 
of ''MyModel'' is removed. This makes 2.42 crash saying that one column is 
not found even though 2.42 does not use the field anywhere in the code.

**Temporary solution**:
Do not makemigrations until de 2.44 release, but it does not scale well 
with many contributors and CI tools (doing their awesome job of making sure 
migrations and models are in sync) will complain.

**Proposed solution**:
Have a ''disabled'' param on Field. When activated this field is not 
fetched from the DB but replaced by a hardcoded value. 
In our use case, ''disabled'' is added at the 2.42 release, then when 2.43 
rolls out and migrates the DB no error is thrown.

**Refs**:
- django-users discussion: 
https://groups.google.com/forum/#!topic/django-users/HY_6rZZ0Kk8
- the same problem discussed in this article, but with a slightly different 
solution: 
https://pankrat.github.io/2015/django-migrations-without-downtimes/#django-wishlist
 
(third item in the wishlist)
- Previous django-developpers discussion: 
https://groups.google.com/d/msg/django-developers/5ofrxeLT95E/XhppgA_KAQAJ

What do you guys think?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/637c3542-c5bd-4493-8b12-44eb61f34a68%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Feature Request][Model, ORM] Disabling a field before removal to support continuous delivery

[Ryan Hiebert]

I'm not sure about the solution you mentioned, but the problem you mention
is one that I definitely do deal with. At my work we have been happy with
using a "safe" migrate command that only runs migrations that are marked as
safe to run before the deployment happens, to address exactly this kind of
scenario. You still need to make sure you don't mix stuff in the same
migration that delete and add a column, for example, but it makes the
process much easier, and we've been able to add the safemigrate command to
our regular automated deployments. The full migrate then only runs when
manually triggered, to deal with removing columns, etc.

You can check out our repository. We've been pretty happy with how it's
working for us. https://github.com/aspiredu/django-safemigrate

On Mon, Jun 24, 2019 at 8:15 AM Matthieu Rudelle 
wrote:

> Hi there,
>
> I can't find any previous ticket proposing a solution to this problem so
> here are my findings:
>
> **Use case**:
> When using continuous delivery several versions of the code can be running
> in parallel on se same DB. Say for instance that release 2.42 is in
> production, 2.43 is about to be rolled out and in this release one field
> (say ''MyModel.my_unused_field'') is not used anymore and was removed.
> Before rolling out 2.43 the DB is migrated and column ''my_unused_field''
> of ''MyModel'' is removed. This makes 2.42 crash saying that one column is
> not found even though 2.42 does not use the field anywhere in the code.
>
> **Temporary solution**:
> Do not makemigrations until de 2.44 release, but it does not scale well
> with many contributors and CI tools (doing their awesome job of making sure
> migrations and models are in sync) will complain.
>
> **Proposed solution**:
> Have a ''disabled'' param on Field. When activated this field is not
> fetched from the DB but replaced by a hardcoded value.
> In our use case, ''disabled'' is added at the 2.42 release, then when 2.43
> rolls out and migrates the DB no error is thrown.
>
> **Refs**:
> - django-users discussion:
> https://groups.google.com/forum/#!topic/django-users/HY_6rZZ0Kk8
> - the same problem discussed in this article, but with a slightly
> different solution:
> https://pankrat.github.io/2015/django-migrations-without-downtimes/#django-wishlist
> (third item in the wishlist)
> - Previous django-developpers discussion:
> https://groups.google.com/d/msg/django-developers/5ofrxeLT95E/XhppgA_KAQAJ
>
> What do you guys think?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/637c3542-c5bd-4493-8b12-44eb61f34a68%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CABpHFHSa4hXVrp7M2MWLcsPXSRnCnvhMQ%3D8D3q1Rg8AsAXU4Cw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Feature Request][Model, ORM] Disabling a field before removal to support continuous delivery

[Dan Davis]

I'd agree that it is a definite use case.  In the dev-ops world, it is
evidently called a "blue torquoise green deployment".   It could be done as
long as the code is not adding a table/field/etc.

My discussion on django-users,
https://groups.google.com/forum/#!topic/django-users/QCmy9reH8cI, also
included this issue, but I was also concerned about:
* Removing a model
* Renaming a field
* etc.

The django-safemigrate 
repository/app seems to be easier to generalize, as such an option could be
added to makemigrations.
Some questions:

   - How does the "safe" field of migrations work with other migrations
   related commands, such as squashmigrations?   It seems to me that only
   migrations that share the same value of "safe" could be squashed.
   - Can makemigrations figure out which migrations are "safe", such as:
  - It is always safe to add a column or table.
  - It is never safe to drop or rename a column or table.


On Mon, Jun 24, 2019 at 9:48 AM Ryan Hiebert  wrote:

> I'm not sure about the solution you mentioned, but the problem you mention
> is one that I definitely do deal with. At my work we have been happy with
> using a "safe" migrate command that only runs migrations that are marked as
> safe to run before the deployment happens, to address exactly this kind of
> scenario. You still need to make sure you don't mix stuff in the same
> migration that delete and add a column, for example, but it makes the
> process much easier, and we've been able to add the safemigrate command to
> our regular automated deployments. The full migrate then only runs when
> manually triggered, to deal with removing columns, etc.
>
> You can check out our repository. We've been pretty happy with how it's
> working for us. https://github.com/aspiredu/django-safemigrate
>
> On Mon, Jun 24, 2019 at 8:15 AM Matthieu Rudelle 
> wrote:
>
>> Hi there,
>>
>> I can't find any previous ticket proposing a solution to this problem so
>> here are my findings:
>>
>> **Use case**:
>> When using continuous delivery several versions of the code can be
>> running in parallel on se same DB. Say for instance that release 2.42 is in
>> production, 2.43 is about to be rolled out and in this release one field
>> (say ''MyModel.my_unused_field'') is not used anymore and was removed.
>> Before rolling out 2.43 the DB is migrated and column ''my_unused_field''
>> of ''MyModel'' is removed. This makes 2.42 crash saying that one column is
>> not found even though 2.42 does not use the field anywhere in the code.
>>
>> **Temporary solution**:
>> Do not makemigrations until de 2.44 release, but it does not scale well
>> with many contributors and CI tools (doing their awesome job of making sure
>> migrations and models are in sync) will complain.
>>
>> **Proposed solution**:
>> Have a ''disabled'' param on Field. When activated this field is not
>> fetched from the DB but replaced by a hardcoded value.
>> In our use case, ''disabled'' is added at the 2.42 release, then when
>> 2.43 rolls out and migrates the DB no error is thrown.
>>
>> **Refs**:
>> - django-users discussion:
>> https://groups.google.com/forum/#!topic/django-users/HY_6rZZ0Kk8
>> - the same problem discussed in this article, but with a slightly
>> different solution:
>> https://pankrat.github.io/2015/django-migrations-without-downtimes/#django-wishlist
>> (third item in the wishlist)
>> - Previous django-developpers discussion:
>> https://groups.google.com/d/msg/django-developers/5ofrxeLT95E/XhppgA_KAQAJ
>>
>> What do you guys think?
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-developers@googlegroups.com.
>> Visit this group at https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-developers/637c3542-c5bd-4493-8b12-44eb61f34a68%40googlegroups.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CABpHFHSa4hXVrp7M2MWLcsPXSRnCnvhMQ%3D8D3q1Rg8AsAXU4Cw%40mail.gmail.com
> 

Re: [Feature Request][Model, ORM] Disabling a field before removal to support continuous delivery

[Ryan Hiebert]

On Mon, Jun 24, 2019, 21:29 Dan Davis  wrote:

>
> Some questions:
>
>- How does the "safe" field of migrations work with other migrations
>related commands, such as squashmigrations?   It seems to me that only
>migrations that share the same value of "safe" could be squashed.
>
> If does not affect squashing. I think this is roughly reasonable, at least
for my uses, although I haven't had any real world use for squash myself. I
recently had to remake the migrations in my repo for $WORK project, and it
made the most sense to keep them as the default safe after deploy, to match
what other apps would get. I think in most cases it won't matter when the
first migration is safe for practical use.


>- Can makemigrations figure out which migrations are "safe", such as:
>   - It is always safe to add a column or table.
>   - It is never safe to drop or rename a column or table.
>
> This is a feature that has sounded neat to me, approximately feasible, but
hasn't yet been worth the time to implement. We've been happy adding the
safety annotations manually.

>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CABpHFHSb2AHKb-XhZcAk6gZ3%2B%2Bt5nz7UkBUNyQ_FC%3DsYGgO_Yw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Feature Request][Model, ORM] Disabling a field before removal to support continuous delivery

[Dan Davis]

>
> If does not affect squashing. I think this is roughly reasonable, at least
> for my uses, although I haven't had any real world use for squash myself. I
> recently had to remake the migrations in my repo for $WORK project, and it
> made the most sense to keep them as the default safe after deploy, to match
> what other apps would get. I think in most cases it won't matter when the
> first migration is safe for practical use.


I don't usually use squashmigrations either, being more likely to simply
rewrite or make them initial.   However, if the functionality of your
module were to be added to Django itself, or become more productized, then
these are the issues to think about, right?

I certainly think the use case (support for something like blue/green
deployments with shared database) is very important.
I'm not a super experienced Django contributor, so I'd love to hear what
more experienced hands have to say about it.

On Mon, Jun 24, 2019 at 11:28 PM Ryan Hiebert  wrote:

>
>
> On Mon, Jun 24, 2019, 21:29 Dan Davis  wrote:
>
>>
>> Some questions:
>>
>>- How does the "safe" field of migrations work with other migrations
>>related commands, such as squashmigrations?   It seems to me that only
>>migrations that share the same value of "safe" could be squashed.
>>
>> If does not affect squashing. I think this is roughly reasonable, at
> least for my uses, although I haven't had any real world use for squash
> myself. I recently had to remake the migrations in my repo for $WORK
> project, and it made the most sense to keep them as the default safe after
> deploy, to match what other apps would get. I think in most cases it won't
> matter when the first migration is safe for practical use.
>
>
>>- Can makemigrations figure out which migrations are "safe", such as:
>>   - It is always safe to add a column or table.
>>   - It is never safe to drop or rename a column or table.
>>
>> This is a feature that has sounded neat to me, approximately feasible,
> but hasn't yet been worth the time to implement. We've been happy adding
> the safety annotations manually.
>
>> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CABpHFHSb2AHKb-XhZcAk6gZ3%2B%2Bt5nz7UkBUNyQ_FC%3DsYGgO_Yw%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFzonYZpeYZGqGk3SMgYSHPk7%2BZ4vQfDNv-GaQJrjFZLivL0gQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Feature Request][Model, ORM] Disabling a field before removal to support continuous delivery

[Paveł Tyślacki]

there are similar to `alter table drop column`  issue:  `alter table rename 
column`, `drop table`, `rename table`. (honestly `alter table drop column` 
and `drop table` a bit different wiith `alter table remane column` and 
`rename table`)

Look like you general flow for migration:
1. change code and create migrations
2. apply migrations
3. apply code for all your instances

the issue in this cases I can describe: we have code that we still used.

for `alter table drop column`, `drop table` next scenario works fine (as 
your temp solution too):

1. remove column/table usage from codebase
2. apply this code for all of your instances
3. apply migration with field/table removal

But if you mix migrations for `alter table add column` and `alter table 
drop column` - you cannot safely apply both migrations simultaneously, and 
your proposal sounds pretty reasonable there.

However if you add disabled option, then make migration, remove field, then 
make migration, then apply this changes to your environment - you will get 
same issue. So to automate deployment process and to get achievable result 
you want I think you need something more complex: deploying atomic changes, 
but for me it sounds more about deployment than django itself.




On Monday, 24 June 2019 16:15:04 UTC+3, Matthieu Rudelle wrote:
>
> Hi there, 
>
> I can't find any previous ticket proposing a solution to this problem so 
> here are my findings: 
>
> **Use case**:
> When using continuous delivery several versions of the code can be running 
> in parallel on se same DB. Say for instance that release 2.42 is in 
> production, 2.43 is about to be rolled out and in this release one field 
> (say ''MyModel.my_unused_field'') is not used anymore and was removed. 
> Before rolling out 2.43 the DB is migrated and column ''my_unused_field'' 
> of ''MyModel'' is removed. This makes 2.42 crash saying that one column is 
> not found even though 2.42 does not use the field anywhere in the code.
>
> **Temporary solution**:
> Do not makemigrations until de 2.44 release, but it does not scale well 
> with many contributors and CI tools (doing their awesome job of making sure 
> migrations and models are in sync) will complain.
>
> **Proposed solution**:
> Have a ''disabled'' param on Field. When activated this field is not 
> fetched from the DB but replaced by a hardcoded value. 
> In our use case, ''disabled'' is added at the 2.42 release, then when 2.43 
> rolls out and migrates the DB no error is thrown.
>
> **Refs**:
> - django-users discussion: 
> https://groups.google.com/forum/#!topic/django-users/HY_6rZZ0Kk8
> - the same problem discussed in this article, but with a slightly 
> different solution: 
> https://pankrat.github.io/2015/django-migrations-without-downtimes/#django-wishlist
>  
> (third item in the wishlist)
> - Previous django-developpers discussion: 
> https://groups.google.com/d/msg/django-developers/5ofrxeLT95E/XhppgA_KAQAJ
>
> What do you guys think?
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/0422ed55-5c7f-4ed2-b096-7df02311b917%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.