Re: Question (potential feature request) on dropping database column in migration

[vishvajit pathak]

Well this is what we call as the downtime.


On Tuesday, 2 October 2018 06:17:10 UTC+5:30, martin_x wrote:
>
> Hi there,
>
> Thanks for making Django a great framework for starters :D
>
> Recently my team have run into problems when trying to remove a database 
> column from the model without stopping our Django server, below is the 
> timeline:
>
> 1. Before migration happens, we have column_A we want to remove, and 
> running on old release_1
> 2. Now we run migration to remove column_A, our webserver is still running 
> on old release_1 and serving requests
> 3. After migration, we ship the new release_2
>
> However, between step 2 and 3, there are requests coming and referencing 
> column_A we want to remove, and throws a exception of course.
>
> So is there anyway I can mark a database column a special state 
> (deprecated, purged in memory, etc), so it has the following affect:
> 1. Won’t generate migration file, which means database wise, nothing has 
> changed
> 2. However, when Django loads the model into memory, it will ignore 
> column_A completely. So query, create, update, etc won’t try to access 
> column_A.
>
> The reason we want to do it that way is so we can achieve no downtime, 
> seamless migration for our application. I believe more people will benefit 
> from this.
>
> Please let me know if more information is needed. Looking forward to 
> hearing from you.
>
> Martin
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/55d789ea-3f7c-4e4b-a682-3ff1745024c7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: django makemessages management command

[vishvajit pathak]

You could include the a python file where you can place all the strings 
which you manually wanted to add and then use the *makemessages *management 
command. I think it will serve the purpose. 

I think this behaviour of commenting out manually added strings by 
*makemessages *management command is correct.


On Monday, 24 September 2018 17:19:08 UTC+5:30, Osama AbuOmar wrote:
>
> About the django *makemessages *management command. I think there should 
> by an option to *NOT *let django comment out manually added translations 
> to the .po files.
> sometimes I need to manually add a translation definition (msgid, msgstr), 
> these are not in any of my template or python files. and when I run 
> makemessages they get commented out. and this is an issue.
> thanks 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/04357d3a-6033-4396-9d2c-791cca9cf742%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Content types shouldn't be created on post_migrate signal

[Arthur Rio]

Hey Marcin,

The problem is that data migration based on app layer (python objects, ie.
Models and Managers here) will cause troubles after some time (when app is
changing).
In the other words - you cannot rely on your app layer when doing database
changes. You should never do that, especially for projects requiring LTS.

In theory I understand the idea, but in practice, migrations, the ORM and
the content type model are all part of Django and I don’t really see how
the app changing could cause troubles. Do you have an example in mind?

Maybe "automatic" is not a proper word. They should be created
automatically, but should be applied "at the right time".

Ok, that we agree on!

CT opt-in should be connected to a signal related to the creation of
migration files (Autodetector?), not to a signal related to their
execution. I.e. pre/post_autodection signals should be introduced.

I think we agree that the solution would be some sort of signal triggered
when a model creation/deletion is detected (in the `makemigrations` phase
as opposed to `migrate`) so that some code is added to the generated
migration. The use of a signal is important to keep things decoupled and
flexible.

The “some code is added to the migration” part still needs to be
determined, either in the shape of insert/delete statements or a RunPython
leveraging the ORM. In both cases, the values to insert (Adding a model) or
to lookup for delete (Removing a model) are just 2 strings, the app label
and the model class name.

After adding contrib.contenttypes, Django should check existence of
django_content_type table. If exists, Django should only check for data
changes and generate series of inserts/deletes. If not, Django should
generate inserts for all models. If CT is removed later, Django should
remove all CT data .

It’s a good idea but I don’t know offhand how we can keep migrations and
content type decoupled to do that (especially the removal).

Finally, I also think the concept could be extended to the permission model
which faces similar issues.


Regards

--

Arthur



On October 4, 2018 at 4:47:19 PM, Marcin Nowak (marcin.j.no...@gmail.com)
wrote:


Hi Arthur.

BTW: RunPython() is another thing, which can break your migrations, and
should not be used (especially not by Django internally), because it relies
on the application layer.

How else can you do a data migration? There is no
> `migrations.InsertIntoTable`,
>

You're right. That's why "Insert" should be (in my opinion) introduced.
Together with "migrations.Delete".

The problem is that data migration based on app layer (python objects, ie.
Models and Managers here) will cause troubles after some time (when app is
changing).
In the other words - you cannot rely on your app layer when doing database
changes. You should never do that, especially for projects requiring LTS.

RunPython() should be used only when you cannot do anything else. In such
case you must accept all consequences. I'm not against RunPython(), but
against doing data migrations using it.

The problem with hypothetical "Insert" operation is with mapping field
types. Insert cannot use Models directly (app layer is changing over a
time), but should "know" how to map arguments (python types, values) to a
database literals. It can be achieved by introducing field mapping for a
every insert or per migration file (something like "model freeze", but only
for used fields).

Also Insert should not accept variables calculated in the app layer (within
a migration) - it should contain only plain/direct data. But using Python
as a language for migrations, will be hard to avoid such possibility. This
is important, because database management should not rely on app layer.
Using variables (i.e. from settings) would result in inconsistent data
changes between different environments.

the only other way currently would be to run a `migrations.RunSql` query
> which may look different based on the database used.
>

RunSQL is not the solution for db agnostic operations. It may be only used
within a project, because db engine used changes rarely (due to the nature
and importance of the data and relational databases, and systems dependent
on the db).
But RunSQL is a handful operation, because SQL is a natural language for db
management. I'm doing many raw sqls in migrations.


> Two things are wrong:
>
>- automatic creation of content types
>
> Why is it wrong to automatically create a content type?
>

Maybe "automatic" is not a proper word. They should be created
automatically, but should be applied "at the right time".


> Content type is an opt-in feature since you can remove it from
> `INSTALLED_APPS`.
>

I know, but it is required by contrib.auth. I saw no project depending on
something else, so CT is optional.. but theoretically ;)


>
>- creation of content types after running migrations
>
> That’s the only real problem for me. Maybe using a signal for
> `migrations.CreateModel` (e.g. post_create_model), instead of using
>

Re: Content types shouldn't be created on post_migrate signal

[Arthur Rio]

For some reason the text is white… Here it is in black:

Hey Marcin,

The problem is that data migration based on app layer (python objects, ie.
Models and Managers here) will cause troubles after some time (when app is
changing).
In the other words - you cannot rely on your app layer when doing database
changes. You should never do that, especially for projects requiring LTS.

In theory I understand the idea, but in practice, migrations, the ORM and
the content type model are all part of Django and I don’t really see how
the app changing could cause troubles. Do you have an example in mind?

Maybe "automatic" is not a proper word. They should be created
automatically, but should be applied "at the right time".

Ok, that we agree on!

CT opt-in should be connected to a signal related to the creation of
migration files (Autodetector?), not to a signal related to their
execution. I.e. pre/post_autodection signals should be introduced.

I think we agree that the solution would be some sort of signal triggered
when a model creation/deletion is detected (in the `makemigrations` phase
as opposed to `migrate`) so that some code is added to the generated
migration. The use of a signal is important to keep things decoupled and
flexible.

The “some code is added to the migration” part still needs to be
determined, either in the shape of insert/delete statements or a RunPython
leveraging the ORM. In both cases, the values to insert (Adding a model) or
to lookup for delete (Removing a model) are just 2 strings, the app label
and the model class name.

After adding contrib.contenttypes, Django should check existence of
django_content_type table. If exists, Django should only check for data
changes and generate series of inserts/deletes. If not, Django should
generate inserts for all models. If CT is removed later, Django should
remove all CT data .

It’s a good idea but I don’t know offhand how we can keep migrations and
content type decoupled to do that (especially the removal).

Finally, I also think the concept could be extended to the permission model
which faces similar issues.


Regards

--

Arthur

On October 5, 2018 at 9:30:58 AM, Arthur Rio (arthur.ri...@gmail.com) wrote:

Hey Marcin,

The problem is that data migration based on app layer (python objects, ie.
Models and Managers here) will cause troubles after some time (when app is
changing).
In the other words - you cannot rely on your app layer when doing database
changes. You should never do that, especially for projects requiring LTS.

In theory I understand the idea, but in practice, migrations, the ORM and
the content type model are all part of Django and I don’t really see how
the app changing could cause troubles. Do you have an example in mind?

Maybe "automatic" is not a proper word. They should be created
automatically, but should be applied "at the right time".

Ok, that we agree on!

CT opt-in should be connected to a signal related to the creation of
migration files (Autodetector?), not to a signal related to their
execution. I.e. pre/post_autodection signals should be introduced.

I think we agree that the solution would be some sort of signal triggered
when a model creation/deletion is detected (in the `makemigrations` phase
as opposed to `migrate`) so that some code is added to the generated
migration. The use of a signal is important to keep things decoupled and
flexible.

The “some code is added to the migration” part still needs to be
determined, either in the shape of insert/delete statements or a RunPython
leveraging the ORM. In both cases, the values to insert (Adding a model) or
to lookup for delete (Removing a model) are just 2 strings, the app label
and the model class name.

After adding contrib.contenttypes, Django should check existence of
django_content_type table. If exists, Django should only check for data
changes and generate series of inserts/deletes. If not, Django should
generate inserts for all models. If CT is removed later, Django should
remove all CT data .

It’s a good idea but I don’t know offhand how we can keep migrations and
content type decoupled to do that (especially the removal).

Finally, I also think the concept could be extended to the permission model
which faces similar issues.


Regards

--

Arthur



On October 4, 2018 at 4:47:19 PM, Marcin Nowak (marcin.j.no...@gmail.com)
wrote:


Hi Arthur.

BTW: RunPython() is another thing, which can break your migrations, and
should not be used (especially not by Django internally), because it relies
on the application layer.

How else can you do a data migration? There is no
> `migrations.InsertIntoTable`,
>

You're right. That's why "Insert" should be (in my opinion) introduced.
Together with "migrations.Delete".

The problem is that data migration based on app layer (python objects, ie.
Models and Managers here) will cause troubles after some time (when app is
changing).
In the other words - you cannot rely on your app layer when doing database
changes. You shou

Re: Can we remove FILE_CHARSET?

[Jon Dufresne]

> Is that always available these days? (I'd guess yes.)

I too would guess yes. I believe any reasonably modern text editor will
support
UTF-8 and even likely default to saving in that encoding. I know mine does.

> Is is something we want to impose? Not sure. Are there people doing
> otherwise? (No idea.)

For templates, it wouldn't be imposed. Users can still override the template
engine's encoding with the 'file_charset' option.

For static files, without imposing it, we're back to the third-party app
concern. Just like DEFAULT_CHARSET, it would difficult to change
FILE_CHARSET
_and_ integrate third party apps. The third party apps have likely encoded
their static files using UTF-8, so setting FILE_CHARSET to some other value
will break.

Cheers,
Jon


On Wed, Oct 3, 2018 at 12:14 PM Carlton Gibson 
wrote:

> Thanks for the follow-up Jon.
>
> I'll let Vasili follow-up on his use-case if possible/relevant.
>
> TBH I'm not at all sure about the SQL data files bit, which is in part why
> I asked here.
> (Encoding issues!)
>
> > Maybe that sentence should be rephrased to "template
> files, static files, and translation catalogs".
>
> OK, so IF it's just this, then I'm on Windows doing development in UTF-8
> no problem (and can't really envisage doing much different as it stands)
> but:
>
> * Is that always available these days? (I'd guess yes.)
> * Is is something we want to impose? Not sure. Are there people doing
> otherwise? (No idea.)
>
> (If we can drop a setting, that'd be 💃🏼)
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/13cc5f04-967b-4f53-92f6-cbe155014edc%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CADhq2b4-7vc-hZgKP_%2BYOmUkbRC%2B8r7i2t74d0-9b0vGrVcmkA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.