Re: Fellow Reports - December 2020

2020-12-16 Thread Mariusz Felisiak
Week ending December 13, 2020. 

*Triaged: *
   https://code.djangoproject.com/ticket/32245 - Replacement for 
./manage.py test --liveserver? (invalid) 
   https://code.djangoproject.com/ticket/32247 - "App does not have 
migrations" message could be more helpful (invalid) 
   https://code.djangoproject.com/ticket/32236 - LoginRequiredMixin and 
PermissionRequiredMixin could use UserPassesTestMixin (wontfix) 
   https://code.djangoproject.com/ticket/32249 - Increment of duration 
field not works properly (duplicate) 
   https://code.djangoproject.com/ticket/32248 - QuerySet.get() leaves 
postgres in 'current transaction is aborted...' state if invalid data is 
passed. (invalid) 
   https://code.djangoproject.com/ticket/32250 - Typo in tutorial 04 
(invalid) 
   https://code.djangoproject.com/ticket/32251 - GEOSGeometry error logging 
when handling a GEOSException (invalid) 
   https://code.djangoproject.com/ticket/32253 - Tuple passed instead of 
string in warnings.warn in validate_key method in base.py (duplicate) 
   https://code.djangoproject.com/ticket/32252 - Using __isnull=True on a 
KeyTransform should not match JSON null on SQLite and Oracle (accepted) 
   https://code.djangoproject.com/ticket/32254 - Support Django Database 
Password Rotation (duplicate) 
   https://code.djangoproject.com/ticket/32257 - Django does not close 
persistent connections on server shutdown (needsinfo) 
   https://code.djangoproject.com/ticket/32258 - Documentation incorrect 
for HttpResponse (invalid) 
   https://code.djangoproject.com/ticket/32256 - squashmigrations optimizer 
crashes when fields' names are swapped using a temporary name (accepted) 

*Reviewed/committed: *
   https://github.com/django/django/pull/13749 - Removed lhs_only argument 
from KeyTransform.preprocess_lhs(). 
   https://github.com/django/django/pull/13753 - Refs #21012 -- Removed 
unnecessary _create_cache() hook. 
   https://github.com/django/django/pull/9272 - Fixed #32233 -- Cleaned-up 
duplicate connection functionality. 
   https://github.com/django/django/pull/13717 - Fixed #32178 -- Allowed 
database backends to skip tests and mark expected failures. 
   https://github.com/django/django/pull/13762 - Refs #31792 -- Updated SQL 
example in Exists() docs. 
   https://github.com/django/django/pull/13757 - Fixed #32252 -- Fixed 
__isnull=True on key transforms on SQLite and Oracle. 
   https://github.com/django/django/pull/13765 - Refs #32178 -- Fixed 
test_mark_expected_failures_and_skips_call teardown. 
   https://github.com/django/django/pull/13759 - Corrected docs regarding 
attributes required for logging in to the admin. 

*Authored: *
   https://github.com/django/django/pull/13758 - Fixed isolation of 
test_runner.EmptyDefaultDatabaseTest.

Best,
Mariusz

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/2c5d3612-f2ac-4a9d-98f6-46b330ccd04cn%40googlegroups.com.


Re: Issues and Improvements to the The Django Administration

2020-12-16 Thread Carles Pina i Estany


Hi,

On Dec/16/2020, Muskan Vaswan wrote:

> Carles Pina I Estany: While I understand what you’re trying to do I
> don’t know why we are trying to avoid COUNT is there a best practice
> associated with this that I don’t know about?

We are (or... I was :-)   ) trying to answer this question as close as
to the database:
-"Are more than X number of rows?"

With COUNT we would answer:
-"How many rows are there?"

Depends on the model, indexes, data in the DB and database the COUNT
operation can take longer than the other way to know if there are more
than X rows. It needs proper checking.

It's more or less what's explained here:
https://docs.datasette.io/en/stable/sql_queries.html#pagination 
but I was thinking in terms of a Django ORM

Sorry if this was a detour on the higher level discussion. My message is
that it probably can be done without COUNT if need to be and show "more
than X rows" instead of the exact number of rows.

Cheers,

-- 
Carles Pina i Estany
https://carles.pina.cat

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20201216084703.GA17389%40pina.cat.


Re: Issues and Improvements to the The Django Administration

2020-12-16 Thread Muskan Vaswan
On Dec 16, 2020, at 2:17 PM, Carles Pina i Estany  wrote:

My message is
that it probably can be done without COUNT if need to be and show "more
than X rows" instead of the exact number of rows.


Well yes this makes sense to me now,
but only in the sense that we would first check whether some query exceeded
the limit or not and only count the exact amount of objects if it's under
the limit.
But even so if you consider the actual implementation, you'd have to make
two queries: one for all the distinct groups of timestamps and action tags
and run a loop on them and so on... This is difficult to explain in words
but basically it would add a lot more queries this way... at least the way
I visualize it happening here.

But also this is something I didn't know about so this was quite
educational, thank you.

Regards
Muskan Vaswan

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAOoM%2B4Nwhzkr2YxfO%3DMrykYKB9q6jHtiP7oFUz%3DEEX4_jseqeg%40mail.gmail.com.


Unapplying migrations in a branch

2020-12-16 Thread Roman Odaisky
Hi All,

I would like to solicit some further comments regarding #32267. The main 
question is:

Should Django support unapplying migrations in a branch?

Typical use case:
$ git switch otherdeveloper/feature
$ manage.py migrate
Applying app.0051_migration_from_feature... OK
Applying app.0052_migration_from_feature... OK
Applying app.0053_merge_master... OK
Applying app.0054_migration_from_feature... OK
$ do some work on the feature branch
$ manage.py migrate ???
$ git switch master

Despite there being a very clear path from the feature branch to the master 
branch, namely, to reverse the migrations that have just been applied, 
there’s no way to tell Django to do that. manage.py migrate app 
0051_migration_from_feature will unapply too little (it will keep 0051 
itself applied) and manage.py migrate app 0050_migration_from_master will 
unapply too much (specifically, all the subsequent migrations from 
master).There’s only this extremely hacky workaround:

migrate  
migrate --fake  
migrate --fake  
migrate  
migrate --fake  

Reversing the migrations does not present a potential for DB state 
corruption because already existing Django mechanisms prevent unapplying a 
migration if its children are still applied. Indeed, Django will happily 
reverse the feature branch migration chain... all except the oldest 
migration. 

What are the possible pitfalls with allowing Django to reverse a migration 
chain, starting with most recent applied migration and ending with a 
certain ancestor of that migration?

The ticket suggests a problem in case someone reverses a migration chain, 
deletes migration files and proceeds with committing new work to that 
branch. This is indeed an error-prone approach that should not be 
supported, for the primary reason that someone who had checked out that 
branch earlier and applied its migrations would be stuck. However, what’s 
wrong with the possibility of simply switching from that branch without 
ever deleting migration files?

What concerns are there pertaining to a possibility of migrating back to a 
node that has multiple children (currently unsupported) that do not apply 
to the possibility of migrating back to a node that has only one child 
(currently supported)?

Ticket: https://code.djangoproject.com/ticket/32267
PR: https://github.com/django/django/pull/13781

-- 
WBR
Roman.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/85a20b24-3f91-412d-9bce-664ba835d576n%40googlegroups.com.


Re: Unapplying migrations in a branch

2020-12-16 Thread Adam Johnson
>
> manage.py migrate app 0050_migration_from_master will unapply too much
> (specifically, all the subsequent migrations from master)


I don't follow you. This is exactly the thing I do, migrating back to the
last common migration.

On Wed, 16 Dec 2020 at 17:21, Roman Odaisky  wrote:

> Hi All,
>
> I would like to solicit some further comments regarding #32267. The main
> question is:
>
> Should Django support unapplying migrations in a branch?
>
> Typical use case:
> $ git switch otherdeveloper/feature
> $ manage.py migrate
> Applying app.0051_migration_from_feature... OK
> Applying app.0052_migration_from_feature... OK
> Applying app.0053_merge_master... OK
> Applying app.0054_migration_from_feature... OK
> $ do some work on the feature branch
> $ manage.py migrate ???
> $ git switch master
>
> Despite there being a very clear path from the feature branch to the
> master branch, namely, to reverse the migrations that have just been
> applied, there’s no way to tell Django to do that. manage.py migrate app
> 0051_migration_from_feature will unapply too little (it will keep 0051
> itself applied) and manage.py migrate app 0050_migration_from_master will
> unapply too much (specifically, all the subsequent migrations from
> master).There’s only this extremely hacky workaround:
>
> migrate  
> migrate --fake  
> migrate --fake  
> migrate  
> migrate --fake  
>
> Reversing the migrations does not present a potential for DB state
> corruption because already existing Django mechanisms prevent unapplying a
> migration if its children are still applied. Indeed, Django will happily
> reverse the feature branch migration chain... all except the oldest
> migration.
>
> What are the possible pitfalls with allowing Django to reverse a migration
> chain, starting with most recent applied migration and ending with a
> certain ancestor of that migration?
>
> The ticket suggests a problem in case someone reverses a migration chain,
> deletes migration files and proceeds with committing new work to that
> branch. This is indeed an error-prone approach that should not be
> supported, for the primary reason that someone who had checked out that
> branch earlier and applied its migrations would be stuck. However, what’s
> wrong with the possibility of simply switching from that branch without
> ever deleting migration files?
>
> What concerns are there pertaining to a possibility of migrating back to a
> node that has multiple children (currently unsupported) that do not apply
> to the possibility of migrating back to a node that has only one child
> (currently supported)?
>
> Ticket: https://code.djangoproject.com/ticket/32267
> PR: https://github.com/django/django/pull/13781
>
> --
> WBR
> Roman.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/85a20b24-3f91-412d-9bce-664ba835d576n%40googlegroups.com
> 
> .
>


-- 
Adam

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAMyDDM0Ru3JHrbeDJPN5tNxeSAqhNCL3ODH4RH27DxOyi_eR6A%40mail.gmail.com.


Re: Unapplying migrations in a branch

2020-12-16 Thread Roman Odaisky
On Wednesday, 16 December 2020 23:38:55 EET Adam Johnson wrote:
> > manage.py migrate app 0050_migration_from_master will unapply too much
> > (specifically, all the subsequent migrations from master)
> 
> I don't follow you. This is exactly the thing I do, migrating back to the
> last common migration.

But the problem with this is, this will unnecessarily unapply and reapply all 
the master migrations, and it might not even be possible if some of those are 
non-reversible, or only reversible with data loss.


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/1746391.WnEzt6Ug8H%40xps.


Re: Unapplying migrations in a branch

2020-12-16 Thread James Bennett
On Wed, Dec 16, 2020 at 1:39 PM Adam Johnson  wrote:
>> manage.py migrate app 0050_migration_from_master will unapply too much 
>> (specifically, all the subsequent migrations from master)
>
>
> I don't follow you. This is exactly the thing I do, migrating back to the 
> last common migration.

I think the issue being described is if other apps have migrations
that *should* be applied in the main branch, but would be unapplied by
migrating back to app/0050. So migrating to app/0050 does not actually
restore the state the main branch expected.

I'm also not sure this is a thing that Django can automate in any way
that's likely to be reliable; too many migrations in large projects
have complex interdependencies, a lot of migrations aren't reversible,
etc.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAL13Cg_dC5Fp%2BCu6xe%2BAzR-xdzwrp96LTZ8SWBwsOt%3DgrObtyg%40mail.gmail.com.


Re: Unapplying migrations in a branch

2020-12-16 Thread Roman Odaisky
On Thursday, 17 December 2020 02:28:02 EET James Bennett wrote:
> On Wed, Dec 16, 2020 at 1:39 PM Adam Johnson  wrote:
> >> manage.py migrate app 0050_migration_from_master will unapply too much
> >> (specifically, all the subsequent migrations from master)> 
> > I don't follow you. This is exactly the thing I do, migrating back to the
> > last common migration.
> I think the issue being described is if other apps have migrations
> that *should* be applied in the main branch, but would be unapplied by
> migrating back to app/0050. So migrating to app/0050 does not actually
> restore the state the main branch expected.

Suppose all these are currently applied:

master ...---0050---0051a---0052a
 \   \
feature   0051b---0052b---0053m---0054b

The issue is about unapplying 0054b, 0053m, 0052b and 0051b while not touching 
the master migrations. Let’s ignore possible dependencies from other apps for 
now.

> I'm also not sure this is a thing that Django can automate in any way
> that's likely to be reliable; too many migrations in large projects
> have complex interdependencies, a lot of migrations aren't reversible,
> etc.

Django will happily reverse all the way to 0051b without giving any thought to 
the above concern. The only thing it won’t do is unapply that one last 
migration. Why can’t we add that one step?


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/2075285.l7tNb9eccj%40xps.


Re: Unapplying migrations in a branch

2020-12-16 Thread Adam Johnson
Ah right, thanks for elaborating. I normally stick with a single app and a
linear migration history.  Avoiding merge migrations is good for a number
of reasons and avoids the situation your diagram covers - see my recent
package django-linear-migrations which help with that:
https://adamj.eu/tech/2020/12/10/introducing-django-linear-migrations/ .

In general though app interdependencies and irreversible migrations
complicate this, as James says.

There’s a tool for using DB “snapshots” locally to save/restore DB state
when switching branches - it’s called stellar:
https://github.com/fastmonkeys/stellar . I think this is the only way to be
reliable. If we were going to make something like this that is Django
specific, we’d normally want to have a working, battle-tested third party
implementation before considering merging it to core.

On Thu, 17 Dec 2020 at 00:36, Roman Odaisky  wrote:

> On Thursday, 17 December 2020 02:28:02 EET James Bennett wrote:
> > On Wed, Dec 16, 2020 at 1:39 PM Adam Johnson  wrote:
> > >> manage.py migrate app 0050_migration_from_master will unapply too much
> > >> (specifically, all the subsequent migrations from master)>
> > > I don't follow you. This is exactly the thing I do, migrating back to
> the
> > > last common migration.
> > I think the issue being described is if other apps have migrations
> > that *should* be applied in the main branch, but would be unapplied by
> > migrating back to app/0050. So migrating to app/0050 does not actually
> > restore the state the main branch expected.
>
> Suppose all these are currently applied:
>
> master ...---0050---0051a---0052a
>  \   \
> feature   0051b---0052b---0053m---0054b
>
> The issue is about unapplying 0054b, 0053m, 0052b and 0051b while not
> touching
> the master migrations. Let’s ignore possible dependencies from other apps
> for
> now.
>
> > I'm also not sure this is a thing that Django can automate in any way
> > that's likely to be reliable; too many migrations in large projects
> > have complex interdependencies, a lot of migrations aren't reversible,
> > etc.
>
> Django will happily reverse all the way to 0051b without giving any
> thought to
> the above concern. The only thing it won’t do is unapply that one last
> migration. Why can’t we add that one step?
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/2075285.l7tNb9eccj%40xps
> .
>
-- 
Adam

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAMyDDM0%2BNJCnxSLxhJQ%2ByfEykVHaq3ez4Onnh1-sUme3CojsAg%40mail.gmail.com.