contrib.auth.Group comment

[Mike Dewhirst]

Auth Groups are fabulous for designing access control into a project. 
However, success in such design relies heavily on either user memory or 
documentation. 

In my case it has to be documentation. 

To that end I believe the contrib.auth.Group model needs a comment field. 
That would let business administrators document what each group or role 
might be responsible for and then system admins could assign correct 
permissions accordingly.

The change required in contrib.auth.models.py is a new line inserted in the 
Group class between "name" and "permissions" as follows ...

comment = models.TextField(_("comment"), default="", blank=True)

I see this as a security improvement because human error in assigning group 
permissions can make security holes.

Cheers

Mike

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/434b7bc6-b416-4ef4-ab08-32ad2d31e9ado%40googlegroups.com.

Re: Defaulting to use BigAutoField in models

[Tom Forbes]

I think we should re-open ticket 31007[1] as there was a consensus in 2017 
during the previous attempt at this[2] and things have not really changed - a 
32bit primary key is still not a sensible default given the calamity that can 
occur when you hit the end of it’s range. 

The sticky point is the implementation - how do we migrate new projects whilst 
minimising disruption to existing projects (who either won’t ever hit the 
limit, or have too much data to migrate wholesale). I think special casing the 
migrations framework is an avenue to explore, so I created this today and to my 
surprise it seems to work quite well: 
https://github.com/orf/django/commit/0a335f208cee1376c25eff55c6f866de122c7839 
.

The obvious effect of this is that for old projects the Django model thinks its 
primary key is 64 bit while the underlying table is only 32 bit. I can’t think 
of much practical impact of this approach: 
- Python 3 doesn’t distinguish between 32/64 bit numbers so code will continue 
to work as-is. 
- Third party migrations that ship with 32 bit columns won’t be impacted, and 
it’s up to them to create migrations if they want to. 
- Adding `id = BigAutoField(primary_key=True)` to a model will create an 
explicit AlterField migration, when projects are able to migrate.

Am I missing something?

> I haven't thought this through, but maybe it's possible to restrict the 
> change to inside AutoField, rather than create a new field class. If its 
> db_parameters() method / db_type() could receive enough context from the 
> migration history, it could potentially determine its own database type.

I’m really not sure how we would do that: wouldn’t it require a pretty big 
refactor? It would definitely be cleaner.

1. https://code.djangoproject.com/ticket/31007 

2. 
https://groups.google.com/forum/#!msg/django-developers/imBJwRrtJkk/P4g0Y87lAgAJ
 



> On 12 Jun 2020, at 12:11, Adam Johnson  wrote:
> 
> I haven't thought this through, but maybe it's possible to restrict the 
> change to inside AutoField, rather than create a new field class. If its 
> db_parameters() method / db_type() could receive enough context from the 
> migration history, it could potentially determine its own database type. For 
> those who want to fixed field sizes, BigAutoField and SmallAutoField could be 
> left, and MediumAutoField added.
> 
> (Whilst poking around I think I spotted a small cleanup: 
> https://code.djangoproject.com/ticket/31698 
>  )
> 
> On Fri, 12 Jun 2020 at 11:40, Tom Forbes  > wrote:
>> I think we should restrict the setting between normal and big auto fields 
>> only. Allowing UUID's would be changing the type, with the potential for 
>> havoc with code incompalities throughout django. It's also not possible to 
>> migrate tables over to the new type.
> 
> That’s a really good point, I got a bit swept up with the idea to realise 
> that it would break a lot of things if applied generally!
> 
>> The autodetector knows if a model is new. It could be that during one 
>> version Django outputs BigAutoField for fields added in CreateModel only.
> 
> We could make every automatic PK become a BigAutoField but include some logic 
> in the migrations framework to ignore changes between an auto_created 
> AutoField -> BigAutoField? This would ignore all existing models in your 
> migration history, but all completely new models would receive BigAutoField 
> PKs from the start. Users who explicitly add a `BigAutoField` to a model with 
> a previously created `AutoField` would see a migration created as the 
> `auto_created` flag is not set. It would also play nicely with foreign keys 
> which also need to be 8 bytes.
> 
> This would take care of third party apps with migrations and not require a 
> setting but the downside is that model state will be slightly different from 
> the database. All models would have `BigAutoField`’s whereas the database 
> would only have an `AutoField`. This feels slightly iffy to me even though I 
> cannot think of a case where it would have any practical effect.
> 
>> On 11 Jun 2020, at 19:22, Adam Johnson > > wrote:
>> 
>> Big +1 on solving this from me.
>> 
>> - The setting would take any dotted path to a class, or a single class name 
>> for a build in field. This would potentially solve [3], and could be useful 
>> to people who want to default to other fields like UUIDs (or a custom 
>> BigAutoField) for whatever reason
>> 
>> I think we should restrict the setting between normal and big auto fields 
>> only. Allowing UUID's would be changing the type, with the potential for 
>> havoc with code incompalities throughout django. It's also not possible to 
>> migrate tables over to the new type.

Proposal: Drop dependency on pytz in favor of zoneinfo

[Paul Ganssle]

Greetings all,

Now that PEP 495  (the fold
attribute, added in Python 3.6) and PEP 615
 (the zoneinfo module, added
in Python 3.9) have been accepted, there's not much reason to continue
using pytz, and its non-standard interface is a major source of bugs
. In
fact, the creator and maintainer of pytz said during the PEP discussions
that he was looking forward to deprecating pytz
.
After merging zoneinfo upstream into CPython, I turned the reference
implementation into a backport to Python 3.6+
, which I continue to maintain
alongside the upstream zoneinfo module, so it is already possible to
migrate Django /today/.

Right now, Django continues to use pytz as its source of time zones,
even for simple zones like UTC; I would like to propose starting the
migration to zoneinfo /now/, to enable Django users to start doing their
own migrations, and so there can be ample time for proper deprecation
cycles. I apologize in advance for the huge wall of text, the TL;DR
summary is:

  * It's important to start this migration as soon as possible, because
it will take a long time and some of the bugs it fixes get worse as
time goes on; right now Django's assumption that all time zones are
from pytz will likely block Django users from migrating before
Django does.
  * The combination of pytz's non-standard interface and Hyrum's Law
 makes this less straightforward than
one would hope. Likely the best solution is to adopt "zoneinfo"
immediately with a wrapper that deprecates the pytz-specific
interface. I have provided a library to do this
.
  * There are some open questions (which may only be open because I
don't know Django's process well enough) in the migration plan:
  o Should this change be put under a feature flag of some sort? If
so, what should be the default values for coming releases?
  o Should /all/ the pytz-specific tests be kept with zoneinfo tests
added, or should they be migrated and pytz tests confined to a
smaller subset?

*Rationale*

Before I get into the detailed migration plan, I'd like to lay out the
case for /why/ this needs to be done. I expect it to be at least
somewhat painful, but and I'd prefer it if y'all were convinced that
it's the right thing to do /before/ you hear about the costs 😉. I am
not the kind of person who thinks that just because something is in the
standard library it is automatically "better" than what's available on
PyPI, but in the case of pytz, I've been recommending people move away
from it from a long time, because most people don't know how to use it
correctly. The issue is that pytz was originally designed to work around
issues with ambiguous and imaginary times that were fixed by PEP 495 in
Python 3.6, and the compromise it made for correctness is that it is not
really compatible with the standard tzinfo interface. That's why you
aren't supposed to directly attach a timezone with the tzinfo argument,
and why datetime arithmetic requires normalization. Any substantial code
base I've ever seen that uses pytz either has bugs related to this or
had a bunch of bugs related to this until someone noticed the problem
and did some sort of large-scale change to fix all the bugs, then
imposed some sort of linting rules.

In addition to the "pytz is hard to use" problems (which, honestly,
should probably be enough), pytz also has a few additional issues that
the maintainer has said will not be fixed (unless pytz becomes a thin
wrapper around zoneinfo, which is at best just using a slower version of
zoneinfo). The biggest issue, which is going to become increasingly
relevant in the coming years, is that it only supports the Version 1
spec in TZif files, which (among other issues), does not support
datetimes after 2038 (or before 1902) — this was deprecated 15 years
ago, and it is unlikely that you will find modern TZif files that don't
have Version 2+ data. Pytz also does not support sub-minute offsets,
which is mostly relevant only in historical time zones. And, of course,
pytz is not compatible with PEP 495 and in some ways really cannot be
made compatible with PEP 495 (at least not easily).

Of course, one reasonable objection here is that Django is a huge
install base with very long release cycles, and it doesn't make sense
for Django to be an experimental "early adopter" of the new library.
This is a reasonable response, but it's actually /because/ it has a huge
install base and long release cycles that it's important for Django to
migrate early, because it can't "turn on a dime" like that. The long
release cycles mean that changes 

HttpResponse headers interface

[Tom Carrick]

I don't find myself using HttpResponse very often, usually I'm using DRF's
Responses. But today I needed to use one, and as I was writing tests, I
ended up somewhat astonished, so with the principle of least astonishment
in mind... I had anticipated that I could check the headers with
`response.headers`, similar to how the new request.headers
 works, but apparently this is
not the case. After reading the docs, I found out that I should just treat
the HttpResponse object itself as if it were a dictionary of headers. This
seems very strange to me, it's not what I expect, but maybe I'm in the
minority.

I have no interest in deprecating the old API, but it would be nice if the
headers were all accessible from a simple headers dict, and perhaps make
this the source of truth, allowing access with any casing but preserving
the original casing for output. It looks like what is currently
HttpResponse._headers was once HttpRequest.headers, but this was 13 years
ago , I don't think it'd be
confusing to add the property back as something different.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAHoz%3DMa9-m%2Bfqj0wqzQ7qW5Aiw3POHtNOp2NTBaHeP_ux5FhLg%40mail.gmail.com.

Re: Proposal: Drop dependency on pytz in favor of zoneinfo

[Ryan Hiebert]

I'm almost exclusively a lurker on this list, but a constant user, and have
in the past a keen observer in the datetime discussions.

I think you've made your case well. I'd be happy for this migration from
Django to be as aggressive as the maintainers are comfortable. I agree that
doing step 1 at a minimum seems like the right thing to me. I'm very
interested in using zoneinfo over pytz for my projects, and I'm willing to
shoulder more risk in order to do so. I hope Django will allow me to make
that choice.

On Wed, Jun 17, 2020 at 9:32 AM Paul Ganssle  wrote:

> Greetings all,
>
> Now that PEP 495  (the fold
> attribute, added in Python 3.6) and PEP 615
>  (the zoneinfo module, added
> in Python 3.9) have been accepted, there's not much reason to continue
> using pytz, and its non-standard interface is a major source of bugs
> . In
> fact, the creator and maintainer of pytz said during the PEP discussions
> that he was looking forward to deprecating pytz
> .
> After merging zoneinfo upstream into CPython, I turned the reference
> implementation into a backport to Python 3.6+
> , which I continue to maintain
> alongside the upstream zoneinfo module, so it is already possible to
> migrate Django *today*.
>
> Right now, Django continues to use pytz as its source of time zones, even
> for simple zones like UTC; I would like to propose starting the migration
> to zoneinfo *now*, to enable Django users to start doing their own
> migrations, and so there can be ample time for proper deprecation cycles. I
> apologize in advance for the huge wall of text, the TL;DR summary is:
>
>- It's important to start this migration as soon as possible, because
>it will take a long time and some of the bugs it fixes get worse as time
>goes on; right now Django's assumption that all time zones are from pytz
>will likely block Django users from migrating before Django does.
>- The combination of pytz's non-standard interface and Hyrum's Law
> makes this less straightforward than one
>would hope. Likely the best solution is to adopt "zoneinfo" immediately
>with a wrapper that deprecates the pytz-specific interface. I have provided
>a library to do this .
>- There are some open questions (which may only be open because I
>don't know Django's process well enough) in the migration plan:
>   - Should this change be put under a feature flag of some sort? If
>   so, what should be the default values for coming releases?
>   - Should *all* the pytz-specific tests be kept with zoneinfo tests
>   added, or should they be migrated and pytz tests confined to a smaller
>   subset?
>
> *Rationale*
>
> Before I get into the detailed migration plan, I'd like to lay out the
> case for *why* this needs to be done. I expect it to be at least somewhat
> painful, but and I'd prefer it if y'all were convinced that it's the right
> thing to do *before* you hear about the costs 😉. I am not the kind of
> person who thinks that just because something is in the standard library it
> is automatically "better" than what's available on PyPI, but in the case of
> pytz, I've been recommending people move away from it from a long time,
> because most people don't know how to use it correctly. The issue is that
> pytz was originally designed to work around issues with ambiguous and
> imaginary times that were fixed by PEP 495 in Python 3.6, and the
> compromise it made for correctness is that it is not really compatible with
> the standard tzinfo interface. That's why you aren't supposed to directly
> attach a timezone with the tzinfo argument, and why datetime arithmetic
> requires normalization. Any substantial code base I've ever seen that uses
> pytz either has bugs related to this or had a bunch of bugs related to this
> until someone noticed the problem and did some sort of large-scale change
> to fix all the bugs, then imposed some sort of linting rules.
>
> In addition to the "pytz is hard to use" problems (which, honestly, should
> probably be enough), pytz also has a few additional issues that the
> maintainer has said will not be fixed (unless pytz becomes a thin wrapper
> around zoneinfo, which is at best just using a slower version of zoneinfo).
> The biggest issue, which is going to become increasingly relevant in the
> coming years, is that it only supports the Version 1 spec in TZif files,
> which (among other issues), does not support datetimes after 2038 (or
> before 1902) — this was deprecated 15 years ago, and it is unlikely that
> you will find modern TZif files that don't have Version 2+ data. Pytz also
>

Re: HttpResponse headers interface

[Adam Johnson]

I have also found this a little odd when writing tests. It would certainly
make it easier to write both normal Django code and tests, and it's a small
addition, so +1 from me.

On Wed, 17 Jun 2020 at 15:35, Tom Carrick  wrote:

> I don't find myself using HttpResponse very often, usually I'm using DRF's
> Responses. But today I needed to use one, and as I was writing tests, I
> ended up somewhat astonished, so with the principle of least astonishment
> in mind... I had anticipated that I could check the headers with
> `response.headers`, similar to how the new request.headers
>  works, but apparently this
> is not the case. After reading the docs, I found out that I should just
> treat the HttpResponse object itself as if it were a dictionary of headers.
> This seems very strange to me, it's not what I expect, but maybe I'm in the
> minority.
>
> I have no interest in deprecating the old API, but it would be nice if the
> headers were all accessible from a simple headers dict, and perhaps make
> this the source of truth, allowing access with any casing but preserving
> the original casing for output. It looks like what is currently
> HttpResponse._headers was once HttpRequest.headers, but this was 13 years
> ago , I don't think it'd be
> confusing to add the property back as something different.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CAHoz%3DMa9-m%2Bfqj0wqzQ7qW5Aiw3POHtNOp2NTBaHeP_ux5FhLg%40mail.gmail.com
> 
> .
>


-- 
Adam

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAMyDDM2HCJKbW%3D3dNkB7BDV4R6eSAGokg%3DPENFd%3D%2BhNwaTt9OQ%40mail.gmail.com.

Re: Defaulting to use BigAutoField in models

[Adam Johnson]

>
> I think special casing the migrations framework is an avenue to explore,
> so I created this today and to my surprise it seems to work quite well:
> https://github.com/orf/django/commit/0a335f208cee1376c25eff55c6f866de122c7839
> .


That is a pretty neat solution. I think it would work quite well.

What about a way to warn users approaching the limit? I think a database
check that detects AutoFields where the next ID is >50% of the way through
would be a good solution. That's the kind of monitoring I've built in the
past.

(50% might sound like it's near the limit, but the table is growing with an
exponential curve, even quite shallow, it's quite close in terms of time.)

I’m really not sure how we would do that: wouldn’t it require a pretty big
> refactor? It would definitely be cleaner.


Yeah I haven't thought it through. Given the small size of your patch and
the lack of expectation for any similar changes, I think special casing the
migrations framework is a fair tradeoff.

On Wed, 17 Jun 2020 at 13:51, Tom Forbes  wrote:

> I think we should re-open ticket 31007[1] as there was a consensus in 2017
> during the previous attempt at this[2] and things have not really changed -
> a 32bit primary key is still not a sensible default given the calamity that
> can occur when you hit the end of it’s range.
>
> The sticky point is the implementation - how do we migrate new projects
> whilst minimising disruption to existing projects (who either won’t ever
> hit the limit, or have too much data to migrate wholesale). I think special
> casing the migrations framework is an avenue to explore, so I created this
> today and to my surprise it seems to work quite well:
> https://github.com/orf/django/commit/0a335f208cee1376c25eff55c6f866de122c7839
> .
>
> The obvious effect of this is that for old projects the Django model
> thinks its primary key is 64 bit while the underlying table is only 32 bit.
> I can’t think of much practical impact of this approach:
> - Python 3 doesn’t distinguish between 32/64 bit numbers so code will
> continue to work as-is.
> - Third party migrations that ship with 32 bit columns won’t be impacted,
> and it’s up to them to create migrations if they want to.
> - Adding `id = BigAutoField(primary_key=True)` to a model will create an
> explicit AlterField migration, when projects are able to migrate.
>
> Am I missing something?
>
> I haven't thought this through, but maybe it's possible to restrict the
> change to inside AutoField, rather than create a new field class. If its
> db_parameters() method / db_type() could receive enough context from the
> migration history, it could potentially determine its own database type.
>
>
> I’m really not sure how we would do that: wouldn’t it require a pretty big
> refactor? It would definitely be cleaner.
>
> 1. https://code.djangoproject.com/ticket/31007
> 2.
> https://groups.google.com/forum/#!msg/django-developers/imBJwRrtJkk/P4g0Y87lAgAJ
>
>
> On 12 Jun 2020, at 12:11, Adam Johnson  wrote:
>
> I haven't thought this through, but maybe it's possible to restrict the
> change to inside AutoField, rather than create a new field class. If its
> db_parameters() method / db_type() could receive enough context from the
> migration history, it could potentially determine its own database type.
> For those who want to fixed field sizes, BigAutoField and SmallAutoField
> could be left, and MediumAutoField added.
>
> (Whilst poking around I think I spotted a small cleanup:
> https://code.djangoproject.com/ticket/31698 )
>
> On Fri, 12 Jun 2020 at 11:40, Tom Forbes  wrote:
>
>> I think we should restrict the setting between normal and big auto fields
>> only. Allowing UUID's would be changing the type, with the potential for
>> havoc with code incompalities throughout django. It's also not possible to
>> migrate tables over to the new type.
>>
>>
>> That’s a really good point, I got a bit swept up with the idea to realise
>> that it would break a lot of things if applied generally!
>>
>> The autodetector knows if a model is new. It could be that during one
>> version Django outputs BigAutoField for fields added in CreateModel only.
>>
>>
>> We could make every automatic PK become a BigAutoField but include some
>> logic in the migrations framework to *ignore* changes between an
>> auto_created AutoField -> BigAutoField? This would ignore all existing
>> models in your migration history, but all completely new models would
>> receive BigAutoField PKs from the start. Users who explicitly add a
>> `BigAutoField` to a model with a previously created `AutoField` would see a
>> migration created as the `auto_created` flag is not set. It would also play
>> nicely with foreign keys which also need to be 8 bytes.
>>
>> This would take care of third party apps with migrations and not require
>> a setting but the downside is that model state will be slightly different
>> from the database. All models would have `BigAutoField`’s whereas the
>> database would only hav