Re: Automatic prefetching in querysets

2017-08-18 Thread Shai Berger 
On Friday 18 August 2017 09:08:11 Anssi Kääriäinen wrote:
> Maybe we should just add the queryset method. This is the smallest atomic
> task that can be done. Even if there's only the queryset method available,
> it's possible to enable prefetches per model by using a Manager.
> 
I disagree on both counts: I don't think it's the smallest atomic task, and 
I'm not so sure it's the right thing to do.

The smallest atomic task, the way I see it, is building the infrastructure 
that would allow adding the queryset method -- but would also allow different 
APIs to be set around it.

And since there is as yet no consensus on the correct API for "end" users, I 
would rather not define one immediately.

Shai.

Re: Should django File wrapper support .next()?

2017-08-18 Thread Adam Johnson 
I looked a bit more and the reason file objects have next() in Py2 is
because "A file object is its own iterator" (
https://docs.python.org/2/library/stdtypes.html#file.next ). This is still
the case in Py3 - "IOBase (and its subclasses) supports the iterator
protocol" ( https://docs.python.org/3/library/io.html#io.IOBase ).

So yes there is a disparity in Django's File. After a quick look at the
code I'm not sure if it's possible to modify it to match though because of
all the logic embedded in __iter__ but it's worth having a try.

On 16 August 2017 at 13:55, Cristiano Coelho 
wrote:

> I forgot about django dropping support for Python2 so I guess you are
> right this doesn't make much sense if next() is removed, but for Python2 it
> would make sense for django's File object to have the same api as the
> object returned by open().
>
>
> El martes, 15 de agosto de 2017, 19:21:08 (UTC-3), Adam Johnson escribió:
>>
>> The next() method is gone from the return value of open() in Python 3
>> and I don't think it was every necessary. As per the iterator protcool (
>> https://docs.python.org/2/library/stdtypes.html#iterator-types ),
>> __iter__ is meant to return an iterator over the object, and *that* is
>> what the next() method - __next__ on Python 3 - should be attached to.
>> Django's File.__iter__ is a generator function, thus it returns a
>> generator, with comes with a next()/__next__().
>>
>> I think you should just try iterating your files by calling iter() on
>> them first, it's Python 3 compatible too.
>>
>> On 27 July 2017 at 17:24, Cristiano Coelho  wrote:
>>
>>> Hello,
>>>
>>> I have recently found an interesting issue, using a project that relies
>>> on different storage backends, when switching from a custom one to django's
>>> file system storage, I found that existing code that would iterate files
>>> with the next() call would start to fail, since although django's File is
>>> iterable, it doesn't define the next method.
>>>
>>> Now I'm wondering if this is on purpose, or a bug. It's odd that every
>>> stream (or almost) from the python library such as everything from the .io
>>> module or simply the object returned by open() (python 2) supports the next
>>> call but the django File wrapper doesn't.
>>>
>>> This happened with django 1.10 and I believe it wasn't changed with
>>> Django 1.11
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to django-develop...@googlegroups.com.
>>> To post to this group, send email to django-d...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/django-developers.
>>> To view this discussion on the web visit https://groups.google.com/d/ms
>>> gid/django-developers/09ffad76-e473-4593-ac84-4bca7f76e92c%
>>> 40googlegroups.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>>
>> --
>> Adam
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/cb7c4945-5932-44a0-8eb8-
> 4571813a0ba3%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Adam

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAMyDDM0arRQc-5kJiDq1FcxiH_%3DxN04jWGNq4RAp0rxMHSDv2A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Automatic prefetching in querysets

2017-08-18 Thread Collin Anderson 
I like that idea - keep it a private API for now and make it a public API
once people have used it a little bit.

On Fri, Aug 18, 2017 at 4:01 AM, Shai Berger  wrote:

> On Friday 18 August 2017 09:08:11 Anssi Kääriäinen wrote:
> > Maybe we should just add the queryset method. This is the smallest atomic
> > task that can be done. Even if there's only the queryset method
> available,
> > it's possible to enable prefetches per model by using a Manager.
> >
> I disagree on both counts: I don't think it's the smallest atomic task, and
> I'm not so sure it's the right thing to do.
>
> The smallest atomic task, the way I see it, is building the infrastructure
> that would allow adding the queryset method -- but would also allow
> different
> APIs to be set around it.
>
> And since there is as yet no consensus on the correct API for "end" users,
> I
> would rather not define one immediately.
>
> Shai.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFO84S4hduThLVTwN-Y0uqVy48aRRndvMgzmGOb24zTD9g0wmQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Default to BigAutoField

2017-08-18 Thread Markus Holtermann 

Thanks for taking the effort to work on this, Kenneth!

I'm don't fully agree with the approach. This essentially forces 3rd
party package authors to make the call about the primary key field size.
While for small to medium size projects BigAutoField is unlikely
required and only comes with additional (storage) costs. Given that the
migrations would need to be part of the 3rd party package there's also
no (trivial) way for project developers to force or change to
SmallAutoField for those packages. The same thing holds the other way
round.

Unfortunately, I don't have another solution at hand.

I realized that I'm a bit late to the discussion and should've chimed
in before all that work was done. Please accept my apologies for that.

/Markus


On Thu, Aug 17, 2017 at 02:43:07PM -0700, Andrew Godwin wrote:

To elaborate on the solution we eventually came up with - we default models
to use a new BigAutoField that migrations will pick up on and generate
migrations to alter columns to, but for safety reasons for those that don't
read release notes, made the migration autodetector ask you if you want to
make these migrations with a slowness warning.

It also tells you how to preserve the old behaviour and avoid new
migrations if you wish (manually set id = SmallAutoField)

I like this approach as it means no new settings or Meta options or
anything, has a reasonable upgrade path, and won't let people unwittingly
wander into giant changes. The downside is that it does add slightly more
friction to the upgrade process.

Andrew

On Thu, Aug 17, 2017 at 2:36 PM, Kenneth Reitz  wrote:


I have opened a pull request:

https://github.com/django/django/pull/8924

Andrew and I came up with a good solution for migrations, together at
DjangoCon.

On Wednesday, June 14, 2017 at 7:36:36 AM UTC-7, Melvyn Sopacua wrote:


On Friday 09 June 2017 15:59:50 Kenneth Reitz wrote:

> However, it should also be noted that those same larger applications

> are the ones that are likely to run into this problem eventually, so

> perhaps forcing the migration is the best path moving forward.





Existing models are the problem. Then again the database knows the truth.
So with a little inspection during apps.get_models we might be able to do
the right thing and even allow migrating in steps.



Apps is also the place to mark an app as migrated.



In fact - couldn't an AppConfig grow a method "get_autoid_type()" and
inject the right one?



You asked fr thoughts, so there's my 2c stream.

--

Melvyn Sopacua


--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/
msgid/django-developers/e3effc41-10e1-42e2-9037-
84c98217cd91%40googlegroups.com

.

For more options, visit https://groups.google.com/d/optout.



--
You received this message because you are subscribed to the Google Groups "Django 
developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uo4Y_pWSf3zAe_4R0GGkDqBv1YGus8Q%2BWPPZ%3DZ6FPwdYQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "Django 
developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20170818124354.GA1898%40inel.local.
For more options, visit https://groups.google.com/d/optout.

Re: Default to BigAutoField

2017-08-18 Thread Andrew Godwin 
On Fri, Aug 18, 2017 at 5:43 AM, Markus Holtermann  wrote:
>
> I'm don't fully agree with the approach. This essentially forces 3rd
> party package authors to make the call about the primary key field size.
> While for small to medium size projects BigAutoField is unlikely
> required and only comes with additional (storage) costs. Given that the
> migrations would need to be part of the 3rd party package there's also
> no (trivial) way for project developers to force or change to
> SmallAutoField for those packages. The same thing holds the other way
> round.
>
> Unfortunately, I don't have another solution at hand.
>
>
This is also true of changing the primary key of third-party packages in
general though - e.g. there's no way I can make everything use UUIDs even
if my database would be way better at those.

I don't see any other solutions that aren't settings doing
spooky-action-at-a-distance to primary keys, and that's something I really
don't want to see.

Andrew


>
> On Thu, Aug 17, 2017 at 02:43:07PM -0700, Andrew Godwin wrote:
>
>> To elaborate on the solution we eventually came up with - we default
>> models
>> to use a new BigAutoField that migrations will pick up on and generate
>> migrations to alter columns to, but for safety reasons for those that
>> don't
>> read release notes, made the migration autodetector ask you if you want to
>> make these migrations with a slowness warning.
>>
>> It also tells you how to preserve the old behaviour and avoid new
>> migrations if you wish (manually set id = SmallAutoField)
>>
>> I like this approach as it means no new settings or Meta options or
>> anything, has a reasonable upgrade path, and won't let people unwittingly
>> wander into giant changes. The downside is that it does add slightly more
>> friction to the upgrade process.
>>
>> Andrew
>>
>> On Thu, Aug 17, 2017 at 2:36 PM, Kenneth Reitz 
>> wrote:
>>
>> I have opened a pull request:
>>>
>>> https://github.com/django/django/pull/8924
>>>
>>> Andrew and I came up with a good solution for migrations, together at
>>> DjangoCon.
>>>
>>> On Wednesday, June 14, 2017 at 7:36:36 AM UTC-7, Melvyn Sopacua wrote:
>>>

 On Friday 09 June 2017 15:59:50 Kenneth Reitz wrote:

 > However, it should also be noted that those same larger applications

 > are the ones that are likely to run into this problem eventually, so

 > perhaps forcing the migration is the best path moving forward.





 Existing models are the problem. Then again the database knows the
 truth.
 So with a little inspection during apps.get_models we might be able to
 do
 the right thing and even allow migrating in steps.



 Apps is also the place to mark an app as migrated.



 In fact - couldn't an AppConfig grow a method "get_autoid_type()" and
 inject the right one?



 You asked fr thoughts, so there's my 2c stream.

 --

 Melvyn Sopacua

 --
>>> You received this message because you are subscribed to the Google Groups
>>> "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to django-developers+unsubscr...@googlegroups.com.
>>> To post to this group, send email to django-developers@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/django-developers.
>>> To view this discussion on the web visit https://groups.google.com/d/
>>> msgid/django-developers/e3effc41-10e1-42e2-9037-
>>> 84c98217cd91%40googlegroups.com
>>> >> 1-10e1-42e2-9037-84c98217cd91%40googlegroups.com?utm_medium=
>>> email&utm_source=footer>
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers  (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-developers@googlegroups.com.
>> Visit this group at https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit https://groups.google.com/d/ms
>> gid/django-developers/CAFwN1uo4Y_pWSf3zAe_4R0GGkDqBv1YGus8Q%
>> 2BWPPZ%3DZ6FPwdYQ%40mail.gmail.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit