Re: Threads and db connection handling question

[Marcin Nowak]

On Saturday, June 4, 2016 at 12:12:42 AM UTC+2, Cristiano Coelho wrote:
>
> Aymeric, I have never said anything about connection pool, I'm talking 
> about thread pooling to perform async work
>

I have similar requirements and issues with Django. It looks like it is 
completely unreliable in async environment, and the worst thing is that I 
can't force Django to create new connection within the thread/process. I'm 
fighting with this almost day by day, enssuring that no db operation is 
performed in asyncs callbacks, and I'm using ram to return results to main 
thread/process and then write them to db. 

Well, they'll say that Django is not designed for that cases. Maybe. My 
advise - don't use Django for medium and big projects. Using Django for 
that job was my biggest mistake.   From version to version it turns more 
into handy tool for blog creators/webdesigners, forcing everyone else to 
(for example) do nasty things with your database ("I'm a Django and I own 
your database, ha ha!").   

Marcin

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/3e6c1b86-fbb8-466f-9fee-c54363bf8ef1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Threads and db connection handling question

[Marcin Nowak]

 

> From version to version it turns more into handy tool for blog 
> creators/webdesigners
>

Don't get me wrong. For smal/short-term projects (up to 1-2yr of operation) 
I'm still using Django and I would recommend it everyone for that kind of 
job. 
For long-term projects the first thing that will kill you is a db migration 
system (due to unnecessary python dependencies inside each migration file).

Marcin 
  

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/52d52fa9-56f7-47f1-ae8f-68ef632d5d9b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: decorator_from_middleware changes

[Tobias McNulty]

On Mon, Jun 20, 2016 at 10:51 AM, Carl Meyer  wrote
> Possible disadvantages of this approach:



> 3. The new behavior may surprise some people accustomed to the old
> behavior, since it means the effect of the middleware decorator won't
> occur immediately where the decorator is applied. E.g. if you apply a
> middleware decorator to a view, and then another custom view decorator
> outside that, the custom view decorator won't see the effects of the
> middleware decorator in the actual response (it would only see a new
> entry in view_func._extra_middleware).

This one seems like the gotcha to me. What if, for example, I have a view
decorator whose effects I specifically don't want to cache, but I do want
to cache the underlying view; is it fair to require that the person write a
middleware and then turn it into a decorator to be able to do that? Are we
effectively saying, "for view decorators to behave the way you might
expect, implement them as middleware"? It seems odd, to me at least, that I
should care what the underlying implementation of a decorator is before I
use it. It also violates the 'strict in/out layering' goal, albeit for
decorators instead of middleware. (A similar argument could be said of
exceptions -- what if I want to trap an exception raised by a
middleware-turned-decorator?)

It might be okay if the decorators themselves were explicit about what they
were doing, for example @cache_page(3600) might become:

@add_middleware(CacheMiddleware, cache_timeout=3600)

However, that's probably a bigger and unnecessary change.

Would it be possible to implement a combination of the approaches, i.e.,
make the delay in applying the middleware-turned-decorator the exception
rather than the rule, perhaps only for TemplateResponses and specifically
for the purpose of supporting a deprecation plan? And then, long-term,
leave it up to middleware/decorator authors & users to decide how best to
implement/layer them, being explicit about the implications of rendering
the response or perhaps more appropriately, "not rendering if you can avoid
it" (i.e., your first strategy)?

Tobias
-- 

Tobias McNulty
Chief Executive Officer

tob...@caktusgroup.com
www.caktusgroup.com

Sent from my mobile.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAMGFDKT7CuRLaS2Z%2BKA-_pSzBOXXGBhj_J1yD9ORe8stwX%3DBGQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Threads and db connection handling question

[Aymeric Augustin]

On 21 Jun 2016, at 12:53, Marcin Nowak  wrote:

> It looks like it is completely unreliable in async environment, and the worst 
> thing is that I can't force Django to create new connection within the 
> thread/process.
> 
> Well, they'll say that Django is not designed for that cases. Maybe. My 
> advise - don't use Django for medium and big projects.

Hello Marcin,

I fail to see how you get from “I’m having trouble using an async framework 
with Django” to “Django only works for small projects”. Indeed, using Django in 
an “async” context requires special care, until channels land — quotes around 
“async” because I don’t know whether you’re talking about gevent or asyncio or 
something else.

However, I know multiple Django-based projects containing millions of lines of 
codes that took hundreds of man.years to build, maintained by dozens of 
developers. That works fine. There’s no fundamental reason making Django 
unsuitable for such projects. It tends to scale better than other technologies 
thanks to strong conventions and limited use of magic.

Regarding database connections, if you weren’t using Django, you’d have to 
create a database connection with something like `psycopg2.connect()` and 
dispose of it with `connection.close()` when you no longer need it. When you’re 
using the Django ORM, database connections are opened automatically when 
needed. You still have to close them manually if you aren’t working within the 
traditional request-response cycle.

If the implicit connection establishment is causing trouble, it shouldn’t be 
hard to write a custom database backend that doesn’t have this behavior. You 
would add a method to establish a connection and raise an exception in 
connection.get_new_connection() 

 if the connection is already established.

You’re writing that you can’t force Django to create a new connection. That’s 
as simple as connection.ensure_connection() 
,
 possibly after connection.close() 

 if you want to close a pre-existing connection. ensure_connection() is a 
private API, but you’re already in unsupported territory if you’re running 
Django within an async framework, so that isn’t making your situation 
significantly worse.

Spreading FUD about Django’s supposed unsuitability for non-trivial projects 
isn’t going to help with your issues and isn’t going to create a context where 
people will look forward to helping you, so please consider avoiding gratuitous 
attacks if you need further help.

Best regards,

-- 
Aymeric.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/7B853AE1-84F1-4165-9BCE-6A668B5C6783%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.

Re: Fate of sql* management commands

[Marcin Nowak]

On Monday, June 1, 2015 at 5:11:27 PM UTC+2, Andrew Godwin wrote:
>
> OK, so I've just gone ahead and done the initial work on this: 
> https://github.com/django/django/pull/4729
>
>
They shot not only in my head - 
http://stackoverflow.com/questions/35455706/equivalent-of-sqlall-in-django-1-9

Generating sql from migrations is useful for translating python migrations 
into SQL (the only one good way to handle database schema).
But old `sql` command generates SQL directly from models, so an application 
must not have new migrations defined.

The old-style commands are still required. I heard from my fellows that 
they are downgrading their projects from Django 1.9 to 1.8 due to missing 
sql commands. Not so funny.

Marcin

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/8f21d96f-c309-46c7-8ec2-19af7189bc12%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Threads and db connection handling question

[Marcin Nowak]

On Tuesday, June 21, 2016 at 2:34:11 PM UTC+2, Aymeric Augustin wrote:
 

> Regarding database connections, if you weren’t using Django, you’d have to 
> create a database connection with something like `psycopg2.connect()` and 
> dispose of it with `connection.close()` when you no longer need it. When 
> you’re using the Django ORM, database connections are opened automatically 
> when needed. You still have to close them manually if you aren’t working 
> within the traditional request-response cycle.
>

Yes, I'm using ORM. The connections are opening silently in main 
process/thread.
I've tried to close them manually, but I've had some kind of problems 
(don't remember what kind of), probably with wrapped `close()`.  


> If the implicit connection establishment is causing trouble, it shouldn’t 
> be hard to write a custom database backend that doesn’t have this behavior. 
>

Maybe this will help. It's worth trying someday.

 

> Spreading FUD about Django’s supposed unsuitability for non-trivial 
> projects isn’t going to help with your issues and isn’t going to create a 
> context where people will look forward to helping you, so please consider 
> avoiding gratuitous attacks if you need further help.
>

This not a FUD but the honest advice. And not an attack but statement of a 
fact. It's nothing personal.

There are much more problems with Django used in some kind of projects, 
especially in long-term projects. I wrote about those many times and almost 
always I've got two answers: 1) do not upgrade  2) this will not be 
changed. Oh, and third - "write yourself", and I'm writing and rewriting 
old good parts of Django and not talking about this (this is only what I 
can do in that case). 

But I belive this is a only matter of time.  I remember how big was 
resistance of implementing `model.reload()`. 7 years of talking about that, 
right? And here we go - it is available from v1.8 as "refresh_from_db()" :) 
 So let's hope for the best :)
 
Marcin

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/559e8fdc-d49c-49c9-b71d-f2d9973ae085%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Fate of sql* management commands

[Tim Graham]

Marcin, what are you hoping to accomplish with your latest mail? As Aymeric 
said in another thread, repeated complaining is not going to help. There's 
already an accepted ticket for this feature. If you want to be productive 
and helpful, pick up Andrew's patch and try to complete it. Alternatively, 
if the business needs justify it, you could offer to pay someone to 
implement it if that suits you better. Tim

On Tuesday, June 21, 2016 at 8:39:54 AM UTC-4, Marcin Nowak wrote:
>
>
>
> On Monday, June 1, 2015 at 5:11:27 PM UTC+2, Andrew Godwin wrote:
>>
>> OK, so I've just gone ahead and done the initial work on this: 
>> https://github.com/django/django/pull/4729
>>
>>
> They shot not only in my head - 
> http://stackoverflow.com/questions/35455706/equivalent-of-sqlall-in-django-1-9
>
> Generating sql from migrations is useful for translating python migrations 
> into SQL (the only one good way to handle database schema).
> But old `sql` command generates SQL directly from models, so an 
> application must not have new migrations defined.
>
> The old-style commands are still required. I heard from my fellows that 
> they are downgrading their projects from Django 1.9 to 1.8 due to missing 
> sql commands. Not so funny.
>
> Marcin
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/c8591966-f49c-47ed-8cd1-82fcb74a360d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Fate of sql* management commands

[Marcin Nowak]

On Tuesday, June 21, 2016 at 3:28:26 PM UTC+2, Tim Graham wrote:
>
> Marcin, what are you hoping to accomplish with your latest mail? As 
> Aymeric said in another thread, repeated complaining is not going to help. 
>

Sorry for that. I'm just overhelmed and tired of "fighting" with missing 
things which were good, then removed, and now someone should reimplement 
them again.
 

> There's already an accepted ticket for this feature. If you want to be 
> productive and helpful, pick up Andrew's patch and try to complete it. 
> Alternatively, if the business needs justify it, you could offer to pay 
> someone to implement it if that suits you better. Tim
>

Well, I'll do something with pleasure. And please don't worry about money. 
But I will not give any patch until you're rejecting change at a 
concept/design stage.

The old sql* commands were good and helpful. To bring them back someone 
must "revert" commits with removals, so there will be an old code 
generating SQL directly from models (ommiting builitin migrations). Do you 
agree?

 
Marcin.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/1c2d2b7e-05dd-45e9-88e6-0515945295ef%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Fate of sql* management commands

[Tim Graham]

No, it's not as simple as reverting the removal commit [0]. All the legacy 
schema generation methods are removed too [1]. We don't want to add them 
back but rather use the SQL generation from the migrations system 
(SchemaEditor classes). Please read this thread closely and look at 
Andrew's pull request [2].

[0] 
https://github.com/django/django/commit/7e8cf74dc74539f40f4cea53c1e8bba82791fcb6
[1] 
https://github.com/django/django/commit/2b039d966f6e61a5ffb5ffac25aa198f9043de3d
[2] https://github.com/django/django/pull/4729

On Tuesday, June 21, 2016 at 9:38:25 AM UTC-4, Marcin Nowak wrote:
>
>
>
> On Tuesday, June 21, 2016 at 3:28:26 PM UTC+2, Tim Graham wrote:
>>
>> Marcin, what are you hoping to accomplish with your latest mail? As 
>> Aymeric said in another thread, repeated complaining is not going to help. 
>>
>
> Sorry for that. I'm just overhelmed and tired of "fighting" with missing 
> things which were good, then removed, and now someone should reimplement 
> them again.
>  
>
>> There's already an accepted ticket for this feature. If you want to be 
>> productive and helpful, pick up Andrew's patch and try to complete it. 
>> Alternatively, if the business needs justify it, you could offer to pay 
>> someone to implement it if that suits you better. Tim
>>
>
> Well, I'll do something with pleasure. And please don't worry about money. 
> But I will not give any patch until you're rejecting change at a 
> concept/design stage.
>
> The old sql* commands were good and helpful. To bring them back someone 
> must "revert" commits with removals, so there will be an old code 
> generating SQL directly from models (ommiting builitin migrations). Do you 
> agree?
>
>  
> Marcin.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/c9f13033-93c0-40fb-9bc0-e9e983b8a2fc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Binding LiveServerTestCase bind to port 0

[Tim Graham]

I'm inclined to make the changes, skip the deprecation, and react 
appropriately if anyone raises a use case that can't be achieved with the 
new "bind to port 0" technique.

I found a few more things that could be removed, for example the "manage.py 
test --liveserver=..." option. Only creating a ticket and release notes 
remain as long as there aren't any concerns.

https://github.com/django/django/pull/6791

On Thursday, June 16, 2016 at 11:09:06 PM UTC-4, charettes wrote:
>
> I've never personally used DJANGO_LIVE_TEST_SERVER_ADDRESS for another 
> purpose than preventing port conflicts which binding to port 0 solves.
>
> It always looked strange to me to rely on an environment variable to 
> achieve this. It feels like it was done this way to avoid introducing yet 
> another setting.
>
> I suppose we could deprecate DJANGO_LIVE_TEST_SERVER_ADDRESS and use a 
> LiveServerTestCase.server_thread_class = LiveServerThread attribute to 
> create the thread and suggest overriding it in the deprecation warning 
> message if new behavior is problematic.
>
> Simon
>
> Le jeudi 16 juin 2016 21:23:05 UTC-4, Tim Graham a écrit :
>>
>> In IRC the other day, Donald pointed out 
>> https://www.dnorth.net/2012/03/17/the-port-0-trick/ and suggested that 
>> Django might be able to take advantage of that technique.
>>
>> I put together a proof of concept that seems to work: 
>> https://github.com/django/django/pull/6791
>>
>> I'm wondering if anyone sees any problems with this approach and if 
>> there's any use case for continuing to support customizing the address 
>> using the DJANGO_LIVE_TEST_SERVER_ADDRESS environment variable.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/4b691593-2892-4dda-9885-b2428b243ad0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Fate of sql* management commands

[Marcin Nowak]

On Tuesday, June 21, 2016 at 4:00:51 PM UTC+2, Tim Graham wrote:
>
> No, it's not as simple as reverting the removal commit [0]. All the legacy 
> schema generation methods are removed too [1]. 
>
We don't want to add them back 
>

I know.  
 

> but rather use the SQL generation from the migrations system (SchemaEditor 
> classes). Please read this thread closely and look at Andrew's pull request 
> [2].
>
>
It is complicated for such simple job (dumping model schema to sql) but 
I'll give a try, just later. 
Currently I must backport JSONField from 1.9 to 1.8, because I can't 
upgrade 1.8.6 to 1.9, because it has removed `sql*` commands.  
 
Marcin.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/81ceed6e-b3c2-41ae-89dd-b41fa4c23ed2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: decorator_from_middleware changes

[Carl Meyer]

Hi Shai,

On 06/20/2016 11:07 PM, Shai Berger wrote:
> Well, we can change decorator_from_middleware to introspect the middleware 
> itself and decide accordingly:
> 
> -- if it does not have process_request or process_response as callable 
> attributes, and it is callable itself, it is a new-style middleware 

"Does not have process_request and process_response as callable
attributes" isn't reliable -- there's nothing preventing a new-style
middleware from having such methods and calling them from `__call__` (in
fact any middleware converted using MiddlewareMixin will fit this
description).

> -- if it does, we can define a new attribute '_dep5' (or something) whose 
> presence will indicate a dep5-compatible middleware. This attribute can be 
> defined on the transition mixin, so that people who use it for transition 
> don't 
> have to care
> 
> -- if it has callable process_request/process_response and doesn't have _dep5 
> then it is an old-style middleware
> 
> Note that decorator_from_middleware already used to instantiate the 
> middleware 
> class anyway, so the check can be applied to a middleware object, not to a 
> middleware factory.
> 
> The whole introspection behavior, including the _dep5 attribute and the code 
> that checks for it, can be dropped when old-style middleware is finally 
> dropped. If someone uses _dep5 manually, it will do them no harm later.

We can indeed introspect to try to auto-detect. It's a bit more complex
due to the different `__init__` signature as well, and it would fail in
case someone defined __call__ on an old-style middleware for some
reason, but I've already written the code for this in
https://github.com/django/django/pull/6765

The main reason I abandoned auto-detection for this second attempt is
that the behavior of these new-style view decorators is different enough
that I'd like to provide better flexibility for third-party projects.
Imagine a third-party project that currently provides both a middleware
and a view decorator, created by decorator_from_middleware. Now they are
adding Django 1.10 compatibility. If we auto-detect, and they inherit
their middleware from the transition mixin so it's cross-compatible,
decorator_from_middleware will suddenly detect it as new-style, leaving
them no way to continue to provide their old decorator with fully
backwards-compatible behavior.

Basically, I think middleware_decorator does something that's
fundamentally different enough from decorator_from_middleware that it is
clearer to make it a separate function and let people make explicit
choices about which one they are using.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/57696C0F.5050804%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature

Re: decorator_from_middleware changes

[Carl Meyer]

Hi Tobias,

On 06/21/2016 06:22 AM, Tobias McNulty wrote:
> On Mon, Jun 20, 2016 at 10:51 AM, Carl Meyer  > wrote
>> Possible disadvantages of this approach:
> 
>  
> 
>> 3. The new behavior may surprise some people accustomed to the old
>> behavior, since it means the effect of the middleware decorator won't
>> occur immediately where the decorator is applied. E.g. if you apply a
>> middleware decorator to a view, and then another custom view decorator
>> outside that, the custom view decorator won't see the effects of the
>> middleware decorator in the actual response (it would only see a new
>> entry in view_func._extra_middleware).
> 
> This one seems like the gotcha to me.

Yeah, I agree.

> What if, for example, I have a
> view decorator whose effects I specifically don't want to cache, but I
> do want to cache the underlying view; is it fair to require that the
> person write a middleware and then turn it into a decorator to be able
> to do that?

Caching happens to be a bad example, because AFAICT nobody should ever
use the cache_page decorator (or cache full responses right after the
view function, as you describe here) due to
https://code.djangoproject.com/ticket/15855 -- unless I suppose they are
pre-adding all the correct Vary headers themselves.

But I think the general point is a good one nonetheless.

> Are we effectively saying, "for view decorators to behave
> the way you might expect, implement them as middleware"? It seems odd,
> to me at least, that I should care what the underlying implementation of
> a decorator is before I use it. It also violates the 'strict in/out
> layering' goal, albeit for decorators instead of middleware. (A similar
> argument could be said of exceptions -- what if I want to trap an
> exception raised by a middleware-turned-decorator?)
> 
> It might be okay if the decorators themselves were explicit about what
> they were doing, for example @cache_page(3600) might become:
> 
> @add_middleware(CacheMiddleware, cache_timeout=3600)

Yes, I did think about this. It makes it more difficult to use, but it
does give a clearer picture of what's actually happening. We aren't
directly decorating the view function, we're just annotating it with an
extra per-view middleware.

> However, that's probably a bigger and unnecessary change.
> 
> Would it be possible to implement a combination of the approaches, i.e.,
> make the delay in applying the middleware-turned-decorator the exception
> rather than the rule, perhaps only for TemplateResponses and
> specifically for the purpose of supporting a deprecation plan? And then,
> long-term, leave it up to middleware/decorator authors & users to decide
> how best to implement/layer them, being explicit about the implications
> of rendering the response or perhaps more appropriately, "not rendering
> if you can avoid it" (i.e., your first strategy)?

I don't think it's worth doing this just for a deprecation path. If
we're OK in the long term with making middleware-turned-decorator
authors responsible for correctly handling TemplateResponse themselves,
we should just go there right away via
https://github.com/django/django/pull/6765 -- we can take care of
backwards-compatibility quite nicely there in the transition mixin. In
fact I'd say the overall backward-compatibility of this approach (#6765)
is better than that of #6805 (the extra-middleware-annotation approach),
due to converting all our builtin decorators-from-middleware to the new
style in the latter.

The downside of #6765 is that the boilerplate for correctly handling
TemplateResponse is pretty ugly [1], and any middleware that accesses
response.content would pretty much need to do that, in case it might
sometime be used in decorator_from_middleware.

To be honest though, I'm still not sure that isn't our best option. Most
middleware I've written don't access response.content, they set headers,
so they'd be unaffected. Some of Django's built-in ones do touch
content, but we can deal with the ugly TemplateResponse boilerplate in
our own middleware.

Carl


 [1]
https://github.com/carljm/django/blob/7d999844bd4b43b82a472634074156c24a4fc7cb/docs/topics/http/middleware.txt#L304

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/576970EC.50704%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature

Re: decorator_from_middleware changes

[Carl Meyer]

On 06/21/2016 10:53 AM, Carl Meyer wrote:
> If we're OK in the long term with making middleware-turned-decorator
> authors responsible for correctly handling TemplateResponse themselves,
> we should just go there right away via
> https://github.com/django/django/pull/6765 -- we can take care of
> backwards-compatibility quite nicely there in the transition mixin. In
> fact I'd say the overall backward-compatibility of this approach (#6765)
> is better than that of #6805 (the extra-middleware-annotation approach),
> due to converting all our builtin decorators-from-middleware to the new
> style in the latter.
> 
> The downside of #6765 is that the boilerplate for correctly handling
> TemplateResponse is pretty ugly [1], and any middleware that accesses
> response.content would pretty much need to do that, in case it might
> sometime be used in decorator_from_middleware.
> 
> To be honest though, I'm still not sure that isn't our best option. Most
> middleware I've written don't access response.content, they set headers,
> so they'd be unaffected. Some of Django's built-in ones do touch
> content, but we can deal with the ugly TemplateResponse boilerplate in
> our own middleware.

We could also leave MiddlewareMixin in place indefinitely, and people
who don't want to do advanced things in __call__ (like using context
managers) can keep inheriting from the mixin and writing process_request
and process_response methods until eternity. Then the mixin takes care
of TemplateResponse deferral and they don't need to worry about it.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/57697232.7010501%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature

Re: status of 1.10 release blockers

[Tim Graham]

The blocker is merged so I'll release the beta in a couple hours if no one 
stops me.

On Monday, June 20, 2016 at 9:10:12 PM UTC-4, Tim Graham wrote:
>
> The remaining blocker:
>
> UserCreationForm doesn't call normalize_email and normalize_username
> https://code.djangoproject.com/ticket/26719
> Status: I sent some final edits to the PR and it should be ready for 
> commit tomorrow.
>
> Other possible blockers:
>
> Different manager for _base_manager and related descriptors
> https://code.djangoproject.com/ticket/26749
> django-hvad is broken by the manager inheritance refactor and it's unclear 
> if it can be adapted for the new behavior.
>
> Update decorator_from_middleware() to work with DEP5 style middleware.
> https://code.djangoproject.com/ticket/26626
> Design decision needed, see 
> https://groups.google.com/d/topic/django-developers/hNQIaYcN3xM/discussion
>
> As long as no new issues pop up by tomorrow, I'm thinking to do the beta 
> release then. We might consider a "beta 2" release around 2 weeks later if 
> we decide to push fixes for the other two issues.
>
> On Saturday, June 18, 2016 at 8:05:32 PM UTC-4, Tim Graham wrote:
>>
>> I'm postponing the release until next week as some release blockers 
>> remain:
>>
>>
>> https://code.djangoproject.com/query?status=!closed&severity=Release%20blocker
>>
>> On Friday, June 17, 2016 at 5:33:04 PM UTC-4, Claude Paroz wrote:
>>>
>>> Le vendredi 17 juin 2016 16:52:50 UTC+2, Tim Graham a écrit :

 There are a few small issues to finish up, but if all goes well, the 
 beta release will happen sometime tomorrow (at least 24 hours from now).

>>>
>>> I'm a bit worried about https://code.djangoproject.com/ticket/26719 
>>> which could have security implications for 1.10. To be investigated...
>>>
>>> Claude
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ce12f95b-4e36-47f3-9810-7f57e8e76941%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: decorator_from_middleware changes

[Shai Berger]

Hi Carl,

On Tuesday 21 June 2016 19:32:15 Carl Meyer wrote:
> Hi Shai,
> 
> On 06/20/2016 11:07 PM, Shai Berger wrote:
> > Well, we can change decorator_from_middleware to introspect the
> > middleware itself and decide accordingly:
> > 
> > -- if it does not have process_request or process_response as callable
> > attributes, and it is callable itself, it is a new-style middleware
> 
> "Does not have process_request and process_response as callable
> attributes" isn't reliable -- there's nothing preventing a new-style
> middleware from having such methods and calling them from `__call__` 

That was an "if", not "if and only if". 

> (in fact any middleware converted using MiddlewareMixin will fit this
> description).
> 

And I suggested a way to deal with them in the next two paragraphs:

> > -- if it does, we can define a new attribute '_dep5' (or something) whose
> > presence will indicate a dep5-compatible middleware. This attribute can
> > be defined on the transition mixin, so that people who use it for
> > transition don't have to care
> > 
> > -- if it has callable process_request/process_response and doesn't have
> > _dep5 then it is an old-style middleware
> > 

...except that on second thought, rather than just looking for its presence, 
we could look for it to be truthy -- so a middleware could explicitly mark 
itself as "old style" for the decorator, by setting it to False.

> 
> We can indeed introspect to try to auto-detect. It's a bit more complex
> due to the different `__init__` signature as well, and it would fail in
> case someone defined __call__ on an old-style middleware for some
> reason, but I've already written the code for this in
> https://github.com/django/django/pull/6765
> 

I missed the difference in __init__, but if you already took care of it...

> The main reason I abandoned auto-detection for this second attempt is
> that the behavior of these new-style view decorators is different enough
> that I'd like to provide better flexibility for third-party projects.
> Imagine a third-party project that currently provides both a middleware
> and a view decorator, created by decorator_from_middleware. Now they are
> adding Django 1.10 compatibility. If we auto-detect, and they inherit
> their middleware from the transition mixin so it's cross-compatible,
> decorator_from_middleware will suddenly detect it as new-style, leaving
> them no way to continue to provide their old decorator with fully
> backwards-compatible behavior.
> 

So, for middlewares where it matters, inherit the mixin and specify the 
attribute as False.

> Basically, I think middleware_decorator does something that's
> fundamentally different enough from decorator_from_middleware that it is
> clearer to make it a separate function and let people make explicit
> choices about which one they are using.
> 

And yet the plan is to deprecate decorator_from_middleware -- that is, "rob" 
people of that choice later on. As long as we intend to do that, I think it 
better to minimize churn.

Shai.

Re: Fate of sql* management commands

[Marcin Nowak]

On Tuesday, June 21, 2016 at 5:49:08 PM UTC+2, Marcin Nowak wrote:
>
>
>
> On Tuesday, June 21, 2016 at 4:00:51 PM UTC+2, Tim Graham wrote:
>>
>> No, it's not as simple as reverting the removal commit [0]. All the 
>> legacy schema generation methods are removed too [1]. 
>>
> We don't want to add them back 
>>
>
> I know.  
>  
>
>> but rather use the SQL generation from the migrations system 
>> (SchemaEditor classes). Please read this thread closely and look at 
>> Andrew's pull request [2].
>>
>>
> It is complicated for such simple job (dumping model schema to sql) but 
> I'll give a try, just later. 
>
>
No, Tim. I will not go through the Andrew's path. It is a "dead end".  
What the `sql` command should do is just simple dump model meta to desired 
sql dialect.

Migrations are designed to manage and apply model changes, so it is quite 
different use case. 

`sql` command should not depend on migrations, because:

   - they may be corrupted  (common case with south/django migrations; they 
   are unreliable)
   - achieving final state is a long journey through winding road,
   - app may not include migrations itself,
   - it is about SQL but not about managing model state/changes.


The only way is to fix SchemaEditor by extractnig sql from create_model(), 
without executing it.  But it's not so easy, because SE requires to be run 
as a context manager and executes deffered sqls at __exit__().  There are 
other SE`s methods which depends on defferend sql execution at 
`__exit__()`. Sorry, but it looks like code written in big hurry.

If SchemaEditor will not be removed in near future, I can try to patch it 
and use as a base of new `sql` command. I think there is no other 
reasonable way to do that. 

Somebody wrote eariler that `sql*` commands can be written as an external 
app/package. They can't. Django internals must be fixed before this, or new 
app must provide complete solution including sql generator like SE (but 
this is a second nonsense, because SE is strongly coupled with db backends).

And finally I would like to remind you, that `sql*` commands were removed 
by you. This was a mistake. Not the first, not the last.  But when you're 
asking me what I'm hoping writing such mails, just please stand in my (and 
others) shoes and think a little about it.

Marcin

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5bd5ef38-83b2-470e-b574-9a533860115a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Fate of sql* management commands

[Marcin Nowak]

On Tuesday, June 21, 2016 at 10:55:13 PM UTC+2, Marcin Nowak wrote:
>
>
> Somebody wrote eariler that `sql*` commands can be written as an external 
> app/package. They can't. Django internals must be fixed before this, or new 
> app must provide complete solution including sql generator like SE (but 
> this is a second nonsense, because SE is strongly coupled with db backends).
>
>
Errata - I've missed "collect_sql" mode, where sql is not executed but 
collected. The patching should not be neccessary. The name of "execute()" 
method is misleading in such context. 

Marcin

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/d811b807-e60f-4fb1-bf10-3d6ae77dc7a3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Fate of sql* management commands

[Shai Berger]

Marcin,

You can write the "sqlall" you want as a shell script -- remove any existing 
migrations,  use "makemigrations" to create an initial migratgion, which will 
exactly reflect your models, and run sqlmigrate on that. You want to do that 
against a clean database, to make sure no migration has already been 
registered for the app in the migration history table. 

I repeat Aymeric's advice that spreading your frustrations over this mailing 
list is counter-productive.

HTH,
Shai.

Re: Fate of sql* management commands

[Marcin Nowak]

>
>
> You can write the "sqlall" you want as a shell script -- remove any 
> existing 
> migrations,  use "makemigrations" to create an initial migratgion, 


Not exactly. I'm not using migrations [.. cut ..]  
 

I repeat Aymeric's advice that spreading your frustrations over this 
> mailing 
> list is counter-productive. 
>
>
Same as removing useful tools. Let's remove `runserver` command for a year 
or two, ok?
In the other thread I just gave advise to guy, who is going through path 
similar to mine. 

[.. cut ..]  

Marcin

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/e6eeb4ae-9b76-48fc-ae87-49a8e1730cf2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Fate of sql* management commands

[Shai Berger]

On Wednesday 22 June 2016 00:48:58 Marcin Nowak wrote:
> > You can write the "sqlall" you want as a shell script -- remove any
> > existing
> > migrations,  use "makemigrations" to create an initial migratgion,
> 
> Not exactly. I'm not using migrations [.. cut ..]
> 

So add another step in the end to remove the temporary migrations folder 
created just in order to enable sqlmigrate. If you're a real purist, add yet 
another one to make sure you don't have a django_migrations table left over.

Really, this is django-users turf.

Shai.

Re: Fate of sql* management commands

[Marcin Nowak]

>
>
> So add another step in the end to remove the temporary migrations folder 
> created just in order to enable sqlmigrate. If you're a real purist, add 
> yet 
> another one to make sure you don't have a django_migrations table left 
> over. 
>
>
Sorry, Shai, I doubt you understand the problem.


Really, this is django-users turf. 


Do what you want. IMO it isn't.

I'm talking here about ressurecting killed tools. And currently I'm working 
on a patch.
I think that redirecting discussion about Django internals to django-users 
list is more counter productive than my earlier thoughts. 

PS. I'm not a purist, but I don't like django_migrations table because it 
is created automatically in foreign database, which Django should not touch 
without direct command from dba architect.
   
Marcin

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ed060117-164f-4814-b587-031c7dbba2e5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Fate of sql* management commands

[Tim Graham]

I'll be happy to look at whatever you come up with or review whatever 
changes you think need to be made in Django to solve the ticket, however, I 
fail to understand how Andrew's proposal isn't suitable. Maybe you could 
give more specific details than "migrations may be corrupted" and "they are 
unreliable". As far as I see, Andrew's proposal doesn't use any on-disk 
migrations so it doesn't require an app to have migrations.

Yes, I did remove the sql* commands because they were scheduled for removal 
when the migrations system was introduced. We didn't want to maintain two 
methods for generating schema. Your previous mail was just rehashing old 
complaints about the lack of this functionality. If I were in your shoes 
and needed this functionality, I would get involved and try to engage in a 
discussion to learn about the reasons for this. Then I would make a 
proposal for how to solve the problem and then implement it. I wouldn't 
complain about it, disparage the design of the Django, and expect someone 
else to write the code that I need. Anyway, it sounds like you're off on 
the right path now.

You have a lot of interesting ideas about where Django could be improved 
but please share them in a polite and constructive fashion. Avoid pointless 
statements like "Sorry, but it looks like code written in big hurry" which 
only serve to offend the people who spend many hours contributing to 
Django. If you continue down that path, I guess all we can do is ignore you 
because we don't like to work with people who can't be polite.

On Tuesday, June 21, 2016 at 4:55:13 PM UTC-4, Marcin Nowak wrote:
>
>
>
> On Tuesday, June 21, 2016 at 5:49:08 PM UTC+2, Marcin Nowak wrote:
>>
>>
>>
>> On Tuesday, June 21, 2016 at 4:00:51 PM UTC+2, Tim Graham wrote:
>>>
>>> No, it's not as simple as reverting the removal commit [0]. All the 
>>> legacy schema generation methods are removed too [1]. 
>>>
>> We don't want to add them back 
>>>
>>
>> I know.  
>>  
>>
>>> but rather use the SQL generation from the migrations system 
>>> (SchemaEditor classes). Please read this thread closely and look at 
>>> Andrew's pull request [2].
>>>
>>>
>> It is complicated for such simple job (dumping model schema to sql) but 
>> I'll give a try, just later. 
>>
>>
> No, Tim. I will not go through the Andrew's path. It is a "dead end".  
> What the `sql` command should do is just simple dump model meta to desired 
> sql dialect.
>
> Migrations are designed to manage and apply model changes, so it is quite 
> different use case. 
>
> `sql` command should not depend on migrations, because:
>
>- they may be corrupted  (common case with south/django migrations; 
>they are unreliable)
>- achieving final state is a long journey through winding road,
>- app may not include migrations itself,
>- it is about SQL but not about managing model state/changes.
>
>
> The only way is to fix SchemaEditor by extractnig sql from create_model(), 
> without executing it.  But it's not so easy, because SE requires to be run 
> as a context manager and executes deffered sqls at __exit__().  There are 
> other SE`s methods which depends on defferend sql execution at 
> `__exit__()`. Sorry, but it looks like code written in big hurry.
>
> If SchemaEditor will not be removed in near future, I can try to patch it 
> and use as a base of new `sql` command. I think there is no other 
> reasonable way to do that. 
>
> Somebody wrote eariler that `sql*` commands can be written as an external 
> app/package. They can't. Django internals must be fixed before this, or new 
> app must provide complete solution including sql generator like SE (but 
> this is a second nonsense, because SE is strongly coupled with db backends).
>
> And finally I would like to remind you, that `sql*` commands were removed 
> by you. This was a mistake. Not the first, not the last.  But when you're 
> asking me what I'm hoping writing such mails, just please stand in my (and 
> others) shoes and think a little about it.
>
> Marcin
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/1231b31e-1f17-4bf1-a2ca-c8e364dac69d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: decorator_from_middleware changes

[Carl Meyer]

Hi Shai,

Apologies, I didn't fully process your suggestion or the reason for the
flag the first time. I think it would work, though I'd be tempted to set
the flag to False on the transition middleware by default, to provide
perfect back-compat for existing uses of decorator_from_middleware until
someone opts in by setting the flag to True or writing their own
new-style middleware that doesn't inherit the mixin.

I'm still not entirely happy with this approach for the reason Tobias
highlighted, though - it doesn't behave like someone might expect a view
decorator to behave, and I think probably it should have an API that
clarifies that what it's really doing is applying per-view middleware.

Still halfway tempted to go with the original "make middleware authors
responsible for this themselves, if they want to access
response.content" approach.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5769C70E.1010308%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature

Re: decorator_from_middleware changes

[Tobias McNulty]

On Tue, Jun 21, 2016 at 7:00 PM, Carl Meyer  wrote:

> Still halfway tempted to go with the original "make middleware authors
> responsible for this themselves, if they want to access
> response.content" approach.
>

I am curious to hear what others think, but I'm leaning this way too.
Having a permanent helper (perhaps MiddewareMixin as you suggested) for
dealing with TemplateResponses certainly sounds like a win, too.

Tobias
-- 


*Tobias McNulty*Chief Executive Officer

tob...@caktusgroup.com
www.caktusgroup.com

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAMGFDKTOoMmA-VdRUMcvgbJJr_q_xWHNoH7Su%2BTiHivsW%2B%2BVxw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[ANNOUNCE] Django 1.10 beta 1 released

[Tim Graham]

We've made the second release on the way to Django's next major release, 
Django 1.10! With a month and a half until the scheduled final release, 
we'll need timely testing from the community to ensure an on-time and 
stable release. Check out the blog post:

https://www.djangoproject.com/weblog/2016/jun/21/django-110-beta-1-released/

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/250f30a7-de13-495d-9409-fc1fc907b716%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.