Re: first() and last(), earliest() and latest()

2013-02-28 Thread Tom Christie 
> +1 for the first syntax.  The others are duplicating functionality 
> that is already present via more aptly named methods.  The first 
> syntax is also more consistent with other ORMs. 

Seconded.
Seems much more obvious, simple and explicit than the other options to me.

On Thursday, 28 February 2013 00:37:53 UTC, Ian wrote:
>
> On Wed, Feb 27, 2013 at 3:34 PM, Wim Feijen > 
> wrote: 
> > Which style do you prefer? 
> > 
> > .filter(last_name__startswith='b').order_by('last_name').first()# 
> clear 
> > and long 
> > .first(last_name__startswith='b').order_by('last_name')# first 
> method 
> > has filter syntax. 
> > .filter(last_name__startswith='b').first('last_name')   # first method 
> has 
> > order_by syntax. 
>
> +1 for the first syntax.  The others are duplicating functionality 
> that is already present via more aptly named methods.  The first 
> syntax is also more consistent with other ORMs. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Database pooling vs. persistent connections

2013-02-28 Thread David Cramer 
Can we please change this so it defaults to off, and just document how to 
turn it on and in what situations you should turn it on?

In my opinion this default-on feature caters to a very specific audience, 
and will cause a lot of unexpected behavior with other users.

Here is the tl;dr of an argument for turning it on:

Connections are expensive, we should persist them.

Here is the tl;dr for turning it off:

Connections are expensive, so we dont have a lot of them.

Immediately for anyone who has configured more workers than they have 
Postgres connections (which I can only imagine is common among people who 
havent setup infrastructure like pgbouncer) things will start blowing up.

Why should this be the default behavior?

Unlike MySQL, which already has cheap connections and doesn't suffer this 
problem, connections in Postgres add quite a large cost, both on creation, 
on allowing them. **Everyone** who cares about their performance on 
Postgres should already be using pgbouncer (or some alternative) to offset 
**both** of these, not just a single problem which is what Django is 
addressing here.

I will happily defend my opinion in person at PyCon in two weeks if it 
still remains a default, and anyone is willing to listen, and if you reach 
out to the other large (and I dont mean scale) Django users I imagine you 
will find a lot of mixed feelings about this default.

On Sunday, February 17, 2013 3:24:52 AM UTC-8, Aymeric Augustin wrote:
>
> **tl;dr** I believe that persistent database connections are a good idea. 
> Please prove me wrong :) 
>
>  
>
> Since I didn't know why the idea of adding a connection pooler to Django 
> was 
> rejected, I did some research before replying to the cx_Oracle 
> SessionPooling 
> thread. 
>
> The best explanation I've found is from Russell: 
>
> > To clarify -- we've historically been opposed to adding connection 
> > pooling to Django is for the same reason that we don't include a web 
> > server in Django -- the capability already exists in third party 
> > tools, and they're in a position to do a much better job at it than us 
> > because it's their sole focus. Django doesn't have to be the whole 
> > stack. 
>
> All the discussions boil down to this argument, and the only ticket on the 
> topic is short on details: https://code.djangoproject.com/ticket/11798 
>
>  
>
> The connection pools for Django I've looked at replace "open a connection" 
> by 
> "take a connection from the pool" and "close a connection" by "return the 
> connection to the pool". This isn't "real" connection pooling: each worker 
> holds a connection for the entire duration of each request, regardless of 
> whether it has an open transaction or not. 
>
> This requires as many connection as workers, and thus is essentially 
> equivalent to persistent database connections, except connections can be 
> rotated among workers. 
>
> Persistent connections would eliminate the overhead of creating a 
> connection 
> (IIRC ~50ms/req), which is the most annoying symptom, without incurring 
> the 
> complexity of a "real" pooler. 
>
> They would be a win for small and medium websites that don't manage their 
> database transactions manually and where the complexity of maintaining an 
> external connection pooler isn't justified. 
>
> Besides, when Django's transaction middelware is enabled, each request is 
> wrapped in a single transaction, which reserves a connection. In this 
> case, a 
> connection pooler won't perform better than persistent connections. 
>
> Obviously, large websites should use an external pooler to multiplex their 
> hundreds of connections from workers into tens of connections to their 
> database and manage their transactions manually. I don't believe 
> persistent 
> connections to the pooler would hurt in this scenario, but if it does, it 
> could be optional. 
>
>  
>
> AFAICT there are three things to take care of before reusing a connection: 
>
> 1) restore a pristine transaction state: transaction.rollback() should do; 
>
> 2) reset all connection settings: the foundation was laid in #19274; 
>
> 3) check if the connection is still alive, and re-open it otherwise: 
> - for psycopg2: "SELECT 1"; 
> - for MySQL and Oracle: connection.ping(). 
>
> Some have argued that persistent connections tie the lifetime of databases 
> connections to the lifetime of workers, but it's easy to store the 
> creation 
> timestamp and re-open the connection if it exceeds a given max-age. 
>
> So -- did I miss something? 
>
> -- 
> Aymeric. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=

Re: Database pooling vs. persistent connections

2013-02-28 Thread Aymeric Augustin 
On 28 févr. 2013, at 20:09, David Cramer  wrote:

> Can we please change this so it defaults to off, and just document how to 
> turn it on and in what situations you should turn it on?
> 

> In my opinion this default-on feature caters to a very specific audience, and 
> will cause a lot of unexpected behavior with other users.

For the record, the reasoning behind the current default is as follows.

By default, Django uses the database session backend. As a consequence, each 
worker opens a database connection early in the request cycle (in 
process_request), and keeps it open until the end of the request. I don't have 
any figures, but I believe that such sites cannot be dismissed as "a very 
specific audience".

When a website grows larger (in terms of traffic), its developers optimize it: 
put sessions into a cache, disable transaction middleware, turn autocommit on, 
handle transactions manually, use a database connection pooler, etc.

The further along this curve you are, the less persistent connections are 
useful; they may even become harmful. However, the further along this curve you 
are, the more you know about database performance too, and the better your 
ability to determine if persistent connections are useful for your application.

The default is targeted at people who wouldn't know what a good value is and 
who will get the most benefit from persistent connections.

You obviously have a strong opinion on what a good value is for your 
applications (namely, 0) and you won't suffer from an unsuitable default.

> Immediately for anyone who has configured more workers than they have 
> Postgres connections (which I can only imagine is common among people who 
> havent setup infrastructure like pgbouncer) things will start blowing up.

This argument was addressed by Alex earlier in the thread and I agree with his 
answer. I also find it more helpful to blow up at application start rather than 
when the traffic exceeds some threshold.

> Why should this be the default behavior?
> 
> Unlike MySQL, which already has cheap connections and doesn't suffer this 
> problem, connections in Postgres add quite a large cost, both on creation, on 
> allowing them. **Everyone** who cares about their performance on Postgres 
> should already be using pgbouncer (or some alternative) to offset **both** of 
> these, not just a single problem which is what Django is addressing here.

"**Everyone** who cares about their performance on Postgres" can take 5 seconds 
to set CONN_MAX_AGE to 0.

Persistent connections will improve the situation a little bit for everyone 
else — people running on a hosted platform, people who don't have the means to 
configure, run and monitor a pooler efficiently, people who don't even know 
what a pooler is, etc.

> I will happily defend my opinion in person at PyCon in two weeks if it still 
> remains a default, and anyone is willing to listen, and if you reach out to 
> the other large (and I dont mean scale) Django users I imagine you will find 
> a lot of mixed feelings about this default.

I won't be at PyCon, but several Django committers will. If a core dev wants to 
change the default to 0, feel free. (Being the author of this feature, I'm 
obviously biased towards turning it on.)

I don't mind being proved wrong, and I chose to commit this early in the 1.6 
release cycle precisely to give plenty of time for such feedback!

-- 
Aymeric.



-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Database pooling vs. persistent connections

2013-02-28 Thread Christophe Pettus 
One comment on the patch (which I generally approve of entirely):

It would be helpful to have a backend method that performers the "restore 
connection between uses" function, rather than just use connection.abort() (of 
course, the default implementation can use that).  For example, on PostgreSQL, 
ABORT; DISCARD ALL is the recommended way of resetting a connection, so being 
able to implement that would be great.

--
-- Christophe Pettus
   x...@thebuild.com

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Database pooling vs. persistent connections

2013-02-28 Thread Christophe Pettus 

On Feb 28, 2013, at 11:09 AM, David Cramer wrote:

> Immediately for anyone who has configured more workers than they have 
> Postgres connections (which I can only imagine is common among people who 
> havent setup infrastructure like pgbouncer) things will start blowing up.

If they have this configuration, it's an error.  The fact that the error is now 
surfacing doesn't make it a correct configuration.

--
-- Christophe Pettus
   x...@thebuild.com

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Database pooling vs. persistent connections

2013-02-28 Thread Aymeric Augustin 
On 28 févr. 2013, at 22:05, Christophe Pettus  wrote:

> It would be helpful to have a backend method that performers the "restore 
> connection between uses" function, rather than just use connection.abort() 
> (of course, the default implementation can use that).  For example, on 
> PostgreSQL, ABORT; DISCARD ALL is the recommended way of resetting a 
> connection, so being able to implement that would be great.

Good point, filed here: https://code.djangoproject.com/ticket/19948

-- 
Aymeric.



-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Database pooling vs. persistent connections

2013-02-28 Thread Michael 
On Thu, Feb 28, 2013 at 4:10 PM, Christophe Pettus  wrote:

>
> On Feb 28, 2013, at 11:09 AM, David Cramer wrote:
>
> > Immediately for anyone who has configured more workers than they have
> Postgres connections (which I can only imagine is common among people who
> havent setup infrastructure like pgbouncer) things will start blowing up.
>
> If they have this configuration, it's an error.  The fact that the error
> is now surfacing doesn't make it a correct configuration.


Why is this an error? It sounds more like a barrier to entry to me. Most
sites won't ever hit the maximum number of database connections, but they
over provision in their app (or they add more apps to the database as they
see their database load isn't so high). Seems to me if we enable this by
default, we are requiring a certain set of hardware that is way more
demanding than the actual use case for the app, just for a small speed
increase.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Database pooling vs. persistent connections

2013-02-28 Thread David Cramer 
It is most definitely not an "error" to have less connections available than 
workers, considering workers may serve different types of requests, and will 
now persist the database connection even after that request has finished. 

It may not be "the most correct", but a concurrent connection is very different 
than a single operation. 


On Thursday, February 28, 2013 at 1:40 PM, Michael wrote:

> On Thu, Feb 28, 2013 at 4:10 PM, Christophe Pettus  (mailto:x...@thebuild.com)> wrote:
> > 
> > On Feb 28, 2013, at 11:09 AM, David Cramer wrote:
> > 
> > > Immediately for anyone who has configured more workers than they have 
> > > Postgres connections (which I can only imagine is common among people who 
> > > havent setup infrastructure like pgbouncer) things will start blowing up.
> > 
> > If they have this configuration, it's an error.  The fact that the error is 
> > now surfacing doesn't make it a correct configuration.
> 
> Why is this an error? It sounds more like a barrier to entry to me. Most 
> sites won't ever hit the maximum number of database connections, but they 
> over provision in their app (or they add more apps to the database as they 
> see their database load isn't so high). Seems to me if we enable this by 
> default, we are requiring a certain set of hardware that is way more 
> demanding than the actual use case for the app, just for a small speed 
> increase. 
> 
> -- 
> You received this message because you are subscribed to a topic in the Google 
> Groups "Django developers" group.
> To unsubscribe from this topic, visit 
> https://groups.google.com/d/topic/django-developers/NwY9CHM4xpU/unsubscribe?hl=en.
> To unsubscribe from this group and all its topics, send an email to 
> django-developers+unsubscr...@googlegroups.com 
> (mailto:django-developers+unsubscr...@googlegroups.com).
> To post to this group, send email to django-developers@googlegroups.com 
> (mailto:django-developers@googlegroups.com).
> Visit this group at http://groups.google.com/group/django-developers?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>  
>  

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Database pooling vs. persistent connections

2013-02-28 Thread Christophe Pettus 

On Feb 28, 2013, at 1:43 PM, David Cramer wrote:

> It is most definitely not an "error" to have less connections available than 
> workers, considering workers may serve different types of requests, and will 
> now persist the database connection even after that request has finished.

If you have more workers than database connections, you have either (a) 
over-configured the number of workers, which is generally a bad thing to do, or 
(b) you are accepting that you will at high-load points get refused 
connections.  I don't see either one as being correct.

--
-- Christophe Pettus
   x...@thebuild.com

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Ticket #17093 -- quarantine global state in django.template

2013-02-28 Thread Christopher Medrela 
Hello, I made all tests green finally (all commits are tested under Python 
2.7.3 and 3.2.3). The problem that took a few days to solve was that the 
default engine should be recomputed after changing TEMPLATE_LOADERS setting 
since its _template_source_loaders internal needs to be recalculated. I 
tried to solve it by adding receiver in template/base.py and then 
circular-imports problem occured. I tried to solve it (that took a few 
days). Fortunately, I asked apollo13 and he helped me and told me that I 
should put the receiver to test/signals.py so I did it.

I also updated master which is important since the hierarchy of test files 
was restructured. I kept old branch and created new called ticket_17093v3 
[new-branch]. The patch is ready to first review (however it still lack 
documentation).

After gathering all pieces of global state into one, I started to do some 
clean up. I noticed that the cache loader unnecessary procrastinates 
resolving loaders list, so I moved the resolving to __init__.

Now there is still a lot of mess in the template system. The mess is about 
passing on origin of template and some APIs. For example, 
TemplateEngine.find_template and BaseLoader.load_template return tuple 
instead of a template. The second element of the tuple is display_name 
which is necessary to compute origin of template (and the origin is 
computed in many places! [computing-origin]). The first element may be not 
a Template instance but... its source.

My plan is that TemplateEngine.find_template and BaseLoader.load_template 
will return only a Template instance (and no display_name!). The Template 
will be created in BaseLoader.load_template and that will be place for 
computing origin of template. That will be much more simpler API than the 
current one. I tried to do this, but there are some problems.

The main problem is somehow buggy (in my opinion) behaviour of 
TemplateEngine.find_template [buggy-find-template]. If a loader raises 
TemplateDoesNotExist, then we catch this exception and check next loader. 
However, this is not what should be done. Look at the case when the 
template exists, but it includes a missing template. Including missing 
template raises the exception and it shouldn't be caught by find_template. 
The problem occured in many tickets [tickets] but the patches didn't solve 
the problem -- they're only a workarounds in my opinion.

I think that find_template should distinguish why the TemplateDoesNotExist 
was raised (because the current template is missing or because the current 
template exists but it includes some missing templates). The simplest 
solution is to ensure that the TemplateDoesNotExist is created with 
template name as its argument (with the exception of 
TemplateEngine.select_template case). Then find_template should check if 
the template name in exception is equal to `name` argument. If it's not 
true then reraise the exception loudly. Otherwise make it silent.

I partially did the changes (however, I didn't pushed it into the branch), 
and it works. My question is if the changes of API are acceptable since 
they are backward incompatible. The changes includes: TemplateDoesNotExist 
(added template_name argument in constructor), TemplateEngine.find_template 
and BaseLoader.load_template (and its subclasses)(changed return values).

Now I would like to ask about GSOC. Last time I told about my idea of 
introducing rendererers. Now I think that it doesn't introduce many 
improvements -- now you can use Django REST Framework (as Tom said) for 
JSON or you can just write your own render_to_response delegating to mako 
template system. So I dropped this idea. Now I wonder what are priorities?

   - Continuing refactoring (and adding new features) of template system. 
   Better error reporting. Logging invalid variable lookups. Compiling Django 
   templates into python functions so they will render faster.
   - Refactoring another part of Django. Is there any wiki page about how 
   Django should be refactored and what principles should I base on when I 
   restructure Django?
   - Improving error reporting.
   - Revamping of validation functionality.
   - Test framework cleanup. Speeding up tests. Splitting long tests 
   migrated from doctests. Testing Django application "in a void" 
   (independently of Django project and everything else). Running all tests 
   against different databases backends in sequence?
   
Please tell me which of these are most interesting for you (and which one 
have the greates chance to be a GSoC project). Which ones are important and 
which ones negligible? The list of ideas from 2012 doesn't contain 
information which ones are priorities. I would like to know to focus on the 
most important first.

[new-branch] https://github.com/chrismedrela/django/tree/ticket_17093v3
[computing-origin] Run `git grep -n make_origin`
[buggy-find-template] 
https://github.com/chrismedrela/django/blob/ticket_17093v3/django/template/base.py#L1362
[t

Re: DjangoCon Europe tickets on sale today!

2013-02-28 Thread Russell Keith-Magee 
On Thu, Feb 28, 2013 at 4:52 AM, Reinout van Rees wrote:

> On 27-02-13 21:21, Andre Terra wrote:
>
>> I wonder if these will ever be made available online for a fee, even if
>> months after the conference. For a hobbyist developer like me, coughing
>> up a few thousand dollars for the tickets, registration and hotel is
>> just impossible, but I would be glad to pay to have access to the
>> knowledge being shared there.
>>
>
> For most Python (and thus Django) conferences, the talks are assumed to be
> available online after a couple of weeks or months. The best site to look
> at is pyvideo: http://pyvideo.org/
>
> http://pyvideo.org/search?**models=videos.video&q=**djangocon
>
> Hm. I notice that the djangocon.eu videos aren't there. 2011's are here:
> http://blip.tv/djangocon-**europe-2011
> 2012's can be downloaded.
> You probably can google around for the rest.
>
> FYI, the 2012 videos are available here:

http://www.klewel.com/conferences/djangocon-2012/

Not sure what needs to be done to get them up on pyvideo.

Yours,
Russ Magee %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.