Offset/limit and looping over a large dataset

[Stefano Crosta]

On Tuesday, March 26, 2013 10:47:34 PM UTC+1, Petite Abeille wrote:
>
>
> On Mar 26, 2013, at 10:03 PM, Alex Gaynor > 
> wrote: 
>
> > For what it's worth, SQL2011 does define OFFSET, finally. 
>
> Perhaps worthwhile mentioning as well : 
>
> "Do not try to implement a scrolling window using LIMIT and OFFSET. Doing 
> so will become sluggish as the user scrolls down toward the bottom of the 
> list." 
> -- Scrolling Cursor, What Not To Do 
> http://www.sqlite.org/cvstrac/wiki?p=ScrollingCursor 
>
> Just saying...


This is already getting slightly off-topic wrt the original discussion on 
EXISTS/IN, so I allowed myself to change the subject.

But this is another important matter I think, and I've already seen mention 
of server-side cursors in these threads. 

I have a question on 
SO 
http://stackoverflow.com/questions/14144408/memory-efficient-constant-and-speed-optimized-iteration-over-a-large-table-in
 
about exactly that problem, it would be great to see server-side cursors 
and/or a smarter way to iterate make it into core. I hacked my procedure by 
I'm sure there are better ways.
I'm decently versed in python and have come to learn django internals, but 
I am zero on database side so I'm finding it difficult to propose a real 
contribution (the best effort I could put together is in the SO question). 
If nudged, I can try and help.

thanks!

Stefano

 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Offset/limit and looping over a large dataset

[Aymeric Augustin]

On 27 mars 2013, at 09:10, Stefano Crosta  wrote:

> it would be great to see server-side cursors and/or a smarter way to iterate 
> make it into core. 

FYI this topic is tracked here: https://code.djangoproject.com/ticket/16614

-- 
Aymeric.



-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Offset/limit and looping over a large dataset

[Anssi Kääriäinen]

On 27 maalis, 10:10, Stefano Crosta  wrote:
> On Tuesday, March 26, 2013 10:47:34 PM UTC+1, Petite Abeille wrote:
>
> > On Mar 26, 2013, at 10:03 PM, Alex Gaynor >
> > wrote:
>
> > > For what it's worth, SQL2011 does define OFFSET, finally.
>
> > Perhaps worthwhile mentioning as well :
>
> > "Do not try to implement a scrolling window using LIMIT and OFFSET. Doing
> > so will become sluggish as the user scrolls down toward the bottom of the
> > list."
> > -- Scrolling Cursor, What Not To Do
> >http://www.sqlite.org/cvstrac/wiki?p=ScrollingCursor
>
> > Just saying...
>
> This is already getting slightly off-topic wrt the original discussion on
> EXISTS/IN, so I allowed myself to change the subject.
>
> But this is another important matter I think, and I've already seen mention
> of server-side cursors in these threads.

Server side cursor would be useful in many situations. However, it is
hard to implement server side cursors in a way the ORM could use them
automatically. See https://code.djangoproject.com/ticket/16614 for
details.

The usual problem with LIMIT/OFFSET is pagination and going deep into
pages list. Server side cursors would only help if you could somehow
keep the same cursor open between requests for different pages.

The solution to large resultset pagination is to sort by some index
you have, and then when you go to next page, you do this:
 
SomeModel.objects.filter(indexedcol__gt=prev_pages_last_obj.indexedcol).order_by('indexedcol')
[0:PAGE_SIZE]
This operation can use the index and is a very efficient way to
retrieve pages. The downside is that this only works for unique
indexed columns (or "unique enough" that duplicates do not matter in
practice). Also, you can't easily give page numbers for pagination,
you can only have links "first, previous, next, last". Still, if you
have a lot of objects then normal pagination of "count all, give links
by page numbers" simply does not scale. The count(*) itself can be too
expensive for large resultsets.

Having in-built support for this type of pagination is something I
would like to see in Django.

If you want to implement an automatically refreshed scrolling window
using AJAX and want the ability to sort by any column and do so
efficiently, then you will need to use server side cursors and have
some way to get the same cursor back for different AJAX requests. In
practice you would need some sort of connection pool where you could
store a connection for reuse, and ask the same connection back when
next request arrives. This seems complex to implement correctly and
doesn't seem like something that belongs into Django...

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Offset/limit and looping over a large dataset

[Stefano Crosta]

Thanks Aymeric, 
that's true, and I should have put that link too.
sadly "akaariai" who's the only one who seemed to understand a bit about it 
(and is a core dev) did not seem interested in bringing this any forward, 
so I thought I'd try to raise some interest again!

On Wednesday, March 27, 2013 9:21:51 AM UTC+1, Aymeric Augustin wrote:
>
> On 27 mars 2013, at 09:10, Stefano Crosta > 
> wrote: 
>
> > it would be great to see server-side cursors and/or a smarter way to 
> iterate make it into core. 
>
> FYI this topic is tracked here: 
> https://code.djangoproject.com/ticket/16614 
>
> -- 
> Aymeric. 
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Offset/limit and looping over a large dataset

[Anssi Kääriäinen]

On 03/27/2013 10:53 AM, Stefano Crosta wrote:

Thanks Aymeric,
that's true, and I should have put that link too.
sadly "akaariai" who's the only one who seemed to understand a bit 
about it (and is a core dev) did not seem interested in bringing this 
any forward, so I thought I'd try to raise some interest again!
The problem is that server side cursors will need dedicated API, and 
there will be database specific problems when using server side cursors. 
On SQLite changes done to rows are visible in the results, on other 
databases not. On PostgreSQL you will need to use WITH HOLD cursors if 
you want to use server side cursors outside transactions (that is, in 
autocommit mode normal server side cursors do not work). When using WITH 
HOLD cursors you must close the cursor explicitly or you will have 
cursor leak... And on MySQL WITH HOLD cursors aren't available at all, 
so you must be in transaction to use server side cursors.


Oracle seems to be the only core DB that will work without problems. In 
fact, using .iterator() on Oracle already works without memory problems.


Maybe the dedicated API could be adding two new keywords to .iterator(): 
server_side_cursor, and with_hold. with_hold=True implies 
server_side_cursor=True. If you use with_hold=True you are responsible 
for closing the iterator, too. The behaviour of server_side_cursor and 
with_hold is database specific - it will be impossible to abstract the 
differences away.


 - Anssi

--
You received this message because you are subscribed to the Google Groups "Django 
developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Offset/limit and looping over a large dataset

[Stefano Crosta]

Thanks Anssi for both your replies. These db interactions definitely are 
really beyond the scope of my knowledge, unluckily.

I did try and implement an iterator such as the one that you describe in 
your other message, based on index column filtering - what I dislike is 
that it's an additional function getting a qs parameter rather then 
naturally sitting in the queryset and (thus?) I did not manage to make it 
very generic...  Eg. some querysets don't even contain the key column 
anymore.. (the outlines of these are in my SO question but ). I'm sure it 
can be done better, especially if it's part of the queryset rather then an 
external function that gets an evaluated qs. I still manage to use it both 
for pagination and for very complex functions that I can't run in the DB.
As you say, it would still be nice to see this kind of generic iterator in 
core, even known the limitations (heck, the standard iterator just exploses 
the process by memory usage, so it's not that much better :) ).

WITH HOLD does not look like a safe solution if you have to explicitly 
close it (would need a separate thread with timeout maybe?), but I also 
read some discussions about creating a connection pool and persistent 
connections, so maybe at some point all these issues will find a common 
base for a solution..

As usual, I'll eagerly keep reading these posts and try to play with code 
on my own hoping to be able to contribute at some point..

Stefano

On Wednesday, March 27, 2013 10:36:02 AM UTC+1, Anssi Kääriäinen wrote:
>
> On 03/27/2013 10:53 AM, Stefano Crosta wrote: 
> > Thanks Aymeric, 
> > that's true, and I should have put that link too. 
> > sadly "akaariai" who's the only one who seemed to understand a bit 
> > about it (and is a core dev) did not seem interested in bringing this 
> > any forward, so I thought I'd try to raise some interest again! 
> The problem is that server side cursors will need dedicated API, and 
> there will be database specific problems when using server side cursors. 
> On SQLite changes done to rows are visible in the results, on other 
> databases not. On PostgreSQL you will need to use WITH HOLD cursors if 
> you want to use server side cursors outside transactions (that is, in 
> autocommit mode normal server side cursors do not work). When using WITH 
> HOLD cursors you must close the cursor explicitly or you will have 
> cursor leak... And on MySQL WITH HOLD cursors aren't available at all, 
> so you must be in transaction to use server side cursors. 
>
> Oracle seems to be the only core DB that will work without problems. In 
> fact, using .iterator() on Oracle already works without memory problems. 
>
> Maybe the dedicated API could be adding two new keywords to .iterator(): 
> server_side_cursor, and with_hold. with_hold=True implies 
> server_side_cursor=True. If you use with_hold=True you are responsible 
> for closing the iterator, too. The behaviour of server_side_cursor and 
> with_hold is database specific - it will be impossible to abstract the 
> differences away. 
>
>   - Anssi 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Is file based cache is safe for concurrent process?

[Ivan]

Hi all.
Can anyone tell, does django.cache locks file for writing by concurrent 
process?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: URL dispatcher fallthrough?

[Tom Christie]

For what it's worth I'd be against this proposal as it stands.

* I haven't seen anything anything that convinces me that a single wrapping 
view isn't a reasonable alternative in the examples cited.
* A `UrlNotMatched` exception sounds like the potential source of 
incredibly non-obvious bugs and surprising behavior.
* Allow a single HTTP call to end up calling into multiple views just seems 
like it would a fundamentally bad design decision.

I really don't see the trade-off of allowing this type of behavior to be 
worth the cost of breaking the current mental model of what a view is and 
does.

Just my personal opinion of course :)

  Tom

On Tuesday, 26 March 2013 17:49:07 UTC, Val Neekman wrote:
>
> +1 here! 
>
> It might also be a great feature to run the 404 through before sending 
> that email out. 
> A replacement and/or complement to IGNORABLE_404_URLS 
>
> Val 
>
>
> On Tue, Mar 26, 2013 at 1:25 PM, Loic Bistuer 
> > 
> wrote: 
> > +1 for me. 
> > 
> > Having a catchall view under a single URL pattern is a tightly coupled 
> > system, more so than allowing independent views to "test the water". 
> > 
> > Django core relies on middleware hacks because the URL dispatcher is 
> missing 
> > this very feature. Having this in core would allow a cleaner 
> implementation 
> > of the fallback mechanisms in contrib.flatpages and contrib.redirects. 
> > 
> > -- 
> > Loic 
> > 
> > On Mar 19, 2013, at 11:18 PM, Adrian Holovaty 
> > > 
> wrote: 
> > 
> > I'd rather not add this to the framework, as it's already possible 
> > with a "wrapper" view (as others have suggested). And on a more 
> > theoretical level, it introduces more coupling between URL patterns 
> > and views. 
> > 
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups 
> > "Django developers" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> an 
> > email to django-develop...@googlegroups.com . 
> > To post to this group, send email to 
> > django-d...@googlegroups.com. 
>
> > Visit this group at 
> http://groups.google.com/group/django-developers?hl=en. 
> > For more options, visit https://groups.google.com/groups/opt_out. 
> > 
> > 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Is file based cache is safe for concurrent process?

[Russell Keith-Magee]

You've already asked this on django-users. Django-developers is for
discussing the development of Django itself -- it shouldn't be used as
"second tier" support if you don't get the answer you want on django-users.

Yours,
Russ Magee %-)

On Wed, Mar 27, 2013 at 8:16 PM, Ivan  wrote:

> Hi all.
> Can anyone tell, does django.cache locks file for writing by concurrent
> process?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers?hl=en
> .
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: URL dispatcher fallthrough?

[meric]

Hi Tom, interested to see why you think a single wrapping view is a 
reasonable alternative for the example I showed above, where you have 
following list of URLs:

//  # front page for country
// / # list of schools and companies with activities in 
that industry, in that country
/// # list of industries company has activities in, in 
that country
 # list of activities company has in that 
industry, in that country
/// # list of activities school has in that country
 # list of activities school has, in that 
industry
/ / # list of schools and companies with activities in that 
industry, in all countries
// # list of industries company has, globally
/// # list of activities company has in that industry, 
globally
// # list of industries the school is involved in, globally
///  # list of activities school has in that industry, 
globally

What is the reasonable alternative? The only way I can think how a wrapping 
view will be able to handle this case, is to write your own router like 

router_view1(slug1=None)
router_view2(slug1=None, slug2=None)
router_view3(slug1=None, slug2=None, slug3=None)

In each case you'll have lots of ifs, try..excepts, for each model, and 
then to appropriate view. For the second and third router_views you'd have 
to have the same thing repeated, but this time it is nested.

Yet, in urls.py, all you have is:

(r'/')
(r'//')
(r'///')

Forcing the user to handle complex logic like this (each user probably 
needs to do it differently), which is not easy to test due to many branches 
involved, IMHO is a big cost.

There's also the question of view_middleware not being applied for the 
appropriate view unless the user specifically looks into how middleware 
gets called and manages handling view middleware themselves in the 
different router_views.

Perhaps you have a better way to implement this than what I can think of? 
This is how we do it in our company, and it'd be great if this can be 
improved.

On Thursday, March 28, 2013 3:28:10 AM UTC+11, Tom Christie wrote:
>
> For what it's worth I'd be against this proposal as it stands.
>
> * I haven't seen anything anything that convinces me that a single 
> wrapping view isn't a reasonable alternative in the examples cited.
> * A `UrlNotMatched` exception sounds like the potential source of 
> incredibly non-obvious bugs and surprising behavior.
> * Allow a single HTTP call to end up calling into multiple views just 
> seems like it would a fundamentally bad design decision.
>
> I really don't see the trade-off of allowing this type of behavior to be 
> worth the cost of breaking the current mental model of what a view is and 
> does.
>
> Just my personal opinion of course :)
>
>   Tom
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Is file based cache is safe for concurrent process?

[Ivan]

Yes, you are right. But I asked for another reason.
For example, I see, that werkzeug's cache write cache trough temporary file 
https://github.com/mitsuhiko/werkzeug/blob/master/werkzeug/contrib/cache.py#L660
But Django cache do not use neither, tmp file or file locking.
So, multiple processes can write a file simultaneously. Also, other 
processes can obtain incomplete data.

I think it's a bug, so I asked a question in this group (django-developers).



четверг, 28 марта 2013 г., 0:55:20 UTC+2 пользователь Russell Keith-Magee 
написал:
>
>
> You've already asked this on django-users. Django-developers is for 
> discussing the development of Django itself -- it shouldn't be used as 
> "second tier" support if you don't get the answer you want on django-users.
>
> Yours,
> Russ Magee %-)
>
> On Wed, Mar 27, 2013 at 8:16 PM, Ivan >wrote:
>
>> Hi all.
>> Can anyone tell, does django.cache locks file for writing by concurrent 
>> process? 
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-develop...@googlegroups.com .
>> To post to this group, send email to 
>> django-d...@googlegroups.com
>> .
>> Visit this group at 
>> http://groups.google.com/group/django-developers?hl=en.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>  
>>  
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: URL dispatcher fallthrough?

[meric]

Tom, you're right about the second and third points though. If the user 
perform any operation with side effect that requires writing to the 
database before the view checks whether the keyword arguments are 
appropriate and decide to raise DoesNotResolve, it can definitely be a 
source of non-obvious bugs and surprising behaviour. For example, a counter 
increment in the view to count how many times the view has been visited, 
placed before checking for whether to raise DoesNotResolve. Multiple such 
views get executed, multiple view counters incremented for one HTTP 
connection.

I can only think of adding a stern warning to the documentation something 
along the lines of "Must not perform any operation requiring a database 
write or any other operation with side effects before the check for 
DoesNotResolve is made.".

Eric

On Thursday, March 28, 2013 3:28:10 AM UTC+11, Tom Christie wrote:
>
> * A `UrlNotMatched` exception sounds like the potential source of 
> incredibly non-obvious bugs and surprising behavior.
> * Allow a single HTTP call to end up calling into multiple views just 
> seems like it would a fundamentally bad design decision.
>
> I really don't see the trade-off of allowing this type of behavior to be 
> worth the cost of breaking the current mental model of what a view is and 
> does.
>
> Just my personal opinion of course :)
>
>   Tom
>
>>  
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

why does django.db.signals map to django.core.signals

[Joseph Curtin]

Hey all, 

   I'm working on adding that pre_syncdb signal, I've come across a 
rother peculiar detail. When I import django.db.signals, it maps to 
django.core.signals. Am I missing something here? I know for example, 
django.conf generates the settings import. Can someone point me to tho 
logic of these path edits?

Cheers,
-Joseph Curtin

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Is file based cache is safe for concurrent process?

[Russell Keith-Magee]

On Thu, Mar 28, 2013 at 7:13 AM, Ivan  wrote:

>
> Yes, you are right. But I asked for another reason.
> For example, I see, that werkzeug's cache write cache trough temporary
> file
> https://github.com/mitsuhiko/werkzeug/blob/master/werkzeug/contrib/cache.py#L660
> But Django cache do not use neither, tmp file or file locking.
> So, multiple processes can write a file simultaneously. Also, other
> processes can obtain incomplete data.
>
> I think it's a bug, so I asked a question in this group
> (django-developers).
>

Ok - *that* is a different matter. For future reference, if you're
suggesting that something might be a bug, it's helpful to make that clear
in your question.

As I said in my response on django-users, the file based cache is primarily
there as a proof of concept that the caching API can be targeted at
multiple backends. However, if someone were to propose a patch to add file
locking or temp file write-throughs, I don't imagine it would be turned
down.

So - feel free to open a ticket for this; and if you want to try your hand
at a patch, this should be a relatively simple patch to create (although
tests could be a little bit complex). As a helper, the
django.core.files.locks module already contains some OS-abstracted code for
file locking.

Yours,
Russ Magee %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: why does django.db.signals map to django.core.signals

[Jeremy Dunck]

I'm not sure what you mean by "maps to".

The basic signal framework is in django.dispatcher. If you're
implementing https://code.djangoproject.com/ticket/11398 then you
probably want to import: from django.dispatcher import Signal.

django.db imports django.core.signals because it is a *consumer* of
the core signals request_finished and request_started.

You probably want to add to django.db.models.signals rather than
django.db in any case.
https://github.com/django/django/blob/master/django/db/models/signals.py



On Wed, Mar 27, 2013 at 4:06 PM, Joseph Curtin  wrote:
> Hey all,
>
>I'm working on adding that pre_syncdb signal, I've come across a rother
> peculiar detail. When I import django.db.signals, it maps to
> django.core.signals. Am I missing something here? I know for example,
> django.conf generates the settings import. Can someone point me to tho logic
> of these path edits?
>
> Cheers,
> -Joseph Curtin
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Is file based cache is safe for concurrent process?

[Ivan]

> As I said in my response on django-users, the file based...

-- Oh, I didn't see it yet. Thanks.

> if you want to try your hand at a patch, this should be a relatively 
simple patch to create (although tests could be a little bit complex).

-- yes, it's a good idea, but i need first to study the Contributing guide 
https://docs.djangoproject.com/en/dev/internals/contributing/writing-code/

Thanks for answer.



четверг, 28 марта 2013 г., 2:23:52 UTC+2 пользователь Russell Keith-Magee 
написал:
>
>
> On Thu, Mar 28, 2013 at 7:13 AM, Ivan >wrote:
>
>>
>> Yes, you are right. But I asked for another reason.
>> For example, I see, that werkzeug's cache write cache trough temporary 
>> file 
>> https://github.com/mitsuhiko/werkzeug/blob/master/werkzeug/contrib/cache.py#L660
>> But Django cache do not use neither, tmp file or file locking.
>> So, multiple processes can write a file simultaneously. Also, other 
>> processes can obtain incomplete data.
>>
>> I think it's a bug, so I asked a question in this group 
>> (django-developers).
>>
>
> Ok - *that* is a different matter. For future reference, if you're 
> suggesting that something might be a bug, it's helpful to make that clear 
> in your question. 
>
> As I said in my response on django-users, the file based cache is 
> primarily there as a proof of concept that the caching API can be targeted 
> at multiple backends. However, if someone were to propose a patch to add 
> file locking or temp file write-throughs, I don't imagine it would be 
> turned down.
>
> So - feel free to open a ticket for this; and if you want to try your hand 
> at a patch, this should be a relatively simple patch to create (although 
> tests could be a little bit complex). As a helper, the 
> django.core.files.locks module already contains some OS-abstracted code for 
> file locking.
>
> Yours,
> Russ Magee %-)
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: why does django.db.signals map to django.core.signals

[Russell Keith-Magee]

Hi Joseph,

There's no logic that does any such thing. There's nothing unusual or
Django-specific going on here - it's just how Python imports work.

There is a module called django.core.signals. It contains the definitions
for a bunch of signals related to the core operation of Django, like the
'connection closed' signal.

There is also a module called django.db. This module contains the logic for
accessing the database.

django.db (strictly, django/db/__init__.py) contains the line "from
django.core import signals". That means that the django.db module contains
a "signals" object in its namespace.

Due to the way Python imports work, you can therefore call "from django.db
import signals" -- because there *is* a signals object in the django.db
module namespace.

However, The django.db module contains an __all__ declaration, so if you
call "from django.db import *", you *won't* get a signals object -- it's
not explicitly included in the symbol export list for the module.

As Jeremy said, this is all a moot point anyway. The post_syncdb signal is
defined  in django.db.models.signals; if you're adding a post_syncdb
signal, *that* is where it should be defined, and *that* is where you
should be importing it from.

Yours,
Russ Magee %-)

On Thu, Mar 28, 2013 at 9:29 AM, Joseph Curtin wrote:

> By 'maps to,' I mean that django.core.signals.__file__ clobbers
> django.db.signals.__file__.
>
> Can you point me to the logic that does this? The 'consumer' logic?
>
>
> On Wed, Mar 27, 2013 at 8:35 PM, Jeremy Dunck  wrote:
>
>> I'm not sure what you mean by "maps to".
>>
>> The basic signal framework is in django.dispatcher. If you're
>> implementing https://code.djangoproject.com/ticket/11398 then you
>> probably want to import: from django.dispatcher import Signal.
>>
>> django.db imports django.core.signals because it is a *consumer* of
>> the core signals request_finished and request_started.
>>
>> You probably want to add to django.db.models.signals rather than
>> django.db in any case.
>> https://github.com/django/django/blob/master/django/db/models/signals.py
>>
>>
>>
>> On Wed, Mar 27, 2013 at 4:06 PM, Joseph Curtin 
>> wrote:
>> > Hey all,
>> >
>> >I'm working on adding that pre_syncdb signal, I've come across a
>> rother
>> > peculiar detail. When I import django.db.signals, it maps to
>> > django.core.signals. Am I missing something here? I know for example,
>> > django.conf generates the settings import. Can someone point me to tho
>> logic
>> > of these path edits?
>> >
>> > Cheers,
>> > -Joseph Curtin
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> Groups
>> > "Django developers" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> an
>> > email to django-developers+unsubscr...@googlegroups.com.
>> > To post to this group, send email to django-developers@googlegroups.com
>> .
>> > Visit this group at
>> http://groups.google.com/group/django-developers?hl=en.
>> > For more options, visit https://groups.google.com/groups/opt_out.
>> >
>> >
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-developers@googlegroups.com.
>> Visit this group at
>> http://groups.google.com/group/django-developers?hl=en.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>>
>
>
> --
> -Joey Curtin
> http://www.jbcurtin.com
> github 
> @jbcurtin
> **
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers?hl=en
> .
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

ATOMIC_REQUESTS, cache invalidation and concurrent process

[Ivan]

With ATOMIC_REQUESTS, after cache invalidation, the concurrent process can 
re-create cache old data, between deleting of cache and commit.

Perhaps this problem should be solved by repeat cache deleting after commit.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.