On Tue, 2007-05-29 at 07:37 +0000, Paul Collier wrote: > Hello all! > > Continuing with the string of posts regarding this year's GSoC, I'm > pleased to be working on new caching functionality for Django's ORM > under the mentorship of Gary Wilson[1]! Big thanks to him and all the > people from the community who made this possible. Now I've just got to > hold up my end of the bargain ;) > > An updated version of my original proposal is available on Django's > wiki[2] and the project repository is available from Google Project > Hosting[3]. > > This project aims to condense common caching idioms into simple > QuerySet methods that deal with cached data intelligently, refill > expired cache, track instance modification/deletion, descend through > relationships, and so on. I've tried to make it accommodate the most > common use cases, be it dumping front-page news articles to cache or > keeping user profile data on hand when rendering a more dynamic site, > but I'd definitely love to hear about how cached data is really used > out in the field. > > Currently, three QuerySet methods, .cache(), .cache_related(), > and .cache_set() handle individual, relation, and aggregate data > caching, respectively. Each can take arguments to specify options like > expiry time. Signals are registered behind the scenes to sync the > cache, and there's also a proposal for transactional caching. Generic > relations will also benefit greatly from .cache_related().
One of my concerns I've had in the past when people have proposed this sort of work is how will object updates interact with cache consistency and correctness. The simplest example (solve this and you've solved them all, I suspect) is if the cache contains information about a Queryset qs and part of that queryset is object o1. If I update o1 in some other part of the code, what assumptions are made about qs? Possibilities are: (1) qs is invalidated from the cache and will be requeried from the database the next time it is needed (consistency + correctness) (2) qs doesn't change until it expires. The cache returns consistent results (in the sense that you can predict what will be returned if you know the state of the system), but not necessarily correct ones, because some or all of the objects in the queryset might have changed. (3) qs is updated somehow so that only necessary changes are requeried on the next access. I suspect this option doesn't scale well, but it's one possibility. Is this one of the things bouncing in your head? [...] > I'm also curious about the progress of QuerySet refactoring, which > would understandably have a huge impact on the code I'm writing. Actually I think this is going to have a lot less impact than you think. The elevator pitch for the QuerySet refactor is that is removes the SQL query consruction code from the QuerySet class and moves it all into another class (that is an attribute of QuerySet). However, all the Django-specific logic of QuerySets, including the methods like filter(), exclude(), etc, will not change. So a QuerySet class will pass off the SQL work and retrieval from the database to another class, but all user-code (including other Django code) still interacts with the QuerySet class itself, because that is responsible for things like turning SQL results back into objects or values or whatever. I would suspect you can avoid messing with the SQL construction that much (I just read through your code so far). What we can do is add a reliable __hash__ method or something similar to the Query class (the thing that does the SQL construction) so that instead of you having to replace QuerySet.iterator() and look at the query as it is constructed, you can just ask the QuerySet.query attribute what its hash is and use that to ascertain identity. I'm not sure what the comments about "working with relations" means in your iterator() implementation. Again, probably won't be hard to do that work by intercepting the QuerySet methods that are called, rather than worrying about the query construction. The good news/bad news (are you a glass half-full or glass half-empty kind of guy?) is that the QuerySet refactor is now my major project, as the unicode branch is winding down and we have to get this sucker done. Regards, Malcolm --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~----------~----~----~----~------~----~------~--~---
