Re: ORM Optimization for filtering by related existence

2014-08-14 Thread Anssi Kääriäinen
On Thu, 2014-08-14 at 02:39 +0300, Shai Berger wrote: > > So, both filters must target the same join > > I don't see how this follows -- separate joins will each have the same set of > related records. True, the query results will be correct for exists lookup (but not for most other lookups if u

Re: ORM Optimization for filtering by related existence

2014-08-13 Thread Shai Berger
On Wednesday 13 August 2014 09:55:33 Anssi Kääriäinen wrote: > > But, how about .filter(Q(somefield__exists=False) | > Q(somefield__othercol=2))? This one should return results only if there > are no somefield related instances at all, or if there are related > instances, then at least one of thos

Re: ORM Optimization for filtering by related existence

2014-08-12 Thread Anssi Kääriäinen
On Wed, 2014-08-13 at 04:38 +0300, Shai Berger wrote: > On Friday 08 August 2014 10:41:20 Anssi Kääriäinen wrote: > > > > As for addition of .exists lookup - how > > should .filter(somefield__exists=False, somefield__othercol=2) work? > > It should be noted that __exists is a little special. For

Re: ORM Optimization for filtering by related existence

2014-08-12 Thread Shai Berger
On Friday 08 August 2014 10:41:20 Anssi Kääriäinen wrote: > > As for addition of .exists lookup - how > should .filter(somefield__exists=False, somefield__othercol=2) work? It should be noted that __exists is a little special. For the False case above, .filter(somefield__exists=False, somefield

Re: ORM Optimization for filtering by related existence

2014-08-08 Thread Anssi Kääriäinen
IIRC relation lookups don't yet use the custom lookups facility, so just writing a custom lookup might not work. We should turn the related lookups to use the new lookup system, too, but that wasn't done as the way relation lookups work is a bit special inside the ORM. As for addition of .exists l

Re: ORM Optimization for filtering by related existence

2014-08-08 Thread Josh Smeaton
Just to be clear, are you (Curtis) suggesting that the isnull lookup should do an EXISTS when looking at a join? I don't think that's a very good idea - having that construct can be extremely useful. I think adding an __exists lookup to relationships would be a great addition - but that's where

Re: ORM Optimization for filtering by related existence

2014-08-08 Thread Anssi Kääriäinen
Unfortunately the problem is a bit deeper in the ORM. In general, when filtering against a multivalued relation (reverse foreign key or m2m relation) Django uses joins instead of subqueries. This can result in duplicating rows if multiple related rows match the filter. The fix recommended in the do

Re: ORM Optimization for filtering by related existence

2014-08-07 Thread Curtis Maloney
Can you create a PR, with docs and tests? If not, we'll need to find someone brave enough to delve into the ORM and add this [quite valid, IMHO] optimisation. -- C On 8 August 2014 11:16, David Butler wrote: > > > On Thursday, August 7, 2014 6:48:22 PM UTC-5, Tim Graham wrote: >> >> Does .fi

Re: ORM Optimization for filtering by related existence

2014-08-07 Thread Collin Anderson
I personally use qs = qs.filter(somefield__gte=1) (works if the id field is an integer) -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+u

Re: ORM Optimization for filtering by related existence

2014-08-07 Thread David Butler
On Thursday, August 7, 2014 6:48:22 PM UTC-5, Tim Graham wrote: > > Does .filter(somefield__isnull=False) not work for what you're trying to > do? > If I do .filter(somefield__isnull=False) it tries to do a LEFT OUTER JOIN on the table for somefield instead of a WHERE EXISTS (...) and if that

Re: ORM Optimization for filtering by related existence

2014-08-07 Thread Josh Smeaton
I think the idea is good, but the implementation you have here isn't desirable. Maybe you could experiment with writing your own custom lookup (https://docs.djangoproject.com/en/dev/howto/custom-lookups/) and see if it fits into the framework that way. AFAIK, anything requiring .extra should be

Re: ORM Optimization for filtering by related existence

2014-08-07 Thread Tim Graham
Does .filter(somefield__isnull=False) not work for what you're trying to do? On Thursday, August 7, 2014 7:43:07 PM UTC-4, David Butler wrote: > > It would be nice if there was some database level optimization for getting > objects that have related objects that exist. > > This is a slow filter:

ORM Optimization for filtering by related existence

2014-08-07 Thread Croepha
It would be nice if there was some database level optimization for getting objects that have related objects that exist. This is a slow filter: qs = [obj for obj in qs if qs.somefield_set.exists()] Could be faster with something like this: qs = qs.filter(somefield__exists=True) Here is some (r