Django ORM query syntax enhancement

[Alexey Zankevich]

Hi all,

This topic is related to the current ORM query syntax with underscores.
There are lots of arguing related to it, anyway it has pros and cons.

Let's take a concrete example of querying a model:

>>> 
GameSession.objects.filter(user__profile__last_login_date__gte=yesterday)


Pros:

1. The syntax is easy to understand
2. Can be extended with custom transforms and lookups

However, there are several cons:

1. Long strings is hard to read, especially if we have fields with 
underscores.
It's really easy to make a mistake by missing one:

>>> GameSession.objects.filter(user_profile__last_login_date__gte=yesterday)

Not easy to catch missing underscore between user and profile, is it? Even
though, it's not easy to say whether it should be "user_profile" attribute 
or
user.profile foreign key.

2. Query strings can't be reused, thus the approach violates DRY principle.
For example, we need to order results by last_login_date:

>>> 
GameSession.objects.filter(user__profile__last_login_date__gte=yesterday) \
.order_by('user__profile__last_login_date')

We can't keep user__profile_login_date as a variable as in the first part 
of the
expression we use a keyword argument, meanwhile in the second part - just a 
string. And thus we just have to type query path twice.

3. Lookup names not natural to Python language and require to be remembered 
or
looked up in documentation. For example, "__gte" or "__lte" lookups tend to 
be
confused with "ge" and "le" due to similarity to methods "__ge__" and 
"__le__".

4. Lookup keywords limited to a single argument only, very inconvenient when
necessary to filter objects by range.

I was thinking a lot trying to solve those issues, keeping in mind Django
approaches. Finally I came up with solution to extend Q objects with dot
expression syntax:

>>> GameSession.objecs.filter(Q.user.profile.last_login_date >= yesterday)

Q is a factory instance for old-style Q objects. Accessing attribute by dot
returns a child factory, calling factory will instantiate old-style Q 
object.

>>> Q


>>> Q.user.profile


>>> Q(user__name='Bob')


It overrides operators, so comparing factory with value returns a related Q
object:

>>> Q.user.name == 'Bob'


Factory has several helper functions for lookups which aren't related to any
Python operators directly:

>>> Q.user.name.icontains('Bob')


And helper to get query path as string, which requred by order_by or
select_related queryset methods: 

>>> Q.user.profile.last_login_date.get_path()
'user__profile__last_login_date'

You can check implementation and more examples here
https://github.com/Nepherhotep/django-orm-sugar

How it solves issues:

#1. Dots hard to confuse with underscores
#2. Query paths can be reused:

>>> factory = Q.user.profile.last_login_date
>>> query = GameSession.objects.filter(factory >= yesterday)
>>> query = query.order_by(factory.get_path())

#3. Not neccessary to remember most of lookup names and use comparison 
operators
instead.
#4. Possible to use multiple keyword arguments:

>>> Q.user.profile.last_login_date.in_range(from_date, to_date)



This approach looked the best for me due to several reasons:

1. It's explicit - it doesn't do anything but generating appropriate Q 
object. 
The result of comparison can be saved as Q object variable.

2. It's short - variants with using model for that will look much longer, 
when
joining two or more filters:

>>> GameSession.objects.user.profile_last_login_date >= yesterday  # awkward

3. Implementation will not require to change querset manager or model 
classes

4. Will still allow to use filters and Q class in the old way:

>>> q = Q(user__profile__last_login_date__gte=yesterday)

or

>>> 
GameSession.objects.filter(user__profile__last_login_date__gte=yesterday)

I'd like to make it as a part of Django ORM syntax and it will not be hard 
to
do, especially taking into account the library is already done and working. 
Anyway, I need your thought about the idea in general, as well as about
particular things like chosen method names - "get_path", "in_range" and etc.
As next step I can create a ticket in the issue tracker, or prepare DEP 
first.
In latter case I need to find a shepherd to work with.

Best regards,
Alexey

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ceeb43c7-b44f-4e81-bd8f-5997df9edb43%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Enhancement to the call_command function to allow the use of modules as arguments

[Mike Lissner]

I recently filed a bug about this 
 and was redirected here for 
discussion. While using call_command (which simplifies calling management 
commands), it occurred to me that the API is a little strange. It currently 
is designed to work like this:

call_command('my_command', *args, **kwargs)

The problem here is that you pass a string into your command, and then 
Django does the magic of converting that into something that can be 
imported, and then uses argparse to parse args and kwargs. I think a better 
API would be:

from my_project.my_app.management.commands import my_command
call_command(my_command, *args, **kwargs)

There are three big advantages of this. First, if you ever change the name 
of your command, a good IDE can realize that it needs to update the import 
statements, and that'll happen automatically. This is good and important in 
larger projects where you can't keep all the code in your head.

Second, this allows code completion from your IDE, making it less likely to 
make a typo or a mistake. Another good thing.

Third, this reduces the amount of string to module importing that Django 
does, and less magic is generally good.

In terms of process, I propose we follow the standard deprecation process. 
At first, it should accept either input, and issue a warning. Over time, it 
should only allow modules.

The bug I filed was closed saying that, there wasn't "sufficient 
justification for the additional complexity that would be required." But 
this should only take a few lines of code to check what the argument is 
(string or module), and then to do the conversion.

I'm curious what the group thinks. This seems like a clear improvement to 
the API to me, but perhaps there's something I'm missing. Happy to discuss.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/9a613577-e0e2-49c8-9a56-ff2a5cb8c94a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Enhancement to the call_command function to allow the use of modules as arguments

[Aymeric Augustin]

Hello Mike,

I saw your ticket a few days ago, thanks for bringing it up here.

> While using call_command (which simplifies calling management commands), it 
> occurred to me that the API is a little strange. It currently is designed to 
> work like this:
> 
> call_command('my_command', *args, **kwargs)

The API is designed like this because it performs the equivalent of 
`django-admin my_command [arg] [arg] […] [--kwarg] [--kwarg] […]`.

> The problem here is that you pass a string into your command, and then Django 
> does the magic of converting that into something that can be imported, and 
> then uses argparse to parse args and kwargs. I think a better API would be:
> 
> from my_project.my_app.management.commands import my_command
> call_command(my_command, *args, **kwargs)

In general I would suggest to factor your business logic into a function that 
you can call independently from the management command. This is better for 
testing as well.

Then the management command is just a thin wrapper that validates the command 
line parameters and calls that function with corresponding arguments.

> There are three big advantages of this. First, if you ever change the name of 
> your command, a good IDE can realize that it needs to update the import 
> statements, and that'll happen automatically. This is good and important in 
> larger projects where you can't keep all the code in your head.

Management commands are usually called from outside the Python project. If you 
rename a management command, “Find -> Replace All” in your source code is 
probably going to take less time than updating all external scripts or 
procedures that depend on it.

> Second, this allows code completion from your IDE, making it less likely to 
> make a typo or a mistake. Another good thing.

Fair enough. [I don’t use an IDE myself. I write tests ;-)]

> Third, this reduces the amount of string to module importing that Django 
> does, and less magic is generally good.

We won’t gain much because django-admin my_command will still need to find 
my_command in one of the installed apps.

> In terms of process, I propose we follow the standard deprecation process. At 
> first, it should accept either input, and issue a warning. Over time, it 
> should only allow modules.

I’m -1 on removing the Python API that’s equivalent to `django-admin 
my_command`. It’s needed for testing management commands that override other 
management commands.

An alternative would be to split call_command() in two functions: one that 
locates the command and one that figures out the default values of the 
arguments. The latter is the piece of code you want to reuse.

> The bug I filed was closed saying that, there wasn't "sufficient 
> justification for the additional complexity that would be required." But this 
> should only take a few lines of code to check what the argument is (string or 
> module), and then to do the conversion.

Django used to have lots of APIs that accepted multiple types of arguments. We 
tend to remove them when possible because they’re a source of bugs and 
confusions and also because there should be one way to do each task. I’m -0 on 
taking the opposite route here. This is a theoretical argument that I’ll 
happily withdraw if a quick type check is the practical option :-)

-- 
Aymeric.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/30336028-2F0E-44B6-9BE3-250E3FC2E9BD%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.

Re: Django Admin New Look

[elky]

Hi guys,

I made double work for vector icons - now we have Font and SVG icons. Let's 
choose what to use.

*Quoting my comment 
 from SVG 
discussion here:*

Well, only week ago I was 100% happy with font icons. But after Collin's 
comment I vote for SVG ( with svg as a source).
Below I listed advantages/disadvantages for both options. 

 

*Font*


   - Customizing. Changing icon color with CSS
  - Additional 100KB (font file + css)
  - Lots of changes in CSS, HTML and JS
  - To support IE8: just add *.ttf file (+ 70KB)
  - Code Diff: 
  
​https://github.com/django/django/compare/master...elky:font-icons?diff=unified&name=font-icons
   

*SVG* 


   - Much less changes
  - Code is elegant - I actually *just replaced* gifs with svg in CSS. 
  No pseudo-elements. No alignment tweaks
  - 25 additional files (requests) but just 19KB in total
  - To support IE8: add fallback in CSS (since we already have gif 
  icons in the repo, my suggestion is to show them in IE8)
  - Code Diff: 
  
​https://github.com/django/django/compare/master...elky:svg-icons?diff=unified&name=svg-icons
   
And the final and main argument to choose SVG is that some apps use a bit 
overridden Django CSS classes so font approach may cause visual issues (I 
checked it with few CMS apps). So font approach is an extra headache for 
app developers. 


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/55509397-cab0-4298-a708-f89c4326fd83%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django ORM query syntax enhancement

[Josh Smeaton]

Hi Alexey,

I find this approach really interesting, and I don't mind the API you've 
created at all. Even if this doesn't make it into Django, I think there's 
enough utility in your library that other people would want to use it. I'm 
not going to comment on specifics just yet like method names, because I 
figure that if we can agree on an approach, the bike-shedding can come 
later.

So first things first. I think we can all, or at least mostly, agree that 
having an alternative syntax for building filters is a good idea. String 
based lookups are cool, and won't go away, but sometimes you want to have 
greater control over what kind of filter you want to build without having 
to register global lookups and transforms. I see similarities with 
prefetch_related. You can provide a string based field lookup, or use a 
Prefetch() object to give users more control over the behaviour of the 
prefetch.

I've been working on a patch (https://github.com/django/django/pull/5090) 
to unify Transforms and Expressions (Func specifically). The work behind 
that can be (and should be) extended to include Lookups -- making them 
fully fledged expression objects. That work will need to happen regardless. 
Once this patch is finished and lands, then we should be able to extend 
.filter() and .exclude() to allow expressions to be composed as filters:

GameSession.objects.filter(GreaterThan('user__profile__last_login_date', 
yesterday))
# or, if last_login was a datetime, and we wanted to compare the date part 
only
GameSession.objects.filter(Equal(Date('user__profile__last_login'), datetime
.now().date))
# or if we wanted to implement __gt__ and __eq__ etc:
GameSession.objects.filter(Date('user__profile__last_login') == datetime.now
().date))

Loic and Anssi have also been working on alternative syntaxes 
(https://github.com/django/django/pull/4953):

GameSession.objects.filter(E('user__profile__last_login').date() == datetime
.now().date)


Article.objects.filter(E('headline').collate('fi') == 'Article1')

Both of these approaches still suffer from "the string problem" that you're 
trying to address, but minimises the final component by extracting the 
lookup and transform components as objects instead. So I think your idea 
here could nicely coexist:

GameSession.objects.filter(Equal(Date(Q.user.profile.last_login), datetime.
now().date))
GameSession.objects.filter(E(Q.user.profile.last_login).date() == datetime.
now().date)

Or, even building this into F expressions rather than Q expressions:

GameSession.objects.filter(Equal(Date(F.user.profile.last_login), datetime.
now().date))
GameSession.objects.filter(E(F.user.profile.last_login).date() == datetime.
now().date)

I think it may look better with F objects, considering they are *F*ield 
references, and since the Lookups (Equal/GTE) parts accept F expressions 
anyway. I'm not too concerned about this particular detail though.

A DEP is probably the right way to go here but I wonder if we should expand 
the scope to include alternative filter syntax as a whole 
(expressions/callable transforms) as well as the dot field reference 
notation you've proposed above. Then we can consider how all these things 
might work together, and clearly document why we've gone one way and not 
another. Obviously, alternatives can exist outside of core where the API is 
available.

I'll be happy to work as the shepherd if needed. But I'd also like some 
input from Loic and Anssi especially, as well as others in the core team 
with some interest in the ORM.

Regards,

On Sunday, 16 August 2015 23:18:26 UTC+10, Alexey Zankevich wrote:
>
> Hi all,
>
> This topic is related to the current ORM query syntax with underscores.
> There are lots of arguing related to it, anyway it has pros and cons.
>
> Let's take a concrete example of querying a model:
>
> >>> 
> GameSession.objects.filter(user__profile__last_login_date__gte=yesterday)
>
>
> Pros:
>
> 1. The syntax is easy to understand
> 2. Can be extended with custom transforms and lookups
>
> However, there are several cons:
>
> 1. Long strings is hard to read, especially if we have fields with 
> underscores.
> It's really easy to make a mistake by missing one:
>
> >>> 
> GameSession.objects.filter(user_profile__last_login_date__gte=yesterday)
>
> Not easy to catch missing underscore between user and profile, is it? Even
> though, it's not easy to say whether it should be "user_profile" attribute 
> or
> user.profile foreign key.
>
> 2. Query strings can't be reused, thus the approach violates DRY principle.
> For example, we need to order results by last_login_date:
>
> >>> 
> GameSession.objects.filter(user__profile__last_login_date__gte=yesterday) \
> .order_by('user__profile__last_login_date')
>
> We can't keep user__profile_login_date as a variable as in the first part 
> of the
> expression we use a keyword argument, meanwhile in the second part - just 
> a 
> string. And thus we just have to type query path twice.
>
> 3. Lookup names