[GSoC] Composite fields once again

[Michal Petrucha]

Hello everyone,

I'm happy to announce that throughout the summer I'll be officially
working on composite fields under the supervision of Andrew Godwin as
part of this year's GSoC.

I'll use this thread to post all my weekly status updates regarding
this project. This email gives a brief overview of the current status
and a rough outline of my plans, as well as a few questions I'd like
to raise with the community.

So, as far as currently available code goes, there's a github repo
containing my work from the GSoC two years ago, which is outdated and
incomplete, but still contains a large amount of working code, most of
which should be possible to adapt to the current state of Django.

The work can be outlined roughly in the following four steps:

1) create a solid implementation of virtual fields
2) refactor ForeignKey/OneToOneField to become virtual fields
3) implement CompositeField
4) make CompositeField work with as many additional parts of Django as
   possible, including GenericForeignKey and inspectdb
5) possibly implement a mechanism to modify the primary key of a model
   instance

All of the above steps are explained in more detail in the proposal I
submitted for the GSoC, available as a gist [1].

Now, for the questions I wanted to raise.


ForeignKey customization API


This one is mostly about the extent to which we want the internal
implementation of ForeignKey to affect the public API. To keep things
backwards compatible, attributes such as db_column or db_index will be
just passed to the auto-generated auxiliary field.

The question is, do we want to make it possible to specify a custom
auxiliary field to a ForeignKey instead of always creating one?

A related question, how should it be possible to customize the
database columns of composite ForeignKeys? Just make ForeignKey accept
a tuple instead of a string? Or just require the user to create the
fields (along with a CompositeField) by hand and pass that as the
auxiliary field? Any nicer possibility?

Either option is rather simple to implement, I just don't really have
a strong opinion, although I find both a little bit unpleasant.


GenericForeignKey and nontrivial field types


As I've indicated in my proposal, just casting any value to a string
and then performing a reversible transformation on such strings may
work well enough for string and integer database columns, not so much
for things like dates, timestamps IP addresses or other similar types.

Any ideas on how to make this work? Should I try to extend the backend
API to include explicit casts for each nontrivial column type to a
string representation equal to the one used by Python? Or should I
just document this as unsupported?


Updatable primary keys
--

This feature is not directly related to the main objective of this
project, which is to implement composite fields. It is just easier for
people to get into a situation where this might be required when using
composite fields.

Full support for this feature would require quite massive changes to
the collectors used cascading deletes -- they'd have to be generalized
to support cascading updates as well. This would introduce a lot of
complexity, obviously.

Jeremy Tillman voiced his opinion against this feature in a comment to
my proposal. He gives valid arguments -- an update of a primary key
would be an expensive operation with potential to kill the performance
of any apps doing it. However, the argument that it is easily done
with a Model.objects.filter(...).update(...) call is not entirely true
as a full implementation of this feature would also cascade. Moreover,
it would make this possible in the admin without much hassle.

So, seeing that there is at least one voice against this feature, I
think it's better to let the community decide if we want it at all.

Either way, it's highly unlikely I'd be able to deliver this feature
as part of this GSoC, the best I can promise is an initial proof of
concept implementation.


Michal


[1] https://gist.github.com/konk/5408673


signature.asc
Description: Digital signature

Re: [GSoC] Composite fields once again

[Luke Sneeringer]

On Jun 2, 2013, at 8:22 AM, Michal Petrucha  wrote:

> GenericForeignKey and nontrivial field types
> 
> 
> As I've indicated in my proposal, just casting any value to a string
> and then performing a reversible transformation on such strings may
> work well enough for string and integer database columns, not so much
> for things like dates, timestamps IP addresses or other similar types.
> 
> Any ideas on how to make this work? Should I try to extend the backend
> API to include explicit casts for each nontrivial column type to a
> string representation equal to the one used by Python? Or should I
> just document this as unsupported?

There's already a `db_type` method that you can override (that receives a 
`connection` object) for the actual database type. It's pretty easy to do 
something to the effect of (for instance):

if 'postgres' in connection['ENGINE']:
return 'uuid'
if 'mysql' in connection['ENGINE']:
return 'char(36)'
[...]

However, having done some work myself on trying to create non-trivial field 
subclasses, it's everything after that which gets difficult. Django provides an 
overridable method to cast the value, but nothing for anything else in the 
expression (the field or the operator), which are hard-coded into the backend. 
(This is a source of frustration for me personally, because it makes it very 
difficult to write classes for, say, PostgreSQL arrays, without either 
resorting to massively ugly syntax or subclassing nearly every single class 
involved in the process of creating a query (Manager, QuerySet, Query, 
WhereNode...)

I ultimately went through the subclass-half-the-world technique quite recently 
(a couple of weeks ago), as I want some non-trivial custom fields for a new 
project I am about to start for my company (sadly, the project is private, 
although I'd be happy to share field code if it would help in any way). What I 
ended up doing is checking the Field subclass for a custom 
`get_db_lookup_expression` method (that's not a Django field method -- I made 
it up), and then my Field subclasses could use that to return a full expression 
in the form "{field} = {value}". If the method is present and I get something 
(other than None) back from that method, then use it, otherwise I pass it on to 
the DatabaseOperators class for its usual processing. Using that method 
prevents me from having to modify a monolithic DatabaseOperators subclass for 
each new field I add (it seems to me that fields should know how to perform 
their lookups).

The other challenge was defining the QuerySet lookup expressions. Django 
essentially hard-codes the things it understands for lookups (e.g. 
Foo.objects.filter(bar__gt=5) being transformed into "select ... from app_foo 
where bar > 5"). The set of lookup suffices (exact, gt, gte, lt, lte, etc.) is, 
sadly, also essentially hard-coded. I wrote an ArrayField to use PostgreSQL 
arrays, and really wanted a way to be able to lookup based on the length of the 
array (so, something like `Foo.objects.exclude(photos__len=0)`, for instance, 
to give me all Foos with no photos). I did manage to make that work, but it was 
a struggle. Also, the set of lookup suffices is universal, even though some of 
them don't make sense on some fields ("year" on IntegerField, for instance).

So, my ideas on getting non-trivial field subclasses to work is basically:

1. Make fields be the arbiter of what lookup types they understand. 
(IntegerFields shouldn't understand "year"; maybe someone's ArrayField subclass 
does understand "len".) This probably needs to be something that can be 
determined on the fly, as composite fields will probably need lookup types 
corresponding to their individual sub-fields.

2. Make lookup types chainable. In the Array "len" example, `photos__len__gt=5` 
makes sense.

3. Make it so fields define how they are looked up based on database engine and 
lookup type.

Moving these things into the Field implementation (rather than in the backend) 
should mean that non-trivial field subclasses become much easier. It'll also 
eliminate the need for, say, django.contrib.gis to have an entirely different 
set of backends -- a large reason gis jumps through those hoops (as best as I 
can tell from reading it; disclaimer: I am not a contributor) is to work around 
the restrictions being described.

I hope that helps your thinking. I have this stuff fresh in my head because 
I've just worked on an implementation for PostgreSQL arrays and composite 
fields that I need for my work. While I've thought a decent bit about 
extensibility (for my own purposes), I haven't open-sourced it largely because 
I know I haven't solved all the problems yet. Having read your e-mail, I now 
hope that I don't have to, as I expect your work to outshine mine. I look 
forward to replacing what I've done with what you do. :-)

One more absolutely massive disclaimer: I am not a core developer or e

Not calling things twice in templates

[Daniele Procida]

The for ... empty pattern in templates is common and useful: 


But this is another common pattern:

{% if my_bonnet.bees %}

{% for bee in my_bonnet.bees %}
{{ bee }}
...

In other words, we have to check first for items in the iterable before 
creating a  or whatever other furniture to put them in.

The problem here is that my_bonnet.bees gets asked for twice, and it could be 
my.bonnet(), some very expensive function.

One solution is to put everything inside:

{% with my_bonnet.bees as bees %}

but now:

* we've used up a precious level of indentation
* we've introduced a new variable in the templates to worry about
* it just feels a bit fussy

Is this enough of an issue to make it worthwhile implementing some other 
approach in the template system?

Daniele

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Not calling things twice in templates

[Jeremy Dunck]

I've had this issue and have used {% with %} or moved the my.bonnet() call
into the view/context.I agree, not ideal, but I was never moved to make
it better in general.

If you're suggesting a general caching layer in the template such that a
given expression is only called once in the course of a template render
(including inheritance, includes, etc.) then I think you would need a way
of whitelisting which expressions were without side effects (similar to
is_safe w/ autoescaping).  It seems like a fair bit of book-keeping, but it
could clearly be added in a backwards-compatible way.




On Sun, Jun 2, 2013 at 2:36 PM, Daniele Procida  wrote:

> The for ... empty pattern in templates is common and useful: <
> https://docs.djangoproject.com/en/dev/ref/templates/builtins/#for-empty>
>
> But this is another common pattern:
>
> {% if my_bonnet.bees %}
> 
> {% for bee in my_bonnet.bees %}
> {{ bee }}
> ...
>
> In other words, we have to check first for items in the iterable before
> creating a  or whatever other furniture to put them in.
>
> The problem here is that my_bonnet.bees gets asked for twice, and it could
> be my.bonnet(), some very expensive function.
>
> One solution is to put everything inside:
>
> {% with my_bonnet.bees as bees %}
>
> but now:
>
> * we've used up a precious level of indentation
> * we've introduced a new variable in the templates to worry about
> * it just feels a bit fussy
>
> Is this enough of an issue to make it worthwhile implementing some other
> approach in the template system?
>
> Daniele
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers?hl=en
> .
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Not calling things twice in templates

[Russell Keith-Magee]

On Mon, Jun 3, 2013 at 5:36 AM, Daniele Procida  wrote:

> The for ... empty pattern in templates is common and useful: <
> https://docs.djangoproject.com/en/dev/ref/templates/builtins/#for-empty>
>
> But this is another common pattern:
>
> {% if my_bonnet.bees %}
> 
> {% for bee in my_bonnet.bees %}
> {{ bee }}
> ...
>
> In other words, we have to check first for items in the iterable before
> creating a  or whatever other furniture to put them in.
>
> The problem here is that my_bonnet.bees gets asked for twice, and it could
> be my.bonnet(), some very expensive function.
>
> One solution is to put everything inside:
>
> {% with my_bonnet.bees as bees %}
>
> but now:
>
> * we've used up a precious level of indentation
> * we've introduced a new variable in the templates to worry about
> * it just feels a bit fussy
>
> Is this enough of an issue to make it worthwhile implementing some other
> approach in the template system?
>
> This specific use case (i.e., the empty UL/OL) has bothered me in the past
as well. However, I'm not sure I see an elegant way to.

I'm open to suggestions, but the only way I can see around the problem is
to put an extra argument on the {% for %} that allows for extra 'wrapper'
content "if not empty" - something like:

{% for bee in my_bonnet.bees pre="" post=""%}
{{ bee }}
{% empty %}
No bees in your bonnet.
{% endfor %}

But I'm really not sure I like the syntax combining markup into tags
arguments. Alternatively, add new sub tags to {% for %}:

{% for bee in my_bonnet.bees %}
{% pre %}

{% body %}
{{ bee }}
{% post %}

{% empty %}
No bees in your bonnet.
{% endfor %}

but that's starting to get verbose, and leaves ambiguous what to do with
content appearing between the opening {% for %} and the {% pre %}

Any other syntax I can think of requires introducing another indentation
level. Have you got a specific suggestion for how to address this?

Russ %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Not calling things twice in templates

[Andre Terra]

Well, Russ, you asked for suggestions, so here's a couple half-hearted
attempts.

Perhaps we could allow for if clauses in the with block or vice-versa? It
could be argued that it would reduce readability and/or induce confusion
with conditional expressions[0], and I would have to agree.

{% with my_bonnet.bees as bees if my_bonnet.bees %}
{# could be confused with conditional expressions #}

{% for bee in bees %}
{{ bee }}
{% endfor %}

{% else %}
No bees!
{% endwith %}

--- or ---

{% if my_bonnet.bees with my_bonnet.bees as bees %}
{# seems a bit repetitive, even though explicit is better than implicit #}

{% for bee in bees %}
{{ bee }}
{% endfor %}

{% else %}
No bees!
{% endif %} {# seems counter-intuitive that this isn't the end of a with
block #}


I'm not too keen on either suggestion, but they seem easier to implement
than the book-keeping alternative. Additional ideas would include having a
magic name for if operands (effectively making if's behave like with's by
default) or creating a different tag altogether.


Cheers,
Andre Terra

[0]:
http://docs.python.org/3/reference/expressions.html#conditional-expressions



On Sun, Jun 2, 2013 at 11:36 PM, Russell Keith-Magee <
russ...@keith-magee.com> wrote:

>
> On Mon, Jun 3, 2013 at 5:36 AM, Daniele Procida  wrote:
>
>> The for ... empty pattern in templates is common and useful: <
>> https://docs.djangoproject.com/en/dev/ref/templates/builtins/#for-empty>
>>
>> But this is another common pattern:
>>
>> {% if my_bonnet.bees %}
>> 
>> {% for bee in my_bonnet.bees %}
>> {{ bee }}
>> ...
>>
>> In other words, we have to check first for items in the iterable before
>> creating a  or whatever other furniture to put them in.
>>
>> The problem here is that my_bonnet.bees gets asked for twice, and it
>> could be my.bonnet(), some very expensive function.
>>
>> One solution is to put everything inside:
>>
>> {% with my_bonnet.bees as bees %}
>>
>> but now:
>>
>> * we've used up a precious level of indentation
>> * we've introduced a new variable in the templates to worry about
>> * it just feels a bit fussy
>>
>> Is this enough of an issue to make it worthwhile implementing some other
>> approach in the template system?
>>
>> This specific use case (i.e., the empty UL/OL) has bothered me in the
> past as well. However, I'm not sure I see an elegant way to.
>
> I'm open to suggestions, but the only way I can see around the problem is
> to put an extra argument on the {% for %} that allows for extra 'wrapper'
> content "if not empty" - something like:
>
> {% for bee in my_bonnet.bees pre="" post=""%}
> {{ bee }}
> {% empty %}
> No bees in your bonnet.
> {% endfor %}
>
> But I'm really not sure I like the syntax combining markup into tags
> arguments. Alternatively, add new sub tags to {% for %}:
>
> {% for bee in my_bonnet.bees %}
> {% pre %}
> 
> {% body %}
> {{ bee }}
> {% post %}
> 
> {% empty %}
> No bees in your bonnet.
> {% endfor %}
>
> but that's starting to get verbose, and leaves ambiguous what to do with
> content appearing between the opening {% for %} and the {% pre %}
>
> Any other syntax I can think of requires introducing another indentation
> level. Have you got a specific suggestion for how to address this?
>
> Russ %-)
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers?hl=en
> .
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Not calling things twice in templates

[Shai Berger]

On Monday 03 June 2013, Russell Keith-Magee wrote:
> Alternatively, add new sub tags to {% for %}:
> 
> {% for bee in my_bonnet.bees %}
> {% pre %}
> 
> {% body %}
> {{ bee }}
> {% post %}
> 
> {% empty %}
> No bees in your bonnet.
> {% endfor %}
> 
> but that's starting to get verbose, and leaves ambiguous what to do with
> content appearing between the opening {% for %} and the {% pre %}
> 
It's also sort of available already:

{% for bee in my_bonnet.bees %}

 {% if forloop.first %}

 {% endif %}

 {{ bee }}

 {% if forloop.last %}
   
 {% endif %}

{% empty %}
 No bees in your bonnet.
{% endfor %}

Shai.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.