Re: pre_save signal on Inherited Model?

[Florian Apolloner]

Hi Leo,

On Wednesday, August 20, 2014 2:48:07 AM UTC+2, Leo Hillman wrote:
>
> — pre_delete registered on parent model: fired
> — pre_save registered on parent model: not fired
>

That sounds inconsistent at least. Can you open a ticket for that?

Cheers,
Florian   

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5d9b393d-db59-4b8e-9a67-f5beda109e7e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: GSoC Meta refactor: Bikeshedding time!!

[Russell Keith-Magee]

On Mon, Aug 18, 2014 at 6:03 PM, Anssi Kääriäinen 
wrote:

> On Monday, August 18, 2014 7:45:17 AM UTC+3, Russell Keith-Magee wrote:
>>
>> I understand what you're driving at here, and I've had similar thoughts
>> over the course of the SoC. The catch is that this makes the API for
>> get_fields() fairly complicated.
>>
>> If every field fits into one specific type, then get_fields() just
>> requires a single boolean flag (do I include fields of type X) for each
>> field type. We can also easily add new field types by adding new booleans
>> to the API.
>>
>> However, if a field fits into multiple categories, then it's impossible
>> (or, at least, exceedingly complicated) to make a single call to
>> get_fields() that will specify all your field requirements. "Get me all
>> non-virtual data fields" requires "virtual=False, data=True, m2m=False",
>> but "Get all virtual data fields that represent m2ms" requires
>> "virtual=True, data=False, m2m=True". You can't pass in both sets of
>> arguments at the same time, so you either have to make multiple calls to
>> get_fields(), or you have to invent some sort of query syntax for
>> get_fields() that allows union queries.
>>
>> Plus, at the end of the day, get_fields() is abstracted behind highly
>> cached and optimised properties for key lookups. These properties are
>> effectively a cached call to get_fields() with a specific set of arguments
>> - so even if get_fields() doesn't expose a "one category per field"
>> requirement, the API will require, at some level, names that have clear
>> (and preferably non-overlapping) membership.
>>
>
> If fields are in multiple categories then users will want to do the full
> range of set operation on the categories. Encoding that in to the API
> doesn't sound promising.
>
>
> I don't think users actually want to get fields based on the suggested
>>> categorization. I feel we get an easier to use and more flexible API if we
>>> have higher level categories and allow fields to match multiple categories.
>>> As a practical example if I want all relation fields, that is going to be
>>> hard using the suggested API. Getting all relation fields is a more
>>> realistic use case than getting related virtual objects.
>>>
>>
>> Quite probably true. As a point of interest, the current (as in, 1.6) API
>> actually doesn't differentiate between category (a) "pure data" and
>> category (b) "relating data (i.e., FK)" fields - if you ask for "data
>> fields" you get pure data *and* foreign keys. So, at least as far as
>> Django's own usage is concerned, you're correct in saying that taxonomy
>> I've described isn't fully required.
>>
>> Daniel's survey of internal usage reveals that there are three use cases
>> for getting a list of fields in Django's internal API:
>>
>>  * Get all data and m2m fields (i.e., categories  a, b, and d). This is
>> effectively "all fields on *this* model"
>>
>>  * Get all data, m2m, related objects, related m2m, and virtual fields
>> (i.e., categories a, b, d, f, g, h, i - excluding c and e because Django
>> doesn't currently have any fields of this type). This is "all fields on
>> this model, or related to this model"
>>
>>  * Get all m2m fields (i.e., category d)
>>
>> So - at the very least, we need names to describe those three groups. My
>> intention with describing a richer taxonomy is to try and give names to
>> other groupings of interest.
>>
>> If we want to have all fields to match single and only single category,
>>> then we need to redefine the categories to make sure ForeignKeys as virtual
>>> fields are possible, and that more esoteric custom join based fields fit in
>>> to the categorization.
>>>
>>
>> Agreed - that's why I threw this out there for discussion :-)
>>
>> Properties like "data", "virtual", "external", "related", "relating" -
>> these are high level concepts describing the way a field manifests.
>> However, that doesn't mean we need to expose these properties as part of
>> the formal API.
>>
>> Part of the underlying problem here -- lets say we roll out Django 1.7
>> with some version of this API, and in 1.8, foreign key fields change to
>> become virtual. That effectively becomes backwards incompatible for queries
>> that are sensitive to a "virtual" flag; but it doesn't change the
>> underlying need to identify that a field is a foreign key. We need to
>> capture the latter use case, but not necessarily the former.
>>
>
> Could we go with a minimal API for get_fields()? Instead of having
> categorization on the get_fields() API, we could provide field flags for
> the categories. With field flags it is straightforward to filter the return
> list of get_fields(). As an example, fetching those fields which are
> relations but which aren't virtual: [f for f in get_fields() if
> f.relational and not f.virtual]. If this path is taken, then I am not sure
> how minimal the get_fields() API should be. We likely need flags for at
> least if the field is defined on local, parent or some remot

Re: GSoC Meta refactor: Bikeshedding time!!

[Marc Tamlyn]

I'd say ArrayField is a straight up data field at the moment. It stores 0-1
lists of data. It's no different to CommaSeparatedIntegerField (seriously,
why does that exists...)

*If* PG gets the relevant update that will allow `integer[] references`
(i.e. ArrayField(ForeignKey)) then this would be different, and would be
more like a m2m field.

There is an argument that it's 0-N anyway, but in the implementation both
within Django and in the database I don't think the distinction is useful
at the point, from an ORM point of view in any case. For a forms point of
view it's quite different.


On 20 August 2014 09:19, Russell Keith-Magee 
wrote:

>
> On Mon, Aug 18, 2014 at 6:03 PM, Anssi Kääriäinen  > wrote:
>
>> On Monday, August 18, 2014 7:45:17 AM UTC+3, Russell Keith-Magee wrote:
>>>
>>> I understand what you're driving at here, and I've had similar thoughts
>>> over the course of the SoC. The catch is that this makes the API for
>>> get_fields() fairly complicated.
>>>
>>> If every field fits into one specific type, then get_fields() just
>>> requires a single boolean flag (do I include fields of type X) for each
>>> field type. We can also easily add new field types by adding new booleans
>>> to the API.
>>>
>>> However, if a field fits into multiple categories, then it's impossible
>>> (or, at least, exceedingly complicated) to make a single call to
>>> get_fields() that will specify all your field requirements. "Get me all
>>> non-virtual data fields" requires "virtual=False, data=True, m2m=False",
>>> but "Get all virtual data fields that represent m2ms" requires
>>> "virtual=True, data=False, m2m=True". You can't pass in both sets of
>>> arguments at the same time, so you either have to make multiple calls to
>>> get_fields(), or you have to invent some sort of query syntax for
>>> get_fields() that allows union queries.
>>>
>>> Plus, at the end of the day, get_fields() is abstracted behind highly
>>> cached and optimised properties for key lookups. These properties are
>>> effectively a cached call to get_fields() with a specific set of arguments
>>> - so even if get_fields() doesn't expose a "one category per field"
>>> requirement, the API will require, at some level, names that have clear
>>> (and preferably non-overlapping) membership.
>>>
>>
>> If fields are in multiple categories then users will want to do the full
>> range of set operation on the categories. Encoding that in to the API
>> doesn't sound promising.
>>
>>
>> I don't think users actually want to get fields based on the suggested
 categorization. I feel we get an easier to use and more flexible API if we
 have higher level categories and allow fields to match multiple categories.
 As a practical example if I want all relation fields, that is going to be
 hard using the suggested API. Getting all relation fields is a more
 realistic use case than getting related virtual objects.

>>>
>>> Quite probably true. As a point of interest, the current (as in, 1.6)
>>> API actually doesn't differentiate between category (a) "pure data" and
>>> category (b) "relating data (i.e., FK)" fields - if you ask for "data
>>> fields" you get pure data *and* foreign keys. So, at least as far as
>>> Django's own usage is concerned, you're correct in saying that taxonomy
>>> I've described isn't fully required.
>>>
>>> Daniel's survey of internal usage reveals that there are three use cases
>>> for getting a list of fields in Django's internal API:
>>>
>>>  * Get all data and m2m fields (i.e., categories  a, b, and d). This is
>>> effectively "all fields on *this* model"
>>>
>>>  * Get all data, m2m, related objects, related m2m, and virtual fields
>>> (i.e., categories a, b, d, f, g, h, i - excluding c and e because Django
>>> doesn't currently have any fields of this type). This is "all fields on
>>> this model, or related to this model"
>>>
>>>  * Get all m2m fields (i.e., category d)
>>>
>>> So - at the very least, we need names to describe those three groups. My
>>> intention with describing a richer taxonomy is to try and give names to
>>> other groupings of interest.
>>>
>>> If we want to have all fields to match single and only single category,
 then we need to redefine the categories to make sure ForeignKeys as virtual
 fields are possible, and that more esoteric custom join based fields fit in
 to the categorization.

>>>
>>> Agreed - that's why I threw this out there for discussion :-)
>>>
>>> Properties like "data", "virtual", "external", "related", "relating" -
>>> these are high level concepts describing the way a field manifests.
>>> However, that doesn't mean we need to expose these properties as part of
>>> the formal API.
>>>
>>> Part of the underlying problem here -- lets say we roll out Django 1.7
>>> with some version of this API, and in 1.8, foreign key fields change to
>>> become virtual. That effectively becomes backwards incompatible for queries
>>> that are sensitive to a "virtual" flag;

Re: GSoC Meta refactor: Bikeshedding time!!

[Ivan Kharlamov]

On 08/20/2014 12:46 PM, Marc Tamlyn wrote:
> I'd say ArrayField is a straight up data field at the moment. It stores
> 0-1 lists of data. It's no different to CommaSeparatedIntegerField
> (seriously, why does that exists...)
> 
> *If* PG gets the relevant update that will allow `integer[] references`
> (i.e. ArrayField(ForeignKey)) then this would be different, and would be
> more like a m2m field.
> 
> There is an argument that it's 0-N anyway, but in the implementation
> both within Django and in the database I don't think the distinction is
> useful at the point, from an ORM point of view in any case. For a forms
> point of view it's quite different.
> 
> 
> On 20 August 2014 09:19, Russell Keith-Magee  > wrote:
> 
> 
> On Mon, Aug 18, 2014 at 6:03 PM, Anssi Kääriäinen
> mailto:anssi.kaariai...@thl.fi>> wrote:
> 
> On Monday, August 18, 2014 7:45:17 AM UTC+3, Russell Keith-Magee
> wrote:
> 
> I understand what you're driving at here, and I've had
> similar thoughts over the course of the SoC. The catch is
> that this makes the API for get_fields() fairly complicated.
> 
> If every field fits into one specific type, then
> get_fields() just requires a single boolean flag (do I
> include fields of type X) for each field type. We can also
> easily add new field types by adding new booleans to the API.
> 
> However, if a field fits into multiple categories, then it's
> impossible (or, at least, exceedingly complicated) to make a
> single call to get_fields() that will specify all your field
> requirements. "Get me all non-virtual data fields" requires
> "virtual=False, data=True, m2m=False", but "Get all virtual
> data fields that represent m2ms" requires "virtual=True,
> data=False, m2m=True". You can't pass in both sets of
> arguments at the same time, so you either have to make
> multiple calls to get_fields(), or you have to invent some
> sort of query syntax for get_fields() that allows union
> queries. 
> 
> Plus, at the end of the day, get_fields() is abstracted
> behind highly cached and optimised properties for key
> lookups. These properties are effectively a cached call to
> get_fields() with a specific set of arguments - so even if
> get_fields() doesn't expose a "one category per field"
> requirement, the API will require, at some level, names that
> have clear (and preferably non-overlapping) membership.
> 
> 
> If fields are in multiple categories then users will want to do
> the full range of set operation on the categories. Encoding that
> in to the API doesn't sound promising.
> 
> 
> I don't think users actually want to get fields based on
> the suggested categorization. I feel we get an easier to
> use and more flexible API if we have higher level
> categories and allow fields to match multiple
> categories. As a practical example if I want all
> relation fields, that is going to be hard using the
> suggested API. Getting all relation fields is a more
> realistic use case than getting related virtual objects.
> 
> 
> Quite probably true. As a point of interest, the current (as
> in, 1.6) API actually doesn't differentiate between category
> (a) "pure data" and category (b) "relating data (i.e., FK)"
> fields - if you ask for "data fields" you get pure data
> *and* foreign keys. So, at least as far as Django's own
> usage is concerned, you're correct in saying that taxonomy
> I've described isn't fully required. 
> 
> Daniel's survey of internal usage reveals that there are
> three use cases for getting a list of fields in Django's
> internal API:
> 
>  * Get all data and m2m fields (i.e., categories  a, b, and
> d). This is effectively "all fields on *this* model"
> 
>  * Get all data, m2m, related objects, related m2m, and
> virtual fields (i.e., categories a, b, d, f, g, h, i -
> excluding c and e because Django doesn't currently have any
> fields of this type). This is "all fields on this model, or
> related to this model"
> 
>  * Get all m2m fields (i.e., category d)
>  
> So - at the very least, we need names to describe those
> three groups. My intention with describing a richer taxonomy
> is to try and give names to other groupings of interest. 
> 
> If we want to have all fields to match single and only
>  

Re: GSoC Meta refactor: Bikeshedding time!!

[Ivan Kharlamov]

On 08/20/2014 03:52 PM, Ivan Kharlamov wrote:
> On 08/20/2014 12:46 PM, Marc Tamlyn wrote:
>> I'd say ArrayField is a straight up data field at the moment. It stores
>> 0-1 lists of data. It's no different to CommaSeparatedIntegerField
>> (seriously, why does that exists...)
>>
>> *If* PG gets the relevant update that will allow `integer[] references`
>> (i.e. ArrayField(ForeignKey)) then this would be different, and would be
>> more like a m2m field.
>>
>> There is an argument that it's 0-N anyway, but in the implementation
>> both within Django and in the database I don't think the distinction is
>> useful at the point, from an ORM point of view in any case. For a forms
>> point of view it's quite different.
>>
>>
>> On 20 August 2014 09:19, Russell Keith-Magee > > wrote:
>>
>>
>> On Mon, Aug 18, 2014 at 6:03 PM, Anssi Kääriäinen
>> mailto:anssi.kaariai...@thl.fi>> wrote:
>>
>> On Monday, August 18, 2014 7:45:17 AM UTC+3, Russell Keith-Magee
>> wrote:
>>
>> I understand what you're driving at here, and I've had
>> similar thoughts over the course of the SoC. The catch is
>> that this makes the API for get_fields() fairly complicated.
>>
>> If every field fits into one specific type, then
>> get_fields() just requires a single boolean flag (do I
>> include fields of type X) for each field type. We can also
>> easily add new field types by adding new booleans to the API.
>>
>> However, if a field fits into multiple categories, then it's
>> impossible (or, at least, exceedingly complicated) to make a
>> single call to get_fields() that will specify all your field
>> requirements. "Get me all non-virtual data fields" requires
>> "virtual=False, data=True, m2m=False", but "Get all virtual
>> data fields that represent m2ms" requires "virtual=True,
>> data=False, m2m=True". You can't pass in both sets of
>> arguments at the same time, so you either have to make
>> multiple calls to get_fields(), or you have to invent some
>> sort of query syntax for get_fields() that allows union
>> queries. 
>>
>> Plus, at the end of the day, get_fields() is abstracted
>> behind highly cached and optimised properties for key
>> lookups. These properties are effectively a cached call to
>> get_fields() with a specific set of arguments - so even if
>> get_fields() doesn't expose a "one category per field"
>> requirement, the API will require, at some level, names that
>> have clear (and preferably non-overlapping) membership.
>>
>>
>> If fields are in multiple categories then users will want to do
>> the full range of set operation on the categories. Encoding that
>> in to the API doesn't sound promising.
>>
>>
>> I don't think users actually want to get fields based on
>> the suggested categorization. I feel we get an easier to
>> use and more flexible API if we have higher level
>> categories and allow fields to match multiple
>> categories. As a practical example if I want all
>> relation fields, that is going to be hard using the
>> suggested API. Getting all relation fields is a more
>> realistic use case than getting related virtual objects.
>>
>>
>> Quite probably true. As a point of interest, the current (as
>> in, 1.6) API actually doesn't differentiate between category
>> (a) "pure data" and category (b) "relating data (i.e., FK)"
>> fields - if you ask for "data fields" you get pure data
>> *and* foreign keys. So, at least as far as Django's own
>> usage is concerned, you're correct in saying that taxonomy
>> I've described isn't fully required. 
>>
>> Daniel's survey of internal usage reveals that there are
>> three use cases for getting a list of fields in Django's
>> internal API:
>>
>>  * Get all data and m2m fields (i.e., categories  a, b, and
>> d). This is effectively "all fields on *this* model"
>>
>>  * Get all data, m2m, related objects, related m2m, and
>> virtual fields (i.e., categories a, b, d, f, g, h, i -
>> excluding c and e because Django doesn't currently have any
>> fields of this type). This is "all fields on this model, or
>> related to this model"
>>
>>  * Get all m2m fields (i.e., category d)
>>  
>> So - at the very least, we need names to describe those
>> three groups. My intention with describing a richer taxonomy
>> is to try and g

Re: GSoC Meta refactor: Bikeshedding time!!

[Ivan Kharlamov]

On 08/20/2014 04:28 PM, Ivan Kharlamov wrote:
> On 08/20/2014 03:52 PM, Ivan Kharlamov wrote:
>> On 08/20/2014 12:46 PM, Marc Tamlyn wrote:
>>> I'd say ArrayField is a straight up data field at the moment. It stores
>>> 0-1 lists of data. It's no different to CommaSeparatedIntegerField
>>> (seriously, why does that exists...)
>>>
>>> *If* PG gets the relevant update that will allow `integer[] references`
>>> (i.e. ArrayField(ForeignKey)) then this would be different, and would be
>>> more like a m2m field.
>>>
>>> There is an argument that it's 0-N anyway, but in the implementation
>>> both within Django and in the database I don't think the distinction is
>>> useful at the point, from an ORM point of view in any case. For a forms
>>> point of view it's quite different.
>>>
>>>
>>> On 20 August 2014 09:19, Russell Keith-Magee >> > wrote:
>>>
>>>
>>> On Mon, Aug 18, 2014 at 6:03 PM, Anssi Kääriäinen
>>> mailto:anssi.kaariai...@thl.fi>> wrote:
>>>
>>> On Monday, August 18, 2014 7:45:17 AM UTC+3, Russell Keith-Magee
>>> wrote:
>>>
>>> I understand what you're driving at here, and I've had
>>> similar thoughts over the course of the SoC. The catch is
>>> that this makes the API for get_fields() fairly complicated.
>>>
>>> If every field fits into one specific type, then
>>> get_fields() just requires a single boolean flag (do I
>>> include fields of type X) for each field type. We can also
>>> easily add new field types by adding new booleans to the API.
>>>
>>> However, if a field fits into multiple categories, then it's
>>> impossible (or, at least, exceedingly complicated) to make a
>>> single call to get_fields() that will specify all your field
>>> requirements. "Get me all non-virtual data fields" requires
>>> "virtual=False, data=True, m2m=False", but "Get all virtual
>>> data fields that represent m2ms" requires "virtual=True,
>>> data=False, m2m=True". You can't pass in both sets of
>>> arguments at the same time, so you either have to make
>>> multiple calls to get_fields(), or you have to invent some
>>> sort of query syntax for get_fields() that allows union
>>> queries. 
>>>
>>> Plus, at the end of the day, get_fields() is abstracted
>>> behind highly cached and optimised properties for key
>>> lookups. These properties are effectively a cached call to
>>> get_fields() with a specific set of arguments - so even if
>>> get_fields() doesn't expose a "one category per field"
>>> requirement, the API will require, at some level, names that
>>> have clear (and preferably non-overlapping) membership.
>>>
>>>
>>> If fields are in multiple categories then users will want to do
>>> the full range of set operation on the categories. Encoding that
>>> in to the API doesn't sound promising.
>>>
>>>
>>> I don't think users actually want to get fields based on
>>> the suggested categorization. I feel we get an easier to
>>> use and more flexible API if we have higher level
>>> categories and allow fields to match multiple
>>> categories. As a practical example if I want all
>>> relation fields, that is going to be hard using the
>>> suggested API. Getting all relation fields is a more
>>> realistic use case than getting related virtual objects.
>>>
>>>
>>> Quite probably true. As a point of interest, the current (as
>>> in, 1.6) API actually doesn't differentiate between category
>>> (a) "pure data" and category (b) "relating data (i.e., FK)"
>>> fields - if you ask for "data fields" you get pure data
>>> *and* foreign keys. So, at least as far as Django's own
>>> usage is concerned, you're correct in saying that taxonomy
>>> I've described isn't fully required. 
>>>
>>> Daniel's survey of internal usage reveals that there are
>>> three use cases for getting a list of fields in Django's
>>> internal API:
>>>
>>>  * Get all data and m2m fields (i.e., categories  a, b, and
>>> d). This is effectively "all fields on *this* model"
>>>
>>>  * Get all data, m2m, related objects, related m2m, and
>>> virtual fields (i.e., categories a, b, d, f, g, h, i -
>>> excluding c and e because Django doesn't currently have any
>>> fields of this type). This is "all fields on this model, or
>>> related to this model"
>>>
>>>  * Get all m2m fields (i.e., category d)
>>>  
>>> So - at the very lea

Re: pre_save signal on Inherited Model?

[charettes]

This might be related to #9318 .

Le mardi 19 août 2014 20:48:07 UTC-4, Leo Hillman a écrit :
>
> I just wanted to write a preliminary post about this before gathering a 
> little more data. When extending a non-abstract model (Python inheritance) 
> it seems to me that a pre_delete signal registered to the parent model is 
> fired when an inherited model is deleted—which would be expected. A 
> pre_save signal registered to the parent model, however, does not seem to 
> be firing when saving the inherited model. I am not sure if this is desired 
> behavior, but it is slightly confusing.
>
> — pre_delete registered on parent model: fired
> — pre_save registered on parent model: not fired
>
> I came upon this case when extending the djcelery PeriodicTask object. 
> When creating an instance of an inheriting model, the pre_save signal was 
> not sent, but the pre_delete signal did seem to be sent.
>
> Any thoughts? I can get around this by manually registering the inheriting 
> model.
>
> aryeh
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/685c7e1d-4d29-41ca-b260-8db91050c0ec%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: GSoC Meta refactor: Bikeshedding time!!

[Anssi Kääriäinen]

On Wednesday, August 20, 2014 11:19:33 AM UTC+3, Russell Keith-Magee wrote:
>
> I think Daniel and I might have come up with a way to meet both these 
> requirements - a minimalist API for get_fields, with at least some 
> protection against the known incoming backwards compatibility issue.
>
> The summary so far: it appears that a complex taxonomy isn't especially 
> helpful - firstly, because any complex taxonomy is going to have edge cases 
> that are hard to categorize, but also because a complex taxonomy leads to a 
> much more complex internal API that is going to be prone to backwards 
> compatibility problems.
>
> So - instead of worrying about 'virtual' and other properties like that, 
> lets look at why the _meta API is fundamentally used - to get a list of 
> fields that need to be handled in data processing. This primarily means 
> forms, but other forms of serialisation are also included. In these use 
> cases, there are always going to be per-field differences (even a CharField 
> and an IntegerField require *slightly* different handling), so we won't 
> focus on internal representations, storage mechanisms, or anything like 
> that. Instead, lets focus on cardinality - a field represents some sort of 
> data that has a cardinality with the object on which it is stored. If 
> something has cardinality 1, you can display a single field. If it's 
> cardinality N, you need to display a list, or some sort of inline.
>
> This results in 3 categories that are mutually exclusive:
>
> a) "Data fields": Fields of cardinality 0-1:
> 
> b) "ManyToMany Fields": Fields that are locally defined that represent a 
> cardinality 0-N relationship with another object:
> 
> c) "Related Objects": Fields that represent a cardinality 0-N relationship 
> with this object, but aren't locally defined:
> 
>
 

> These three types are mutually exclusive - you either have cardinality 1 
> *or* cardinality N, not both; and you're either locally defined on this 
> object or you're not. I can't think of an example of "cardinality 1 data 
> that isn't defined on this object", but it would fit into this taxonomy if 
> it were needed; I also can't think of a field definition that would span 
> models.
>

The reverse of OneToOneField is a cardinality 1 data that isn't defined on 
this object.

>
> In addition to this basic classification, a field can be marked as 
> "hidden". The immediate use for this is to hide the related_name='+' case 
> of a FK or M2M. Looking forward, it would be used to mask fields that 
> exist, but aren't intended to be user visible - for example, in the 
> potential future case where a ForeignKey is split in two, or a Composite 
> Key, there would be a "hidden" integer field (or fields) storing the actual 
> data, and a virtual (but non-hidden) field that is the public API for 
> manipulating the relationship. This would also be backwards compatible, 
> because the "visible" field list hasn't changed.
>

There are use cases that do not fit this categorization. For example when 
instantiating a model from database you will need to supply the hidden 
integer field data for a foreign key, but you must skip the foreign key 
field itself. That is, a model with relation to author is initialized as 
MyModel(pk=1, author_first_name='foo', author_last_name='bar') (technically 
this is done through *args for performance reasons), not with MyModel(pk=1, 
author=author_instance). Similar considerations likely apply to 
serialization of models.

Form fields for a model is another consideration. If one wants those fields 
that should have a field in a form, that is currently defined as [f for f 
in model._meta.fields if f.editable]. The editable fields set doesn't 
necessarily match the above categorization. In fact, I believe if we 
inspect Django's code base it will be clear there can't be any 
categorization where fields belong to only one category, but which fulfills 
all use cases in Django. It is like trying to categorize animals for every 
use case. If you want mammals, then categorization to sea and land 
creatures will not work. If you want sea creatures, then categorization to 
mammals and fish is useless.

The point is that I am convinced we will need to provide field flags to 
complement the get_fields() API no matter what API we choose for 
get_fields(). In fact, if we define and document a sane set of field flags, 
then the get_fields() API isn't that important, it just needs to be useful 
for the most common use cases.
 

> Fields are also tracked according to their parentage; this is used by 
> tools interacting with inheritance relationships to know which fields are 
> actually on this model, and which are inherited from a base class.
>
> This yields the following formal API for _meta:
>
>  * get_fields(data, many_to_many, related, include_hidden, include_parents)
>
>  * @property data_fields (=> get_fields(data=True, many_to_many=False, 
> related=False, include_hidden=False, include_parents=True)

Re: GSoC Meta refactor: Bikeshedding time!!

[Shai Berger]

On Wednesday 20 August 2014 10:29:49 Anssi Kääriäinen wrote:
> On Wednesday, August 20, 2014 11:19:33 AM UTC+3, Russell Keith-Magee wrote:
> > 
> > This yields the following formal API for _meta:
> >  * get_fields(data, many_to_many, related, include_hidden,
> > include_parents)
> >  
> >  * @property data_fields 
> > 
> >  * @property many_to_many_fields 
> > 
> >  * @property related_objects 
> >  (=> get_fields(data=False, many_to_many=False, related=True,
> >  include_hidden=False, include_parents=True)
> > 
> +1 if we also consider defining and documenting an useful set of field flags.

+1 Anssi.

>
> I wonder if a better name for the related category exists. My first
> instinct is that foreign key fields should match the related flag. Could it
> be made cleaner that these are relations defined on remote model? Maybe
> just remote_relations could work?
> 

Since this is a bikeshedding thread, I'll say that the name "related_objects" 
makes me itch a little as well -- mostly, because each of the objects returned 
by the property is not a related-object, but rather a manager of related-
objects.

I was considering "related_managers", but that is sort-of mixing the "what" 
with the "how", and also thinking about the possible Array(FK) fields, I prefer 
"related_collections".

Shai.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/2540029.o0OKDSAJdQ%40deblack.
For more options, visit https://groups.google.com/d/optout.

[ANNOUNCE] Django security releases issued

[James Bennett]

Today we've issued releases to address four security issues reported to us.
Full disclosure is on the djangoproject.com weblog:

https://www.djangoproject.com/weblog/2014/aug/20/security/

All users are encouraged to upgrade.

Additionally, for anyone who missed it, last week we published an advisory
regarding the nature of the remove_tags template filter and function:

https://www.djangoproject.com/weblog/2014/aug/11/remove-tags-advisory/

If you are using remove_tags, it is recommended you read that advisory.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAL13Cg_rw8UKCNT8QV9aiMYXMHy47dPVHwu%2BBHNorq2fRjtybg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: GSoC Meta refactor: Bikeshedding time!!

[Russell Keith-Magee]

On Wed, Aug 20, 2014 at 8:28 PM, Ivan Kharlamov 
wrote:

> On 08/20/2014 03:52 PM, Ivan Kharlamov wrote:
> > On 08/20/2014 12:46 PM, Marc Tamlyn wrote:
> >> I'd say ArrayField is a straight up data field at the moment. It stores
> >> 0-1 lists of data. It's no different to CommaSeparatedIntegerField
> >> (seriously, why does that exists...)
> >>
> >> *If* PG gets the relevant update that will allow `integer[] references`
> >> (i.e. ArrayField(ForeignKey)) then this would be different, and would be
> >> more like a m2m field.
> >>
> >> There is an argument that it's 0-N anyway, but in the implementation
> >> both within Django and in the database I don't think the distinction is
> >> useful at the point, from an ORM point of view in any case. For a forms
> >> point of view it's quite different.
> >>
> >>
> >> On 20 August 2014 09:19, Russell Keith-Magee  >> > wrote:
> >>
> >>
> >> On Mon, Aug 18, 2014 at 6:03 PM, Anssi Kääriäinen
> >> mailto:anssi.kaariai...@thl.fi>> wrote:
> >>
> >> On Monday, August 18, 2014 7:45:17 AM UTC+3, Russell Keith-Magee
> >> wrote:
> >>
> >> I understand what you're driving at here, and I've had
> >> similar thoughts over the course of the SoC. The catch is
> >> that this makes the API for get_fields() fairly complicated.
> >>
> >> If every field fits into one specific type, then
> >> get_fields() just requires a single boolean flag (do I
> >> include fields of type X) for each field type. We can also
> >> easily add new field types by adding new booleans to the
> API.
> >>
> >> However, if a field fits into multiple categories, then it's
> >> impossible (or, at least, exceedingly complicated) to make a
> >> single call to get_fields() that will specify all your field
> >> requirements. "Get me all non-virtual data fields" requires
> >> "virtual=False, data=True, m2m=False", but "Get all virtual
> >> data fields that represent m2ms" requires "virtual=True,
> >> data=False, m2m=True". You can't pass in both sets of
> >> arguments at the same time, so you either have to make
> >> multiple calls to get_fields(), or you have to invent some
> >> sort of query syntax for get_fields() that allows union
> >> queries.
> >>
> >> Plus, at the end of the day, get_fields() is abstracted
> >> behind highly cached and optimised properties for key
> >> lookups. These properties are effectively a cached call to
> >> get_fields() with a specific set of arguments - so even if
> >> get_fields() doesn't expose a "one category per field"
> >> requirement, the API will require, at some level, names that
> >> have clear (and preferably non-overlapping) membership.
> >>
> >>
> >> If fields are in multiple categories then users will want to do
> >> the full range of set operation on the categories. Encoding that
> >> in to the API doesn't sound promising.
> >>
> >>
> >> I don't think users actually want to get fields based on
> >> the suggested categorization. I feel we get an easier to
> >> use and more flexible API if we have higher level
> >> categories and allow fields to match multiple
> >> categories. As a practical example if I want all
> >> relation fields, that is going to be hard using the
> >> suggested API. Getting all relation fields is a more
> >> realistic use case than getting related virtual objects.
> >>
> >>
> >> Quite probably true. As a point of interest, the current (as
> >> in, 1.6) API actually doesn't differentiate between category
> >> (a) "pure data" and category (b) "relating data (i.e., FK)"
> >> fields - if you ask for "data fields" you get pure data
> >> *and* foreign keys. So, at least as far as Django's own
> >> usage is concerned, you're correct in saying that taxonomy
> >> I've described isn't fully required.
> >>
> >> Daniel's survey of internal usage reveals that there are
> >> three use cases for getting a list of fields in Django's
> >> internal API:
> >>
> >>  * Get all data and m2m fields (i.e., categories  a, b, and
> >> d). This is effectively "all fields on *this* model"
> >>
> >>  * Get all data, m2m, related objects, related m2m, and
> >> virtual fields (i.e., categories a, b, d, f, g, h, i -
> >> excluding c and e because Django doesn't currently have any
> >> fields of this type). This is "all fields on this model, or
> >> related to this model"
> >>
> >> 

Re: GSoC Meta refactor: Bikeshedding time!!

[Russell Keith-Magee]

On Thu, Aug 21, 2014 at 1:29 AM, Anssi Kääriäinen 
wrote:

> On Wednesday, August 20, 2014 11:19:33 AM UTC+3, Russell Keith-Magee wrote:
>
>> I think Daniel and I might have come up with a way to meet both these
>> requirements - a minimalist API for get_fields, with at least some
>> protection against the known incoming backwards compatibility issue.
>>
>> The summary so far: it appears that a complex taxonomy isn't especially
>> helpful - firstly, because any complex taxonomy is going to have edge cases
>> that are hard to categorize, but also because a complex taxonomy leads to a
>> much more complex internal API that is going to be prone to backwards
>> compatibility problems.
>>
>> So - instead of worrying about 'virtual' and other properties like that,
>> lets look at why the _meta API is fundamentally used - to get a list of
>> fields that need to be handled in data processing. This primarily means
>> forms, but other forms of serialisation are also included. In these use
>> cases, there are always going to be per-field differences (even a CharField
>> and an IntegerField require *slightly* different handling), so we won't
>> focus on internal representations, storage mechanisms, or anything like
>> that. Instead, lets focus on cardinality - a field represents some sort of
>> data that has a cardinality with the object on which it is stored. If
>> something has cardinality 1, you can display a single field. If it's
>> cardinality N, you need to display a list, or some sort of inline.
>>
>> This results in 3 categories that are mutually exclusive:
>>
>> a) "Data fields": Fields of cardinality 0-1:
>> 
>>
>> b) "ManyToMany Fields": Fields that are locally defined that represent a
>> cardinality 0-N relationship with another object:
>> 
>>
>> c) "Related Objects": Fields that represent a cardinality 0-N
>> relationship with this object, but aren't locally defined:
>> 
>>
>
>
>> These three types are mutually exclusive - you either have cardinality 1
>> *or* cardinality N, not both; and you're either locally defined on this
>> object or you're not. I can't think of an example of "cardinality 1 data
>> that isn't defined on this object", but it would fit into this taxonomy if
>> it were needed; I also can't think of a field definition that would span
>> models.
>>
>
> The reverse of OneToOneField is a cardinality 1 data that isn't defined on
> this object.
>

And the obvious answer was looking right at me :-). I had that mentally
wrapped into (c) (because historically O2O is handled as a redundant case
of FK). This suggests to me that either (a) the "related" flag is more
about "objects that have a relationship with this one", rather than being
specifically about cardinality, or (b) there's another group for 0-1
cardinality reverse relationships.


> In addition to this basic classification, a field can be marked as
>> "hidden". The immediate use for this is to hide the related_name='+' case
>> of a FK or M2M. Looking forward, it would be used to mask fields that
>> exist, but aren't intended to be user visible - for example, in the
>> potential future case where a ForeignKey is split in two, or a Composite
>> Key, there would be a "hidden" integer field (or fields) storing the actual
>> data, and a virtual (but non-hidden) field that is the public API for
>> manipulating the relationship. This would also be backwards compatible,
>> because the "visible" field list hasn't changed.
>>
>
> There are use cases that do not fit this categorization. For example when
> instantiating a model from database you will need to supply the hidden
> integer field data for a foreign key, but you must skip the foreign key
> field itself. That is, a model with relation to author is initialized as
> MyModel(pk=1, author_first_name='foo', author_last_name='bar') (technically
> this is done through *args for performance reasons), not with MyModel(pk=1,
> author=author_instance). Similar considerations likely apply to
> serialization of models.
>


> Form fields for a model is another consideration. If one wants those
> fields that should have a field in a form, that is currently defined as [f
> for f in model._meta.fields if f.editable]. The editable fields set doesn't
> necessarily match the above categorization. In fact, I believe if we
> inspect Django's code base it will be clear there can't be any
> categorization where fields belong to only one category, but which fulfills
> all use cases in Django. It is like trying to categorize animals for every
> use case. If you want mammals, then categorization to sea and land
> creatures will not work. If you want sea creatures, then categorization to
> mammals and fish is useless.
>
> Sure - but let me say in advance that the API I've proposed here isn't a
purely theoretical exercise. Daniel has implemented this API to prove that
it's sufficient to meet all Django's existing use cases. The three
categories (plus the two include qualifiers) I've described meets that
criterion. Ho

Re: GSoC Meta refactor: Bikeshedding time!!

[Russell Keith-Magee]

On Thu, Aug 21, 2014 at 2:21 AM, Shai Berger  wrote:

> On Wednesday 20 August 2014 10:29:49 Anssi Kääriäinen wrote:
> > On Wednesday, August 20, 2014 11:19:33 AM UTC+3, Russell Keith-Magee
> wrote:
> > >
> > > This yields the following formal API for _meta:
> > >  * get_fields(data, many_to_many, related, include_hidden,
> > > include_parents)
> > >
> > >  * @property data_fields
> > >
> > >  * @property many_to_many_fields
> > >
> > >  * @property related_objects
> > >  (=> get_fields(data=False, many_to_many=False, related=True,
> > >  include_hidden=False, include_parents=True)
> > >
> > +1 if we also consider defining and documenting an useful set of field
> flags.
>
> +1 Anssi.
>
> >
> > I wonder if a better name for the related category exists. My first
> > instinct is that foreign key fields should match the related flag. Could
> it
> > be made cleaner that these are relations defined on remote model? Maybe
> > just remote_relations could work?
> >
>
> Since this is a bikeshedding thread, I'll say that the name
> "related_objects"
> makes me itch a little as well -- mostly, because each of the objects
> returned
> by the property is not a related-object, but rather a manager of related-
> objects.
>
> I was considering "related_managers", but that is sort-of mixing the "what"
> with the "how", and also thinking about the possible Array(FK) fields, I
> prefer
> "related_collections".
>

Hi Shai,

I'll accept that there are probably better names for this - but as I said
in my response to Anssi, until we actually know what "it" is, it will be
more productive to focus on the categories and flags that need to exist.
Once we've got those categories nailed down, the names will hopefully be
more obvious (or, at least, more obviously a bikeshed).

Russ %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAJxq84-LdY15%2Bfp4afOo-QT5aHODoWGsdN28v%2B0Qs2gPycqpUg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: GSoC Meta refactor: Bikeshedding time!!

[Anssi Kääriäinen]

On Thu, 2014-08-21 at 11:07 +0800, Russell Keith-Magee wrote:




> The point is that I am convinced we will need to provide field
> flags to complement the get_fields() API no matter what API we
> choose for get_fields(). In fact, if we define and document a
> sane set of field flags, then the get_fields() API isn't that
> important, it just needs to be useful for the most common use
> cases.
>   
> Well, no - it needs to be useful for *all* the use cases *in Django's
> codebase*. The end goal here is to provide a formal API definition so
> that someone else can take the specification, make a duck that quacks
> exactly like it, and use it because it is compatible with Django's
> internals.
> 
> 
> As an indicative goal - I'm thinking a good GSoC project for next year
> would be to implement a Django-compatible model layer for SQLAlchemy.
> That means the student will need to implement get_fields() (or
> whatever API we end up with) to sufficient depth that they can expose
> a SQLAlchemy model in Django's Admin, using Django's forms. Daniel's
> proof-of-concept project wrapping an email API demonstrates that this
> isn't a theoretical goal - it's has the potential of being real. 

Unfortunately I don't see SQLAlchemy models in Admin as realistic at all
by just providing correct _meta API. Of course, the fields returned by
get_fields() calls will also need to quack the right way. Case in point,
Daniel's Gmail PoC uses Django fields internally.

Maybe there is some miscommunication here - what I meant by saying that

In fact, if we define and document a sane set of field flags,
then the get_fields() API isn't that important, it just needs to
be useful for the most common use cases.

meant that if we have sane field flags, then one can always use
get_fields(**return_all_possible_fields), and then use flags on fields
to get a set of fields one is interested in.

It is trivially true that get_fields() isn't sufficient for all use
cases in Django without relying on field flags. One can't even get the
fields for a form without relying on field flags filtering.

So, get_fields() doesn't need to return correct list of fields for all
use cases in Django, it is sufficient that one can get the correct list
of fields with further filtering.

I am not sure if you were objecting to anything else than "it
[get_fields()] just needs to be useful for the most common use cases". I
hope I cleared up what I meant by that.

I remain convinced we should document field flags in addition to the
get_fields() API.

 - Anssi


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/1408602872.11410.300.camel%40TTY32.
For more options, visit https://groups.google.com/d/optout.

Re: GSoC Meta refactor: Bikeshedding time!!

[Anssi Kääriäinen]

On Thu, 2014-08-21 at 11:07 +0800, Russell Keith-Magee wrote:




> Sure - and the purpose of this thread is to tease out what those
> "useful" flags are. At the moment, it's not clear to me where exactly
> the conceptual holes lie from your perspective. As I said, the API
> I've proposed here is sufficient to meet all *current* use cases in
> the code base.
>  
> As best as I can make out, it appears you see a problem with the
> concept of "hidden" - because in various circumstances, different
> fields will be "hidden" in different ways (especially in the
> composite/virtual foreign key future). Taking that "future virtualised
> foreign key" case - if you're dealing with the database, it's the
> virtual field that needs to be hidden, because the database only cares
> about fields with an actual column/table underneath it; but if you're
> dealing with a form, you don't want the field for the underlying
> field, you want the virtual field. However, given a virtual field
> representation, I imagine it is possible to get back to the field (or
> fields) that hold the underlying representation; all that is important
> is that you can iterate over a list of "fields", and from there,
> determine a list of column names. The fact that the column name comes
> from a different underlying column isn't important; what's important
> is that the "foreign key" is only counted once in the introspection
> process.
> 
> 
> So - what I really need here is a counterproposal from someone
> familiar with the composite key work. I'm not bound to any of the
> details of the proposal I've given here - I'm just relating the end
> point of work from SoC. It works with the current use cases exposed by
> Django, but when this hits master, we're going to need to live with it
> long term, so I want to make sure we're not boxing ourselves into a
> corner, or introducing categorisations that aren't representative.

No counterproposal from me - as I said earlier I think the get_fields()
API as defined is sufficient.

We already have one field that doesn't play well with the categorization
when it comes to model serialization or initialization. That field is
GenericForeignKey. If it is returned by get_fields(data=True), then
get_fields(data=True) can't be used for model serialization or
initialization without filtering the return set further down using
flags.

In short, I see all concrete fields as an important category itself. We
can't provide that with the proposed categorization.

The easiest way to get all the concrete fields would be [f for f in
get_fields(**return_all_fields) if f.concrete]. It is also notable that
ordering matters, and for that reason inspecting the underlying fields
of a virtual field doesn't sound promising.

For the ForeignKey backwards compatibility problem - maybe we could get
the virtual fields patch committed for 1.8. This way we wouldn't have
any backwards compatibility problem to begin with. The composite fields
patch was otherwise close to committable, but the problem was that it
didn't play well with migrations. I have no idea what the actual
problems were. Anybody have more knowledge why composite fields do not
play well with migrations, and what is needed to solve the problems?

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/1408604445.11410.321.camel%40TTY32.
For more options, visit https://groups.google.com/d/optout.