Ticket #8161 - Please help me fix this.
Hi all,
I wanted to fix this bug. I just went through the code to see what was
happening. For the same sample code given in the ticket, these were my
findings:
The schema generated looks like this and there is no issue with it:
CREATE TABLE "testapp_childmodel" (
"parentmodel_ptr_id" integer NOT NULL PRIMARY KEY REFERENCES
"testapp_parentmodel" ("id"),
"text" text NOT NULL
);
CREATE TABLE "testapp_parentmodel" (
"id" integer NOT NULL PRIMARY KEY,
"slug" varchar(50) NOT NULL
);
CREATE TABLE "testapp_parentmodel_links" (
"id" integer NOT NULL PRIMARY KEY,
"from_parentmodel_id" integer NOT NULL REFERENCES "testapp_parentmodel"
("id"),
"to_parentmodel_id" integer NOT NULL REFERENCES "testapp_parentmodel"
("id"),
UNIQUE ("from_parentmodel_id", "to_parentmodel_id")
);
CREATE INDEX "testapp_parentmodel_slug" ON "testapp_parentmodel" ("slug");
I inserted test1, test2 and test3(all names being values of slug field) to
ParentModel and linked test1 and test3 to test2 through links field. There
seems to be no issues. Then I inserted testc1, testc2 and testc3 to
ChildModel and linked testc2 and testc3 to testc1 through inherited links
field. Now I can see only to links, i.e
I get these results in the interactive interpreter:
>>> from dummy.testapp.models import ParentModel, ChildModel
>>> c1 = ChildModel.objects.get(slug='testc1')
>>> c2 = ChildModel.objects.get(slug='testc2')
>>> c3 = ChildModel.objects.get(slug='testc3')
>>> c1.links.all()
[]
>>> c2.links.all()
[]
>>> c3.links.all()
[]
This means only to links are being queried and not from links for the
inherited model. I just saw how the queries are working and these are the
queries constructed for c1, c2 and c3 respectively.
SELECT "testapp_parentmodel"."id", "testapp_parentmodel"."slug" FROM
"testapp_parentmodel" INNER JOIN "testapp_parentmodel_links" ON
("testapp_parentmodel"."id" =
"testapp_parentmodel_links"."to_parentmodel_id") WHERE
"testapp_parentmodel_links"."from_parentmodel_id" = 4 LIMIT 21
SELECT "testapp_parentmodel"."id", "testapp_parentmodel"."slug" FROM
"testapp_parentmodel" INNER JOIN "testapp_parentmodel_links" ON
("testapp_parentmodel"."id" =
"testapp_parentmodel_links"."to_parentmodel_id") WHERE
"testapp_parentmodel_links"."from_parentmodel_id" = 5 LIMIT 21
SELECT "testapp_parentmodel"."id", "testapp_parentmodel"."slug" FROM
"testapp_parentmodel" INNER JOIN "testapp_parentmodel_links" ON
("testapp_parentmodel"."id" =
"testapp_parentmodel_links"."to_parentmodel_id") WHERE
"testapp_parentmodel_links"."from_parentmodel_id" = 6 LIMIT 21
>From what I see the queries must be actually UNION of two queries which
fetch data both from, "from" and "to" fields in the
"testapp_parentmodel_links" table.It should look some thing like this, from
what I have understood:
SELECT "testapp_parentmodel"."id", "testapp_parentmodel"."slug" FROM
"testapp_parentmodel" INNER JOIN "testapp_parentmodel_links" ON
("testapp_parentmodel"."id" =
"testapp_parentmodel_links"."to_parentmodel_id") WHERE
"testapp_parentmodel_links"."from_parentmodel_id" = 4 LIMIT 21
UNION
SELECT "testapp_parentmodel"."id", "testapp_parentmodel"."slug" FROM
"testapp_parentmodel" INNER JOIN "testapp_parentmodel_links" ON
("testapp_parentmodel"."id" =
"testapp_parentmodel_links"."from_parentmodel_id") WHERE
"testapp_parentmodel_links"."to_parentmodel_id" = 4 LIMIT 21
Is this an elegant approach? If so can some one help me implement this? I
went through the code of QuerySet, Manager classes and other parts of
django.db.models. How to go about from here? I am very keen on fixing this
issue. Kindly help me.
--
Thanks and regards,
Madhusudan.C.S
Blogs at: www.madhusudancs.info
Official Email ID: madhusu...@madhusudancs.info
P.S: I am really sorry if the text is not wrapped. I am still not
understanding how wrapping works here. Will correct it from next mail. Hope
Auto text wrapping works.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---
Re: Ticket #8161 - Please help me fix this.
Hi Alex,
First of all thanks a lot for the reply.
On Wed, Mar 11, 2009 at 9:11 PM, Alex Gaynor wrote:
>
> On Wed, Mar 11, 2009 at 10:31 AM, Madhusudan C.S
> wrote:
>
>> Hi all,
>> I wanted to fix this bug. I just went through the code to see what
>> was happening. For the same sample code given in the ticket, these were my
>> findings:
>>
>> The schema generated looks like this and there is no issue with it:
>> CREATE TABLE "testapp_childmodel" (
>> "parentmodel_ptr_id" integer NOT NULL PRIMARY KEY REFERENCES
>> "testapp_parentmodel" ("id"),
>> "text" text NOT NULL
>> );
>> CREATE TABLE "testapp_parentmodel" (
>> "id" integer NOT NULL PRIMARY KEY,
>> "slug" varchar(50) NOT NULL
>> );
>> CREATE TABLE "testapp_parentmodel_links" (
>> "id" integer NOT NULL PRIMARY KEY,
>> "from_parentmodel_id" integer NOT NULL REFERENCES
>> "testapp_parentmodel" ("id"),
>> "to_parentmodel_id" integer NOT NULL REFERENCES "testapp_parentmodel"
>> ("id"),
>> UNIQUE ("from_parentmodel_id", "to_parentmodel_id")
>> );
>>
>> CREATE INDEX "testapp_parentmodel_slug" ON "testapp_parentmodel" ("slug");
>>
>> I inserted test1, test2 and test3(all names being values of slug field) to
>> ParentModel and linked test1 and test3 to test2 through links field. There
>> seems to be no issues. Then I inserted testc1, testc2 and testc3 to
>> ChildModel and linked testc2 and testc3 to testc1 through inherited links
>> field. Now I can see only to links, i.e
>> I get these results in the interactive interpreter:
>>
>> >>> from dummy.testapp.models import ParentModel, ChildModel
>> >>> c1 = ChildModel.objects.get(slug='testc1')
>> >>> c2 = ChildModel.objects.get(slug='testc2')
>> >>> c3 = ChildModel.objects.get(slug='testc3')
>> >>> c1.links.all()
>> []
>> >>> c2.links.all()
>> []
>> >>> c3.links.all()
>> []
>>
>> This means only to links are being queried and not from links for the
>> inherited model. I just saw how the queries are working and these are the
>> queries constructed for c1, c2 and c3 respectively.
>>
>> SELECT "testapp_parentmodel"."id", "testapp_parentmodel"."slug" FROM
>> "testapp_parentmodel" INNER JOIN "testapp_parentmodel_links" ON
>> ("testapp_parentmodel"."id" =
>> "testapp_parentmodel_links"."to_parentmodel_id") WHERE
>> "testapp_parentmodel_links"."from_parentmodel_id" = 4 LIMIT 21
>>
>> SELECT "testapp_parentmodel"."id", "testapp_parentmodel"."slug" FROM
>> "testapp_parentmodel" INNER JOIN "testapp_parentmodel_links" ON
>> ("testapp_parentmodel"."id" =
>> "testapp_parentmodel_links"."to_parentmodel_id") WHERE
>> "testapp_parentmodel_links"."from_parentmodel_id" = 5 LIMIT 21
>>
>> SELECT "testapp_parentmodel"."id", "testapp_parentmodel"."slug" FROM
>> "testapp_parentmodel" INNER JOIN "testapp_parentmodel_links" ON
>> ("testapp_parentmodel"."id" =
>> "testapp_parentmodel_links"."to_parentmodel_id") WHERE
>> "testapp_parentmodel_links"."from_parentmodel_id" = 6 LIMIT 21
>>
>> From what I see the queries must be actually UNION of two queries which
>> fetch data both from, "from" and "to" fields in the
>> "testapp_parentmodel_links" table.It should look some thing like this, from
>> what I have understood:
>>
>> SELECT "testapp_parentmodel"."id", "testapp_parentmodel"."slug" FROM
>> "testapp_parentmodel" INNER JOIN "testapp_parentmodel_links" ON
>> ("testapp_parentmodel"."id" =
>> "testapp_parentmodel_links"."to_parentmodel_id") WHERE
>> "testapp_parentmodel_links"."from_parentmodel_id" = 4 LIMIT 21
>> UNION
>> SELECT "testapp_parentmodel"."id", "testapp_parentmodel"."slug" FROM
>> "testapp_parentmodel" INNER JOIN "testapp_parentmodel_links" ON
&g
Re: Ticket #8161 - Please help me fix this.
Hi Alex,
Sorry for wasting your time :(
On Thu, Mar 12, 2009 at 1:15 AM, Madhusudan C.S wrote:
> Hi Alex,
> First of all thanks a lot for the reply.
>
> On Wed, Mar 11, 2009 at 9:11 PM, Alex Gaynor wrote:
>
>
>>
>> On Wed, Mar 11, 2009 at 10:31 AM, Madhusudan C.S
>> wrote:
>>
>>> Hi all,
>>> I wanted to fix this bug. I just went through the code to see what
>>> was happening. For the same sample code given in the ticket, these were my
>>> findings:
>>>
>>> The schema generated looks like this and there is no issue with it:
>>> CREATE TABLE "testapp_childmodel" (
>>> "parentmodel_ptr_id" integer NOT NULL PRIMARY KEY REFERENCES
>>> "testapp_parentmodel" ("id"),
>>> "text" text NOT NULL
>>> );
>>> CREATE TABLE "testapp_parentmodel" (
>>> "id" integer NOT NULL PRIMARY KEY,
>>> "slug" varchar(50) NOT NULL
>>> );
>>> CREATE TABLE "testapp_parentmodel_links" (
>>> "id" integer NOT NULL PRIMARY KEY,
>>> "from_parentmodel_id" integer NOT NULL REFERENCES
>>> "testapp_parentmodel" ("id"),
>>> "to_parentmodel_id" integer NOT NULL REFERENCES "testapp_parentmodel"
>>> ("id"),
>>> UNIQUE ("from_parentmodel_id", "to_parentmodel_id")
>>> );
>>>
>>> CREATE INDEX "testapp_parentmodel_slug" ON "testapp_parentmodel"
>>> ("slug");
>>>
>>> I inserted test1, test2 and test3(all names being values of slug field)
>>> to ParentModel and linked test1 and test3 to test2 through links field.
>>> There seems to be no issues. Then I inserted testc1, testc2 and testc3 to
>>> ChildModel and linked testc2 and testc3 to testc1 through inherited links
>>> field. Now I can see only to links, i.e
>>> I get these results in the interactive interpreter:
>>>
>>> >>> from dummy.testapp.models import ParentModel, ChildModel
>>> >>> c1 = ChildModel.objects.get(slug='testc1')
>>> >>> c2 = ChildModel.objects.get(slug='testc2')
>>> >>> c3 = ChildModel.objects.get(slug='testc3')
>>> >>> c1.links.all()
>>> []
>>> >>> c2.links.all()
>>> []
>>> >>> c3.links.all()
>>> []
>>>
>>> This means only to links are being queried and not from links for the
>>> inherited model. I just saw how the queries are working and these are the
>>> queries constructed for c1, c2 and c3 respectively.
>>>
>>> SELECT "testapp_parentmodel"."id", "testapp_parentmodel"."slug" FROM
>>> "testapp_parentmodel" INNER JOIN "testapp_parentmodel_links" ON
>>> ("testapp_parentmodel"."id" =
>>> "testapp_parentmodel_links"."to_parentmodel_id") WHERE
>>> "testapp_parentmodel_links"."from_parentmodel_id" = 4 LIMIT 21
>>>
>>> SELECT "testapp_parentmodel"."id", "testapp_parentmodel"."slug" FROM
>>> "testapp_parentmodel" INNER JOIN "testapp_parentmodel_links" ON
>>> ("testapp_parentmodel"."id" =
>>> "testapp_parentmodel_links"."to_parentmodel_id") WHERE
>>> "testapp_parentmodel_links"."from_parentmodel_id" = 5 LIMIT 21
>>>
>>> SELECT "testapp_parentmodel"."id", "testapp_parentmodel"."slug" FROM
>>> "testapp_parentmodel" INNER JOIN "testapp_parentmodel_links" ON
>>> ("testapp_parentmodel"."id" =
>>> "testapp_parentmodel_links"."to_parentmodel_id") WHERE
>>> "testapp_parentmodel_links"."from_parentmodel_id" = 6 LIMIT 21
>>>
>>> From what I see the queries must be actually UNION of two queries which
>>> fetch data both from, "from" and "to" fields in the
>>> "testapp_parentmodel_links" table.It should look some thing like this, from
>>> what I have understood:
>>>
>>> SELECT "testapp_parentmodel"."id", "testapp_parentmodel"."slug" FROM
>>> "testapp_parentmodel" INNER JOIN "testapp_parentmodel_links" ON
>>> (
[GSoC] Serialization Refactor
Hi all, I am a prospective GSoC student who is interested in working on Django this summer. I have fully read 2 the mails about GSoC(Malcolm's mail to a prospective student on django-gsoc list and Jacob's mail for all the prospective students on -devel list.) Thanks to both of you for such an informative and detailed mail. It really helped me. Coming back, I have been using Django from last 8 months and started using Google App Engine some time back in January. Ever since I started using it I felt the need for Django's models to be supported since it requires a duplication of efforts to learn App Engine Models when we already know Django and wanted to see that support in Django badly. And now when I decided to participate in GSoC as a Django student and saw the ideas list I was very happy to see that being listed as one of the ideas. By the time I made some preparations to discuss about that on the list here, I read Malcolm's mail about the same to one of the Student prospect and his note on the Wiki page. It really did disappointed me, but now I understand how huge the project is and how it might turn to be difficult to some one who doesn't know Django internals *extremely* well. But some how I don't want to give up working on Django. I found that the serialization refactor idea interests me. Can some one please tell me who the person-of-contact or most-likely-to-be-mentor for that idea is or should I discuss about that on the list with everyone here in general? Also can I get some more information about it. I went through wadofstuff linked on the ideas page and all the examples given in their website. It looks seriously interesting to me. Can some one please tell me what exactly Django is expecting for it from a student? I just want to add few other things I have in mind for this idea. But this seems to be becoming a very long mail. I will write my additional points in the next mail and will give a small intro about myself in a separate mail as well. -- Thanks and regards, Madhusudan.C.S P.S. Really sorry for such a long mail. Blogs at: www.madhusudancs.info Official Email ID: madhusu...@madhusudancs.info --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
[GSoC] An Introduction about me
Hi all, This is an introduction about myself. Since Jacob said, "And if we don't know them at all, it's hard to trust they'll get things done." I am writing my involvement in Django and other FOSS communities in general here to let you all know something about me. Hope this helps you people to tell me what I need to learn and how to go about the idea I am interested in. I have been interested in contributing to Django even before GSoC ever flashed to me. In January or so, I badly felt the need for Multiple Primary Key support (Ticket #373) and pinged David Crammer about it on #django-dev, since he had done some work on it. Back then I started to read the Django ORM code, but could not write any code after that. Thanks to University coursework :( Later from early March I have been trying to contribute something to Django. (I am free from then and will be mostly free henceforth and totally free in Summer without any other commitments.) I have had some discussions about fixing ticket #8161 on django-devel list (http://is.gd/obr2) but unfortunately it was fixed. So was looking for few other things and I thought I will apply for GSoC as Django student since I felt it lowers the barrier to get started. I am mostly interested in ORM related ideas since I have read most of django.db.* code. (I am sorry, I am not claiming I am very well versed in Django ORM, but I have a fair idea of how the code is written and structured). I am searching for other ORM related ideas from the ticket list. I will get back to you all whenever I find I something interesting. I am involved in FOSS communities from 3 years now and have Python experience of around 1.25 years. I have contributed few patches to projects like Melange (http://code.google.com/p/soc/source/browse/trunk/AUTHORS, the app on which this year GSoC is run, built on Django), KDE Step (http://is.gd/oci7), GNUSim8085(worked for Windows port), RTEMS and quite a few other FOSS projects. -- Thanks and regards, Madhusudan.C.S Blogs at: www.madhusudancs.info Official Email ID: madhusu...@madhusudancs.info --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
[GSoC] Serialization Refactor
Hi all, I just wrote 2 mails about myself and my wish to participate in GSoC as a Django student. Sorry if I am spamming your inboxes. I just want to keep my mails short so people who don't want to read everything in there can skip the mails that are irrelevant to them. Please correct me where ever I am wrong and if I am not doing it the way it must be done here in Django. I hope I understand what Malcolm and Jacob meant when they said this. We make changes because there are use-cases > for them, not because we can. So any proposal should > be driven by trying to fix some existing problem, not > creating a "wouldn't it be nice if...?" situation. I want to work on Serialization Refactor for GSoC. Since what the Django community requires exactly from that idea is still not clear to me, I am requesting any of you to explain a bit on what is expected of that project? In the mean time, before I get the response let me add few more ideas to it. I am proposing this idea as a Django user initially, opening it up for discussion for the rest of the community. I personally feel this is a missing feature in Django and want to see it happen as a "Django user" for sure. (Also please tell me if it is worth opening a ticket on this and sending this idea to Django users list as well for additional feedback? ) Let me begin my idea with an interesting Use Case I have as a Django User(Hope many other users would have felt the same). I am not sure if this already exists in Django. I assume it doesn't from what I have learnt. Please correct me if I am wrong. I have a Web app written in Django which gets its data in a Serialized format. The data source is actually a third- party script which fetches an HTML page from a website parses the data and supplies it in json format to us. (The page parsed is actually an University Result sheet, for which the script has no access to its results Database). I now want to store this data into my Database. But along with the data provided by JSON I need to add some additional administration stuff into the database table for each serialized data I get. One can easily ask me, why can't I use deserialization. But the problem here as I have understood(may be I am wrong, please correct me if so) is, whenever I deserialize the stream data I get, I can only obtain a DeserializedObject that contains a Django object which should contain the full Model data including any PK fields that exist in the model, but not the subset of fields. This is not the case here. I just want to make the Serialized data, I get, a part of the Database Table, say a subset of fields in the table along with other fields too, for example like the time at which this data was recorded in the Database, some indexing stuff among other things. One can also ask me to write a Custom Field which stores the serialized data in a String (i.e as varchar) format or something like that. But from what I understand (from the docs) I can use custom fields for single fields but not for data that must be split over several fields. Thats exactly what is required here, since I get the marks in JSON, I must able to obtain class average over a particular subject and stuff which becomes difficult if I store JSON data as string. Since I need to deserialize the entire string each time I need access to a single field in it. So the solution that appears to me now is to add a Serialization Field support to Django Models. Say something like JSONField and provide Meta Data for the JSON Field Structure in some way, say defining a class for its structure (as we do for ModelAdmin) or providing this in Class Meta inside the Model Definition. This can be done since we will already, at least, know what will be the format of serialized data we recieve (quite obvious, we need to know this, since we cannot process any random serialized data). Hope this is somewhat similar in idea to what is pointed out as ModelAdmin in the ideas list on the wiki page. I would like to add support for JSON and Python serialization through this project during Summer Of Code period and take take up XML and YAML post GSoC since I feel if we include those also it would be too much for 12 weeks project. Just my estimate :( Python Serialization support has another interesting use case I feel. If we allow Python buitin types, at least types like lists, tuples and dictionary fields in Django Models, we will be providing the highest level of Object Oriented Abstraction for Relational Databases. We will make the lives of Django users easier by allowing them to use those Python types easily without having to worry too much about Normalization. But how we implement them will also be interesting and tricky. It obviously requires many design decisions from Django Community. One idea that I get now is to apply the same kind of Normalization we apply to the list of values we have to put it into a Relational Database, like creating a new table for list items and creating a foreign key re
Re: [GSoC] Serialization Refactor
Hi Malcolm and all, On Sat, Mar 21, 2009 at 8:16 AM, Malcolm Tredinnick < malc...@pointy-stick.com> wrote: > > > I want to work on Serialization Refactor for GSoC. Since what > > the Django community requires exactly from that idea is still > > not clear to me, I am requesting any of you to explain a bit > > on what is expected of that project? > > Yes, that seems to be the problem here (in fact, it was what I was > thinking to myself when reading your second mail). > > I thought this problem was going to arise. The one-line suggestions on > the SoC wiki page aren't particularly specific, unfortunately. They also > aren't QA'd in any real way for practicality or difficulty, so it's a > bit of a combination of wishlist and brainstorming. A starting point for > further research, really. The confusion there is our fault, but if you > view it as a starting point for thinking, that will be a good point. > Ah OK. I did not see it that way, sorry. I thought some one who wrote there on the wiki had specific set of things on mind. Will definitely work on it. Thanks. That particular item appears to be very poorly named in the wiki page. > It's not about refactoring at all (which is changing code around to make > new functionality easier, or remove redundancy). It's about adding new > features to the serializers. Enhancing, extending and changing in > various places, not refactoring. Getting it now :) Now, there are a bunch of things that could be worked on in the > serialization space. Have a look at the currently open tickets in that > area (I mean, read them *all*): > > http://code.djangoproject.com/query?status=new&status=assigned&status=reopened&component=Serialization&order=priority I had already seen most of the tickets in Serialization and ORM before making this post. Since it is a huge list combined, I had only glanced through the tickets there. Will get into each of them and will study them in detail. You'll see a few consistent patterns for feature requests and > awkwardness there (beyond the things that are just basic bugs we have to > fix at some point). It's also worth having a look at mailing list posts > (and tickets) that refer to "fixtures", since that's where those things > are used in Django. You'll start to see problems there with items like > content type values changing or references to pk value in other models > that change upon loading. Oh OK. I think most of the problems of this kind are already filed as tickets in the tracker. I remember seeing them, like #6233, #7052, #9422, IIRC. Those are the tickets I have bookmarked here :) I will also look into Mailing list posts for them. > We'd like to change the serialisation format > to be a lot more robust when it's referring to other models in any way. > One possibility is to use a label instead of a value for those fields. > It can even be designed to be backwards compatible (by adding a version > field to any new format). > Oh OK. I think I need to study a bit more about this and jump into discussion. I will read the related code. Adding support for non-model fields to be serialised is another option. I am confused here a bit. Can you please tell me what you meant by saying non-model field? Does it mean Foreign Keys or something like that? Most of the things I see on the DjangoFullSerializers project appear to > be covered in the tickets in Trac. So the question then becomes whether > the goal would be to merge in DjangoFullSerializers, but keeping things > backwards compatible for existing users. Or to take the good ideas and > merge them in in a more piecemeal fashion. Or work through the general > problems raised in the serializer and fixture tickets and posts on the > mailing list. > > Hopefully that gives you a bunch of ideas for a bit more research. Yeah thanks. It definitely gave me a lot of ideas. Working on them right away. > [...] > > > > So the solution that appears to me now is to add a > > Serialization Field support to Django Models. Say something > > like JSONField and provide Meta Data for the JSON Field > > Structure in some way, say defining a class for its structure > > (as we do for ModelAdmin) or providing this in Class Meta > > inside the Model Definition. This can be done since we will > > already, at least, know what will be the format of serialized > > data we recieve (quite obvious, we need to know this, since > > we cannot process any random serialized data). Hope this is > > somewhat similar in idea to what is pointed out as ModelAdmin > > in the ideas list on the wiki page. > > Hmm .. fields that provide serialized data aren't really anything to do > with the serializer. You can already write them now. In fact, people > have. I'm not sure where Meta factors into this, either. Writing a new > custom field type isn't really Summer of Code project (it's Weekend of > Code difficulty, really). Oh is it possible to deserialize the stream data obtained from external source into various number of f
[GSoC] Ticket #5929 (Previously Serialization Refactor)
Hi Malcolm and all, Doesn't ticket #5929 (http://code.djangoproject.com/ticket/5929) look very similar to what I proposed for Python? Allowing Python built in complex datatype objects to be mapped to Relational Database (which includes lists, dictionaries and tuples)? May be an additional abstraction along with what is proposed in the ticket? That too this ticket being accepted by Jacob gives me a rare kind of feeling that this might be a feature that can be implemented along with what I have written above in my previous mails. I request you all to give your feedback? -- Thanks and regards, Madhusudan.C.S Blogs at: www.madhusudancs.info Official Email ID: madhusu...@madhusudancs.info --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: [GSoC] Ticket #5929 (Previously Serialization Refactor)
Hi Malcolm, Thanks a lot for replying. On Sun, Mar 22, 2009 at 3:08 AM, Malcolm Tredinnick < malc...@pointy-stick.com> wrote: > > > Hi Malcolm and all, > >Doesn't ticket #5929 (http://code.djangoproject.com/ticket/5929) > > look very similar to what I proposed for Python? Allowing Python > > built in complex datatype objects to be mapped to Relational > > Database (which includes lists, dictionaries and tuples)? May > > be an additional abstraction along with what is proposed in the > > ticket? > > That's a valid feature request and something we'll definitely implement, > but it's not an obvious consequence of your previous post, since it's > really got nothing to do with serialisation. It's a well-known ticket / > feature request, but there are plenty of other well-known tickets and I > haven't assumed they're part of your proposal, either. I am extremely sorry. I did not mean it was an obvious consequence of my previous post. I had written something similar to support for Complex Datatypes too in my previous post though not obvious, (excerpt from it below). """If we allow Python buitin types, at least types like lists, tuples and dictionary fields in Django Models, we will be providing the highest level of Object Oriented Abstraction for Relational Databases. One idea that I get now is to apply the same kind of Normalization we apply to the list of values we have to put it into a Relational Database, like creating a new table for list items ...""" I actually requested you to tell me, what I have assumed about the ticket is right or wrong, i.e I requested you to tell me if these ideas are in any similar? I am really if I sounded as you said. I've been trying to still work out what it is you're proposing to work > on. "Something to do with serialisation" and something to solve a > problem you're having with an external app seems to be the impression > I've gotten so far. You need to come up with something a bit more > concrete. "I am going to solve these particular problems as represented > by these bugs and/or these mailing list posts showing a common use-case > pattern." is the type of thing that would be encouraging. > > To be very frank I haven't myself come up with anything concrete till now, since the ideas list themselves don't tell anything in specific, but just give a hint to what might be lead to a potential GSoC project. I am just in the process of brainstorming and pooling ideas and discussing with you and the rest of the Django-community(I am afraid since it seems like no one else is interested in this discussion) to come up with something concrete. This process seems to be going slow. Is it possible at all, if you don't mind to give me some IRC time, say for about 20-30 mins or so I can discuss something with you? Btw I am "madrazr" on #django-dev. -- Thanks and regards, Madhusudan.C.S Blogs at: www.madhusudancs.info Official Email ID: madhusu...@madhusudancs.info --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
[GSoC] Proposal for discussion about Serialization requirements and requesting for Review
Hi all,
After some discussions with Malcolm on this list and doing some
research based on the pointers he gave me I have come up with a
rough plan of what I want to do this summer for Django. Since we
are running out of time, I have come up with a *rough draft* of the
proposal without full discussion with the Django community about the
features that can be implemented. So this is in no way a *Complete
Proposal* and I don't want to submit until some discussion on this
happens really. Also the required proposal format asks to put the
links of the devel list discussions that led to the proposal, which I don't
have except Malcolm's mails. So I kindly request you all to review my
proposal thoroughly and suggest me what I can add or subtract from
the proposal. If my propositions and assumptions are true and how I
can correct myself, so that I can submit my proposal to Google.
*Note: *
Django doesn't serialize inherited Model fields in the Child Model. I
asked
on IRC why this decision was taken but got no response. I searched the
devel list too, but did not get anything on it. I want to add it to my
proposal, but before doing it I wanted to know why this decision was
taken. Will it be a workable and necessary solution to add that to my
proposal?
Same is the case for Ticket #10201. Can someone please tell me why
microsecond data was dropped?
Also I am leaving adding extras option to serializers since a patch for it
has already been submitted(Ticket #5711) and looks like a working
solution. If you all want something extra to be done there to
commit it to django trunk, please tell me, I will work on that a bit
and add it to the proposal.
Here is my long long long proposal:
Title: Restructuring of existing Serialization format and improvisation of
APIs
~
Abstract
~
Greetings!
I wish to provide Django, a better support for Serialization by building
upon the
existing Serialization framework. This project includes extending the format
of the
Serialized output that existing Serializer produces by allowing in-depth
traversal of
Relation Fields in a given Model. The project also includes extending the
existing API
to specify the depth of the relations to be serialized, the name of the
related model
to be serialized. The API also provides for backwards compatibility to allow
older
versions of serialized output to work with the to-be introduced changes. All
the
changes will be made keeping in mind 2 important things.
1. All the changes should be backwards compatible (can only break when a
very
important requirement that improves the serialization by many folds
cannot be
implemented without making backwards incompatible changes and django
community gives a GO Green signal for doing so).
2. The serialized data should be useful not just for use withing Django
apps but
also for exporting the data for external use and processing.
~~~
Why?
~~~
- The existing format of the serialized output firstly doesn't specify the
name of the
Primary Key(PK henceforth), which is a problem for fields which are
implicitly set
as PKs (Ticket #10295).
- The existing format only specifies the PK of the related field, but
doesn't traverse it
in depth to specify its fields (Ticket #4656).
- There are no APIs for the above said requirement.
- The inherited models fields are not serialized.
Situations/problems arising from attempting to fix the above problems
- When we allow Serialization to follow relations, it becomes unnatural if
the related Model is included in every relating model data. The data
becomes extremely redundant. Consider the following example.
class Poll2(models.Model):
question = models.CharField(max_length=200)
pub_date = models.DateTimeField('date published')
def __unicode__(self):
return self.question
class Choice2(models.Model):
poll = models.ForeignKey(Poll)
choice = models.CharField(max_length=200)
votes = models.IntegerField()
def __unicode__(self):
return self.choice
The serializing Choice2 Model might look something like below if we allow
following-of-Relations:
[
{
"pk": 1,
"model": "testapp.choice2",
"fields": {
"votes": 1,
"poll": [
{
"pk": 1,
"model": "testapp.poll2",
"fields": {
"question": "What's Up?",
"pub_date": "2009-03-01 06:00:00"
}
}
]
"choice": "Django"
}
},
{
"pk": 2,
"model": "testapp.choice2",
"fields": {
"votes": 2,
"poll": [
{
"pk": 1,
"model": "testapp.poll2",
"fields": {
"question": "What's Up?",
"pub_date": "2009-03-01 06:00:00"
Re: Proposal for discussion about Serialization requirements and requesting for Review
Hi Koen, On Thu, Mar 26, 2009 at 11:35 PM, koenb wrote: > > On 26 mrt, 17:48, "Madhusudan C.S" wrote: > > Hi all, > > [snipped] > > I can't seem to find anything on the problem concerning contenttypes > in your proposal (see [1] for some recent discussion). > It would be nice to see that solved too, since it is one of the > inconveniences that has bitten me several times. > > Thanks a lot for that link. I will work on that in detail and add it to my proposal. -- Thanks and regards, Madhusudan.C.S Blogs at: www.madhusudancs.info Official Email ID: madhusu...@madhusudancs.info --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: [GSoC] Proposal for discussion about Serialization requirements and requesting for Review
Hi all, What a blunder :( I submitted my proposal the way I will have to submit to socghop.appspot.com with lines manually wrapped at 80 chars per line and the groups wrapp it at 75 chars making my proposal look as ugly as possible. Did not realize that it was 75 chars here. Please excuse me, tell me if my proposal is unreadable I will resubmit it with lines wrapped at 70 chars or so. -- Thanks and regards, Madhusudan.C.S Blogs at: www.madhusudancs.info Official Email ID: madhusu...@madhusudancs.info --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: [GSoC] Proposal for discussion about Serialization requirements and requesting for Review
Hi Malcolm, On Fri, Mar 27, 2009 at 9:05 AM, Malcolm Tredinnick < malc...@pointy-stick.com> wrote: > > On Fri, 2009-03-27 at 00:23 +0530, Madhusudan C.S wrote: > > Hi all, > > > > What a blunder :( I submitted my proposal the way I will > > have to submit to socghop.appspot.com with lines manually wrapped > > at 80 chars per line and the groups wrapp it at 75 chars making > > my proposal look as ugly as possible. Did not realize that it was > > 75 chars here. Please excuse me, tell me if my proposal is > > unreadable I will resubmit it with lines wrapped at 70 chars > > or so. > > > Why manually wrap it at all? Email clients have been able to handle > wrapping lines sensibly on behalf of the sender for about 20 years now. > Just type normally and only hit Return between paragraphs. > > Right. I get it now. Won't do that blunder again :( Some of my friends who participated in previous years of GSoC had told me to manually wrap the text since they felt the text would look ugly after submission to Google's app if it is not wrapped with some small paragraphs appearing as a single huge line and also since wrapping gives a neatly presented look too :( -- Thanks and regards, Madhusudan.C.S Blogs at: www.madhusudancs.info Official Email ID: madhusu...@madhusudancs.info --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: [GSoC] Proposal for discussion about Serialization requirements and requesting for Review
f fields attribute(Thanks to
DjangoFullSerializers for giving this idea).
To solve the ticket #5711, I propose a method extra_fields() which returns a
dictionary. It must return dictionary instead of a tuple because most of the
times the extra fields are computed/derived fields. Example below:
class PollSerializer(newserializer.JSONSerializer):
#...
def extra_fields(self):
pub_date_recent = pub_date > '2009-03-15'
return {'is_recent': pub_date_recent}
One can also specify how a Primary Key can be serialized with the method def
pk_serialize() which returns a dictionary. This should address the ticket
#102. Example below:
class PollSerializer(newserializer.JSONSerializer):
#...
def pk_serialize(self):
return {'pk': pk_value, 'pk name': 'id'}
The dictionary can contain any number of items, but the stress is for the
use of *pk_value* at least once to serialize the PK value somewhere. I am
still unsure, if I should make this a method or an attribute. Can some one
kindly give suggestions?
The serialized output after over-riding the pk_serialize() method looks
something like below.
{
"pk": 1,
"pk name": 'id'
"model": "testapp.poll2",
"fields": {
"pub_date": "2009-03-01 06:00:00",
"question": "What's Up?"
}
}
An additional model_extras() method can be overridden, which by default
returns nothing in the Parent classes. But in the over-ridden method of the
derived class this can return a dictionary of values which are added to the
Model's serialized data. An example of this can be version number of the
serialized format. API example:
class PollSerializer(newserializer.JSONSerializer):
#...
def model_extras(self):
return {'version': '2.1'}
Finally coming to the big thing, Ticket #4656, I propose 3 Class attributes
for this. First one being select_related (as per your suggestion) which is a
dictionary. The key of the dictionary being the name of the Relation
Attribute and the value is a dictionary. This dictionary can have keys -
'fields' or 'excludefields', whose values are tuples of strings, which
indicate the name of the fields in that model to be selected or excluded. If
this dictionary is empty, it serializes the entire model, by using its
Serialization class similar to this one, if at all defined or using the
existing serializers.
Example:
class ChoiceSerializer(newserializer.JSONSerializer):
#...
select_related = {'poll': {'fields': ('question')}}
NOTE: I am not very sure if I can implement this in the SoC timeline, but I
will include it in the API proposal, if I run out of time I will continue
with this after GSoC. If time permits, well and good, I will implement this
too. The value of 'fields' key in the above dictionary is a tuple of strings
which clearly means I cannot follow a relation on that model. So I wish to
also allow dictionaries in this tuple along with the strings. This
dictionary is again a select_related kind of nested dictionary which can
follow the relation in that realtion and so on.
For the Book, Author, City example you gave, it can looks like this:
class BookSerializer(newserializer.JSONSerializer):
#...
select_related = {
'author': {
'fields': ('name', 'age', {
'city':{
'fields': ('cityname', ...)
}
})
}
}
*END NOTE*
Rest of the following are in the SoC timeline.
The second of the 3 attributes, is the inline_related attribute which can be
set to True. In the parent class this is false. If it is set to true,
Serializer will serialize the select_related relations inline.
The third attribute is the reverse_related. It is again a dictionary,
similar in structure to the select_related dictionary, with keys being the
name of the Model that relates to this model. For example:
class PollSerializer(newserializer.JSONSerializer):
#...
reverse_related = {'choice': {
'fields': ('choice', 'votes')
}}
Last but not the least always exists ;-)
The user registers this PollSerializer class with our serializer framwork,
similar to ModelAdmin as:
serializer.register.model(Poll, PollSerializer)
Now a question arises, what if the user wants to change only the
serialization format i.e notation, nothing else in the entire app? Should he
do the donkey's coding job of copy pasting list_separtor and dict_separator?
I feel he need not. For that I propose the following. The solution is to
define a Serializer class, say AppnameSerializer with what ever app specific
customization he wants(provided by the API) and th
Re: [GSoC] Proposal for discussion about Serialization requirements and requesting for Review
Hello all, Also I would like to add again that, I am madrazr on #django-dev. Whenever I tried to ask something I haven't got any response till now. I am not complaining, I understand it is mainly because of timezone problems. I just want to inform anyone who wants to tell me directly on my face ;-) anything about my proposal that I am available for that :D I will be around whenever I am logged into the channel. -- Thanks and regards, Madhusudan.C.S Blogs at: www.madhusudancs.info Official Email ID: madhusu...@madhusudancs.info --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: [GSoC] Proposal for discussion about Serialization requirements and requesting for Review
Hi Russell,
After some thinking again, I have re-worked on my proposal and come up
with the following idea. Here is my draft proposal. I have also submitted it
to socghop.appspot.com
Let us consider the following two models for discussion through out:
class Poll(models.Model):
question = models.CharField(max_length=200)
pub_date = models.DateTimeField('date published')
creator = models.CharField(max_length=200)
valid_for = models.IntegerField(max_length=200)
def __unicode__(self):
return self.question
class Choice(models.Model):
poll = models.ForeignKey(Poll)
choice = models.CharField(max_length=200)
votes = models.IntegerField()
def __unicode__(self):
return self.choice
This projects begins by providing ModelAdmin and Feeds framework
like APIs for Serializers where the user now will be able to construct
a class for specifying custom serialization formats. I propose the API
based on the following ideas.
The user will first define a Class inherited from the Serializer
framework. The parent class is a generic base Serializer class. The
user defined class is then passed as a parameter to the serialize
method we call when we want to serialize the Models. Within this class
the user will be able to specify the customized serialization format
in which he desires the output. Since Python supports majorly three
data structures, Lists, Tuples and Dictionaries, this format can
contain any of these data structures in any possible order. Examples:
Example 1:
class PollSerializer(Serializer):
custom_format = [("question", "valid_for", "id")]
The output in this case will be a list of tuples containing the values
of question, valid_for and id fields. Here the strings are the names
of the fields in the model.
OR
Example 2:
class PollSerializer2(Serializer):
custom_format = (["question", {
"valid_for_number_of_days": "valid_for"
"Poll ID": "id"
}])
The output in this case will be a tuple of lists containing the values
of question and a dictionary which contains valid_for and id fields
as values and their description as keys of a dictionary.
The implementation although not trivial, will work as follows:
(This is not final. Final implementation will be worked out by
discussing with the community)
- The custom_format will be checked for the type. The top level
structure will be decided from this type. "{}" if dictionary, "()"
if tuple and "[]" if list. In case of XML, the root tag will be
django-objects. Also its children will have tag name as "object"
and include model="Model Name" in the tag. This is same as the
existing XML Serializer till here.
- Further the type of the only item within the top-level structure
is determined. All the django objects serialized will be of this
type. In case of XML, the children of "object" tag will be the tags
having the name "field". The tags will also have name="fieldname"
and type="FieldType" attributes within this tag. Additionally if
these field tags are items of the dictionary, they will have a
description="dictionary_key" attribute in the field tag.
- Further each item within the inner object("question","valid_for"
and "id" in the first example) is checked for the type and the
serialized output will have corresponding type. This is implemented
recursively from this level. In case of XML, however, the name of
the tag for further level groupings will have to be chosen in some
consistent way. My suggestion for now is to name the tags as
"field1" for the third level in the original custom format structure,
"field2" for the fourth level in the original custom format
structure, and so on.
For the second example above, we call the serializer as follows:
serializer.serialize("json", Poll.objects.all(),
custom_serializer=PollSerializer2)
The output looks as follows:
(
["What's Up?", {
"valid_for_number_of_days": "30"
"Poll ID": "1"
}
],
["Elections 2009", {
"valid_for_number_of_days": "60"
"Poll ID": "2"
}
]
)
Also if we use XML,
serializer.serialize("xml", Poll.objects.all(),
custom_serializer=PollSerializer2)
The output looks as follows:
What's Up?
30
1
Elections 2009
60
2
Further when a user wants to include extra fields in the serialized
data like additional non-model fields or computed fields, he needs
to specify the name of the method in the class that returns the value
of this field as the value of that item in his format. It should not
be a String. So that we can check if the item value is callable
and if so we can call that method and use the return value for
s
