Re: HttpRequest.read() issues
I've created two tickets for this, with patches and tests... http://code.djangoproject.com/ticket/15762 - WSGIRequest should wrap the test client wsgi.input in LimitedStream http://code.djangoproject.com/ticket/15763 - MultiPartParser's LimitBytes is now redundant. It's possible that I've misunderstood and you can't assume that it'll be okay to do request.read(BUFFER_SIZE) without respecting CONTENT_LENGTH, although if that's the case then presumably that's an issue itself? (Eg. because it means that you can't hand the request over to some arbitrary parser that just treats it like any other file-like object) Cheers, t. -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Fixing makemessages for Javascript
Jannis and Łukasz have both suggested the same thing: use Babel instead
of xgettext. I understand why: it's a more complete solution than what I
have proposed, which is at heart still a way to trick xgettext into
parsing source code it doesn't natively understand.
I have no experience with Babel, so I don't know what work lies ahead to
integrate it with Django. I have used my code in makemessages, and it
works well.
Questions now include:
1) What can we get done in 1.3.1? Is integrating Babel something that
would have to wait for 1.4?
2) Who is the best expert on Babel and Django that could comment on the
work needed?
3) Are there other opinions about the two paths forward? Are there other
options?
I would like very much to get this problem solved.
--Ned.
On 4/4/2011 6:42 PM, Jannis Leidel wrote:
On 04.04.2011, at 23:15, Ned Batchelder wrote:
Last week I re-encountered the problems with using makemessages on Javascript
files, and lost a couple of half-days to trying to figure out why some of my
translatable messages weren't being found and deposited into my .po files.
After fully understanding the extent of Django's current hack, I decided to
take a stab at providing a better solution.
Background: today, Javascript source files are parsed for messages by running a
"pythonize" regex over them, and giving the resulting text to xgettext,
claiming it is Perl. The pythonize regex simply changes any //-style comment on its own
line into a #-style comment. This strange accommodation leaves a great deal of valid
Javascript syntax in place to confuse the Perl parser in xgettext. As a result,
seemingly innocuous Javascript will result in lost messages:
gettext("xyzzy 1");
var x = y;
gettext("xyzzy 2");
var x = z;
gettext("xyzzy 3");
In this sample, messages 1 and 3 are found, and message 2 is not, because
y;ABC;abc; is valid Perl for a transliteration operator. Digging into
this, every time I thought I finally understood the full complexity of the
brokenness, another case would pop up that didn't make sense. The full
horror of Perl syntax
(http://perldoc.perl.org/perlop.html#Quote-and-Quote-like-Operators , for
example) means that it is very difficult to treat non-Perl code as Perl and
expect everything to be OK. This is polyglot programming at its worst.
This needs to be fixed. To that end, I've written a Javascript lexer
(https://bitbucket.org/ned/jslex) with the goal of using it to pre-process
Javascript into a form more suitable for xgettext. My understanding of why we
claim Javascript is Perl is that Perl has regex literals like Javascript does,
and so xgettext stands the best chance of parsing Javascript as Perl. Clearly
that's not working well. My solution would instead remove the regex literals
from the Javascript, and then have xgettext treat it as C.
Thanks Ned, I meant the post about this issue here after 1.3, since we also
talked about this during the Pycon sprint, especially since we seem to have hit
a few more problems with the recent gettext 0.18.1.1 (such as a seemingly
stricter Perl lexer) -- which I encountered while I applied the final
translation updates right before 1.3 but didn't have time to investigate yet.
The bottom line is that I think we should rethink the way we look for
translateable strings instead of working around the limitations of xgettext.
1. Is this the best path forward? Ideally xgettext would support Javascript
directly. There's code out there to add Javascript to xgettext, but I don't
know what shape that code is in, or if it's reasonable to expect Django
installations to use bleeding-edge xgettext. Is there some better solution
that someone is pursuing?
We can't really expect Django users to upgrade to the most recent (or even an
unreleased) version of gettext, We've bumped the minimum required version in
Django 1.2 to 0.15 once all OSes were covered with installers. Which made me
talk to Armin Ronacher about using Babel instead of GNU gettext, since it has a
JavaScript lexer and is in use in Sphinx and Trac. [1] In that sense, I
wholeheartedly encourage you to take a stab at it for 1.4 -- if you think
that's a good idea.
Having a Python based library (assuming it works similarly) seems like a better
fit to Django than relying on a C program.
2. Is there some other badness that will bite us if we tell xgettext that the
modified Javascript is C? With a full Javascript lexer, I feel pretty
confident that we could solve issues if they do come up, but I'd like to know
now what they are.
I feel this is much better solved once and fall all than to keep misusing
xgettext.
Jannis
1: http://babel.edgewall.org/browser/trunk/babel/messages/jslexer.py
--
You received this message because you are subscribed to the Google Groups "Django
developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to
django-d
Re: [GSoC] Revised form rendering
On Sun, Apr 3, 2011 at 11:46 PM, Gregor Müllegger wrote:
> Hi,
>
> 2011/4/2 Russell Keith-Magee :
>> On Fri, Apr 1, 2011 at 11:57 PM, Gregor Müllegger
>> wrote:
>> Firstly, while it looks fine for a small example, I can see how it
>> would rapidly deteriorate if you have lots of fields, or lots of
>> custom field requirements for each field. Django's template language
>> doesn't allow you to split tags over multiple lines, so what happens
>> when my tag runs over 80 characters (this simple example is already a
>> 69 characters)?
>
> (Having the possibility of multiline tags would be nice in many other cases as
> well ... but that's another topic :-)
I'm not convinced it would be nice. Template tags are intended to be
short and pithy, not space in which you can define your own
domain-specific language. My feeling is that if a template tag
proposal needs to invoke the "if only tags could span multiple lines"
clause, the tag is already doing too much in it's definition.
The one exception I will grant to this is comments -- requiring {%
comment %} {% endcomment %}, or {# #} on every line is just daft,
IMHO. However, it's a daftness that hasn't progressed into enough of
an itch to drive me to scratch it myself. It's also unrelated to your
proposal :-)
> I don't see the use of more than two or three modifiers in a template tag in a
> day to day use.
Well.. I gave you an example -- a form with 10 fields, and I want to
render 8 of them.
> However it's of course possible to produce complex statements
> that would need to span multiple lines. And it's already possible with the
> syntax described in the proposal.
>
> You can use a {% formblock %} to extract the modifiers out of the form tag:
>
> {% formblock using layout "uni_form" %}
> {% formblock using widget template "calendar" for myform.birthday %}
> {% form myform using fields "firstname" "lastname" "birthday" %}
> {% endformblock %}
> {% endformblock %}
>
> Some might not like the syntax since you would encourage another level of
> indention and another two lines for the closing tags. But you can use the
> "configure" statement as described in the media handling section of my
> proposal:
>
> {% form myform configure layout "uni_form" %}
> {% form myform configure widget template "calendar" for myform.birthday %}
> {% form myform using fields "firstname" "lastname" "birthday" %}
I hadn't quite grasped how you were using formblock in your previous
example. I now see what you're trying to do, and it makes a lot more
sense. However, I'm not completely sold on certain aspects of this
syntax.
In particular, I have two objections:
* it feels like it's trying to cram too much into a single template tag.
* it requires a lot of duplication of "form myform configure" or
"formblock myform using"
Consider a slight variation that uses more tags:
{% form myform configure %}
{% layout "uni_form" %}
{% widget "calendar" for DateField %}
{% autocomplete "lastname" from "/users/lastnames" %}
{% endform %}
Then, when it comes time to render:
{% form myform using %}
{% field "firstname" %}
And some HTML decoration
{% field "lastname" %}
{% endform %}
or, using an iterative approach:
{% form myform using %}
{% for field in myform %}
{% field field %}
{% endfor %}
{% endform %}
or, using default layout:
{% form myform %}
This approach:
* cuts down on the duplication, either in the form of indented form
blocks, or in duplicated 'tag opening' content
* avoids the need for a DSL embedded in the tag definition
* avoids potential grammar ambiguities -- the less grammar there is
in a tag, the less likely there will be edge cases in the parsing.
>> Secondly, you appear to be using a single template tag for rendering
>> both forms *and* fields. Is there a motivation behind this?
>
> Yes there is :-) The motiviation is, that {% form myform.birthday %} is just a
> shorthand for {% form myform using fields "birthday" %}. You can also use it
> to iterate over a form (like in your example below) and outputting it field by
> field:
>
> {% for field in myform %}
> {% form field %}
> {% endfor %}
>
> I also don't see a need for an extra {% formfield %} tag. I don't see any
> cases in which a {% formfield %} would be shorter, more flexible, easier to
> use. It might be more readable, but I think it would end up as a stripped down
> copy/alias of the form tag.
See my previous comments about cramming everything into a single tag.
Also, there's an implementation issue here; if the first argument to
form can be a field or a form, then your implementation will need to
have logic to differentiate the two -- either isinstance checks, or
duck-type checks. Either way, it's a problem that is avoided by being
explicit: {% form %} operates on forms, {% field %} operates on
fields.
>> Rendering modifiers
>>
>>
> But like I already wrote above: I see your concerns with adding yet anoth
Using OneToOneField in reverse from values or values_list is rejected with "Invalid field name" (repost)
My apologies for reposting this - I originally posted this at the end
of January and as soon as I posted it I realised that with the push
for bug squashing for the 1.3 release going on I probably wouldn't get
any attention (and thus comment or replies :), so I'm trying a repost
now that 1.3 is out. If someone knowledgeable about the query code in
django could comment on whether the behaviour is by design or an
oversight that would be useful!
(Below where I say "reverse name" I mean using that reverse name from
the model that the OneToOneField relates to)
-
Working on some code I've hit that the reverse name for a
OneToOneField cannot be used, it seems, in values_list or values. I
couldn't see why there would need to be such an exclusion, given it is
a one-to-one relationship, so am wondering if it's just an oversight
in the code or a deliberate exclusion?
The portion of code that is rejecting doing e.g.
"values_list('onetoonereversename')" is in setup_joins where the if
line reads:
if not allow_many and (m2m or not direct):
The values of "m2m" and "direct" come from opts.get_field_by_name and
it seems that the reverse of a OneToOne gives the same results as the
reverse of a ForeignKey, as of course both are not m2m and are not
direct. Looking at the code later on in setup_joins it describes the
path where direct is false and m2m is false as being "One-to-many
field (ForeignKey defined on the target model)", so it looks like the
logic I've quoted above of "if not allow_many and (m2m or not direct)"
has the "not direct" clause in to reject reverse ForeignKey
relationships and in doing so also rejects reverse OneToOne
relationships (and with the latter case I suspect it need not!).
If someone a bit more knowledgeable about the query machinery could
comment on whether it does look like an oversight or is a deliberate
exclusion, then if it's the former I'll log it as a bug.
Assuming the rest of the code is ok in the case of a reverse OneToOne
then something along the lines of "if not allow_many and (m2m or (not
direct and not isinstance(getattr(field,'field',None),OneToOneField))"
instead could work to allow OneToOne reverses through in setup_joins
when allow_many is False.
Regards,
Matt
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
Re: Customizable Serialization
On Thu, Mar 31, 2011 at 7:38 PM, Vivek Narayanan wrote:
> Hi Russ,
>
> Thanks for the suggestions once again, I've thought of changing the
> model for handling nested fields.
>
> Each model can have a no of serializers, and they can be plugged in to
> other serializers
> and in this way nested models can be handled instead of cycling
> through a tree of arbitrary depth.
> Each serializer object would have a ``dump_object()`` method that
> would return a python object.
> This method is called when a serializer is plugged into another in a
> nested model.
>
>>My point is that it should be *possible* to define a "generic"
>>serialization strategy -- after all, that's what Django does right
>>now. If arguments like this do exist, they should essentially be
>>arguments used to instantiate a specific serialization strategy,
>>rather than something baked into the serialization API.
>
> Yes, I'll be adding arguments ``attrs`` and ``exclude`` to the
> __init__ method of the serializer.
>
> Here is how the API can be used for generating the formats in those 2
> examples:
>
> """
> For the first option.
> """
>
> class OriginalOutput(JSONSerializer):
> wrap_fields = "fields"
> indent = 4
>
> def meta2_pk(self, obj):
> #get pk
> def meta2_model(self, obj):
> #get model name
So, by my count, here's a list of your implicit assumptions in this
syntactical expression:
* the fields are embedded in a substructure within the main model serialization
* ... and this is something that is sufficiently common that it
deserves first-class representation in your serializer syntax
* The pk is serialized as a top-level attribute
* ... and it's serialized using the attribute name 'pk'
* The model name is serialized as a top-level attribute
* ... and it's serialized using the attribute name 'model'
* indentation is something that has been defined as an attribute of
the serializer, rather than as a configurable item decided at
rendering time (like it is currently)
There are also a bunch of implicit rules about the ways foreign keys,
m2ms and so on are rendered. These aren't so concerning, since there
needs to be some defaults if the syntax is going to be even remotely
terse. However, it bears repeating that they are implicit.
> """
> The second option.
> """
>
> class Base(JSONSerializer):
> """
> A base serializer with formatting options.
> """
> indent = 4
>
> class Editor(Base):
> """
> This can be used to serialize a Person model.
> Note the company field, if no such Serializer object is specified,
> the foreign key is returned.
>
> By specifying the ``attrs`` argument while initializing the
> object, one can restrict the fields
> being serialized. Otherwise, all available fields and metadata
> will be serialized.
> """
> company = Company(from_field = 'company', attrs = ['name',
> 'founded'])
ok - so:
* Where is "Company" defined?
* Why aren't the attributes of Company embedded one level deep?
* Why is the attribute named company, and then an argument is passed
to the Company class with the value of 'company'?
> class Author(Base):
> """
> The dump_object method is called when a serializer is plugged into
> another and returns a python object,
> this is normally a dict but this can be overridden.
> """
> def dump_object(self, obj):
> return obj.first_name + ' ' + obj.last_name
* Why is there a difference between a serializer that returns a list,
a serializer that returns a dictionary, and a serializer that returns
a string?
> class BookDetails(Base):
> """
> This is the serializer that will yield the final output.
>
> Since the 'authors' field is M2M, the Author serializer will be
> applied over the list of authors.
>
> Aliases is a dict that maps attribute names to their labels during
> serialization.
>
> """
> wrap_all = "book details"
> authors = Author(from_field = 'authors')
> editor = Editor(from_field = 'editor', attrs = ['firstname',
> 'lastname', 'coworkers', 'phone', 'company'])
> aliases = {
> 'title': 'book_title',
> 'acount': 'author_count'
> }
>
> def meta2_acount(self, obj):
> # return count of authors
>
>
> The serialized data can be obtained by calling BookDetails.serialize()
> or OriginalOutput.serialize().
>
> Some of the options I didn't cover in the above example are:
>
> * A ``fieldset`` option, which is a list of fields to be serialized.
> If it is not set, all fields are part of the
> serializer. This can be additionally restricted during initialization
> as shown above.
>
> * Reverse relations can be used in the same way as other pluggable
> serializers by
> specifying the ``from_field`` argument as the related name during
> initialization.
>
> I hope this will add more flexibility.
I'm sorry, but this seems extremely confused to me.
You have fields with names like "wrap_all" and "wrap_fields". These
are presumably fixed names that will be interpreted b
Re: Customizable Serialization Proposal
On Sun, Apr 3, 2011 at 6:58 AM, DaNmarner wrote: > I've got some more crystalized details about my proposal. > > Here are the target result if this proposal is implementated: > > 1. User could configure serialization for each model. > > 2. In fact, user could configure the serialization details down to > each field > of each model. That means: > > 3. The key name of each field would be freely defined, the value would > be > freely and dynamically defined. > > 4. Nested structure could be represented as either fk or get > serialized as > substructures in the result. > > 5. The depth of serializaton can be defined, again, down to each > field. > > 6. Various bonuses. e.g. "sorters" for serialized result, values > that's not > from the query set, etc. > > > First off, the current assumption on the structure of a Model, as far > as > serialization is concerned, should be changed. I propose that we look > at the > model as a flat key-value structure, that is, pk and model name (and > potentially other metadata) are no different to a user-defined field. > > That is the base of everything below. So from now on, I'll stop > distinguishing > user-defined fields, pk and metadata in my discussion, and call them > "fields" > instead. > > I want to add APIs targeting two level of abstraction: > > "class SerializedModel": used to customize how each model is > serialized. > > "class SerializedField": used to customize how each field is > serialized. > > They are to be subclassed by user, serving as configuration for > serialization. > SerializedModel will be passed as options to the actual serializaton > module and > be referenced to through out the process. > > How to achieved those goals in my claim? Let's see what's in those > classes. > > The meat of things lies in the SerializedField class, I want to > introduce the > proposed attributes of this class one by one, along each I'll try > explain my > reasoning behind it and it's relevance to my goals. > > Some "regular" parts of a field: > > "key_name": the name to be used for this field, defaults to the field > name(pk, > model, album). This allows user to represent the field in whatever > name they > want. > > "with_type": a boolean, this let user deside whether the type of this > field > should be part of the result. > > "value_format": default to string repr of the field value, but if the > user > want, she can assign a function to value_format to twist the value > however she > like and return. More on this attr below. > > "belongs_to": where in the result does the field go, default to top > level. If > we were to mimic the current serialization format, this value for all > the > user-defined fields would be "field". > > "as_tag": a boolean. This is here specifically for XMLSerializer. If > set false, > it causes the field to be rendered as attrs. There's more on this part > below. > > If the field references to other models, these addtional attrs for > SerializedField would be used: > > "rel_name": similar to "key_name", but it is for the relation type, if > renderd. > defaults to "rel", as it would be if we were to mimic current XML > format. > > "rel_to_name": similar to "key_name" and "rel_name", the key in the > result > whose value is the model this field points to. > > "with_rel": boolean, determines whether the two fields above shoud be > part of > the result. > > "embed_level": an integer. This is my answer to the question "how to > handle > nested objects?". And it deserves its own paragraph. > > For a reference-typed field, the value of SerializedField.value_format > should > be an instance of SerializedField. If "embed_level" is 0, then only > the foreign > key will be rendered. If it is greater than 0, then the serializer > should go > ahead and serialize whatever this field points to according to the > info > contained in "value_format". "embed_level" would be initialized each > time an > instance of SerializedField is instantiated, which happens when the > serializer > discovers embed_level > 0. Another related note is, when "embed_level" >> 0, > "as_tag" would be forced to be True. > > > Compare to SerializedField, SerializedModel is much simpler. It's > function is > no more than holding info about which field should be included in the > serialization, and mapping the wanted field to its SerializedField > instance. > Inspired by contrib.admin.ModelAdmin, SerializedModel has two > attributes: > > "exclude": a list of unwelcomed fields referenced by their str names. > > "fields": a dict whose keys are str names of wanted fields, and the > values are > SerializedField instances. > > If nessacery, a "sorter" fields which accept a sorter function for the > result > serialization might be included as well. > > The new serialization framework would include default > SerializationField for > all the "native" field types, so that users don't have to create one > for each > fields. > > Here's an illustration of how a simple use case looks like,
Re: Prepared statements proof of concept
On Sat, Apr 2, 2011 at 10:04 PM, Marti Raudsepp wrote: > Hi list, > > I wrote a proof of concept patch to add prepared statement support to > Django for the PostgreSQL backend. Note that it's just a hack to see > if this approach could work at all, I know it's badly written. :) > > The patch is quite simple and so far has worked with all queries > generated by Django, for a few different applications. It added a > noticeable speed boost, though I haven't done any repeatable > benchmarks. The main advantage is skipping redundant planning stages > of similar queries -- especially for web pages which may involve > complex queries -- many JOINs etc -- but only fetch a page of 25 rows. > > Here's the hack: > https://bitbucket.org/intgr/django-queue/src/308dee4377c6/prepared_initial.patch > Screenshot in action: http://ompldr.org/vODIzdQ/django_prepared.png > > Now I'm wondering how to approach a solution that would be mergeable > into Django core. > > I get the impression that Django core developers have been opposed to > built-in connection pooling. However, prepared statements are mostly > useless without persistent connections. Is there any chance that > prepared statements would be accepted into core or is this a show > stopper? I'm willing to study Django's internals and take on a fair > bit of work to implement this feature. To clarify -- we've historically been opposed to adding connection pooling to Django is for the same reason that we don't include a web server in Django -- the capability already exists in third party tools, and they're in a position to do a much better job at it than us because it's their sole focus. Django doesn't have to be the whole stack. In principle, I am (and, I suspect the same is true of most of the core team) open to any suggestion that exposes a performance feature of the underlying data store. However, absent of details, it's difficult to say whether a proposal would gain traction. If it's a feature that is only of benefit in pooled environments, the barrier to entry will be higher. However, it might be enough to stimulate some discussion into how to improve Django's support for connection pooled environments -- especially if you can demonstrate some real-world performance benefits. Acceptance will also depend on the invasiveness on the change you're proposing -- if you need to gut SQL compilers to make it work in an elegant way, the level of enthusiasm probably won't be high. The user-facing API will also matter -- how do you propose to wrap the underlying feature in a mechanism that makes sense in ORM terms. It's also worth pointing out that proposals to add APIs are generally looked upon more favorably than specific feature additions -- especially for 'edge case' improvements. Adding a feature means the core team inherits a feature we have to look after; adding an APIs lets others implement and maintain the features, and we just have to keep the API consistent. I don't know if this approach will be viable in your case, but it's worth considering. So, the answer is a definite "maybe" :-) If you can provide some more details, we might be able to provide a more definitive answer. Yours, Russ Magee %-) -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Sprints DjangoCon Europe
Hi all, I'm sure you've all heard of the upcoming DjangoCon Europe. Of special interest to Django Developers will be the sprints, on the 9th & 10th of June. We've got a very special location: http://djangocon.eu/venues/#sprint_venue Of further interest: the sprints will run for 48 hours straight. We even have field beds... I think it would be great if we can have some (or a couple) of themes to work on / towards. Can we have the sprints coincide with "a bigger picture" in Django? If this is too soon, I'd be glad to re-ask the same question a month in advance... regards, Klaas -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Sprints DjangoCon Europe
Nice!!! But, at over 500 euros, and the hotel costing 140 euros per night, I probably won't be able to attend.. (just simply can't justify spending 1000 euros on a conference, no matter how awesome it is lol). I'll look forward to seeing the slides / pictures though! Cal On Tue, Apr 5, 2011 at 9:58 PM, Klaas van Schelven < klaasvanschel...@gmail.com> wrote: > Hi all, > > I'm sure you've all heard of the upcoming DjangoCon Europe. Of special > interest to Django Developers will be the sprints, on the 9th & 10th > of June. We've got a very special location: > http://djangocon.eu/venues/#sprint_venue > Of further interest: the sprints will run for 48 hours straight. We > even have field beds... > > I think it would be great if we can have some (or a couple) of themes > to work on / towards. Can we have the sprints coincide with "a bigger > picture" in Django? > > If this is too soon, I'd be glad to re-ask the same question a month > in advance... > > regards, > Klaas > > -- > You received this message because you are subscribed to the Google Groups > "Django developers" group. > To post to this group, send email to django-developers@googlegroups.com. > To unsubscribe from this group, send email to > django-developers+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/django-developers?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Customizable Serialization Proposal
> 1) It's almost illegible. Between your mail client's munging of line > wrapping, and the completely unreadable code samples, it's very > difficult to tell if you have presented a good idea. Pardon the format. I actually auto-wrapped the text with vim and copy pasted at the first time. Realizing the result is ugly, I actually deleted it from the thread (using the google group web) and reposted without extra formatting. I was going to apologize for the duplication but didn't want to further spam everyone on this list. I believe what you read is the first copy, and the 2nd would be better. > 2) You haven't done the one thing that the Django GSoC wiki recommends > -- provide, as a proof of your concept, an expression of Django's own > serializers using your proposed serialization syntax. > > If you'd done any research on the topic, you'd find that I've raised > the second point every single time this topic is proposed -- in fact, > there's a currently active django-dev discussion with another student > proposing a serialization project. I regret that the unfortunate format stopped you from reading through my proposal, but my code example merely presented a case where the "extra" customization is provided: excluding fields, renaming key names, serializing nested objects. If none of above needs exist, then the user *don't* need to write anything. A default configuration would be used. And that, I assumed from the beginning, would be the current format the serializers use. -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Customizable Serialization Proposal
As a side note: I encountered the use case for customizing serialization last summer when building my own blog project (I wanted an app that could import/export my models from/to wordpress/blogger XML. The API I proposed largely emerged from my thought back then. I'm willing to commit efforts to make Django able to do that in all seriousness, through GSoC or not. On Apr 5, 6:44 pm, DaNmarner wrote: > > 1) It's almost illegible. Between your mail client's munging of line > > wrapping, and the completely unreadable code samples, it's very > > difficult to tell if you have presented a good idea. > > Pardon the format. I actually auto-wrapped the text with vim and copy > pasted at the first time. Realizing the result is ugly, I actually > deleted it from the thread (using the google group web) and reposted > without extra formatting. I was going to apologize for the duplication > but didn't want to further spam everyone on this list. > > I believe what you read is the first copy, and the 2nd would be > better. > > > 2) You haven't done the one thing that the Django GSoC wiki recommends > > -- provide, as a proof of your concept, an expression of Django's own > > serializers using your proposed serialization syntax. > > > If you'd done any research on the topic, you'd find that I've raised > > the second point every single time this topic is proposed -- in fact, > > there's a currently active django-dev discussion with another student > > proposing a serialization project. > > I regret that the unfortunate format stopped you from reading through > my proposal, but my code example merely presented a case where the > "extra" customization is provided: excluding fields, renaming key > names, serializing nested objects. > > If none of above needs exist, then the user *don't* need to write > anything. A default configuration would be used. And that, I assumed > from the beginning, would be the current format the serializers use. -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Customizable Serialization Proposal
On Wed, Apr 6, 2011 at 7:44 AM, DaNmarner wrote: >> 1) It's almost illegible. Between your mail client's munging of line >> wrapping, and the completely unreadable code samples, it's very >> difficult to tell if you have presented a good idea. > > Pardon the format. I actually auto-wrapped the text with vim and copy > pasted at the first time. Realizing the result is ugly, I actually > deleted it from the thread (using the google group web) and reposted > without extra formatting. I was going to apologize for the duplication > but didn't want to further spam everyone on this list. > > I believe what you read is the first copy, and the 2nd would be > better. > >> 2) You haven't done the one thing that the Django GSoC wiki recommends >> -- provide, as a proof of your concept, an expression of Django's own >> serializers using your proposed serialization syntax. >> >> If you'd done any research on the topic, you'd find that I've raised >> the second point every single time this topic is proposed -- in fact, >> there's a currently active django-dev discussion with another student >> proposing a serialization project. > > I regret that the unfortunate format stopped you from reading through > my proposal, but my code example merely presented a case where the > "extra" customization is provided: excluding fields, renaming key > names, serializing nested objects. > > If none of above needs exist, then the user *don't* need to write > anything. A default configuration would be used. And that, I assumed > from the beginning, would be the current format the serializers use. Granted. However, saying "It will exist" isn't a compelling argument for us to pick your project. I already know it will exist. What I don't know is what it will look like under your proposal. So - show me what it would look like. Show me how I, as a third party, would use your proposed syntax to define output that would match Django's existing serialization scheme. Yes, this serialization format will exist as a built in default; but I should be able to reproduce that format in my own code if I want. And then, for bonus points, show me how to define a serializer that matches the pathological case that I posted in the other thread. The point here is to provide a practical demonstration that your proposed syntax is flexible and clear enough to do the job it needs to do. I don't doubt for a second that you want to do this project, and that *you* think that you can do it. However, you still need to convince *us* of that fact. We're not going to accept a proposal without a clear indication that your proposal is at least heading in the right direction are at least on the right track. Yours Russ Magee %-) -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: [GSoC] Composite fields: proposal, part 1
I started writing the draft for a full proposal, however, I don't have time to finish it today as I have to revise for tomorrow's exam. I will try to finish it in 12 hours at most since I know I'm already posting it a little bit too late to make it possible to review it thoroughly. Anyway, I'll post at least the part I have already written so that at least something can be commented on for now. Just to make sure there won't be any formatting issues, I uploaded it here: http://people.ksp.sk/~johnny64/GSoC-full-proposal GSoC 2011 Proposal: Composite Fields About me My name is Michal Petrucha. I'm an undergrad student of computer science at the Faculty of Mathematics, Physics and Informatics, Comenius University, Bratislava, Slovakia. As a high school student I participated in programming contests such as the Olympiad in Informatics. While developping an application for internal use by the organizers of several Slovak high school programming contests I got into a situation where having support for composite primary keys would help greatly, therefore I decided to implement it with some extra added value. Synopsis Django's ORM is a powerful tool which suits perfectly most use-cases, however, there are cases where having exactly one primary key column per table induces unnecessary redundancy. One such case is the many-to-many intermediary model. Even though the pair of ForeignKeys in this model identifies uniquely each relationship, an additional field is required by the ORM to identify individual rows. While this isn't a real problem when the underlying database schema is created by Django, it becomes an obstacle as soon as one tries to develop a Django application using a legacy database. Since there is already a lot of code relying on the pk property of model instances and the ability to use it in QuerySet filters, it is necessary to implement a mechanism to allow filtering of several actual fields by specifying a single filter. The proposed solution is a virtual field type, CompositeField. This field type will enclose several real fields within one single object. From the public API perspective this field type will share several characteristics of other field types, namely: - CompositeField.unique This will create a unique index on the enclosed fields in the database, deprecating the 'unique_together' Meta attribute. - CompositeField.db_index This option will create a non-unique index on the enclosed fields. - CompositeField.primary_key This option will tell the ORM that the primary key for the model is composed of the enclosed fields. - Retrieval and assignment Retrieval of the CompositeField value will return a namedtuple containing the actual values of underlying fields; assignment will assign given values to the underlying fields. - QuerySet filtering Supplying an iterable the same way as with assignment to an 'exact'-type filter will match only those instances where each underlying field value equals the corresponding supplied value. Implementation -- Specifying a CompositeField in a Model ~~ The constructor of a CompositeField will accept the supported options as keyword parameters and the enclosed fields will be specified as positional parameters. The order in which they are specified will determine their order in the namedtuple representing the CompositeField value (i. e. when retrieving and assigning the CompositeField's value; see example below). unique and db_index ~~~ Implementing these will require some modifications in the backend code. The table creation code will have to handle virtual fields as well as local fields in the table creation and index creation routines respectively. When the code handling CompositeField.unique is finished, the models.options.Options class will have to be modified to create a unique CompositeField for each tuple in the Meta.unique_together attribute. The code handling unique checks in models.Model will also have to be updated to reflect the change. Retrieval and assignment Jacob has actually already provided a skeleton of the code that takes care of this as seen in [1]. I'll only summarize the behaviour in a brief example of my own. class SomeModel(models.Model): first_field = models.IntegerField() second_field = models.CharField(max_length=100) composite = models.CompositeField(first_field, second_field) >>> instance = new SomeModel(first_field=47, second_field="some string") >>> instance.composite SomeModel_composite(first_field=47, second_field='some string') >>> instance.composite = (74, "other string") >>> instance.first_field, instance.second_field (74, 'other string') One thing that bugs me is the name of the namedtuple -- it is a class, which means InitialCaps is the right way, however, its n
Re: Customizable Serialization Proposal
> So - show me what it would look like. Show me how I, as a third party, > would use your proposed syntax to define output that would match > Django's existing serialization scheme. Yes, this serialization format > will exist as a built in default; but I should be able to reproduce > that format in my own code if I want. # Fair enough. if code > literacy: # here so_be_it() class DefaultSerializedField(object): belongs_to = "field" as_tag = true class DefaultSerializedSpecialField(DefaultSerializedField): " For pk and modelname " belongs_to = None as_tag = false class DefaultSerializedModel(object): exclude = [] fields = None def __init__(self,embed_level): self.embed_level = embed_level #Somewhere in the dark forest of BaseSerializer implemented if not config_is_passed_as_an_arg: config = DefaultSerializedModel(embed_level=0) for item in queryset: pk_config = getattr(config, 'pk', DefaultSerializedSpecialField()) self.process_pk(pk_config) mn_config = getattr(config, 'modelname', DefaultSerializedSpecialField()) self.process_model_name(mn_config) fields = getattr(config, 'fields',()) for name, field_config in fields.: if not name_of(field) in item.exclude: self.process_field(name, field_config) if not fields: default_config = DefaultSerializedField() for field in regular_field_of(item): if not name_of(field) in item.exclude: self.process_field(name_of(field), default_config) The idea is not to abandon all current assumption about how serialization proceeds, but to do so only to a degree such that we get the flexibility needed. -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Customizable Serialization Proposal
On Wed, Apr 6, 2011 at 11:31 AM, DaNmarner wrote: >> So - show me what it would look like. Show me how I, as a third party, >> would use your proposed syntax to define output that would match >> Django's existing serialization scheme. Yes, this serialization format >> will exist as a built in default; but I should be able to reproduce >> that format in my own code if I want. > > # Fair enough. > > if code > literacy: # here > so_be_it() > > > class DefaultSerializedField(object): > belongs_to = "field" > as_tag = true > > class DefaultSerializedSpecialField(DefaultSerializedField): > " For pk and modelname " > belongs_to = None > as_tag = false > > class DefaultSerializedModel(object): > exclude = [] > fields = None > def __init__(self,embed_level): > self.embed_level = embed_level > > > #Somewhere in the dark forest of BaseSerializer implemented > > if not config_is_passed_as_an_arg: > config = DefaultSerializedModel(embed_level=0) > > for item in queryset: > pk_config = getattr(config, 'pk', DefaultSerializedSpecialField()) > self.process_pk(pk_config) > > mn_config = getattr(config, 'modelname', > DefaultSerializedSpecialField()) > self.process_model_name(mn_config) > > fields = getattr(config, 'fields',()) > for name, field_config in fields.: > if not name_of(field) in item.exclude: > self.process_field(name, field_config) > if not fields: > default_config = DefaultSerializedField() > for field in regular_field_of(item): > if not name_of(field) in item.exclude: > self.process_field(name_of(field), default_config) > > The idea is not to abandon all current assumption about how > serialization proceeds, but to do so only to a degree such that we get > the flexibility needed. No. The idea is to *explicitly encode* the current assumptions, which implicitly allows someone else to define a different set of assumptions. The purpose of this project is to define (and then implement) a DSL for serialization. Django's existing serialization format is just one output product that can be defined using that DSL; ideally, the DSL would be sufficiently expressive to allow any output format to be defined. I would also note that you haven't taken the bait I have repeatedly offered (once in a reply to you, once in the other currently active thread, and many times in the past). Django's serialization language is just one example of an output format. Showing that you can handle a pathological case is a much more rigorous test. Yours, Russ Magee %-) -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Customizable Serialization Proposal
> The purpose of this project is to define (and then implement) a DSL > for serialization. If a DSL is what you are looking for, then I withdraw my proposal. The idea of hosting yet another DSL inside Django project scares me a little. I'll just implement my idea as a separate project, I guess. Thank you for your time. -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: [GSoC] Composite fields: proposal, part 1
Am 06.04.2011 um 02:45 schrieb Michal Petrucha:
[snip]
> unique and db_index
> ~~~
> Implementing these will require some modifications in the backend code.
> The table creation code will have to handle virtual fields as well as
> local fields in the table creation and index creation routines
> respectively.
>
> When the code handling CompositeField.unique is finished, the
> models.options.Options class will have to be modified to create a unique
> CompositeField for each tuple in the Meta.unique_together attribute. The
> code handling unique checks in models.Model will also have to be updated
> to reflect the change.
Instead of auto-creating CompositeFields for each composite index you could
just append to _meta.unique_together in CompositeFields.
This prevents cluttering the set of field for introspection. Or is there a
benefit in using full CompositeFields for each index?
> Retrieval and assignment
>
>
> Jacob has actually already provided a skeleton of the code that takes care
> of this as seen in [1]. I'll only summarize the behaviour in a brief
> example of my own.
>
>class SomeModel(models.Model):
>first_field = models.IntegerField()
>second_field = models.CharField(max_length=100)
>composite = models.CompositeField(first_field, second_field)
>
instance = new SomeModel(first_field=47, second_field="some string")
instance.composite
>SomeModel_composite(first_field=47, second_field='some string')
instance.composite = (74, "other string")
instance.first_field, instance.second_field
>(74, 'other string')
Just a small bikeshed: CompositeField should take the fields as a tuple instead
of *varargs.
> QuerySet filtering
> ~~
>
> This is where the real fun begins.
>
> The fundamental problem here is that Q objects which are used all over the
> code that handles filtering are designed to describe single field lookups.
> On the other hand, CompositeFields will require a way to describe several
> individual field lookups by a single expression.
>
> Since the Q objects themselves have no idea about fields at all and the
> actual field resolution from the filter conditions happens deeper down the
> line, inside models.sql.query.Query, this is where we can handle the
> filters properly.
>
> There is already some basic machinery inside Query.add_filter and
> Query.setup_joins that is in use by GenericRelations, this is
> unfortunately not enough. The optional extra_filters field method will be
> of great use here, though it will have to be extended.
>
> Currently the only parameters it gets are the list of joins the
> filter traverses, the position in the list and a negate parameter
> specifying whether the filter is negated. The GenericRelation instance can
> determine the value of the content type (which is what the extra_filters
> method is used for) easily based on the model it belongs to.
>
> This is not the case for a CompositeField -- it doesn't have any idea
> about the values used in the query. Therefore a new parameter has to be
> added to the method so that the CompositeField can construct all the
> actual filters from the iterable containing the values.
>
> Afterwards the handling inside Query is pretty straightforward. For
> CompositeFields (and virtual fields in general) there is no value to be
> used in the where node, the extra_filters are responsible for all
> filtering, but since the filter should apply to a single object even after
> join traversals, the aliases will be set up while handling the "root"
> filter and then reused for each one of the extra_filters.
As you noted, the problem with GFK, CompositeFields, and virtual fields in
general is mostly that support for filtering (read: custom lookups) requires
changes deep inside the ORM.
I have a half-baked proposal that would make such fields much simpler to
implement but would require a large refactoring of the ORM. It might well be
that it turns out to be impractical, impossible, or too big to be included in
your summer of code project. That being said, here's the plan:
Lookups are handled by the field class. That involves moving significant bits
of code from db.models.sql.query into the field classes. Ideally, Query knows
anything about fieldtypes. It just delegates lookups (including related field
lookups) to the field implementation via:
def add_lookup(self, query, alias, lookup_bits, value, **kwargs):
"""
- `query` is the db.sql.query.Query instance
- `alias` is the table alias of the "current" table; that is:
the table which this lookup must be performed on
- `lookup_bits` is a sequence of lookups, e.g.
"related_field__some_field__icontains".split("__")
- `value` is the filter argument
- `**kwargs`: It might be necessary to pass more information,
but certainly not arbitrary data. I just hav
