Re: [GSoC] Revised form rendering

[Gregor Müllegger]

Hi, thanks for your fast feedback. I was really looking forward to your input,
I haven't reached you in IRC before.

2011/4/1 Carl Meyer :
> Hi Gregor,
>
> As you've probably seen in past threads, this is an area where I'm quite
> motivated to see some improvement. I think you've got quite a strong
> proposal here in general, and you've clearly done your homework, so my
> comments below dive directly into the details:

(I've shorten my proposal a bit in the quotes to make the whole email more
readable)

> On 04/01/2011 11:57 AM, Gregor Müllegger wrote:
>> I suggest reading this proposal online: https://gist.github.com/898375
>> It's exactly the same as below but formated nicely.
>>
>>
>> GSoC 2011 Proposal - Revised form rendering
>> 
>>
>> ...
>>
>> Motiviation
>> ---
>>
>> ...
>>
>>    I want to tell the designer how he can do this without my help in the
>>    template.
>
> I agree with your Motivations section - in particular this final line,
> which sums up the core motivation as I see it.
>
>> Goals I want to accomplish
>> --
>>
>> ...
>>
>> **1. No HTML in python source**
>>
>> ...
>>
>
> This all looks quite good to me.
>
>> **2. Reordering/skipping fields**
>>
>> ...
>>
>> The goal will be to make these modifiers defineable. Which means that you
>> can create your own modifiers to support some of your extravagant form
>> rendering needs. To support this we will need to have a rendering modifier
>> *registry* or something similiar, where you can ideally load new ones with 
>> the
>> existing ``{% load %}`` tag. The same will apply for custom *chrome*,
>> described below.
>
> This last bit worries me; it has the potential to add unneeded
> complexity. Do you have any actual use cases for the "custom rendering
> modifier" that can't be handled adequately by your proposal otherwise?
> The concept of what a "rendering modifier" actually _is_ and what it can
> do both seem poorly defined here. I'd rather see this level of
> flexibility left out (and potentially added later) than done without a
> clear driving concept and motivation.
>

I understand your worries and I clearly don't want to invent a YAGNI case. But
I think it fits nicely into the proposal (of it's current state). The reasons
are chrome. Chrome are just a specialized versions of "rendering modifiers"
(btw: I don't like the name but it's the best I came up with yet). And if you
can make it possible to register your own chrome, then it might be only a very
small overhead (if any) to do the same for rendering modifiers.


I want to make clear what a rendering modifier would be on the python level. I
thought about a FormRenderer class that handles the rendering of the form
based on the some predefined rules like what default layout is used. How to
render widgets and so on.

Rendering modifiers are then decorated around the FormRenderer. An example:
In the template::

{% form myform using layout "p" and widget "CalendarInput" for
myform.birthday %}

The FormRenderer iterates over the fields in myform and wants to render them
with layout "default" and with the widgets defined in the form itself. It
asks then the form modifiers "layout" and "widget" that were used in the form
tag how they want to modify the rendering of a specific field. The "layout"
modifier would always return "use layout p", the "widget" modifier always
returns "use the default widget" except when FormRenderer reaches the birthday
field, then "widget" would return "use CalendarInput widget".

(Sorry for the very easy language, maybe my english is to limited to express
it in a not so verbose way.)


I haven't thought in detail about additional usecases for the rendering
modifiers that might life in the app ecosystem. But while writing this
response, there came some of the possibilities into my mind. Maybe there is
the potential for a doctype modifier::

   {% form myform using doctype "html4" %}

This will add a new template directory to the template path while rendering
the form. So that it renders HTML4 compatible widgets if available in the new
template directory. And if not it falls back to the default templates used by
the new template based widgets.

Something more exotic (and complex to implement but still possible) is a
"validation" rendering modifier::

   {% form myform using validation %}

This will walk through the form and outputs at the end some javascript that
validates the form on the client side based on the used form fields.
This will need a port of python validators into javascript, but that isn't
that complex (I already did a proof of concept for that kind of behaviour in a
template tag).
And thats something that wouldn't get into django core but could life nicely
in **django-clientsidevalidation** :-)


This is why I think that the registry for rendering modifiers is useful. We
would need a registry (i.e. a dict that holds the "name to modifier" mapping)
anyway. So why shouldn't 

uWSGI documentation

[James Pic]

Hello everybody,

Do you think uWSGI deserves a place in the official django documentation ?

I think it should because it's easier, safer, faster and more secure
than flup or mod_wsgi. Also, it made my sysadmin life really easy and
that's something cool to share with the community.

In this case, please consider the attached draft
(django/docs/howto/deployment/uwsgi.txt): it's not perfect and
probably not commitable as-is, but I am ready to listen to feedback
and improve it. Otherwise it'll just make another blog post.

What do you think ?

Cheers

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.


How to use Django with uWSGI


.. highlight:: bash

uWSGI_ is a fast, self-healing and developer/sysadmin-friendly application
container server coded in pure C. It comes with all the advantages of FastCGI:

* sane, runs in a site-specific process other than the web server
* secure, can run with a site-specific user and group
* fast, uWSGI_ is fastest in most setups

_uWSGI comes with additional advantages:

* it provides a simple Web server,
* it can gracefully restart the process, without loosing any requests unlike
  a fastcgi process restart
* lower memory footprint than flup or mod_wgsi
* provides a safe and fast `cache backend for django
  `_

The full list of advantages are on the _uWSGI homepage.

.. _uWSGI: http://projects.unbit.it/uwsgi/

Prerequisite: general concept
=

uWSGI_ provides a command to start a project process. The Web server uses the
process through a socket file created by uWSGI_. The system administrator
controls the process using the pid written by uWSGI_ in the pidfile.

If the Web server doesn't serve the application correctly, the system
administrator should ensure that:

* the Web server log file: it should tell if the Web server managed to connect
  to the uWSGI_ process or not
* if the process socket is a file: the Web server process should have
  write permission on the socket file
* if the process socket is a file: uWSGI_ sometimes won't erase the socket
  file on interruption, remove the file and start uWSGI again in that case
* uWSGI_ can start the process on the foreground which will make errors easily
  visible to the system administrator

Prerequisite: uWSGI
===

`uWSGI wiki`_ describes several
installation procedures. The simplest way is to run this command::

# install current stable version
pip install uwsgi
# install LTS (long term support)
pip install http://projects.unbit.it/downloads/uwsgi-lts.tar.gz

Managing the uWSGI server
=

uWSGI operates on a client-server model, and in most cases you'll be starting
the uWSGI process on your own. Your Web server (be it Nginx, or otherwise)
only contacts your Django-uWSGI process when the server needs a dynamic page
to be loaded. Because the daemon is already running with the code in memory,
it's able to serve the response very quickly.

A Web server can connect to an uWSGI_ server in one of two ways: It can use
either a Unix domain socket (a "named pipe" on Win32 systems), or it can use a
TCP socket. What you choose is a manner of preference; a TCP socket is usually
easier due to permissions issues.

Starting the server
---

uWSGI is highly configurable and thus there are many ways to start the
process. For example, uwsgi version 0.9.6.8 provides a hundred switches.
This guide demonstrates the most important of them, but does not intent to
substitute the official manual and online documentation.

Starting a process with uWSGI takes an additional python script. The script
can be in the project directory, named 'youruwsgi.py'::

from django.core.handlers.wsgi import WSGIHandler

# set application for WSGI processing
application = WSGIHandler()

To communicate with other applications like a Web server, uWSGI_ uses a
socket. You'll need to specify either a :djadminopt:`socket`, a
:djadminopt:`protocol` or both :djadminopt:`host` and :djadminopt:`port`.
Then, when you set up your Web server, you'll just need to point it at the
host/port or socket you specified when starting the FastCGI server.

.. admonition:: Choosing the socket

The easiest is to set the socket to a high level (>1024) local port like
127.0.0.1:3000. If the socket is a file, the system administrator must
ensure that the Web server process has rwx privileges on that file.

To start one uWSGI process to handle requests with your project::

uwsgi --home=/path/to/your/pro

Prepared statements proof of concept

[Marti Raudsepp]

Hi list,

I wrote a proof of concept patch to add prepared statement support to
Django for the PostgreSQL backend. Note that it's just a hack to see
if this approach could work at all, I know it's badly written. :)

The patch is quite simple and so far has worked with all queries
generated by Django, for a few different applications. It added a
noticeable speed boost, though I haven't done any repeatable
benchmarks. The main advantage is skipping redundant planning stages
of similar queries -- especially for web pages which may involve
complex queries -- many JOINs etc -- but only fetch a page of 25 rows.

Here's the hack:
https://bitbucket.org/intgr/django-queue/src/308dee4377c6/prepared_initial.patch
Screenshot in action: http://ompldr.org/vODIzdQ/django_prepared.png

Now I'm wondering how to approach a solution that would be mergeable
into Django core.

I get the impression that Django core developers have been opposed to
built-in connection pooling. However, prepared statements are mostly
useless without persistent connections. Is there any chance that
prepared statements would be accepted into core or is this a show
stopper? I'm willing to study Django's internals and take on a fair
bit of work to implement this feature.

What are the steps to get there?

This is what I currently think needs to be done:
1. Implement some sort of persistent database connection
2. Add prepared statement support for PostgreSQL, default to on for
persistent connections?
3. Except queries with LIKE, extra(), etc, that might not work well
with prepared statements
4. Fix up CursorDebugWrapper to make prepared statements more
transparent for the developer

Some more ideas:
* QuerySet method to force enable/disable preparing
* QuerySet or Q attributes to force certain literals to be constants
* Skip SQL building for prepared statements, by caching statements
based on sql.Query attributes
* Similar prepared statement support provided by MySQL

Regards,
Marti

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Revised form rendering

[Russell Keith-Magee]

On Sat, Apr 2, 2011 at 5:20 AM, Carl Meyer  wrote:
>
> On 04/01/2011 05:09 PM, Mikhail Korobov wrote:
>> Implementation: 
>> https://bitbucket.org/kmike/django-widget-tweaks/src/0e9bac3c71bd/widget_tweaks/templatetags/widget_tweaks.py
>
> That's quite a neat app - I have some similar template filters
> implemented in some of my own projects. It doesn't cover the full range
> of Gregor's proposal -- but I actually think most of Gregor's proposal
> would be possible to implement outside Django. Perhaps we should
> consider whether it would be preferable to start it off that way, or
> focus the proposal more on where Django itself really needs to change.

I think this would be a good way to structure the project -- i.e.,
keep a clear conceptual separation between hooks and configuration
options that need to be added to Django in order to make an external
form rendering library possible, and the library that actually
exploits those hooks and configuration options.

Alex took a similar approach with the NoSQL work last year -- there is
a fairly clear separation between the "query refactor" changes, and
the "MongoDB module" that exploits those changes.

I don't think this needs to change the scope of Gregor's proposal very
much. If we change the focus to be just on the "Django side", we run
the risk of architecture astronauting. Developing a non-trivial
implementation that demonstrates the need for any given refactor helps
to clarify exactly why any given change to Django's APIs are needed.

Yours,
Russ Magee %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: uWSGI documentation

[Łukasz Rekucki]

On 2 April 2011 13:23, James Pic  wrote:
> Hello everybody,
>
> Do you think uWSGI deserves a place in the official django documentation ?
>
> I think it should because it's easier, safer, faster and more secure
> than flup or mod_wsgi. Also, it made my sysadmin life really easy and
> that's something cool to share with the community.
>
> In this case, please consider the attached draft
> (django/docs/howto/deployment/uwsgi.txt): it's not perfect and
> probably not commitable as-is, but I am ready to listen to feedback
> and improve it. Otherwise it'll just make another blog post.

+1. More docs didn't hurt anybody. At least while there is someone to
maintain them.

>
> What do you think ?
>

My first impression is that you're focusing too much on all those
switches. That section got me totally lost. That should be simplified.
Also, do I really need to type all that stuff? I probably won't be
doing that on production, so showing how to place it in a
configuration file would be better.

I would also drop the whole "advantages" section. The purpose of the
page is to show how to deploy a Django app on uWSGI, not compare it
against other solutions. You also mention some features, but never
show how to enable/use them which is confusing.

PS. Do you think having a `runuwsgi` command similar to `runfcgi`
would be useful ?

-- 
Łukasz Rekucki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: RFC #2705 (Support for SELECT FOR UPDATE)

[Dan Fairs]

Hi,

Now 1.3's out the door, I'd like this to be looked at again, if possible.

  http://code.djangoproject.com/ticket/2705



I'll re-apply Ramiro's patch locally and make sure everything still checks out 
for me, and report back.


The latest version of the patch still applied cleanly for me and passed 
tests when I tried it again late last week (r15043).


I hope Jacob's comments about the tests has now been addressed - a 
mistake in transaction setup was masking an error. The tests should now 
be sane, and pass for me on MySQL, PostgreSQL, SQLite and Oracle.


I'd welcome review and feedback on improving the patch; I'd welcome 
seeing it merged even more. :)


Cheers,
Dan
--
Dan Fairs | dan.fa...@gmail.com | www.fezconsulting.com

--
You received this message because you are subscribed to the Google Groups "Django 
developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Revised form rendering

[Gregor Müllegger]

2011/4/2 Russell Keith-Magee :
> I think this would be a good way to structure the project -- i.e.,
> keep a clear conceptual separation between hooks and configuration
> options that need to be added to Django in order to make an external
> form rendering library possible, and the library that actually
> exploits those hooks and configuration options.
>
> Alex took a similar approach with the NoSQL work last year -- there is
> a fairly clear separation between the "query refactor" changes, and
> the "MongoDB module" that exploits those changes.
>
> I don't think this needs to change the scope of Gregor's proposal very
> much. If we change the focus to be just on the "Django side", we run
> the risk of architecture astronauting. Developing a non-trivial
> implementation that demonstrates the need for any given refactor helps
> to clarify exactly why any given change to Django's APIs are needed.
>
> Yours,
> Russ Magee %-)

I agree on this. The best practice seems to keep changes outside of django if
possible until proven useful. Which can be done cleanly with the templatetag
at least. I didn't integrate plans on what goes inside django and what will be
outside of it since I didn't know if GSoC projects are accepted that work most
of the time on an app than on django itself. But if thats possible like with
Alex's work last year, thats fine for me.

Let me split up quickly what might go into django directly and what could live
outside if my proposal gets accepted:

1. Layouts
The form layouts should be included in django itself, refactoring
{{ form.as_table }}, labels, errorlists etc to make django code HTML free :-)

2. The template tag
This one clearly goes outside of django. Including a template tag is pretty
easy. Merging this in later would be also a no brainer.

3. Rendering modifiers
Thats also something that will go into an app. It's tight pretty closely to
the template tag. But this might require some internal changes to forms to
support all of the aspects described in the proposal (I don't see any
necessary changes yet, but it's like Russell describes, we don't know what
needs to be done until we have a full featured, non-trivial implementation).

4. Changes to widgets
I could see some additional APIs to widgets that get included into django
directly. Like a clean API to add attributes (instead of modifying attrs
directly), an API to add classes, an API to change the template that is used
to render the widget ...
All thirdparty apps that provide additional form features like
"django-widget-tweaks" would benefit.

5. Admin integration
This is a big point that is difficult to do. Creating a "admin" form layout is
still possible. But using the template tag and its mechanics inside of the
admin might not be useful until the template tag is merged into django.
However we might use the admin just as proof of concept, developing
integration of the app in another branch of django. But thats not ideal IMO.

6. Documentation
Writing documentation is ofcourse straight forward for the stuff that goes
into django directly. The thirdparty stuff will be documented like any other
reusable app. I don't see very big problems there. It might just generate some
extra work merging it in afterwards if the app code is accepted for django
core. But thats really the tiniest problem and acceptable if we can produce a
more mature product.


Gregor

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Prepared statements proof of concept

[Marti Raudsepp]

On Sat, Apr 2, 2011 at 17:04, Marti Raudsepp  wrote:
> The patch is quite simple and so far has worked with all queries
> generated by Django, for a few different applications. It added a
> noticeable speed boost

Oh, I forgot to mention, I'm using the patch together with the
persistent connections snippet from here:
http://djangosnippets.org/snippets/1707/

Regards,
Marti

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: [GSoC] Revised form rendering

[Russell Keith-Magee]

On Fri, Apr 1, 2011 at 11:57 PM, Gregor Müllegger  wrote:
> I suggest reading this proposal online: https://gist.github.com/898375
> It's exactly the same as below but formated nicely.
>
>
> GSoC 2011 Proposal - Revised form rendering
> 
>
> Hi my name is Gregor Müllegger. I'm a Computer Science student in Germany at
> the University of Augsburg currently in the fourth year of my studies. I first
> came to django shortly before 0.96 was released and a lots of awesomeness was
> introduced with the magic removal branch.

Hi Gregor,

Firstly, I'd like to echo Carl's sentiments -- this is a strong
proposal that shows you've researched the history of Django
discussions on this topic, and given it some thought of your own.

Carl has already raised several of the points that I noticed on my
read through. However, I do have some of my own queries.

Syntax
~~

You've proposed a syntax that goes something like:

{% form myform using layout "p" and fields "firstname" "lastname" %}
{% form fancyform.favourite_color using layout "p" %}

I have two questions about this syntax.

Firstly, while it looks fine for a small example, I can see how it
would rapidly deteriorate if you have lots of fields, or lots of
custom field requirements for each field. Django's template language
doesn't allow you to split tags over multiple lines, so what happens
when my tag runs over 80 characters (this simple example is already a
69 characters)?

Secondly, you appear to be using a single template tag for rendering
both forms *and* fields. Is there a motivation behind this?

Rendering modifiers


I share Carl's concern about exactly how and why these are necessary;
even after your most recent post, I'm still not completely clear on
what they mean in practice.

On the one hand, I can see that there may be a use case for passing
arguments into the template rendering process. However, you seem to be
proposing a mechanism by which developers could define Python code
that alter widget rendering by performing such tasks as:

 * selecting a different rendering directory
 * changing the template that is used for a widget
 * changing the layout scheme
 * prepending/appending content onto a widget

I think I understand your motivation here, but my inclination is that
this will end up being a very complex system that won't end up being
useful in practice.

For example -- consider the case of validation. In an ideal world, it
would be nice to be able to just throw a switch and say "turn on
client side validation for this field", and then have that validation
work regardless of the rendering of the field itself. However, in
practice, it seems to me that the implementation of validation will be
tightly bound to the rendering of the field itself -- i.e, your widget
template would really need to be a "HTML4 text input with validation",
not a "text widget" with the HTML4 option and the validation option
enabled.

I can certainly see value in allowing template designers to pass in
keyword arguments when rendering fields on a template, allowing chrome
developers to define templates that implement various features (e.g.,
define a TextInput template that optionally has validation). However,
making this a complex registry and code solution seems like overkill
to me.

Form rows
~

If you look at the existing as_* form layout tools, there is a
distinction between gross form layout, and the layout of an individual
form row.

{{ form_header }}
{% for field in form %}
{{ pre_field }}}
{{ field }}
{{ post_field }}
{% endfor %}
{{ form_footer }}

Conceptually, the rendering for pre/post field could vary depending on
the field. How is this handled in your proposal? Is it part of the
form renderer? Or part of the widget rendering? This overlaps with
your discussion about error templates -- since errors (and their
rendering) is one of the major pre/post rendering activities for both
forms and fields.

CSS/Media
~

I'm not wild about the idea of having to include the same form content
twice in order to get CSS and JS included as where it is required.

I can see two options here.

Firstly, follow the lead of the {% cycle %} tag -- use the {% form %}
tag to define what the form will look like, but not actually render
where it is defined; assign the rendered properties to a context
variable, then reference that context variable when you want to
actually render something related to the form.

{% form myform  as myform_rendered %}

{{ myform_rendered.media.css }}

{{ myform_rendered.media.js }}

{{ myform_rendered.html }}

This would be fairly easy to implement, but I'm not sure it would be
especially flexible.

Secondly, it may be possible to play a trick on the template compiler.
Jonas Obrist suggested this trick to me at Djangocon (US) last year;
however, I haven't had a chance to dig into it any more.

The trick goes something like this:

When you parse a template tag like an {%

Re: [GSoC] Revised form rendering

[burc...@gmail.com]

On Sat, Apr 2, 2011 at 10:45 PM, Russell Keith-Magee
 wrote:
> Secondly, it may be possible to play a trick on the template compiler.
> Jonas Obrist suggested this trick to me at Djangocon (US) last year;
> however, I haven't had a chance to dig into it any more.
>
> The trick goes something like this:
>
> When you parse a template tag like an {% if %}, the template parser
> scans forward until it reaches an {% endif %}. All the node content
> that is discovered is then provided to the if node as the subtemplate
> for the if.
>
> However, there's no specific requirement that you need to parse until
> you reach a specific tag -- you could just scan to the end of the
> document, and treat the rest of the document as if it were the nested
> content of an if that runs to the end of the page.
>
> So -- you should be able to write a "CSS" template tag that reads to
> the end of the template, and in the process, discovers all the {% form
> %} nodes on the template. From that, you can determine all the CSS
> requirements that your HTML document has.
>
> It requires a bit of a twist in the document tree -- the entire
> document effectively becomes a child of the CSS node in the template
> -- but it should allow you to get access to form rendering
> requirements before the form is actually rendered on the page.

Hi Russell,

Maybe a bit different thing, but if we're talking about forms media
customization,

I'd love the following solution implemented for css and js:
Tag {% media for form1 %} or {% css for form1 %} which would remember
all previous css/js references, displaying only ones not rendered yet.
For example, lots of my widgets are referring jquery js and css, and
there's no need to load jquery 10 times.
And if I loaded jquery before forms, I would like to be able to say
{% js "{{STATIC_URL}}/media/jquery-1.4.2.min.js" %} or {% css
"/media/css/jquery-tweaks.css" %}

Gregor,

Regarding proposal itself,

the idea to make django form rendering based on templates is good, but
I think it should be simple, modular and not much verbose.
I'd like more to see more modular, but easier syntax:
1) If you have to render form in your custom way, you have to write
all form fields (you'll tweak their order and widgets anyway later, so
it's a plus).
2) If you have to render field in your custom way, you have to write
your own widget or widget template.
3) Any advanced form customization can be made using templates.

Something like this:
{% renderform form1 using layout "uni_form" %} {# errors output could
be changed with layout templates only #}
  {# will use template layouts/app1/uni_form/charfield/x.html if it
was charfield. #}
  {% render field1 using widget template "x" set class="field1" %}
  {% render field3 %}
  {% render field2 %}
{% endrenderform %}

-- 
Best regards, Yuri V. Baburov, Skype: yuri.baburov, MSN: bu...@live.com

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Customizable Serialization Proposal

[DaNmarner]

I've got some more crystalized details about my proposal.

Here are the target result if this proposal is implementated:

1. User could configure serialization for each model.

2. In fact, user could configure the serialization details down to
each field
of each model. That means:

3. The key name of each field would be freely defined, the value would
be
freely and dynamically defined.

4. Nested structure could be represented as either fk or get
serialized as
substructures in the result.

5. The depth of serializaton can be defined, again, down to each
field.

6. Various bonuses. e.g. "sorters" for serialized result, values
that's not
from the query set, etc.


First off, the current assumption on the structure of a Model, as far
as
serialization is concerned, should be changed. I propose that we look
at the
model as a flat key-value structure, that is, pk and model name (and
potentially other metadata) are no different to a user-defined field.

That is the base of everything below. So from now on, I'll stop
distinguishing
user-defined fields, pk and metadata in my discussion, and call them
"fields"
instead.

I want to add APIs targeting two level of abstraction:

"class SerializedModel": used to customize how each model is
serialized.

"class SerializedField": used to customize how each field is
serialized.

They are to be subclassed by user, serving as configuration for
serialization.
SerializedModel will be passed as options to the actual serializaton
module and
be referenced to through out the process.

How to achieved those goals in my claim? Let's see what's in those
classes.

The meat of things lies in the SerializedField class, I want to
introduce the
proposed attributes of this class one by one, along each I'll try
explain my
reasoning behind it and it's relevance to my goals.

Some "regular" parts of a field:

"key_name": the name to be used for this field, defaults to the field
name(pk,
model, album). This allows user to represent the field in whatever
name they
want.

"with_type": a boolean, this let user deside whether the type of this
field
should be part of the result.

"value_format": default to string repr of the field value, but if the
user
want, she can assign a function to value_format to twist the value
however she
like and return. More on this attr below.

"belongs_to": where in the result does the field go, default to top
level. If
we were to mimic the current serialization format, this value for all
the
user-defined fields would be "field".

"as_tag": a boolean. This is here specifically for XMLSerializer. If
set false,
it causes the field to be rendered as attrs. There's more on this part
below.

If the field references to other models, these addtional attrs for
SerializedField would be used:

"rel_name": similar to "key_name", but it is for the relation type, if
renderd.
defaults to "rel", as it would be if we were to mimic current XML
format.

"rel_to_name": similar to "key_name" and "rel_name", the key in the
result
whose value is the model this field points to.

"with_rel": boolean, determines whether the two fields above shoud be
part of
the result.

"embed_level": an integer. This is my answer to the question "how to
handle
nested objects?". And it deserves its own paragraph.

For a reference-typed field, the value of SerializedField.value_format
should
be an instance of SerializedField. If "embed_level" is 0, then only
the foreign
key will be rendered. If it is greater than 0, then the serializer
should go
ahead and serialize whatever this field points to according to the
info
contained in "value_format". "embed_level" would be initialized each
time an
instance of SerializedField is instantiated, which happens when the
serializer
discovers embed_level > 0. Another related note is, when "embed_level"
> 0,
"as_tag" would be forced to be True.


Compare to SerializedField, SerializedModel is much simpler. It's
function is
no more than holding info about which field should be included in the
serialization, and mapping the wanted field to its SerializedField
instance.
Inspired by contrib.admin.ModelAdmin, SerializedModel has two
attributes:

"exclude": a list of unwelcomed fields referenced by their str names.

"fields": a dict whose keys are str names of wanted fields, and the
values are
SerializedField instances.

If nessacery, a "sorter" fields which accept a sorter function for the
result
serialization might be included as well.

The new serialization framework would include default
SerializationField for
all the "native" field types, so that users don't have to create one
for each
fields.

Here's an illustration of how a simple use case looks like, taking the
models
from the tutorial as an example:

class SerializedChoiceText(DefaultSerializedCharField): key_name =
"choice_text" value_fmt = lambda x: x.lowercase() as_tag = True #
default value
belongs_to = "field"

class SerializedChoice(DefaultSerializedModel): exclude = ["votes"]
fields = {
'pk': DefaultSe

Re: Customizable Serialization Proposal

[DaNmarner]

After a few days' consideration, I've got some more crystalized
details about my proposal.

Here are the target result if this proposal is implementated:

1. User could configure serialization for each model.

2. In fact, user could configure the serialization details down to
each field of each model. That means:

3. The key name of each field would be freely defined, the value would
be freely and dynamically defined.

4. Nested structure could be represented as either fk or get
serialized as substructures in the result.

5. The depth of serializaton can be defined, again, down to each
field.

6. Various bonuses. e.g. "sorters" for serialized result, values
that's not from the query set, etc.


First off, the current assumption on the structure of a Model, as far
as serialization is concerned, should be changed. I propose that we
look at the model as a flat key-value structure, that is, pk and model
name (and potentially other metadata) are no different to a user-
defined field.

That is the base of everything below. So from now on, I'll stop
distinguishing user-defined fields, pk and metadata in my discussion,
and call them "fields" instead.

I want to add APIs targeting two level of abstraction:

"class SerializedModel": used to customize how each model is
serialized.

"class SerializedField": used to customize how each field is
serialized.

They are to be subclassed by user, serving as configuration for
serialization. SerializedModel will be passed as options to the actual
serializaton module and be referenced to through out the process.

How to achieved those goals in my claim? Let's see what's in those
classes.

The meat of things lies in the SerializedField class, I want to
introduce the proposed attributes of this class one by one, along each
I'll try explain my reasoning behind it and it's relevance to my
goals.

Some "regular" parts of a field:

"key_name": the name to be used for this field, defaults to the field
name(pk, model, album). This allows user to represent the field in
whatever name they want.

"with_type": a boolean, this let user deside whether the type of this
field should be part of the result.

"value_format": default to string repr of the field value, but if the
user want, she can assign a function to value_format to twist the
value however she like and return. More on this attr below.

"belongs_to": where in the result does the field go, default to top
level. If we were to mimic the current serialization format, this
value for all the user-defined fields would be "field".

"as_tag": a boolean. This is here specifically for XMLSerializer. If
set false, it causes the field to be rendered as attrs. There's more
on this part below.

If the field references to other models, these addtional attrs for
SerializedField would be used:

"rel_name": similar to "key_name", but it is for the relation type, if
renderd. defaults to "rel", as it would be if we were to mimic current
XML format.

"rel_to_name": similar to "key_name" and "rel_name", the key in the
result whose value is the model this field points to.

"with_rel": boolean, determines whether the two fields above shoud be
part of the result.

"embed_level": an integer. This is my answer to the question "how to
handle nested objects?". And it deserves its own paragraph.

For a reference-typed field, the value of SerializedField.value_format
should be an instance of SerializedField. If "embed_level" is 0, then
only the foreign key will be rendered. If it is greater than 0, then
the serializer should go ahead and serialize whatever this field
points to according to the info contained in "value_format".
"embed_level" would be initialized each time an instance of
SerializedField is instantiated, which happens when the serializer
discovers embed_level > 0. Another related note is, when "embed_level"
> 0, "as_tag" would be forced to be True.


Compare to SerializedField, SerializedModel is much simpler. It's
function is no more than holding info about which field should be
included in the serialization, and mapping the wanted field to its
SerializedField instance. Inspired by contrib.admin.ModelAdmin,
SerializedModel has two attributes:

"exclude": a list of unwelcomed fields referenced by their str names.

"fields": a dict whose keys are str names of wanted fields, and the
values are SerializedField instances.

If nessacery, a "sorter" fields which accept a sorter function for the
result serialization might be included as well.

The new serialization framework would include default
SerializationField for all the "native" field types, so that users
don't have to create one for each fields.

Here's an illustration of how a simple use case looks like, taking the
models from the tutorial as an example:

class SerializedChoiceText(DefaultSerializedCharField):
key_name = "choice_text"
value_fmt = lambda x: x.lowercase()
as_tag = True # default value
belongs_to = "field"

class SerializedChoice(DefaultSerializedModel):
exclude