Re: High Level Discussion about the Future of Django

[Graham Dumpleton]

On Apr 5, 4:37 pm, Russell Keith-Magee 
wrote:
> However, we can't seriously start talking about Python 3 until:
>
>  * all the downstream vendors (DB-API implementations, mod_wsgi, etc)
> have viable Python 3 implementations, and

Hmmm, mod_wsgi has had working Python 3.0 support for over a year. I
even did a quick test of Martin's original go at porting Django to
Python 3.0 and at least got the home page working.

I accept though that that is but one required bit and there is a lot
more other important stuff besides it.

:-)

Graham

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: NoSQL Support for the ORM

[Waldemar Kornewald]

Hey Alex,

On Apr 7, 2:11 am, Alex Gaynor  wrote:
> Non-relational database support for the Django ORM
> ==
>
> Note:  I am withdrawing my proposal on template compilation.  Another
> student
> has expressed some interest in working on it, and in any event I am
> now more
> interested in working on this project.

It's great that you want to work on this project. Since I want to see
this feature in Django, I'm offering mentoring help with the NoSQL
part. You know Django's ORM better than me, so I probably can't really
help you there, but I can help to make sure that your modifications
will work well on NoSQL DBs. Just in case this is necessary, I'll
apply as a GSoC mentor before it's too late (if I remember correctly,
in 2007 we could still allow new mentors even at this late stage)?

> Method
> ~~
>
> The ORM architecture currently has a ``QuerySet`` which is backend
> agnostic, a
> ``Query`` which is SQL specific, and a ``SQLCompiler`` which is
> backend
> specific (i.e. Oracle vs. MySQL vs. generic).  The plan is to change
> ``Query``
> to be backend agnostic by delaying the creation of structures that are
> SQL
> specific, specifically join/alias data.  Instead of structures like
> ``self.where``, ``self.join_aliases``, or ``self.select`` all working
> in terms
> of joins and table aliases the composition of a query would be stored
> in terms
> of a tree containing the "raw" filters, as passed to the filter calls,
> with
> things like ``Field.get_prep_value`` called appropriately.  The
> ``SQLCompiler``
> will be responsible for computing the joins for all of these data-
> structures.

Could you please elaborate on the data structures? In the end, non-
relational backends shouldn't have to reproduce large parts of the
SQLQuery code just to emulate a JOIN. When we tried to do a similar
refactoring we quickly faced the problem that we needed something
similar to setup_joins() and other SQLQuery features. We'd also have
to create code for grouping filters into individual queries on tables.
The Query class should take care of as much of the common stuff as
possible, so nonrel backends can potentially emulate every single SQL
feature (e.g., via MapReduce or whatever) with the least effort.
Otherwise this refactoring would actually have more disadvantages than
our current SQLCompiler-based approach in Django-nonrel (as ridiculous
as that sounds).

However, it's important that all of the emulated features are handled
not by the backend, but by a reusable code layer which sits on top of
the nonrel backends. It would be wasteful to let every backend
developer write his own JOIN emulation and denormalization and
aggregate code, etc.. The refactored ORM should at least still allow
for writing some kind of "proxy" backend that sits on top of the
actual nonrel backend and takes care of SQL features emulation. I'm
not sure if it's a good idea to integrate the emulation into Django
itself because then progress will be slowed down.

Ideally, we should provide a simplified API for nonrel backends,
similar to the one that we recently published for Django-nonrel, so a
backend could be written in two days instead of two weeks. We can port
our work over to the refactored ORM, so this you don't have to deal
with this (except if it should be officially integrated into Django).

In addition to these changes you'll also need to take care of a few
other things:

Many NoSQL DBs provide a simple "upsert"-like behavior where on save()
they either create a new entity if none exists with that primary key
or update the existing entity if one exists. However, on save() Django
first checks if an entity exists. This would be inefficient and
unnecessary, so the backend should be able to turn that behavior off.

On delete() Django also deletes related objects. This can be a costly
operation, especially if you have a large number of entities. Also,
the queries that collect the related entities can conflict with
transaction support at least on App Engine and it might also be very
inefficient on HBase. IOW, it's not sufficient to let the user handle
the deletion for large datasets. So, non-relational (and maybe also
relatinoal) DBs should be able to defer and split up the deletion
process into background tasks - which would also simplify the
developer's job because he doesn't have to take care of manually
writing background tasks for large datasets, so it's a good addition
in general.

I'm not sure how to handle multi-table inheritance. It could be done
with JOIN emulation, but this would be very inefficient.
Denormalization is IMHO not the answer to this problem, either. Should
Django simply fail to execute such a query on those backends or should
the user make sure that he doesn't use multi-table inheritance
unnecessarily in his code?

Bye,
Waldemar Kornewald

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, se

Re: [GSOC] NoSQL Support for the ORM

[Russell Keith-Magee]

On Wed, Apr 7, 2010 at 8:11 AM, Alex Gaynor  wrote:
> Non-relational database support for the Django ORM
> ==
>
> Note:  I am withdrawing my proposal on template compilation.  Another student
> has expressed some interest in working on it, and in any event I am now more
> interested in working on this project.
>
> About Me
> 
>
> I'm a sophomore computer science student at Rensselaer Polytechnic Institute.
> I'm a frequent contributor to Django (including last year's successful 
> multiple
> database GSoC project) and other related projects; I'm also a committer on 
> both
> `Unladen Swallow `_ and
> `PyPy `_.
>
> Background
> ~~
>
> As the person responsible for large swaths of multiple database support I am
> intimately familiar with the architecture of the ORM, the code itself, and the
> various concerns that need to be accounted for (pickleability, etc.).
>
> Rationale
> ~
>
> Non-relational databases tend to support some subset of the operations that 
> are
> supported on relational databases, therefore it should be possible to perform
> these operations on all databases.  Some people are of the opinion that we
> shouldn't bother to support these databases, because they can't perform all
> operations, I'm of the opinion that the abstraction is already a little leaky,
> we may as well exploit this for a common API where possible, as well as giving
> users of these databases the admin and models forms for free.
>
> Method
> ~~
>
> The ORM architecture currently has a ``QuerySet`` which is backend agnostic, a
> ``Query`` which is SQL specific, and a ``SQLCompiler`` which is backend
> specific (i.e. Oracle vs. MySQL vs. generic).  The plan is to change ``Query``
> to be backend agnostic by delaying the creation of structures that are SQL
> specific, specifically join/alias data.  Instead of structures like
> ``self.where``, ``self.join_aliases``, or ``self.select`` all working in terms
> of joins and table aliases the composition of a query would be stored in terms
> of a tree containing the "raw" filters, as passed to the filter calls, with
> things like ``Field.get_prep_value`` called appropriately.  The 
> ``SQLCompiler``
> will be responsible for computing the joins for all of these data-structures.

I can see the intention here, and I can see how this approach could be
used to solve the problem. However, my initial concern is that normal
SQL users will end up carrying around a lot of extra overhead so that
they can support backends that they will never use.

Have you given any thought to how complex the datastructures inside
Query will need to be, and how complex and/or expensive the conversion
process will be?

Other issues that spring to mind:

 * What about nonSQL datatypes? List/Set types are a common feature of
Non-SQL backends, and are The Right Way to solve a whole bunch of
problems. How do you propose to approach these datatypes? What (if
any) overlap exists between the use of set data types and m2m? Is
there any potential overlap between supporting List/Set types and
supporting Arrays in SQL?

 * How does a non-SQL backend integrate with syncdb and other setup
tools? What about inspectdb?

 * What about basic connection management? Is the existing Connection
API likely to be compatible, or will modifications be required?

 * Why the choice of MongoDB specifically? Do you have particular
experience with MongoDB? Does MongoDB have features that make it a
good choice?

 * Given that you're only proposing a single proof-of-concept backend,
have you given any thought to the needs of other backends? It's not
hard to envisage that Couch, Cassandra, GAE etc will all have slightly
different requirements and problems. Is there a common ground that
exists between all data store backends? If there isn't, how do you
know that what you are proposing will be sufficient to support them?

There's also the issue of specificity in your proposal; I'll take you
at your word that what you have proposed is a draft that requires
elaboration.

Yours,
Russ Magee %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Talk on django to an enterprise audience

[Russell Keith-Magee]

On Tue, Apr 6, 2010 at 2:46 AM, Lakshman Prasad  wrote:
> Hi,
> I will be speaking about django "Building Reusable Applications using
> django" at the Great Indian Developer summit (link) in a couple of weeks.
> GIDS is a large conference, with lot of enterprise audience. To most of
> them, this will be an introduction to django. I have spoken about django to
> local user groups and at some conferences before (1 2 3).
> So, wrt this talk at the conference,
> * I have seen several threads that touch about on
> "django-for-the-enterprise", I solicit the community feedback about the
> specific content that needs to be highlighted. If core developers and/or
> BDFLs have something to say, I will be glad.

I'm not sure exactly what it is that you would like us to say.

The threads you mention are mostly about identifying the potential
sticking points for enterprise customers. To my reading, the outcome
of those threads has been that for the most part, these problems
aren't show stoppers, and in most cases, there are solutions or
workaround - it's mostly just a matter of following appropriate
conventions for an 'enterprise' project setup.

That said, we're open to any suggestions; if someone with "enterprise"
requirements wants to make a suggestion on something that needs to be
improved, we're only too happy to entertain those suggestions.

Yours,
Russ Magee %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Talk on django to an enterprise audience

[Lakshman Prasad]

> these problems aren't show stoppers, and in most cases, there are solutions
> or workaround


Thanks. This is exactly a very good summary, that I can share (with how to,
where possible).

if someone with "enterprise" requirements wants to make a suggestion on
> something that needs to be improved, we're only too happy to entertain those
> suggestions.


This coming from a core developer, than a user (me), I can now say it much
more confidently. Thanks again!

On Wed, Apr 7, 2010 at 4:41 PM, Russell Keith-Magee
wrote:

> On Tue, Apr 6, 2010 at 2:46 AM, Lakshman Prasad 
> wrote:
> > Hi,
> > I will be speaking about django "Building Reusable Applications using
> > django" at the Great Indian Developer summit (link) in a couple of weeks.
> > GIDS is a large conference, with lot of enterprise audience. To most of
> > them, this will be an introduction to django. I have spoken about django
> to
> > local user groups and at some conferences before (1 2 3).
> > So, wrt this talk at the conference,
> > * I have seen several threads that touch about on
> > "django-for-the-enterprise", I solicit the community feedback about the
> > specific content that needs to be highlighted. If core developers and/or
> > BDFLs have something to say, I will be glad.
>
> I'm not sure exactly what it is that you would like us to say.
>
> The threads you mention are mostly about identifying the potential
> sticking points for enterprise customers. To my reading, the outcome
> of those threads has been that for the most part, these problems
> aren't show stoppers, and in most cases, there are solutions or
> workaround - it's mostly just a matter of following appropriate
> conventions for an 'enterprise' project setup.
>
> That said, we're open to any suggestions; if someone with "enterprise"
> requirements wants to make a suggestion on something that needs to be
> improved, we're only too happy to entertain those suggestions.
>
> Yours,
> Russ Magee %-)
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To post to this group, send email to django-develop...@googlegroups.com.
> To unsubscribe from this group, send email to
> django-developers+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/django-developers?hl=en.
>
>


-- 
Lakshman
http://uswaretech.com/

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: GSOC proposal for "App loading"

[Russell Keith-Magee]

On Mon, Apr 5, 2010 at 4:11 PM, Dagvadorj Galbadrakh
 wrote:
> Hello group,
>
>
>
> I want to attend to this year's Google Summer of Code program with
> "App loading". The following is a part from my proposal. It offers
> more simplistic approach than heavily discussed one with App() in
> INSTALLED_APPS, i.e., multiple instances of an app is declared as
> ('foo.polls', 'mayor'). I'd appreciate any comments and feedbacks.

I don't want to appear harsh, but there really isn't much to comment
on here. You've given an example of how the code would look to end
users, and some very shallow details of how that example would be
interpreted, but almost no information on how this change would affect
the rest of Django, or the complications that you anticipate.

For example, the Wiki description of this problem lists 2 specific
problems that you need to address. It's not clear to me that you've
addressed either of these.

It also fails to address the reasons why App loading is interesting in
the first place - issues like translation of application names, and
specification of application settings.

The only detail that I can see that I can comment on is the process of
name munging that you use to avoid collisions - and I can't say I like
that very much at all.

On a superficial level, I'm really not a fan of having a trailing
underscore everywhere.

On a more substantial level, you haven't provided any indication why
this approach is necessary instead of simply allowing the end-user to
specify the name that they want the application to be represented as
-- that is, if I want to use the name gsoc_2009, then why not specify
('gsoc', 'gsoc_2009')? This leaves the flexibility in the hands of the
end user, rather than embedded in a convention that I can guarantee
*someone* will want to break out of.

Plus, at the end of the day, it doesn't prevent name collisions, anyway:
INSTALLED_APPS = (
   'foo.gsoc_2009_',
   ('foo.gsoc', '2009'),
)

Yours,
Russ Magee %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: GSoC: App Loading

[Russell Keith-Magee]

On Mon, Apr 5, 2010 at 5:35 AM, Arthur Koziel  wrote:
> Hi,
> I’m going to apply for GSoC with the goal of refactoring the app loading
> [0]. I’ve been looking at Django’s current app loading implementation
> (db.models.loading), Vinay Sajip’s patch in #3591 [1] and the notes on the
> InstalledAppsRevision wiki page [2]. Here's what I'm planning to change. Any
> feedback is appreciated.
> Step 1
> -
> Deal with the two main problems of the current app loading implementation,
> and make it possible to:
> - deploy several instances of the same app
> - deploy two apps with the same name
> The INSTALLED_APPS setting will be changed to:
>
> INSTALLED_APPS = InstalledApps(
>     'django.contrib.auth', # strings will be converted to App()

Strings will be converted to App()... by What? When?

What happens to existing code that is using "for app in INSTALLED_APS"
as an idiom?

>     App('django.contrib.contenttypes'),
>     App('django.contrib.sessions'),
>     App('django.contrib.sites'),
>     App('django.contrib.messages'),
>     App('django.contrib.admin'),
>     App('django.contrib.admin', name=‘demo’),
> )
>
> The InstalledApps and App classes will be added to core.apps. The
> db.models.loading module will be deprecated and redirect to core.apps.

Is there a specific reason for this move? Is it just shuffling
deckchairs, or is there a purpose behind the move?

> The structure of the InstalledApps class will be similiar to the
> db.models.loading.AppCache class. Instead of storing the model
> modules/instances for each app (app_store/app_models), this class will store
> the App instances. An App instance will store the model modules/instances.
> The methods of the old AppCache class (get_apps(), get_app(), ...) will be
> kept for backwards compatibility but modified to reflect the change
> mentioned above.
> Step 2
> -
> As mentioned on the InstalledAppsRevision wiki page [2],
> model._meta.app_label is overloaded and used for:
>
> - Admin app display name (by .title())
> - Admin permissions prefix
> - DB table creation prefix
> - Dumpdata/Loaddata fixture prefix identifier
> - When explicitly set, used to associate models elsewhere in the app
> structure.
> - RelatedObject explicit cross-app identification.
> - contrib.contenttypes identification
>
> The goals of Step 2 is to add options to the App class and modify other
> parts of Django to not rely so heavily on app_label. For example, a
> `db_prefix` option could be added to the App class and the ORM code could be
> modified to use db_prefix instead of app_label:
>
> INSTALLED_APPS = InstalledApps(
>     App('django.contrib.admin', db_prefix='chewie'),
> )
>
> In order for this to work, each model instance needs to have an app instance
> associated (via _meta.app). The orm code can then use
> model._meta.app.db_prefix to create the table.

I agree with Alex - there's a lot more detail needed here. How will I
get access to the App instance that a model belongs to? How will
legacy attributes like db_prefix be proxied? What is the order of
precedence when a model and an app disagree on db_prefix?

For the record - I don't think the "multiple versions of a single app"
problem is one worth solving. I'm not even convinced it can be solved
at all, for all the reasons Florian identified, and then some.
Although it may have been part of the original use case for #3591,
much of that use case has since been replaced by namespaced URLs and
class-based applications. If you're going to tackle this problem, I'd
rather see you concentrate on issues like:

 * Translating application names
 * Solving the "two applications called auth" problem
 * Providing configurability for db-level naming convention at the
deployment level (e.g., so you can override db_prefix and table names
when you deploy an app)
 * Providing configurability for 'app level' settings; for example,
when deploying contrib.comments, setting the name of the model that
will be used for content.

> Step 3
> -
> The current app loading implementation in Django was designed to load model
> classes. However, there are many applications which just provide
> templatetags or management commands and don’t use Django’s ORM at all. These
> applications have to ship with an empty models.py file in order to be loaded
> correctly. The goal of Step 3 is to make it possible to load applications
> without a models.py file.

I would have thought this step would have been a logical consequence
of Step 1; if you're going to massively rewrite app loading, it would
seem unusual to embed all the old problems in your rewrite.

Yours,
Russ Magee %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl

Re: GSOC proposal for "App loading"

[Dagvadorj Galbadrakh]

Thanks for the comment. It will really help. I will try to address
wide variety of consequences of any change in the code in my proposal.

As for the last comment, it really brings up conflict as I was also
thinking yesterday. Firstly, my solution would be changing
importlib.import_module to have 3 parameters and naming a new instance
of foo.bar [in other words, (foo.bar, mayor)] application as
foo.bar.mayor. For example:

def import_module(name, package=None, clone=None)
   ...
   if not clone None:
  new_module = __import__(name.split('.')[-1], globals(),
locals(), ['models'], -1)
  sys.modules[name] = new_module
  return sys.modules[name]
   ...

Then, I thought that changing import_module is not necessary and
thought of importing the original app to the new instance in place,
like:

...
for app_name in settings.INSTALLED_APPS:
 if (type(app_name) == types.TupleType):
a = app_name[0] + "." + app_name[1]
sys.modules[a] = __import__(app_namep[0])
...
not using sys.modules exactly, of course.

It would be the most straight-forward way without interrupting the
existing use of the current app loading mechanism.


On Wed, Apr 7, 2010 at 2:25 PM, Russell Keith-Magee
 wrote:
> On Mon, Apr 5, 2010 at 4:11 PM, Dagvadorj Galbadrakh
>  wrote:
>> Hello group,
>>
>>
>>
>> I want to attend to this year's Google Summer of Code program with
>> "App loading". The following is a part from my proposal. It offers
>> more simplistic approach than heavily discussed one with App() in
>> INSTALLED_APPS, i.e., multiple instances of an app is declared as
>> ('foo.polls', 'mayor'). I'd appreciate any comments and feedbacks.
>
> I don't want to appear harsh, but there really isn't much to comment
> on here. You've given an example of how the code would look to end
> users, and some very shallow details of how that example would be
> interpreted, but almost no information on how this change would affect
> the rest of Django, or the complications that you anticipate.
>
> For example, the Wiki description of this problem lists 2 specific
> problems that you need to address. It's not clear to me that you've
> addressed either of these.
>
> It also fails to address the reasons why App loading is interesting in
> the first place - issues like translation of application names, and
> specification of application settings.
>
> The only detail that I can see that I can comment on is the process of
> name munging that you use to avoid collisions - and I can't say I like
> that very much at all.
>
> On a superficial level, I'm really not a fan of having a trailing
> underscore everywhere.
>
> On a more substantial level, you haven't provided any indication why
> this approach is necessary instead of simply allowing the end-user to
> specify the name that they want the application to be represented as
> -- that is, if I want to use the name gsoc_2009, then why not specify
> ('gsoc', 'gsoc_2009')? This leaves the flexibility in the hands of the
> end user, rather than embedded in a convention that I can guarantee
> *someone* will want to break out of.
>
> Plus, at the end of the day, it doesn't prevent name collisions, anyway:
> INSTALLED_APPS = (
>   'foo.gsoc_2009_',
>   ('foo.gsoc', '2009'),
> )
>
> Yours,
> Russ Magee %-)
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Django developers" group.
> To post to this group, send email to django-develop...@googlegroups.com.
> To unsubscribe from this group, send email to 
> django-developers+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/django-developers?hl=en.
>
>



-- 
Dagvadorj GALBADRAKH

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: GSoC: App Loading

[Vinay Sajip]

On Apr 7, 12:40 pm, Russell Keith-Magee 
wrote:
>
> Strings will be converted to App()... by What? When?
>

This is already done by the #3591 patch, in
django.db.models.loading.py.

> What happens to existing code that is using "for app in INSTALLED_APS"
> as an idiom?
>

The #3591 patch does this by replacing settings.INSTALLED_APPS with
get_installed_app_paths, which obviously wouldn't work for 3rd party
code out in the wild. ISTM that to keep things backward-compatible,
you need two lists - INSTALLED_APPS and an internal list of App()
instances. You could for instance allow users to specify a mixture of
strings and App instances in settings.INSTALLED_APPS, and once the
loading has happened, this list would contain only strings, whereas an
internal list would contain only App instances and be accessed via a
new function such as get_installed_apps().

> >     App('django.contrib.contenttypes'),
> >     App('django.contrib.sessions'),
> >     App('django.contrib.sites'),
> >     App('django.contrib.messages'),
> >     App('django.contrib.admin'),
> >     App('django.contrib.admin', name=‘demo’),
> > )
>
> > The InstalledApps and App classes will be added to core.apps. The
> > db.models.loading module will be deprecated and redirect to core.apps.
>
> Is there a specific reason for this move? Is it just shuffling
> deckchairs, or is there a purpose behind the move?
>

If apps are to have first-class-citizen status, then this I suppose
would serve to emphasize that status, but other than that I'm not sure
it's necessary.

>
> I agree with Alex - there's a lot more detail needed here. How will I
> get access to the App instance that a model belongs to? How will
> legacy attributes like db_prefix be proxied? What is the order of
> precedence when a model and an app disagree on db_prefix?
>

Agreed that a more detailed specification is required, but wouldn't at
least some of that detailing be part of the actual GSoC work?

> For the record - I don't think the "multiple versions of a single app"
> problem is one worth solving. I'm not even convinced it can be solved
> at all, for all the reasons Florian identified, and then some.

I agree, but IIRC it was quoted as one reason why this feature was
left out of the running in the recent Django revisions. It doesn't
make a lot of sense to me for any app where models are concerned, and
I believe there would be no point in duplicating the models for
multiple instances of an app.

> Although it may have been part of the original use case for #3591,
> much of that use case has since been replaced by namespaced URLs and
> class-based applications. If you're going to tackle this problem, I'd
> rather see you concentrate on issues like:

Class-based applications? Can someone point me to some of those?
Googling for "django class-based applications" doesn't throw up
anything.

>  * Translating application names

This is handled in the #3591 patch AFAIK.

>  * Solving the "two applications called auth" problem
>  * Providing configurability for db-level naming convention at the
> deployment level (e.g., so you can override db_prefix and table names
> when you deploy an app)
>  * Providing configurability for 'app level' settings; for example,
> when deploying contrib.comments, setting the name of the model that
> will be used for content.
>


> > Step 3
> > -
> > The current app loading implementation in Django was designed to load model
> > classes. However, there are many applications which just provide
> > templatetags or management commands and don’t use Django’s ORM at all. These
> > applications have to ship with an empty models.py file in order to be loaded
> > correctly. The goal of Step 3 is to make it possible to load applications
> > without a models.py file.
>
> I would have thought this step would have been a logical consequence
> of Step 1; if you're going to massively rewrite app loading, it would
> seem unusual to embed all the old problems in your rewrite.
>

I think that would all come out in the wash; for example the #3591
patch doesn't require a models.py, and the app instance has slots for
two modules: the app package itself and the models sub-package or
module within it. It should be very easy to handle the case of an app
with no models.

Regards,

Vinay Sajip

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: GSOC proposal for "App loading"

[Vinay Sajip]

On Apr 7, 12:41 pm, Dagvadorj Galbadrakh  wrote:
>
> Then, I thought that changing import_module is not necessary and
> thought of importing the original app to the new instance in place,
> like:

The very fact that you considered replacing import_module would make
me a bit nervous :-/

>
> ...
> for app_name in settings.INSTALLED_APPS:
>      if (type(app_name) == types.TupleType):
>         a = app_name[0] + "." + app_name[1]
>         sys.modules[a] = __import__(app_namep[0])
> ...
> not using sys.modules exactly, of course.
>

Perhaps you should review prior work in this area first, so as not to
repeat work already done? As Alex Gaynor suggested, you could benefit
from looking at this other thread:

http://groups.google.com/group/django-developers/browse_frm/thread/4cca2086dd485879

which shows evidence of a greater awareness of at least some of the
issues, without leaping straight to implementation details: as a
result, it comes across as more considered.

Regards,

Vinay Sajip

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: GSoC: App Loading

[Russell Keith-Magee]

On Wed, Apr 7, 2010 at 8:23 PM, Vinay Sajip  wrote:
>
> On Apr 7, 12:40 pm, Russell Keith-Magee 
> wrote:
>>
>> I agree with Alex - there's a lot more detail needed here. How will I
>> get access to the App instance that a model belongs to? How will
>> legacy attributes like db_prefix be proxied? What is the order of
>> precedence when a model and an app disagree on db_prefix?
>>
>
> Agreed that a more detailed specification is required, but wouldn't at
> least some of that detailing be part of the actual GSoC work?

It's fair to assume that there will always be *some* details that are
worked out as part of the GSoC work. My intention isn't to try and
nail down all the details before the project starts - it's to try and
work out how much Arthur has thought about the problem, and how well
he can communicate what he is thinking.

The GSoC application process is a lot like a job interview. Although
Django itself isn't paying the stipend to students, we are going to
commit a lot of resources to mentoring a student. As a result, we
aren't going to take on students just to make up the numbers. Asking
questions like this one allows me to kill 3 birds with 1 stone - I get
to improve the design, *and* I get to see the design skills of the
prospective student in action, *and* I get to see how well the
prospective student is able to communicate those designs to the wider
community.

>> For the record - I don't think the "multiple versions of a single app"
>> problem is one worth solving. I'm not even convinced it can be solved
>> at all, for all the reasons Florian identified, and then some.
>
> I agree, but IIRC it was quoted as one reason why this feature was
> left out of the running in the recent Django revisions. It doesn't
> make a lot of sense to me for any app where models are concerned, and
> I believe there would be no point in duplicating the models for
> multiple instances of an app.

Not to my reading. The reason for rejection at official voting has
always been "incomplete design". I can see that the subject of
multi-deployed apps has come up in the past, but it's also been
identified as a *hard* problem, and possibly one that doesn't need
solving. Unless someone can provide a particularly compelling use
case, I'm inclined to put it on the ignore pile.

>> Although it may have been part of the original use case for #3591,
>> much of that use case has since been replaced by namespaced URLs and
>> class-based applications. If you're going to tackle this problem, I'd
>> rather see you concentrate on issues like:
>
> Class-based applications? Can someone point me to some of those?

django.contrib.admin is the most notable example.

Yours,
Russ Magee %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: GSoC: App Loading

[Jannis Leidel]

>>> For the record - I don't think the "multiple versions of a single app"
>>> problem is one worth solving. I'm not even convinced it can be solved
>>> at all, for all the reasons Florian identified, and then some.
>> 
>> I agree, but IIRC it was quoted as one reason why this feature was
>> left out of the running in the recent Django revisions. It doesn't
>> make a lot of sense to me for any app where models are concerned, and
>> I believe there would be no point in duplicating the models for
>> multiple instances of an app.
> 
> Not to my reading. The reason for rejection at official voting has
> always been "incomplete design". I can see that the subject of
> multi-deployed apps has come up in the past, but it's also been
> identified as a *hard* problem, and possibly one that doesn't need
> solving. Unless someone can provide a particularly compelling use
> case, I'm inclined to put it on the ignore pile.

Having tried to solve this problem myself a while ago, I agree -- multiple app 
instances are out of scope of the summer of code.

Jannis

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: GSoC: App Loading

[Vinay Sajip]

On Apr 7, 2:18 pm, Russell Keith-Magee  wrote:
>
> It's fair to assume that there will always be *some* details that are
> worked out as part of the GSoC work. My intention isn't to try and
> nail down all the details before the project starts - it's to try and
> work out how much Arthur has thought about the problem, and how well
> he can communicate what he is thinking.
>
> The GSoC application process is a lot like a job interview. Although
> Django itself isn't paying the stipend to students, we are going to
> commit a lot of resources to mentoring a student. As a result, we
> aren't going to take on students just to make up the numbers. Asking
> questions like this one allows me to kill 3 birds with 1 stone - I get
> to improve the design, *and* I get to see the design skills of the
> prospective student in action, *and* I get to see how well the
> prospective student is able to communicate those designs to the wider
> community.
>

Seems eminently sensible to me :-)

> >> For the record - I don't think the "multiple versions of a single app"
> >> problem is one worth solving. I'm not even convinced it can be solved
> >> at all, for all the reasons Florian identified, and then some.
>
> > I agree, but IIRC it was quoted as one reason why this feature was
> > left out of the running in the recent Django revisions. It doesn't
> > make a lot of sense to me for any app where models are concerned, and
> > I believe there would be no point in duplicating the models for
> > multiple instances of an app.
>
> Not to my reading. The reason for rejection at official voting has
> always been "incomplete design". I can see that the subject of
> multi-deployed apps has come up in the past, but it's also been
> identified as a *hard* problem, and possibly one that doesn't need
> solving. Unless someone can provide a particularly compelling use
> case, I'm inclined to put it on the ignore pile.
>

Agree - and the incompleteness of the design was IIRC (and perhaps I
don't) partly because it didn't address the multiple app instances
issue. That's one of the issues mentioned in

http://code.djangoproject.com/wiki/InstalledAppsRevision

and that page was referred to as containing what needed to be
addressed in a viable design.

>
> > Class-based applications? Can someone point me to some of those?
>
> django.contrib.admin is the most notable example.
>

Oh right, I thought you were talking about something else :-)

Regards,

Vinay Sajip

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: GSoC: App Loading

[Dagvadorj Galbadrakh]

On Wed, Apr 7, 2010 at 2:40 PM, Russell Keith-Magee
 wrote:
> If you're going to tackle this problem, I'd
> rather see you concentrate on issues like:
>
>  * Translating application names
>  * Solving the "two applications called auth" problem
>  * Providing configurability for db-level naming convention at the
> deployment level (e.g., so you can override db_prefix and table names
> when you deploy an app)
>  * Providing configurability for 'app level' settings; for example,
> when deploying contrib.comments, setting the name of the model that
> will be used for content.
>
> Yours,
> Russ Magee %-)
>

So shortly do you imply that the "app loading" problem is indeed the
problem of integrating several application via db tables through
configuration in deployment?


-- 
Dagvadorj GALBADRAKH

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: GSoC: App Loading

[Arthur Koziel]

On Apr 7, 2010, at 1:40 PM, Russell Keith-Magee wrote:

> On Mon, Apr 5, 2010 at 5:35 AM, Arthur Koziel  wrote:
>> Hi,
>> I’m going to apply for GSoC with the goal of refactoring the app loading
>> [0]. I’ve been looking at Django’s current app loading implementation
>> (db.models.loading), Vinay Sajip’s patch in #3591 [1] and the notes on the
>> InstalledAppsRevision wiki page [2]. Here's what I'm planning to change. Any
>> feedback is appreciated.
>> Step 1
>> -
>> Deal with the two main problems of the current app loading implementation,
>> and make it possible to:
>> - deploy several instances of the same app
>> - deploy two apps with the same name
>> The INSTALLED_APPS setting will be changed to:
>> 
>> INSTALLED_APPS = InstalledApps(
>> 'django.contrib.auth', # strings will be converted to App()
> 
> Strings will be converted to App()... by What? When?
> 

The InstalledApps() class will do the converting during the initialization. As 
Vinay mentioned, his patch already does this in line 157.

> What happens to existing code that is using "for app in INSTALLED_APS"
> as an idiom?

That's the main reason why INSTALLED_APPS will be an InstalledApps instance. 
This way, an __iter__ function could be added that goes through an internal 
dict of Apps and yields their module path as a string.

> 
>> App('django.contrib.contenttypes'),
>> App('django.contrib.sessions'),
>> App('django.contrib.sites'),
>> App('django.contrib.messages'),
>> App('django.contrib.admin'),
>> App('django.contrib.admin', name=‘demo’),
>> )
>> 
>> The InstalledApps and App classes will be added to core.apps. The
>> db.models.loading module will be deprecated and redirect to core.apps.
> 
> Is there a specific reason for this move? Is it just shuffling
> deckchairs, or is there a purpose behind the move?

Yes, I wanted to make it optional for Apps to have models associated. And if an 
App doesn't require models, what's the point of it to live in db.models?

> 
>> The structure of the InstalledApps class will be similiar to the
>> db.models.loading.AppCache class. Instead of storing the model
>> modules/instances for each app (app_store/app_models), this class will store
>> the App instances. An App instance will store the model modules/instances.
>> The methods of the old AppCache class (get_apps(), get_app(), ...) will be
>> kept for backwards compatibility but modified to reflect the change
>> mentioned above.
>> Step 2
>> -
>> As mentioned on the InstalledAppsRevision wiki page [2],
>> model._meta.app_label is overloaded and used for:
>> 
>> - Admin app display name (by .title())
>> - Admin permissions prefix
>> - DB table creation prefix
>> - Dumpdata/Loaddata fixture prefix identifier
>> - When explicitly set, used to associate models elsewhere in the app
>> structure.
>> - RelatedObject explicit cross-app identification.
>> - contrib.contenttypes identification
>> 
>> The goals of Step 2 is to add options to the App class and modify other
>> parts of Django to not rely so heavily on app_label. For example, a
>> `db_prefix` option could be added to the App class and the ORM code could be
>> modified to use db_prefix instead of app_label:
>> 
>> INSTALLED_APPS = InstalledApps(
>> App('django.contrib.admin', db_prefix='chewie'),
>> )
>> 
>> In order for this to work, each model instance needs to have an app instance
>> associated (via _meta.app). The orm code can then use
>> model._meta.app.db_prefix to create the table.
> 
> I agree with Alex - there's a lot more detail needed here. How will I
> get access to the App instance that a model belongs to? How will
> legacy attributes like db_prefix be proxied? What is the order of
> precedence when a model and an app disagree on db_prefix?
> 
> For the record - I don't think the "multiple versions of a single app"
> problem is one worth solving. I'm not even convinced it can be solved
> at all, for all the reasons Florian identified, and then some.
> Although it may have been part of the original use case for #3591,
> much of that use case has since been replaced by namespaced URLs and
> class-based applications. If you're going to tackle this problem, I'd
> rather see you concentrate on issues like:
> 
> * Translating application names
> * Solving the "two applications called auth" problem
> * Providing configurability for db-level naming convention at the
> deployment level (e.g., so you can override db_prefix and table names
> when you deploy an app)
> * Providing configurability for 'app level' settings; for example,
> when deploying contrib.comments, setting the name of the model that
> will be used for content.

Okay, I'll focus on these issues rather than multiple app instances.

> 
>> Step 3
>> -
>> The current app loading implementation in Django was designed to load model
>> classes. However, there are many applications which just provide
>> templatetags or management commands and don’t use Django’s ORM at all. These
>

Re: GSoC: App Loading

[Jannis Leidel]

Am 07.04.2010 um 13:40 schrieb Russell Keith-Magee:

> On Mon, Apr 5, 2010 at 5:35 AM, Arthur Koziel  wrote:
>> Hi,
>> I’m going to apply for GSoC with the goal of refactoring the app loading
>> [0]. I’ve been looking at Django’s current app loading implementation
>> (db.models.loading), Vinay Sajip’s patch in #3591 [1] and the notes on the
>> InstalledAppsRevision wiki page [2]. Here's what I'm planning to change. Any
>> feedback is appreciated.
>> Step 1
>> -
>> Deal with the two main problems of the current app loading implementation,
>> and make it possible to:
>> - deploy several instances of the same app
>> - deploy two apps with the same name
>> The INSTALLED_APPS setting will be changed to:
>> 
>> INSTALLED_APPS = InstalledApps(
>> 'django.contrib.auth', # strings will be converted to App()
> 
> Strings will be converted to App()... by What? When?
> 
> What happens to existing code that is using "for app in INSTALLED_APS"
> as an idiom?

I'm worried that creating the app instances with an ``app()`` callable would 
require an import and contradicts with the convention of having dotted paths 
for any configurable extension. That's why I think INSTALLED_APPS should be 
kept as an iterable of dotted paths which is converted to an app registry 
during the start-up (which already happens for wildcard entries like 
"django.contrib.*"). Each entry of the list would be a string with one of the 
following values:

- Dotted path to an app module (current default, e.g. 'django.contrib.auth'). 
During the start-up an anonymous app instance would be initialized, using the 
models module of the loaded app module.

- Dotted path to an app class (e.g. 'django.contrib.auth.AuthApp'). During the 
start-up a named app instance would be created, either with the models module 
of the module the app class is located in, the model attribute of the class or 
none (model-less app).

- Dotted path to an app instance (e.g. 'django.contrib.admin.site'). During the 
start-up an existing app instance would be handled like it has been loaded in 
one of the former two cases.

Iterating over the INSTALLED_APPS setting would still be supported given the 
fact each app instance has it's own label that sets its models db_prefix, so 
the check for a custom app would be:

if "acme.auth" in INSTALLED_APPS:
# do something

An app class could look like this:

from django.contrib import auth
from django.conf.urls.defaults import patterns
from django.utils.translation import ugettext_lazy as _

class AcmeAuthApp(auth.AuthApp):
label = 'acme' # sets the db_prefix
models = auth.models
verbose_name = _('acme auth')

def get_urls(self):
return patterns('acme.auth.views',
url(r'^login/$', 'login', name="acme_auth_login"),
)

Jannis

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: GSoC: App Loading

[Arthur Koziel]

On Apr 7, 2010, at 1:40 PM, Russell Keith-Magee wrote:

> I agree with Alex - there's a lot more detail needed here. How will I
> get access to the App instance that a model belongs to? How will
> legacy attributes like db_prefix be proxied? What is the order of
> precedence when a model and an app disagree on db_prefix?
> 
> For the record - I don't think the "multiple versions of a single app"
> problem is one worth solving. I'm not even convinced it can be solved
> at all, for all the reasons Florian identified, and then some.
> Although it may have been part of the original use case for #3591,
> much of that use case has since been replaced by namespaced URLs and
> class-based applications. If you're going to tackle this problem, I'd
> rather see you concentrate on issues like:
> 
> * Translating application names

The App class could use "verbose_name" and "verbose_name_plural" options. Since 
a lot of projects already do something like "_ = lambda s: s" in their 
settings.py file, we could do the following:

INSTALLED_APPS = InstalledApps(
App('django.contrib.admin', verbose_name=_('Adminstration'), 
verbose_name_plural=_('Administrations'))

> * Solving the "two applications called auth" problem

The "two applications called auth" problem is mainly a problem with the keys of 
the app_models dict in AppCache. The keys are mod.__name__.split('.')[-2], 
which means that for "django.contrib.auth.models" the key will be "auth". Now, 
if there is another models module "foobar.auth.models" the same key will be 
used and the models from foobar.auth will be appended to the "auth" key. So, 
for example, if both applications would have one model each, Django would 
internally load this as one application with 2 models.

Anyway, this is easy to solve. As I mentioned in the proposal I plan to remove 
the app_models dict from AppCache and track App instances instead. Each App 
instance would have a model attribute. There would be no overlap like in the 
app_models dict.

> * Providing configurability for db-level naming convention at the
> deployment level (e.g., so you can override db_prefix and table names
> when you deploy an app)
> * Providing configurability for 'app level' settings; for example,
> when deploying contrib.comments, setting the name of the model that
> will be used for content.

The App class could just take keyword arguments. To actually use this options 
would be up to the application developer. For example, django-disqus currently 
expects the user to set the following variables in settings.py:

DISQUS_SHORTNAME = 'foo'
DISQUS_API_KEY = '123123'

This could be done with:

INSTALLED_APPS = InstalledApps(
App('django_disqus', shortname='foo', api_key='123123')
)

which I personally like better. We could then provide an get_app() function to 
get the App instance and access the options:

get_app('django_disqus').options.shortname

Arthur

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: NoSQL Support for the ORM

[Alex Gaynor]

On Wed, Apr 7, 2010 at 4:43 AM, Waldemar Kornewald  wrote:
> Hey Alex,
>
> On Apr 7, 2:11 am, Alex Gaynor  wrote:
>> Non-relational database support for the Django ORM
>> ==
>>
>> Note:  I am withdrawing my proposal on template compilation.  Another
>> student
>> has expressed some interest in working on it, and in any event I am
>> now more
>> interested in working on this project.
>
> It's great that you want to work on this project. Since I want to see
> this feature in Django, I'm offering mentoring help with the NoSQL
> part. You know Django's ORM better than me, so I probably can't really
> help you there, but I can help to make sure that your modifications
> will work well on NoSQL DBs. Just in case this is necessary, I'll
> apply as a GSoC mentor before it's too late (if I remember correctly,
> in 2007 we could still allow new mentors even at this late stage)?
>
>> Method
>> ~~
>>
>> The ORM architecture currently has a ``QuerySet`` which is backend
>> agnostic, a
>> ``Query`` which is SQL specific, and a ``SQLCompiler`` which is
>> backend
>> specific (i.e. Oracle vs. MySQL vs. generic).  The plan is to change
>> ``Query``
>> to be backend agnostic by delaying the creation of structures that are
>> SQL
>> specific, specifically join/alias data.  Instead of structures like
>> ``self.where``, ``self.join_aliases``, or ``self.select`` all working
>> in terms
>> of joins and table aliases the composition of a query would be stored
>> in terms
>> of a tree containing the "raw" filters, as passed to the filter calls,
>> with
>> things like ``Field.get_prep_value`` called appropriately.  The
>> ``SQLCompiler``
>> will be responsible for computing the joins for all of these data-
>> structures.
>
> Could you please elaborate on the data structures? In the end, non-
> relational backends shouldn't have to reproduce large parts of the
> SQLQuery code just to emulate a JOIN. When we tried to do a similar
> refactoring we quickly faced the problem that we needed something
> similar to setup_joins() and other SQLQuery features. We'd also have
> to create code for grouping filters into individual queries on tables.
> The Query class should take care of as much of the common stuff as
> possible, so nonrel backends can potentially emulate every single SQL
> feature (e.g., via MapReduce or whatever) with the least effort.
> Otherwise this refactoring would actually have more disadvantages than
> our current SQLCompiler-based approach in Django-nonrel (as ridiculous
> as that sounds).
>
> However, it's important that all of the emulated features are handled
> not by the backend, but by a reusable code layer which sits on top of
> the nonrel backends. It would be wasteful to let every backend
> developer write his own JOIN emulation and denormalization and
> aggregate code, etc.. The refactored ORM should at least still allow
> for writing some kind of "proxy" backend that sits on top of the
> actual nonrel backend and takes care of SQL features emulation. I'm
> not sure if it's a good idea to integrate the emulation into Django
> itself because then progress will be slowed down.
>
> Ideally, we should provide a simplified API for nonrel backends,
> similar to the one that we recently published for Django-nonrel, so a
> backend could be written in two days instead of two weeks. We can port
> our work over to the refactored ORM, so this you don't have to deal
> with this (except if it should be officially integrated into Django).
>

No.  I am vehemently opposed to attempting to extensively emulate the
features of a relational database in a non-relational one.  People
talk about the "object relational" impedance mismatch, much less the
"object-relational non-relational" one.  I have no interest in
attempting to support any attempts at emulating features that just
don't exist on the databases they're being emulated on.

> In addition to these changes you'll also need to take care of a few
> other things:
>
> Many NoSQL DBs provide a simple "upsert"-like behavior where on save()
> they either create a new entity if none exists with that primary key
> or update the existing entity if one exists. However, on save() Django
> first checks if an entity exists. This would be inefficient and
> unnecessary, so the backend should be able to turn that behavior off.
>
> On delete() Django also deletes related objects. This can be a costly
> operation, especially if you have a large number of entities. Also,
> the queries that collect the related entities can conflict with
> transaction support at least on App Engine and it might also be very
> inefficient on HBase. IOW, it's not sufficient to let the user handle
> the deletion for large datasets. So, non-relational (and maybe also
> relatinoal) DBs should be able to defer and split up the deletion
> process into background tasks - which would also simplify the
> developer's job because he doesn't have to take care of manually
> w

Re: [GSOC] NoSQL Support for the ORM

[Alex Gaynor]

On Wed, Apr 7, 2010 at 6:47 AM, Russell Keith-Magee
 wrote:
> On Wed, Apr 7, 2010 at 8:11 AM, Alex Gaynor  wrote:
>> Non-relational database support for the Django ORM
>> ==
>>
>> Note:  I am withdrawing my proposal on template compilation.  Another student
>> has expressed some interest in working on it, and in any event I am now more
>> interested in working on this project.
>>
>> About Me
>> 
>>
>> I'm a sophomore computer science student at Rensselaer Polytechnic Institute.
>> I'm a frequent contributor to Django (including last year's successful 
>> multiple
>> database GSoC project) and other related projects; I'm also a committer on 
>> both
>> `Unladen Swallow `_ and
>> `PyPy `_.
>>
>> Background
>> ~~
>>
>> As the person responsible for large swaths of multiple database support I am
>> intimately familiar with the architecture of the ORM, the code itself, and 
>> the
>> various concerns that need to be accounted for (pickleability, etc.).
>>
>> Rationale
>> ~
>>
>> Non-relational databases tend to support some subset of the operations that 
>> are
>> supported on relational databases, therefore it should be possible to perform
>> these operations on all databases.  Some people are of the opinion that we
>> shouldn't bother to support these databases, because they can't perform all
>> operations, I'm of the opinion that the abstraction is already a little 
>> leaky,
>> we may as well exploit this for a common API where possible, as well as 
>> giving
>> users of these databases the admin and models forms for free.
>>
>> Method
>> ~~
>>
>> The ORM architecture currently has a ``QuerySet`` which is backend agnostic, 
>> a
>> ``Query`` which is SQL specific, and a ``SQLCompiler`` which is backend
>> specific (i.e. Oracle vs. MySQL vs. generic).  The plan is to change 
>> ``Query``
>> to be backend agnostic by delaying the creation of structures that are SQL
>> specific, specifically join/alias data.  Instead of structures like
>> ``self.where``, ``self.join_aliases``, or ``self.select`` all working in 
>> terms
>> of joins and table aliases the composition of a query would be stored in 
>> terms
>> of a tree containing the "raw" filters, as passed to the filter calls, with
>> things like ``Field.get_prep_value`` called appropriately.  The 
>> ``SQLCompiler``
>> will be responsible for computing the joins for all of these data-structures.
>
> I can see the intention here, and I can see how this approach could be
> used to solve the problem. However, my initial concern is that normal
> SQL users will end up carrying around a lot of extra overhead so that
> they can support backends that they will never use.
>
> Have you given any thought to how complex the datastructures inside
> Query will need to be, and how complex and/or expensive the conversion
> process will be?
>

I see no reason they need to be any more complex than the current
ones.  You have a tree that represents filters (combined where and
having, this means that the SQLCompiler is responsible for splitting
these up, which I think will make fixing some other bugs easier (i.e.
disjunction with a filter on aggregates currently doesn't work)).
There's already quite a lot of stuff that's computed later, such as
select_related's transformation into JOINs.

> Other issues that spring to mind:
>
>  * What about nonSQL datatypes? List/Set types are a common feature of
> Non-SQL backends, and are The Right Way to solve a whole bunch of
> problems. How do you propose to approach these datatypes? What (if
> any) overlap exists between the use of set data types and m2m? Is
> there any potential overlap between supporting List/Set types and
> supporting Arrays in SQL?
>

Is there overlap between List/Set and Arrays in SQL?  Probably.  In my
opinion there's no reason, once we have a good clean seperation of
concerns in the architecture that implementing a ListField would be
particularly hard.  If we happened to include one in Django, all the
better (from the perspective of interoperability).

>  * How does a non-SQL backend integrate with syncdb and other setup
> tools? What about inspectdb?
>

Most, but not all non-relational databases don't require table setup
the way relational DBs do.  MongoDB doesn't require anything at all,
by contrast Cassandra requires an XML configuration file.  How to
handle these is a little touchy, but basically I think syncdb should
stay conceptually pure, generating "tables", if extra config is needed
backends should ship custom management commands.

As for inspectdb it only really makes sense on backends that have
structured "tables", so they could implement it, and other backends
could punt.

>  * What about basic connection management? Is the existing Connection
> API likely to be compatible, or will modifications be required?
>

No, it's not.  non-relational data

Re: [GSOC] NoSQL Support for the ORM

[lasizoillo]

2010/4/7 Alex Gaynor :

>  * 2 weeks - begin working on a backend for a non-relational database 
> (probably
>   MongoDB)

Pymodels[1] have backends for MogoDB and Tokyo Tyrant/Cabinet. Maybe
some things can be reused in backend.

http://bitbucket.org/neithere/pymodels/

Regards,

Javi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: [GSOC] NoSQL Support for the ORM

[Alex Gaynor]

On Wed, Apr 7, 2010 at 2:19 PM, lasizoillo  wrote:
> 2010/4/7 Alex Gaynor :
>
>>  * 2 weeks - begin working on a backend for a non-relational database 
>> (probably
>>   MongoDB)
>
> Pymodels[1] have backends for MogoDB and Tokyo Tyrant/Cabinet. Maybe
> some things can be reused in backend.
>
> http://bitbucket.org/neithere/pymodels/
>
> Regards,
>
> Javi
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Django developers" group.
> To post to this group, send email to django-develop...@googlegroups.com.
> To unsubscribe from this group, send email to 
> django-developers+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/django-developers?hl=en.
>
>

I don't really see how, they use a completely different API.

Alex

-- 
"I disapprove of what you say, but I will defend to the death your
right to say it." -- Voltaire
"The people's good is the highest law." -- Cicero
"Code can always be simpler than you think, but never as simple as you
want" -- Me

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: NoSQL Support for the ORM

[Waldemar Kornewald]

On Wed, Apr 7, 2010 at 5:12 PM, Alex Gaynor  wrote:
> No.  I am vehemently opposed to attempting to extensively emulate the
> features of a relational database in a non-relational one.  People
> talk about the "object relational" impedance mismatch, much less the
> "object-relational non-relational" one.  I have no interest in
> attempting to support any attempts at emulating features that just
> don't exist on the databases they're being emulated on.

This decision has to be based on the actual needs of NoSQL developers.
Did you actually work on non-trivial projects that needed
denormalization and in-memory JOINs and manually maintained counters?
I'm not making this up. The "dumb" key-value store API is not enough.
People are manually writing lots of code for features that could be
handled by an SQL emulation layer. Do we agree until here?

Then, the question boils down to: Is the ORM the right place to handle
those features?

We see more advantages in moving those features into the ORM instead
of some separate API:
No matter whether you do denormalization or an in-memory JOIN, you end
up emulating an SQL-like JOIN. When you're maintaining a counter you
again do a simple and very common operation supported by SQL:
counting. Django's ORM already provides that functionality. Django's
current reusable apps already use that functionality. Developers
already know Django's ORM and thus also that functionality. By moving
these features into the ORM
* existing Django apps will either work directly on NoSQL or at least
be much easier to port
* Django apps written for NoSQL will be portable across all NoSQL DBs
without any code changes and in the worst case require only minor
changes to switch to SQL
* the resulting code is shorter and easier to understand than with a
separate API which would only add another layer of indirection you'd
have to think about *every* (!) single time you work with models (and
if you have to think about this while writing model code you end up
with potentially a lot more bugs, as is actually the case in practice)
* developers won't have to use and learn a different models API (you'd
only need to learn an API for specifying "optimization" rules, but the
models would still be the same)

App Engine's indexes are not that different from what we propose. Like
many other NoSQL DBs, the datastore doesn't create indexes for all
possible queries. Sometimes you'll need a composite index to make
certain queries work. On Cassandra, CouchDB, Redis, and many other
"crippled" NoSQL DBs you solve this problem by maintaining even the
most trivial DB indexes with manually written indexing *code* (and I
mean *anything* that filters on fields other than the primary key). I
bet five years ago database developers would've called anyone nuts
who'd seriously suggest that nonsense, but somehow the NoSQL hype
makes developers forget about productivity. Anyway, on App Engine,
instead of writing code for those trivial indexes you add a simple
index definition to your index.yaml (actually, it's automatically
generated for you based on the queries you execute) and suddenly the
normal query API supports the respective filter rules transparently
(with exactly the same API; this is in strong contrast to Cassandra,
etc. which also make you manually write code for traversing those
manually implemented indexes! basically, they make you implement a
little specialized DB for every project and this is no joke, but the
sad truth). Now, our goal is to bring App Engine's indexing
definitions to the next level and allow to specify denormalization and
other "advanced" indexing rules which make more complicated queries
work transparently, again via the same API that everyone already
knows.

Instead of seeing this as object-relational non-relational mapping you
should see this as an object-relational mapping for a type of DB that
needs explicitly specified indexing rules for complex queries (which,
if you really think about it, exactly describes what working with
NoSQL DBs is like).

>> In addition to these changes you'll also need to take care of a few
>> other things:
>>
>> Many NoSQL DBs provide a simple "upsert"-like behavior where on save()
>> they either create a new entity if none exists with that primary key
>> or update the existing entity if one exists. However, on save() Django
>> first checks if an entity exists. This would be inefficient and
>> unnecessary, so the backend should be able to turn that behavior off.
>>
>> On delete() Django also deletes related objects. This can be a costly
>> operation, especially if you have a large number of entities. Also,
>> the queries that collect the related entities can conflict with
>> transaction support at least on App Engine and it might also be very
>> inefficient on HBase. IOW, it's not sufficient to let the user handle
>> the deletion for large datasets. So, non-relational (and maybe also
>> relatinoal) DBs should be able to defer and split up the deletion
>> process into b

[JOB] Front-End Developer - Contract/Telecommute | 40-50/hour

[OSS]

Front-End Developer - Contract/Telecommute | 40-50/hour

My client is a B2B media company and they are looking to hire a Front-
End Web Developer for a few upcoming projects, ranging from an online
publication development project to social media applications. This is
contract work for now, but it might have the potential to turn into a
permanent position for the right person.

Here's a quick run down of the important stuff they are looking for,
ideally:

* Solid JavaScript development experience. JQuery is a huge plus.
* Solid PHP experience. WordPress plug-in development would be great.
* Python experience would be a huge plus. Django experience an even
bigger plus.
* Solid HTML/CSS. This almost goes without saying.
* Experience working with templating languages would be great.
* Strong visual design skills. You should be able to craft amazing-
looking, functional user-interfaces even if you're not working off an
explicit mockup a designer has given you.
* The ability to learn and adapt quickly to new languages, APIs,
development requirements, etc.

Above all else, what they require is a strong commitment to
productivity. They are not hiring somebody because their team lacks
the skill sets mentioned above; they are hiring somebody because they
have a lot of projects on their plate and they need to increase their
output.  They are only interested in working with people who are
passionate about helping them increase the quality and quantity of
development work they do.

Although my client has an office in NYC, your geographic U.S. location
is not important, since the development team is scattered around the
country.  Wherever you live, you'll have to be comfortable working
with others remotely.  You must be able to contribute as little as 20
or as many as 40+ hours per week.

To be considered, please submit your RESUME and HOURLY requirements to
beau[AT]open-source-staffing.com

Thank you,
Beau J. Gould
--
Open Source Staffing
http://www.open-source-staffing.com
http://www.facebook.com/beau.gould
beau[AT]open-source-staffing.com

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: [GSOC] NoSQL Support for the ORM

[Waldemar Kornewald]

On Wed, Apr 7, 2010 at 5:22 PM, Alex Gaynor  wrote:
>> Other issues that spring to mind:
>>
>>  * What about nonSQL datatypes? List/Set types are a common feature of
>> Non-SQL backends, and are The Right Way to solve a whole bunch of
>> problems. How do you propose to approach these datatypes? What (if
>> any) overlap exists between the use of set data types and m2m? Is
>> there any potential overlap between supporting List/Set types and
>> supporting Arrays in SQL?
>>
>
> Is there overlap between List/Set and Arrays in SQL?  Probably.  In my
> opinion there's no reason, once we have a good clean seperation of
> concerns in the architecture that implementing a ListField would be
> particularly hard.  If we happened to include one in Django, all the
> better (from the perspective of interoperability).

Do all SQL DBs provide an array type? PostgreSQL has it and I think it
can exactly mimic NoSQL lists, but I couldn't find an equivalent in
sqlite and MySQL. Does this possibly stand in the way of integrating
an official ListField into Django or is it OK to have a field that
isn't supported on all DBs? Or can we fall back to storing the list
items in separate entities in that case?

>>  * How does a non-SQL backend integrate with syncdb and other setup
>> tools? What about inspectdb?
>>
>
> Most, but not all non-relational databases don't require table setup
> the way relational DBs do.  MongoDB doesn't require anything at all,
> by contrast Cassandra requires an XML configuration file.  How to
> handle these is a little touchy, but basically I think syncdb should
> stay conceptually pure, generating "tables", if extra config is needed
> backends should ship custom management commands.

Essentially, I agree, but I would add things like auto-generated
CouchDB views to the syncdb process (since syncdb on SQL already takes
care of creating indexes, too).

>>  * Why the choice of MongoDB specifically? Do you have particular
>> experience with MongoDB? Does MongoDB have features that make it a
>> good choice?
>>
>
> MongoDB offers a wide range of filtering options, which from my
> perspective means it presents a greater test of the flexibility of the
> developed APIs.  For this reason GAE would also be a good choice.
> Something like Riak or Cassandra, which basically only have native
> support for get(pk=3) would be a poor test of the flexibility of the
> API.

MongoDB really is a good choice. Out-of-the-box (without manual index
definitions) it provides more features than GAE and most other NoSQL
DBs. MongoDB and GAE should also have the simplest backends.

Why should the Cassandra/CouchDB/Riak/Redis/etc. backend only support
pk=... queries? There's no reason why the backend couldn't maintain
indexes for the other fields and transparently support filters on any
field. I mean, you don't really want developers to manually create and
query separate indexing models for mapping one field value to its
respective primary key in the primary model table. We can do much
better than that.

>>  * Given that you're only proposing a single proof-of-concept backend,
>> have you given any thought to the needs of other backends? It's not
>> hard to envisage that Couch, Cassandra, GAE etc will all have slightly
>> different requirements and problems. Is there a common ground that
>> exists between all data store backends? If there isn't, how do you
>> know that what you are proposing will be sufficient to support them?
>>
>
> To a certain extent this is a matter of knowing the featuresets of the
> databases and, hopefully, having a mentor who is knowledgeable about
> them.  The reality is under the GSOC time constraints attempting to
> write complete backends for multiple databases would probably be
> impossible.

Well, you might be able to quickly adapt the MongoDB backend to GAE
(within GSoC time constraints) due to their similarity. Anyway, there
is common ground between the NoSQL DBs, but this highly depends on
what problem we agree to solve. If we only provide exactly the
features that each DB supports natively, they'll appear dissimilar
because they take very different approaches to indexing and if this
isn't abstracted and automated NoSQL support doesn't really make sense
with Django. OTOH, if the goal is to make an abstraction around their
indexes they can all look very similar from the perspective of
Django's ORM (of course they have different "features" like sharding
or eventual consistency or being in-memory DBs or supporting fast
writes or reads or having transactions or ..., but in the end only few
of these features have any influence on Django's ORM, at all).

Bye,
Waldemar Kornewald

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com

Re: GSoC: App Loading

[Russell Keith-Magee]

On Wed, Apr 7, 2010 at 9:45 PM, Dagvadorj Galbadrakh
 wrote:
> On Wed, Apr 7, 2010 at 2:40 PM, Russell Keith-Magee
>  wrote:
>> If you're going to tackle this problem, I'd
>> rather see you concentrate on issues like:
>>
>>  * Translating application names
>>  * Solving the "two applications called auth" problem
>>  * Providing configurability for db-level naming convention at the
>> deployment level (e.g., so you can override db_prefix and table names
>> when you deploy an app)
>>  * Providing configurability for 'app level' settings; for example,
>> when deploying contrib.comments, setting the name of the model that
>> will be used for content.
>>
>> Yours,
>> Russ Magee %-)
>>
>
> So shortly do you imply that the "app loading" problem is indeed the
> problem of integrating several application via db tables through
> configuration in deployment?

I'm afraid I don't even understand the question.

The app loading problem is potentially many things. The list of bullet
points I gave is an indication of a working subset of those problems
that I think is worth pursuing for GSoC.

Yours,
Russ Magee %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Front-End Developer - Contract/Telecommute | 40-50/hour

[David Cramer]

On Apr 7, 4:47 pm, OSS  wrote:
> Front-End Developer - Contract/Telecommute | 40-50/hour
>
> My client is a B2B media company and they are looking to hire a Front-
> End Web Developer for a few upcoming projects, ranging from an online
> publication development project to social media applications. This is
> contract work for now, but it might have the potential to turn into a
> permanent position for the right person.
>
> Here's a quick run down of the important stuff they are looking for,
> ideally:
>
> * Solid JavaScript development experience. JQuery is a huge plus.
> * Solid PHP experience. WordPress plug-in development would be great.
> * Python experience would be a huge plus. Django experience an even
> bigger plus.
> * Solid HTML/CSS. This almost goes without saying.
> * Experience working with templating languages would be great.
> * Strong visual design skills. You should be able to craft amazing-
> looking, functional user-interfaces even if you're not working off an
> explicit mockup a designer has given you.
> * The ability to learn and adapt quickly to new languages, APIs,
> development requirements, etc.
>
> Above all else, what they require is a strong commitment to
> productivity. They are not hiring somebody because their team lacks
> the skill sets mentioned above; they are hiring somebody because they
> have a lot of projects on their plate and they need to increase their
> output.  They are only interested in working with people who are
> passionate about helping them increase the quality and quantity of
> development work they do.
>
> Although my client has an office in NYC, your geographic U.S. location
> is not important, since the development team is scattered around the
> country.  Wherever you live, you'll have to be comfortable working
> with others remotely.  You must be able to contribute as little as 20
> or as many as 40+ hours per week.
>
> To be considered, please submit your RESUME and HOURLY requirements to
> beau[AT]open-source-staffing.com
>
> Thank you,
> Beau J. Gould
> --
> Open Source 
> Staffinghttp://www.open-source-staffing.comhttp://www.facebook.com/beau.gould
> beau[AT]open-source-staffing.com

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Front-End Developer - Contract/Telecommute | 40-50/hour

[Russell Keith-Magee]

On Thu, Apr 8, 2010 at 11:45 AM, David Cramer  wrote:
> 

... which they've been given several times in the past. I've just
blocked their account.

Yours,
Russ Magee %-)

> On Apr 7, 4:47 pm, OSS  wrote:
>> Front-End Developer - Contract/Telecommute | 40-50/hour
>>
>> My client is a B2B media company and they are looking to hire a Front-
>> End Web Developer for a few upcoming projects, ranging from an online
>> publication development project to social media applications. This is
>> contract work for now, but it might have the potential to turn into a
>> permanent position for the right person.
>>
>> Here's a quick run down of the important stuff they are looking for,
>> ideally:
>>
>> * Solid JavaScript development experience. JQuery is a huge plus.
>> * Solid PHP experience. WordPress plug-in development would be great.
>> * Python experience would be a huge plus. Django experience an even
>> bigger plus.
>> * Solid HTML/CSS. This almost goes without saying.
>> * Experience working with templating languages would be great.
>> * Strong visual design skills. You should be able to craft amazing-
>> looking, functional user-interfaces even if you're not working off an
>> explicit mockup a designer has given you.
>> * The ability to learn and adapt quickly to new languages, APIs,
>> development requirements, etc.
>>
>> Above all else, what they require is a strong commitment to
>> productivity. They are not hiring somebody because their team lacks
>> the skill sets mentioned above; they are hiring somebody because they
>> have a lot of projects on their plate and they need to increase their
>> output.  They are only interested in working with people who are
>> passionate about helping them increase the quality and quantity of
>> development work they do.
>>
>> Although my client has an office in NYC, your geographic U.S. location
>> is not important, since the development team is scattered around the
>> country.  Wherever you live, you'll have to be comfortable working
>> with others remotely.  You must be able to contribute as little as 20
>> or as many as 40+ hours per week.
>>
>> To be considered, please submit your RESUME and HOURLY requirements to
>> beau[AT]open-source-staffing.com
>>
>> Thank you,
>> Beau J. Gould
>> --
>> Open Source 
>> Staffinghttp://www.open-source-staffing.comhttp://www.facebook.com/beau.gould
>> beau[AT]open-source-staffing.com
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Django developers" group.
> To post to this group, send email to django-develop...@googlegroups.com.
> To unsubscribe from this group, send email to 
> django-developers+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/django-developers?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.