Re: [GSoC] Application Loading
On Fri, Apr 9, 2010 at 12:33 AM, Nick Sandford wrote:
> An App Loading mechanism for Django
>
>
> About Me
> --
> Hi everyone,
>
> My name is Nick Sandford, I'm an electrical engineering student at the
> University of Western Australia.
>
> Background
> ---
>
> I haven't been a particularly active contributor to any open source project -
> here is where I hope to change that. In my current work at Christ
> Church Grammar School we use Django heavily for most of our internal
> projects. I've
> followed django-dev closely for a couple of years and I have poked around with
> its internals on various occasions.
Ok - I clearly need to get a Perth DjUG going...
> Plan
> -
>
> Implement an improved application loading mechanism into Django.
>
> Rationale
> -
>
> Django current application loading code is inflexible in many cases. There
> exists a need for a number of extra features to be added to the way
> applications are handled such as:
>
> * The ability to internationalise application names.
> * The ability to customise application names similar to ``verbose_name`` in
> ``model._meta``
> * Deploy the same application multiple times in a single project.
> * Deploy two different applications with the same name.
> * Manage settings within applications.
>
> Method
> ---
>
> I don't intend to solve the third dot point of the previous list - it seems
> like a difficult problem, possibly to be tackled post-GSoC. What I do intend
> to do is to move towards 'application classes' more akin to
> ``django.contrib.admin``.
>
> New syntax in ``settings.INSTALLED_APPS`` will be introduced to address the
> previous issues. Some examples of accepted syntax:
>
> .. sourcecode:: python
>
> INSTALLED_APPS = (
> 'django.contrib.auth',
> app('blog.BlogApplication', 'blog_1', 'My blog', 'app1_'),
> app('blog.BlogApplication', 'blog_2', 'Some other blog', 'app2_'),
> app(path='tagging.Tagging', verbose_name='My tagging application'),
> app('categories', db_prefix='cat_'),
> app({'path': 'django.contrib.admin.AdminApplication',
> 'label': 'admin',
> 'verbose_name': 'Secret Admin'}),
> )
>
> The ``app`` function will take four arguments, three of which are optional.
> These are ``path``, ``label``, ``verbose_name``, and ``db_prefix``. It will
> return an instance of an ``Application`` object, which will contain all of an
> installed application's information. ``path`` will be the dotted path to a
> subclass of ``Application``. The downside is that ``settings.py`` requires
> an import, which may be against style rules.
>
> .. sourcecode:: python
>
> def app(path, label=None, verbose_name=None, db_prefix='')
> if not path or not isinstance(path, basestring):
> raise ImproperlyConfigured('Application path must be string.')
> application_class = import_module(path)
> return application_class(path, label, verbose_name, db_prefix)
>
> ``INSTALLED_APPS`` will then be a tuple containing strings or ``Application``
> instances. The application loading code will iterate over ``INSTALLED_APPS``
> and construct an internal cache of ``Application`` instances to be used with
> ``get_models``, etc. For backwards compatibility, if an element of the tuple
> is
> a string, an instance of a base ``Application`` class will be created with
> sane
> defaults (similar to ``app_label`` at the moment).
>
> The ``Application`` class will be very similar to ``django.contrib.admin``'s
> ``AdminSite`` and will hopefully simplify application settings. If you write
> views and urls directly on the ``Application`` class itself, instead of an
> ``from django.conf import settings`` and subsequent ``settings.API_KEY``, you
> could just reference ``self.api_key`` for instance. This wouldn't be the
> required, just an optional extra.
>
> Model classes will get a ``_meta.app`` attribute, which will be an instance
> of the model's ``Application`` class. Models should only be associated with
> one
> application.
>
> I agree with moving ``django.db.models.loading`` to ``core.apps``, since we'll
> no longer require applications to have a ``models.py``. A reference to the
> new functions in ``core.apps`` will still live in ``django.db.models.loading``
> for backwards compatibility.
>
> A subclass of ``Application`` might look like:
>
> .. sourcecode:: python
>
> from django.views.simple import direct_to_template
> from djang.core.apps import Application
> from blog.models import Entry, Category
>
> class BlogApplication(Application):
> models = [Entry, Category]
> api_key = 'default'
>
> def entry_detail(self, slug, request,
> template_name='blog/entry_detail.html'):
> entry = get_object_or_404(Entry, slug=slug)
> context = {
> 'entry': entry,
> 'api_key': self.api_key,
> }
> return dir
Re: [GSOC] NoSQL Support for the ORM
On Thu, Apr 8, 2010 at 5:55 AM, Waldemar Kornewald wrote: > On Wed, Apr 7, 2010 at 5:22 PM, Alex Gaynor wrote: >>> Other issues that spring to mind: >>> >>> * What about nonSQL datatypes? List/Set types are a common feature of >>> Non-SQL backends, and are The Right Way to solve a whole bunch of >>> problems. How do you propose to approach these datatypes? What (if >>> any) overlap exists between the use of set data types and m2m? Is >>> there any potential overlap between supporting List/Set types and >>> supporting Arrays in SQL? >>> >> >> Is there overlap between List/Set and Arrays in SQL? Probably. In my >> opinion there's no reason, once we have a good clean seperation of >> concerns in the architecture that implementing a ListField would be >> particularly hard. If we happened to include one in Django, all the >> better (from the perspective of interoperability). > > Do all SQL DBs provide an array type? PostgreSQL has it and I think it > can exactly mimic NoSQL lists, but I couldn't find an equivalent in > sqlite and MySQL. Does this possibly stand in the way of integrating > an official ListField into Django or is it OK to have a field that > isn't supported on all DBs? Or can we fall back to storing the list > items in separate entities in that case? No - Array types aren't available everywhere. However, it would be nice to be able to support them (even if not in core); if this GSoC lays the groundwork to make this possible, then it's worth looking at. I was more interested in the m2m issue - the 'natural' way to handle m2m on some NoSQL isn't to have a separate relation, it's to maintain a list/set of related references. Yours, Russ Magee %-) -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: GSoC: App Loading
On Wed, Apr 7, 2010 at 10:26 PM, Jannis Leidel wrote: > > Am 07.04.2010 um 13:40 schrieb Russell Keith-Magee: > >> On Mon, Apr 5, 2010 at 5:35 AM, Arthur Koziel >> wrote: >>> Hi, >>> I’m going to apply for GSoC with the goal of refactoring the app loading >>> [0]. I’ve been looking at Django’s current app loading implementation >>> (db.models.loading), Vinay Sajip’s patch in #3591 [1] and the notes on the >>> InstalledAppsRevision wiki page [2]. Here's what I'm planning to change. Any >>> feedback is appreciated. >>> Step 1 >>> - >>> Deal with the two main problems of the current app loading implementation, >>> and make it possible to: >>> - deploy several instances of the same app >>> - deploy two apps with the same name >>> The INSTALLED_APPS setting will be changed to: >>> >>> INSTALLED_APPS = InstalledApps( >>> 'django.contrib.auth', # strings will be converted to App() >> >> Strings will be converted to App()... by What? When? >> >> What happens to existing code that is using "for app in INSTALLED_APS" >> as an idiom? > > I'm worried that creating the app instances with an ``app()`` callable would > require an import and contradicts with the convention of having dotted paths > for any configurable extension. Agreed. Starting down the path of requiring imports in the settings file isn't something I'd like to encourage. > That's why I think INSTALLED_APPS should be kept as an iterable of dotted > paths which is converted to an app registry during the start-up (which > already happens for wildcard entries like "django.contrib.*"). Each entry of > the list would be a string with one of the following values: > > - Dotted path to an app module (current default, e.g. 'django.contrib.auth'). > During the start-up an anonymous app instance would be initialized, using the > models module of the loaded app module. > > - Dotted path to an app class (e.g. 'django.contrib.auth.AuthApp'). During > the start-up a named app instance would be created, either with the models > module of the module the app class is located in, the model attribute of the > class or none (model-less app). > > - Dotted path to an app instance (e.g. 'django.contrib.admin.site'). During > the start-up an existing app instance would be handled like it has been > loaded in one of the former two cases. I like this; although I think it has some potential to cause a whole different bunch of headaches. At the moment, "for app in settings.INSTALLED_APPS" is guaranteed to give you a list of app names; identifying the difference between an app module, and app class and an app instance isn't going to be trivial from the point of view of an end developer -- since, at the end of the day, they're all just strings. Yours, Russ Magee %-) -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: [GSOC] Application to update the Test Suite
On Fri, Apr 9, 2010 at 8:59 AM, Paul McMillan wrote: > I've written an application to improve Django's Test Suite. > http://code.djangoproject.com/wiki/SummerOfCode2010#Testingupdates > > My application is here (also pasted below for convenience): > http://socghop.appspot.com/gsoc/student_proposal/show/google/gsoc2010/paulmcmillan/t127077396156 > > I'm hoping for constructive feedback particularly in the area of > maintaining test integrity. I realize this is a fairly late > application, but the task itself is relatively non-controversial. > > Thanks! > -Paul > > = > The problem > > Django's testing suite has become fragmented over time. The modeltests > and regressiontests directories serve similar purposes and should be > joined. Nobody wants to do concerted work to improve testing because > it isn't a showstopping bug or a new feature. GSOC provides an > opportunity to support this kind of work. > > Proposal > > I will merge the code in modeltests and regressiontests, turning > doctests into unit tests and cleaning up existing code as I go. I will > use coverage.py to verify that the re-written tests cover the same > areas of code as the previous ones did, as well as identifying areas > without coverage. > > Many GSOC proposals are for large feature improvements that will > involve a major code branch and merge. This project can instead be > done as a series of incremental patches, reducing the impact of a > failure to complete and providing more immediate results. This is one of the appealing aspect to housekeeping style projects. However, in this case, you may want to give some thought to the backporting complexity that you will be introducing. Yes, when you have finished converting the "basic" doctest into a unittest suite, we can commit that to trunk. However, at the point we do this, there is a massive disconnect between trunk and the stable branches. If someone needs to fix a bug, they will essentially need to write the test twice; once for the old framework, and once for the new. Do you have any suggestions on what we should be doing to mitigate this complexity? > In order to avoid introducing new bugs, I will configure a variety of > virtual machines for testing. At a minimum, I will test in Ubuntu, > Windows XP, and Mac OS X, with the most recent versions of Python 2.4, > 2.5, and 2.6. On each of these systems, I will test against the > supported database backends (PostgreSQL, MySQL, SQLite, Oracle). I > will build scripts to automate the test running in these environments > so that testing can continue prior to patches being merged to trunk. We have a buildbot currently in development; if you're have the resources to contribute, it would be good if these could be integrated permanently into the build farm. > Since Django 1.2 is dropping support for Python 2.3, there is extra > code in the test suite to support 2.3 that can be removed. This should > improve the speed of test running. If it does, it will be by a trivial fraction. The extra overheads cause by Py2.3 requirements are fairly minor. > Internally, when a doctest is run, Python converts it into a unittest > before actually executing it. It might be possible to use this code to > automate the conversion. Converting to unittests should improve test > run time because it eliminates the parsing necessary to convert each > doctest into a unittest. The #python IRC channel anecdotally agrees on > this point. The #python IRC channel needs to think about the problem a little more. Yes - a simple unittest will be faster than a comparable doctest. The problem is that this isn't what will be happening. In a doctest, there is one prolonged setup process; as the doctest runs, the test setup evolves with the test. In a unittest, each testable item needs to be isolated, which means running setUp and tearDown multiple times. Each setup/teardown cycle means creating a bunch of database objects, and then rolling back the database to destroy those objects. The setup/teardown process is effectively done once in a doctest. It is done N times on a unittest with N tests. This is a potentially expensive operation, and it *will* slow down test runs. While this project is *very* appealing from a project point of view -- we get the benefit of individually failing and isolated tests -- but the risk of test runtime blowouts is huge. Any suggestions on how to tackle this problem (and speed up test runs in general) would be almost as valuable as the doctest->unittest conversion itself. Yours, Russ Magee %-) -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: NoSQL Support for the ORM
On Thu, Apr 8, 2010 at 11:03 PM, flo...@gmail.com wrote: > On Apr 8, 12:32 pm, Waldemar Kornewald wrote: > >> What I'm proposing is not a complete emulation of all features at all >> cost, but simply an automation of the things that are possible and in >> wide use on nonrel DBs. Moreover, you'd only use these features where >> actually needed, so this would be a helper that replaces exactly the >> code you'd otherwise write by hand - nothing more. Denormalization, >> counters, etc. indeed go over the network, but people still do it >> because there is no alternative (CouchDB being an exception, but there >> we can auto-generate a view, so the index is created on the DB: same >> game, different location). > > "Denormalization, counters, etc." is a completely orthogonal problem. > Solving those problems would help even those who are using relational > databases, in fact. But just because it's useful, and precisely > because it's orthogonal, means it doesn't belong in this summer of > code project. I guess we have a misunderstanding here. I never wanted to have it in this GSoC project. It's clearly out of scope. I just want to make sure that emulation will not be more difficult. > I think what you're going to run into is that since CouchDB, > Cassandra, MongoDB, GAE, Redis, Riak, Voldemort, etc. are all so > vastly different, that attempting to do *any* emulation will result in > serious pain down the line. You are absolutely right that those DBs are vastly different (*at the low level*) and that's why Django's ORM would suck as a crippled low-level replacement for the native NoSQL APIs. I can't imagine how you would map, for instance, Redis' list management features to the ORM. Depending on your problem you won't get around using Redis' native API to solve a problem efficiently, no matter how extensive the emulation layer is. Unless I've misunderstood, Alex wants to build just a low-level API replacement, but it doesn't make any sense because the native NoSQL APIs are much better at this task than Django's ORM will ever be. But there's a second use-case: Working with object-like data (with relations between objects). That's where people write indexing code by hand (column indexes, denormalization indexes, counters, etc.) which is very unproductive. This use-case is pretty common and it maps pretty well to Django's ORM *at the high level* because the high-level usage can look the same on all DBs: * get by username: User.objects.filter(username=...) * join via denormalization index: Profile.objects.filter(age=21, user__username=...) * keep counter for number of votes for each video: video.vote_set.count() * etc.. An abstraction/emulation layer can save you a lot of work because you won't have to maintain the required indexes by hand and those indexes don't make your query code more complicated. Also, it makes your code portable (except where you needed the native API for optimization). This also means that there is a clear separation of purpose: the native API for a few optimizations and Django's ORM for object-like data. We already use that distinction when we combine raw SQL with the ORM. It's no problem to automatically maintain such indexes on NoSQL DBs. As long as you're free to store whatever you want in the DB and run background tasks you have full control over everything. > It simply doesn't seem reasonable to claim that whatever refactoring > Alex does, will make "it more difficult or even impossible to > implement an emulation layer" because all he would be doing is > decoupling SQL from the query class. That can *only* make your goal > easier. When I said the ORM refactoring should not make it more difficult to implement the emulation layer Alex said that he was "vehemently opposed" to emulating SQL features extensively. I don't know what "extensively" means, but if we have to make a design decision during the refactoring and Alex says "I don't care about that feature for NoSQL" we might end up with an ORM that actually makes it more difficult. That's why I find it important that we agree on a common goal for the refactoring. Bye, Waldemar Kornewald -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: NoSQL Support for the ORM
On Apr 9, 9:06 am, Russell Keith-Magee wrote: > No - Array types aren't available everywhere. However, it would be > nice to be able to support them (even if not in core); if this GSoC > lays the groundwork to make this possible, then it's worth looking at. We already implemented a ListField in django-nonrel which is backend independent. This can be used as a starting point. > I was more interested in the m2m issue - the 'natural' way to handle > m2m on some NoSQL isn't to have a separate relation, it's to maintain > a list/set of related references. Your right, for many situations you use lists for m2m but in general you get problems with big lists because you would have to fetch big entities and on App Engine you can't have more than 5000 entries in a list. In that case you have to split the list over multiple entities. Additionally when making queries which need filters on both models of the m2m relation it is better to use an intermediary table with denormalization (for example via ManyToManyField). -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
GSoC
Hello to everyone, I'm not a skilled developer and i used Djnago only a couple of times but, however, i have a project a would like to develop with this framework, and as python is the programming language i am more familiar with i would like to send and attempt for the google summer of code. ___School___ I'm coming from a media design and comunication master in the Netherlands, Rotterdam, under the Hogeschool University. I'm using python more or less since one year and half so my programming skills are kind of "young" and i probably have too many ideas, so if my plan sounds "terrible" to you i understand and i only ask for a hint about how to begin realizing it. __Plan__ I am working at this website: www.pythonspeculativeinterface.org And although i still have some problem of design the main task i would like the application to deal with is the database of the wiki. As you can see in the first form of the website you can type some words and get a definition of it. This short text should be retrieved by the django application inside the wiki database. As long as i know this could be realized in two ways: 1) Routing the two databases, one of django and the other of the wiki. Or else setting up the wiki database as managed = False and using manytomany options to view the content of the wiki database. It sounds really artificial to me because i'm still not really familiar with databases settings and what so ever but this solution could be the best in order to keep the content of the form synchronized with the content of the wiki. I don't have the exact code example to do but i can attempt a timeline: 3 weeks Studing the mediawiki database and implementing the needed parts of it in django's database. 1 week debugging 4 weeks writing and debugging the django interface to the web form 2 weeks website redesign (now the website uses python in the cgi, javascript and php, i would like to reduce the numbers of language used to python and javascript only). 2) Writing a script which uses XML-RPC or a xml interface to the wiki database and namespaces in general, to get the content needed to the script from the wiki. It is a sort of "transduction" and it seems to me a more simple approach to the problem, because it doesn't directly involve databases synchronization which is the main problem to me in the first solution. I found something about namespaces and xml in the Ideas of SoC in Django page about it, here is the link http://code.djangoproject.com/wiki/SummerOfCode2010#Customizableserialization Timeline 4 weeks writing and debugging the script for the XML-RPC interface to the database 3 website redesign (now the website uses python in the cgi, javascript and php, i would like to reduce the numbers of language used to python and javascript only). Criticism, hints and other groups where to post the problem if you find it not relevant for the GSoC, or if you think i am not at the level to accomplish the task, are all welcome. Thanks Agnese -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: GSoC
On Fri, Apr 9, 2010 at 7:48 PM, tab1ta wrote: > Hello to everyone, > I'm not a skilled developer and i used Djnago only a couple of times > but, however, i have a project a would like to develop with this > framework, and as python is the programming language i am more > familiar with i would like to send and attempt for the google summer > of code. > > ___School___ > I'm coming from a media design and comunication master in the > Netherlands, Rotterdam, under the Hogeschool University. I'm using > python more or less since one year and half so my programming skills > are kind of "young" and i probably have too many ideas, so if my plan > sounds "terrible" to you i understand and i only ask for a hint about > how to begin realizing it. > > __Plan__ > > I am working at this website: > www.pythonspeculativeinterface.org > And although i still have some problem of design the main task i would > like the application to deal with is the database of the wiki. Hi Agnese, As the Django's wiki says [1], we're looking for projects that add value to Django itself - not application/CMS projects that use Django. It sounds to me like this project falls firmly in the category of a project that uses Django. [1] http://code.djangoproject.com/wiki/SummerOfCode2010 Yours, Russ Magee %-) -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Porting _django_bash_completion to Windows Powershell
One final question about the bash script's functionality--I cannot
test easily on Linux now, so I'd love to hear some feedback.
As far as I can tell, django-admin.py sqlall will only show
completions if DJANGO_SETTINGS_MODULE is set. Is this correct?
On Mar 17, 10:14 pm, Arthur Koziel wrote:
> On Mar 17, 2010, at 3:35 PM, guillermooo wrote:
>
> > autocomplete() is executed always, regardless whether the user has
> > requested completions or not. The only early exit point of
> > autocomplete() is reached if DJANGO_AUTO_COMPLETE is false. Otherwise,
> > sys.exit(1) returns to the console. How is this avoided when you do
> > something like `manage.py sqlall`? Is the variable
> > DJANGO_AUTO_COMPLETE implicitly unset after each execution of the
> > completion bash script?
>
> 36 COMPREPLY=( $( COMP_WORDS="${COMP_WORDS[*]}" \
> 37 COMP_CWORD=$COMP_CWORD \
> 38 DJANGO_AUTO_COMPLETE=1 $1 ) )
>
> The Python function is executed in a Bash subshell (that's what the outer
> round braces are for). COMP_WORDS/COMP_CWORD and DJANGO_AUTO_COMPLETE are
> local variables and available only within this subshell.
>
> Arthur
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
Re: [GSoC] Application Loading
An updated proposal:
An App Loading mechanism for Django
About Me
--
Hi everyone,
My name is Nick Sandford, I'm an electrical engineering student at the
University of Western Australia.
Background
---
I haven't been a particularly active contributor to any open source project -
here is where I hope to change that. In my current work at Christ Church
Grammar School we use Django heavily for most of our internal projects. I've
followed django-dev closely for a couple of years and I have poked around with
its internals on various occasions.
Plan
-
Implement an improved application loading mechanism into Django.
Rationale
-
Django current application loading code is inflexible in many cases. There
exists a need for a number of extra features to be added to the way
applications are handled such as:
* The ability to internationalise application names.
* The ability to customise application names similar to ``verbose_name`` in
``model._meta``
* Deploy the same application multiple times in a single project.
* Deploy two different applications with the same name.
* Manage settings within applications.
Method
---
I don't intend to solve the third dot point of the previous list - it seems
like a difficult problem, possibly to be tackled post-GSoC. What I do intend
to do is to move towards 'application classes' more akin to
``django.contrib.admin``.
I plan to introduce a new file into the project structure - ``apps.py`` which
will be used to register installed applications with the ``AppCache``. I will
also move to introduce a new setting in ``settings.py``, ``APP_LOADER`` which
will be a string containing a dotted path to an application loading class. This
will serve to facilitate a move from ``settings.INSTALLED_APPS`` to the new
``apps.py``. ``APP_LOADER`` will take one of two values:
'django.core.apps.loader.legacy', which will use ``settings.INSTALLED_APPS`` or
'django.core.apps.loader.registry', which would use the new registry-based
approach. Only applications registered in ``apps.py`` in the project directory
will be registered with the ``AppCache``. The default APP_LOADER setting for
1.3 will be 'django.core.apps.loader.legacy'.
``apps.py`` will contain purely application-related setup. This should be handy
to manage application settings and to ensure ``settings.py`` doesn't get too
full of application-specific settings. This will allow us to register an app
in one of three ways something like:
.. sourcecode:: python
from django.core import apps
from blog import BlogApplication
class MyBlogApplication(BlogApplication):
api_key = 'testing'
another_blog = BlogApplication(label='blog2',
verbose_name=_('Another blog'),
api_key='anothertest')
# an application class.
apps.register(MyBlogApplication, label='blog', verbose_name=_('My blog'))
# an application instance.
apps.register(another_blog)
# old-style application.
apps.register('django.contrib.auth')
This could allow us not to touch ``settings.INSTALLED_APPS`` at all. If people
need to iterate over ``settings.INSTALLED_APPS``, a move to ``get_apps`` will
be recommended.
The ``Application`` class will be very similar to ``django.contrib.admin``'s
``AdminSite`` and will hopefully simplify application settings. If you write
views and urls directly on the ``Application`` class itself, instead of an
``from django.conf import settings`` and subsequent ``settings.API_KEY``, you
could just reference ``self.api_key`` for instance. This wouldn't be the
required, just an optional extra.
In addition to providing application settings an ``Application`` will be
required to explicitly define their models. These models could live outside
the traditional application structure, provided they are included in the
``models`` list on ``Application``. The current structure promotes strictly
requiring models for an application to reside in that application's
``models.py``. I wouldn't be promoting changing the current status quo with
regard to this, but it would be possible to deviate from it and include models
defined outside of the applications ``models.py`` on an ``Application``.
An application class would have other benefits, as evidenced by a short look at
``django.contrib.admin.AdminSite``. It could concievably be used to define a
set of 'base' add/edit/delete views for each model on an ``Application``. It
could even be used to set up an application's urls. This could make an
``Application`` class unwieldy and large however, so it won't be forced and
people can continue using views.py and urls.py as before.
Model classes will get a ``_meta.app`` attribute, which will be an instance
of the model's ``Application`` class. Models should only be associated with one
application.
There are a number of places within Django that rely on ``app_label`` and
that will need to be addressed
Re: [GSoC] Application Loading
On Fri, Apr 9, 2010 at 2:59 PM, Russell Keith-Magee
wrote:
> On Fri, Apr 9, 2010 at 12:33 AM, Nick Sandford
> wrote:
>> An App Loading mechanism for Django
>>
>>
>> About Me
>> --
>> Hi everyone,
>>
>> My name is Nick Sandford, I'm an electrical engineering student at the
>> University of Western Australia.
>>
>> Background
>> ---
>>
>> I haven't been a particularly active contributor to any open source project -
>> here is where I hope to change that. In my current work at Christ
>> Church Grammar School we use Django heavily for most of our internal
>> projects. I've
>> followed django-dev closely for a couple of years and I have poked around
>> with
>> its internals on various occasions.
>
> Ok - I clearly need to get a Perth DjUG going...
>
>> Plan
>> -
>>
>> Implement an improved application loading mechanism into Django.
>>
>> Rationale
>> -
>>
>> Django current application loading code is inflexible in many cases. There
>> exists a need for a number of extra features to be added to the way
>> applications are handled such as:
>>
>> * The ability to internationalise application names.
>> * The ability to customise application names similar to ``verbose_name`` in
>> ``model._meta``
>> * Deploy the same application multiple times in a single project.
>> * Deploy two different applications with the same name.
>> * Manage settings within applications.
>>
>> Method
>> ---
>>
>> I don't intend to solve the third dot point of the previous list - it seems
>> like a difficult problem, possibly to be tackled post-GSoC. What I do intend
>> to do is to move towards 'application classes' more akin to
>> ``django.contrib.admin``.
>>
>> New syntax in ``settings.INSTALLED_APPS`` will be introduced to address the
>> previous issues. Some examples of accepted syntax:
>>
>> .. sourcecode:: python
>>
>> INSTALLED_APPS = (
>> 'django.contrib.auth',
>> app('blog.BlogApplication', 'blog_1', 'My blog', 'app1_'),
>> app('blog.BlogApplication', 'blog_2', 'Some other blog', 'app2_'),
>> app(path='tagging.Tagging', verbose_name='My tagging application'),
>> app('categories', db_prefix='cat_'),
>> app({'path': 'django.contrib.admin.AdminApplication',
>> 'label': 'admin',
>> 'verbose_name': 'Secret Admin'}),
>> )
>>
>> The ``app`` function will take four arguments, three of which are optional.
>> These are ``path``, ``label``, ``verbose_name``, and ``db_prefix``. It will
>> return an instance of an ``Application`` object, which will contain all of an
>> installed application's information. ``path`` will be the dotted path to a
>> subclass of ``Application``. The downside is that ``settings.py`` requires
>> an import, which may be against style rules.
>>
>> .. sourcecode:: python
>>
>> def app(path, label=None, verbose_name=None, db_prefix='')
>> if not path or not isinstance(path, basestring):
>> raise ImproperlyConfigured('Application path must be string.')
>> application_class = import_module(path)
>> return application_class(path, label, verbose_name, db_prefix)
>>
>> ``INSTALLED_APPS`` will then be a tuple containing strings or ``Application``
>> instances. The application loading code will iterate over ``INSTALLED_APPS``
>> and construct an internal cache of ``Application`` instances to be used with
>> ``get_models``, etc. For backwards compatibility, if an element of the tuple
>> is
>> a string, an instance of a base ``Application`` class will be created with
>> sane
>> defaults (similar to ``app_label`` at the moment).
>>
>> The ``Application`` class will be very similar to ``django.contrib.admin``'s
>> ``AdminSite`` and will hopefully simplify application settings. If you write
>> views and urls directly on the ``Application`` class itself, instead of an
>> ``from django.conf import settings`` and subsequent ``settings.API_KEY``, you
>> could just reference ``self.api_key`` for instance. This wouldn't be the
>> required, just an optional extra.
>>
>> Model classes will get a ``_meta.app`` attribute, which will be an instance
>> of the model's ``Application`` class. Models should only be associated with
>> one
>> application.
>>
>> I agree with moving ``django.db.models.loading`` to ``core.apps``, since
>> we'll
>> no longer require applications to have a ``models.py``. A reference to the
>> new functions in ``core.apps`` will still live in
>> ``django.db.models.loading``
>> for backwards compatibility.
>>
>> A subclass of ``Application`` might look like:
>>
>> .. sourcecode:: python
>>
>> from django.views.simple import direct_to_template
>> from djang.core.apps import Application
>> from blog.models import Entry, Category
>>
>> class BlogApplication(Application):
>> models = [Entry, Category]
>> api_key = 'default'
>>
>> def entry_detail(self, slug, request,
>> template_name='blog/entry_detail.html'
Re: [GSoC] Application Loading
On Fri, Apr 9, 2010 at 9:20 PM, Nick Sandford wrote:
> An updated proposal:
>
> An App Loading mechanism for Django
>
>
> About Me
> --
> Hi everyone,
>
> My name is Nick Sandford, I'm an electrical engineering student at the
> University of Western Australia.
>
> Background
> ---
>
> I haven't been a particularly active contributor to any open source project -
> here is where I hope to change that. In my current work at Christ Church
> Grammar School we use Django heavily for most of our internal projects. I've
> followed django-dev closely for a couple of years and I have poked around with
> its internals on various occasions.
>
> Plan
> -
>
> Implement an improved application loading mechanism into Django.
>
> Rationale
> -
>
> Django current application loading code is inflexible in many cases. There
> exists a need for a number of extra features to be added to the way
> applications are handled such as:
>
> * The ability to internationalise application names.
> * The ability to customise application names similar to ``verbose_name`` in
> ``model._meta``
> * Deploy the same application multiple times in a single project.
> * Deploy two different applications with the same name.
> * Manage settings within applications.
>
> Method
> ---
>
> I don't intend to solve the third dot point of the previous list - it seems
> like a difficult problem, possibly to be tackled post-GSoC. What I do intend
> to do is to move towards 'application classes' more akin to
> ``django.contrib.admin``.
>
> I plan to introduce a new file into the project structure - ``apps.py`` which
> will be used to register installed applications with the ``AppCache``. I will
> also move to introduce a new setting in ``settings.py``, ``APP_LOADER`` which
> will be a string containing a dotted path to an application loading class.
> This
> will serve to facilitate a move from ``settings.INSTALLED_APPS`` to the new
> ``apps.py``. ``APP_LOADER`` will take one of two values:
> 'django.core.apps.loader.legacy', which will use ``settings.INSTALLED_APPS``
> or
> 'django.core.apps.loader.registry', which would use the new registry-based
> approach. Only applications registered in ``apps.py`` in the project directory
> will be registered with the ``AppCache``. The default APP_LOADER setting for
> 1.3 will be 'django.core.apps.loader.legacy'.
>
> ``apps.py`` will contain purely application-related setup. This should be
> handy
> to manage application settings and to ensure ``settings.py`` doesn't get too
> full of application-specific settings. This will allow us to register an app
> in one of three ways something like:
>
> .. sourcecode:: python
>
> from django.core import apps
> from blog import BlogApplication
>
> class MyBlogApplication(BlogApplication):
> api_key = 'testing'
>
> another_blog = BlogApplication(label='blog2',
> verbose_name=_('Another blog'),
> api_key='anothertest')
>
> # an application class.
> apps.register(MyBlogApplication, label='blog', verbose_name=_('My blog'))
> # an application instance.
> apps.register(another_blog)
> # old-style application.
> apps.register('django.contrib.auth')
>
> This could allow us not to touch ``settings.INSTALLED_APPS`` at all. If people
> need to iterate over ``settings.INSTALLED_APPS``, a move to ``get_apps`` will
> be recommended.
>
> The ``Application`` class will be very similar to ``django.contrib.admin``'s
> ``AdminSite`` and will hopefully simplify application settings. If you write
> views and urls directly on the ``Application`` class itself, instead of an
> ``from django.conf import settings`` and subsequent ``settings.API_KEY``, you
> could just reference ``self.api_key`` for instance. This wouldn't be the
> required, just an optional extra.
>
> In addition to providing application settings an ``Application`` will be
> required to explicitly define their models. These models could live outside
> the traditional application structure, provided they are included in the
> ``models`` list on ``Application``. The current structure promotes strictly
> requiring models for an application to reside in that application's
> ``models.py``. I wouldn't be promoting changing the current status quo with
> regard to this, but it would be possible to deviate from it and include models
> defined outside of the applications ``models.py`` on an ``Application``.
>
> An application class would have other benefits, as evidenced by a short look
> at
> ``django.contrib.admin.AdminSite``. It could concievably be used to define a
> set of 'base' add/edit/delete views for each model on an ``Application``. It
> could even be used to set up an application's urls. This could make an
> ``Application`` class unwieldy and large however, so it won't be forced and
> people can continue using views.py and urls.py as before.
>
> Model classes will get
Re: Application Loading
On Apr 9, 2:09 am, Nick Sandford wrote: > It's a great place to start, and gives a good idea about where to look for > problems. I'm not sure about get_installed_app_paths, rather than modifying > settings.INSTALLED_APPS to contain a tuple of strings pointing to all > application paths. If we end up using get_installed_app_paths it would break > all of those using for app in settings.INSTALLED_APPS in third party code. You're right: get_installed_app_paths needs to be replaced with e.g. get_installed_apps() to get the app instances, while INSTALLED_APPS remains a list of path strings. That will avoid backward compatibility problems with 3rd-party code. > You've taken the route of assuming applications won't know their models > until runtime, I've expected that people explicitly assign models to apps. > Not exactly, it was more a case of keeping the changes as small as possible, at the time of the original patch. It's certainly better to have models explicitly associated with apps - it would make some things easier, especially in a multi-db setup. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
GSoC Proposal: Web Unifying Markup (templating) Language, or WUML
Proposal Title
--
Web Unifying Markup (templating) Language, or WUML
Background
--
As a freelance developer having built entire platforms of Django, I
find the most time consuming part of web development to be the front
end development. Essentially, Javascript, HTML, and CSS. Of course
there are tools that might help. For example, SASS for CSS, JQuery for
Javascript, and recently, an exciting new library known as the SHPAML
surfaced that is pretty similar to HAML, but more pythonic, and
definitely more transparent. (http://shpaml.webfactional.com/)
However, what does not change is that these components are still
distinct. Everytime I have to modify a property of an element, I'd
still have to go to my CSS file, or SASS file. Same goes to JS, or
HTML.
But thats not all, if we were to use SHPAML and SASS, every time we
have to test the interface, we have to manually compile, or have a 3rd
party script to compile all the relevant files back into its HTML/CSS/
JS components. And that means before every F5, we have to run a mini
script. It gets annoying after a while.
This is when I hope WUML can come in.
Essentially, I hope to have a unified templating language that will
coexists with template tags of Django, whereby I can streamline my
development into a pythonic element-centric experience. But its better
to demonstrate with examples.
Examples
html
body
div#header
| this is some verbose text inside this div
+ background-color: {{ backgroundcolor }}
+ pointer: cursor
^ alert("true");
^ this.hide();
will be compiled into
> A HTML FILE
this is some verbose text inside this div
> A CSS FILE
html body div#header {
background-color: #FFF;
pointer: cursor;
}
> A JS FILE (using JQuery)
$(document).ready({function(){
$("div#header").live("click",function() {
alert("true");
this.hide(); //hide here has to be defined in another js file.
});
});
Arguments against WUML (or why it is helpful)
-
Some people might argue that this defeats the purpose of abstraction
of different components into its respective components. JS/HTML/CSS
existed distinctly for a reason. While I agree strongly with this
point, but I hope to point out that WUML is not replacing any of those
components.
Here are some reasons why WUML might be interesting:
1) A javascript coder, a HTML writer, and a CSS scriptor, can still
work on their individual files, no one is stopping them. What WUML
does is, it provides a shorthand in an element-centric manner to
various of these components in a pythonic sense. The 3 different
people can now instead of working on 3 different files, they can work
on the same code, and view distinctly what are the relevant details to
each element they should take note of. Besides, with the state of the
art revision control any decent programmer should use these days,
merging will make this much easier.
2) WUML is s much shorter.
3) WUML is pythonic, at least syntatically-wise.
4) WUML will greatly boost any frontend developer's lead time.
Implementation Details
--
In a gist, I will basically be merging the best components of JS/CSS/
HTML, and providing a markup syntax for them in a pythonic fashion.
Details:
- Architecture: The WUML layer will live atop the 2 main components,
namely SASS and SHPAML. Hopefully, we can use the python-sass library,
instead of its generic binary backend for compilation. This way, the
WUML layer will interface the 2 pythonic layer, and provide a whole
new markup language. The template compiler which I hope will be worked
on this GSoC, will live atop WUML layer, and hence compile the WUML
compliant template everytime a change is made.
- Compilation: I would hope to work with a co-developer who would be
working on template compilation, and provide an extension of
compilation of WUML into their relevant separate components. This way,
compilation can take place dynamically and when needed, instead of
manually.
Some Notes
--
- Old templates will still be compliant.
- Using SHPAML's principle, all code can still be hardcoded. That
means one would be able to mix in generic HTML code, or CSS styles
into the SHPAML, and only use WUML when needed. And hence, non
intrusive.
Deliverables
At the end of the summer, I hope to deliver an entire markup language
that will be compilable using Django's default templating language, by
which I hope it'll help bolster various of Django's design
philosophies:
- Less code.
- Quick development.
- DRY.
- Discourage redundancy
- Be decoupled from HTML
- Assume designer's competency
- Extensibility
Timeline
First half of GSoC: Deliver a working library of WUML that will
compile code into its seperate components
Second half of GSoC: Work with fellow student in synergising WUML into
the template compiler.
--
You received this message because you are subscrib
Re: Porting _django_bash_completion to Windows Powershell
For the two Django devs on Windows out there, you might want to use this: http://bitbucket.org/guillermooo/powershell-utilities-for-django-developers/ Tests, docs, etc. to come soon. -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Application to update the Test Suite
I've revised the proposal, but am inlining more complete answers. > If someone needs to fix a bug, they will essentially need to > write the test twice; once for the old framework, and once for the > new. Do you have any suggestions on what we should be doing to > mitigate this complexity? Unfortunately some mismatch seems inevitable for major changes like the proposed one. I will include comments indicating where corresponding test code in the previous branches can be found to help with this problem. If the test code is a completely new chunk of code, it will be written in the unittest style which should not be too hard to add to the old test suite. Many of the bits of code that are currently doctests are fairly well vetted by now, and so should hopefully have few changes necessary for backports. If it's just a minor couple line testing change, that shouldn't be too hard to make into a doctest if that's more appropriate. Better ideas are, of course welcome. > We have a buildbot currently in development; if you're have the > resources to contribute, it would be good if these could be integrated > permanently into the build farm. That might be possible. I was intending to pull up vmware instances on my development machine, but we should discuss how best I could contribute to that. > The #python IRC channel needs to think about the problem a little more. I figured that was probably the case, given that you included that aspect in the project suggestion... > While this project is *very* appealing from a project point of view -- > we get the benefit of individually failing and isolated tests -- but > the risk of test runtime blowouts is huge. Any suggestions on how to > tackle this problem (and speed up test runs in general) would be > almost as valuable as the doctest->unittest conversion itself. This is a bit of a sticky problem. I'll be thinking about it between now and when the project (potentially) starts, but my first thoughts are as follows: We know that the existing doctests work when run sequentially. There may be bugs that are revealed as they're run individually, but to a first order, we know that some tests can safely be run sequentially without the setup/teardown each time. Since we're worried that this process is going to greatly inflate runtime, why not investigate options that let us avoid it where appropriate? Specifically, why not structure things such that we can lump tests together into groups that can safely be run sequentially, but also allow a "full" test run that runs each tested item individually as is currently the case for unittests? There are a couple problems with this approach. The first is that there are now 2 different ways to "pass the tests" - the quick run and the full run. Developers will still have to run the "full" run sometimes, negating at least some time saved. On the other hand, the proposed "quick" run doesn't provide LESS coverage than the existing doctests, so it might be good enough, particularly for tests which are currently in the regression test category. Unfortunately, it's probably not broadly appropriate for a majority of existing unit tests. We could make the "quick" option the standard for unit tests, with the "full" option reserved for assistance in bug hunting when there's an identified error. It needs more concerted thought, but I'd be interested in feedback. Maybe it's a terrible idea. This subject should probably have a dedicated thread here for better exposure and more specific discussion. Hopefully others have good ideas. In any case, I should do some profiling work as I'm doing the conversion to identify particularly inefficient areas and clean them up. This is part of the advantage of doing it by hand rather than as an automated process - another set of eyes to look for problems. I reworked the proposal timeline slightly to move a week from the end to the beginning to implement major changes (such as the above idea) to improve performance before I get deeply into the conversion. -Paul -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: GSoC Proposal: Web Unifying Markup (templating) Language, or WUML
I have some remarks/thoughts.
- How does this make code re-use better?
For instance, how do you use the same CSS styles along different files.
Or am I required to move the re-usable CSS/JS code in a normal file?
- Can you give an example on how you see the template tags being used in here?
>From what I've read in the other proposals, your timeline should be
divided into weeks.
It shows a better understanding of the problem.
-- Gert
Mobile: +32 498725202
Web: http://gert.selentic.net
On Fri, Apr 9, 2010 at 20:14, nubela wrote:
> Proposal Title
> --
>
> Web Unifying Markup (templating) Language, or WUML
>
> Background
> --
>
> As a freelance developer having built entire platforms of Django, I
> find the most time consuming part of web development to be the front
> end development. Essentially, Javascript, HTML, and CSS. Of course
> there are tools that might help. For example, SASS for CSS, JQuery for
> Javascript, and recently, an exciting new library known as the SHPAML
> surfaced that is pretty similar to HAML, but more pythonic, and
> definitely more transparent. (http://shpaml.webfactional.com/)
>
> However, what does not change is that these components are still
> distinct. Everytime I have to modify a property of an element, I'd
> still have to go to my CSS file, or SASS file. Same goes to JS, or
> HTML.
>
> But thats not all, if we were to use SHPAML and SASS, every time we
> have to test the interface, we have to manually compile, or have a 3rd
> party script to compile all the relevant files back into its HTML/CSS/
> JS components. And that means before every F5, we have to run a mini
> script. It gets annoying after a while.
>
> This is when I hope WUML can come in.
>
> Essentially, I hope to have a unified templating language that will
> coexists with template tags of Django, whereby I can streamline my
> development into a pythonic element-centric experience. But its better
> to demonstrate with examples.
>
> Examples
>
>
> html
> body
> div#header
> | this is some verbose text inside this div
> + background-color: {{ backgroundcolor }}
> + pointer: cursor
> ^ alert("true");
> ^ this.hide();
>
> will be compiled into
>
>> A HTML FILE
>
>
>
> this is some verbose text inside this div
>
>
>
>> A CSS FILE
>
> html body div#header {
> background-color: #FFF;
> pointer: cursor;
> }
>
>
>> A JS FILE (using JQuery)
>
> $(document).ready({function(){
> $("div#header").live("click",function() {
> alert("true");
> this.hide(); //hide here has to be defined in another js file.
> });
> });
>
>
> Arguments against WUML (or why it is helpful)
> -
>
> Some people might argue that this defeats the purpose of abstraction
> of different components into its respective components. JS/HTML/CSS
> existed distinctly for a reason. While I agree strongly with this
> point, but I hope to point out that WUML is not replacing any of those
> components.
>
> Here are some reasons why WUML might be interesting:
>
> 1) A javascript coder, a HTML writer, and a CSS scriptor, can still
> work on their individual files, no one is stopping them. What WUML
> does is, it provides a shorthand in an element-centric manner to
> various of these components in a pythonic sense. The 3 different
> people can now instead of working on 3 different files, they can work
> on the same code, and view distinctly what are the relevant details to
> each element they should take note of. Besides, with the state of the
> art revision control any decent programmer should use these days,
> merging will make this much easier.
>
> 2) WUML is s much shorter.
>
> 3) WUML is pythonic, at least syntatically-wise.
>
> 4) WUML will greatly boost any frontend developer's lead time.
>
> Implementation Details
> --
>
> In a gist, I will basically be merging the best components of JS/CSS/
> HTML, and providing a markup syntax for them in a pythonic fashion.
>
> Details:
>
> - Architecture: The WUML layer will live atop the 2 main components,
> namely SASS and SHPAML. Hopefully, we can use the python-sass library,
> instead of its generic binary backend for compilation. This way, the
> WUML layer will interface the 2 pythonic layer, and provide a whole
> new markup language. The template compiler which I hope will be worked
> on this GSoC, will live atop WUML layer, and hence compile the WUML
> compliant template everytime a change is made.
>
> - Compilation: I would hope to work with a co-developer who would be
> working on template compilation, and provide an extension of
> compilation of WUML into their relevant separate components. This way,
> compilation can take place dynamically and when needed, instead of
> manually.
>
> Some Notes
> --
>
> - Old templates will still be compliant.
>
> - Using SHPAML's principle, all code can still be hardcoded. That
> means one would be able to mix in generic
Re: Application to update the Test Suite
Maybe it's an overly simplistic question, but: what makes the tests slow currently? It's not simply the volume of them. It's more than possible for Python to race through hundreds of tests per second under the right conditions. Some of Django's test modules obviously zip along, but others seem to take a rather long time to complete individual tests. Has anyone identified which tests are the slowest and what is making them so slow? Is it the length of the tests? The parsing? The fixtures that are loaded? The data generated in the test itself? Without knowing what gives the current test suite its performance characteristics aren't we all just guessing at how a rewrite of it will compare? Perhaps a good test would be to take one test module and convert it from doc tests to unit tests and profile the before and after versions to see how they compare. While the micro-scale may not ultimately compare to the macro, it would at least be a starting benchmark. I don't have any answers here, but that's what occurs to me about performance... All the best, - Gabriel On Apr 9, 12:51 pm, Paul McMillan wrote: > I've revised the proposal, but am inlining more complete answers. > > > If someone needs to fix a bug, they will essentially need to > > write the test twice; once for the old framework, and once for the > > new. Do you have any suggestions on what we should be doing to > > mitigate this complexity? > > Unfortunately some mismatch seems inevitable for major changes like > the proposed one. I will include comments indicating where > corresponding test code in the previous branches can be found to help > with this problem. If the test code is a completely new chunk of code, > it will be written in the unittest style which should not be too hard > to add to the old test suite. Many of the bits of code that are > currently doctests are fairly well vetted by now, and so should > hopefully have few changes necessary for backports. If it's just a > minor couple line testing change, that shouldn't be too hard to make > into a doctest if that's more appropriate. Better ideas are, of course > welcome. > > > We have a buildbot currently in development; if you're have the > > resources to contribute, it would be good if these could be integrated > > permanently into the build farm. > > That might be possible. I was intending to pull up vmware instances on > my development machine, but we should discuss how best I could > contribute to that. > > > The #python IRC channel needs to think about the problem a little more. > > I figured that was probably the case, given that you included that > aspect in the project suggestion... > > > While this project is *very* appealing from a project point of view -- > > we get the benefit of individually failing and isolated tests -- but > > the risk of test runtime blowouts is huge. Any suggestions on how to > > tackle this problem (and speed up test runs in general) would be > > almost as valuable as the doctest->unittest conversion itself. > > This is a bit of a sticky problem. I'll be thinking about it between > now and when the project (potentially) starts, but my first thoughts > are as follows: > > We know that the existing doctests work when run sequentially. There > may be bugs that are revealed as they're run individually, but to a > first order, we know that some tests can safely be run sequentially > without the setup/teardown each time. Since we're worried that this > process is going to greatly inflate runtime, why not investigate > options that let us avoid it where appropriate? Specifically, why not > structure things such that we can lump tests together into groups that > can safely be run sequentially, but also allow a "full" test run that > runs each tested item individually as is currently the case for > unittests? > > There are a couple problems with this approach. The first is that > there are now 2 different ways to "pass the tests" - the quick run and > the full run. Developers will still have to run the "full" run > sometimes, negating at least some time saved. On the other hand, the > proposed "quick" run doesn't provide LESS coverage than the existing > doctests, so it might be good enough, particularly for tests which are > currently in the regression test category. Unfortunately, it's > probably not broadly appropriate for a majority of existing unit > tests. We could make the "quick" option the standard for unit tests, > with the "full" option reserved for assistance in bug hunting when > there's an identified error. > > It needs more concerted thought, but I'd be interested in feedback. > Maybe it's a terrible idea. This subject should probably have a > dedicated thread here for better exposure and more specific > discussion. Hopefully others have good ideas. > > In any case, I should do some profiling work as I'm doing the > conversion to identify particularly inefficient areas and clean them > up. This is part of the advantage of doing it by hand ra
Re: GSoC Proposal: Web Unifying Markup (templating) Language, or WUML
To be honest, my biggest question is "is this really about Django?"
It seems like this proposal could have gone to anybody, or it could be
a standalone app.
I also would like to know if you have examples of other successful
implementations of such a system that incorporate HTML, CSS, *and* JS
into one. Striking out into this territory and finding it's a much
larger job than expected just means the code branch is more likely to
die.
I'm particularly concerned about your discussion of javascript being
included. Though Django has chosen jQuery for its admin there is no
such requirement for the rest of any Django site's codebase. Writing a
meta-language for javascript that is framework-agnostic (or framework-
independent) is going to be extremely hard.
You talk about extensibility, but don't give any examples of how this
system would *actually* be extended.
How do you propose for your parser to determine the difference between
this new WUML markup and non-escaped similar-looking markup in legacy
templates? Backwards compatibility is not a simple issue here.
Adding dependencies on more external Python libraries also doesn't sit
well with me. I don't want to have to install SASS and SHPAML
libraries, nor do I want to be reliant upon the development directions
of those projects.
This proposal is certainly ambitious. I'll give you that.
All the best,
- Gabriel
On Apr 9, 11:14 am, nubela wrote:
> Proposal Title
> --
>
> Web Unifying Markup (templating) Language, or WUML
>
> Background
> --
>
> As a freelance developer having built entire platforms of Django, I
> find the most time consuming part of web development to be the front
> end development. Essentially, Javascript, HTML, and CSS. Of course
> there are tools that might help. For example, SASS for CSS, JQuery for
> Javascript, and recently, an exciting new library known as the SHPAML
> surfaced that is pretty similar to HAML, but more pythonic, and
> definitely more transparent. (http://shpaml.webfactional.com/)
>
> However, what does not change is that these components are still
> distinct. Everytime I have to modify a property of an element, I'd
> still have to go to my CSS file, or SASS file. Same goes to JS, or
> HTML.
>
> But thats not all, if we were to use SHPAML and SASS, every time we
> have to test the interface, we have to manually compile, or have a 3rd
> party script to compile all the relevant files back into its HTML/CSS/
> JS components. And that means before every F5, we have to run a mini
> script. It gets annoying after a while.
>
> This is when I hope WUML can come in.
>
> Essentially, I hope to have a unified templating language that will
> coexists with template tags of Django, whereby I can streamline my
> development into a pythonic element-centric experience. But its better
> to demonstrate with examples.
>
> Examples
>
>
> html
> body
> div#header
> | this is some verbose text inside this div
> + background-color: {{ backgroundcolor }}
> + pointer: cursor
> ^ alert("true");
> ^ this.hide();
>
> will be compiled into
>
> > A HTML FILE
>
>
>
> this is some verbose text inside this div
>
>
>
> > A CSS FILE
>
> html body div#header {
> background-color: #FFF;
> pointer: cursor;
>
> }
> > A JS FILE (using JQuery)
>
> $(document).ready({function(){
> $("div#header").live("click",function() {
> alert("true");
> this.hide(); //hide here has to be defined in another js file.
> });
>
> });
>
> Arguments against WUML (or why it is helpful)
> -
>
> Some people might argue that this defeats the purpose of abstraction
> of different components into its respective components. JS/HTML/CSS
> existed distinctly for a reason. While I agree strongly with this
> point, but I hope to point out that WUML is not replacing any of those
> components.
>
> Here are some reasons why WUML might be interesting:
>
> 1) A javascript coder, a HTML writer, and a CSS scriptor, can still
> work on their individual files, no one is stopping them. What WUML
> does is, it provides a shorthand in an element-centric manner to
> various of these components in a pythonic sense. The 3 different
> people can now instead of working on 3 different files, they can work
> on the same code, and view distinctly what are the relevant details to
> each element they should take note of. Besides, with the state of the
> art revision control any decent programmer should use these days,
> merging will make this much easier.
>
> 2) WUML is s much shorter.
>
> 3) WUML is pythonic, at least syntatically-wise.
>
> 4) WUML will greatly boost any frontend developer's lead time.
>
> Implementation Details
> --
>
> In a gist, I will basically be merging the best components of JS/CSS/
> HTML, and providing a markup syntax for them in a pythonic fashion.
>
> Details:
>
> - Architecture: The WUML layer will live atop the 2 main com
Re: GSoC Proposal: Web Unifying Markup (templating) Language, or WUML
On Sat, Apr 10, 2010 at 10:10 AM, Gabriel Hurley wrote: > To be honest, my biggest question is "is this really about Django?" This, by itself, is the reason that this proposal will be rejected. Django is a server side framework. We have deliberately avoided making any pronouncements on the right Javascript framework, or AJAX library, or CSS framework, or any other tool that can be used on the client side. Regardless of the merits of the proposal itself, it simply isn't within the problem domain that Django is trying to solve. Yours, Russ Magee %-) -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Application to update the Test Suite
On Fri, Apr 9, 2010 at 5:29 PM, Gabriel Hurley wrote: > Maybe it's an overly simplistic question, but: what makes the tests > slow currently? It's not simply the volume of them. It's more than > possible for Python to race through hundreds of tests per second under > the right conditions. Tests are slow largely because you have to load fixtures into the database and roll them back for every test case. This is required to guarantee a consistent initial state for each test. Speeding tests up would be great, but a lot of work has already been done in this area (e.g., tests are performed in a transaction that's rolled back in teardown, if possible, since that's faster then dropping and reloading each table) so I'm not sure how much low-hanging fruit is left. Keep in mind that it's better to have slow tests that do the right thing than fast ones that don't. Of course fast tests that do the right thing would be ideal. Mike -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Application to update the Test Suite
On Sat, Apr 10, 2010 at 8:29 AM, Gabriel Hurley wrote: > Maybe it's an overly simplistic question, but: what makes the tests > slow currently? It's not simply the volume of them. It's more than > possible for Python to race through hundreds of tests per second under > the right conditions. > > Some of Django's test modules obviously zip along, but others seem to > take a rather long time to complete individual tests. Has anyone > identified which tests are the slowest and what is making them so > slow? Is it the length of the tests? The parsing? The fixtures that > are loaded? The data generated in the test itself? Yes. :-) There are multiple factors at play here. The first is obvious if you run the test suite right now using different backends: SQLite is much faster than Postgres, which is much faster than MySQL/MyISAM. This is because SQLite uses an in-memory store, so it isn't disk bound; Postgres is disk bound, but is able to use transactions to optimize test setup and teardown; MySQL is also disk bound, but doesn't support transactions, so there are a lot of "CREATE DATABASE; do one thing; DESTROY DATABASE" calls in the lifepan of the test suite. The move to using transaction based tests in Django 1.1 sped things up a lot (for Postgres and SQLite, anyway); but there is still room to improve. The biggest single problem that I am aware of is that even though tests are composed in TestCases that have common setup/teardown routines, each test is handled separately. As a result, if you have a test case with 10 tests and a complex fixture setup, the fixtures are read from disk 10 times, loaded into the database 10 times, and rolled back out of the database 10 times. Ideally, this should only happen once, but Python's unittest doesn't (currently) provide easy hooks to do TestCase-wide setup. So - a couple of options that are worth looking at: * Reduce the amount of parsing that is needed. Keep the pre-parsed results of loading fixtures in memory, so that fixtures don't need to be repeatedly parsed. This is logged as #9449. * Reduce the amount of loading/unloading that is needed. One way to do this is to look at integrating support for unittest2 [1]. This has been logged as #12991. In addition to adding lots of nifty features like test skipping and better assertion diffs, unittest2 adds support for TestCase-wide (and module wide) setups. [1] http://www.voidspace.org.uk/python/articles/unittest2.shtml > Without knowing what gives the current test suite its performance > characteristics aren't we all just guessing at how a rewrite of it > will compare? > > Perhaps a good test would be to take one test module and convert it > from doc tests to unit tests and profile the before and after versions > to see how they compare. While the micro-scale may not ultimately > compare to the macro, it would at least be a starting benchmark. Also a good idea. Some quality time with the profiler would almost certainly reveal extra room for optimization. Yours, Russ Magee %-) -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Application to update the Test Suite
On Sat, Apr 10, 2010 at 2:21 AM, Russell Keith-Magee wrote: > On Sat, Apr 10, 2010 at 8:29 AM, Gabriel Hurley wrote: >> Maybe it's an overly simplistic question, but: what makes the tests >> slow currently? It's not simply the volume of them. It's more than >> possible for Python to race through hundreds of tests per second under >> the right conditions. >> >> Some of Django's test modules obviously zip along, but others seem to >> take a rather long time to complete individual tests. Has anyone >> identified which tests are the slowest and what is making them so >> slow? Is it the length of the tests? The parsing? The fixtures that >> are loaded? The data generated in the test itself? > > Yes. :-) > > There are multiple factors at play here. The first is obvious if you > run the test suite right now using different backends: SQLite is much > faster than Postgres, which is much faster than MySQL/MyISAM. This is > because SQLite uses an in-memory store, so it isn't disk bound; > Postgres is disk bound, but is able to use transactions to optimize > test setup and teardown; MySQL is also disk bound, but doesn't support > transactions, so there are a lot of "CREATE DATABASE; do one thing; > DESTROY DATABASE" calls in the lifepan of the test suite. > > The move to using transaction based tests in Django 1.1 sped things up > a lot (for Postgres and SQLite, anyway); but there is still room to > improve. The biggest single problem that I am aware of is that even > though tests are composed in TestCases that have common setup/teardown > routines, each test is handled separately. As a result, if you have a > test case with 10 tests and a complex fixture setup, the fixtures are > read from disk 10 times, loaded into the database 10 times, and rolled > back out of the database 10 times. Ideally, this should only happen > once, but Python's unittest doesn't (currently) provide easy hooks to > do TestCase-wide setup. > Spoke too late ;) Python 2.7 SVN now has setUpClass and tearDownClass. > So - a couple of options that are worth looking at: > > * Reduce the amount of parsing that is needed. Keep the pre-parsed > results of loading fixtures in memory, so that fixtures don't need to > be repeatedly parsed. This is logged as #9449. > > * Reduce the amount of loading/unloading that is needed. One way to > do this is to look at integrating support for unittest2 [1]. This has > been logged as #12991. In addition to adding lots of nifty features > like test skipping and better assertion diffs, unittest2 adds support > for TestCase-wide (and module wide) setups. > > [1] http://www.voidspace.org.uk/python/articles/unittest2.shtml > >> Without knowing what gives the current test suite its performance >> characteristics aren't we all just guessing at how a rewrite of it >> will compare? >> >> Perhaps a good test would be to take one test module and convert it >> from doc tests to unit tests and profile the before and after versions >> to see how they compare. While the micro-scale may not ultimately >> compare to the macro, it would at least be a starting benchmark. > > Also a good idea. Some quality time with the profiler would almost > certainly reveal extra room for optimization. > > Yours, > Russ Magee %-) > > -- > You received this message because you are subscribed to the Google Groups > "Django developers" group. > To post to this group, send email to django-develop...@googlegroups.com. > To unsubscribe from this group, send email to > django-developers+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/django-developers?hl=en. > > Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Voltaire "The people's good is the highest law." -- Cicero "Code can always be simpler than you think, but never as simple as you want" -- Me -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Application to update the Test Suite
On Sat, Apr 10, 2010 at 2:25 PM, Alex Gaynor wrote: > On Sat, Apr 10, 2010 at 2:21 AM, Russell Keith-Magee > wrote: >> On Sat, Apr 10, 2010 at 8:29 AM, Gabriel Hurley wrote: >>> Maybe it's an overly simplistic question, but: what makes the tests >>> slow currently? It's not simply the volume of them. It's more than >>> possible for Python to race through hundreds of tests per second under >>> the right conditions. >>> >>> Some of Django's test modules obviously zip along, but others seem to >>> take a rather long time to complete individual tests. Has anyone >>> identified which tests are the slowest and what is making them so >>> slow? Is it the length of the tests? The parsing? The fixtures that >>> are loaded? The data generated in the test itself? >> >> Yes. :-) >> >> There are multiple factors at play here. The first is obvious if you >> run the test suite right now using different backends: SQLite is much >> faster than Postgres, which is much faster than MySQL/MyISAM. This is >> because SQLite uses an in-memory store, so it isn't disk bound; >> Postgres is disk bound, but is able to use transactions to optimize >> test setup and teardown; MySQL is also disk bound, but doesn't support >> transactions, so there are a lot of "CREATE DATABASE; do one thing; >> DESTROY DATABASE" calls in the lifepan of the test suite. >> >> The move to using transaction based tests in Django 1.1 sped things up >> a lot (for Postgres and SQLite, anyway); but there is still room to >> improve. The biggest single problem that I am aware of is that even >> though tests are composed in TestCases that have common setup/teardown >> routines, each test is handled separately. As a result, if you have a >> test case with 10 tests and a complex fixture setup, the fixtures are >> read from disk 10 times, loaded into the database 10 times, and rolled >> back out of the database 10 times. Ideally, this should only happen >> once, but Python's unittest doesn't (currently) provide easy hooks to >> do TestCase-wide setup. >> > > Spoke too late ;) Python 2.7 SVN now has setUpClass and tearDownClass. Yes, it does, but we still need to support Python 2.4, so we can't just start using those features -- we need to have a migration strategy. However, if we determine that unittest2 will give us a major performance boost, I have no particular objection to shipping unittest2 with Django as a support mechanism for Python 2.4-6 (in the same way that we ship a copy of doctest.py). Yours, Russ Magee %-) -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
