Re: What The Enterprise wants from Django
2010/1/20 Jacob Kaplan-Moss : > On Wed, Jan 20, 2010 at 12:31 PM, Luke Plant wrote: >> I don't understand how avoiding the settings.py mechanism will produce >> *more* flexibility. > > The problem -- at least as I see it -- is that of a intertwingulment > of "application" settings with "ops" settings. > > As organizations get larger, roles stratify more. Thus you end up with > a pretty big gulf between the ops team and the application developers. > In these environments, application developers probably shouldn't be > messing with things like the `DATABASE_*` settings -- in some > organizations its even a breach of policy for the developers to *know* > the database password! In the past I've used code like the following in settings files: # we're going to allow overriding of settings via a yml file. # This makes live nicer for anyone managing the box # and means settings can be overloaded without redeploying the site SETTINGS_OVERIDE = "/etc/$(project).yml" try: file = open(SETTINGS_OVERIDE) for key, value in yaml.load(file).items(): globals()[key]=value except: # we don't always have the file around or need the setting # defined so best to be quite if things go wrong pass You can also make this more limited, by allowing only the overriding certain settings. You can then use tools like puppet on the ops side to push config files in to /etc (or wherever suites you system). Example here http://morethanseven.net/2009/02/15/let-you-sys-admin-override-your-django-settings/ This pattern is also used in my project templates paster stuff: http://github.com/garethr/django-project-templates and http://pypi.python.org/pypi/django-project-templates/ > > OTOH, ops types don't know what the heck to do with `INSTALLED_APPS`... > > And then there's middleware: some middleware (caching, etags) might > need to be controlled by ops; some by apps. > > Obviously you can *do* this with Django, and I'd argue it's not > particularly hard. Still, it's something that bigger orgs have to > figure out themselves, and that can be a barrier to adoption. > > The enterprisey world solves this with standards like WAR and servers > like Glassfish: if I'm an application developer, all I need to do to > deploy is somehow produce a WAR. And if I'm a ops person, I just take > that WAR and slap it into Glassfish. It's rarely that easy in > practice, of course, but the execs making the tech decisions don't > know that. > > Jacob > > -- > You received this message because you are subscribed to the Google Groups > "Django developers" group. > To post to this group, send email to django-develop...@googlegroups.com. > To unsubscribe from this group, send email to > django-developers+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/django-developers?hl=en. > > > > -- Gareth Rushgrove Web Geek morethanseven.net garethrushgrove.com -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: What The Enterprise wants from Django
2010/1/21 Russell Keith-Magee : > > It strikes me that this is just one example of a whole class of "best > practice" stuff that we should document, but don't at present - like > project layout, automated deployment, continuous integration, > configuration management, etc. > > We've historically avoided having documentation on how to use Django > with external tools (except when we have to, as in the case of > mod_wsgi). A big part of the solution to the problem you've described > is to establish a vetted best practice, and documenting that practice > on our website as a community resource. I'd certainly be interested in contributing to this effort. I have lots of this information scattered about on my blog, internal company notes and presentations from DJUGL and the likes. I've avoided trying to get it on the wiki for the reasons you describe. What is the best way of going about documenting this so it can be vetted by the community and enshrined as somewhat official best practice - without the work in progress being used before it's been stamped in someway? > > The rest of the solution falls to the toolchain outside of Django > itself. It still needs to be solved, but it isn't a Django problem > specifically. One option here might be to take the Apache model, and > use the DSF to incubate tools that have benefits to Django. However, > that's part of a much bigger discussion. > > Yours, > Russ Magee %-) > > -- > You received this message because you are subscribed to the Google Groups > "Django developers" group. > To post to this group, send email to django-develop...@googlegroups.com. > To unsubscribe from this group, send email to > django-developers+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/django-developers?hl=en. > > > > -- Gareth Rushgrove Web Geek morethanseven.net garethrushgrove.com -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: TimeField for duration
On 01/21/2010 01:31 AM, Ian Kelly wrote: > 2010/1/20 Łukasz Rekucki : >> 2010/1/21 Jerome Leclanche : >>> Hi Lukasz, thanks for the wrap up >>> >>> Main issue is precision. I personally very, very often work with >>> milliseconds, and I can imagine many use cases where I'd need to work >>> with microseconds. How many of these backends do not at least support >>> millisecond precision? >> From my understanding of the docs PostgreSQL can do microseconds >> (6digits), Oracle - nanoseconds (9digits), SQLite3 seems to accept >> miliseconds in string representation "HH:MM:SS.SSS". >> >> And the worst case is MySQL: >> >> """A trailing .uu microseconds part of TIME values is allowed >> under the same conditions as for other temporal values, as described >> in Section 10.3.1, “The DATETIME, DATE, and TIMESTAMP Types”. This >> includes the property that any microseconds part is discarded from >> values stored into TIME columns.""" >> >> After seeing all this mess, i'm +1 on using BigIntField. Good work. > > In other words, all the same limitations that we already handle for > TimeFields. This looks to me like a green light to just use the > native datatypes. > > I see another big downside to the BigIntField approach: you can't add > TimeFields and IntervalFields at the database level (e.g. with F > expressions). I'm not sure how PostgreSQL or MySQL would handle this > if you tried it, but I know what Oracle would do: it would accept the > SQL as valid, it would interpret the IntervalField value as a decimal > number of /days/, and it would return a completely incorrect result. I personally agree with this. Native datatypes, if possible, are also cleaner than obscure integer mappings. The TIME and/or INTERVAL precision depends on the backend, that's all (although that may be confusing). And people who need to handle nanoseconds or so can just add custom numeric fields to their models. -- Olivier -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Feedback requested: Multi-db database routing
Hi Russell, Thanks, I'm rather happy too. It's exactly what we wanted! DB Routing API is even better than some fixed db routing format. I just thought about any model with GFK, like TagField from django-tagging -- with multidb, to be effectively reusable, it has to know in what DB to save and load, depending on its client ContentType. Since it's solution for the problems discussed in http://groups.google.com/group/django-developers/browse_frm/thread/7904c7da7cb0085f/d063d2086299635e, could you please think about one more thing discussed there: syncdb targeting to correct databases. Something like a db_router.get_all_databases_for_model(model) call. It was suggested in the thread. Another issue is bothering me: db_router.allow_relation doesn't convert in my head into any algorithm of its usage. Docs in this thread didn't help me either when it's called and what for. So I took a look into implementation, it looks like it only works as a validator: ...if not router.allow_relation(obj, self.instance): raise ValueError(...) But how one would suggest in which db to save object depending on the related object? I.e., let's talk about this example with GFK. Thanks! On Wed, Jan 20, 2010 at 10:06 PM, Russell Keith-Magee wrote: > Ok - so that's the patch. Any comments, queries, or criticisms are welcome. > > Yours, > Russ Magee %-) -- Best regards, Yuri V. Baburov, ICQ# 99934676, Skype: yuri.baburov, MSN: bu...@live.com -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Feedback requested: Multi-db database routing
On Thu, Jan 21, 2010 at 8:21 PM, Yuri Baburov wrote: > Hi Russell, > > Thanks, I'm rather happy too. > It's exactly what we wanted! > > DB Routing API is even better than some fixed db routing format. > I just thought about any model with GFK, like TagField from django-tagging -- > with multidb, to be effectively reusable, it has to know in what DB > to save and load, depending on its client ContentType. > > Since it's solution for the problems discussed in > http://groups.google.com/group/django-developers/browse_frm/thread/7904c7da7cb0085f/d063d2086299635e, > could you please think about one more thing discussed there: syncdb > targeting to correct databases. I completely acknowledge how such a setting could be useful for any sort of metaprogramming situation - i.e., in the case of an automated admin interface only showing the models that are known to be available, or in the case of syncdb. My hesitation in implementing something to address this problem is driven by pragmatism. Firstly, I haven't seen an proposal that includes an elegant way to express this database allocation information. All the formats that I've seen amount to reproducing either INSTALLED_APPS, or worse, having an analog of INSTALLED_APPS down to the level of individual models. I have no desire to introduce a requirement for an exceptionally complex data structure, especially for the single database case. Secondly, while I can see how this information could be useful, it doesn't change the fact that the information is just as easy to manage manually. Syncdb doesn't do anything special -- it just invokes sqlall and loaddata initial_data on your database. The admin can be completely configured to include (or not include) any object you want. I know this isn't ideal, but it does *work*. > Something like a db_router.get_all_databases_for_model(model) call. > It was suggested in the thread. Making this a programatic operation is an interesting idea, as I said when it was proposed. Putting this callable operation in the router avoids my previous hesitation, as it means we don't need to put callable code in the settings file. I have some questions regarding implementation, though: How do you see this interacting with the stack of routers? i.e., is it additive over all items in the stack? How do you handle the case of removing objects from a database? For example, if I want all models in contrib.auth to be on the 'users' database, I can encode that rule in my AuthRouter, However if the bucket case assigns all models to the default database, that undoes the good work that the AuthRouter has done. How do you encode the information of "yes, I've considered the default database as a candidate for auth.User, and I've rejected it"? > Another issue is bothering me: > db_router.allow_relation doesn't convert in my head into any algorithm > of its usage. > Docs in this thread didn't help me either when it's called and what for. > So I took a look into implementation, it looks like it only works as a > validator: > ...if not router.allow_relation(obj, self.instance): raise ValueError(...) > But how one would suggest in which db to save object depending on the > related object? You are correct - allow_relation() is purely a validation hook, not a database allocator. It is used to determine if mybook.author = john is a valid operation - in this example, allow_relation(mybook, john) would be invoked. On a single database situation, both objects are on the default database, so the relation is allowed. If you have mybook on database d1 and john on database d2, then the relation will be prevented, unless you define a router that allows that specific cross-database pair - for example a master/slave router would allow the relation if d1 and d2 are part of the same slave pool. > I.e., let's talk about this example with GFK. There's nothing specific about generic foreign keys here - a GFK is a relation between two objects, and they either need to be on the same database, or on two databases that a router agrees are compatible. allow_relation() is what does that check. Yours, Russ Magee %-) -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: TimeField for duration
On 01/21/2010 01:31 AM, Ian Kelly wrote:
> In other words, all the same limitations that we already handle for
> TimeFields. This looks to me like a green light to just use the
> native datatypes.
I wrote a simple DurationField for my own purpose, it is not meant to be
included in Django as-is, but could serve as a proof-of-concept. See attached
duration.py.
I'm using Django r12271 with MySQL and this DurationField is mapped to a TIME
column. I can retrieve records which contain a day time, that is: the TIME
column is less than 24h. But it fails when it is over 24h.
The following exception is raised:
ValueError: hour must be in 0..23
The traceback was not helpful, saying that it comes from:
File "/[...]/django/db/backends/mysql/base.py", line 86, in execute
return self.cursor.execute(query, args)
After some debugging, I found that this exception is raised by the
datetime.time() constructor, from django/db/backends/util.py line 62:
def typecast_time(s): # does NOT store time zone information
if not s: return None
hour, minutes, seconds = s.split(':')
if '.' in seconds: # check whether seconds have a fractional part
seconds, microseconds = seconds.split('.')
else:
microseconds = '0'
return datetime.time(int(hour), int(minutes), int(seconds),
int(float('.'+microseconds) * 100))
This typecast_time() method is used as a converter by MySQLdb, as explained in
django/db/backends/mysql/base.py:
# MySQLdb-1.2.1 returns TIME columns as timedelta -- they are more like
# timedelta in terms of actual behavior as they are signed and include days --
# and Django expects time, so we still need to override that. [...]
I would quite like to receive this timedelta object as an argument to
DurationField.to_python(). But as shown above, it is converted to a time object
at low-level.
So, this is a bit of a user question, but I how can I make my DurationField work
without actually modifying the Django source code?
--
Olivier
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
from django.db import models
import datetime
from django.utils.translation import ugettext_lazy as _
class Duration(datetime.timedelta):
def __str__(self):
hours = self.days * 24
minutes = self.seconds / 60
seconds = self.seconds % 60
return "%.2d:%.2d:%.2d" % (hours, minutes, seconds)
@staticmethod
def fromstr(str):
test = re.match('^([0-9]+)(?::([0-9]+)(?::([0-9]+))?)?$', str)
if test:
groups = test.groups()
return Duration(hours=int(groups[0]), minutes=int(groups[1]), seconds=int(groups[2]))
else:
raise ValueError("Malformed duration string: " + str)
def as_seconds(self):
return self.days * 24 * 3600 + self.seconds
# The following is based on Django TimeField
class DurationField(models.Field):
description = _("Duration")
__metaclass__ = models.SubfieldBase
default_error_messages = {
'invalid': _('Enter a valid duration in HH:MM[:ss[.uu]] format.'),
}
def get_internal_type(self):
return 'TimeField'
def to_python(self, value):
if value is None:
return None
if isinstance(value, datetime.time):
return Duration(hours=value.hour, minutes=value.minute, seconds=value.second)
if isinstance(value, datetime.datetime):
# Not usually a good idea to pass in a datetime here (it loses
# information), but this can be a side-effect of interacting with a
# database backend (e.g. Oracle), so we'll be accommodating.
return self.to_python(value.time())
try:
value = Duration.fromstr(value)
except ValueError:
raise exceptions.ValidationError(self.error_messages['invalid'])
def get_prep_value(self, value):
return self.to_python(value)
def get_db_prep_value(self, value, connection, prepared=False):
# Casts times into the format expected by the backend
return unicode(value)
def value_to_string(self, obj):
val = self._get_val_from_obj(obj)
if val is None:
data = ''
else:
data = unicode(val)
return data
def formfield(self, **kwargs):
defaults = {'form_class': forms.TimeField}
defaults.update(kwargs)
return super(DurationField, self).formfield(**defaults)
Re: Feedback requested: Multi-db database routing
Russell Keith-Magee wrote:
class MasterSlaveRouter(object):
def db_for_read(self, model, db=None):
# Point all read operations to a random slave
return random.choice(['slave1','slave2'])
def db_for_write(self, model, db=None):
# Point all write operations to the master
return 'master'
def allow_relation(self, obj1, obj2):
# Allow any relation between two objects in the db pool
db_list = ('master','slave1','slave2')
if obj1 in db_list and obj2 in db_list:
return True
return None
Then, in your settings file, add the following:
DATABASE_ROUTERS = ['path.to.AuthRouter', 'path.to.MasterSlaveRouter']
And you're set.
It's more of a side note.
I understand that this is only an example but examples like this one
might give an impression that Django now solves master-slave replication
out of the box. It's not.
The hard thing about replication is that switching between master and
slaves doesn't depend on models and doesn't depend on SQL operators. It
depends on some vague notion of "operation context": if we are in some
logical multi-step process of changing state then all writes *and* reads
should happen on a master or we won't have a consistent transaction.
That said, this routing looks to me like a convenient and may be even
necessary plumbing. It just doesn't solve the whole problem.
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
Re: What The Enterprise wants from Django
On Tuesday 19 January 2010 21:26:17 Jacob Kaplan-Moss wrote: > So there we are. This is very much a brain dump, and I don't really > expect any concrete action to result from it. However, I found some > really interesting stuff there, and I thought I'd share. Thanks for your clarification elsewhere in this thread. One question I have (for Jacob and anyone else with experience): I would have guessed that a big issue with Django from an enterprise perspective is its use of 'singletons'. Though we don't use that terminology, we have dozens of them in the code - every other setting we have implies the existence of a singleton, since the setting can have only one value. For example, EMAIL_BACKEND means that we have one email component singleton which is used everywhere. If you wanted one set of views to use a different email backend, you are out of luck. The 'admin' app was the first major component to allow multiple instances, but there are various other apps which might need this change in theory. How much is this an issue in practice? I imagine that for most sub-systems, most people actually want singletons, but for the cases you don't, it is a big pain. Personally I think that, while there has been an unhealthy proliferation of settings which we need to trim, Django has hit the sweet spot between easy set up and configuration on the one hand, and flexibility on the other. The alternative - turning everything into components with dependency injection everywhere - could be horrific for our sweet spot. One solution to this problem would be to have multiple Django instances hosting different parts of your URL space, allowing them to have multiple settings files. This has obvious limitations, but from the perspective of a large project, perhaps it's not too bad - I imagine that large sites are already used to carving up the URL space to be served by completely different technologies in some cases, at least for hosts running LAMP or similar. Regards, Luke -- "Procrastination: Hard work often pays off after time, but laziness always pays off now." (despair.com) Luke Plant || http://lukeplant.me.uk/ -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: What The Enterprise wants from Django
On Jan 21, 2010, at 9:58 AM, Luke Plant wrote: > On Tuesday 19 January 2010 21:26:17 Jacob Kaplan-Moss wrote: > >> So there we are. This is very much a brain dump, and I don't really >> expect any concrete action to result from it. However, I found some >> really interesting stuff there, and I thought I'd share. > > Thanks for your clarification elsewhere in this thread. > > One question I have (for Jacob and anyone else with experience): > > I would have guessed that a big issue with Django from an enterprise > perspective is its use of 'singletons'. Though we don't use that > terminology, we have dozens of them in the code - every other setting > we have implies the existence of a singleton, since the setting can > have only one value. For example, EMAIL_BACKEND means that we have > one email component singleton which is used everywhere. If you wanted > one set of views to use a different email backend, you are out of > luck. > > The 'admin' app was the first major component to allow multiple > instances, but there are various other apps which might need this > change in theory. > > How much is this an issue in practice? Speaking as a web developer using Django in a *very* enterprisey corporation, I will say this singleton issue has been a problem for me. As a self-inflicted example, I wrote an LDAP backend that looks up the necessary settings from settings.py. I need to rewrite it to store the settings in the db, because I need to be able to query multiple LDAP sources. I run into similar issues with lots of different apps. So yes, singletons are an issue for me. The difference between ops settings vs. app settings has also been a problem. The lack of established, vetted best practices (for deployment, layout, everything Russ mentioned) also has bitten me regularly. In part because I wind up spending a lot of time doing the research to figure out what I need to do, in part because the state of the art moves forward at a rapid pace. But, to put it in perspective, I'm the only person who even *can* deploy a Django app at my place of work. Which is not great, but I'm also a Linux sys-admin, so it is at least in my job description. But none of my fellow SAs have any clue what to do to get stuff working on one of our corporate RHEL 5 boxen, and no inclination to learn. Anyway, I'm glad to see this discussion taking place. Django indeed has a sweet spot, and it isn't really for enterprise installations -- though it is *very* good in that sweet spot. There are places where the early design decisions to favor convention over configuration impose a lack of flexibility in favor of simplicity. A definite trade-off. It makes it harder to use in places where flexibility and integration are required. Here's a big question, though: given that satisfying the enterprise crowd, of which I am an occasional part, will require additional "stuff" that may bring along with it additional complexity, making it potentially harder to get started for newbies, is that a direction that the project should head? Is it worth satisfying the enterprise crowd to increase adoption, at the cost of turning into the kind of configuration hell that J2EE servers exemplify? Or, more positively, is it possible to add the required enterprisey bits while still maintaining a certain level of simplicity? Note that Django is already headed towards greater complexity (multi-db, email backends), due to requirements from the current community. My two cents, appropriately adjusted for the current economic situation. ---Peter Herndon -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: What The Enterprise wants from Django
> I would have guessed that a big issue with Django from an enterprise > perspective is its use of 'singletons'. > > How much is this an issue in practice? Yes, that's been my experience. For complex deployments with tens or hundreds of apps, it is often the case that it is absolutely essential to have settings on a per-app basis. Unfortunately those settings very often clash. > It strikes me that this is just one example of a whole class of "best > practice" stuff that we should document, but don't at present - like > project layout, automated deployment, continuous integration, > configuration management, etc. It would help smaller companies I think, folks who need to deploy a couple of SOA nodes or a site with ten or twenty apps across a handful of servers. When it comes to huge deployments, it is so diverse. Best practice for Google isn't best practice for wisconsinhairstylists.com Case studies are helpful if folks can share them. But its a political minefield (or unhelpful pontification, depending on the care you take) to get into the business of canonizing deployment structures! -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: What The Enterprise wants from Django
On Thu, Jan 21, 2010 at 8:58 AM, Luke Plant wrote: > I would have guessed that a big issue with Django from an enterprise > perspective is its use of 'singletons'. [snip] > How much is this an issue in practice? Other than databases, it hasn't been much of an issue with most of the BigCos I've been involved with. That's because... > One solution to this problem would be to have multiple Django > instances hosting different parts of your URL space, allowing them to > have multiple settings files. Bingo. All of the larger Django installations I've worked with haven't been just one project -- they've been dozens (hundreds in one case) of little bits, often cobbled together by different teams and sometimes different companies (in-house vs. contractors vs. off-the-shelf products). Django's URL/view decoupling looks *really* nice in these cases -- Django apps, properly written, don't care *where* in the URL space they live -- so this means that Django fits in well in these heterogenous environments. And I've yet to encounter a larger company that's anything but wildly heterogeneous. Peter's question below about enterprisy complexity and the dangers of becoming J2EE are good ones; I'm glad they're part of the discussion. It's difficult to know where to draw the line between simplicity and flexibility. Right now we have exactly 90 documented settings, some of which point at classes or interfaces that themselves have more complexity. To me this already sounds way too high, but it's nothing compared to the configuration options available in other environments -- PHP has about 600 php.ini settings! -- so there's obviously a matter of taste here. It's also important to keep in mind that part of the reason Django's had (some) success in corporate environments is *because* of the simplicity, not despite it. I've heard, over and over again, that teams that have switched to Django have done so at least in part because of the simplicity and ease-of-use. So we should be careful about setting up a false dichotomy where us hippie open source freaks only value simplicity and build toys and those square suit types who love FactoryManagerFactories because CIO magazine tells them to. The real world, as always, is a lot more complex. Jacob -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: What The Enterprise wants from Django
On Thu, Jan 21, 2010 at 10:24 AM, sago wrote: > Case studies are helpful if folks can share them. Just a quick note that the DSF is working on gathering case studies; getting folks who have permission to share is indeed a tricky issue. Not much is set up yet, but I, at least, would be *thrilled* to help companies using Django put together case studies that could be shared publicly. Until we've got something more formal in place, consider it a standing offer -- drop me a line if you'd like to share a success story. Jacob -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: What The Enterprise wants from Django
I agree that managing settings gets to be a bit difficult in many environments - even non Fortune 1000 environments. One of the things we've played with in Satchmo is a database backed solution that allows you to define reasonable defaults - even override-able in the settings - but allow people to edit through the admin interface. Bruce has been working to split it out from Satchmo here - http://bitbucket.org/bkroeze/django-livesettings/overview/ I'm not saying this is the end all be all solution but it's something we've been using for a bit so it should be useful in informing the direction the community wants to take. I also have some ideas for how it should be improved or tweaked but in my opinion it's a nice start and it would be good to see what people think about using it in other apps. It's also consistent with the Django philosophy of building it outside of core and seeing if it reaches critical mass for the community. -Chris -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: What The Enterprise wants from Django
Hi! My previous experience was from Java J2EE world and as Jacob said we adopted Django because it really shines solving some complexity problems associated to J2EE world as MVC patter, templates, etc., but I agree that a common way to configure external applications would be nice. I'll also have been searching a good RIA to connect with Django. One of the frameworks I like best is Qooxdoo http://qooxdoo.org/, perhaps the best javascript opensource RIA frameworks I have seen. In order to connect with Qooxdoo just Json-RCP or xml-rcp is needed. Perhaps we'll need an special support for this in Django to mark a view as rcp as some external projects do as RCP4Django or other. I'd like to see Qooxdoo as the default Django RIA (not the default javascript framework) as Qooxdoo people uses also Python and the framework fits quite well in what I think is an enterprise RIA framework. I know that this could be against the Django policy of not to bless a javascript framework, but in my opinion if we want to shine at RIA you have to choose. We have created also some HTTP-XML based web services in Django in a fraction of time we needed to create them in Java. For that we are using Django without ORM, but we have found a great help to allow documenting the web service and create the test pages. Anyway, I'm glad to see Fortune companies are interested in Django :) -- Antoni Aloy López Blog: http://trespams.com Site: http://apsl.net -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Proposal: errors with URLs in them
One of my favourite errors in Python is this one: File "settings.py", line 2 SyntaxError: Non-ASCII character '\xce' in file settings.py on line 2, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details This is when you've declared a non-ASCII character without the encoding declared. In fact I don't even bother to look up the syntax to make it work, it's easier to put the text in, let it fail and then let python tell me the URL to the syntax. There are common errors that I see being made in putting together Django applications and I'd like to see if we can provide more helpful errors to the user. There are obviously lots of places we can't do this, but I think in some areas we can. One example from #django last night that i've seen a few times is leaving a comma out in INSTALLED_APPS so you have: INSTALLED_APPS = ( 'django.contrib.auth' 'django.contrib.contenttypes', ...) This leads to: Error: No module named authdjango.contrib.contenttypes Which is a pretty confusing error. Supposing this said: Error: No module named authdjango.contrib.contenttypes. See http://djangoproject.com/errors/32 for help. And that page gave lots of detail: This means Django can't find an app from installed apps. This could be because: - You've misspelled it - You've missed a comma in INSTALLED_APPS ... We've already got a page that seems similar to this at: http://code.djangoproject.com/wiki/BetterErrorMessages Ideally such errors would be part of the documentation. This is of course won't be possible for all errors or all parts of Django, but would work for some of the more common errors that crop up regularly. Would this be a course worth pursuing? -- Andy McKay, @clearwind http://clearwind.ca/djangoski -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Feedback requested: Multi-db database routing
On Wed, Jan 20, 2010 at 11:06 AM, Russell Keith-Magee wrote: > I've just uploaded a new patch improving the multi-db interface. This > patch introduces a way to easily configure the database selection > process, and enables you to assign a database selection mechanism to > models not under your control (e.g., changing database usage for the > contrib.auth app). Brilliant! Thanks Russell. +1 Paul McLanahan -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: What The Enterprise wants from Django
On 2010-01-21, at 9:15 AM, chris.moff...@gmail.com wrote: > I agree that managing settings gets to be a bit difficult in many > environments - even non Fortune 1000 environments. One of the things > we've played with in Satchmo is a database backed solution that allows > you to define reasonable defaults - even override-able in the settings > - but allow people to edit through the admin interface. I've seen that route before and I have to say that I would be strongly against that. The only saving grace could be multi-db which could allow you to have a specific settings database that's separate from the core. Having data and settings intermingled in the same area just causes too many problems. Further, it detracts from the ability to run the project without the database around. In the scenario of a larger environment with more delineation of roles, it may not be possible. -- Andy McKay, @clearwind http://clearwind.ca/djangoski -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Feedback requested: Multi-db database routing
Hi Russell, On Thu, Jan 21, 2010 at 7:10 PM, Russell Keith-Magee wrote: > On Thu, Jan 21, 2010 at 8:21 PM, Yuri Baburov wrote: >> Hi Russell, >> >> Thanks, I'm rather happy too. >> It's exactly what we wanted! >> >> DB Routing API is even better than some fixed db routing format. >> I just thought about any model with GFK, like TagField from django-tagging -- >> with multidb, to be effectively reusable, it has to know in what DB >> to save and load, depending on its client ContentType. >> >> Since it's solution for the problems discussed in >> http://groups.google.com/group/django-developers/browse_frm/thread/7904c7da7cb0085f/d063d2086299635e, >> could you please think about one more thing discussed there: syncdb >> targeting to correct databases. > > I completely acknowledge how such a setting could be useful for any > sort of metaprogramming situation - i.e., in the case of an automated > admin interface only showing the models that are known to be > available, or in the case of syncdb. > > My hesitation in implementing something to address this problem is > driven by pragmatism. > > Firstly, I haven't seen an proposal that includes an elegant way to > express this database allocation information. All the formats that > I've seen amount to reproducing either INSTALLED_APPS, or worse, > having an analog of INSTALLED_APPS down to the level of individual > models. I have no desire to introduce a requirement for an > exceptionally complex data structure, especially for the single > database case. > > Secondly, while I can see how this information could be useful, it > doesn't change the fact that the information is just as easy to manage > manually. Syncdb doesn't do anything special -- it just invokes sqlall > and loaddata initial_data on your database. The admin can be > completely configured to include (or not include) any object you want. > I know this isn't ideal, but it does *work*. Well, it does work, if you're django creator :) But Django newbie can easily fail with this "simple" task. And, of course, extensibility with 3rd party plugins is not covered. Ideally, included plugins should "just work". Guessing what to expect here? a) Plugin doesn't know any of your database aliases b) Plugin can is set to use some specific database(s) >> Something like a db_router.get_all_databases_for_model(model) call. >> It was suggested in the thread. > > Making this a programatic operation is an interesting idea, as I said > when it was proposed. Putting this callable operation in the router > avoids my previous hesitation, as it means we don't need to put > callable code in the settings file. I have some questions regarding > implementation, though: > > How do you see this interacting with the stack of routers? i.e., is it > additive over all items in the stack? > > How do you handle the case of removing objects from a database? For > example, if I want all models in contrib.auth to be on the 'users' > database, I can encode that rule in my AuthRouter, However if the > bucket case assigns all models to the default database, that undoes > the good work that the AuthRouter has done. How do you encode the > information of "yes, I've considered the default database as a > candidate for auth.User, and I've rejected it"? Ok, if something is "addictive", it can be often split to separate items. Then, db_router.allow_syncdb(database, model) with choices "Allow" (True), "Deny" (False) and "Don't care" (None) is suggested. With default to "Allow" if no router told anything specific. Syncdb isn't very speed demanding operation, so, guess, this solution will work rather good. >> Another issue is bothering me: >> db_router.allow_relation doesn't convert in my head into any algorithm >> of its usage. >> Docs in this thread didn't help me either when it's called and what for. >> So I took a look into implementation, it looks like it only works as a >> validator: >> ...if not router.allow_relation(obj, self.instance): raise ValueError(...) >> But how one would suggest in which db to save object depending on the >> related object? > > You are correct - allow_relation() is purely a validation hook, not a > database allocator. It is used to determine if mybook.author = john is > a valid operation - in this example, allow_relation(mybook, john) > would be invoked. > > On a single database situation, both objects are on the default > database, so the relation is allowed. If you have mybook on database > d1 and john on database d2, then the relation will be prevented, > unless you define a router that allows that specific cross-database > pair - for example a master/slave router would allow the relation if > d1 and d2 are part of the same slave pool. I meant, docs you sent in the beginning of this thread should be enhanced with such explanation :) >> I.e., let's talk about this example with GFK. > > There's nothing specific about generic foreign keys here - a GFK is a > relation between two objects, and they either need to be on
Re: Proposal: errors with URLs in them
On Fri, Jan 22, 2010 at 1:35 AM, Andy McKay wrote: > One of my favourite errors in Python is this one: > > File "settings.py", line 2 > SyntaxError: Non-ASCII character '\xce' in file settings.py on line 2, > but no encoding declared; see http://www.python.org/peps/pep-0263.html > for details > > This is when you've declared a non-ASCII character without the encoding > declared. In fact I don't even bother to look up the syntax to make it work, > it's easier to put the text in, let it fail and then let python tell me the > URL to the syntax. > > There are common errors that I see being made in putting together Django > applications and I'd like to see if we can provide more helpful errors to the > user. There are obviously lots of places we can't do this, but I think in > some areas we can. One example from #django last night that i've seen a few > times is leaving a comma out in INSTALLED_APPS so you have: > > INSTALLED_APPS = ( > 'django.contrib.auth' > 'django.contrib.contenttypes', > ...) > > This leads to: > > Error: No module named authdjango.contrib.contenttypes > > Which is a pretty confusing error. Supposing this said: > > Error: No module named authdjango.contrib.contenttypes. See > http://djangoproject.com/errors/32 for help. > > And that page gave lots of detail: > > This means Django can't find an app from installed apps. This could be > because: > > - You've misspelled it > > - You've missed a comma in INSTALLED_APPS > > ... > > We've already got a page that seems similar to this at: > > http://code.djangoproject.com/wiki/BetterErrorMessages > > Ideally such errors would be part of the documentation. > > This is of course won't be possible for all errors or all parts of Django, > but would work for some of the more common errors that crop up regularly. > Would this be a course worth pursuing? I'd say yes - with some caveats. Firstly, we need to be careful about casting too wide a net in catching errors. We don't want to accidentally catch errors and give a solution that is wrong. We also don't want to hide useful errors under a 'helpful' blanket (e.g., the way that Django sometimes masks import errors). If we're going to introduce specific error messages, we need to make sure they actually refer to a specific problem. Secondly, we have the problem of getting people to actually read the error message. I completely agree with you that the Python PEP-0263 error is a great example; however, we have two very good counterexamples in our own code: - MySQLdb doesn't appear to be installed - Accessor for field 'foo' clashes with field "Bar.whiz'. Add a related_name argument to the definition for foo' I've lost count of the number of people that have posted to Django-users, dumped those error messages and said "What do I do?". I'm always hesitant to blame the user for problems like this. If the user isn't following instructions, maybe the instructions aren't clear enough. Maybe the error message in question is just too pithy -- perhaps if they redirected to a single page in the docs that gave a verbose explanation of the problem (including examples and workarounds where appropriate), we might be able to avoid a few of these questions. However, in the words of Cool Hand Luke: "some folks, you just can't reach". :-) So - if you can produce a patch that demonstrates 5-10 examples of patterns that would benefit from an elaborate error message, I certainly think that patch could find it's way into trunk. More broadly, *any* patch that improves error handling is most welcome. Looking at the wiki page you reference, many of those errors are simple cases of an Attribute/Value error that should be caught and rethrown with a more meaningful error. Even if we don't put an explanatory page in the docs, those errors should be improved. Yours, Russ Magee %-) -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Feedback requested: Multi-db database routing
On Fri, Jan 22, 2010 at 2:03 AM, Yuri Baburov wrote: > Hi Russell, > > On Thu, Jan 21, 2010 at 7:10 PM, Russell Keith-Magee > wrote: >> Secondly, while I can see how this information could be useful, it >> doesn't change the fact that the information is just as easy to manage >> manually. Syncdb doesn't do anything special -- it just invokes sqlall >> and loaddata initial_data on your database. The admin can be >> completely configured to include (or not include) any object you want. >> I know this isn't ideal, but it does *work*. > Well, it does work, if you're django creator :) > But Django newbie can easily fail with this "simple" task. I'm all for making Django easy to use for new users, but if your first Django project is a heterogeneous multi-database setup, then I don't think it's unreasonable to expect that you might have a learning curve. Remember, this problem doesn't exist *at all* if you're just working through the tutorial or working on a single database solution. It only exists if you're using multidb. > And, of course, extensibility with 3rd party plugins is not covered. > Ideally, included plugins should "just work". > Guessing what to expect here? > a) Plugin doesn't know any of your database aliases > b) Plugin can is set to use some specific database(s) As I said last time, I completely acknowledge that generic metaprogramming (like that done by the admin interface) isn't possible without a metadata description like the one you want. However, I'm not going to introduce a bad interface that everyone will need to use just to solve a problem that only exists for one specific subset of problems. If you can come up with a clean solution for the problem, I'll commit it. >>> Something like a db_router.get_all_databases_for_model(model) call. >>> It was suggested in the thread. >> >> Making this a programatic operation is an interesting idea, as I said >> when it was proposed. Putting this callable operation in the router >> avoids my previous hesitation, as it means we don't need to put >> callable code in the settings file. I have some questions regarding >> implementation, though: >> >> How do you see this interacting with the stack of routers? i.e., is it >> additive over all items in the stack? >> >> How do you handle the case of removing objects from a database? For >> example, if I want all models in contrib.auth to be on the 'users' >> database, I can encode that rule in my AuthRouter, However if the >> bucket case assigns all models to the default database, that undoes >> the good work that the AuthRouter has done. How do you encode the >> information of "yes, I've considered the default database as a >> candidate for auth.User, and I've rejected it"? > Ok, if something is "addictive", it can be often split to separate items. > Then, db_router.allow_syncdb(database, model) > with choices "Allow" (True), "Deny" (False) and "Don't care" (None) > is suggested. With default to "Allow" if no router told anything specific. > Syncdb isn't very speed demanding operation, so, guess, > this solution will work rather good. That could work. I want to make sure the router code I've already published is finished ready for the beta, but I'll try and take a look at this over the next couple of days. If you want to try your hand at a patch in the interim, please do. > I meant, docs you sent in the beginning of this thread > should be enhanced with such explanation :) Certainly. I was working on a draft of the docs last night, and this is one of the points that I have elaborated upon. >>> I.e., let's talk about this example with GFK. >> >> There's nothing specific about generic foreign keys here - a GFK is a >> relation between two objects, and they either need to be on the same >> database, or on two databases that a router agrees are compatible. >> allow_relation() is what does that check. > No, I meant a bit different thing: > You have GFK-ed model, you need to express, that > a) "for saving, use the same database as for related object" > b) "for saving, use this fixed database (or any of these databases)" > c) "for loading, use the same database as for related object" > d) "for loading, use this fixed database (or any of these databases)" > e) "for deleting, use the same database as for related object" -- bonus! > -- hm, can be that related object was deleted already? > f) "for deleting, use this fixed database (or any of these databases)" -- > bonus! > AFAIU, by default, for a)-f) will be used default DB. > Customizations b), d), f) are covered by .db_for_read() and > .db_for_write() calls. > What about a), c), e)? Again, there is nothing GFK specific here - normal foreign keys have exactly the same problems. GFKs are just unusual in that they are defined using two model fields rather than one. Django's behavior is the same whether you're talking about FKs or GFKs (although the implementation is different) The problem with these use cases is that they treat the related object as somet
Re: Feedback requested: Multi-db database routing
On Thu, Jan 21, 2010 at 9:54 PM, Ivan Sagalaev
wrote:
> Russell Keith-Magee wrote:
>>
>> class MasterSlaveRouter(object):
>> def db_for_read(self, model, db=None):
>> # Point all read operations to a random slave
>> return random.choice(['slave1','slave2'])
>>
>> def db_for_write(self, model, db=None):
>> # Point all write operations to the master
>> return 'master'
>>
>> def allow_relation(self, obj1, obj2):
>> # Allow any relation between two objects in the db pool
>> db_list = ('master','slave1','slave2')
>> if obj1 in db_list and obj2 in db_list:
>> return True
>> return None
>>
>> Then, in your settings file, add the following:
>>
>> DATABASE_ROUTERS = ['path.to.AuthRouter', 'path.to.MasterSlaveRouter']
>>
>> And you're set.
>
> It's more of a side note.
>
> I understand that this is only an example but examples like this one might
> give an impression that Django now solves master-slave replication out of
> the box. It's not.
>
> The hard thing about replication is that switching between master and slaves
> doesn't depend on models and doesn't depend on SQL operators. It depends on
> some vague notion of "operation context": if we are in some logical
> multi-step process of changing state then all writes *and* reads should
> happen on a master or we won't have a consistent transaction.
Sure. The router I published doesn't handle replication lag either.
As with any complex situation, it will always fall to the user to
ensure that they are paying attention to what they are doing. The
interaction of transactions and multiple databases is always going to
be tricky, and will require the end user to be aware of the
consequences of their actions.
The introduction of database routers doesn't remove or override the
ability to manually specify database use. If you know you're in a
transaction and it's important that queries need to be constrained to
the master, you can do so by issuing queries directly against the
master.
Of course, it would be nice if Django were able to automagically
detect when a master read was required by virtue of transaction use.
I'm just not sure how we would do this. Any suggestions will be
gratefully received.
> That said, this routing looks to me like a convenient and may be even
> necessary plumbing. It just doesn't solve the whole problem.
I'll make sure the docs include an appropriate admonition.
Yours,
Russ Magee %-)
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
Possible bug in messages framework?
Hey guys-
I've been using Django 1.2's new messages framework a lot lately, and
it's awesome. but, i think I may have discovered a bug (or possibly
I'm doing something wrong, but if so, I sure as hell can't figure out
what it is). It seems to me the messages are never delivered to the
user when there's an HttpResponseRedirect after the message is set.
For example:
def checkin(request, venue_id):
venue = get_object_or_404(Venue, id=venue_id)
checkin = CheckIn(singer=request.singer, venue=venue,
date_created=datetime.datetime.now())
checkin.save()
message = "Skadoosh! You're checked in to %s." % venue.name
messages.success(request, message)
return HttpResponseRedirect(request.META['HTTP_REFERER'])
This doesn't work. When the user is redirected, they don't see the
message. On the other hand, this works fine:
def checkin(request, venue_id):
venue = get_object_or_404(Venue, id=venue_id)
checkin = CheckIn(singer=request.singer, venue=venue,
date_created=datetime.datetime.now())
checkin.save()
message = "Skadoosh! You're checked in to %s." % venue.name
messages.success(request, message)
return render_to_response("path/to/template.html", { 'message':
message }, context_instance=RequestContext(request))
This is all using the default backend (LegacyFallbackStorage).
Thoughts? Bug, or am I doing something wrong? Thanks in advance, all!
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
Re: Possible bug in messages framework?
After a little more playing around, I've discovered that this is not an issue if I use the session storage -- so it seems to be related to cookie storage. -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Possible bug in messages framework?
I wonder if this is related? http://groups.google.com/group/django-developers/browse_thread/thread/5613e9a03d92c902/738a3b81e405dc78#738a3b81e405dc78 On Jan 21, 2010, at 10:55 PM, j...@jeffcroft.com wrote: > After a little more playing around, I've discovered that this is not > an issue if I use the session storage -- so it seems to be related to > cookie storage. > > -- > You received this message because you are subscribed to the Google Groups > "Django developers" group. > To post to this group, send email to django-develop...@googlegroups.com. > To unsubscribe from this group, send email to > django-developers+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/django-developers?hl=en. > -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Possible bug in messages framework?
Sean- Can't say for sure it's related, but I can verify that I was using Safari (Mobile) and that my message had commas in them...so it definitely could be. Switching to session storage solved the problem for me, and that backend works fine for my needs. So, it's no longer pressing issue for me, but there definitely seems to be something wrong in the cookie storage backend. Jeff On Jan 21, 9:07 pm, Sean Brant wrote: > I wonder if this is > related?http://groups.google.com/group/django-developers/browse_thread/thread... > > On Jan 21, 2010, at 10:55 PM, j...@jeffcroft.com wrote: > > > > > After a little more playing around, I've discovered that this is not > > an issue if I use the session storage -- so it seems to be related to > > cookie storage. > > > -- > > You received this message because you are subscribed to the Google Groups > > "Django developers" group. > > To post to this group, send email to django-develop...@googlegroups.com. > > To unsubscribe from this group, send email to > > django-developers+unsubscr...@googlegroups.com. > > For more options, visit this group > > athttp://groups.google.com/group/django-developers?hl=en. -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
