Hi Jan, It sounds like you've made great progress. We have an informal trac and hg repo set up at trac and hg dot woe-beti.de respectively. you're more than welcome to add your documentation there! Let me know if you want an hg repo tp play with too and I'll sort it out for you.
Cheers, Ben 2008/7/8 Jan Oberst <[EMAIL PROTECTED]>: > > Hi guys, > > I've been heavily swamped with work for college, so I missed this > thead and the few others on multiple databases. Sorry. > > I have implemented a proof-of-concept database scaling solution for > Django. It tackles all kind of scaling issues I have seen in Django. > It's purpose is mainly to find out if we could scale up Django at all. > I didn't worry too much about syntax and the way it's supposed to > integrate into Django - I just hacked away in Django code to make it > work the fastest possible way I could think of. > > > The solution covers the largest part of Simon's #2 problem. I added a > few attributes and config parameters to the ORM so you can decide > which models are hosted on which server. One model can be hosted on 20 > servers with the actual location depending on a foreign key value. > > We're using it to store data for different groups on different servers > for a more horizontal scaling. For example if a photo got a ForeignKey > to group A it will be routed to server 15 because of some logic. > > You can also route objects 1-1000 to server 1 and 1001-2000 to server > 2. > > > I have also added database denormalization, caching foreign key > querysets to the DB, bulk prefetching, in-model privacy checks and a > few other things. > > A large percentage of the stuff probably isn't suitable for Django- > trunk. Most of it tackles quite specific and hard scaling issues, but > I guess there's a way to build it more modular and make it work for > more people. After all I'm new to Django-developers and also to > opening up my work. > > If some of you are interested in the code and would benefit from it I > would be more than happy to share. > > Just posting a big pile of code probably won't help you too much, so I > thought I'd write a few lines documentation about each part and post > them here. Does that sound reasonable? > > Jan > > On May 22, 4:59 pm, Simon Willison <[EMAIL PROTECTED]> wrote: > > I have to admit I'm slightly worried about the multi-database > > proposal, because at the moment it doesn't seem to solve either of the > > multi-db problems I'm concerned about. > > > > The proposal at the moment deals with having different models live in > > different databases - for example, the Forum application lives on DB1 > > while the Blog application lives on DB2. > > > > I can see how this could be useful, but the two database problems that > > keep me up at night are the following: > > > > 1. Replication - being able to send all of my writes to one master > > machine but spread all of my reads over several slave machines. > > Thankfully Ivan Sagalaev's confusingly named mysql_cluster covers this > > problem neatly without modification to Django core - it's just an > > alternative DB backend which demonstrates that doing this isn't > > particularly hard:http://softwaremaniacs.org/soft/mysql_cluster/en/ > > > > 2. Sharding - being able to put User entries 1-1000 on DB1, whereas > > User entries 1001-2000 live on DB2 and so on. > > > > I'd love Django to have built-in abilities to solve #1 - it's a really > > important first-step onscalingup to multiple databases, and it's > > also massively easier than any other part of the multi-db problem. > > > > I wouldn't expect a magic solution to #2 because it's so highly > > dependent on the application that is being built, but at the same time > > it would be nice to see a multi-db solution at least take this in to > > account (maybe just by providing an easy tool to direct an ORM request > > to a specific server based on some arbitrary logic). > > > > I may have misunderstood the proposal, but I think it's vital that the > > above two use cases are considered. Even if they can't be solved > > outright, providing tools that custom solutions to these cases can be > > built with should be a priority for multi-db support. > > > > Cheers, > > > > Simon > > > -- Regards, Ben Ford [EMAIL PROTECTED] +447792598685 --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~----------~----~----~----~------~----~------~--~---
