Re: Django, The Web Framework for perfectionists and innovative with rechargeable batteries.
1) Vito had a legitimate question. 2) It is hard to find anything in archives especially with a limited English. 3) He asked his question about supporting new technologies. 4) Jacob answered and explained the situation giving a lot of details and examples. 5) Vito thanked for the answer and explain his position: perfectionists != conservatives, and suggested that new trends should be ... discussed. Now we can move on, right? No, the tread started to grow while people were making more and more assumptions about the guy's sinister motives. Was it really necessary to argue with Google Translate or whatever he was using? This is a retorical question => please do not try to answer. Cheers, Eugene Lazutkin http://lazutkin.com/ On 07/30/2010 04:02 PM, Jacob Kaplan-Moss wrote: > On Fri, Jul 30, 2010 at 3:39 PM, Mark Bucciarelli wrote: >> The devs here _extremely_ patient with whiners [1]. >> >> m >> >> [1] People with lots of ideas about how you should >> spend your free time, and spend hours writing emails >> trying to convince you how smart they are, but can't >> be bothered to contribute a single line of code to the >> project. > > That's pretty harsh, and not exactly right. There are plenty of ways > to contribute to open source that don't involving writing code -- bug > reports, documentation, evangelism, blogging, twittering, speaking at > conferences, starting companies around the project, ... > > There's even room for people who have good ideas but don't have the > time to implement them. The key is knowing how to make suggestions in > a way that doesn't sound like your entitling yourself to a bunch of > volunteers' time. > > There is *ABSOLUTELY* nothing wrong with new community members -- or > old ones, or anyone -- making suggestions and advocating for a > particular direction or feature. But it needs to be done in a > respectful, concrete, and polite manner. > > As long as you remember that everyone's a volunteer, and that you > don't have any right to tell anyone how to spend their time [1], then > it's just fine. > > Jacob > > [1] Unless you'd like to send sacks of money to developers. If you > have sacks of cash, we'll probably do what you want. Depending on the > size of the sacks and the denomination of the bills, of course. > -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Memory usage depends on number of fields in model, in a strange way
Nope, I am testing Django memory usage for different kind of objects, and create 10 instances to minimize per-object measurement error, memory usage pattern is exactly the same for 100 or 1000 objects. On Jul 31, 9:30 am, hcarvalhoalves wrote: > """ > class U(models.Model): > name = models.TextField(max_length = 255, blank = False) > name2 = models.TextField(max_length = 255, blank = False) > name3 = models.TextField(max_length = 255, blank = False) > ... > data = [U() for x in xrange(10)] > """ > > What is this actually trying to test? If Python allocation works? > > Yes, you'll start getting all sort of weird memory usage patterns from > the Python VM if you hold references to 10 bogus objects without > garbage collecting in-between. > > On 30 jul, 21:04, OverKrik wrote: > > > > > Thx Alex, should I post minimal testapp just to make sure? I really > > tried triggering this with my own classes, but had no luck even with > > pretty complex ones. > > > On Jul 31, 3:50 am, Alex Gaynor wrote: > > > > This isn't something django can help, it has to do with the overallocation > > > patterns for python dicts. You would see this for any pytho obj with the > > > right number of fields. > > > > Alex > > > > On Jul 30, 2010 7:17 PM, "Jacob Kaplan-Moss" wrote: > > > > On Fri, Jul 30, 2010 at 5:59 PM, OverKrik wrote: > > > > Hi, I've been profiling Djan... > > > > Hm, in a quick test I can't seem to reproduce this, and I can't see > > > how it'd be happening. Can you post more details -- your model, how > > > you're measuring the memory usage, which database you're using, etc? > > > > Thanks! > > > > Jacob > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "Django developers" g... -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Memory usage depends on number of fields in model, in a strange way
Why are you testing? Have you seen memory leak somewhere? I ask it because. I had problems with memory leaks in configuration (Mysql, Apache, mod_python) but it was bad configuration of Mysql server and Apache. If you send information about your test environment. We can better reproduce you problem. For example: My test enveroment: Hardware - Intel Core i7 920 @ 2.67GHz Quad-Core Hyper-Threading-Technologie - 8 GB DDR3 RAM - 2 x 750 GB SATA-II HDD (Software-RAID 1) Apache = mod_python Configuration -- cat /etc/apache2/apache2.conf Mysql == Configuration -- cat /etc/mysql/my.cnf 2010/7/31 OverKrik : > Nope, I am testing Django memory usage for different kind of objects, > and create 10 instances to minimize per-object measurement error, > memory usage pattern is exactly the same for 100 or 1000 objects. > > On Jul 31, 9:30 am, hcarvalhoalves wrote: >> """ >> class U(models.Model): >> name = models.TextField(max_length = 255, blank = False) >> name2 = models.TextField(max_length = 255, blank = False) >> name3 = models.TextField(max_length = 255, blank = False) >> ... >> data = [U() for x in xrange(10)] >> """ >> >> What is this actually trying to test? If Python allocation works? >> >> Yes, you'll start getting all sort of weird memory usage patterns from >> the Python VM if you hold references to 10 bogus objects without >> garbage collecting in-between. >> >> On 30 jul, 21:04, OverKrik wrote: >> >> >> >> > Thx Alex, should I post minimal testapp just to make sure? I really >> > tried triggering this with my own classes, but had no luck even with >> > pretty complex ones. >> >> > On Jul 31, 3:50 am, Alex Gaynor wrote: >> >> > > This isn't something django can help, it has to do with the >> > > overallocation >> > > patterns for python dicts. You would see this for any pytho obj with the >> > > right number of fields. >> >> > > Alex >> >> > > On Jul 30, 2010 7:17 PM, "Jacob Kaplan-Moss" wrote: >> >> > > On Fri, Jul 30, 2010 at 5:59 PM, OverKrik wrote: >> > > > Hi, I've been profiling Djan... >> >> > > Hm, in a quick test I can't seem to reproduce this, and I can't see >> > > how it'd be happening. Can you post more details -- your model, how >> > > you're measuring the memory usage, which database you're using, etc? >> >> > > Thanks! >> >> > > Jacob >> >> > > -- >> > > You received this message because you are subscribed to the Google Groups >> > > "Django developers" g... > > -- > You received this message because you are subscribed to the Google Groups > "Django developers" group. > To post to this group, send email to django-develop...@googlegroups.com. > To unsubscribe from this group, send email to > django-developers+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/django-developers?hl=en. > > -- Sergej Dergatsjev E-mail: sergej.dergats...@gmail.com Skype: SergejDergatsjev Site: http://www.eecho.info -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Memory usage depends on number of fields in model, in a strange way
Hi, I am not having any memory leaks, just checking how much memory every part of django use out of interest.(And because I am writing an article about how Django ORM rocks :) ). On Jul 31, 3:40 pm, Sergej dergatsjev eecho wrote: > Why are you testing? > > Have you seen memory leak somewhere? > > I ask it because. > I had problems with memory leaks in configuration (Mysql, Apache, > mod_python) but it was bad configuration of Mysql server and Apache. > > If you send information about your test environment. > We can better reproduce you problem. > > For example: > > My test enveroment: > > Hardware > > > - Intel Core i7 920 @ 2.67GHz Quad-Core Hyper-Threading-Technologie > - 8 GB DDR3 RAM > - 2 x 750 GB SATA-II HDD (Software-RAID 1) > > Apache > = > mod_python > Configuration > -- > cat /etc/apache2/apache2.conf > > Mysql > == > Configuration > -- > cat /etc/mysql/my.cnf > > 2010/7/31 OverKrik : > > > > > > > Nope, I am testing Django memory usage for different kind of objects, > > and create 10 instances to minimize per-object measurement error, > > memory usage pattern is exactly the same for 100 or 1000 objects. > > > On Jul 31, 9:30 am, hcarvalhoalves wrote: > >> """ > >> class U(models.Model): > >> name = models.TextField(max_length = 255, blank = False) > >> name2 = models.TextField(max_length = 255, blank = False) > >> name3 = models.TextField(max_length = 255, blank = False) > >> ... > >> data = [U() for x in xrange(10)] > >> """ > > >> What is this actually trying to test? If Python allocation works? > > >> Yes, you'll start getting all sort of weird memory usage patterns from > >> the Python VM if you hold references to 10 bogus objects without > >> garbage collecting in-between. > > >> On 30 jul, 21:04, OverKrik wrote: > > >> > Thx Alex, should I post minimal testapp just to make sure? I really > >> > tried triggering this with my own classes, but had no luck even with > >> > pretty complex ones. > > >> > On Jul 31, 3:50 am, Alex Gaynor wrote: > > >> > > This isn't something django can help, it has to do with the > >> > > overallocation > >> > > patterns for python dicts. You would see this for any pytho obj with > >> > > the > >> > > right number of fields. > > >> > > Alex > > >> > > On Jul 30, 2010 7:17 PM, "Jacob Kaplan-Moss" > >> > > wrote: > > >> > > On Fri, Jul 30, 2010 at 5:59 PM, OverKrik wrote: > >> > > > Hi, I've been profiling Djan... > > >> > > Hm, in a quick test I can't seem to reproduce this, and I can't see > >> > > how it'd be happening. Can you post more details -- your model, how > >> > > you're measuring the memory usage, which database you're using, etc? > > >> > > Thanks! > > >> > > Jacob > > >> > > -- > >> > > You received this message because you are subscribed to the Google > >> > > Groups > >> > > "Django developers" g... > > > -- > > You received this message because you are subscribed to the Google Groups > > "Django developers" group. > > To post to this group, send email to django-develop...@googlegroups.com. > > To unsubscribe from this group, send email to > > django-developers+unsubscr...@googlegroups.com. > > For more options, visit this group > > athttp://groups.google.com/group/django-developers?hl=en. > > -- > Sergej Dergatsjev > E-mail: sergej.dergats...@gmail.com > Skype: SergejDergatsjev > Site:http://www.eecho.info -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Django, The Web Framework for perfectionists and innovative with rechargeable batteries.
So anyways... Google App Engine is a special case. It is not nearly as robust a database solution as something like CouchDB or MongoDB - but then again it was never meant to be. The direction from the beginning was on scaling, even if it meant dropping features. The BigTable ORM lacks many of the functions necessary for implementing the Django ORM on that database. Python-only solutions to these gaps would not scale as well as low-level database APIs, and as such would probably not scale well enough to be useful. The situation has not improved because Google doesn't see a need. App Engine fits in an even more specialized niche than most NoSQL solutions. That said, a lot of work has been done to make Django useful on that platform. My own web site runs Django 1.2.1 on App Engine. Things like auth, caching, and sessions have been implemented without porting the ORM. So really, the only barrier to "pure" Django on App Engine is the model, which needs to be rewritten for each application used on GAE. Now django-nonrel is doing some excellent work to try to implement a workable ORM on App Engine. However, I'm sure they are facing an uphill battle because of the constraints of the platform. I have no doubt that they will indeed produce a workable solution in time, though. I've been doing lots of research with Django on GAE, and am actually using it in production. If you're interesting in this particular stack, please contact me off-list and I can attempt to answer any questions that you have. -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Django, The Web Framework for perfectionists and innovative with rechargeable batteries.
On Sat, Jul 31, 2010 at 9:04 AM, tiemonster wrote: > I've been doing lots of research with Django on GAE, and am actually > using it in production. If you're interesting in this particular > stack, please contact me off-list and I can attempt to answer any > questions that you have. I'd love to see a public write-up of your project, techniques you used, and how you're liking the results. Maybe blog about it, or post it here? Jacob -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Memory usage depends on number of fields in model, in a strange way
On Sat, Jul 31, 2010 at 7:20 AM, OverKrik wrote: > Hi, I am not having any memory leaks, just checking how much memory > every part of django use out of interest.(And because I am writing an > article about how Django ORM rocks :) ). You've constructed a somewhat artificial situation -- in the real world, allocating 100,000 instances without any GC would almost always be a bug -- and so you're seeing somewhat artificial results. In practice, the number of fields is pretty irrelevant to memory consumption. As Alex said, what you're actually seeing in this situation is Python's dictionary pre-allocation at work (because Django stores the set of fields in a dictionary). I'll try to explain what's happening: Initially, an empty dictionary is allocated with 8 slots. Dictionaries are resized when they reach 2/3rds full, so the dict is resized when you add the fifth item to the dict. (You see this when you add the fourth field because there's an implied primary key field.) When dictionaries are resized, the space is doubled. However, there's a special memory optomization for min-sized (8-slot) dicts, so you see a much bigger jump from 4 to 5 fields than at any other point in the allocation. If you want to read more about how Python dicts work, you can read check out http://svn.python.org/projects/python/branches/py3k/Objects/dictnotes.txt -- it's got some pretty detailed information there. Jacob -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Memory usage depends on number of fields in model, in a strange way
Jacob, thank you for an excellent explanation. On Jul 31, 7:36 pm, Jacob Kaplan-Moss wrote: > On Sat, Jul 31, 2010 at 7:20 AM, OverKrik wrote: > > Hi, I am not having any memory leaks, just checking how much memory > > every part of django use out of interest.(And because I am writing an > > article about how Django ORM rocks :) ). > > You've constructed a somewhat artificial situation -- in the real > world, allocating 100,000 instances without any GC would almost always > be a bug -- and so you're seeing somewhat artificial results. In > practice, the number of fields is pretty irrelevant to memory > consumption. > > As Alex said, what you're actually seeing in this situation is > Python's dictionary pre-allocation at work (because Django stores the > set of fields in a dictionary). I'll try to explain what's happening: > > Initially, an empty dictionary is allocated with 8 slots. Dictionaries > are resized when they reach 2/3rds full, so the dict is resized when > you add the fifth item to the dict. (You see this when you add the > fourth field because there's an implied primary key field.) When > dictionaries are resized, the space is doubled. However, there's a > special memory optomization for min-sized (8-slot) dicts, so you see a > much bigger jump from 4 to 5 fields than at any other point in the > allocation. > > If you want to read more about how Python dicts work, you can read > check > outhttp://svn.python.org/projects/python/branches/py3k/Objects/dictnotes... > -- it's got some pretty detailed information there. > > Jacob -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Django, The Web Framework for perfectionists and innovative with rechargeable batteries.
On Sat, Jul 31, 2010 at 4:04 PM, tiemonster wrote: > So anyways... > > Google App Engine is a special case. It is not nearly as robust a > database solution as something like CouchDB or MongoDB - but then > again it was never meant to be. The direction from the beginning was > on scaling, even if it meant dropping features. The BigTable ORM lacks > many of the functions necessary for implementing the Django ORM on > that database. Python-only solutions to these gaps would not scale as > well as low-level database APIs, and as such would probably not scale > well enough to be useful. The situation has not improved because > Google doesn't see a need. App Engine fits in an even more specialized > niche than most NoSQL solutions. Which features does App Engine's datastore lack that would be necessary for Django's ORM? Why would a Python-only solution not scale well enough? You probably mean it does scale, but it's slightly less efficient because background tasks and map/reduce are handled via requests (it's a distributed system, after all), but the overhead should still be small compared to the DB overhead. What I actually like about GAE's datastore is that it forces you much earlier to think in terms of scalability. MongoDB users might suddenly realize with a 10GB database that something is wrong and needs to be rewritten and migrated. In GAE you normally run into these issues with just a few 1000 entities or earlier. There's only one issue that can show up much later (several 10,000 entities) and I've described it here: http://www.allbuttonspressed.com/blog/django/2010/01/An-App-Engine-limitation-you-didn-t-know-about Unfortunately, GAE's upcoming extended datastore API (the one which is supposed to solve the exploding indexes problem) will use merge-joins more extensively, so that limitation will probably affect more people. > Now django-nonrel is doing some excellent work to try to implement a > workable ORM on App Engine. However, I'm sure they are facing an > uphill battle because of the constraints of the platform. I have no > doubt that they will indeed produce a workable solution in time, > though. Is our current solution not workable for you? What's missing? In our experience, Django's ORM maps pretty well to GAE's API. You can access almost all GAE features via Django's ORM in a portable way. Auth, caching, and sessions work unmodified. Many small reusable apps work unmodified (e.g., django-socialauth, django-openid-auth, etc.). What's left is mostly batch-put and cursors and entity groups. The last one would be easy to add. The first two would ideally get an equivalent in Django's ORM before they're implemented, but it should be possible to provide separate functions for those features, too. I'm just asking because I'm interested in hearing from other people which problems we need to solve to make Django-nonrel and Django's official NoSQL project practical for them. Bye, Waldemar Kornewald -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Ticket #14007: Automatically discover models within a package without using the app_label Meta attribute
I the first time I went through the trouble of refactoring a large set of models into a package I couldn't figure out why my models were "missing". Took me a while to trawl through a number of blog posts to come across the app_label requirement/mechanism. At the time, I don't think it was documented, but it certainly is now. I don't see the difficulty in extending the base model class one time for your app and currying all your models by extending that subclass. I've never tried, but it seems like it should work depending on the level of introspection done during discovery. -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
