Re: Bug in DateTimeInput: localization formatting decided during start-up?
Ticket created. http://code.djangoproject.com/ticket/16038 -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Django Triage Project
Dear Django Developers I need to implement a triage process for security system, I have read the django documents that said it has a triage system for its tickets. that is exactly what I need. Did any body know how can I find the django triage system source code to use in my project Best Regards Mohammad Hamidi Esfahani -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Django Triage Project
On Mon, May 16, 2011 at 2:04 PM, Mohammad Hamidi Esfahani wrote: > Dear Django Developers > > I need to implement a triage process for security system, I have read > the django documents that said it has a triage system for its tickets. > that is exactly what I need. > > Did any body know how can I find the django triage system source code > to use in my project Django uses Trac to organize it's tickets. It isn't a Django-based project, but it does use Python. Yours, Russ Magee %-) -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: RFC: Composite fields API
On Thu, May 12, 2011 at 5:16 PM, Carl Meyer wrote: > > On 05/12/2011 06:41 AM, Michal Petrucha wrote: > > On Thu, May 12, 2011 at 02:49:03PM +0100, Tom Evans wrote: > The value of a CompositeField will be represented by an instance of a > > CompositeValue class. This will be a descendant of tuple and will > > resemble namedtuple present in Python >= 2.5. It will support > > iteration, numbered indexing and access to individual field values > > using attributes corresponding to underlying field names. The order of > > values will be the same as the order of fields specified in the model > > definition. > > Yes, Tom is right of course - now that Python 2.5 is minimal version, we > can just use namedtuple. > As far as I can tell, namedtuple was added in Python 2.6, not 2.5, so a compatibility class may still be necessary. http://docs.python.org/library/collections.html#collections.namedtuple -- Regards, Ian Clelland -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Test optimizations (2-5x as fast)
Ahoy, Alex! Thanks for the quick response. > 1. Class-level fixture setup > >> This is the one I'm most interested. I did a patch a number of months ago >> to do the fixture parsing, but not DB insertion on a per-class basis. I >> didn't find that to be a big win. However, I'm going to be working on a >> patch to do bulk inserts (that is a single execute/executemany call for all >> objects to be inserted), which could be a big win for fixture loading, so >> I'd kind of like to do that first, to see how big a win this is after that. >> This is obviously more specialized, and invasive (IMO), so if we can get >> most of the win without it that might be good enough. Could you explain what you mean by "to do the fixture parsing"? Did you try to speed up or cache the JSON parsing or something? It's the per-class setup that yielded the biggest win for 2 of our largest sites: support.mozilla.com and addons.mozilla.com. We have (unsurprisingly) several tests in each class, and this avoids redoing all the I/O for each test. (CPU is practically free in an I/O-using situation like this, so I went straight for the disk writes.) As for bulk inserts, that would be great to have as well! I'd be surprised if they were a huge win, since, in an MVCC DB, the writes typically happen on commit, and there wouldn't be any fewer of those. On the other hand, it's another way to cut traffic, and engines like MyISAM should benefit more, since they commit immediately. I'd love to see numbers on it. Have you had a chance to bench it? > Speeding up tests is defintely of interest to me, so thanks for the great > work! You're welcome! I scratched a personal itch, but I hope others can benefit as well. Toward that, should I work up a Django patch, or would the core team rather I release my work as a pluggable package? I realistically have the time to do only one. Cheers, Erik -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Test optimizations (2-5x as fast)
On Mon, May 16, 2011 at 7:36 PM, Erik Rose wrote: > Toward that, should I work up a Django patch, or would the core team rather I > release my work as a pluggable package? Patch, please! Fast is good :) Jacob -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Test optimizations (2-5x as fast)
> How about caching the test databases? The database state could be cached > after model setup (which takes some time if you've got lots of them) + > initial data fixture setup, and after the setup for each test case (fixtures > + setUp() method). > > So, in the best case, no database setup is required at all to run tests -- > which encourages test driven development :-) So that would be 11 separate DBs for our tests, and you'd just switch between them? Interesting idea. Or are you proposing caching the results of queries for each test class, essentially mocking out the DB? Perhaps some numbers would illuminate: I clock the total setup and teardown time for support.mozilla.com's 1064 tests at 2.59 seconds after my optimizations. So I'm pretty happy with that. :-) CPU use for the test run is 76% according to the `time` commands, so there's a little more I/O to kill but not much. Cheers, Erik -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Test optimizations (2-5x as fast)
Woo, thanks for the constructive suggestions! > Also, one thing I'm quickly noticing (I'm a bit confused why its > setup_class and not setUpClass as well), I was writing to nose's hooks; didn't realize Django used unittest2 now! > but this wont work with > postgres without changing the DELETE code to work like the test > runner's TRUNCATE foo, bar; (due to foreign key constraints). Absolutely. I assume this is what you fix below > You can do something like this to handle the > flushing: > >sql_list = connection.ops.sql_flush(no_style(), > tables, connection.introspection.sequence_list()) >for sql in sql_list: >cursor.execute(sql) Brilliant! Thanks! Say, can you think of any backends in which you actually have to reset the sequences after truncating? That seems like an interesting decoupling to me. MySQL, anyway, does the reset implicitly; perhaps we can optimize its sql_flush routine. > Unfortunately, you're still reliant that nothing was created with > signals that uses constraints. For us this is very common, and I can't > imagine we're an edge case there Can you tell me of these signals? Which ones? I don't think we use them, but I don't want to overlook them. Erik -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Test optimizations (2-5x as fast)
Postgres requires resetting the sequences I believe. I just assume Oracle/MSSQL are probably similar. Regarding the signals, basically we have a bunch of post_save type things, which tend to store aggregate data for certain conditions. These get populated (in some cases) in our tests, but don't correspond to a fixture or a model in the same app. -- David Cramer http://justcramer.com On Mon, May 16, 2011 at 9:09 PM, Erik Rose wrote: > Woo, thanks for the constructive suggestions! > >> Also, one thing I'm quickly noticing (I'm a bit confused why its >> setup_class and not setUpClass as well), > > I was writing to nose's hooks; didn't realize Django used unittest2 now! > >> but this wont work with >> postgres without changing the DELETE code to work like the test >> runner's TRUNCATE foo, bar; (due to foreign key constraints). > > Absolutely. I assume this is what you fix below > >> You can do something like this to handle the >> flushing: >> >> sql_list = connection.ops.sql_flush(no_style(), >> tables, connection.introspection.sequence_list()) >> for sql in sql_list: >> cursor.execute(sql) > > Brilliant! Thanks! Say, can you think of any backends in which you actually > have to reset the sequences after truncating? That seems like an interesting > decoupling to me. MySQL, anyway, does the reset implicitly; perhaps we can > optimize its sql_flush routine. > >> Unfortunately, you're still reliant that nothing was created with >> signals that uses constraints. For us this is very common, and I can't >> imagine we're an edge case there > > Can you tell me of these signals? Which ones? I don't think we use them, but > I don't want to overlook them. > > Erik -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Test optimizations (2-5x as fast)
> Regarding the signals, basically we have a bunch of post_save type > things, which tend to store aggregate data for certain conditions. > These get populated (in some cases) in our tests, but don't correspond > to a fixture or a model in the same app. Ah, gotcha. So, a couple solutions off the top of my head: either just punt and truncate everything (except auth_permission and django_content_type). That would be a shame, since it takes awhile. OTOH, if you use fixture grouping (optimization #2), that's a lot more tolerable. Or maybe we could monitor what tables are getting insertions somehow. I'll give it some thought. -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Test optimizations (2-5x as fast)
On Fri, 2011-05-13 at 16:57 -0700, Erik Rose wrote: > tl;dr: I've written an alternative TestCase base class which makes > fixture-using tests much more I/O efficient on transactional DBs, and I'd > like to upstream it. > > Greetings, all! This is my first django-dev post, so please be gentle. :-) I > hack on support.mozilla.com, a fairly large Django site with about 1000 > tests. Those tests make heavy use of fixtures and, as a result, used to take > over 5 minutes to run. So, I spent a few days seeing if I could cut the > amount of DB I/O needed. Ultimately, I got the run down to just over 1 > minute, and almost all of those gains are translatable to any Django site > running against a transactional DB. No changes to the apps themselves are > needed. I'd love to push some of this work upstream, if there's interest (or > even lack of opposition ;-)). > > The speedups came from 3 main optimizations: > > 1. Class-level fixture setup > > Given a transaction DB, there's no reason to reload fixtures via dozens of > SQL statements before every test. I made use of setup_class() and > teardown_class() (yay, unittest2!) to change the flow for TestCase-using > tests to this: > a. Load the fixtures at the top of the class, and commit. > b. Run a test. > c. Roll back, returning to pristine fixtures. Go back to step b. > d. At class teardown, figure out which tables the fixtures loaded into, > and expressly clear out what was added. > > Before this optimization: 302s to run the suite > After: 97s. > > Before: 37,583 queries > After: 4,116 > > On top of that, an additional 4s was saved by reusing a single connection > rather than opening and closing them all the time, bringing the final number > down to 93s. (We can get away with this because we're committing any > on-cursor-initialization setup, whereas the old TestCase rolled it back.) > > Here's the code: > https://github.com/erikrose/test-utils/blob/master/test_utils/__init__.py#L121. > I'd love to generalize it a bit (to fall back to the old behavior with > non-transactional backends, for example) and offer it as a patch to Django > proper, replacing TestCase. Thoughts? > > (If you notice that copy-and-paste of loaddata sitting off to the side in > another module, don't fret; in the patch, that would turn into a refactoring > of loaddata to make the computation of the fixture-referenced tables > separately reusable.) > > > 2. Fixture grouping > > I next observed that many test classes reused the same sets of fixtures, > often via subclassing. After the previous optimization, our tests still > loaded fixtures 114 times, even though there were only 11 distinct sets of > them. So, I thought: why not write a custom testrunner that buckets the > classes by fixture set and advises the classes that, unless they're the first > or last in a bucket, they shouldn't bother tearing down or setting up the > fixtures, respectively? This took the form of a custom nose plugin (we use > nose for all our Django stuff), and it took another quarter off the test run: > > Before: 97s > After: 74s > > Of course, test independence is still preserved. We're just factoring out > pointlessly repeated setup. > > I don't really have plans to upstream this unless someone calls for it, but > I'll be making it available soon, likely as part of django-nose. > > > 3. Startup optimizations > > At this point, it was bothering me that, just to run a single test, I had to > wait through 15s of DB initialization (mostly auth_permissions and > django_content_type population)—stuff which was already perfectly valid from > the previous test run. So, building on some work we had already done in this > direction, I decided to skip the teardown of the test DB and, symmetrically, > the setup on future runs. If you make schema changes, just set an env var, > and it wipes and remakes the DB like usual. I could see pushing this into > django-nose as well, but it's got the hackiest implementation and can > potentially confuse users. I mention it for completeness. > > Before: startup time 15s > After: 3s (There's quite a wide variance due to I/O caching luck.) > > Code: https://github.com/erikrose/test-utils/commit/b95a1b7 > > > If you read this far, you get a cookie! I welcome your feedback on merging > optimization #1 into core, as well as any accusations of insanity re: #2 and > #3. FWIW, everything works great without touching any of the tests on 3 of > our Django sites, totaling over 2000 tests. I would be very happy to test this against Oracle database to see is how much patch improves speed since previously running tests against Oracle has been a real pain specially all db recreate stuff took a long long time. -- Jani Tiainen -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from
