On Fri, Mar 25, 2011 at 12:51 PM, Stephen J. Turnbull <step...@xemacs.org> wrote: > Eg, much as I normally respect Barry's intuitions, his proposal (to > remove costly tests, without reference to the possibility of missing > something important) is IMHO absolutely the wrong criterion. I don't > really know about Python's test suite, but in XEmacs the time- > expensive tests are mostly the ones that involve timeouts and lots of > system calls because they interact with the OS -- and those are > precisely the finicky areas where a small change can occasion an > unexpected bug. For XEmacs, those bugs also are more likely than > average to be showstoppers for dependents, in the sense that until > they're fixed, the dependents can't be tested at all. So the cost of > *not* running those tests is relatively high, too.
For Python, our most expensive, slow tests are generally process related or IO related (over time the threading related ones have been largely fixed to use Event based signalling rather than relying on timeouts, so they're significantly faster than they once were). These slow tests are *also* typically the most platform dependent tests, so a "green light" from a local test run is generally pretty inconclusive - you don't really find out whether you borked something until you get green lights on the buildbots for platforms other than those the patch was developed on. So I still see value in having a standard "smoke test" that runs through the platform independent stuff, to reduce the number of minor errors that needlessly cause the buildbots to go red. The idea would be to create a tiered test suite along the following lines: 1. The buildbots: run the entire (-uall) test suite across a variety of platforms. Performed for every commit pushed to hg.python.org/cpython. 2. Complete local test: run the entire (-uall) test suite on a local working copy. Recommended before first committing a fix or change to a branch. 3. Basic local test: run the test suite with no additional resources enabled on a local working copy. Current closest equivalent to a "smoke test" 4. Proposed "smoke test": quick test of platform independent code for use when merging heads after a push race 5. Specific tests: run specified tests for modules being worked on. Used during development to check fix validity and feature degree of completion. With the volume of platform dependent code we have in CPython and the standard library, the only way we're ever likely to achieve consistently green buildbots is to move to a "staging repo" model, where commits only get made to the central repo after they have already passed muster on the buildbots at least once. I think that's actually a good way for us to go in the long run, but I also think separating out a fast set of platform independent is a decent idea. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
