[Python-Dev] Disable tests in unittest (issue3202)

[Justin Mazzola Paluska]

Hi,

I just reported issue3202 in the bugracker
(http://bugs.python.org/issue3202) with a patch that implements a way
to disable unittest.TestCases using a decorator.  Benjamin Peterson
suggested that I bring up the issue on python-dev.

The idea behind the patch is that it’s sometimes useful to disable
tests without removing them from the TestCase.  For example, a
co-worker and I have a module with a few tests that will fail for the
forseeable future because we haven’t had a chance to implement the
features the tests are testing.  The tracebacks for these “not
implemented yet” tests obscure real failures in other tests.

Normally, we’d just rename the failing methods from something starting
with “test” to something like “_test”, so unittest ignores them.
However, doing so removes all traces of the tests when you re-run the
test suite, so the disabled tests are easy to forget.

Instead, issue3202 allows you to write something like:

  from unittest import main, TestCase, disabled
  
  class SampleTest(TestCase):
  
  def test_addition(self):
  
  self.assertEqual(2, 1+1)
  
  def test_broken(self):

  # this is a real failure in our code  
  self.assertEqual(5, 2+2)

  @disabled
  def test_not_implemented(self):
  
  # Test of a feature not implemented yet.
  doit()
  if __name__ == '__main__':
  main()

which has one test disabled.  Running the test suite outputs

%python sample_tests2.py 
.FD
==
FAIL: test_broken (__main__.SampleTest)
--
Traceback (most recent call last):
  File "sample_tests2.py", line 12, in test_broken
self.assertEqual(5, 2+2)
AssertionError: 5 != 4

--
Ran 3 tests in 0.001s

FAILED (failures=1)

showing only the real failure and a simple “D” for the disabled test.
The disabled test is still there, but doesn’t get in your way.

JUnit4 has a similar decorator for its tests, @Ignore.

The patch in issue3202 implements the decorator @disabled and a few
modifications to the classes in the unittest module.

What does the Python hivemind think?  I’m also open to enhancing it if
the list has any ideas.
—Justin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Disable tests in unittest (issue3202)

[Justin Mazzola Paluska]

On Thu, Jun 26, 2008 at 10:51:49AM +1000, Jonathan Lange wrote:
> On Thu, Jun 26, 2008 at 7:13 AM, Justin Mazzola Paluska <[EMAIL PROTECTED]> 
> wrote:
> I think it's really worth looking at the approach that bzrlib's tests
> take here (see bzrlib.tests.ExtendedTestResult and the
> out-of-date-but-helpful http://bazaar-vcs.org/BzrExtendTestSuite)
> 
> Instead of disabling the tests, their practice is to run the tests but
> have them raise KnownFailure. When they write a test that they know
> will fail, they have it raise this exception. The extended TestResult
> object catches this exception in addFailure and reports it
> appropriately.[1]

I wasn’t aware of bzrlib’s extentions to unittest — many of them,
including KnownFailure, TestSkipped,
bzrlib.test.ExtendedTestResult.expectFailure, and the sundry list of
bzrlib.test.TestCase.assert* look useful enough to belong in the
standard library.


> I can see that disabling still has some value, but I'd rather have
> KnownFailure first.

I think disabling and the bzr extensions have different use cases.
KnownFailure lets someone with an intimate knowledge of the the
failing test case write down that they expect the test to fail.  The
disabled decorator lets someone less familiar with the test cases to
easily disable tests that are getting in their way, without needing to
understand the body of the test function (or worse, mis-understanding
it and breaking it in a hard-to-understand way).

I agree that in the ideal world, we’d like to have both.  In fact, if
we had the bzr extensions, it’d be easy to write the disabled
decorator in terms of their KnownFailure:

def disabled(func):
@wraps(func):
def wrapper(*args, **kwargs):
raise KnownFailure("%s disabled" % func.__name__)
return wrapper

However, unless Python’s unittest is going to incorporate the useful
features of bzrlib’s TestCase object[*], it probably makes sense to
implement the disabled feature separately.
—Justin

[*] Perhaps in the long term, this is the right thing to do as
unittest seems to have fallen behind in useful feature count as its
“conceptual” parent JUnit.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com