The value of a unittest test is not in how well they pass, but in how well
they fail.
While looking at possibly helping with the str_uni branch when that was
going on I found that in some cases unittest failure results can take a
little bit (or a lot) of work to figure out just what was failing, where
and why.
While helping Eric test the new format function and class I came up with a
partial solution which may be a bases for further improvements. Eric told
me it did help quite a bit. So I think it's worth looking into.
Since we were running over a hundred different options over several
different implementations to make sure they all passed and failed in the
same way, we were using data based test cases so we could easily test the
same data with each version. Unfortunately that has a drawback that the
traceback doesn't show what data was used when testing exceptions.
Additionally when something did fail it was not always obvious what and why
it was failing.
One of the conclusions I came to is it would be better if tests did not
raise standard python exceptions unless the test itself has a problem. By
having tests raise special *Test_Only* exceptions, it can make the output
of the test very much clearer.
Here are the added Test_Only Excepitons. These would only be in the
unittest module to catch the following situations.
Wrong_Result_Returned
Unexpected_Exception_Raised
No_Exception_Raised
Wrong_Exception_Raised
And two new functions that use them.
assertTestReturns(expect, test, message)
assertTestRaises(expect, test, message)
These additions would not effect any existing tests. To use these requires
the code to be tested to be wrapped in a function with no arguments. And
it is the same format for both assertTestReturns and assertTestRaises.
for data in testdata:
expect, a, b, c = data
def test():
return foo(a, b, c)
assertTestReturns(expect, test, repr(data))
Replacing all existing tests with this form isn't reasonable but adding
this as an option for those who want to use it is very easy to do.
The test file I used to generate the following output is attached.
Cheers,
Ron
###
#
# Test output using standard assertEquals and assertRaises.
#
* The data has the form [(ref#, expect, args, kwds), ...]
* The ref# is there to help find the failing test for situation where
you may have dozens of almost identical data. It's not required but
helpful to have.
* I didn't include actual bad testcase tests in these examples, but if
some generated exceptions similar to the that of the failing tests, I think
it could add a bit more confusion to the situation than the not too
confusing example here.
$ python ut_test.py
EEFFFFFF
======================================================================
ERROR: test_A (__main__.test1_normal_failures)
----------------------------------------------------------------------
Traceback (most recent call last):
File "ut_test.py", line 100, in test_A
result = some_function(*args, **kwds)
File "ut_test.py", line 62, in some_function
baz = kwds['baz']
KeyError: 'baz'
#
# This fails as a test "error" instead of a test "fail".
# What was args and kwds here?
#
======================================================================
ERROR: test_B (__main__.test1_normal_failures)
----------------------------------------------------------------------
Traceback (most recent call last):
File "ut_test.py", line 108, in test_B
self.assertRaises(expect, test, args, kwds)
File "unittest.py", line 320, in failUnlessRaises
callableObj(*args, **kwargs)
File "ut_test.py", line 107, in test
return some_function(*args, **kwds)
File "ut_test.py", line 62, in some_function
baz = kwds['baz']
KeyError: 'baz'
#
# Same as above. Fails as a test "error", unkown arguments
# values for some_function().
#
======================================================================
FAIL: test_C (__main__.test1_normal_failures)
----------------------------------------------------------------------
Traceback (most recent call last):
File "ut_test.py", line 114, in test_C
self.assertRaises(expect, test, args, kwds)
AssertionError: KeyError not raised
#
# What was args, and kwds values?
#
======================================================================
FAIL: test_D (__main__.test1_normal_failures)
----------------------------------------------------------------------
Traceback (most recent call last):
File "ut_test.py", line 120, in test_D
repr((n, expect, args, kwds)))
AssertionError: (8, ('Total baz:', 4), [1, 2], {'baz': 'Total baz:'})
#
# This one is ok.
#
###
#
# Test output using the added methods and test only exceptions with
# the same test data.
#
* Test errors only occur on actual test "errors".
* The reason for the fail is explained in all cases for test "fails".
* The only time you get an actual python exception is when the test
it self has a problem. Otherwise you get an test_exception that
refers to the exception in the actual code.
======================================================================
FAIL: test_A (__main__.test2_new_failures)
----------------------------------------------------------------------
Traceback (most recent call last):
File "ut_test.py", line 131, in test_A
repr((n, expect, args, kwds)))
File "ut_test.py", line 36, in assertTestReturns
result = test()
File "ut_test.py", line 129, in test
return some_function(*args, **kwds)
File "ut_test.py", line 62, in some_function
baz = kwds['baz']
Unexpected_Exception_Raised: KeyError('baz',)
Reference:
(2, ('Total baz:', 3), [1, 2], {'raz': 'Total baz:'})
======================================================================
FAIL: test_B (__main__.test2_new_failures)
----------------------------------------------------------------------
Traceback (most recent call last):
File "ut_test.py", line 138, in test_B
repr((n, expect, args, kwds)))
File "ut_test.py", line 45, in assertTestRaises
result = test()
File "ut_test.py", line 136, in test
return some_function(*args, **kwds)
File "ut_test.py", line 62, in some_function
baz = kwds['baz']
Wrong_Exception_Raised: KeyError('baz',)
Reference:
(4, <type 'exceptions.IndexError'>, [1, 2], {'raz': 'Total baz:'})
======================================================================
FAIL: test_C (__main__.test2_new_failures)
----------------------------------------------------------------------
Traceback (most recent call last):
File "ut_test.py", line 145, in test_C
repr((n, expect, args, kwds)))
File "ut_test.py", line 52, in assertTestRaises
raise self.No_Exception_Raised(result, ref)
No_Exception_Raised: returned -> ('Total baz:', 3)
Reference:
(6, <type 'exceptions.KeyError'>, [1, 2], {'baz': 'Total baz:'})
======================================================================
FAIL: test_D (__main__.test2_new_failures)
----------------------------------------------------------------------
Traceback (most recent call last):
File "ut_test.py", line 152, in test_D
repr((n, expect, args, kwds)))
File "ut_test.py", line 41, in assertTestReturns
raise self.Wrong_Result_Returned(result, ref)
Wrong_Result_Returned: ('Total baz:', 3)
Reference:
(8, ('Total baz:', 4), [1, 2], {'baz': 'Total baz:'})
----------------------------------------------------------------------
Ran 8 tests in 0.004s
FAILED (failures=6, errors=2)
import sys
import unittest
import traceback
#
# Additions to unittest.TestCase class.
#
class TestCase(unittest.TestCase):
failureException = unittest.TestCase.failureException
class Unexpected_Exception_Raised(failureException):
def __init__(self, exc, ref):
self.exc = exc
self.ref = ref
def __str__(self):
return '\n'.join([repr(self.exc), '\nReference:', self.ref])
class Wrong_Exception_Raised(Unexpected_Exception_Raised): pass
class No_Exception_Raised(failureException):
def __init__(self, result, ref=""):
self.result = repr(result)
self.ref = ref
def __str__(self):
return "returned -> " + '\n'.join([self.result,
'\nReference:', self.ref])
class Wrong_Result_Returned(No_Exception_Raised):
def __str__(self):
return '\n'.join([self.result, '\nReference:', self.ref])
def assertTestReturns(self, test, expect, ref=""):
try:
result = test()
except Exception, e:
e0, e1, e2 = sys.exc_info()
raise self.Unexpected_Exception_Raised, (e, ref), e2
if result != expect:
raise self.Wrong_Result_Returned(result, ref)
def assertTestRaises(self, test, expect, ref=""):
try:
result = test()
except Exception, e:
if isinstance(e, expect):
return e
else:
e0, e1, e2 = sys.exc_info()
raise self.Wrong_Exception_Raised, (e, ref), e2
raise self.No_Exception_Raised(result, ref)
#
# A minimal function to test in order to generate varous errors
# depending on the data.
#
def some_function(*args, **kwds):
bar = args[0] + args[1]
baz = kwds['baz']
return (baz, bar)
#
# Data lists like this could be hundreds of lines.
# The ref# is there to help distinguish nearly identical
# tests from each other and help find the specific failing item.
#
# (ref#, expected, args, kwds)
test_data_A = [
(1, ('Total baz:', 3), [1, 2], {'baz':'Total baz:'}), # pass
(2, ('Total baz:', 3), [1, 2], {'raz':'Total baz:'}) # Unexpected exc
]
test_data_B = [
(3, KeyError, [1, 2], {'raz':'Total baz:'}), # pass
(4, IndexError, [1, 2], {'raz':'Total baz:'}) # Wrong exception
]
test_data_C = [
(5, KeyError, [1, 2], {'raz':'Total baz:'}), # pass
(6, KeyError, [1, 2], {'baz':'Total baz:'}) # No exception
]
test_data_D = [
(7, ('Total baz:', 3), [1, 2], {'baz':'Total baz:'}), # pass
(8, ('Total baz:', 4), [1, 2], {'baz':'Total baz:'}) # Wrong value
]
class test1_normal_failures(TestCase):
def test_A(self):
for n, expect, args, kwds in test_data_A:
result = some_function(*args, **kwds)
self.assertEqual(expect, result,
repr((n, expect, args, kwds)))
def test_B(self):
for n, expect, args, kwds in test_data_B:
def test(args, kwds):
return some_function(*args, **kwds)
self.assertRaises(expect, test, args, kwds)
def test_C(self):
for n, expect, args, kwds in test_data_C:
def test(args, kwds):
return some_function(*args, **kwds)
self.assertRaises(expect, test, args, kwds)
def test_D(self):
for n, expect, args, kwds in test_data_D:
result = some_function(*args, **kwds)
self.assertEqual(expect, result,
repr((n, expect, args, kwds)))
class test2_new_failures(TestCase):
def test_A(self):
for n, expect, args, kwds in test_data_A:
def test():
return some_function(*args, **kwds)
self.assertTestReturns(test, expect,
repr((n, expect, args, kwds)))
def test_B(self):
for n, expect, args, kwds in test_data_B:
def test():
return some_function(*args, **kwds)
self.assertTestRaises(test, expect,
repr((n, expect, args, kwds)))
def test_C(self):
for n, expect, args, kwds in test_data_C:
def test():
return some_function(*args, **kwds)
self.assertTestRaises(test, expect,
repr((n, expect, args, kwds)))
def test_D(self):
for n, expect, args, kwds in test_data_D:
def test():
return some_function(*args, **kwds)
self.assertTestReturns(test, expect,
repr((n, expect, args, kwds)))
def runtests():
unittest.main(__name__)
if __name__ == "__main__":
runtests()
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
