logilab.common.testlib??? =================================== This library is developed primarily by Logilab. It has the following advantages: * it is based on unittest2 THE Python standard library test * to which it adds a number of interesting features First Contact ============= Assume that your tests are written using unittest. Switching to logilab.common.testlib starts with replacing unittest.TestCase by logilab.common.testlib as source class of your tests. The starting point unittest: .. code-block:: python import unittest as ut class MyTest(ut.TestCase): pass Now becomes: .. code-block:: python class MyTest(lct.TestCase): pass We are now ready to start with this test environment! Automatic discovery of tests ============================ Let's start with a directory full of tests using logilab.common.testlib:: $ > tree dummy_tests dummy_test/ | --- [jpht 4096] test | --- [jpht 235] test_one.py | --- [jpht 235] test_two.py Launching all your tests is as simple as:: $ > cd dummy_test $ > pytest It can even be shorter if you use the following layout for your project:: $ > tree dummy_project dummy_project/ | --- [jpht 4096] test | --- [jpht 0] test_one.py | --- [jpht 0] test_two.py 1 directory, 2 files Launching all your tests can be done this way:: $ > cd dummy_project $ > pytest **pytest** is looking for a *test* directory by default. If none is find, the current directory is used. You can be more specific and select which directory to use with the *-t* option or even choose which test to run:: $ > pytest test/test_one.py TestOne will only execute TestOne in test_one.py. **pytest** can even do more for you. Imagine you have such a case:: $ > pytest going into dummy_project/test test_one.py =========================== ======= ===================== FF. =========================== =========================================== FAIL: test_fail (test_one.TestOne ) ------------------------------------------------- --------------------- Traceback (most recent call last) File "/ usr/lib/pymodules/python2.6/logilab/common/testlib.py" line 1258, in _proceed testfunc (* args, ** kwargs) File "dummy_project/test/test_one.py", line 7, in test_fail self.assertTrue (False) File "/ usr / lib / pymodules/python2.6/unittest2/case.py ", line 427, in assertTrue self.failureException raise(msg) AssertionError: False Is Not True no stdout no stderr ============================================== ======================== FAIL: test_fail2 (test_one.TestOne) ------------------ -------------------------------------------------- - Traceback (most recent call last) File "/ usr/lib/pymodules/python2.6/logilab/common/testlib.py", line 1258, in _proceed testfunc (* args, ** kwargs) File "dummy_project/test/test_one.py ", line 9, in test_fail2 self.assertTrue (False) File" / usr/lib/pymodules/python2.6/unittest2/case.py ", line 427, in assertTrue raise self.failureException (msg) AssertionError: False Is Not True no stdout no stderr test_two.py =========================== ================ ============ FF. ==================================== ================================== FAIL: test_fail (test_two.TestOne) -------- -------------------------------------------------- ------------ Traceback (most recent call last) File "/ usr/lib/pymodules/python2.6/logilab/common/testlib.py", line 1258, in _proceed testfunc (* args , ** kwargs) File "dummy_project/test/test_two.py", line 7, in test_fail self.assertTrue (False) File "/ usr/lib/pymodules/python2.6/unittest2/case . py ", line 427, in assertTrue self.failureException raise(msg) AssertionError: False Is Not True no stdout no stderr ============================================== ======================== FAIL: test_fail2 (test_two.TestOne) ------------------ -------------------------------------------------- - Traceback (most recent call last) File "/ usr/lib/pymodules/python2.6/logilab/common/testlib.py", line 1258, in _proceed testfunc (* args, ** kwargs) File "dummy_project/test/test_two.py ", line 9, in test_fail2 self.assertTrue (False) File" / usr/lib/pymodules/python2.6/unittest2/case.py ", line 427, in assertTrue raise self.failureException (msg) AssertionError: False Is Not True no stdout no stderr ********************************************** ********************************* Ran in six test cases 0.01s (0.01s CPU), 4failures 0 moduleOK (2 failed) failures: dummy_project/test/test_one [2 / 3] dummy_project/test/test_two [2 / 3] You're alone with 4 errors which must now identify in your code. That often leads to run commands one by one in a python shell to identify the blocking point. Isn't it the aim of the debuuger?! And many are now trying to use the -i option pytest:: $ > pytest -i going into dummy_project/test =========================== test_one.py ============================ FF. ====================================================================== FAIL: test_fail (test_one.TestOne) ---------------------------------------------------------------------- Traceback (most recent call last) File "/usr/lib/pymodules/python2.6/logilab/common/testlib.py", line 1258, in _proceed testfunc(*args, **kwargs) File "dummy_project/test/test_one.py", line 7, in test_fail self.assertTrue(False) File "/usr/lib/pymodules/python2.6/unittest2/case.py", line 427, in assertTrue raise self.failureException(msg) AssertionError: False is not True no stdout no stderr ====================================================================== FAIL: test_fail2 (test_one.TestOne) ---------------------------------------------------------------------- Traceback (most recent call last) File "/usr/lib/pymodules/python2.6/logilab/common/testlib.py", line 1258, in _proceed testfunc(*args, **kwargs) File "dummy_project/test/test_one.py", line 9, in test_fail2 self.assertTrue(False) File "/usr/lib/pymodules/python2.6/unittest2/case.py", line 427, in assertTrue raise self.failureException(msg) AssertionError: False is not True no stdout no stderr Choose a test to debug: 0 : test_fail (test_one.TestOne) 1 : test_fail2 (test_one.TestOne) Type 'exit' (or ^D) to quit Enter a test name: pytest offers you the opportunity to enter the debugger. You now have access to the contents of variables, you can use the up command of the debugger to rewind execution, etc... Pretty cool. You will also notice 2 goodies you won't find somewhere else: * You can access the debug for all tests that failed; simply choose which one to debug. * try the list command in the debugger: you have access to the whole code of your function / method and a colored print out! Very handy when your functions are longer than the few lines proposed by the standard debugger. Identify errors in your code becomes a breeze and writing tests a pleasure! The coverage rate ================= logilab.common.testlib provides a second key command when writing tests: pycoverage. The name reveals its functionality: determine tests coverage rate. **pycoverage** is used in conjunction with **pytest**. First step, start with the *pytest --coverage* to collect information:: $ > pytest --coverage going into easy_coding/trunk/test ======================= test_type_checker.py ======================= ...................... ========================= test_plotting.py ========================= ...................... ======================= test_misc_module.py ======================== ..... ========================= test_observer.py ========================= ... ******************************************************************************* Ran 52 test cases in 202.69s (202.38s CPU) All 4 modules OK coverage information stored, use it with pycoverage -ra Second step, analyze data using pycoverage. Let's start with some statistics for a specific module:: $ > pycoverage -a easy_coding/plotting Name Stmts Exec Cover %Missing --------------------------------------------------- easy_coding.plotting 63 62 98% 100% One line is not covered. The *-r* option will let you know which one:: $ > pycoverage -r easy_coding/plotting.py $ > cat easy_coding/plotting.py,cover ... > def plot(self, xvalues, yvalues, axis=0, format='', label='None', *args, **kw): > """ > Add an x-y plot to the graph > :type xvalues: list or array > :type yvalues: list or array > :param axis: which y-axis to use for plotting the values > :type axis: int > :param format: line style to use > :type format: str > :param label: legend associated with the set of values > :type label: str > """ > self._is_axis(axis) > if not isinstance(label, str): ! raise TypeError, 'title should be a string' > if self._hold == False and self._plots[axis].lines: > self._plots[axis].lines.pop() > line = self._plots[axis].plot(xvalues, yvalues, format, label=label, > *args, **kw) > if axis != 0: > self._labels[line[0]] = label + ' - axis y%i'% (axis+1) > else: > self._labels[line[0]] = label ... Line starting with exclamation mark is the one you are looking for. **pytest** helps you run your tests and debug the underlying code; **pycoverage** helps you write relevant tests, i.e. tests covering most of your code. And even more ============= **pytest** and **pycoverage** already make TDD easy. But **testlib** has even more for you. More assertions --------------- * assertLess|Greater * assertLess|GreaterEqual * assertDirEqual * assertFileEqual * ... The *within_tempdir* decorator ------------------------------ Imagine you want to test a function reading a configuration file: .. code-block:: python def load_configfile(filename, case_sensitive=False): if not osp.isfile(filename): raise ValueError, 'input file does not exist' # File exist; we can load it if case_sensitive: config = CaseSensitiveConfigParser() else: config = SafeConfigParser() try: config.read(filename) except ParsingError: raise TypeError, 'invalid ini file syntax' return config One solution would be to create a configuration file stored somewhere in the project tree. Another is to use **within_tempdir** and generate files in the tests; advantage only tests have to be maintained and they become independent of the project tree! Your test might look like this: .. code-block:: python import os.path as osp import logilab.common.testlib have lct TestLoadConfigfile class (lct.TestCase): @ lct.within_tempdir test_a_valid_file def (self): # you are now in a temporary directory, # you can work with relative paths With open ('myfile,' w ') as fh : fh.write ('[test] \ nkey1 = val1 \ n) self.assertTrue (osp.isfile (' myfile ') @ lct.within_tempdir test_an_invalid_file def with (self):open (' myfile ',' w ') as fh: fh.write ('invalid file') self.assertRaises (TypeError, load_configfile 'myfile') You no longer have to worry about paths or deleting temporary files and directories at the end of your test: **within_tempdir** handles everything for you! Here we are with this first introduction. You can have a look to the source code if you want to discover more.