One year ago, Martin Michlmayr made a speach at the Debian Conference 1 about Quality Assurance within Debian and the progress that have been made so far. He mentioned that we lack regression tests for packages, but as far as I know, no solution has been found.
What are regression tests? -------------------------- Regression tests are tests that are made to ensure that the behaviour of a given program has not changed across releases. Why? ---- Currently, we have some way to check a package before it gets installed. That is what Lintian (and Linda as well ;-) is here for: checks are performed on the content of the binary and source package. However, we have no way to check a package after its installation. And in most of the cases, Lintian checks cannot prevent software breakages. Until now, developers need to test packages manually: installation, upgrade, removal, purge, failures and so on. This can be very painful and boring when you have different ways to configure your packages, especially when they have to be configured through debconf. Furthermore, some packages may perform some complex tasks like database creation/upgrades, web server configuration, mail aliases management, etc and user configuration and environment may vary, making the success of such tasks unpredictable. What? ----- The main idea is to make tests on package installation and after their installation is completed. Here is a list of possible tests that come to my mind (in no special order): - tests on maintainer scripts: maintainer scripts perform some very simple tasks like: changing ownerships and permissions, starting daemons, creating users/groups, etc. They are likely to change across releases, and as they are mostly written in shell, they are error prone. This could be as complex as checking that a database has been properly configured, that a LDAP directory has been properly populated, that a web server has been correctly modified. - debconf tests: there are usually many possible answer to debconf question and different paths within. Some tests can be written to simulate user interaction, providing a broader set of configuration answer. Checks would be made with respect to some given answers. - configuration files tests: some behaviours may vary with respect to configuration options. Tests would provide a set of configuration files and their matching expected behaviour. - application-specific tests: sometimes Debian customize applications and we may want to see if nothing has been broken. - ... How? ---- In a chrooted environment, we can install deboostrap and package dependencies through APT. Then, we need some program that would execute some simple scripts in a pseudo language (for safety purpose, tests must be error prone as much as possible) describing what tests have to be performed, and how they would be performed. This is only a short summary :-) Do we need this? ----------------- Although this may look boring for small packages, I have many reasons to think that this can be usefull for complex packages. We can imagine running such test overnight for packages maintained via CVS. Comments are welcome. Thanks. Cheers, PS: Please, no CC on reply, I'm subscribed to the list. -- Jérôme Marant
