Yaroslav Halchenko writes ("Re: package testing, autopkgtest, and all that"):
> First a brief question:
> > The source package provides a test metadata file
> > debian/tests/control. This is a file containing zero or more
> > RFC822-style stanzas, along these lines:
>
> Do you still have somewhere that awk package demo package which had
> debian/tests/ ? Currently our archive does not carry any package having
> debian/tests/ (unfortunately).Probably. I'll see if I can dig it out. > All of the above approaches seems to separate testing "materials" > from the actual packages. Both mago and checkbox come with user > interfaces, thus enabling users to test/validate their own systems > without requiring setting up any virtual environment. And they ship > their tests along (there seems to be some discovery mechanisms but I > haven't checked in detail yet). These tools are mostly orthogonal to and even complementary to autopkgtest. There are a whole bunch of other automatic testing utilities like xnee, etc., which solve other parts of the automatic testing problem and can be usefully combined with autopkgtest. I wasn't aware of "qa-regression-testing": > - qa-regression-testing: collection of unit and regression tests for > various components (from kernel to xine and apache) of the > systems. Is not designed for distribution to the users (yet?) That seems to want to collect the tests in one central place. Debian (and to an extent our derivatives) have some difficulty with centralised collections of information about multiple different pieces of software. Our social and technical structures are based around more-or-less freestanding packages. So it makes more sense in our context to locate tests with the package. > On the other hand, Ian's autopkgtest aims to provide a unified testing > framework built around packages, so that it becomes possible for us, > maintainers, to equip packages with necessary tests batteries; which could be > reused by others (e.g. QA teams). Yes. > Unfortunately the core aspect of the current autopkgtest - relying > on tests in source packages -- imho to be not the ideal solution to > target both sides of the userbase (i.e. maintainers/QA vs mortals). I'm not sure I know what you mean. > ,--- > | Q. Why put the tests in the source package ? > | > | A. Other possibilities include a special .deb generated by the > | source (which is a bit strange and what happens to this .deb and > | will make it even harder to reuse upstream test suites), or > | putting them in the .deb to be tested (definitely wrong - most > | people won't want the tests and they might be very large) or > | having them floating about separately somewhere (which prevents us > | from sharing and exchanging tests with other parts of the Free > | Software community). The source package is always available when > | development is taking place. > `--- > > Ian -- could you please unroll your arguments a bit? I still do not > see why source packages are beneficial besides build-time testing > (which we often do already without any additional framework) You're asking why the tests should be in source packages rather than binary packages. The underlying point you seem to be missing is that a major aim of autopkgtest is that it should be straightforward to provide autopkgtest bindings for existing upstream test suites. So, firstly, binary packages have a much bigger overhead, and more complicated restrictions, compared to adding new material to source packages: - Binary packages get entries in a large number of databases both in our central infrastructure and on users' systems - Binary packages are highly visible in ways that test infrastructure doesn't need to be - They are relatively complicated to produce - They can only be installed in particular locations - Only one version can be present on a system at once (unless even more complicated things are done) The additional complexity and effort doesn't buy us anything, and the restrictions are pointless. Secondly: very often the tests will want to use other material from the source package. Upstream packages often come with tests, or strange example programs, or other material, which is useful for testing but not useful for users, and which the upstream package does not install. So if the tests were in binary packages, often we'd have to construct a weird binary package which contained all or part of the built source tree. This would be very ugly and also bulky. And additional violence would have to be done to the upstream test suite to try to make it work when "installed", rather than when run out of the source tree as it expects. This is probably going to be much harder than simply arranging for the tests to test the installed rather than built copy (if they don't already). Thirdly: having the tests in a source package makes it easier to run one version of the tests against a different version of the package. This is because we typically have different distribution channels and configuration for the sources of source packages, but also because there is a good chance that the dependencies are likely to be more relaxed. It would be quite reasonable to run the tests from unstable on the code in testing. That gets you quicker updates to the tests, and is something I definitely wanted to support. Fourthly: doing it this way makes the whole automated testing system more easily shareable with upstream and with non-Debian-derived distros. A different distro can use different metadata, but we can share the underlying tests. This is less true if our tests involve messing with "make install" and extra binary packages. So overall, it's easier this way. To counter your objections: > [with tests in binary packages one could do] > sudo apt-get install apache2-tests > adt-run apache2 We already have a good architecture and good machinery for installing sources. > To accomplish above with tests only in source packages, would > require apt-src like infrastructure (and do version/dependencies > tracking on them), We already have all of that infrastructure. I didn't have to write any substantial amount of new dependency-handling code. > and moreover why should I care to keep sources on my (user's) system > at all! I think that users who want to do testing can reasonably be asked to install the source code. Ian. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/19785.26347.347850.910...@chiark.greenend.org.uk
