Re: Automated testing - design and interfaces

2005-11-24 Thread Ian Jackson
Anthony Towns writes ("Re: Automated testing - design and interfaces"): > On Mon, Nov 21, 2005 at 06:22:37PM +, Ian Jackson wrote: > > This is no good because we want the test environment to be able to > > tell which tests failed, so the test cases have to be e

Re: Automated testing - design and interfaces

2005-11-23 Thread Robert Collins
On Wed, 2005-11-23 at 18:16 +1000, Anthony Towns wrote: > On Mon, Nov 21, 2005 at 06:22:37PM +, Ian Jackson wrote: > > > Note that it's often better to have a single script run many tests, so > > > you probably want to allow tests to pass back some summary information, > > > or include the last

Re: Automated testing - design and interfaces

2005-11-23 Thread Anthony Towns
On Mon, Nov 21, 2005 at 06:22:37PM +, Ian Jackson wrote: > > Note that it's often better to have a single script run many tests, so > > you probably want to allow tests to pass back some summary information, > > or include the last ten lines of its output or similar. Something like: > > foo F

Re: Automated testing - design and interfaces

2005-11-21 Thread Ian Jackson
Anthony Towns writes ("Re: Automated testing - design and interfaces"): > On Thu, Nov 17, 2005 at 06:43:32PM +, Ian Jackson wrote: > > The source package provides a test metadata file debian/tests/ > > control. This is a file containing zero or more RFC822-style &

Re: Automated testing - design and interfaces

2005-11-20 Thread Robert Collins
On Thu, 2005-11-17 at 14:36 -0800, Steve Langasek wrote: > [let's get this over to a technical list like it was supposed to be ;)] > > Following your exit status based approach you could add to stanzas > > something like: > > > Expected-Status: 0 > > > I found the above requirement the very mi

Re: Automated testing - design and interfaces

2005-11-18 Thread Stefano Zacchiroli
On Thu, Nov 17, 2005 at 02:36:06PM -0800, Steve Langasek wrote: > FWIW, I don't see that there's a clear advantage to having the test harness > *expect* non-zero exit values (or non-empty output as you also suggested). As I understood it, the proposed approach is about standardizing and easing the

Re: Automated testing - design and interfaces

2005-11-17 Thread Anthony Towns
Bcc'ed to -project; followups to -devel. On Thu, Nov 17, 2005 at 06:43:32PM +, Ian Jackson wrote: > Note that the point is to be able to test the _actual package_, _as > installed_ (eg on a testbed system). This is much better than testing > the package from the source treeu during build time

Re: Automated testing - design and interfaces

2005-11-17 Thread Steve Langasek
[let's get this over to a technical list like it was supposed to be ;)] On Thu, Nov 17, 2005 at 10:43:34PM +0100, Stefano Zacchiroli wrote: > On Thu, Nov 17, 2005 at 06:43:32PM +, Ian Jackson wrote: > > This means execute debian/tests/fred, debian/tests/bill, etc., > > each with no argumen