bug#23521: XFAIL
On 19 May 2016 at 00:04, Mathieu Lirzin wrote: >> It is often easier to write expected-to-fail tests this way (so that >> they can all look the same), rather than have to have, for example, an >> extra driver that converts expected errors into success codes for the >> automake test harness. > > What do you mean precisely by “an extra driver”? This would be a reference to a "custom test driver". https://www.gnu.org/software/automake/manual/html_node/Overview-of-Custom-Test-Drivers-Support.html#Overview-of-Custom-Test-Drivers-Support
bug#23521: XFAIL
On 20 May 2016 at 15:58, Gavin Smith wrote: > On 19 May 2016 at 00:04, Mathieu Lirzin wrote: > >> It is often easier to write expected-to-fail tests this way (so that > >> they can all look the same), rather than have to have, for example, an > >> extra driver that converts expected errors into success codes for the > >> automake test harness. > > > > What do you mean precisely by “an extra driver”? > > This would be a reference to a "custom test driver". > > https://www.gnu.org/software/automake/manual/html_node/Overview-of-Custom-Test-Drivers-Support.html#Overview-of-Custom-Test-Drivers-Support > Thanks, that's what I meant. -- http://rrt.sc3d.org
bug#23521: XFAIL
Reuben Thomas writes: > On 19 May 2016 at 00:04, Mathieu Lirzin wrote: > > > It is often easier to write expected-to-fail tests this way (so > that > > they can all look the same), rather than have to have, for > example, an > > extra driver that converts expected errors into success codes > for the > > automake test harness. > > What do you mean precisely by “an extra driver”? > > > A custom test driver. OK, I wasn't sure. Indeed a custom test driver seems a bit heavy just checking failures. IMO the solution Peter proposed is nice and simple. -- Mathieu Lirzin
bug#23521: XFAIL
On 20 May 2016 at 16:49, Mathieu Lirzin wrote: > Reuben Thomas writes: > > > On 19 May 2016 at 00:04, Mathieu Lirzin wrote: > > > > > It is often easier to write expected-to-fail tests this way (so > > that > > > they can all look the same), rather than have to have, for > > example, an > > > extra driver that converts expected errors into success codes > > for the > > > automake test harness. > > > > What do you mean precisely by “an extra driver”? > > > > > > A custom test driver. > > OK, I wasn't sure. Indeed a custom test driver seems a bit heavy just > checking failures. IMO the solution Peter proposed is nice and simple. > What Peter proposed is essentially a custom test driver: I would not expect to duplicate the logic to check the return code &c. in each test expected to fail; rather, I would put it in a custom test driver that would handle expected fails and mark them as passes. (My expected fails are all of the same type, i.e. a non-zero exit code. It might additionally be useful, as Peter suggests, to check that an expected error message is produced.) -- http://rrt.sc3d.org
