On Tue, 19 May 2020 at 00:36, Jacob Bachmeyer <jcb62...@gmail.com> wrote: > > Rob Savoye wrote: > > On 5/18/20 2:16 PM, Jacob Bachmeyer wrote: > > > >> My work on DejaGnu is one of my hobbies. However, DejaGnu is a > >> relatively stable project, so I do not expect it to require much time. > >> > > > > To adequately test DejaGnu, you need to be able to build and test the > > GNU toolchain, both native and cross. That's necessary to test patches, > > or reproduce bugs. Once all that is setup, then yes testing patches > > isn't too time consuming. Building several toolchains and setting up > > cross testing is what sucks up most of the time. You don't need to be a > > toolchain developer, but you do need to be able to work in that environment. > > > What is the minimal set needed for sufficient testing? (Which > packages? GCC, GDB, ...? How many versions? Latest release/Past three > releases/All supported releases/Something else? I have/have access to > AMD64 (with 32-bit compatibility available); otherwise I will need to > ask for an account on the GCC Compile Farm Andrew Pinski mentioned.) > > How to run the testsuites with a testing version of DejaGnu? Is it as > simple as specifying > "RUNTEST=/path/to/testing/version/of/dejagnu/runtest" as an argument to I generally do: PATH=/path/to/testing-dejagnu:$PATH make check
IIRC, that's what ABE does too. > "make check"? Is running diff on the *.log (or *.sum) files after > running "make check" with both "old" and "new" DejaGnu sufficient to > verify a proposed release? > > > Luckily you can use my ABE tool (also GPLv3), which I built for > > Linaro, and it fully automates the process of building and testing > > native or cross toolchains. > It seems to have disappeared from the Linaro wiki... is it currently > posted anywhere? ABE is still there: https://git.linaro.org/toolchain/abe.git/ > > I agree you've been deeper into the DejaGnu > > code than anybody else these days, but unless you run the toolchain > > testsuites in a cross build from git, you can never really be sure if > > something doesn't break in obscure ways. Maybe others disagree with me > > on this, but testing the toolchain is the primary purpose of DejaGnu. > > > Are released versions sufficient or do the tests also need to be run > with Git checkouts of the toolchain packages? > > > I'm open to an additional co-maintainer if you can survive getting > > fully setup. ABE makes it easy, you just need a ton of disk space. While > > it is possible to add bug fixes or features without much toolchain > > testing, it's impossible to do a stable release without it. With > > DejaGnu, all releases are long-term support... and we will need to do a > > release after the patch backlog is caught up. > > > Part of the reason I am now involved with writing tests for DejaGnu > itself is to reduce this burden. Right now, the only way to know if a > change causes problems is to run numerous testsuites for other (large) > packages and "see if it breaks". > I'll let Rob/Ben comment on what is "sufficient" testing, I believe GDB is the most challenging. Christophe > > -- Jacob >
