The recent and ongoing discussion over is_remote (and why is that the
only is* predicate with an underscore in the name?) and its
less-than-entirely-accurate implementation started me thinking about the
topic. What would be a better way to handle remote testing?
We will need to preserve backwards compatibility for testsuites, or at
least provide for a transition period to move to the new model.
Ideally, the only changes affecting testsuites should be that existing
workarounds become unnecessary going forward. At the very least, the
GDB workaround Pedro Alves mentioned must not break the new implementation.
Test configuration (global init files) need not remain backwards
compatible -- a site can update their init files when installing the
upgraded DejaGnu. Ideally, "old model" configuration should be
automatically migrated in memory upon load.
A rough sketch to start a discussion for a future solution:
Add an abstraction layer (that probably will support existing API calls
-- I *think* DejaGnu already has this) that wraps a logical connection.
Remote testing uses a "transfer" connection
(ftp/kermit/tftp/sftp/scp/rcp/rsync/etc.) and a "control" connection
(telnet/rsh/ssh/kermit/snmp/etc.). Local testing uses a "local-unix" or
"local-posix" connection that provides both interfaces, but implements
"transfer" as no-ops. Ideally, testsuites should not care whether they
are running in local or remote mode.
We want to support Canadian cross environments with a separate test
driver machine (so possibly four different platforms, but DejaGnu itself
is relatively platform-independent, needing only a POSIX host) and any
combination of build/host/target for toolchain testing. Note that
multiarch machines are likely to be the future: AMD64 already does this
today, and RISC-V systems could have all of RV32/RV64/RV128 on the same
machine, all with varying extensions either by configuration or
heterogeneous hardware. Cross toolchains are likely to increase in
importance. So we have toolchain "build"/"host"/"target" and a separate
DejaGnu "driver" in the picture, all of which can be same or different,
so a testsuite could use up to three remote connections.
This leads to a few scenarios off the top of my head that DejaGnu needs
to support:
1. Toolchain: compilers. These (notably GCC but possibly others as
well) can need the full complexity of the system, with possible need to
build the tested program on one remote system, transfer the executables
to another remote system, use the executables on "host" to produce test
programs, transfer the test programs to a third remote system, and run
the test programs on "target". The simplifying factor here is than
compilers are batch programs, so interaction is limited.
2. Toolchain: debuggers. These (probably only GDB, but possibly
others for specific languages) can be as complex as compilers, but are
also interactive programs for which the tests can depend on a compiler
to build test programs on "host" for "target". The debugger itself also
runs on "host" (possibly a different "host" from the associated
compiler!) and connects (by some means independent of DejaGnu) to
"target" for debugging the test programs.
3. Unit testing. These are probably less complex, since there is only
a need to build (possibly on a remote "build") the unit test programs,
transfer them to the "host", run them on the "host", and collect the
results. Small complexities are introduced when gcov(1) or similar
tools are used to validate unit test coverage.
4. General system testing. These are again similar to unit testing:
programs are built on "build", transferred, and run on "host". These
are to unit tests as debuggers are to compilers: general system testing
is likely to be "interactive" rather than specific "run this program".
Debuggers are particularly interesting, since they potentially introduce
fifth and sixth(!) systems into the picture: the "build" and "host"
systems for the associated compiler, which could be different from the
"build" and "host" systems for the debugger. This further expands if
there are multiple associated compilers. This suggests that we should
split build/host/target into a hierarchical structure, with plain
build/host/target as defaults for the "main tool" being tested, but
allowing "subtools" (like the compiler in the case of a debugger) to
have their own build/host/target sets. This could be particularly
relevant for multi-arch systems, where a compiler notionally built for
one target may well be capable of producing output either directly for
or that is compatible with another target. (I expect RISC-V to be able
to make extensive use of this eventually.)
The possibility of needing remote builds raises a question of scope: on
one hand, it is easy to say that building tested programs is
out-of-scope for DejaGnu, but on the other hand, DejaGnu already needs
most of the relevant functionality in order to support remote testing of
compilers. The incremental code is small: a routine that uses the
already-needed remote connection infrastructure to transfer the sources
to a build machine, run "configure" and "make", and collect the output
binaries.
The complexity associated with these changes is probably unmanageable
with DejaGnu's current design, which is why I advocate for moving the
DejaGnu core and core libraries out of the global Tcl namespace. (Doing
that (backwards compatibly) is on my local TODO list, by the way.)
Eventually, (far future) DejaGnu could offer a "strict" mode, where test
scripts are run in safe interpreters with access to only the DejaGnu
API. If the API is properly designed, a testsuite that runs in "strict"
mode should be independent of local or remote configuration. Depending
on the details of the GDB testsuite, "strict" mode may also require some
notion of "test groups" -- some test scripts being applicable only on
certain "boards".
Lastly, it unlikely that this discussion would even be possible if the
existing DejaGnu, with all its limitations, had not been written.
Thanks to Rob and Cygnus for laying that groundwork.
-- Jacob
_______________________________________________
DejaGnu mailing list
DejaGnu@gnu.org
https://lists.gnu.org/mailman/listinfo/dejagnu
