===================================================== Avocado Test Loader: Test types and extended status =====================================================
The current architecture of the Avocado Test Loader has a number of shortcomings. First, although not directly related to the major topic being present here, it's worth noticing that the various test loader implementations are still not implemented the same way that other extensible areas of Avocado are. Basically, it does *not* describe and interface at `avocado.core.plugin_interfaces`, which should be followed by the actual test loader implementations. This can be explained by the fact that the test loader implementation preceeds the "new" plugin architecture, and has not yet been ported. Given that this document will suggest changes to the Avocado Test Loader subsystem, it makes sense to also consider porting it to the "new" architeture. This task can will be tracked here: https://trello.com/c/H3hSd3Lm/1123-test-loader-adapt-implementation-to-the-new-plugin-architecture But the most urgent violation in the architecture is one the most visible to users: "test types" which are not really tests. Abuse of test types =================== A sample command execution can quickly show the current breakage when it comes to test types:: $ avocado list -V /dev/null Type Test NOT_A_TEST /dev/null: Not an INSTRUMENTED (avocado.Test based), PyUNITTEST (unittest.TestCase based) or SIMPLE (executable) test !GLIB /dev/null: No GLib-like tests found !GOLANG /dev/null: Go binary not found. !ROBOT /dev/null: Data source does not exist. TEST TYPES SUMMARY ================== !GLIB: 1 !GOLANG: 1 !ROBOT: 1 ACCESS_DENIED: 0 BROKEN_SYMLINK: 0 EXTERNAL: 0 GLIB: 0 GOLANG: 0 INSTRUMENTED: 0 MISSING: 0 NOT_A_TEST: 1 PYUNITTEST: 0 ROBOT: 0 SIMPLE: 0 A partial list of the problems that this output reveals: 1) `NOT_A_TEST` and `!ROBOT` (and many others) listed under "Type" are not really test types, but representations of the loader "findings" on a given test reference, which I'll call "test reference resolution result" from now on. The same applies to `ACCESS_DENIED`, `BROKEN_SYMLINK` and `MISSING`, which are listed as test types, but in fact are not. 2) `NOT_A_TEST` is a "test reference resolution result" that is only used for `INSTRUMENTED` and `SIMPLE` tests (although the given message only makes it clear for the INSTRUMENTED test type) 3) The "TEST TYPE SUMMARY" reports a total count of 2 for one given reference, not because two tests were found, but because they were *not* found by two different loaders (FileLoader and RobotLoader). One may argue that having well defined "test types" such as `ACCESS_DENIED` are useful so that the user can tell what was wrong with the loading of the test. But, the fact is that the same information can be given without categorizing it as a test type. Expected behavior ================= The following command output would be produced if the problems listed previously did not exist. Fist, if an only a test reference that can not be resolved into a valid test type were given, output similar to the following would be expected:: $ avocado list /dev/null Unable to resolve reference(s): '/dev/null' Resolution process results: =========================== Reference Plugin Result Info /dev/null file /dev/null yaml_testsuite /dev/null robot /dev/null external The key point here is that *no test* was found when a number of plugins attempted to find one. Plugins should be free to add additional information, such as:: $ avocado list /does/not/exist Unable to resolve reference(s): '/does/not/exist' Resolution process results: =========================== Reference Plugin Result Info /does/not/exist file FileNotFoundError: [Errno 2] No such file or directory /does/not/exist yaml_testsuite Failed to parse YAML file /does/not/exist robot /does/not/exist external Proposed Solution ================= Instead of delving into too much theory, a PR has been posted with a RFC implementation of a new resolver architecture. To avoid disruption, that PR introduces a "resolve" command (instead of "list") and hooks itself into the "nrunner" experimental test runner. Feedback to this and to the PR "[RFC] Introduce avocado.core.resolver as an alternative to loader" are extremely welcome.
