Hello all, In case you haven't noticed it, I'd like to draw your attention to D32930, where we're proposing a new test framework for lldb-server tests. The discussion has so far been about low-level implementation details, so you don't have to read through it if you don't feel like to (but I do encourage it)
However, I'd like to explain some of the high-level motivations which led to our proposed design and open a discussion on them here. I'll do this in an FAQ form: :) - why new framework?: Lldb-server tests were never really suited for the dotest.py model anyway (for example they end up creating an SBDebugger, only to be completely ignoring it and opening a socket connection to lldb-server directly). Perhaps for this reason, they are also harder to write than usual lldb tests, and a lot more messy internally (e.g., after Chris's ipv6 patch, which caused all lldb-server connections in the test to fail, I've learned that the test harness will attempt the lldb-server connection 400(!!!) times before conceding defeat). The test suite operation is also very illogical when it comes to doing remote tests: to test lldb-server on a remote target, you first have to build lldb-server for the target, THEN you have to build lldb for the HOST, and THEN you run dotest.py in the HOST build folder while passing funny dotest.py arguments. - why lldb-server ?: We'd like this to be a first step in porting the existing test off of the dotest.py test runner. Unlike the full test suite, the number of lldb-server tests is not that big, so porting them is an task achievable in a not too long timeframe, and it can serve as a proof of concept when considering further steps. Also, lldb-server already performs a relatively well-defined and simple task, which means it fits the llvm model of testing isolated components of functionality without the need for a massive refactor. - why c++ (aka, if the existing test suite is broken, why not just fix it) ?: There are two fundamental issues with the current test suite which cannot be easily "fixed". The first one is the remote execution (which is where a large part of the test harness complexity comes from). By writing the test in c++ we can run the test *and* lldb-server remotely (***), avoiding the network communication and flakyness that comes with it. The other issue is the fact that it needs to have a completely independent reimplementation of the gdb-remote protocol. Sure, some duplication is expected from tests, but it does not have to be that big. If we write the test in c++ we can reuse parts of the gdb-remote client code (thereby increasing test coverage of that as well), and only resort to manual packet parsing when we really want to (e.g., when testing the server response to a malformed packet or similar). - ok, but why not have the test described by a text file, and have a c++ FileCheck-like utility which interprets it?: Due to the unpredictable (e.g. we cannot control the addresses of objects in the inferior), and interactive nature of the tests, we believe they would be easier to write imperatively, instead of a more declarative FileCheck style. E.g. for one test you need to send qRegisterInfoN for N=1,... until you find the pc register then pluck a field with that number from a stop-reply packet, and compare that the result of another packet, while possible reversing endianness. To describe something like this in a text file, you will either need primitives to describe loops, conditionals, etc (which will then tend towards implementing a full scripting language), or have a very high-level primitive operation which does exactly this, which will tend towards having many specialized primitive operations. regards, pavel (***) To achieve this, we want to propose adding the ability to execute tests remotely to llvm-lit, which we hope will be useful to more people. I'll write more about this in a separate email with llvm-dev included. _______________________________________________ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
