Thank you for this work. I will add a usual requirement - please make sure that "check-lldb-unit" works in standalone builds.
On 15.05.2017 16:33, Pavel Labath wrote: > Hello all, > > In case you haven't noticed it, I'd like to draw your attention to > D32930, where we're proposing a new test framework for lldb-server > tests. The discussion has so far been about low-level implementation > details, so you don't have to read through it if you don't feel like > to (but I do encourage it) > > However, I'd like to explain some of the high-level motivations which > led to our proposed design and open a discussion on them here. I'll do > this in an FAQ form: :) > > - why new framework?: > > Lldb-server tests were never really suited for the dotest.py model > anyway (for example they end up creating an SBDebugger, only to be > completely ignoring it and opening a socket connection to lldb-server > directly). Perhaps for this reason, they are also harder to write than > usual lldb tests, and a lot more messy internally (e.g., after Chris's > ipv6 patch, which caused all lldb-server connections in the test to > fail, I've learned that the test harness will attempt the lldb-server > connection 400(!!!) times before conceding defeat). The test suite > operation is also very illogical when it comes to doing remote tests: > to test lldb-server on a remote target, you first have to build > lldb-server for the target, THEN you have to build lldb for the HOST, > and THEN you run dotest.py in the HOST build folder while passing > funny dotest.py arguments. > > > - why lldb-server ?: > We'd like this to be a first step in porting the existing test off of > the dotest.py test runner. Unlike the full test suite, the number of > lldb-server tests is not that big, so porting them is an task > achievable in a not too long timeframe, and it can serve as a proof of > concept when considering further steps. Also, lldb-server already > performs a relatively well-defined and simple task, which means it > fits the llvm model of testing isolated components of functionality > without the need for a massive refactor. > > - why c++ (aka, if the existing test suite is broken, why not just fix it) ?: > There are two fundamental issues with the current test suite which > cannot be easily "fixed". The first one is the remote execution (which > is where a large part of the test harness complexity comes from). By > writing the test in c++ we can run the test *and* lldb-server remotely > (***), avoiding the network communication and flakyness that comes > with it. The other issue is the fact that it needs to have a > completely independent reimplementation of the gdb-remote protocol. > Sure, some duplication is expected from tests, but it does not have to > be that big. If we write the test in c++ we can reuse parts of the > gdb-remote client code (thereby increasing test coverage of that as > well), and only resort to manual packet parsing when we really want to > (e.g., when testing the server response to a malformed packet or > similar). > > - ok, but why not have the test described by a text file, and have a > c++ FileCheck-like utility which interprets it?: > Due to the unpredictable (e.g. we cannot control the addresses of > objects in the inferior), and interactive nature of the tests, we > believe they would be easier to write imperatively, instead of a more > declarative FileCheck style. E.g. for one test you need to send > qRegisterInfoN for N=1,... until you find the pc register then pluck a > field with that number from a stop-reply packet, and compare that the > result of another packet, while possible reversing endianness. To > describe something like this in a text file, you will either need > primitives to describe loops, conditionals, etc (which will then tend > towards implementing a full scripting language), or have a very > high-level primitive operation which does exactly this, which will > tend towards having many specialized primitive operations. > > regards, > pavel > > (***) To achieve this, we want to propose adding the ability to > execute tests remotely to llvm-lit, which we hope will be useful to > more people. I'll write more about this in a separate email with > llvm-dev included. >
signature.asc
Description: OpenPGP digital signature
_______________________________________________ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
