I think the discussion has veered of from the "lldb-server testing" topic into the more murky "general testing" waters. I'll put my thoughts on those first, and then get back to lldb-server test plans.
On 31 May 2017 at 18:37, Zachary Turner via Phabricator <revi...@reviews.llvm.org> wrote: > What *would* help would be to ignore how difficult it may or may not be, and > just take a couple of tests and re-write them in some DSL that you invent > specifically for this purpose that is as concise as possible yet as > expressive as you need, and we go from there. I did this with a couple of > fairly hairy tests a few months ago and it didn't seem that bad to me. All DSL examples I've seen from you were toy-ish in one aspect or another and I could not see a straightforward (or even a complicated) way to extend them to handle some of the hairy test scenarios. If you want to see a more complicated example look at <https://reviews.llvm.org/D33426>, which has been crawling for a couple of weeks, large part of it due to the complexity of coming up with a reliable test vector. I don't see a way to create a reasonable DSL for this test -- you'd either have to add loops and conditions and very fine-grained primitives for accessing SBAPI (at which point you might as well use SBAPI itself), or you need to introduce a very high level primitive operation, which would drive the threads to a desired state (and end up with a very large number of primitives most of which are used only in a single test). What I think that makes writing a lot of the tests difficult is the fact that all of the tests are responsible for building their own executables and driving them to the state where you want to perform the actual test. This makes the test very undeterministic and environment-dependent. - to run a libstdc++ data formatter test, you need to have a system capable of running libstdc++ and have the library installed. To write a test that verifies that the data formatter handles corrupt STL data structures reasonably, you need to write some very non-standard-compliant code to corrupt them yourself. And if the STL structure layout changes, and you want your data formatter to handle both layouts, you cannot write tests that will verify that both of them still work. - to write a test for the handling of some DWARF corner case, you need to figure out how to coax the compiler into producing the dwarf you want. And even then, you have no guarantee that when you update the compiler, it will still produce the same dwarf that you were interested in testing. - etc. Now, I think you actually agree with most of this. What differs (I believe) is the conclusion we draw from it. My conclusion is that there is no point in trying to invent a DSL yet. A lot of other things need to happen first. For example, my ideal testing strategy for data formatters would not involve running a process *at all*. All the data formatters (at least the sane ones which do not evaluate expressions) need is something that provides contents of some memory and a bit of debug info. This is pretty much what a core file does. However, we're not ready to start using core files everywhere yet, because we don't have a reasonable way to generate small cores (they can be small, if you try enough, but it's not easy), and checking in binaries is gross (so we need to have a textual representation of them and tool for converting to/from it). At that point it would be easy to write a DSL for data formatter tests. For example, you could have a yaml description of the state of the stl container and a tool which ingests that and spits out a "fake" core file. Then you load that core file in lldb, run a bunch of "frame variable" commands and FileCheck their output. No run commands necessary, no makefiles, and the test will work on every platform. However, this requires a lot of prep work, and I don't see anyone committing to doing that right now. As for lldb-server tests, I think the case for a lldb-server DSL is much weaker than that for many other things that lldb does. That DSL would need to handle all of the 100 or so packets that the program supports, and need to be able to construct them using quite complicated expressions. It's possible, but I don't believe it's worth the effort right now. You're welcome to try, I can give you the requirements (first one: it needs to be able to run remotely), but we're not going to attempt it right now. We hope to address a couple of other problematic issues with the new test suite (complicated test framework, Makefiles, code duplication in the tests, paralelization problems) and I believe any future DSL for these tests can build on this by providing adding a front-end to the test client we are adding here. _______________________________________________ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
