If we want to add a testsuite runner which takes a source file, a place to put a breakpoint, the name of a variable to examine, and it runs through those in SB API, I'm all in favor. I don't know if that's going to add a lot of test coverage to lldb, but I have no problem with such a thing.
My goal isn't to make test case writing hard. My goal is to make the testsuite a benefit to the project, instead of a boat anchor. My non-goal is what testsuite harness is used to run the SB API tests - whether it's lit or whatever, I don't care. > On Sep 14, 2016, at 11:19 PM, Jason Molenda <jmole...@apple.com> wrote: > > It's great to make writing tests easier. We'd all love to have more tests. > > If "writing tests easier" is "command line output scraping", that's only > hurting the project in the long term. I'm telling you this from years of > experience on a project where we did exactly this. It's great to have > examples where we're testing that "p 5" prints 5. That's awesome and I'm > sure it won't break. But that's not more than a tiny fraction of what you > need to test in a debugger. > > It's important to separate "let's use lit" and "let's write command line > scraping tests", unless lit means "command line scraping tests", in which > case it's fine to conflate the two. No one is going to be up in arms about > lit as a test harness -- but using that as a way to get lots of command line > scraping tests added to lldb is disingenuous. We've had this discussion many > times on the mailing lists, and all of us who had to work on a scraping > testsuite have pushed back hard against the siren song of these tests because > we've learned what a big mistake it is. > > Talking about python 2 v python 3 compatibility or speed of the testsuite -- > these are implementation side issues that I don't have an opinion on. If we > want to write our SB API in a different way to solve those issues -- want to > write the tests in C++? Fine, whatever. I think that would just make it > slower to write new tests. Can tests in lit use the SB API exclusively? > Then I don't care about adding lit. > > I remain firmly against command line scraping tests, it only adds debt to the > project long term. It's a lesson learned from gdb that bears not repeating. > > J > > >> On Sep 14, 2016, at 9:36 PM, Zachary Turner <ztur...@google.com> wrote: >> >> There's also a cost to *not* writing test cases, which is paid >> proportionally to how difficult it is to write test cases. >> >> That said, aren't the existing lldbinline tests an example of the behavior >> you're objecting to? And that is a relatively recent addition which I have >> never seen or heard of any objections to. >> >> On Wed, Sep 14, 2016 at 9:07 PM Jason Molenda <jmole...@apple.com> wrote: >> >> On Sep 14, 2016, at 5:13 PM, Zachary Turner <ztur...@google.com> wrote: >> >>> I don't blame you for being scared of command tests. I don't support their >>> use in the current LLDB test suite either, for exactly the same reasons you >>> and Jason have expressed. But I do think it's possible to come up with >>> something that a) doesn't suffer from the same problems, b) allows testing >>> a ton of extra functionality that is not currently testable through the >>> api, and c) doesn't rely on python at all. If I'm wrong I'll eat crow :) >> >> >> I think your examples are reductive to "printing values is easy to keep >> consistent" -- but that's only a small part of what a debugger testsuite >> needs to do. >> >> If we want to use lit to drive tests written in terms of SB API, then I'm >> open to seeing how this tool can be used in lldb. >> >> If lit can only be used to write command output tests, then I believe, based >> on years and years of experience with exactly that kind of test suite in >> with gdb, that this is a very poor addition to lldb. It will only hamper >> future lldb development as we want to improve the lldb command line UI - as >> it did with gdb. Every change to the command line UI breaks hundreds of >> "easy to write" tests. And who is responsible for cleaning all those up? >> The people who wrote these easy-to-write command line output matching tests? >> No, it'll be the person trying to improve the debugger, which is a big >> disincentive to changing anything else you'll need to wade through the swamp >> of thousands of accumulated test cases. >> >> There's a cost to writing a test case. Either it is paid up front when you >> write the test in terms of SB API, or it is paid repeatedly over time as >> people change the UI to the command line tool. >> >> I have nothing against lit per se, I'm open to seeing what an SB API based >> test case in that harness looks like. I have nothing but objections to >> adding significant new additions to the testsuite that are locked to the >> command line UI behavior of lldb today. >> >> >> J > _______________________________________________ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
