On Tue, Dec 1, 2015 at 10:43 AM, Renato Golin <renato.go...@linaro.org> wrote:
> On 1 December 2015 at 18:29, David Blaikie <dblai...@gmail.com> wrote: > > Are they things the test-suite couldn't (either technically or > > philosophically) cover, or only that it doesn't cover it at the moment, > but > > could do so? > > IMO, it's a philosophical issue. The test-suite is a whole-program > execution, and all that matters is the end result, the output of the > program, not the program itself. > > Those tests are mostly large (> 3 lines of code) so, if we were trying > to objdump and match for instruction patterns, any optimisation > variance could destroy them, as well as you may find the pattern > elsewhere in the same object, not belonging to the function you want > (inlining, etc). Not sure I follow - I'm not suggesting adding objdump/instruction checking to existing large programs in the test-suite, but adding other tests to the test-suite that do this and have appropriate input for it to be a reliable/useful form of testing. > All in all, all the problems with reducing C code > would be pertinent here, so a big hammer to kill small issues. On the > other hand, the Clang based asm/obj examples are all small, and > hand-crafted to suit the problem being tested. > > We could, however, add a new directory in the test-suite for that kind > of tests, like we already have for regressions in C/C++. If a separate directory makes it more clear what the purpose is, sure, seems fine to me, but I don't have strong or informed opinions on exactly how the test-suite is laid out. > That would > dissociate the front-end from the back-end, and essentially any > compiler used would have to abide by the rules of the test (which may > not be true for some compilers). > This is perhaps an interesting part/point: In theory I would think that these tests should be able to be phrased in a compiler-agnostic way, since they're testing the compiler as a whole and testing behavior that the source code demands, no? > I don't mind where they end up, really, but we need that kind of test > somewhere. Clang is an easy place because we know we have it built > when the Clang repo is checked out. > > > > That would sort of defeat the point of having the testing and projects > > separated though - it would tie the tests together and produce most of > the > > undesirable outcomes of having single end-to-end tests. > > Right, I agree, but since the relationship between Clang and LLVM is > non-trivial, and there are lots of changes that need to be done on > both sides, I can't see why we shouldn't have tests that span across > projects. > > Clearly, the sanitizer tests need Clang and Compiler-RT. Some Libc++ > tests need Compiler-RT, others need libunwind, on ARM I need Clang to > build it. There are a lot of dependencies that are not there by > accident, but by design. > > The more you move tests away from the big targets (LLVM, Clang), the > less they're ran by people committing patches, and the harder will be > to pick up the failures. Having said that, a decent buildbot coverage > would account for most of those problems, but it would also put a > stress on slow/expensive/experimental hardware support. It's a > balance, I think. > Why would it require slow/expensive/experimental hardware? (& if it does require that, how do you expect average developers to run them on a regular basis?) > > > > (it would mean that at least the pure (or at least non-Clang) LLVM > developer > > would have a test to run where they would not if the test remained in > Clang > > only) > > That is a good point. Right now, RT/libc++ tests in that category are > controlled by enabling/disabling support via CMake flags. We may be > able to do the same with Clang (if LLVM is not built with it, disable > asm/obj tests). Moving them all to a regression package in the > test-suite would be another option, but one that would take a lot > longer... > > cheers, > --renato >
_______________________________________________ cfe-commits mailing list cfe-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
