Hi Renato, > by a non-trivial amount.
Why do you think it would be non trivial? Some simple lit tests aren't exactly arduous on most targets. James On Tue, 1 Dec 2015 at 19:39, Renato Golin <renato.go...@linaro.org> wrote: > On 1 December 2015 at 19:09, David Blaikie <dblai...@gmail.com> wrote: > > Not sure I follow - I'm not suggesting adding objdump/instruction > checking > > to existing large programs in the test-suite, but adding other tests to > the > > test-suite that do this and have appropriate input for it to be a > > reliable/useful form of testing. > > Sorry, I was building to that... But currently, the test-suite > compiles the code into an executable, then run it. Changing that > premise means we'll have something other than the test-suite we always > had, and may bring other problems, like too many hats for the > infrastructure to keep, especially now that James is moving it to > integrate with lit. We may decide this is the way to go, but we need > to make clear all the pros and cons of doing so beforehand. > > > > This is perhaps an interesting part/point: In theory I would think that > > these tests should be able to be phrased in a compiler-agnostic way, > since > > they're testing the compiler as a whole and testing behavior that the > source > > code demands, no? > > There's where things get interesting. Say we change the test-suite to > instead of compiling and running, just compile to asm/obj and > grep/objdump to match instructions, ELF sections, debug symbols, etc. > > Some tests are easy to make compiler independent. Intrinsics tests > fall into that category. Most currently in Clang tests could move > directly to the test-suite as soon as we have such infrastructure. > > Other tests, however, are specific to Clang+LLVM that may make no > sense to GCC or ICC, ARMCC etc. Those tests, would have to be in yet > another separate directory, that is only enabled if the compiler is > Clang, assuming the back-end is always LLVM. > > These are all options that, for me, have equal value with keeping them > in Clang. AFAICS, they stir the same kind of problems wherever they > are tested. > > > >> would account for most of those problems, but it would also put a > >> stress on slow/expensive/experimental hardware support. It's a > >> balance, I think. > > > > Why would it require slow/expensive/experimental hardware? (& if it does > > require that, how do you expect average developers to run them on a > regular > > basis?) > > You misunderstood. It would "put a stress on", not "require". > Basically, it would increase the cost of testing on those classes of > hardware more than off-the-shelf hardware by a non-trivial amount. > > This goes back to our discussion on buildbots, where some targets are > harder to find hardware than others, and when you do, they're slower. > Meaning the more variations we need to test, the more hardware you > have to have to make it in any reasonable time. While a single 64-core > Xeon can cope with most testing (including self-hosting, test-suite, > sanitizers, libc++) on a per-commit basis, there is nothing similar, > widely available, today for most other targets. > > But that's another discussion entirely, let's not dwell into that again. :) > > cheers, > --renato >
_______________________________________________ cfe-commits mailing list cfe-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
