Re: [lldb-dev] LLDB finish_swig and Python module installation broken for multi-configuration generators
On Thu, 2019-10-10 at 23:11 +, Ted Woodward wrote: > Starting in r372835 (Fix installing Python modules on systems using > /usr/lib), the finish_swig and installation steps using multi-configuration > generators (like Visual Studio or XCode) are broken. > > The problem is the symlink and install steps look for liblldb and > site-packages the wrong directories. > > For the symlink, I get this output: > Making symlink from ..\..\..\bin\liblldb.dll to > i:\obj\lib\site-packages\lldb\_lldb.pyd > ... > WindowsError: [Error 2] The system cannot find the file specified: > '..\\..\\..\\bin\\liblldb.dll' > > Because the cwd is I:\obj\Lib\site-packages\lldb, not the correct > I:\obj\Release\Lib\site-packages\lldb. liblldb.dll is in i:\obj\Release\bin, > not i:\obj\bin. Also, the link itself should be in i:\obj\Release\lib, not > i:\obj\lib. > > The problem is the cmake code went from using LLVM_LIBRARY_OUTPUT_INTDIR to > CMAKE_BINARY_DIR. CMAKE_BINARY_DIR is not complete for multi-configuration > generators - you need to specify the configuration as well. > [...] > Michal, please try this and see if it still works for the issue you fixed in > r372835. Actually, I see that Haibo has already addressed this 2 days ago. Could you test if his solution (r374250) works for you? -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part ___ lldb-dev mailing list lldb-dev@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] [Openmp-dev] [cfe-dev] [llvm-dev] RFC: End-to-end testing
"Robinson, Paul via Openmp-dev" writes:
> David Greene, will you be at the LLVM Dev Meeting? If so, could you sign
> up for a Round Table session on this topic? Obviously lots to discuss
> and concerns to be addressed.
That's a great idea. I will be there. I'm also hoping to help run a
routable on complex types so we'll need times that don't conflict. What
times work well for folks?
> (1) Executable tests. These obviously require an execution platform; for
> feasibility reasons this means host==target and the guarantee of having
> a linker (possibly but not necessarily LLD) and a runtime (possibly but
> not necessarily including libcxx). Note that the LLDB tests and the
> debuginfo-tests project already have this kind of dependency, and in the
> case of debuginfo-tests, this is exactly why it's a separate project.
Ok. I'd like to learn more about debuginfo-tests and how they're set
up.
> (2) Non-executable tests. These are near-identical in character to the
> existing clang/llvm test suites and I'd expect lit to drive them. The
> only material difference from the majority(*) of existing clang tests is
> that they are free to depend on LLVM features/passes. The only difference
> from the majority of existing LLVM tests is that they have [Obj]{C,C++} as
> their input source language.
Right. These are the kinds of tests I've been thinking about.
-David
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] [cfe-dev] [llvm-dev] RFC: End-to-end testing
Renato Golin via cfe-dev writes: > On Thu, 10 Oct 2019 at 22:26, David Greene wrote: >> That would be a shame. Where is test-suite run right now? Are there >> bots? How are regressions reported? > > There is no shame in making the test-suite better. That's not what I meant, sorry. I mean it would be a shame not to be able to put end-to-end tests next to the code they test. Tests that are separated from code either tend to not get written/committed or tend to not get run pre-merge. > We do have bots running them in full CI for multiple targets, yes. > Regressions are reported and fixed. The benchmarks are also followed > by a smaller crowd and regression on those are also fixed (but > slower). How are regressions reported? On llvm-dev? > I'm not proposing to move e2e off to a dark corner, I'm proposing to > have a scaled testing strategy that can ramp up and down as needed, > without upsetting the delicate CI and developer balance. I don't think I quiet understand what you mean. CI is part of development. If tests break we have to fix them, regardless of where the tests live. > Sure, e2e tests are important, but they need to catch bugs that the > other tests don't catch, not being our front-line safety net. Of course. > A few years back there was a big effort to clean up the LIT tests from > duplicates and speed up inefficient code, and a lot of tests are > removed. If we just add the e2e today and they never catch anything > relevant, they'll be the next candidates to go. I'm confused. On the one hand you say you don't want to put e2e tests in a dark corner, but here you speculate they could be easily removed. Presumably a test was added because there was some failure that other tests did not catch. It's true that once a test is fixed it's very likely it will never break again. Is that a reason to remove tests? If something end-to-end breaks in the field, it would be great if we could capture a component-level test for it. That would be my first goal. I agree it can be tempting to stop at an e2e test level and not investigate further down. We definitely want to avoid that. My guess is that over time we'll gain experience of what a good e2e test is for the LLVM project. > The delta that e2e can test is really important, but really small and > fairly rare. So running it less frequent (every few dozen commits) > will most likely be enough for anything we can possibly respond to > upstream. I think that's probably reasonable. > Past experiences have, over and over, shown us that new shiny CI toys > get rusty, noisy, and dumped. I don't think e2e testing is shiny or new. :) > We want to have the tests, in a place anyone can test, that the bots > *will* test periodically, and that don't annoy developers often enough > to be a target. What do you mean by "annoy?" Taking too long to run? > In a nutshell: > * We still need src2src tests, to ensure connection points (mainly > IR) are canonical and generic, avoiding hidden contracts Yes. > * We want the end2end tests to *add* coverage, not overlap with or > replace existing tests Yes, but I suspect people will disagree about what constitutes "overlap." > * We don't want those tests to become a burden to developers by > breaking on unrelated changes and making bots red for obscure reasons Well, tests are going to break. If a test is too fragile it should be fixed or removed. > * We don't want them to be a burden to our CI efforts, slowing down > regular LIT testing and becoming a target for removal I certainly think less frequent running of tests could help with that if it proves to be a burden. > The orders of magnitude for number of commits we want to run tests are: > * LIT base, linker, compiler-RT, etc: ~1 > * Test-suite correctness, end-2-end: ~10 > * Multi-stage build, benchmarks: ~100 > > We already have that ratio (somewhat) with buildbots, so it should be > simple to add e2e to the test suite at the right scale. Would it be possible to keep them in the monorepo but have bots that exercise those tests at the test-suite frequency? I suspect that if e2e tests live in test-suite very few people will ever run them before merging to master. >> > The last thing we want is to create direct paths from front-ends to >> > back-ends and make LLVM IR transformation less flexible. >> >> I'm not sure I follow. Can you explain this a bit? > > Right, I had written a long paragraph about it but deleted in the > final version of my email. :) > > The main point is that we want to avoid hidden contracts between the > front-end and the back-end. > > We want to make sure all front-ends can produce canonical IR, and that > the middle-end can optimise the IR and that the back-end can lower > that to asm in a way that runs correctly on the target. As we have > multiple back-ends and are soon to have a second official front-end, > we want to make sure we have good coverage on the multi-step tests > (AST to IR, IR to asm, etc). A
Re: [lldb-dev] [cfe-dev] [llvm-dev] RFC: End-to-end testing
Hi David, You answer some of your own questions down below, so I'll try to collate the responses and shorten my reply. On Fri, 11 Oct 2019 at 15:20, David Greene wrote: > How are regressions reported? On llvm-dev? They're buildbots, exactly like any other. Direct email, llvm-commits, irc, bugzilla. There is no distinction, broken bots need to be fixed. llvm-dev is not the place to report bugs. > I'm confused. On the one hand you say you don't want to put e2e tests > in a dark corner, but here you speculate they could be easily removed. > Presumably a test was added because there was some failure that other > tests did not catch. It's true that once a test is fixed it's very > likely it will never break again. Is that a reason to remove tests? Sorry, my point is about the dynamics between number of tests, their coverage, time to run, frequency of *unrelated* breakage, etc. There are no set rules, but there is a back-pressure as developers and bot owners tend to breakages. > What do you mean by "annoy?" Taking too long to run? Tests that break more often are looked at more often, and if their breakages overlap with other simpler tests, than developers will begin to question their importance. Tests that take too long to run will be looked into, and if they don't add much, they can be asked for removal. That pressure is higher in the LIT side than in the test-suite. I'm trying to find a place where we can put the tests that will run at the appropriate frequency and have the lowest probability of upsetting CI and developers, so we can evolve them into what they *need* to be, not cap it from the start and end up with something sub-par. > Would it be possible to keep them in the monorepo but have bots that > exercise those tests at the test-suite frequency? I suspect that if e2e > tests live in test-suite very few people will ever run them before > merging to master. Bots are pretty dumb: either they run something or they don't. But more importantly, if we split the e2e tests in LIT, then people won't run them before merging to master anyway. Truth is, we don't *need* to. That's the whole point of having a fast CI and the community really respects that. As long as we have the tests running every few dozen commits, and bot owner and developers work to fix them in time, we're good. Furthermore, the test-suite already has e2e tests in there, so it is the right place to add more. We can have more control of which tools and libraries to use, how to check for quality, etc. > I still think the kinds of e2e tests I'm thinking of are much closer to > the existing LIT tests in the monorepo than things in test-suite. I > expect them to be quite small. Adding tests to LIT means all fast bots will be slower. Adding them to the test-suite means all test-suite bots will still take the about same time. If the tests only need to be ran once ever few dozen commits, then having them on LIT is clearly not the right place. > They wouldn't necessarily need to run as > part of check-all (and indeed, I've been told that no one runs check-all > anyway because it's too fragile). check-all doesn't need to check everything that is in the repo, but everything that is built. So if you build llvm+clang, then you should *definitely* check both. "make check" doesn't do that. With the monorepo this may change slightly, but we still need a way to test everything that our patches touch, including clang, rt, and others. I always ran check-all before every patch, FWIW. ___ lldb-dev mailing list lldb-dev@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] [cfe-dev] [llvm-dev] RFC: End-to-end testing
On Thu, Oct 10, 2019 at 2:21 PM David Greene via cfe-dev < cfe-...@lists.llvm.org> wrote: > Florian Hahn via llvm-dev writes: > > >> - Performance varies from implementation to implementation. It is > >> difficult to keep tests up-to-date for all possible targets and > >> subtargets. > > > > Could you expand a bit more what you mean here? Are you concerned > > about having to run the performance tests on different kinds of > > hardware? In what way do the existing benchmarks require keeping > > up-to-date? > > We have to support many different systems and those systems are always > changing (new processors, new BIOS, new OS, etc.). Performance can vary > widely day to day from factors completely outside the compiler's > control. As the performance changes you have to keep updating the tests > to expect the new performance numbers. Relying on performance > measurements to ensure something like vectorization is happening just > isn't reliable in our experience. Could you compare performance with vectorization turned on and off? > > > With tests checking ASM, wouldn’t we end up with lots of checks for > > various targets/subtargets that we need to keep up to date? > > Yes, that's true. But the only thing that changes the asm generated is > the compiler. > > > Just considering AArch64 as an example, people might want to check the > > ASM for different architecture versions and different vector > > extensions and different vendors might want to make sure that the ASM > > on their specific cores does not regress. > > Absolutely. We do a lot of that sort of thing downstream. > > >> - Partially as a result, but also for other reasons, performance tests > >> tend to be complicated, either in code size or in the numerous code > >> paths tested. This makes such tests hard to debug when there is a > >> regression. > > > > I am not sure they have to. Have you considered adding the small test > > functions/loops as micro-benchmarks using the existing google > > benchmark infrastructure in test-suite? > > We have tried nightly performance runs using LNT/test-suite and have > found it to be very unreliable, especially the microbenchmarks. > > > I think that might be able to address the points here relatively > > adequately. The separate micro benchmarks would be relatively small > > and we should be able to track down regressions in a similar fashion > > as if it would be a stand-alone file we compile and then analyze the > > ASM. Plus, we can easily run it and verify the performance on actual > > hardware. > > A few of my colleagues really struggled to get consistent results out of > LNT. They asked for help and discussed with a few upstream folks, but > in the end were not able to get something reliable working. I've talked > to a couple of other people off-list and they've had similar > experiences. It would be great if we have a reliable performance suite. > Please tell us how to get it working! :) > > But even then, I still maintain there is a place for the kind of > end-to-end testing I describe. Performance testing would complement it. > Neither is a replacement for the other. > > >> - Performance tests don't focus on the why/how of vectorization. They > >> just check, "did it run fast enough?" Maybe the test ran fast enough > >> for some other reason but we still lost desired vectorization and > >> could have run even faster. > >> > > > > If you would add a new micro-benchmark, you could check that it > > produces the desired result when adding it. The runtime-tracking > > should cover cases where we lost optimizations. I guess if the > > benchmarks are too big, additional optimizations in one part could > > hide lost optimizations somewhere else. But I would assume this to be > > relatively unlikely, as long as the benchmarks are isolated. > > Even then I have seen small performance tests vary widely in performance > due to system issues (see above). Again, there is a place for them but > they are not sufficient. > > > Also, checking the assembly for vector code does also not guarantee > > that the vector code will be actually executed. So for example by just > > checking the assembly for certain vector instructions, we might miss > > that we regressed performance, because we messed up the runtime checks > > guarding the vector loop. > > Oh absolutely. Presumably such checks would be included in the test or > would be checked by a different test. As always, tests have to be > constructed intelligently. :) > > -David > ___ > cfe-dev mailing list > cfe-...@lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev > ___ lldb-dev mailing list lldb-dev@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] LLDB finish_swig and Python module installation broken for multi-configuration generators
The build step works with r374250, but the install step fails. It needs
CMAKE_CFG_INTDIR added to:
install(DIRECTORY ${CMAKE_BINARY_DIR}/${LLDB_PYTHON_RELATIVE_PATH}/
DESTINATION ${LLDB_PYTHON_RELATIVE_PATH}
COMPONENT lldb-scripts)
> -Original Message-
> From: Michał Górny
> Sent: Friday, October 11, 2019 8:35 AM
> To: Ted Woodward ; LLDB d...@lists.llvm.org>; h...@google.com; antonio.afo...@gmail.com
> Subject: [EXT] Re: LLDB finish_swig and Python module installation broken for
> multi-configuration generators
>
> On Thu, 2019-10-10 at 23:11 +, Ted Woodward wrote:
> > Starting in r372835 (Fix installing Python modules on systems using
> > /usr/lib),
> the finish_swig and installation steps using multi-configuration generators
> (like
> Visual Studio or XCode) are broken.
> >
> > The problem is the symlink and install steps look for liblldb and
> > site-packages
> the wrong directories.
> >
> > For the symlink, I get this output:
> > Making symlink from ..\..\..\bin\liblldb.dll to
> > i:\obj\lib\site-packages\lldb\_lldb.pyd
> > ...
> > WindowsError: [Error 2] The system cannot find the file specified:
> '..\\..\\..\\bin\\liblldb.dll'
> >
> > Because the cwd is I:\obj\Lib\site-packages\lldb, not the correct
> I:\obj\Release\Lib\site-packages\lldb. liblldb.dll is in i:\obj\Release\bin,
> not
> i:\obj\bin. Also, the link itself should be in i:\obj\Release\lib, not
> i:\obj\lib.
> >
> > The problem is the cmake code went from using
> LLVM_LIBRARY_OUTPUT_INTDIR to CMAKE_BINARY_DIR. CMAKE_BINARY_DIR
> is not complete for multi-configuration generators - you need to specify the
> configuration as well.
> >
> [...]
> > Michal, please try this and see if it still works for the issue you fixed in
> r372835.
>
> Actually, I see that Haibo has already addressed this 2 days ago. Could you
> test if his solution (r374250) works for you?
>
> --
> Best regards,
> Michał Górny
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] [cfe-dev] [llvm-dev] RFC: End-to-end testing
Sean Silva via cfe-dev writes: >> We have to support many different systems and those systems are always >> changing (new processors, new BIOS, new OS, etc.). Performance can vary >> widely day to day from factors completely outside the compiler's >> control. As the performance changes you have to keep updating the tests >> to expect the new performance numbers. Relying on performance >> measurements to ensure something like vectorization is happening just >> isn't reliable in our experience. > > Could you compare performance with vectorization turned on and off? That might catch more things but now you're running tests twice and it still won't catch some cases. -David ___ lldb-dev mailing list lldb-dev@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
[lldb-dev] Offline build bots
Hi,
All the *nix build bots (lldb-x86_86-{debian,fedora}, netbsd-amd64) have
been offline for a while now. Some as far as July 12.
I’m not sure if this is being tracked or whom to contact about this.
Thank you.
- Afonso
--
Best regards,
António Afonso
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
