Are you able to provide a one liner that helps clarify this? python -m unittest discover -s QMTest --pattern "Test*.py" produces a very quick failure.
Andrew On 20 December 2017 at 16:39, Bill Deegan <[email protected]> wrote: > 4. We're already using unittest. QMtest is just the name of the directory > at this point. (You used to have to install qmtest to run the test suite) > > On Wed, Dec 20, 2017 at 11:28 AM, Daniel Moody <[email protected]> > wrote: > >> 1. are you referring to the use of runtest.py command, or how the various >> builders and environments are started? >> 2. agreed >> 3. With the free CI services you are limited to whatever they offer, and >> at this point splitting up the tests is a workaround, instead of rewriting >> the entire test suite. Even then, scaling it up, it may still be necessary >> to split up tests across jobs. >> 4. agreed, a well known standard python testing framework would be great. >> I like pytest. >> >> My goal currently is to at least get basic CI on github working with as >> little rework as possible. >> >> On Wed, Dec 20, 2017 at 11:20 AM, Bill Deegan <[email protected]> >> wrote: >> >>> >>> >>> On Wed, Dec 20, 2017 at 11:05 AM, Andrew Featherstone < >>> [email protected]> wrote: >>> >>>> My preferred way of working on this problem: >>>> >>>> >>>> 1. Get parity with what happens at http://buildbot.scons.org/#/. I >>>> can't work out from there how to read the actual commands being run in >>>> the >>>> CI job, so I'm mostly replicating what happens in the Travis-based >>>> builds. >>>> >>>> >>> Basically: >>> 1) git clone >>> 2) python2.7 runtest.py -a >>> >>> Current docs and packages aren't being built (but should be) >>> >>>> >>>> 1. Get all of scons' build processes onto cloud-based CI. >>>> Everything that is at http://scons.org/pages/download.html should >>>> come out of a CI job. >>>> >>>> Sounds good. >>> We used to build dev packages an make available a while back. >>> setup.py needs a rewrite and a few other pieces of logic can be removed. >>> >>> >>>> 1. >>>> 2. Get the tests running faster. This needs some profiling, but >>>> from the logs it reads like every test is run in a separate process, >>>> which >>>> feels pretty expensive to me. There's also references to 2008-era blog >>>> posts which I take with a large grain of salt. We need the tests >>>> running as >>>> a pre-requisite of course. Devising way to slice up long running tests >>>> feels like solving the wrong problem. >>>> >>>> My guess is that most of the tests could have >>> DefaultEnvironment(tools=[]), and all Environment()'s could have a more >>> limited list of tools. That would make the test faster to much faster >>> depending on the test. Likely more speedup on win32 than on linux due to >>> msvc detection logic. >>> >>> >>>> >>>> 1. Move away from a QMTest to something more commonplace. The >>>> canonical home of QMTest seems to be https://github.com/MentorEmbed >>>> ded/qmtest, last updated in 2011. I think it's time to look at >>>> something else, e.g. plain old unittest or pytest. >>>> >>>> QMtest is basically gone. Tests are basically unittests now. >>> >>> >>>> >>>> 1. >>>> >>>> Perhaps it's worth splitting off another thread for roadmapping? >>>> >>> >>> Sure. >>> >>> >>> >>>> >>>> Andrew >>>> >>>> On 20 December 2017 at 14:00, Daniel Moody <[email protected]> wrote: >>>> >>>>> What about this: >>>>> >>>>> --interval-start FLOAT Percentile as float (0.0 - 1.0) of what >>>>> test to start on from the >>>>> determined list of all tests >>>>> --interval-end FLOAT Percentile as float (0.0 - 1.0) of what >>>>> test to start on from the >>>>> determined list of all >>>>> tests >>>>> >>>>> interval-start default is 0 >>>>> interval-end default is 1 >>>>> >>>>> A little more flexible I think. >>>>> >>>>> >>>>> >>>>> On Wed, Dec 20, 2017 at 8:41 AM, Bill Deegan < >>>>> [email protected]> wrote: >>>>> >>>>>> Maybe add a flag directly to runtest.py to split up the tests. >>>>>> --test_mod 4 --test_index 0..3 ? >>>>>> >>>>>> Or something to that affect. >>>>>> >>>>>> On Wed, Dec 20, 2017 at 8:27 AM, Daniel Moody <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hey Andrew, >>>>>>> >>>>>>> I also was working on this. I think in your case the builds timed >>>>>>> out due to issues with vswhere: >>>>>>> https://github.com/Microsoft/vswhere/issues/87 >>>>>>> https://github.com/Microsoft/vswhere/issues/91 >>>>>>> >>>>>>> In this case we will need to use the Visual Studio 2017 image. >>>>>>> >>>>>>> Also just to note; the issue mentioned above is also an issue with >>>>>>> SCons MSVS detection in general now for SCons 3 and newer since that is >>>>>>> when it was switched to use vswhere. >>>>>>> >>>>>>> I liked some of the things you had in your script so I took those >>>>>>> and merge them into my script. I also implemented a script that will >>>>>>> split >>>>>>> the build up into multiple jobs like the travis script does, however I >>>>>>> have >>>>>>> not been able to get appveyor to do multi-line scripts correctly so it >>>>>>> is >>>>>>> all in a one liner at the moment: >>>>>>> https://github.com/dmoody256/scons/blob/AppveyorCI/.appveyor.yml >>>>>>> https://ci.appveyor.com/project/dmoody256/scons/build/1.0.53 >>>>>>> >>>>>>> Currently python 3 will fail from several tests so I have them >>>>>>> commented out. >>>>>>> >>>>>>> >>>>>>> On Wed, Dec 20, 2017 at 8:00 AM, Bill Deegan < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Parallel should help. >>>>>>>> >>>>>>>> On my buildbot worker (with 2 other builds running single threaded >>>>>>>> tests) it takes 2:05. >>>>>>>> So on a reasonably modern machine, -j2 should finish in under an >>>>>>>> hour if not, try -j3? >>>>>>>> >>>>>>>> Or we can split up runs as we've done with the travis run.. >>>>>>>> >>>>>>>> -Bill >>>>>>>> >>>>>>>> On Wed, Dec 20, 2017 at 5:54 AM, Andrew Featherstone < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> I've been trying to get AppVeyor working for Windows-based CI, but >>>>>>>>> we're hitting their 1 hour time limit (see >>>>>>>>> https://github.com/ajf58/scons/blob/appveyor/.appveyor.yml and >>>>>>>>> https://ci.appveyor.com/project/ajf58/scons. >>>>>>>>> >>>>>>>>> My next pass at this will be trying to run the unit tests in >>>>>>>>> parallel (as the Windows VM has two cores available). Either that, or >>>>>>>>> we >>>>>>>>> split the job matrix a different way, e.g. run tests grouped by >>>>>>>>> something >>>>>>>>> else other than Python version. >>>>>>>>> >>>>>>>>> Thoughts? >>>>>>>>> >>>>>>>>> Andrew >>>>>>>>> >>>>>>>>> On 18 December 2017 at 22:51, Bill Deegan < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Daniel, >>>>>>>>>> >>>>>>>>>> Can we get travis to test with py2.7, 3.5, and 3.6 ? >>>>>>>>>> >>>>>>>>>> -Bill >>>>>>>>>> >>>>>>>>>> On Wed, Dec 6, 2017 at 12:03 AM, Bill Deegan < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Thanks! >>>>>>>>>>> That's pretty cool. >>>>>>>>>>> I'll try to get the coverage hooked up soon. >>>>>>>>>>> That'll also be very useful.. >>>>>>>>>>> >>>>>>>>>>> On Tue, Dec 5, 2017 at 8:27 PM, Jonathon Reinhart < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Yes, it should automatically do that. >>>>>>>>>>>> >>>>>>>>>>>> See this (merged) PR from one of my projects: >>>>>>>>>>>> https://github.com/JonathonReinhart/scuba/pull/98 >>>>>>>>>>>> >>>>>>>>>>>> Towards the bottom you'll see a "View Details" button. >>>>>>>>>>>> Clicking that will expand a box showing the results of all the >>>>>>>>>>>> "checks" that ran. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Dec 5, 2017 at 11:13 PM, Bill Deegan < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Is there a way to get travis to post the results back into the >>>>>>>>>>>>> pull request? >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Scons-dev mailing list >>>>>>>>>>>> [email protected] >>>>>>>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Scons-dev mailing list >>>>>>>>>> [email protected] >>>>>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Scons-dev mailing list >>>>>>>>> [email protected] >>>>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Scons-dev mailing list >>>>>>>> [email protected] >>>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Scons-dev mailing list >>>>>>> [email protected] >>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev >>>>>>> >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Scons-dev mailing list >>>>>> [email protected] >>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev >>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> Scons-dev mailing list >>>>> [email protected] >>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> Scons-dev mailing list >>>> [email protected] >>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev >>>> >>>> >>> >>> _______________________________________________ >>> Scons-dev mailing list >>> [email protected] >>> https://pairlist2.pair.net/mailman/listinfo/scons-dev >>> >>> >> >> _______________________________________________ >> Scons-dev mailing list >> [email protected] >> https://pairlist2.pair.net/mailman/listinfo/scons-dev >> >> > > _______________________________________________ > Scons-dev mailing list > [email protected] > https://pairlist2.pair.net/mailman/listinfo/scons-dev > >
_______________________________________________ Scons-dev mailing list [email protected] https://pairlist2.pair.net/mailman/listinfo/scons-dev
