Re: [PATCH] test-lib: make '--stress' more bisect-friendly

2019-02-08 Thread Jeff King
On Fri, Feb 08, 2019 at 07:33:07PM +0100, SZEDER Gábor wrote: > On Fri, Feb 08, 2019 at 11:49:37AM -0500, Jeff King wrote: > > One other thing I've noticed with it: I forget to add my custom > > --root=/var/ram/git-tests when I invoke it, so my hard disk goes > > crazy (and the tests often r

Re: [PATCH] test-lib: make '--stress' more bisect-friendly

2019-02-08 Thread Jeff King
On Fri, Feb 08, 2019 at 07:23:19PM +0100, SZEDER Gábor wrote: > > Picking an is tough. Too low and you get a false negative, too high > > and you can wait forever, especially if the script is long. But I don't > > think there's any real way to auto-scale it, except by seeing a few of > > the fail

Re: [PATCH] test-lib: make '--stress' more bisect-friendly

2019-02-08 Thread SZEDER Gábor
On Fri, Feb 08, 2019 at 11:49:37AM -0500, Jeff King wrote: > One other thing I've noticed with it: I forget to add my custom > --root=/var/ram/git-tests when I invoke it, so my hard disk goes > crazy (and the tests often run slower!). I'm not sure if there's a > convenient fix. OTOH, that

Re: [PATCH] test-lib: make '--stress' more bisect-friendly

2019-02-08 Thread SZEDER Gábor
On Fri, Feb 08, 2019 at 11:47:33AM -0500, Jeff King wrote: > On Fri, Feb 08, 2019 at 12:50:45PM +0100, SZEDER Gábor wrote: > > > - Make it exit with failure if a failure is found. > > > > - Add the '--stress-limit=' option to repeat the test script > > at most N times in each of the paral

Re: [PATCH] test-lib: make '--stress' more bisect-friendly

2019-02-08 Thread Jeff King
On Fri, Feb 08, 2019 at 11:47:33AM -0500, Jeff King wrote: > > This is a case when an external stress script works better, as it can > > easily check commits in the past... if someone has such a script, > > that is. > > Heh, I literally just implemented this kind of max-count in my own > "stress

Re: [PATCH] test-lib: make '--stress' more bisect-friendly

2019-02-08 Thread Jeff King
On Fri, Feb 08, 2019 at 12:50:45PM +0100, SZEDER Gábor wrote: > - Make it exit with failure if a failure is found. > > - Add the '--stress-limit=' option to repeat the test script > at most N times in each of the parallel jobs, and exit with > success when the limit is reached. > [...

[PATCH] test-lib: make '--stress' more bisect-friendly

2019-02-08 Thread SZEDER Gábor
Let's suppose that a test somehow becomes flaky between 'master' and 'pu', and tends to fail within the first 50 repetitions when run with '--stress'. In such a case we could use 'git bisect' to find the culprit: if the test script fails with '--stress', then the commit is definitely bad, but if i