Re: [PROPOSAL] eliminate file count loophole in PR StressNewTest

[Dan Smith]

Won't this make it impossible to merge refactoring changes that touch a lot
of tests?

-Dan

On Sat, Feb 29, 2020, 12:37 PM Robert Houghton  wrote:

> Yes, as it should
>
> On Sat, Feb 29, 2020, 12:25 Dan Smith  wrote:
>
> > Doesn't the build fail when concourse times out?
>
>

Re: [PROPOSAL] eliminate file count loophole in PR StressNewTest

[Owen Nichols]

Impossible, no. Inconvenient, perhaps, but a small price to pay for being
able to trust that green means green.

With or without this proposed change, if anyone is having trouble getting
their PR to pass StressNew, please bring it up on the dev list and we can
discuss the appropriate solution on a case-by-case basis (e.g. increasing
timeout, fixing the logic that identifies changed test files, splitting
into multiple PRs, authorizing an override, etc).

On Sun, Mar 1, 2020 at 8:56 AM Dan Smith  wrote:

> Won't this make it impossible to merge refactoring changes that touch a lot
> of tests?
>
> -Dan
>
> On Sat, Feb 29, 2020, 12:37 PM Robert Houghton 
> wrote:
>
> > Yes, as it should
> >
> > On Sat, Feb 29, 2020, 12:25 Dan Smith  wrote:
> >
> > > Doesn't the build fail when concourse times out?
> >
> >
>

Re: [PROPOSAL] eliminate file count loophole in PR StressNewTest

[Anthony Baker]

What percentage of PR’s are currently subject to the 25 test file rule?  How 
many would be subject to the concourse timeout?

I’d like to understand the scope of the impact for this change.

Anthony


> On Mar 1, 2020, at 8:58 AM, Owen Nichols  wrote:
> 
> Impossible, no. Inconvenient, perhaps, but a small price to pay for being
> able to trust that green means green.
> 
> With or without this proposed change, if anyone is having trouble getting
> their PR to pass StressNew, please bring it up on the dev list and we can
> discuss the appropriate solution on a case-by-case basis (e.g. increasing
> timeout, fixing the logic that identifies changed test files, splitting
> into multiple PRs, authorizing an override, etc).
> 
> On Sun, Mar 1, 2020 at 8:56 AM Dan Smith  wrote:
> 
>> Won't this make it impossible to merge refactoring changes that touch a lot
>> of tests?
>> 
>> -Dan
>> 
>> On Sat, Feb 29, 2020, 12:37 PM Robert Houghton 
>> wrote:
>> 
>>> Yes, as it should
>>> 
>>> On Sat, Feb 29, 2020, 12:25 Dan Smith  wrote:
>>> 
 Doesn't the build fail when concourse times out?
>>> 
>>> 
>> 

Re: [PROPOSAL] eliminate file count loophole in PR StressNewTest

[Owen Nichols]

We don’t tend to look too closely at successful PR checks to see whether they 
actually checked anything at all.

One example I found is 
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-pr/jobs/StressNewTestOpenJDK11/builds/5957
 
:
32 is too many changed tests to stress test. Allowing this job to pass without 
stress testing.

Here are 92 more examples (url’s omitted for brevity — use the example above as 
a template and just replace the last 4 digits):
26 is too many changed tests to stress test. Allowing this job to pass without 
stress testing.  (build 6243)
26 is too many changed tests to stress test. Allowing this job to pass without 
stress testing.  (build 6249)
26 is too many changed tests to stress test. Allowing this job to pass without 
stress testing.  (build 6402)
27 is too many changed tests to stress test. Allowing this job to pass without 
stress testing.  (build 6262)
27 is too many changed tests to stress test. Allowing this job to pass without 
stress testing.  (build 6430)
27 is too many changed tests to stress test. Allowing this job to pass without 
stress testing.  (build 6439)
27 is too many changed tests to stress test. Allowing this job to pass without 
stress testing.  (build 6449)
27 is too many changed tests to stress test. Allowing this job to pass without 
stress testing.  (build 6454)
27 is too many changed tests to stress test. Allowing this job to pass without 
stress testing.  (build 6458)
27 is too many changed tests to stress test. Allowing this job to pass without 
stress testing.  (build 6459)
28 is too many changed tests to stress test. Allowing this job to pass without 
stress testing.  (build 6224)
28 is too many changed tests to stress test. Allowing this job to pass without 
stress testing.  (build 6441)
28 is too many changed tests to stress test. Allowing this job to pass without 
stress testing.  (build 6448)
28 is too many changed tests to stress test. Allowing this job to pass without 
stress testing.  (build 6452)
29 is too many changed tests to stress test. Allowing this job to pass without 
stress testing.  (build 6102)
29 is too many changed tests to stress test. Allowing this job to pass without 
stress testing.  (build 6177)
30 is too many changed tests to stress test. Allowing this job to pass without 
stress testing.  (build 5939)
30 is too many changed tests to stress test. Allowing this job to pass without 
stress testing.  (build 5940)
30 is too many changed tests to stress test. Allowing this job to pass without 
stress testing.  (build 5949)
30 is too many changed tests to stress test. Allowing this job to pass without 
stress testing.  (build 6473)
31 is too many changed tests to stress test. Allowing this job to pass without 
stress testing.  (build 5953)
31 is too many changed tests to stress test. Allowing this job to pass without 
stress testing.  (build 6187)
31 is too many changed tests to stress test. Allowing this job to pass without 
stress testing.  (build 6470)
31 is too many changed tests to stress test. Allowing this job to pass without 
stress testing.  (build 6471)
31 is too many changed tests to stress test. Allowing this job to pass without 
stress testing.  (build 6474)
31 is too many changed tests to stress test. Allowing this job to pass without 
stress testing.  (build 6475)
32 is too many changed tests to stress test. Allowing this job to pass without 
stress testing.  (build 5958)
32 is too many changed tests to stress test. Allowing this job to pass without 
stress testing.  (build 6173)
32 is too many changed tests to stress test. Allowing this job to pass without 
stress testing.  (build 6236)
32 is too many changed tests to stress test. Allowing this job to pass without 
stress testing.  (build 6237)
32 is too many changed tests to stress test. Allowing this job to pass without 
stress testing.  (build 6242)
33 is too many changed tests to stress test. Allowing this job to pass without 
stress testing.  (build 6246)
33 is too many changed tests to stress test. Allowing this job to pass without 
stress testing.  (build 6248)
33 is too many changed tests to stress test. Allowing this job to pass without 
stress testing.  (build 6250)
33 is too many changed tests to stress test. Allowing this job to pass without 
stress testing.  (build 6251)
33 is too many changed tests to stress test. Allowing this job to pass without 
stress testing.  (build 6254)
33 is too many changed tests to stress test. Allowing this job to pass without 
stress testing.  (build 6255)
34 is too many changed tests to stress test. Allowing this job to pass without 
stress testing.  (build 6139)
34 is too many changed tests to stress test. Allowing this job to pass without 
stress testing.  (build 6141)
34 is too many changed tests to stress test. Allowing this job to pass without 
stress testing.  (build 62