mumrah commented on PR #17019: URL: https://github.com/apache/kafka/pull/17019#issuecomment-2315635864
@chia7712 thanks for the feedback. > The quota is limited and so it could impact the other flow (normal PR and CI) Since this workflow is run manually, I think the impact would be limited. Also, as long as the caller isn't running a whole module's tests, it should only run of a few minutes. I've set a timeout of 1hr to the job to prevent using up too much run time. > Also, README offers a simple command I didn't realize that :) I think this method of repeating a test is not actually very useful since it's just running the Gradle command over and over. Often times, flaky tests only appear when the system is under load. Invoking Gradle in a loop gives too much time for the system to "settle" in between runs. This is why I normally use (and recommend) the IntelliJ "Run Until Failed" option while running tests in IntelliJ (not Gradle). It runs in a tight loop and puts some load on the system. Even still, I've run into plenty of cases where a test is _only_ failing in CI. Not having the ability to run a single test in CI makes it really hard to debug such cases. It essentially means each trial of your bugfix requires waiting for a full CI run. A workaround to this I've used in the past is to alter the Jenkinsfile to just run the tests I want. I think a dedicated job for running a single test is a better option for developers. Another benefit of having this new workflow is it gives us easily shareable evidence when submitting test fixes. Instead of a reviewer looking at a single CI run for pass/fail, the author can give a link to a 10x deflake run which gives stronger evidence of the fix. Let me know what you think -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
