I tried to gather some data for us to discuss this topic in this thread. I tried to count number of un-necessary builds by looking at most recent (as of 12, March 9 PM PST) 50 PRs merged to master and 50 PRs. Identifying un-necessary builds is bit subjective. I tried to be more conservative where I didn't count a build as un-necessary if I was in doubt. Hence, I was not able to automate, but I made an effort to go through PRs manually and use below criteria to identify un-necessary commits triggering the builds.
1. Explicitly marked as WIP / do not review PR 2. Incremental WIP commit and finally commenting a commit “trigger CI” 3. Multiple commits to address all comments from single review. This is assuming we see a comment, address them, commit, next the following comment 4. Sequence of documentation only changes I found there were around 42 avoidable builds from most recent 50 merged PRs and around 86 builds from recent 50 open PRs. I synced up with other contributors (Joe Evans, Chai) from Amazon who is contributing to MXNet CI system. I was told that on an average it costs around $84 per build and on an average 6 commits per merged PR (for year 2019). Going by that, it is approximately 1/6 builds are avoidable. [100 / 300 + 300 ] Usability should be top most priority. But, since either a reviewer or pr author can trigger the bot, is it really a hurdle for pr author or reviewer to call a bot to trigger CI? Given that PR author and reviewer is already actively commenting various details such as - PR description, review comments and responses, adding labels etc. Me too curious to know the behavior for Tao's above use case. Best, Sandeep On Thu, Mar 12, 2020 at 7:18 PM Tao Lv <[email protected]> wrote: > Is it possible for re-triggering a single job to be abused? For example, > the author spends two days re-triggering a flaky job to make it pass. But > other jobs which have passed the validation may be broken by other commits > during the two day without being noticed. And finally the PR is merged with > underlying problems. > > On Fri, Mar 13, 2020 at 6:19 AM Marco de Abreu <[email protected]> > wrote: > > > In the end it only comes down to money, considering that the system is > auto > > scaling, making the execution time constant. > > > > If we're trading money for usability, I certainly would prefer usability. > > I'd rather recommend to spend time on parallelizing test execution or > > getting rid of integration tests in the PR stage instead reducing the > costs > > by making people not use it. But taking a step back to requiring people > to > > manually trigger CI again doesn't feel right. > > > > I'm happy to see that bot deployed, but I do not agree with removing the > > auto trigger functionality for new commits. > > > > -Marco > > > > Chaitanya Bapat <[email protected]> schrieb am Do., 12. März 2020, > > 22:47: > > > > > @Marco Thanks for pointing that out. > > > Tomorrow i.e. Friday, March 13, 2020 at 3:00 PM - 3:30 PM in > (UTC-08:00) > > > Pacific Time (US & Canada). > > > > > > > When do we expect this bot to be deployed? > > > @Lin If all goes well in the next week I can deploy it to public Apache > > > (provided I get permissions from Apache Infra) > > > > > > @Marco Thanks for your feedback. > > > > CI system has to support the community without requiring people to > > > constantly shepherd every single run > > > We have data for the number of times CI was triggered unnecessarily > which > > > includes > > > - Entire build triggered instead of specific build > > > - CI triggered when PR is still work in progress or not yet ready (say > - > > > intermediate commits) > > > At the end its a trade-off > > > Money, Resources, Time to build for each and every commit vs Pain of > > > triggering builds > > > > > > > > > > Scan trigger plugin would poll SCM. Can we use plugin at scale? > > > > > > 1. I haven't tested it on scale. But I think with the current scale of > > > MXNet repo (191 open PRs i.e. checking for changes to 191 branches - It > > > should be manageable) > > > 2. What's the purpose of the plugin? tldr; Branch discovery or branch > > > indexing. > > > Scan trigger plugin comes into the picture only once per PR per job > > (i.e. 8 > > > times per PR for 8 jobs). It is basically done when a new PR is made > and > > > the job (say unix-cpu hasn't discovered the new PR branch yet). That's > > it. > > > So it shouldn't be a problem for public MXNet repo. > > > > > > Thanks, > > > Chai > > > > > > > > > On Thu, 12 Mar 2020 at 14:22, Marco de Abreu <[email protected]> > > > wrote: > > > > > > > Btw you forgot to set a date and time for the metting > > > > > > > > > > > > On Thu, Mar 12, 2020 at 10:18 PM Marco de Abreu < > > [email protected] > > > > > > > > wrote: > > > > > > > > > Thanks Chai, I generally like the idea of the bot. But I'm not a > > > > supporter > > > > > of the idea to disable any automatic triggering (disabling the > > webhook > > > is > > > > > also not an option, considering that this will disable master > > > triggers). > > > > > The CI system has to support the community without requiring people > > to > > > > > constantly shepherd every single run. Disabling automatic > triggering > > > > seems > > > > > like a step back to me. > > > > > > > > > > Instead, I'd recommend that CI gets triggered upon every commit as > > > usual, > > > > > but people have the possibility to call a "command" (i.e. make a > > > message > > > > > which results in the bot setting a label) to disable CI until they > > > revoke > > > > > it. But the standard should still be that a new commit triggers a > new > > > CI > > > > > run. > > > > > > > > > > https://plugins.jenkins.io/multibranch-scan-webhook-trigger/ > seems > > > like > > > > > this would poll SCM. This will incur high quota restrictions. Are > you > > > > sure > > > > > that you can use that plugin at scale? > > > > > > > > > > -Marco > > > > > > > > > > > > > > > On Thu, Mar 12, 2020 at 10:04 PM Lin Yuan <[email protected]> > > wrote: > > > > > > > > > >> Chai, > > > > >> > > > > >> Awesome work. When do we expect this bot to be deployed? > > > > >> > > > > >> Best, > > > > >> > > > > >> Lin > > > > >> > > > > >> On Thu, Mar 12, 2020 at 2:00 PM Chaitanya Bapat < > > [email protected] > > > > > > > > >> wrote: > > > > >> > > > > >> > Hello MXNet community, > > > > >> > > > > > >> > I have built an MXNet Bot <https://github.com/mxnet-bot> that > > > allows > > > > PR > > > > >> > Authors, Committers and Jenkins Admins to trigger CI manually. > > > > >> > It handles 2 problems > > > > >> > 1. Manual CI trigger instead of existing automated CI trigger > > > > >> > 2. Gives permissions to PR Authors (in addition to MXNet > > Committers > > > > and > > > > >> > Jenkins Admins) > > > > >> > > > > > >> > Design Doc : > > > > >> > https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot > > > > >> > > > > > >> > I urge you all to attend the demonstration meeting and lend your > > > views > > > > >> on > > > > >> > the same. > > > > >> > > > > > >> > Thank you, > > > > >> > Chai > > > > >> > > > > > >> > *Meeting Details*: > > > > >> > ==============Conference Bridge Information============== > > > > >> > You have been invited to an online meeting, powered by Amazon > > Chime. > > > > >> > *Chime meeting ID*: *9272158344* > > > > >> > Join via Chime clients (manually): Select 'Meetings > Join a > > > Meeting', > > > > >> and > > > > >> > enter 9272158344 > > > > >> > Join via Chime clients (auto-call): If you invite auto-call as > > > > attendee, > > > > >> > Chime will call you when the meeting starts, select 'Answer' > > > > >> > *Join via browser screen share*: https://chime.aws/9272158344 > > > > >> > *Join via phone* (US): +1-929-432-4463,,,9272158344# > > > > >> > *Join via phone (US toll-free)*: +1-855-552-4463,,,9272158344# > > > > >> > International dial-in: https://chime.aws/dialinnumbers/ > > > > >> > In-room video system: Ext: 62000, Meeting PIN: 9272158344# > > > > >> > > > > > >> > -- > > > > >> > *Chaitanya Prakash Bapat* > > > > >> > *+1 (973) 953-6299* > > > > >> > > > > > >> > [image: https://www.linkedin.com//in/chaibapat25] > > > > >> > <https://github.com/ChaiBapchya>[image: > > > > >> https://www.facebook.com/chaibapat > > > > >> > ] > > > > >> > <https://www.facebook.com/chaibapchya>[image: > > > > >> > https://twitter.com/ChaiBapchya] < > https://twitter.com/ChaiBapchya > > > > >> >[image: > > > > >> > https://www.linkedin.com//in/chaibapat25] > > > > >> > <https://www.linkedin.com//in/chaibapchya/> > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > > -- > > > *Chaitanya Prakash Bapat* > > > *+1 (973) 953-6299* > > > > > > [image: https://www.linkedin.com//in/chaibapat25] > > > <https://github.com/ChaiBapchya>[image: > > https://www.facebook.com/chaibapat > > > ] > > > <https://www.facebook.com/chaibapchya>[image: > > > https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya > > >[image: > > > https://www.linkedin.com//in/chaibapat25] > > > <https://www.linkedin.com//in/chaibapchya/> > > > > > > -- Sandeep Krishnamurthy
