Re: Proposal for using a multi-headed tree instead of inbound (updated)

2013-04-05 Thread jmaher
My thoughts on why the average build time is shorter on try vs inbound is inbound includes pgo builds and debug builds which have other steps. The try server builds are not usually doing pgo. ___ dev-platform mailing list dev-platform@lists.mozilla.org

Re: Proposal for using a multi-headed tree instead of inbound

2013-04-04 Thread Jesse Ruderman
On 4/4/13 7:12 AM, Kartikaya Gupta wrote: On 13-04-03 23:05 , Jesse Ruderman wrote: I suggest adding an Auto branch between Try and Central. Advantages: [snip] * In Scenario D (when subtle patch interactions cause build or test failures), automation can move on to another set of Try landings,

Re: Proposal for using a multi-headed tree instead of inbound

2013-04-04 Thread Gregory Szorc
On 4/4/13 8:07 AM, Kartikaya Gupta wrote: On 13-04-03 19:49 , Jesse Ruderman wrote: +1. But can we do this with rebased changesets instead of "trivial" merge changesets? While the core of hg can handle merges, pretty much none of the tools we rely on for understanding history (hg {log, grep, d

Re: Proposal for using a multi-headed tree instead of inbound

2013-04-04 Thread Kartikaya Gupta
On 13-04-03 19:49 , Jesse Ruderman wrote: +1. But can we do this with rebased changesets instead of "trivial" merge changesets? While the core of hg can handle merges, pretty much none of the tools we rely on for understanding history (hg {log, grep, diff, bisect}) handle them well. Thinking

Re: Proposal for using a multi-headed tree instead of inbound

2013-04-04 Thread Kartikaya Gupta
(Moving thread from from m.d.tree-managment to m.d.platform) On 13-04-03 19:03 , Jeff Hammel wrote: Without wanting to start a contraversy, is there a particular reason to use multiple trees here vs e.g. a checkin-needed flag on bugzilla or similar? I'm not quite sure what you're suggesting. A

Re: Proposal for using a multi-headed tree instead of inbound

2013-04-04 Thread Kartikaya Gupta
On 13-04-03 23:05 , Jesse Ruderman wrote: I suggest adding an Auto branch between Try and Central. Advantages: [snip] * In Scenario D (when subtle patch interactions cause build or test failures), automation can move on to another set of Try landings, giving sheriffs time react without the pre

Re: Proposal for using a multi-headed tree instead of inbound

2013-04-04 Thread jmaher
the 325 jobs per push come from manually counting jobs on tbpl (ignoring pgo). remember to use showall=1. The total stats from gps include try which has much fewer test jobs per push and inbound coalescing. ___ dev-platform mailing list dev-platfo

Re: Proposal for using a multi-headed tree instead of inbound

2013-04-04 Thread Neil
Gregory Szorc wrote: Here is the percent of total builder time we spent performing jobs broken down by tree: inbound 43.98% try 27.48% central 5.24% ... We have an inbound and try problem. After patches have passed try, do people then push them to inbound, because they don't want

Re: Proposal for using a multi-headed tree instead of inbound

2013-04-04 Thread Jesse Ruderman
On 4/3/13 10:45 PM, Kartikaya Gupta wrote: > On 13-04-03 19:49 , Jesse Ruderman wrote: > > But can we do this with rebased changesets instead of "trivial" merge > > changesets? While the core of hg can handle merges, pretty much none of > > the tools we rely on for understanding history (hg

Re: Proposal for using a multi-headed tree instead of inbound

2013-04-03 Thread Kartikaya Gupta
Thanks for the many comments! I replied to a bunch of stuff below. On 13-04-03 18:34 , Justin Lebar wrote: This is a really interesting idea. git is smart enough to let you pull only the csets that end up getting merged into master. I don't know how multiple heads works with hg, but I wonder i

Re: Proposal for using a multi-headed tree instead of inbound

2013-04-03 Thread Ehsan Akhgari
On 2013-04-03 10:59 PM, Jeff Hammel wrote: So I'm not sure I understand: > 1. This will incur a significant increase in our infra resource usage since all of these patches have to do a full try run. We simply cannot afford that in today's world where we're struggling against wait times and inf

Re: Proposal for using a multi-headed tree instead of inbound

2013-04-03 Thread Jesse Ruderman
I suggest adding an Auto branch between Try and Central. Advantages: * Pulling from Central is safe, because it only gets csets that passed both Try (as individual developer pushes) and Auto (as a group). * Infrastructure load will be slightly lower, because everyone's pushes to Try will hav

Re: Proposal for using a multi-headed tree instead of inbound

2013-04-03 Thread Clint Talbert
On 4/3/2013 6:33 PM, jmaher wrote: I looked at the data used to calculate the offenders, and I found: total type, total jobs, total duration, total hours try builders, 3525, 12239477, 3399.8547 try testers, 71821, 121294315, 33692.8652778 inbound builders, 7862, 30877533, 8577.0925 inbound t

Re: Proposal for using a multi-headed tree instead of inbound

2013-04-03 Thread Jeff Hammel
On 04/03/2013 06:33 PM, Ehsan Akhgari wrote: On 2013-04-03 9:10 PM, Clint Talbert wrote: On 4/3/2013 4:28 PM, Joshua Cranmer 🐧 wrote: On 4/3/2013 4:31 PM, Kartikaya Gupta wrote: For what it's worth, I do recall there being release engineering talk about some sort of "autoland" feature (which

Re: Proposal for using a multi-headed tree instead of inbound

2013-04-03 Thread Trevor Saunders
On Wed, Apr 03, 2013 at 08:55:36PM -0400, Ehsan Akhgari wrote: > On 2013-04-03 7:44 PM, Joshua Cranmer 🐧 wrote: > >On 4/3/2013 5:36 PM, L. David Baron wrote: > >>On Wednesday 2013-04-03 17:31 -0400, Kartikaya Gupta wrote: > >Instead of running {mochitest-*,reftest,crashtest,xpcshell,marionette} > >

Re: Proposal for using a multi-headed tree instead of inbound

2013-04-03 Thread jmaher
I looked at the data used to calculate the offenders, and I found: total type, total jobs, total duration, total hours try builders, 3525, 12239477, 3399.8547 try testers, 71821, 121294315, 33692.8652778 inbound builders, 7862, 30877533, 8577.0925 inbound testers, 121641, 182883638, 50801.0105

Re: Proposal for using a multi-headed tree instead of inbound

2013-04-03 Thread Ehsan Akhgari
On 2013-04-03 9:10 PM, Clint Talbert wrote: On 4/3/2013 4:28 PM, Joshua Cranmer 🐧 wrote: On 4/3/2013 4:31 PM, Kartikaya Gupta wrote: For what it's worth, I do recall there being release engineering talk about some sort of "autoland" feature (which would automatically land any patch that passe

Re: Proposal for using a multi-headed tree instead of inbound

2013-04-03 Thread Clint Talbert
On 4/3/2013 4:28 PM, Joshua Cranmer 🐧 wrote: On 4/3/2013 4:31 PM, Kartikaya Gupta wrote: For what it's worth, I do recall there being release engineering talk about some sort of "autoland" feature (which would automatically land any patch that passed try or something), and I recall (my memory

Re: Proposal for using a multi-headed tree instead of inbound

2013-04-03 Thread Ehsan Akhgari
On 2013-04-03 7:11 PM, Gregory Szorc wrote: On 4/3/13 3:36 PM, L. David Baron wrote: On Wednesday 2013-04-03 17:31 -0400, Kartikaya Gupta wrote: 1. Take the latest green m-c change, commit your patch(es) on top of it, and push it to try. 2. If your try push is green, flag it for eventual merge

Re: Proposal for using a multi-headed tree instead of inbound

2013-04-03 Thread Ehsan Akhgari
On 2013-04-03 6:36 PM, L. David Baron wrote: On Wednesday 2013-04-03 17:31 -0400, Kartikaya Gupta wrote: 1. Take the latest green m-c change, commit your patch(es) on top of it, and push it to try. 2. If your try push is green, flag it for eventual merge to m-c and you're done. 3. If your try pu

Re: Proposal for using a multi-headed tree instead of inbound

2013-04-03 Thread Ehsan Akhgari
On 2013-04-03 7:44 PM, Joshua Cranmer 🐧 wrote: On 4/3/2013 5:36 PM, L. David Baron wrote: On Wednesday 2013-04-03 17:31 -0400, Kartikaya Gupta wrote: 1. Take the latest green m-c change, commit your patch(es) on top of it, and push it to try. 2. If your try push is green, flag it for eventual m

Re: Proposal for using a multi-headed tree instead of inbound

2013-04-03 Thread Gregory Szorc
On 4/3/13 4:11 PM, Gregory Szorc wrote: I pulled the raw builder logs from https://secure.pub.build.mozilla.org/builddata/buildjson/ and assembled a tab-separated file of all the builds for 2013-03-17 through 2013-03-23: https://people.mozilla.com/~gszorc/builds-20130317-20130323.txt.bz2 Yes

Re: Proposal for using a multi-headed tree instead of inbound

2013-04-03 Thread Trevor Saunders
On Wed, Apr 03, 2013 at 04:59:31PM -0700, Jeff Hammel wrote: > On 04/03/2013 04:44 PM, Joshua Cranmer 🐧 wrote: > >On 4/3/2013 5:36 PM, L. David Baron wrote: > >>On Wednesday 2013-04-03 17:31 -0400, Kartikaya Gupta wrote: > >>>1. Take the latest green m-c change, commit your patch(es) on top of > >>

Re: Proposal for using a multi-headed tree instead of inbound

2013-04-03 Thread Chris AtLee
On 16:11, Wed, 03 Apr, Gregory Szorc wrote: On 4/3/13 3:36 PM, L. David Baron wrote: On Wednesday 2013-04-03 17:31 -0400, Kartikaya Gupta wrote: 1. Take the latest green m-c change, commit your patch(es) on top of it, and push it to try. 2. If your try push is green, flag it for eventual merge

Re: Proposal for using a multi-headed tree instead of inbound

2013-04-03 Thread Jeff Hammel
On 04/03/2013 04:44 PM, Joshua Cranmer 🐧 wrote: On 4/3/2013 5:36 PM, L. David Baron wrote: On Wednesday 2013-04-03 17:31 -0400, Kartikaya Gupta wrote: 1. Take the latest green m-c change, commit your patch(es) on top of it, and push it to try. 2. If your try push is green, flag it for eventual

Re: Proposal for using a multi-headed tree instead of inbound

2013-04-03 Thread Justin Lebar
> But can we do this with rebased changesets instead of "trivial" merge > changesets? Personally, I think merges are a Very Good idea because they bake into the tree information that is currently contained only in the pushlog. This improves bisect from my perspective, although I'll grant it might

Re: Proposal for using a multi-headed tree instead of inbound

2013-04-03 Thread Jesse Ruderman
+1. But can we do this with rebased changesets instead of "trivial" merge changesets? While the core of hg can handle merges, pretty much none of the tools we rely on for understanding history (hg {log, grep, diff, bisect}) handle them well. ___ de

Re: Proposal for using a multi-headed tree instead of inbound

2013-04-03 Thread Joshua Cranmer 🐧
On 4/3/2013 5:36 PM, L. David Baron wrote: On Wednesday 2013-04-03 17:31 -0400, Kartikaya Gupta wrote: 1. Take the latest green m-c change, commit your patch(es) on top of it, and push it to try. 2. If your try push is green, flag it for eventual merge to m-c and you're done. 3. If your try push

Re: Proposal for using a multi-headed tree instead of inbound

2013-04-03 Thread jmaher
> > This seems like it would lead to a substantial increase in > build/test load -- one that I suspect we don't currently have the > hardware to support. This is because it would require a full > build/test run for every push, which we avoid today because many > builds and tests get merged on inb

Re: Proposal for using a multi-headed tree instead of inbound

2013-04-03 Thread Joshua Cranmer 🐧
On 4/3/2013 4:31 PM, Kartikaya Gupta wrote: The developer workflow I'm proposing requires there be a tree that is allowed to have multiple heads. The "try" tree we have now satisfies this requirement, so I'm using that in my proposal, but we could just as easily use inbound or some other tree p

Re: Proposal for using a multi-headed tree instead of inbound

2013-04-03 Thread Gregory Szorc
On 4/3/13 3:36 PM, L. David Baron wrote: On Wednesday 2013-04-03 17:31 -0400, Kartikaya Gupta wrote: 1. Take the latest green m-c change, commit your patch(es) on top of it, and push it to try. 2. If your try push is green, flag it for eventual merge to m-c and you're done. 3. If your try push i

Re: Proposal for using a multi-headed tree instead of inbound

2013-04-03 Thread Justin Lebar
If anything this should improve the experience of bisecting, because you'll be able to bisect known-good csets on m-c and only at the end step in to the merge csets which may or may not be good. Right now we say that when people push a patch queue to m-c every patch should be green, but in practic

Re: Proposal for using a multi-headed tree instead of inbound

2013-04-03 Thread Gary Kwong
Another potential problem with this approach is that we will have more merge changes in m-c, which generally screws with hg bisect. Personally I already have enough trouble with hg bisect to the point where I don't use it because I can't trust it. This may be a legitimate problem for some, but it'

Re: Proposal for using a multi-headed tree instead of inbound

2013-04-03 Thread L. David Baron
On Wednesday 2013-04-03 17:31 -0400, Kartikaya Gupta wrote: > 1. Take the latest green m-c change, commit your patch(es) on top of > it, and push it to try. > 2. If your try push is green, flag it for eventual merge to m-c and > you're done. > 3. If your try push is not green, update your patch(es)

Re: Proposal for using a multi-headed tree instead of inbound

2013-04-03 Thread Justin Lebar
This is a really interesting idea. git is smart enough to let you pull only the csets that end up getting merged into master. I don't know how multiple heads works with hg, but I wonder if we could make hg smart enough to do the same thing. Otherwise, maybe it's time to switch to git. On Wed,

Re: Proposal for using a multi-headed tree instead of inbound

2013-04-03 Thread Gregory Szorc
On 4/3/13 2:31 PM, Kartikaya Gupta wrote: Excellent write-up! I think a re-examination of our tree management is long overdue, especially with all the recent closures on inbound. My suggested process *requires* a tree which allows multiple heads, which is why I suggest "try" instead of "inbou