Re: Recent build time improvements due to unified sources

2013-11-29 Thread Gregory Szorc
On 11/29/13, 5:12 AM, Ehsan Akhgari wrote: On 11/28/2013, 4:55 PM, Mike Hoye wrote: On 11/28/2013, 4:45 PM, Nicholas Nethercote wrote: I'm pretty sure that gps was saying "if you're paid to work by Mozilla, get a faster machine". More generally, we're all in furious agreement: fast builds are g

Re: Recent build time improvements due to unified sources

2013-11-28 Thread Mike Hommey
On Thu, Nov 28, 2013 at 05:12:53PM -0500, Ehsan Akhgari wrote: > On 11/28/2013, 4:55 PM, Mike Hoye wrote: > >On 11/28/2013, 4:45 PM, Nicholas Nethercote wrote: > >>I'm pretty sure that gps was saying "if you're paid to work by > >>Mozilla, get a faster machine". More generally, we're all in furious

Re: Recent build time improvements due to unified sources

2013-11-28 Thread Ehsan Akhgari
On 11/28/2013, 4:55 PM, Mike Hoye wrote: On 11/28/2013, 4:45 PM, Nicholas Nethercote wrote: I'm pretty sure that gps was saying "if you're paid to work by Mozilla, get a faster machine". More generally, we're all in furious agreement: fast builds are good; achieving them via multiple means is wo

Re: Recent build time improvements due to unified sources

2013-11-28 Thread Mike Hoye
On 11/28/2013, 4:45 PM, Nicholas Nethercote wrote: I'm pretty sure that gps was saying "if you're paid to work by Mozilla, get a faster machine". More generally, we're all in furious agreement: fast builds are good; achieving them via multiple means is worthwhile; those with the option of getti

Re: Recent build time improvements due to unified sources

2013-11-28 Thread Nicholas Nethercote
On Thu, Nov 28, 2013 at 1:12 PM, Ehsan Akhgari wrote: > > Right. FWIW, this is probably as good a forum as any to voice my objection > on the recent trend of recommending people to buy new hardware to get faster > builds. There is a *lot* that we can still do in order to improve our build > time

Re: Recent build time improvements due to unified sources

2013-11-28 Thread Ehsan Akhgari
On 11/28/2013, 1:21 PM, Mike Hoye wrote: On 11/28/2013, 12:14 PM, Ehsan Akhgari wrote: Please file a bug (and CC me) with more details about where the build fails, how much memory the machine has, and how much memory each compiler invocation consumes (test with make -j1). We can adjust the num

Re: Recent build time improvements due to unified sources

2013-11-28 Thread Mike Hoye
On 11/28/2013, 12:14 PM, Ehsan Akhgari wrote: Please file a bug (and CC me) with more details about where the build fails, how much memory the machine has, and how much memory each compiler invocation consumes (test with make -j1). We can adjust the number of files we build in one chunk per d

Re: Recent build time improvements due to unified sources

2013-11-28 Thread Ehsan Akhgari
On 11/28/2013, 2:29 AM, Gabriele Svelto wrote: On 28/11/2013 08:09, Gregory Szorc wrote: Peak RSS likely has increased significantly (hundreds to gigabytes range). OK, that's what I was curious about. Memory is cheap and getting cheaper. Nobody paid by Mozilla to develop Firefox should have

Re: Recent build time improvements due to unified sources

2013-11-28 Thread Gabriele Svelto
On 28/11/2013 10:57, Neil wrote: I often build in a VM. I allocate 2 CPUs and 2GB of RAM to it, which seems to be enough even to link xul.dll with debug symbols, although it takes a few minutes. (Linking it without symbols takes just seconds.) Given the amount of memory needed to link I haven't c

Re: Recent build time improvements due to unified sources

2013-11-28 Thread Neil
Gabriele Svelto wrote: Another common case of OOMs are people building inside a VM. Just yesterday I helped another mozillian figure out why his FxOS build had started to fail. It turned out he was building inside a VM with too many jobs and too little memory dedicated to it. I often build

Re: Recent build time improvements due to unified sources

2013-11-27 Thread Gregory Szorc
On 11/28/13, 2:29 PM, Gabriele Svelto wrote: On 28/11/2013 08:09, Gregory Szorc wrote: Peak RSS likely has increased significantly (hundreds to gigabytes range). OK, that's what I was curious about. Memory is cheap and getting cheaper. Nobody paid by Mozilla to develop Firefox should have a

Re: Recent build time improvements due to unified sources

2013-11-27 Thread Gabriele Svelto
On 28/11/2013 08:09, Gregory Szorc wrote: Peak RSS likely has increased significantly (hundreds to gigabytes range). OK, that's what I was curious about. Memory is cheap and getting cheaper. Nobody paid by Mozilla to develop Firefox should have a machine with less than 16 GB. Adding 25%+ to b

Re: Recent build time improvements due to unified sources

2013-11-27 Thread Gregory Szorc
On 11/28/13, 12:21 PM, Gabriele Svelto wrote: On 28/11/2013 06:06, Gregory Szorc wrote: On de5aa094b55f, we're now down to: Wall 7:37 (457) User 45:38 (2738) Sys3:54 (234) Total 49:32 (2972) That's with WebRTC and ICU enabled. Looking at my own stats while building I was wonderi

Re: Recent build time improvements due to unified sources

2013-11-27 Thread Gabriele Svelto
On 28/11/2013 06:06, Gregory Szorc wrote: On de5aa094b55f, we're now down to: Wall 7:37 (457) User 45:38 (2738) Sys3:54 (234) Total 49:32 (2972) That's with WebRTC and ICU enabled. Looking at my own stats while building I was wondering if anybody has looked at peak memory consum

Re: Recent build time improvements due to unified sources

2013-11-27 Thread Gregory Szorc
On 11/21/13, 10:38 PM, Gregory Szorc wrote: On 11/20/13, 9:57 PM, Gregory Szorc wrote: On 11/19/2013 10:08 PM, Gregory Szorc wrote: On 11/18/13, 11:15 PM, Gregory Szorc wrote: Do builds feel like they've gotten faster in the last few weeks^hours? It's because they have. When I did my MacBook

Re: Recent build time improvements due to unified sources

2013-11-21 Thread Nicholas Nethercote
On Thu, Nov 21, 2013 at 7:38 AM, Gregory Szorc wrote: > > You people are sick. I go to bed, wake up and my builds have gotten faster And I've gone from 7.5 minutes to 6.75 minutes in the past day or two. Nick ___ dev-platform mailing list dev-platform@

Re: Recent build time improvements due to unified sources

2013-11-21 Thread Gregory Szorc
On 11/20/13, 9:57 PM, Gregory Szorc wrote: On 11/19/2013 10:08 PM, Gregory Szorc wrote: On 11/18/13, 11:15 PM, Gregory Szorc wrote: Do builds feel like they've gotten faster in the last few weeks^hours? It's because they have. When I did my MacBook Pro comparison [1] with a changeset from Oct

Re: Recent build time improvements due to unified sources

2013-11-21 Thread Michael Shal
> From: "Neil" > There used to be a limitation that source files had to be in the VPATH. > This limitation obviously does not apply to unified sources (the > compiler will use the -I path to find the source.) so you shouldn't have > a problem setting UNIFIED_SOURCES in a parent moz.build file. In

Re: Recent build time improvements due to unified sources

2013-11-21 Thread Neil
Zack Weinberg wrote: On 2013-11-20 12:37 PM, Benoit Jacob wrote: Talking about ideas for further extending the impact of UNIFIED_SOURCES, it seems that the biggest limitation at the moment is that sources can't be unified between different moz.build's. Because of that, source directories tha

Re: Recent build time improvements due to unified sources

2013-11-20 Thread Gregory Szorc
On 11/19/2013 10:08 PM, Gregory Szorc wrote: > On 11/18/13, 11:15 PM, Gregory Szorc wrote: >> Do builds feel like they've gotten faster in the last few weeks^hours? >> It's because they have. >> >> When I did my MacBook Pro comparison [1] with a changeset from Oct 28, >> build times on my 2013 2.6

Re: Recent build time improvements due to unified sources

2013-11-20 Thread Benoit Jacob
2013/11/20 Ehsan Akhgari > On 2013-11-20 5:27 PM, Robert O'Callahan wrote: > >> On Thu, Nov 21, 2013 at 11:06 AM, Zack Weinberg wrote: >> >> On 2013-11-20 12:37 PM, Benoit Jacob wrote: >>> >>> Talking about ideas for further extending the impact of UNIFIED_SOURCES, it seems that the

Re: Recent build time improvements due to unified sources

2013-11-20 Thread Ehsan Akhgari
On 2013-11-20 5:27 PM, Robert O'Callahan wrote: On Thu, Nov 21, 2013 at 11:06 AM, Zack Weinberg wrote: On 2013-11-20 12:37 PM, Benoit Jacob wrote: Talking about ideas for further extending the impact of UNIFIED_SOURCES, it seems that the biggest limitation at the moment is that sources can't

Re: Recent build time improvements due to unified sources

2013-11-20 Thread Robert O'Callahan
On Thu, Nov 21, 2013 at 11:06 AM, Zack Weinberg wrote: > On 2013-11-20 12:37 PM, Benoit Jacob wrote: > >> Talking about ideas for further extending the impact of UNIFIED_SOURCES, >> it >> seems that the biggest limitation at the moment is that sources can't be >> unified between different moz.bui

Re: Recent build time improvements due to unified sources

2013-11-20 Thread Zack Weinberg
On 2013-11-20 12:37 PM, Benoit Jacob wrote: Talking about ideas for further extending the impact of UNIFIED_SOURCES, it seems that the biggest limitation at the moment is that sources can't be unified between different moz.build's. Because of that, source directories that consist of many small su

Re: Recent build time improvements due to unified sources

2013-11-20 Thread Gregory Szorc
On 11/20/13, 9:49 AM, Benoit Jacob wrote: 2013/11/20 Gregory Szorc mailto:g...@mozilla.com>> On Nov 20, 2013, at 9:37, Benoit Jacob mailto:jacob.benoi...@gmail.com>> wrote: > Talking about ideas for further extending the impact of UNIFIED_SOURCES, it > seems that the bigges

Re: Recent build time improvements due to unified sources

2013-11-20 Thread Ehsan Akhgari
While this analysis is interesting, it doesn't measure the impact of the unified builds project directly, so I decided that now that a good chunk of code is being compiled in unified mode we should get some specific numbers on the improvements. What I did was I took inbound as of ab70db6b27c8,

Re: Recent build time improvements due to unified sources

2013-11-20 Thread Ehsan Akhgari
On 2013-11-20 12:09 PM, Chris Peterson wrote: It might be useful to add a files_per_unified_file parameter to mozconfig or mach build. People could benchmark different values of files_per_unified_file (trading off clobber vs incremental build times). The same parameter could also be used to disab

Re: Recent build time improvements due to unified sources

2013-11-20 Thread Ehsan Akhgari
On 2013-11-20 12:37 PM, Benoit Jacob wrote: Talking about ideas for further extending the impact of UNIFIED_SOURCES, it seems that the biggest limitation at the moment is that sources can't be unified between different moz.build's. Because of that, source directories that consist of many small su

Re: Recent build time improvements due to unified sources

2013-11-20 Thread Gregory Szorc
On Nov 20, 2013, at 9:37, Benoit Jacob wrote: > Talking about ideas for further extending the impact of UNIFIED_SOURCES, it > seems that the biggest limitation at the moment is that sources can't be > unified between different moz.build's. Because of that, source directories > that consist of man

Re: Recent build time improvements due to unified sources

2013-11-20 Thread Benoit Jacob
2013/11/20 Gregory Szorc > On Nov 20, 2013, at 9:37, Benoit Jacob wrote: > > > Talking about ideas for further extending the impact of UNIFIED_SOURCES, > it > > seems that the biggest limitation at the moment is that sources can't be > > unified between different moz.build's. Because of that, so

Re: Recent build time improvements due to unified sources

2013-11-20 Thread Patrick McManus
I was skeptical of this work - so I need to say now that it is paying dividends bigger and faster than I thought it could. very nice! On Wed, Nov 20, 2013 at 3:38 AM, Nicholas Nethercote wrote: > On September 12, a debug clobber build on my new Linux desktop took > 12.7 minutes. Just then it t

Re: Recent build time improvements due to unified sources

2013-11-20 Thread Benoit Jacob
Talking about ideas for further extending the impact of UNIFIED_SOURCES, it seems that the biggest limitation at the moment is that sources can't be unified between different moz.build's. Because of that, source directories that consist of many small sub-directories do not benefit much from UNIFIED

Re: Recent build time improvements due to unified sources

2013-11-20 Thread Chris Peterson
On 11/19/13, 10:08 PM, Gregory Szorc wrote: And 24 hours later, m-c (4f993fa378eb) is getting faster: Wall 8:47 (527) User 52:41 (3161) Sys4:38 (278) Total 57:19 (3439) Unified builds currently coalesce source files in batches of 16. It might be useful to add a files_per_unified_file

Re: Recent build time improvements due to unified sources

2013-11-20 Thread Nicholas Nethercote
On September 12, a debug clobber build on my new Linux desktop took 12.7 minutes. Just then it took 7.5 minutes. Woo! Nick ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform

Re: Recent build time improvements due to unified sources

2013-11-19 Thread Gregory Szorc
On 11/18/13, 11:15 PM, Gregory Szorc wrote: Do builds feel like they've gotten faster in the last few weeks^hours? It's because they have. When I did my MacBook Pro comparison [1] with a changeset from Oct 28, build times on my 2013 2.6 GHz MacBook Pro were as follows: Wall 11:13 (673) User

Re: Recent build time improvements due to unified sources

2013-11-19 Thread Mike Hommey
On Mon, Nov 18, 2013 at 11:15:16PM -0800, Gregory Szorc wrote: > Do builds feel like they've gotten faster in the last few weeks^hours? > It's because they have. > > When I did my MacBook Pro comparison [1] with a changeset from Oct 28, > build times on my 2013 2.6 GHz MacBook Pro were as follows:

Recent build time improvements due to unified sources

2013-11-18 Thread Gregory Szorc
Do builds feel like they've gotten faster in the last few weeks^hours? It's because they have. When I did my MacBook Pro comparison [1] with a changeset from Oct 28, build times on my 2013 2.6 GHz MacBook Pro were as follows: Wall 11:13 (673) User 69:55 (4195) Sys6:04 (364) Total 75:59 (4