Re: Unified builds

2013-11-26 Thread Neil
Ehsan Akhgari wrote: It's hard to say exactly what's going on here without knowing more about how this build is produced. These are intermittent failures on TBPL, mostly starred by philor. It would be really great if you could file a bug about this with more details on how to reproduce this

Re: Unified builds

2013-11-25 Thread Ehsan Akhgari
On 2013-11-23 7:22 AM, ISHIKAWA,chiaki wrote: (2013/11/23 1:41), Ehsan Akhgari wrote: On 2013-11-21 1:12 PM, ISHIKAWA,chiaki wrote: (2013/11/22 2:17), Ehsan Akhgari wrote: FWIW if this proves to be common, it's a huge problem since it would affect our crash stats etc... :( Well, if the prob

Re: Unified builds

2013-11-23 Thread ISHIKAWA,chiaki
(2013/11/23 1:41), Ehsan Akhgari wrote: On 2013-11-21 1:12 PM, ISHIKAWA,chiaki wrote: (2013/11/22 2:17), Ehsan Akhgari wrote: FWIW if this proves to be common, it's a huge problem since it would affect our crash stats etc... :( Well, if the problem is related to the symptom that I observed w

Re: Unified builds

2013-11-22 Thread Mike Hommey
On Fri, Nov 22, 2013 at 07:24:14PM -0500, Ehsan Akhgari wrote: > On 2013-11-22 4:25 PM, Gregory Szorc wrote: > >On Nov 22, 2013, at 13:16, Ehsan Akhgari wrote: > > > >>On Fri, Nov 22, 2013 at 1:02 PM, Boris Zbarsky wrote: > >> > >>>On 11/22/13 11:41 AM, Ehsan Akhgari wrote: > >>> > Hmm to the

Re: Unified builds

2013-11-22 Thread Ehsan Akhgari
On 2013-11-22 4:25 PM, Gregory Szorc wrote: On Nov 22, 2013, at 13:16, Ehsan Akhgari wrote: On Fri, Nov 22, 2013 at 1:02 PM, Boris Zbarsky wrote: On 11/22/13 11:41 AM, Ehsan Akhgari wrote: Hmm to the best of my knowledge, we don't generate the *.i files unless if you explicitly request th

Re: Unified builds

2013-11-22 Thread Gregory Szorc
On Nov 22, 2013, at 13:16, Ehsan Akhgari wrote: > On Fri, Nov 22, 2013 at 1:02 PM, Boris Zbarsky wrote: > >> On 11/22/13 11:41 AM, Ehsan Akhgari wrote: >> >>> Hmm to the best of my knowledge, we don't generate the *.i files unless >>> if you explicitly request them. Is that what you did in th

Re: Unified builds

2013-11-22 Thread Ehsan Akhgari
On Fri, Nov 22, 2013 at 1:02 PM, Boris Zbarsky wrote: > On 11/22/13 11:41 AM, Ehsan Akhgari wrote: > >> Hmm to the best of my knowledge, we don't generate the *.i files unless >> if you explicitly request them. Is that what you did in that build? >> > > ccache obvously always generates .i files,

Re: Unified builds

2013-11-22 Thread Boris Zbarsky
On 11/22/13 11:41 AM, Ehsan Akhgari wrote: Hmm to the best of my knowledge, we don't generate the *.i files unless if you explicitly request them. Is that what you did in that build? ccache obvously always generates .i files, since those are what it uses as its cache key. And then it compil

Re: Unified builds

2013-11-22 Thread Ehsan Akhgari
On 2013-11-22 8:33 AM, Mike Hommey wrote: On Thu, Nov 14, 2013 at 05:49:33PM -0500, Ehsan Akhgari wrote: I've started to work on a project in my spare time to switch us to use unified builds for C/C++ compilation. The way that unified builds work is by using the UNIFIED_SOURCES instead of the S

Re: Unified builds

2013-11-22 Thread Ehsan Akhgari
On 2013-11-21 1:12 PM, ISHIKAWA,chiaki wrote: (2013/11/22 2:17), Ehsan Akhgari wrote: FWIW if this proves to be common, it's a huge problem since it would affect our crash stats etc... :( Well, if the problem is related to the symptom that I observed with the use of -gsplit-dwarf option with

Re: Unified builds

2013-11-22 Thread Mike Hommey
On Thu, Nov 14, 2013 at 05:49:33PM -0500, Ehsan Akhgari wrote: > I've started to work on a project in my spare time to switch us to use > unified builds for C/C++ compilation. The way that unified builds work is > by using the UNIFIED_SOURCES instead of the SOURCES variable in moz.build > files.

Re: Unified builds

2013-11-21 Thread Ehsan Akhgari
On 2013-11-21 10:06 AM, Michael Shal wrote: From: "Gregory Szorc" Unified sources have probably saved sufficient CPU cycles across all of automation to more than offset having a non-unified build-only job. I'll defer to the Platform Team (Ehsan?) to request that from Release Engineering. How m

Re: Unified builds

2013-11-21 Thread Ehsan Akhgari
On 2013-11-21 12:00 PM, ISHIKAWA,chiaki wrote: (2013/11/22 1:47), Ehsan Akhgari wrote: On 2013-11-21 11:41 AM, ISHIKAWA,chiaki wrote: (2013/11/15 7:49), Ehsan Akhgari wrote: I've started to work on a project in my spare time to switch us to use unified builds for C/C++ compilation. The way th

Re: Unified builds

2013-11-21 Thread Gregory Szorc
On 11/21/13, 7:06 AM, Michael Shal wrote: From: "Gregory Szorc" Unified sources have probably saved sufficient CPU cycles across all of automation to more than offset having a non-unified build-only job. I'll defer to the Platform Team (Ehsan?) to request that from Release Engineering. How man

Re: Unified builds (regarding periodic clobber)

2013-11-21 Thread Karl Tomlinson
Gregory Szorc writes: > Do we need periodic clobber? It's just wallpapering over legit > clobber-needed issues, which doesn't exactly help us fix them. The problem is that unfixed bugs in dependences mean that code bugs can sometimes not show up until the next clobber. There is then a huge regre

Re: Unified builds

2013-11-21 Thread Michael Shal
> From: "Gregory Szorc" > Unified sources have probably saved sufficient CPU cycles across all of > automation to more than offset having a non-unified build-only job. I'll > defer to the Platform Team (Ehsan?) to request that from Release > Engineering. How many CPU cycles would we have saved ac

Re: Unified builds (regarding periodic clobber)

2013-11-21 Thread Gregory Szorc
On 11/21/13, 8:40 AM, Ehsan Akhgari wrote: On 2013-11-21 10:44 AM, Ed Morley wrote: On 21 November 2013 15:38:28, Ehsan Akhgari wrote: We don't, we clobber periodically. I'm not sure why that is better than never clobbering... We've been force-clobbering 1-2 times a day in automation for sev

Re: Unified builds

2013-11-21 Thread Ehsan Akhgari
On 2013-11-21 11:41 AM, ISHIKAWA,chiaki wrote: (2013/11/15 7:49), Ehsan Akhgari wrote: I've started to work on a project in my spare time to switch us to use unified builds for C/C++ compilation. The way that unified builds work is by using the UNIFIED_SOURCES instead of the SOURCES variable in

Re: Unified builds

2013-11-21 Thread Ed Morley
On 21 November 2013 15:38:28, Ehsan Akhgari wrote: We don't, we clobber periodically. I'm not sure why that is better than never clobbering... We've been force-clobbering 1-2 times a day in automation for several months now sadly, due to bug 928195 and similar prior. (Patches almost ready to

Re: Unified builds

2013-11-21 Thread Ehsan Akhgari
On 2013-11-21 10:44 AM, Ed Morley wrote: On 21 November 2013 15:38:28, Ehsan Akhgari wrote: We don't, we clobber periodically. I'm not sure why that is better than never clobbering... We've been force-clobbering 1-2 times a day in automation for several months now sadly, due to bug 928195 and

Re: Unified builds

2013-11-21 Thread ISHIKAWA,chiaki
(2013/11/15 7:49), Ehsan Akhgari wrote: I've started to work on a project in my spare time to switch us to use unified builds for C/C++ compilation. The way that unified builds work is by using the UNIFIED_SOURCES instead of the SOURCES variable in moz.build files. With that, the build system c

Re: Unified builds (regarding periodic clobber)

2013-11-21 Thread Ed Morley
> On 11/21/13, 8:40 AM, Ehsan Akhgari wrote: >> Yes, but our periodic clobber has been in effect long before that bug >> (and in fact for as long as I can remember.) Yes, but it's only once a week rather than a couple of times a day :-) On 21/11/2013 16:45, Gregory Szorc wrote: Do we need perio

Re: Unified builds

2013-11-21 Thread ISHIKAWA,chiaki
(2013/11/22 1:47), Ehsan Akhgari wrote: On 2013-11-21 11:41 AM, ISHIKAWA,chiaki wrote: (2013/11/15 7:49), Ehsan Akhgari wrote: I've started to work on a project in my spare time to switch us to use unified builds for C/C++ compilation. The way that unified builds work is by using the UNIFIED_S

Re: Unified builds

2013-11-21 Thread ISHIKAWA,chiaki
(2013/11/22 2:17), Ehsan Akhgari wrote: FWIW if this proves to be common, it's a huge problem since it would affect our crash stats etc... :( Well, if the problem is related to the symptom that I observed with the use of -gsplit-dwarf option with ccache, then we may be in for a big surprise.

Re: Unified builds

2013-11-20 Thread Ehsan Akhgari
On 2013-11-20 5:17 PM, Gregory Szorc wrote: On 11/20/13, 2:02 PM, Zack Weinberg wrote: On 2013-11-18 5:41 PM, Ehsan Akhgari wrote: I wouldn't go all the way to that extreme. The list of broken system headers is unfortunately quite large, and we already have lots of this pattern in the tree:

Re: Unified builds

2013-11-20 Thread Gregory Szorc
On 11/20/13, 2:02 PM, Zack Weinberg wrote: On 2013-11-18 5:41 PM, Ehsan Akhgari wrote: I wouldn't go all the way to that extreme. The list of broken system headers is unfortunately quite large, and we already have lots of this pattern in the tree: #ifdef MyFunction // screw you, windows.h/X.h

Re: Unified builds

2013-11-20 Thread Zack Weinberg
On 2013-11-18 5:41 PM, Ehsan Akhgari wrote: I wouldn't go all the way to that extreme. The list of broken system headers is unfortunately quite large, and we already have lots of this pattern in the tree: #ifdef MyFunction // screw you, windows.h/X.h/etc. #undef MyFunction #endif A small not

Re: Unified builds

2013-11-19 Thread Gregory Szorc
On 11/14/13, 2:49 PM, Ehsan Akhgari wrote: I've started to work on a project in my spare time to switch us to use unified builds for C/C++ compilation. The way that unified builds work is by using the UNIFIED_SOURCES instead of the SOURCES variable in moz.build files. With that, the build syste

Re: Unified builds

2013-11-18 Thread L. David Baron
On Tuesday 2013-11-19 17:08 +1300, Robert O'Callahan wrote: > Fortunately two static variables with the same name in the same translation > unit is an error in C++, at least with gcc. Ah, indeed. I'd tested in C, where it wasn't an error, but I also see an error with gcc in C++. -David -- 𝄞

Re: Unified builds

2013-11-18 Thread Robert O'Callahan
On Tue, Nov 19, 2013 at 4:04 PM, L. David Baron wrote: > I expect that otherwise we'd get pretty frequent bustage with people > forgetting to add #includes, and that bustage would then show up > when we add or remove files, which would make it a huge pain to add > or remove files. > > But I'm act

Re: Unified builds

2013-11-18 Thread Ehsan Akhgari
On 2013-11-18 10:16 PM, Mike Hommey wrote: On Tue, Nov 19, 2013 at 11:04:25AM +0800, L. David Baron wrote: On Monday 2013-11-18 18:44 -0500, Ehsan Akhgari wrote: On 2013-11-17 7:50 PM, L. David Baron wrote: On Sunday 2013-11-17 16:45 -0800, Jonas Sicking wrote: On Thu, Nov 14, 2013 at 2:49 PM

Re: Unified builds

2013-11-18 Thread Ehsan Akhgari
On 2013-11-18 10:04 PM, L. David Baron wrote: On Monday 2013-11-18 18:44 -0500, Ehsan Akhgari wrote: On 2013-11-17 7:50 PM, L. David Baron wrote: On Sunday 2013-11-17 16:45 -0800, Jonas Sicking wrote: On Thu, Nov 14, 2013 at 2:49 PM, Ehsan Akhgari wrote: I've started to work on a project in

Re: Unified builds

2013-11-18 Thread Mike Hommey
On Tue, Nov 19, 2013 at 11:04:25AM +0800, L. David Baron wrote: > On Monday 2013-11-18 18:44 -0500, Ehsan Akhgari wrote: > > On 2013-11-17 7:50 PM, L. David Baron wrote: > > >On Sunday 2013-11-17 16:45 -0800, Jonas Sicking wrote: > > >>On Thu, Nov 14, 2013 at 2:49 PM, Ehsan Akhgari > > >>wrote: >

Re: Unified builds

2013-11-18 Thread L. David Baron
On Monday 2013-11-18 18:44 -0500, Ehsan Akhgari wrote: > On 2013-11-17 7:50 PM, L. David Baron wrote: > >On Sunday 2013-11-17 16:45 -0800, Jonas Sicking wrote: > >>On Thu, Nov 14, 2013 at 2:49 PM, Ehsan Akhgari > >>wrote: > >>>I've started to work on a project in my spare time to switch us to use

Re: Unified builds

2013-11-18 Thread Ehsan Akhgari
On 2013-11-17 7:50 PM, L. David Baron wrote: On Sunday 2013-11-17 16:45 -0800, Jonas Sicking wrote: On Thu, Nov 14, 2013 at 2:49 PM, Ehsan Akhgari wrote: I've started to work on a project in my spare time to switch us to use unified builds for C/C++ compilation. The way that unified builds wo

Re: Unified builds

2013-11-18 Thread Ehsan Akhgari
On 2013-11-17 7:45 PM, Jonas Sicking wrote: On Thu, Nov 14, 2013 at 2:49 PM, Ehsan Akhgari wrote: I've started to work on a project in my spare time to switch us to use unified builds for C/C++ compilation. The way that unified builds work is by using the UNIFIED_SOURCES instead of the SOURCES

Re: Unified builds

2013-11-18 Thread Ehsan Akhgari
On 2013-11-18 9:53 AM, Boris Zbarsky wrote: On 11/18/13 9:49 AM, Benoit Jacob wrote: 2) Keep these cpp files, that #include system headers, in plain old SOURCES, not in UNIFIED_SOURCES. Maybe that's the right way to go. Enforce that no .cpp file that's in UNIFIED_SOURCES ever includes the b

Re: Unified builds

2013-11-18 Thread Ehsan Akhgari
On 2013-11-18 3:52 PM, Robert O'Callahan wrote: On Tue, Nov 19, 2013 at 3:31 AM, Boris Zbarsky wrote: While true, in the new setup we have a different problem: adding or removing a .cpp file makes other random .cpp files not compile. I don't think we should worry much about this until we ha

Re: Unified builds

2013-11-18 Thread Ehsan Akhgari
On 2013-11-17 7:53 PM, Jonas Sicking wrote: On Sat, Nov 16, 2013 at 1:34 AM, Ms2ger wrote: One issue I only thought of today: this means that Source2.cpp can use all headers included into Source1.cpp–until enough files are added between Source1 and Source2 that Source2 ends up in the next unif

Re: Unified builds

2013-11-18 Thread Ehsan Akhgari
On 2013-11-18 4:33 AM, Neil wrote: Ehsan Akhgari wrote: I've started to work on a project in my spare time to switch us to use unified builds for C/C++ compilation. The way that unified builds work is by using the UNIFIED_SOURCES instead of the SOURCES variable in moz.build files. With that,

Re: Unified builds

2013-11-18 Thread Ehsan Akhgari
On 2013-11-17 5:39 PM, Mike Hommey wrote: On Sat, Nov 16, 2013 at 10:34:38AM +0100, Ms2ger wrote: On 11/14/2013 11:49 PM, Ehsan Akhgari wrote: I've started to work on a project in my spare time to switch us to use unified builds for C/C++ compilation. The way that unified builds work is by usi

Re: Unified builds

2013-11-18 Thread Robert O'Callahan
On Tue, Nov 19, 2013 at 3:31 AM, Boris Zbarsky wrote: > While true, in the new setup we have a different problem: adding or > removing a .cpp file makes other random .cpp files not compile. > I don't think we should worry much about this until we have more data on how often it's a problem in pra

Re: Unified builds

2013-11-18 Thread Chris Peterson
On 11/16/13, 1:34 AM, Ms2ger wrote: One way around it would be not to unify sources in automation. On one hand, this could cause more bustage when changes that built locally turn out not to have enough includes; on the other, it might be better than having to fix up a dozen unrelated files whenev

Re: Unified builds

2013-11-18 Thread Neil
Boris Zbarsky wrote: On 11/17/13 5:26 PM, Ehsan Akhgari wrote: I don't think that we need to try to fix this problem any more than the general problem of denoting our dependencies explicitly. It's common for you to remove an #include from a header and find dozens of .cpp files in the tree t

Re: Unified builds

2013-11-18 Thread Boris Zbarsky
On 11/18/13 9:49 AM, Benoit Jacob wrote: 2) Keep these cpp files, that #include system headers, in plain old SOURCES, not in UNIFIED_SOURCES. Maybe that's the right way to go. Enforce that no .cpp file that's in UNIFIED_SOURCES ever includes the broken system headers, like we already do fo

Re: Unified builds

2013-11-18 Thread Benoit Jacob
2013/11/18 Boris Zbarsky > On 11/17/13 5:26 PM, Ehsan Akhgari wrote: > >> I don't think that we need to try to fix this problem any more than the >> general problem of denoting our dependencies explicitly. It's common for >> you to remove an #include from a header and find dozens of .cpp files i

Re: Unified builds

2013-11-18 Thread Boris Zbarsky
On 11/17/13 5:26 PM, Ehsan Akhgari wrote: I don't think that we need to try to fix this problem any more than the general problem of denoting our dependencies explicitly. It's common for you to remove an #include from a header and find dozens of .cpp files in the tree that implicitly depended on

Re: Unified builds

2013-11-18 Thread Neil
Ehsan Akhgari wrote: I've started to work on a project in my spare time to switch us to use unified builds for C/C++ compilation. The way that unified builds work is by using the UNIFIED_SOURCES instead of the SOURCES variable in moz.build files. With that, the build system creates files su

Re: Unified builds

2013-11-17 Thread Benoit Jacob
Here is a wiki page to track our progress on this front and our means of synchronizing on this work: https://wiki.mozilla.org/Platform/Porting_to_unified_sources Benoit 2013/11/14 Ehsan Akhgari > I've started to work on a project in my spare time to switch us to use > unified builds for C/C++

Re: Unified builds

2013-11-17 Thread Jonas Sicking
On Sat, Nov 16, 2013 at 1:34 AM, Ms2ger wrote: > > One issue I only thought of today: this means that Source2.cpp can use all > headers included into Source1.cpp–until enough files are added between > Source1 and Source2 that Source2 ends up in the next unified file. This > hasn't been much of an

Re: Unified builds

2013-11-17 Thread L. David Baron
On Sunday 2013-11-17 16:45 -0800, Jonas Sicking wrote: > On Thu, Nov 14, 2013 at 2:49 PM, Ehsan Akhgari > wrote: > > I've started to work on a project in my spare time to switch us to use > > unified builds for C/C++ compilation. The way that unified builds work is > > by using the UNIFIED_SOURC

Re: Unified builds

2013-11-17 Thread Jonas Sicking
On Thu, Nov 14, 2013 at 2:49 PM, Ehsan Akhgari wrote: > I've started to work on a project in my spare time to switch us to use > unified builds for C/C++ compilation. The way that unified builds work is > by using the UNIFIED_SOURCES instead of the SOURCES variable in moz.build > files. With tha

Re: Unified builds

2013-11-17 Thread Mike Hommey
On Sat, Nov 16, 2013 at 10:34:38AM +0100, Ms2ger wrote: > On 11/14/2013 11:49 PM, Ehsan Akhgari wrote: > >I've started to work on a project in my spare time to switch us to use > >unified builds for C/C++ compilation. The way that unified builds work is > >by using the UNIFIED_SOURCES instead of t

Re: Unified builds

2013-11-17 Thread Ehsan Akhgari
On Sat, Nov 16, 2013 at 5:26 AM, Gabriele Svelto wrote: > On 14/11/2013 23:49, Ehsan Akhgari wrote: > >> The advantage of this is that it speeds up the compilation (I've measured >> between 6-15x speed improvement depending on the code in question >> locally.) >> > > Another advantage would be tha

Re: Unified builds

2013-11-17 Thread Ehsan Akhgari
On Sat, Nov 16, 2013 at 4:34 AM, Ms2ger wrote: > On 11/14/2013 11:49 PM, Ehsan Akhgari wrote: > >> I've started to work on a project in my spare time to switch us to use >> unified builds for C/C++ compilation. The way that unified builds work is >> >> by using the UNIFIED_SOURCES instead of the

Re: Unified builds

2013-11-16 Thread Gabriele Svelto
On 14/11/2013 23:49, Ehsan Akhgari wrote: The advantage of this is that it speeds up the compilation (I've measured between 6-15x speed improvement depending on the code in question locally.) Another advantage would be that for mostly self-contained modules we could get most of the benefits of

Re: Unified builds

2013-11-16 Thread Ms2ger
On 11/14/2013 11:49 PM, Ehsan Akhgari wrote: I've started to work on a project in my spare time to switch us to use unified builds for C/C++ compilation. The way that unified builds work is by using the UNIFIED_SOURCES instead of the SOURCES variable in moz.build files. With that, the build sys

Re: Unified builds

2013-11-15 Thread Ehsan Akhgari
On 2013-11-15 4:58 AM, Neil wrote: Ehsan Akhgari wrote: we go down from 1m51s on my machine to 31s And how does that compare to the case where you touch one .cpp file in each subdirectory of layout? It depends on which file, obviously, but here is an example with a random file: Before th

Re: Unified builds

2013-11-15 Thread Neil
Ehsan Akhgari wrote: we go down from 1m51s on my machine to 31s And how does that compare to the case where you touch one .cpp file in each subdirectory of layout? -- Warning: May contain traces of nuts. ___ dev-platform mailing list dev-platform@

Re: Unified builds

2013-11-14 Thread Ehsan Akhgari
On 2013-11-14 7:46 PM, Mike Hommey wrote: On Thu, Nov 14, 2013 at 07:01:03PM -0500, Ehsan Akhgari wrote: On 2013-11-14 6:29 PM, Chris Peterson wrote: On 11/14/13, 2:49 PM, Ehsan Akhgari wrote: The way that unified builds work is by using the UNIFIED_SOURCES instead of the SOURCES variable in m

Re: Unified builds

2013-11-14 Thread Mike Hommey
On Thu, Nov 14, 2013 at 07:01:03PM -0500, Ehsan Akhgari wrote: > On 2013-11-14 6:29 PM, Chris Peterson wrote: > >On 11/14/13, 2:49 PM, Ehsan Akhgari wrote: > >>The way that unified builds work is > >>by using the UNIFIED_SOURCES instead of the SOURCES variable in moz.build > >>files. With that, th

Re: Unified builds

2013-11-14 Thread Ehsan Akhgari
On 2013-11-14 6:29 PM, Chris Peterson wrote: On 11/14/13, 2:49 PM, Ehsan Akhgari wrote: The way that unified builds work is by using the UNIFIED_SOURCES instead of the SOURCES variable in moz.build files. With that, the build system creates files such as: // Unified_cpp_path_0.cpp #include "So

Re: Unified builds

2013-11-14 Thread Chris Peterson
On 11/14/13, 2:49 PM, Ehsan Akhgari wrote: The way that unified builds work is by using the UNIFIED_SOURCES instead of the SOURCES variable in moz.build files. With that, the build system creates files such as: // Unified_cpp_path_0.cpp #include "Source1.cpp" #include "Source2.cpp" // ... Are