On 2016-11-19 4:26 PM, Mike Hommey wrote: > On Sat, Nov 19, 2016 at 04:12:22PM -0500, Ehsan Akhgari wrote: >> On 2016-11-18 7:22 PM, Gregory Szorc wrote: >>> On Fri, Nov 18, 2016 at 3:50 PM, Ehsan Akhgari <ehsan.akhg...@gmail.com >>> <mailto:ehsan.akhg...@gmail.com>> wrote: >>> >>> Here are the moz.build processing times on my machine when I add one >>> line to a LOCAL_INCLUDES in one moz.build file: >>> >>> 0:02.97 Reticulating splines... >>> 0:37.16 Finished reading 3131 moz.build files in 23.09s >>> 0:37.16 Processed into 9325 build config descriptors in 3.62s >>> 0:37.16 RecursiveMake backend executed in 6.50s >>> 0:37.16 2951 total backend files; 0 created; 2 updated; 2949 >>> unchanged; 0 deleted; 54 -> 1110 Makefile >>> 0:37.16 FasterMake backend executed in 0.50s >>> 0:37.16 12 total backend files; 0 created; 0 updated; 12 unchanged; 0 >>> deleted >>> 0:37.16 Total wall time: 34.22s; CPU time: 16.20s; Efficiency: 47%; >>> Untracked: 0.50s >>> >>> >>> 0:37.89 Finished reading 3131 moz.build files in 21.90s >>> 0:37.89 Processed into 9327 build config descriptors in 3.82s >>> 0:37.89 RecursiveMake backend executed in 7.85s >>> 0:37.89 2951 total backend files; 0 created; 4 updated; 2947 >>> unchanged; 0 deleted; 54 -> 1110 Makefile >>> 0:37.89 FasterMake backend executed in 0.59s >>> 0:37.89 12 total backend files; 0 created; 0 updated; 12 unchanged; 0 >>> deleted >>> 0:37.89 Total wall time: 34.70s; CPU time: 17.70s; Efficiency: 51%; >>> Untracked: 0.55s >>> >>> The clear offender is "reading moz.build files". What's involved in >>> this process? Is there some low hanging fruit for making it faster? >>> Are there things non-build developers can do (such as reducing the >>> number of moz.build files) that would help? > > Please note that "reading moz.build files" is actually misleading, > because usually half of the time is spent not reading moz.build files. > That also seems like a lot of time to spend on reading a "small" number > of files (because the printed number of files is actually wrong, see bug > 1208380). It would be interesting to see a profile for that "reading > moz.build files" phase.
If there is an easy thing I can do to capture such a profile, please let me know. I don't know where in the code you're referring to, and I have also never really profiled Python code. :-) >>> >>> Is there any way for us to avoid reprocessing the world for every single >>> moz.build file change? >>> >>> >>> Please tell us a bit more about your machine. Because on my machine... >>> >>> Linux >>> $ ./mach build-backend >>> 0:00.08 >>> /home/gps/src/firefox/obj-x86_64-pc-linux-gnu/_virtualenv/bin/python >>> /home/gps/src/firefox/obj-x86_64-pc-linux-gnu/config.status >>> Reticulating splines... >>> Finished reading 3027 moz.build files in 3.79s >>> Processed into 9028 build config descriptors in 1.54s >>> RecursiveMake backend executed in 3.00s >>> 2829 total backend files; 0 created; 0 updated; 2829 unchanged; 0 >>> deleted; 54 -> 1065 Makefile >>> FasterMake backend executed in 0.22s >>> 12 total backend files; 0 created; 0 updated; 12 unchanged; 0 deleted >>> Total wall time: 8.76s; CPU time: 7.26s; Efficiency: 83%; Untracked: 0.21s >>> >>> Windows 10 >>> $ ./mach build-backend >>> 0:00.55 >>> c:/dev/src/firefox/obj-i686-pc-mingw32\_virtualenv\Scripts\python.exe >>> c:/dev/src/firefox/obj-i686-pc-mingw32\config.status >>> Reticulating splines... >>> Finished reading 3041 moz.build files in 7.01s >>> Processed into 9841 build config descriptors in 3.19s >>> RecursiveMake backend executed in 3.81s >>> 2860 total backend files; 0 created; 1 updated; 2859 unchanged; 0 >>> deleted; 62 -> 1069 Makefile >>> FasterMake backend executed in 0.38s >>> 11 total backend files; 0 created; 1 updated; 10 unchanged; 0 deleted >>> VisualStudio backend executed in 1.71s >>> Generated Visual Studio solution at >>> c:/dev/src/firefox/obj-i686-pc-mingw32\msvc\mozilla.sln >>> Total wall time: 16.43s; CPU time: 16.43s; Efficiency: 100%; Untracked: >>> 0.33s >>> >>> The numbers seem to indicate your machine is a bit old or slow? >> >> Hmm, my machine is a MacBook Pro (this is my main development machine). >> It's a MacBook Pro (Retina, 15-inch, Mid 2014), 2.8Ghz processor, 16GB >> Memory, 1TB APPLE SSD SM1024F, Running Sierra 10.12.1. >> >> My machine wasn't entirely idle, I had things like a Terminal with a >> bunch of vim's open, Nightly, Chrome, Thunderbird, LimeChat. But these >> numbers are in the usual ballpark I get. >> >>> FWIW, bug 1308982 is on our radar for speeding up moz.build reading. >> >> That's great! I think that will help people like me a lot. Thankfully >> Chris seems to be currently working on it. >> >> But I'm still not sure why we can't only re-process the moz.build file >> that has changed... > > Not sure there's a bug on file, but we've been talking about doing this > for a long time. It's not exactly trivial, but we'll get there > eventually. Thank you, that is good to know. Cheers, Ehsan _______________________________________________ dev-builds mailing list dev-builds@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-builds
