On Sat, Nov 19, 2016 at 1:32 PM, Ehsan Akhgari <ehsan.akhg...@gmail.com> wrote:
> On 2016-11-19 4:26 PM, Mike Hommey wrote: > > On Sat, Nov 19, 2016 at 04:12:22PM -0500, Ehsan Akhgari wrote: > >> On 2016-11-18 7:22 PM, Gregory Szorc wrote: > >>> On Fri, Nov 18, 2016 at 3:50 PM, Ehsan Akhgari < > ehsan.akhg...@gmail.com > >>> <mailto:ehsan.akhg...@gmail.com>> wrote: > >>> > >>> Here are the moz.build processing times on my machine when I add > one > >>> line to a LOCAL_INCLUDES in one moz.build file: > >>> > >>> 0:02.97 Reticulating splines... > >>> 0:37.16 Finished reading 3131 moz.build files in 23.09s > >>> 0:37.16 Processed into 9325 build config descriptors in 3.62s > >>> 0:37.16 RecursiveMake backend executed in 6.50s > >>> 0:37.16 2951 total backend files; 0 created; 2 updated; 2949 > >>> unchanged; 0 deleted; 54 -> 1110 Makefile > >>> 0:37.16 FasterMake backend executed in 0.50s > >>> 0:37.16 12 total backend files; 0 created; 0 updated; 12 > unchanged; 0 > >>> deleted > >>> 0:37.16 Total wall time: 34.22s; CPU time: 16.20s; Efficiency: > 47%; > >>> Untracked: 0.50s > >>> > >>> > >>> 0:37.89 Finished reading 3131 moz.build files in 21.90s > >>> 0:37.89 Processed into 9327 build config descriptors in 3.82s > >>> 0:37.89 RecursiveMake backend executed in 7.85s > >>> 0:37.89 2951 total backend files; 0 created; 4 updated; 2947 > >>> unchanged; 0 deleted; 54 -> 1110 Makefile > >>> 0:37.89 FasterMake backend executed in 0.59s > >>> 0:37.89 12 total backend files; 0 created; 0 updated; 12 > unchanged; 0 > >>> deleted > >>> 0:37.89 Total wall time: 34.70s; CPU time: 17.70s; Efficiency: > 51%; > >>> Untracked: 0.55s > >>> > >>> The clear offender is "reading moz.build files". What's involved > in > >>> this process? Is there some low hanging fruit for making it > faster? > >>> Are there things non-build developers can do (such as reducing the > >>> number of moz.build files) that would help? > > > > Please note that "reading moz.build files" is actually misleading, > > because usually half of the time is spent not reading moz.build files. > > That also seems like a lot of time to spend on reading a "small" number > > of files (because the printed number of files is actually wrong, see bug > > 1208380). It would be interesting to see a profile for that "reading > > moz.build files" phase. > > If there is an easy thing I can do to capture such a profile, please let > me know. I don't know where in the code you're referring to, and I have > also never really profiled Python code. :-) > You can profile any Python process by running it with: $ python -mcProfile <program> <args> e.g. $ python -mcProfile mach build-backend You probably want to sort the output using "tottime" or "cumtime" (total and cumulative) respectively. e.g. $ python -mcProfile -s cumtime mach build-backend That's the best profiler that Python has built in. Unfortunately, the profiler works by registering a hook every time a function is called or finishes. If your code contains a lot of function calls, the profiler overhead can skew results towards functions that are called frequently. I've found statistical/sampling profilers do a much better job (and don't slow down your code during profiling as much). There is a "statprof" profiler floating around. Unfortunately, we'd need to modify `mach` to invoke it, as it doesn't provide a CLI API for recording (only reporting). And the version on PyPI is somewhat old. The version vendored in the Mercurial repository at https://www.mercurial-scm.org/repo/hg/raw-file/default/mercurial/statprof.py contains a number of improvements, including support for showing the "hot path" of code execution. I would not be opposed to vendoring it in mozilla-central and adding `mach --profile` support. Let us know what cProfile says and we can go from there. > > >>> > >>> Is there any way for us to avoid reprocessing the world for every > single > >>> moz.build file change? > >>> > >>> > >>> Please tell us a bit more about your machine. Because on my machine... > >>> > >>> Linux > >>> $ ./mach build-backend > >>> 0:00.08 > >>> /home/gps/src/firefox/obj-x86_64-pc-linux-gnu/_virtualenv/bin/python > >>> /home/gps/src/firefox/obj-x86_64-pc-linux-gnu/config.status > >>> Reticulating splines... > >>> Finished reading 3027 moz.build files in 3.79s > >>> Processed into 9028 build config descriptors in 1.54s > >>> RecursiveMake backend executed in 3.00s > >>> 2829 total backend files; 0 created; 0 updated; 2829 unchanged; 0 > >>> deleted; 54 -> 1065 Makefile > >>> FasterMake backend executed in 0.22s > >>> 12 total backend files; 0 created; 0 updated; 12 unchanged; 0 deleted > >>> Total wall time: 8.76s; CPU time: 7.26s; Efficiency: 83%; Untracked: > 0.21s > >>> > >>> Windows 10 > >>> $ ./mach build-backend > >>> 0:00.55 > >>> c:/dev/src/firefox/obj-i686-pc-mingw32\_virtualenv\Scripts\python.exe > >>> c:/dev/src/firefox/obj-i686-pc-mingw32\config.status > >>> Reticulating splines... > >>> Finished reading 3041 moz.build files in 7.01s > >>> Processed into 9841 build config descriptors in 3.19s > >>> RecursiveMake backend executed in 3.81s > >>> 2860 total backend files; 0 created; 1 updated; 2859 unchanged; 0 > >>> deleted; 62 -> 1069 Makefile > >>> FasterMake backend executed in 0.38s > >>> 11 total backend files; 0 created; 1 updated; 10 unchanged; 0 deleted > >>> VisualStudio backend executed in 1.71s > >>> Generated Visual Studio solution at > >>> c:/dev/src/firefox/obj-i686-pc-mingw32\msvc\mozilla.sln > >>> Total wall time: 16.43s; CPU time: 16.43s; Efficiency: 100%; Untracked: > >>> 0.33s > >>> > >>> The numbers seem to indicate your machine is a bit old or slow? > >> > >> Hmm, my machine is a MacBook Pro (this is my main development machine). > >> It's a MacBook Pro (Retina, 15-inch, Mid 2014), 2.8Ghz processor, 16GB > >> Memory, 1TB APPLE SSD SM1024F, Running Sierra 10.12.1. > >> > >> My machine wasn't entirely idle, I had things like a Terminal with a > >> bunch of vim's open, Nightly, Chrome, Thunderbird, LimeChat. But these > >> numbers are in the usual ballpark I get. > >> > >>> FWIW, bug 1308982 is on our radar for speeding up moz.build reading. > >> > >> That's great! I think that will help people like me a lot. Thankfully > >> Chris seems to be currently working on it. > >> > >> But I'm still not sure why we can't only re-process the moz.build file > >> that has changed... > > > > Not sure there's a bug on file, but we've been talking about doing this > > for a long time. It's not exactly trivial, but we'll get there > > eventually. > > Thank you, that is good to know. > > Cheers, > Ehsan > >
_______________________________________________ dev-builds mailing list dev-builds@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-builds
