There were a few complaints on this list about the performance of the build system on Windows; this mail provides some data on the problem.
I. Why is Cygwin make slow? 1. GNU make calls vfork(2), which is implemented[1] as fork(2) on Cygwin which, among other things, copies the entire process memory of the parent, and does several context switches. According to Task Manager, our non-recursive make process uses 110MB of memory, and copying that for every spawned process adds up. 2. to check time stamps, make calls stat(2) which is implemented by Cygwin as a series of Win32 system calls, followed perhaps by some mapping of Win32 user-ids to something more POSIXy, or opening the file to see if it starts with "#!", none of which matters to make. 3. Another problem of Cygwin make is token deflation: the job-server implementation (by which spawned make processes in bundled externals can build in parallel) is not reliable and is notorious for losing tokens, which is why it is currently recommended to use a higher -jN than on other platforms to avoid idle CPUs. II. GNU make configurations The following configurations were tested, pointing out any performance relevant differences from upstream releases: 1. LO patched Cygwin GNU make 3.82 downloaded from dev-www.libreoffice (purely guessing here, based on the git log:) - back-ported upstream 4.0 glob performance regression fix 2. LO patched Cygwin GNU make 3.82 from gnu-make-lo-3.82 branch - back-ported upstream 4.0 $(file) function - back-ported upstream 4.0 glob performance regression fix - built-in "cp" commands: heuristic to recognize the particular invocations of cp(1) that happen during the build and open-code these directly in make's C code git clone git://gerrit.libreoffice.org/gnu-make-lo git checkout gnu-make-lo-3.82 ./configure make 3. LO patched Cygwin GNU make 4.0 from gnu-make-lo-4.0 branch - built-in "cp" commands (as above) - "depcache" feature to speed-up incremental builds with a more space- efficient generated dependency file format git clone git://gerrit.libreoffice.org/gnu-make-lo git checkout gnu-make-lo-4.0 ./configure make 4. upstream Win32 GNU make 4.0+ (upstream current master 85047eb9044d4b72d50e6620c505c675d55ab98b) This is built with Visual Studio 2012; it's not entirely obvious how to build it, for example it requires a sed program in the PATH; this should work: git clone git://git.savannah.gnu.org/make.git [go to Start menu, click on "VS2012 x86 Native Tools Command Prompt"] set PATH=%PATH%;C:\Cygwin\bin cd .../path/to/make build_w32.bat --without-guile should result in a WinRel/gnumake.exe There were quite a few issues with both shell quoting and over-long command lines; one advantage of Cygwin is apparently that it uses a different way to transfer the argument vector to spawned processes; with Win32 CreateProcess the limit appears to be 8k and there is no argument vector, but simply a single command line string that the called process then has to split up again somehow; see also [3]. These issues are fixed on current LO master (and "gengal" can now read arguments from a response file). The only required patch that is not yet in master is: https://gerrit.libreoffice.org/#/c/9698/ Note1: a Win32 build of the upstream 4.0 release crashed with memory corruption, that's why i switched to current master, which does not have this problem. Note2: one would think that defining HAVE_CYGWIN_SHELL is a good idea when using Cygwin's shell, but experience indicates otherwise. In particular, it resulted in more quoting issues, contrary to what is stated in the documentation. Note3: there were problems with the very old Cygwin DLL 1.7.18 i had installed, crashing in sh.exe with message "fork: can't reserve memory for stack ... Win32 error 487" similar to [2], that went away after upgrading to current Cygwin DLL 1.7.30. III. Measurements These were taken on a Fedora 20 KVM with 12 logical cores and 16GB RAM, running Windows 7, with Cygwin DLL 1.7.30 and Visual Studio 2012. autogen.input: MAKE=E:/make/make-85047eb-msvc.exe #MAKE=/cygdrive/e/make/make-3.82-lo-cygwin.exe #MAKE=/cygdrive/e/make/make-4.0-lo-cygwin.exe --with-external-tar=/cygdrive/e/lo/ext_sources --with-lang=fr es --disable-ccache --enable-dbgutil --enable-werror --enable-pch --enable-ext-languagetool --enable-ext-wiki-publisher --enable-ext-nlpsolver --enable-ext-numbertext --with-ant-home=/cygdrive/c/Program Files/apache-ant-1.8.4 --with-junit=/home/mstahl/junit-4.10.jar --enable-crashdump --with-help --with-myspell-dicts --with-parallelism=20 --with-doxygen=E:/doxygen/doxygen/bin/doxygen --with-package-format=msi We measure build from scratch on an otherwise idle system of revision aa8b268935b07c9bf95a596ee84c76eed4c03634 (plus necessary build system fixes) via "time make check", taking note of how many job-server tokens were lost, and then the make overhead of an incremental rebuild by invoking make with --dry-run (so no commands are actually executed, make only checks which targets need to be rebuilt) via: sed -i Makefile -e "s/(build_goal)/(build_goal) --dry-run/" time make check legend: F = from-scratch build time T = job-server tokens at end of build I = incremental build make overhead time (4x) 1. "old" LO patched Cygwin GNU make 3.82 F: 111m49s T: 17/20 I: 3m24s 1m30s 1m31s 1m31s F: 113m58s T: 19/20 I: 3m32s 1m25s 1m22s 1m26s F: 112m23s T: 18/20 I: 3m25s 1m30s 1m27s 1m31s F: 116m15s 2. LO patched Cygwin GNU make 3.82 from gnu-make-lo-3.82 branch F: 88m12s T: 18/20 I: 2m56s 1m02s 1m01s 1m00s F: (112m58s) T: 19/20 I: 2m57s 1m00s 1m01s 1m01s ^^^^^^^^^ measurement error, failed test waits for click on dialog F: 89m40s T: 20/20 I: 3m02s 1m00s 1m00s 1m00s 3. LO patched Cygwin GNU make 4.0 from gnu-make-lo-4.0 branch F: 89m25s T: 16/20 I: 2m54s 0m48s 0m49s 0m50s F: 90m15s T: 15/20 I: 2m50s 0m48s 0m49s 0m49s F: 94m32s T: 20/20 I: 2m51s 0m49s 0m50s 0m49s 4. upstream Win32 GNU make 4.0+ F: 83m40s F: 85m19s F: 85m53s I: 0m49s 0m27s 0m28s 0m27s F: 80m17s I: 0m47s 0m34s 0m34s 0m34s F: 84m34s I: 0m45s 0m28s 0m29s 0m29s F: 84m23s I: 0m44s 0m29s 0m29s 0m28s Win32 make did not emit messages about missing tokens; hopefully that indicates no tokens were lost, and not that it wouldn't print an error if this happens. IV. Conclusion Using a native Win32 GNU make provides faster from-scratch rebuilds than any previous attempt to improve build performance, and significantly reduces the incremental re-build overhead of make on Windows by a factor of 2x-3x. Since there are not many developers using Windows anyway, i'd propose to have a flag day to switch to requiring Win32 make on master, dropping support for Cygwin make. This gerrit patch does the switch: https://gerrit.libreoffice.org/#/c/9698/ Of course, this first requires preparing tinderboxes, and testing by others if the Win32 make works well everywhere; please do so if you are able to do a Windows build of LO. [1] http://cygwin.com/cgi-bin/cvsweb.cgi/src/winsup/cygwin/fork.cc?rev=1.244&content-type=text/x-cvsweb-markup&cvsroot=src [2] https://cygwin.com/ml/cygwin/2011-02/msg00416.html [3] http://blogs.msdn.com/b/twistylittlepassagesallalike/archive/2011/04/23/everyone-quotes-arguments-the-wrong-way.aspx _______________________________________________ LibreOffice mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/libreoffice
