We recently dealt with something similar in Gecko. See https://bugzilla.mozilla.org/show_bug.cgi?id=1152046
- Kyle On Mon, Sep 28, 2015 at 2:41 PM, Gregory Szorc <g...@mozilla.com> wrote: > As part of profiling a Python process on Windows using Process Monitor.exe, > I noticed that closing file handles using CloseFile() takes 1+ms. Contrast > this with other I/O related system calls like WriteFile() that tend to take > ~1us. (I /think/ it only takes a longer time if the file has been written > to.) This is on Windows 10 running natively (no VM) when writing to an SSD. > Files are opened with _fopen() in "a+" mode if it matters. I can also repro > in "a" mode. > > When writing thousands of files in rapid succession, this 1+ms pause > (assuming synchronous I/O) piles up. Assuming a 1ms pause, writing 100,000 > files spends 100s in CloseFile()! The process profile also shows the bulk > of the time in CloseFile(), so this is a real hot spot. > > Both Mercurial and the Firefox build system need to write thousands of > files in rapid order. Both are using Python. Both should experience > significant performance improvements if we find a more efficient way to do > bulk file writing (which currently is over 2x slower than Linux and OS X). > > Short of going full overlapped I/O (which will be a PITA with Python), the > best I can think of is to perform CloseFile() on a background thread or > pool of background threads. > > I was curious if anyone has dug into optimizing the writing of thousands of > files on Windows and can share any insights so we can make the Firefox > build system and Mercurial significantly faster on Windows. > > gps > _______________________________________________ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
