Re: CloseFile() latency on Windows

2015-09-29 Thread Ehsan Akhgari
On 2015-09-29 1:47 PM, Gregory Szorc wrote: You'd be surprised. :-) Windows doesn't really have a notion of open file limits similar to Unix. File handles opened using _open can go up to a maximum of 2048. fopen has a cap of 512 which can be raised up to 2048 using _setmaxs

Re: CloseFile() latency on Windows

2015-09-29 Thread Gregory Szorc
On Tue, Sep 29, 2015 at 6:05 AM, Ehsan Akhgari wrote: > On 2015-09-29 12:52 AM, Gregory Szorc wrote: > >> On Mon, Sep 28, 2015 at 6:45 PM, Ehsan Akhgari > > wrote: >> >> On 2015-09-28 5:41 PM, Gregory Szorc wrote: >> >> When writing thousands of files i

Re: CloseFile() latency on Windows

2015-09-29 Thread Neil
Gregory Szorc wrote: Files are opened with _fopen() in "a+" mode if it matters. I can also repro in "a" mode. ... Short of going full overlapped I/O Overlapped I/O isn't supported for operations that change the valid data length of the file. http://blogs.msdn.com/b/oldnewthing/archive/

Re: CloseFile() latency on Windows

2015-09-29 Thread Ehsan Akhgari
On 2015-09-29 9:05 AM, Ehsan Akhgari wrote: You'd be surprised. :-) Windows doesn't really have a notion of open file limits similar to Unix. File handles opened using _open can go up to a maximum of 2048. fopen has a cap of 512 which can be raised up to 2048 using _setmaxstdio(). *But* these

Re: CloseFile() latency on Windows

2015-09-29 Thread Ehsan Akhgari
On 2015-09-29 12:52 AM, Gregory Szorc wrote: On Mon, Sep 28, 2015 at 6:45 PM, Ehsan Akhgari mailto:ehsan.akhg...@gmail.com>> wrote: On 2015-09-28 5:41 PM, Gregory Szorc wrote: When writing thousands of files in rapid succession, this 1+ms pause (assuming synchronous I/O) pil

Re: CloseFile() latency on Windows

2015-09-28 Thread Gregory Szorc
On Mon, Sep 28, 2015 at 6:45 PM, Ehsan Akhgari wrote: > On 2015-09-28 5:41 PM, Gregory Szorc wrote: > >> When writing thousands of files in rapid succession, this 1+ms pause >> (assuming synchronous I/O) piles up. Assuming a 1ms pause, writing 100,000 >> files spends 100s in CloseFile()! The proc

Re: CloseFile() latency on Windows

2015-09-28 Thread Ehsan Akhgari
On 2015-09-28 5:41 PM, Gregory Szorc wrote: When writing thousands of files in rapid succession, this 1+ms pause (assuming synchronous I/O) piles up. Assuming a 1ms pause, writing 100,000 files spends 100s in CloseFile()! The process profile also shows the bulk of the time in CloseFile(), so this

Re: CloseFile() latency on Windows

2015-09-28 Thread Kyle Huey
We recently dealt with something similar in Gecko. See https://bugzilla.mozilla.org/show_bug.cgi?id=1152046 - Kyle On Mon, Sep 28, 2015 at 2:41 PM, Gregory Szorc wrote: > As part of profiling a Python process on Windows using Process Monitor.exe, > I noticed that closing file handles using Clos

CloseFile() latency on Windows

2015-09-28 Thread Gregory Szorc
As part of profiling a Python process on Windows using Process Monitor.exe, I noticed that closing file handles using CloseFile() takes 1+ms. Contrast this with other I/O related system calls like WriteFile() that tend to take ~1us. (I /think/ it only takes a longer time if the file has been writte