[issue9611] FileIO not 64-bit safe under Windows

STINNER Victor added the comment: Issue #9015 has been marked as a duplicate of this issue. -- ___ Python tracker ___ ___ Python-bugs-

[issue9611] FileIO not 64-bit safe under Windows

STINNER Victor added the comment: I backported fixes to 2.7, and also add a new fix to FileIO.read(). I don't see anything else to do on this issue, so I close it. Note: read() and write() methods the file object in 2.7 are 64-bit safe on any OS. They use fread() and frwrite() which take a le

[issue9611] FileIO not 64-bit safe under Windows

Roundup Robot added the comment: New changeset 7acdf9f5eb31 by Victor Stinner in branch '3.2': Issue #9611, #9015: FileIO.read() clamps the length to INT_MAX on Windows. http://hg.python.org/cpython/rev/7acdf9f5eb31 New changeset e8646f120330 by Victor Stinner in branch 'default': (merge 3.2) I

[issue9611] FileIO not 64-bit safe under Windows

Roundup Robot added the comment: New changeset 6abbc5f68e20 by Victor Stinner in branch '2.7': Issue #9611, #9015: FileIO.read(), FileIO.readinto(), FileIO.write() and http://hg.python.org/cpython/rev/6abbc5f68e20 -- nosy: +python-dev ___ Python trac

[issue9611] FileIO not 64-bit safe under Windows

STINNER Victor added the comment: r87917 removes (useless and dangerous) conversion to size_t. -- ___ Python tracker ___ ___ Python-bu

[issue9611] FileIO not 64-bit safe under Windows

STINNER Victor added the comment: r87734 fixes stdprinter.write(). -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubsc

[issue9611] FileIO not 64-bit safe under Windows

STINNER Victor added the comment: As asked by Antoine, I commited my patch: r87722. ... But I don't know if it fixes the issue or not, I don't have access to a Windows with more than 4 GB of memory. -- ___ Python tracker

[issue9611] FileIO not 64-bit safe under Windows

Changes by STINNER Victor : Removed file: http://bugs.python.org/file19898/read_write_32bits.patch ___ Python tracker ___ ___ Python-bugs-list

[issue9611] FileIO not 64-bit safe under Windows

STINNER Victor added the comment: (oops, the patch contained unrelated changes: remove trailing spaces) -- Added file: http://bugs.python.org/file19899/read_write_32bits.patch ___ Python tracker ___

[issue9611] FileIO not 64-bit safe under Windows

STINNER Victor added the comment: I agree that clamping is a nice solution, and attached patch implements it. About the question of the loop: FileIO.readall() uses while(1) without checking for signals. It looks like new_buffersize() maximum size is not BIGCHUNK but (BIGCHUNK-1)*2: /

[issue9611] FileIO not 64-bit safe under Windows

Martin v. Löwis added the comment: Am 04.11.2010 22:28, schrieb Amaury Forgeot d'Arc: > > Amaury Forgeot d'Arc added the comment: > >> Why do you think this would be somehow an example for a blocking stream >> that does not write all data without raising an exception? > Well, that's what "cla

[issue9611] FileIO not 64-bit safe under Windows

Antoine Pitrou added the comment: > On a second thought... is there another example where a *blocking* > stream does not write all the data without raising an exception? It can happen with pipes or sockets, if some buffer can only store part of the data. Or any regular stream if a signal is rec

[issue9611] FileIO not 64-bit safe under Windows

Amaury Forgeot d'Arc added the comment: > Why do you think this would be somehow an example for a blocking stream > that does not write all data without raising an exception? Well, that's what "clamping" means, isn't it? -- ___ Python tracker

[issue9611] FileIO not 64-bit safe under Windows

Martin v. Löwis added the comment: > On a second thought... is there another example where a *blocking* stream > does not write all the data without raising an exception? Why do you think this would be somehow an example for a blocking stream that does not write all data without raising an exc

[issue9611] FileIO not 64-bit safe under Windows

Amaury Forgeot d'Arc added the comment: On a second thought... is there another example where a *blocking* stream does not write all the data without raising an exception? -- ___ Python tracker ___

[issue9611] FileIO not 64-bit safe under Windows

Martin v. Löwis added the comment: > If I'm not mistaken, WriteFile takes the length as a DWORD, which is > still 32 bits under Win64. Oops, ignore me, then... I agree that clamping is fine, assuming the buffering layer takes that into account. -- _

[issue9611] FileIO not 64-bit safe under Windows

Antoine Pitrou added the comment: > I propose a different solution: On Windows, instead of calling > write(), we call WriteFile directly. If I'm not mistaken, WriteFile takes the length as a DWORD, which is still 32 bits under Win64. -- ___ Python t

[issue9611] FileIO not 64-bit safe under Windows

Martin v. Löwis added the comment: I propose a different solution: On Windows, instead of calling write(), we call WriteFile directly. We try to faithfully follow the CRT implementation as much as possible, knowing that what we write to actually is a binary file (in particular, the file handl

[issue9611] FileIO not 64-bit safe under Windows

Antoine Pitrou added the comment: > About the issue, I'd suggest to just clamp the length to 2**32-1, and > don't bother retrying; leave this to the buffered layer. Yes, I think it's reasonable. (or perhaps clamp to something page-aligned, such as 2**32-4096). Also, the signal issue which was r

[issue9611] FileIO not 64-bit safe under Windows

Changes by STINNER Victor : -- nosy: +haypo ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.o

[issue9611] FileIO not 64-bit safe under Windows

Amaury Forgeot d'Arc added the comment: Fortunately, the lower-level write() has no such bug, at least when used in binary mode as FileIO does: it's almost a direct call to WriteFile(). This issue is more considered because it's not a bug in the Microsoft CRT, but in the Python implementation

[issue9611] FileIO not 64-bit safe under Windows

Martin Spacek added the comment: We've got a near duplicate Issue9015. This thread seems more considered though :) Note that writing more than 2**32-1 bytes at once results in a hung process with 100% CPU in 64-bit Windows, which has to be killed with Task Manager. So I think that qualifies a

[issue9611] FileIO not 64-bit safe under Windows

Antoine Pitrou added the comment: Le dimanche 15 août 2010 à 18:53 +, Martin v. Löwis a écrit : > Martin v. Löwis added the comment: > > > Well, the loop stops when an error status is returned by the raw IO > > layer. At that point, the buffered IO layer re-raises the error after a > > bit

[issue9611] FileIO not 64-bit safe under Windows

Martin v. Löwis added the comment: > Well, the loop stops when an error status is returned by the raw IO > layer. At that point, the buffered IO layer re-raises the error after a > bit of internal cleanup. Assume the following case: 1. writing starts, and writes some data 2. Ctrl-C is pressed,

[issue9611] FileIO not 64-bit safe under Windows

Antoine Pitrou added the comment: > > Most people use buffered I/O, and the buffered layer automatically retries. > > I see. I think this is already slightly problematic: if you send an > interrupt, it won't oblige. IMO, any such loop ought to be > interruptable. Well, the loop stops when an e

[issue9611] FileIO not 64-bit safe under Windows

Martin v. Löwis added the comment: > Most people use buffered I/O, and the buffered layer automatically retries. I see. I think this is already slightly problematic: if you send an interrupt, it won't oblige. IMO, any such loop ought to be interruptable. -- ___

[issue9611] FileIO not 64-bit safe under Windows

Antoine Pitrou added the comment: The write() man page says: The number of bytes written may be less than count if, for example, there is insufficient space on the underlying physical medium, or the RLIMIT_FSIZE resource limit is encountered (see setrlimit(2)), or the

[issue9611] FileIO not 64-bit safe under Windows

Martin v. Löwis added the comment: Using chunked writes is tricky. If the first write succeeds, and the second one fails, how do you report the result? -- nosy: +loewis ___ Python tracker _

[issue9611] FileIO not 64-bit safe under Windows

Changes by Antoine Pitrou : -- nosy: +brian.curtin, tim.golden ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: h

[issue9611] FileIO not 64-bit safe under Windows

Antoine Pitrou added the comment: os.write() (in posixmodule.c) is also affected. os.read(), however, is limited to 32-bit inputs. -- ___ Python tracker ___

[issue9611] FileIO not 64-bit safe under Windows

New submission from Antoine Pitrou : Modules/_io/fileio.c assumes that read() and write() allow a Py_ssize_t length, but under Windows the length is an int, limiting chunk size to 2GB. It should be easy enough to read or write in multiple chunks, although testing might be difficult. -