[issue25626] Gzip fails for file over 2**32 bytes

Martin Panter added the comment: Thank you Serhiy for you help, and Ben for reporting this. -- resolution: -> fixed stage: commit review -> resolved status: open -> closed ___ Python tracker __

[issue25626] Gzip fails for file over 2**32 bytes

Roundup Robot added the comment: New changeset 80f6f77a7cc3 by Martin Panter in branch '3.5': Issue #25626: Change zlib to accept Py_ssize_t and cap to UINT_MAX https://hg.python.org/cpython/rev/80f6f77a7cc3 New changeset afa1b6cd77a5 by Martin Panter in branch 'default': Issue #25626: Merge zli

[issue25626] Gzip fails for file over 2**32 bytes

Changes by Serhiy Storchaka : -- assignee: -> martin.panter stage: patch review -> commit review ___ Python tracker ___ ___ Python-bu

[issue25626] Gzip fails for file over 2**32 bytes

Serhiy Storchaka added the comment: Thank you Martin, the patch LGTM. The difference between __int__() and __index__() is that float has the former, but not the latter. For better compatibility we have to use __int__() in maintained releases. But obviously __index__() is more appropriate sema

[issue25626] Gzip fails for file over 2**32 bytes

Martin Panter added the comment: New patch addressing comments. I used the undocumented API _PyLong_FromNbInt() to call __int__() rather than __index__(). -- Added file: http://bugs.python.org/file41107/zlib-Py_ssize_t.2.patch ___ Python tracker

[issue25626] Gzip fails for file over 2**32 bytes

Serhiy Storchaka added the comment: Added new comments. -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http

[issue25626] Gzip fails for file over 2**32 bytes

Martin Panter added the comment: Thanks for your testing Serhiy. I can reproduce this on 32-bit Linux (also by building with CC="gcc -m32"). It is easy to produce the bug with flush(), but with Decompress.decompress() it would need over 2 GiB of memory and data to expand the buffer enough to t

[issue25626] Gzip fails for file over 2**32 bytes

Serhiy Storchaka added the comment: Ah, I found yet one bug. >>> zlib.decompress(zlib.compress(b'abcd'), 0, sys.maxsize+1) Traceback (most recent call last): File "", line 1, in SystemError: Negative size passed to PyBytes_FromStringAndSize There are similar bugs in decompressor's methods de

[issue25626] Gzip fails for file over 2**32 bytes

Martin Panter added the comment: Actually it did make an existing bug a bit worse. The bug is triggered with valid sizes greater than LONG_MAX, due to an exception not being cleared, and my change made this bug more dramatic, turning the OverflowError into a crash. This new patch should fix th

[issue25626] Gzip fails for file over 2**32 bytes

Serhiy Storchaka added the comment: You can commit your patch right now (it shouldn't make things worse) and add new tests later. -- ___ Python tracker ___ _

[issue25626] Gzip fails for file over 2**32 bytes

Martin Panter added the comment: Okay. For the gzip module, I cannot easily test this myself. Quickly looking at other cases, I guess it would look something like this, but I need to spend some time understanging the bigmemtest decorator properly: @unittest.skipIf(sys.maxsize < _4G, "Requires

[issue25626] Gzip fails for file over 2**32 bytes

Serhiy Storchaka added the comment: Could you please add tests for other two functions? And tests for the gzip module? -- ___ Python tracker ___

[issue25626] Gzip fails for file over 2**32 bytes

Martin Panter added the comment: I am inclined to commit my patch to get this fixed in the upcoming 3.5.1 release, unless anyone thinks that could be a problem. -- components: +Extension Modules ___ Python tracker

[issue25626] Gzip fails for file over 2**32 bytes

Martin Panter added the comment: 3.4 shouldn’t be affected by the gzip regression. In 3.4 the gzip module does not set max_length to limit the decompression size. In 3.4 the underlying zlib module will also raise OverflowError if max_lenth=2**32, and my patch could be applied, but I don’t thin

[issue25626] Gzip fails for file over 2**32 bytes

Serhiy Storchaka added the comment: Is 3.4 affected? A uint_converter was added in issue18294. -- ___ Python tracker ___ ___ Python-bu

[issue25626] Gzip fails for file over 2**32 bytes

Changes by Serhiy Storchaka : -- nosy: +serhiy.storchaka ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https:

[issue25626] Gzip fails for file over 2**32 bytes

Martin Panter added the comment: Ben: By adding the cap, it means when the lower-level zlib module is asked to decode say 5 GiB, it will decode at most 4 GiB (rather than raising an exception). The 4 GiB limit (UINT_MAX to be precise) is due to the underlying zlib library; it’s not Python’s do

[issue25626] Gzip fails for file over 2**32 bytes

Ben Cipollini added the comment: @Martin: yes, that reproduces the problem. >I think the ideal fix would be to cap the limit at 2**32 - 1 in the zlib >library. If this cap is implemented, is there any workaround how we can efficiently open gzipped files over 4GB? Such files exist, exist in hi

[issue25626] Gzip fails for file over 2**32 bytes

Martin Panter added the comment: Thanks for the report. Can you confirm if this demo illustrates your problem? For me, I only have 2 GiB of memory so I get a MemoryError, which seems reasonable for my situation. from gzip import GzipFile from io import BytesIO file = BytesIO() writer = GzipFil

[issue25626] Gzip fails for file over 2**32 bytes

Changes by SilentGhost : -- components: +Library (Lib) nosy: +nadeem.vawda, twouters versions: +Python 3.6 ___ Python tracker ___ ___

[issue25626] Gzip fails for file over 2**32 bytes

Changes by Matthew Brett : -- nosy: +Matthew.Brett ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail

[issue25626] Gzip fails for file over 2**32 bytes

New submission from Ben Cipollini: Gzip fails when opening a file more than 2**32 bytes. This is a new issue in Python 3.5. We hit this opening large neuroimaging files from the Human Connectome Project. See https://github.com/nipy/nibabel/issues/362 for more details. When size is > 2**32, we