[issue5689] Support xz compression in tarfile module

Changes by Nadeem Vawda : -- status: open -> closed ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.

[issue5689] Support xz compression in tarfile module

Roundup Robot added the comment: New changeset b86b54fcb5c2 by Lars Gustäbel in branch 'default': Issue #5689: Avoid excessive memory usage by using the default lzma preset. http://hg.python.org/cpython/rev/b86b54fcb5c2 -- ___ Python tracker

[issue5689] Support xz compression in tarfile module

Amaury Forgeot d'Arc added the comment: Ping. Windows buildbots are still failing with MemoryError because of this preset=9. The patch looks good to me as well. -- nosy: +amaury.forgeotdarc ___ Python tracker ___

[issue5689] Support xz compression in tarfile module

Changes by Nikolaus Rath : -- nosy: -Nikratio ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.pytho

[issue5689] Support xz compression in tarfile module

Nadeem Vawda added the comment: Patch looks good to me. -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http

[issue5689] Support xz compression in tarfile module

Lars Gustäbel added the comment: Yes, that's much better. Thanks for the tip. -- Added file: http://bugs.python.org/file24086/lzma-preset.diff ___ Python tracker ___

[issue5689] Support xz compression in tarfile module

Changes by Lars Gustäbel : Removed file: http://bugs.python.org/file24084/lzma-preset.diff ___ Python tracker ___ ___ Python-bugs-list mailing

[issue5689] Support xz compression in tarfile module

Nadeem Vawda added the comment: Yes, that's a good idea. I've been testing a similar change, and it seems to drop the peak memory usage for test_tarfile from around 810MB down to under 200MB. It looks like 2GB genuinely isn't enough to reliably use LZMA compression with preset=9. You might want

[issue5689] Support xz compression in tarfile module

Lars Gustäbel added the comment: Wouldn't it be better then to use a default compresslevel of 6 in tarfile? I used level 9 in my patch without a particular reason, just because I thought 9 must be better than 6 ;-) -- Added file: http://bugs.python.org/file24084/lzma-preset.diff

[issue5689] Support xz compression in tarfile module

Nadeem Vawda added the comment: Not to worry - as I said in my previous message, I can reproduce the error on my own XP machine. I also noticed that running test_tarfile alone doesn't trigger the errors, which leads me to suspect that the failure is due to some interaction with another test get

[issue5689] Support xz compression in tarfile module

Paul Moore added the comment: A simple rebuild and test run of that test in debug mode didn't fail... I'll run the full test suite as a check, but that could take some time - that buildslave isn't the fastest in the world... -- ___ Python tracker

[issue5689] Support xz compression in tarfile module

Antoine Pitrou added the comment: Perhaps Paul can try to reproduce and diagnose the issue directly on the buildbot? -- nosy: +pmoore ___ Python tracker ___

[issue5689] Support xz compression in tarfile module

Nadeem Vawda added the comment: This failure seems to crop up often, but not on every run: http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%203.x/builds/3941/steps/test/logs/stdio http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%203.x/builds/3940/steps/test/logs/stdio

[issue5689] Support xz compression in tarfile module

STINNER Victor added the comment: There is failure on a XP buildbot. I don't know if it is a sporadic issue or not. http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%203.x/builds/3921/steps/test/logs/stdio == ERROR:

[issue5689] Support xz compression in tarfile module

Lars Gustäbel added the comment: Please, go ahead! -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://ma

[issue5689] Support xz compression in tarfile module

Éric Araujo added the comment: Lars, as part of a small doc patch I want to change this in tarfile.rst: The :mod:`tarfile` module makes it possible to read and write tar archives, including those using gzip or bz2 compression. -(:file:`.zip` files can be read and written using the :mod:

[issue5689] Support xz compression in tarfile module

Nadeem Vawda added the comment: Great stuff! I'll close this issue along with issue 6715 once the buildbot stuff is all sorted out. -- ___ Python tracker ___

[issue5689] Support xz compression in tarfile module

Lars Gustäbel added the comment: Thanks for the review, guys! I can't close this issue yet because it depends on #6715. -- resolution: -> fixed stage: needs patch -> committed/rejected ___ Python tracker

[issue5689] Support xz compression in tarfile module

Roundup Robot added the comment: New changeset 899a8c7b2310 by Lars Gustäbel in branch 'default': Issue #5689: Add support for lzma compression to the tarfile module. http://hg.python.org/cpython/rev/899a8c7b2310 -- nosy: +python-dev ___ Python track

[issue5689] Support xz compression in tarfile module

Éric Araujo added the comment: Patch looks great. I did a review on Rietveld. -- ___ Python tracker ___ ___ Python-bugs-list mailing

[issue5689] Support xz compression in tarfile module

STINNER Victor added the comment: Some comments about 2011-12-08-tarfile-lzma.diff: > elif self.buf.startswith(b"\x5d\x00\x00\x80") or self.buf.startswith(b"... Micro-optimization: you can use self.buf.startswith((b"\x5d\x00\x00\x80", b"\xfd7zXZ")) here. > raise ValueError("mode must be 'r'

[issue5689] Support xz compression in tarfile module

Lars Gustäbel added the comment: For those who want to test it first, I post the current state of the patch here. It is ready for commit, there are no failing tests. If nobody objects, I will apply it this weekend. -- Added file: http://bugs.python.org/file23880/2011-12-08-tarfile-lzm

[issue5689] Support xz compression in tarfile module

Éric Araujo added the comment: I’m perfectly happy to let Lars do it next week or next month, there is no rush. The existing patch may even require very little or no change, as Nadeem’s module (in the stdlib) provides the same classes than the other lzma module which was used by the patch.

[issue5689] Support xz compression in tarfile module

Martin v. Löwis added the comment: There is plenty of time until 3.3. OTOH, if Eric wants to work on it now: you got a week :-) Do recognize that there is a patch to start from already. -- ___ Python tracker _

[issue5689] Support xz compression in tarfile module

Lars Gustäbel added the comment: I will be happy to, but my spare time is limited right now, so this could take about a week. If this is a problem, please go ahead. -- ___ Python tracker __

[issue5689] Support xz compression in tarfile module

Éric Araujo added the comment: Python now has an lzma module. Lars, do you have the time to update your patch or should I do it? -- ___ Python tracker ___ _

[issue5689] Support xz compression in tarfile module

Lars Gustäbel added the comment: Attached is a patch with the current state of my work on lzma integration into tarfile (17 test errors). -- assignee: -> lars.gustaebel keywords: +patch Added file: http://bugs.python.org/file23162/2011-09-15-tarfile-lzma.diff

[issue5689] Support xz compression in tarfile module

Changes by Éric Araujo : -- components: +Library (Lib) -Extension Modules nosy: +nadeem.vawda title: please support lzma compression as an extension and in the tarfile module -> Support xz compression in tarfile module versions: +Python 3.3 -Python 3.2 ___