[Python-Dev] Lack of sequential decompression in the zipfile module
Though I am an avid Python programmer, I've never forayed into the area of developing Python itself, so I'm not exactly sure how all this works. I was confused (and somewhat disturbed) to discover recently that the zipfile module offers only one-shot decompression of files, accessible only via the read() method. It is my understanding that the module will handle files of up to 4 GB in size, and the idea of decompressing 4 GB directly into memory makes me a little queasy. Other related modules (zlib, tarfile, gzip, bzip2) all offer sequential decompression, but this does not seem to be the case for zipfile (even though the underlying zlib makes it easy to do). Since I was writing a script to work with potentially very large zipped files, I took it upon myself to write an extract() method for zipfile, which is essentially an adaption of the read() method modeled after tarfile's extract(). I feel that this is something that should really be provided in the zipfile module to make it more usable. I'm wondering if this has been discussed before, or if anyone has ever viewed this as a problem. I can post the code I wrote as a patch, though I'm not sure if my file IO handling is as robust as it needs to be for the stdlib. I'd appreciate any insight into the issue or direction on where I might proceed from here so as to fix what I see as a significant problem. Thanks, Derek ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Bug in smtpd.SMTPChannel.smtp_MAIL
smtpd.SMTPChannel contains a bug such that when connected to an SMTPServer (or any subclass thereof), issuing a MAIL command with no argument closes the socket and gives this error on the server: error: uncaptured python exception, closing channel (:'NoneType' object is unsubscriptable [/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/asyncore.py|read|68] [/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/asyncore.py|handle_read_event|390] [/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/asynchat.py|handle_read|137] [/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/smtpd.py|found_terminator|158] [/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/smtpd.py|smtp_MAIL|224] [/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/smtpd.py|__getaddr|212]) The desired result is of course is to respond with a 501 Syntax error. The problem arises because the arg parameter passed to each smtp_* command handler function is None when there is no argument. arg is passed on to __getaddr which attempts a slice on the None object. I think the most elegant solution would be to insert a "if not arg" check in __getaddr and return. The existing smtp_MAIL code will then issue the 501 Syntax error. -Derek Shockey ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bug in smtpd.SMTPChannel.smtp_MAIL
I did actually submit a patch last night after I emailed the list. The issue ID is 1307. I used the ternary operator (because I like it and because it was the shortest solution), but if this is not desirable for backwards compatibility, I could obviously rewrite it another way. -Derek Shockey On 10/21/07, Guido van Rossum <[EMAIL PROTECTED]> wrote: > > Could you submit a patch to the bug tracker (bugs.python.org)? > > 2007/10/20, Derek Shockey <[EMAIL PROTECTED]>: > > smtpd.SMTPChannel contains a bug such that when connected to an > SMTPServer > > (or any subclass thereof), issuing a MAIL command with no argument > closes > > the socket and gives this error on the server: > > > > error: uncaptured python exception, closing channel < smtpd.SMTPChannel > > connected 127.0.0.1:58587 at 0x847d8> ( > 'exceptions.TypeError'>:'NoneType' object is unsubscriptable > > > [/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/asyncore.py|read|68] > > > [/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/asyncore.py|handle_read_event|390] > > > [/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/asynchat.py|handle_read|137] > > > [/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/smtpd.py|found_terminator|158] > > > [/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/smtpd.py|smtp_MAIL|224] > > > [/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/smtpd.py|__getaddr|212]) > > > > The desired result is of course is to respond with a 501 Syntax error. > The > > problem arises because the arg parameter passed to each smtp_* command > > handler function is None when there is no argument. arg is passed on to > > __getaddr which attempts a slice on the None object. > > > > I think the most elegant solution would be to insert a "if not arg" > check in > > __getaddr and return. The existing smtp_MAIL code will then issue the > 501 > > Syntax error. > > > > > > -Derek Shockey > > > > ___ > > Python-Dev mailing list > > Python-Dev@python.org > > http://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > > http://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] ints not overflowing into longs?
I just found an unexpected behavior and I'm wondering if it is a bug. In my 2.7.2 interpreter on OS X, built and installed via MacPorts, it appears that integers are not correctly overflowing into longs and instead are yielding bizarre results. I can only reproduce this when using the exponent operator with two ints (declaring either operand explicitly as long prevents the behavior). >>> 2**100 0 >>> 2**100L 1267650600228229401496703205376L >>> 20**20 -2101438300051996672 >>> 20L**20 1048576L >>> 10**20 7766279631452241920 >>> 10L**20L 1L To confirm I'm not crazy, I tried in the 2.7.1 and 2.6.7 installations included in OS X 10.7, and also a 2.7.2+ (not sure what the + is) on an Ubuntu machine and didn't see this behavior. This looks like some kind of truncation error, but I don't know much about the internals of Python and have no idea what's going on. I assume since it's only in my MacPorts installation, it must be build configuration issue that is specific to OS X, perhaps only 10.7, or MacPorts. Am I doing something wrong, and is there a way to fix it before I compile? I could find any references to this problem as a known issue. Thanks, Derek ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ints not overflowing into longs?
Thank you, I narrowed it down from there and got a properly working build. I gather the problem is that in Xcode 4.2 the default compiler was changed to clang, but the version of clang bundled with it has a bug that breaks overflows in intobject.c. In case anyone else hits this, I fixed this in MacPorts by forcing it to use gcc. Edit the portfile (port edit python27) and add this anywhere after the 5th or so line: configure.compiler llvm-gcc-4 -Derek On Wed, Nov 2, 2011 at 7:41 PM, Guido van Rossum wrote: > Apparently Macports is still using a buggy compiler. I reported a > similar issue before and got this reply from Ned Delly: > > """ > Thanks for the pointer. That looks like a duplicate of Issue11149 (and > Issue12701). Another manifestation of this was reported in Issue13061 > which also originated from MacPorts. I'll remind them that the > configure change is likely needed for all Pythons. It's still safest to > stick with good old gcc-4.2 on OS X at the moment. > """ > > (Those issues are on bugs.python.org.) > > --Guido > > On Wed, Nov 2, 2011 at 7:32 PM, Derek Shockey wrote: >> I just found an unexpected behavior and I'm wondering if it is a bug. >> In my 2.7.2 interpreter on OS X, built and installed via MacPorts, it >> appears that integers are not correctly overflowing into longs and >> instead are yielding bizarre results. I can only reproduce this when >> using the exponent operator with two ints (declaring either operand >> explicitly as long prevents the behavior). >> >>>>> 2**100 >> 0 >>>>> 2**100L >> 1267650600228229401496703205376L >> >>>>> 20**20 >> -2101438300051996672 >>>>> 20L**20 >> 1048576L >> >>>>> 10**20 >> 7766279631452241920 >>>>> 10L**20L >> 1L >> >> To confirm I'm not crazy, I tried in the 2.7.1 and 2.6.7 installations >> included in OS X 10.7, and also a 2.7.2+ (not sure what the + is) on >> an Ubuntu machine and didn't see this behavior. This looks like some >> kind of truncation error, but I don't know much about the internals of >> Python and have no idea what's going on. I assume since it's only in >> my MacPorts installation, it must be build configuration issue that is >> specific to OS X, perhaps only 10.7, or MacPorts. >> >> Am I doing something wrong, and is there a way to fix it before I >> compile? I could find any references to this problem as a known issue. >> >> Thanks, >> Derek >> ___ >> Python-Dev mailing list >> Python-Dev@python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/guido%40python.org >> > > > > -- > --Guido van Rossum (python.org/~guido) > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ints not overflowing into longs?
I believe you're right. The 2.7.2 MacPorts portfile definitely passes the -fwrapv flag to clang, but the bad behavior still occurs with exponents. I verified the current head of the 2.7 branch does not have this problem when built with clang, so I'm assuming that issue12973 resolved this with a patch to int_pow() and that it will be out in the next release. -Derek On Thu, Nov 3, 2011 at 4:30 AM, Antoine Pitrou wrote: > On Wed, 2 Nov 2011 19:41:30 -0700 > Guido van Rossum wrote: >> Apparently Macports is still using a buggy compiler. > > If I understand things correctly, this is technically not a buggy > compiler but Python making optimistic assumptions about the C standard. > (from issue11149: "clang (as with gcc 4.x) assumes signed integer > overflow is undefined. But Python depends on the fact that signed > integer overflow wraps") > > I'd happily call that a buggy C standard, though :-) > > Regards > > Antoine. > > >> I reported a >> similar issue before and got this reply from Ned Delly: >> >> """ >> Thanks for the pointer. That looks like a duplicate of Issue11149 (and >> Issue12701). Another manifestation of this was reported in Issue13061 >> which also originated from MacPorts. I'll remind them that the >> configure change is likely needed for all Pythons. It's still safest to >> stick with good old gcc-4.2 on OS X at the moment. >> """ >> >> (Those issues are on bugs.python.org.) >> >> --Guido >> >> On Wed, Nov 2, 2011 at 7:32 PM, Derek Shockey >> wrote: >> > I just found an unexpected behavior and I'm wondering if it is a bug. >> > In my 2.7.2 interpreter on OS X, built and installed via MacPorts, it >> > appears that integers are not correctly overflowing into longs and >> > instead are yielding bizarre results. I can only reproduce this when >> > using the exponent operator with two ints (declaring either operand >> > explicitly as long prevents the behavior). >> > >> >>>> 2**100 >> > 0 >> >>>> 2**100L >> > 1267650600228229401496703205376L >> > >> >>>> 20**20 >> > -2101438300051996672 >> >>>> 20L**20 >> > 1048576L >> > >> >>>> 10**20 >> > 7766279631452241920 >> >>>> 10L**20L >> > 1L >> > >> > To confirm I'm not crazy, I tried in the 2.7.1 and 2.6.7 installations >> > included in OS X 10.7, and also a 2.7.2+ (not sure what the + is) on >> > an Ubuntu machine and didn't see this behavior. This looks like some >> > kind of truncation error, but I don't know much about the internals of >> > Python and have no idea what's going on. I assume since it's only in >> > my MacPorts installation, it must be build configuration issue that is >> > specific to OS X, perhaps only 10.7, or MacPorts. >> > >> > Am I doing something wrong, and is there a way to fix it before I >> > compile? I could find any references to this problem as a known issue. >> > >> > Thanks, >> > Derek >> > ___ >> > Python-Dev mailing list >> > Python-Dev@python.org >> > http://mail.python.org/mailman/listinfo/python-dev >> > Unsubscribe: >> > http://mail.python.org/mailman/options/python-dev/guido%40python.org >> > >> >> >> >> -- >> --Guido van Rossum (python.org/~guido) > > > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/derek.shockey%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ints not overflowing into longs?
You're right; among my many tests I think I muddled the situation with a stray CFLAGS variable in my environment. Apologies for the misinformation. The current MacPorts portfile does not add -fwrapv. Adding -fwrapv to OPT in the Makefile solves the problem. I confirmed by manually building the v2.7.2 tag with clang and -fwrapv, and the overflow behavior is correct. I've notified the MacPorts package maintainer. -Derek On Thu, Nov 3, 2011 at 11:07 AM, Stefan Krah wrote: > Derek Shockey wrote: >> I believe you're right. The 2.7.2 MacPorts portfile definitely passes >> the -fwrapv flag to clang, but the bad behavior still occurs with >> exponents. > > Really? Even without the fix for issue12973 the -fwrapv flag > should be sufficient, as reported in issue13061 and Issue11149. > > For clang version 3.0 (trunk 139691) on FreeBSD this is the case. > > > Stefan Krah > > > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/derek.shockey%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
