Re: [Python-Dev] Fast Implementation for ZIP decryption

[CJ Kucera]

Hello list, resurrecting a rather old thread from here:

http://mail.python.org/pipermail/python-dev/2009-August/091450.html

I've actually come across a use case where faster zipfile decryption
would be very helpful to me (though one could certainly argue that it's
a frivolous case).  I'm writing a map editor for a game whose graphics
datafiles are stored inside an encrypted zipfile.  I've received the
game developer's blessing for my editor to be able to read inside the
zipfile, so legality isn't an issue here, but the pure-Python decryption
currently available in the 'zipfile' module really is dog-slow.

When I have my app load directly from already-uncompressed files on the
HD, it's rare for a map to take more than 1 second to load; when the app
has to use the pure-Python decryption routines to load all the graphics
images, it can take over a minute (and this is on a 3GHz Core 2 Duo).

Bundling the necessary graphics with my app isn't an option, nor is
asking users to crack the zipfile encryption themselves to provide an
uncompressed directory to read from.

So anyway, I'd at least love to have a peek at the C code mentioned in
the thread; if nothing more, perhaps I could bundle it into something
that could be installed optionally by someone who wants it.  Does anyone
still have this code available somewhere?

Thanks a bunch,
CJ

-- 
WOW: Flemmy|   "Happiness isn't good enough for me!  I
p...@apocalyptech.com   |  demand euphoria!"
24.24.2.3171   |  - Calvin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Fast Implementation for ZIP decryption

[CJ Kucera]

CJ Kucera wrote:
> Hello list, resurrecting a rather old thread from here:
> 
> http://mail.python.org/pipermail/python-dev/2009-August/091450.html

... and one final update from me, mostly just so Google and the like
will pick it up.  I did actually end up packaging up something I called
"czipfile," which is just the stock Python Lib/zipfile.py optimized in
Cython, which provides very fast zipfile decryption.  It's available
here:

http://pypi.python.org/pypi/czipfile

So, anyone with a need for fast zipfile decryption can now head over
that way.

FWIW, while playing around with things, I found that you can get some
pretty nontrivial performance improvements in pure Python by unrolling
the decryption loop.  For instance, line 608 in Lib/zipfile.py used to
be:

newdata = ''.join(map(self.decrypter, newdata))

Substituting the following results in some significant speed
improvements (though still quite a lot slower than the C-based
extension) -

key0 = self.decrypter.key0
key1 = self.decrypter.key1
key2 = self.decrypter.key2
crctable = self.decrypter.crctable
datalist = []
for c in newdata:
k = key2 | 2
cord = ord(c) ^ (((k * (k^1)) >> 8) & 255)
datalist.append(chr(cord))
key0 = ((key0 >> 8) & 0xff) ^ crctable[(key0 ^ cord) & 0xff]
key1 = (key1 + (key0 & 255)) & 4294967295
key1 = (key1 * 134775813 + 1) & 4294967295
key2 = ((key2 >> 8) & 0xff) ^ crctable[(key2 ^ ((key1 >> 24) & 
255)) & 0xff]
self.decrypter.key0 = key0
self.decrypter.key1 = key1
self.decrypter.key2 = key2
newdata = ''.join(datalist)

Anyone looking to speed up decryption who didn't want to absolutely
depend on the user having czipfile installed might want to consider
bundling their own modified version of Lib/zipfile.py with the above
changes.

Okay, that's all.  Enjoy!

-CJ

-- 
WOW: Flemmy|   "The ships hung in the sky in much the same
p...@apocalyptech.com   |way that bricks don't." - Douglas Adams,
24.24.2.3171   | _The Hitchhiker's Guide To The Galaxy_
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com