[issue14684] zlib set dictionary support inflateSetDictionary
New submission from Sam Rushing :
Google's SPDY protocol requires the use of a pre-defined compression
dictionary. The current zlib module doesn't expose the two functions for
setting the dictionary.
This patch is minimal in the sense that it only exposes the two functions, but
unfortunately the sequence of zlib calls required is clumsy: a call to
inflate() must fail first (with an error of Z_NEED_DICT):
import zlib
zdict = b"thequickbrownfoxjumped\x00"
c = zlib.compressobj()
c.set_dictionary (zdict)
cd = c.compress (b"the quick brown fox jumped over the candlestick")
cd += c.flush()
d = zlib.decompressobj()
try:
print (d.decompress (cd))
except zlib.error as what:
if what.args[0].startswith ('Error 2 '):
d.set_dictionary (zdict)
print (d.flush())
Obviously a better way to catch/match Z_NEED_DICT would be nice.
A much nicer solution would allow you to associate the dictionary at creation
time, rather than having to wait for an error condition. I can only guess that
the zlib authors designed it that way for a reason?
--
components: Extension Modules
files: zlib_set_dictionary.patch
keywords: patch
messages: 159492
nosy: Sam.Rushing
priority: normal
severity: normal
status: open
title: zlib set dictionary support inflateSetDictionary
type: enhancement
versions: Python 3.4
Added file: http://bugs.python.org/file25387/zlib_set_dictionary.patch
___
Python tracker
<http://bugs.python.org/issue14684>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14684] zlib set dictionary support inflateSetDictionary
Sam Rushing added the comment: I'm currently reworking this so that the dictionaries are provided in the constructor, and inflateSetDictionary() is called automatically. I've gone over the zlib RFC's and zlibmodule.c, and I'm fairly certain that whatever usage mode might involve multiple calls to SetDictionary() couldn't be supported by the zlib object anyway. r.e. 3.3/3.4 - I merely chose the highest version number I saw, since this diff is against HEAD (as recommended by the docs). It's been quite a few years since I've submitted a patch to CPython! -- ___ Python tracker <http://bugs.python.org/issue14684> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14684] zlib set dictionary support inflateSetDictionary
Sam Rushing added the comment: Ok, here's the patch. It has a single short test. For use with SPDY, it's necessary to test that the following stream data also correctly decompresses, I'll attach that to the next comment. -- Added file: http://bugs.python.org/file25446/zlib_set_dictionary_2.patch ___ Python tracker <http://bugs.python.org/issue14684> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14684] zlib set dictionary support inflateSetDictionary
Sam Rushing added the comment: This test is rather large, since it includes the predefined SPDY draft 2 dictionary, and some real-world data. Not sure what the policy is on including so much data in a test. If there's enough time I could make a smaller test that also verifies the correct behavior on a stream... -- Added file: http://bugs.python.org/file25447/tz2.py ___ Python tracker <http://bugs.python.org/issue14684> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14684] zlib set dictionary support inflateSetDictionary
Sam Rushing added the comment: Argh, probably need to add the 'dict' field to the copy() method. -- ___ Python tracker <http://bugs.python.org/issue14684> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14684] zlib set dictionary support inflateSetDictionary
Sam Rushing added the comment: Updated version of the patch: extends the test, including a test of the streaming behavior needed for SPDY (both compression and decompression). Also wik: copy()/uncopy() are aware of the 'dict' attribute. -- ___ Python tracker <http://bugs.python.org/issue14684> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14684] zlib set dictionary support inflateSetDictionary
Changes by Sam Rushing : Added file: http://bugs.python.org/file25449/zlib_set_dictionary_3.patch ___ Python tracker <http://bugs.python.org/issue14684> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14684] zlib set dictionary support inflateSetDictionary
Sam Rushing added the comment: renames dict->zdict, splits the test, adds BEGIN/END around inflate call. -- Added file: http://bugs.python.org/file25472/zlib_set_dictionary_4.patch ___ Python tracker <http://bugs.python.org/issue14684> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14684] zlib set dictionary support inflateSetDictionary
Sam Rushing added the comment: I think other than the disagreement about whether the dictionary constructor arg should be a buffer object, it's good to go. To restate my position: the need is for an immutable string of bytes, and that's exactly what PyBytes_Type is for. I see no advantage to allowing a buffer object, which will require extra code to check that it is both a buffer object and set to be readonly. I believe the argument for aesthetics does not apply, as the constant dictionary constructor argument is a morally different kind of parameter, comparable to (say) the compression level. You folks are of course welcome to change it, though. 8^) -- ___ Python tracker <http://bugs.python.org/issue14684> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
