[issue43547] support ZIP files with zeroed out fields (e.g. for reproducible builds)
Change by Felix C. Stegerman : -- nosy: +obfusk ___ Python tracker <https://bugs.python.org/issue43547> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43547] support ZIP files with zeroed out fields (e.g. for reproducible builds)
Change by Felix C. Stegerman : -- keywords: +patch pull_requests: +23737 stage: -> patch review pull_request: https://github.com/python/cpython/pull/24979 ___ Python tracker <https://bugs.python.org/issue43547> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43547] support ZIP files with zeroed out fields (e.g. for reproducible builds)
Felix C. Stegerman added the comment: I've created a draft PR; RFC :) Also: * setting the date to (1980,0,0,0,0,0) already works; * the main issue seems to be that external_attr cannot be 0 atm. -- ___ Python tracker <https://bugs.python.org/issue43547> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43547] support ZIP files with zeroed out fields (e.g. for reproducible builds)
Felix C. Stegerman added the comment: I've closed the PR for now. Using a carefully crafted ZipInfo object doesn't work because ZipFile modifies its .external_attr when set to 0. Using something like this quickly hacked together ZipInfo subclass does work: class ZeroedZipInfo(zipfile.ZipInfo): def __init__(self, zinfo): for k in self.__slots__: setattr(self, k, getattr(zinfo, k)) def __getattribute__(self, name): if name == "date_time": return (1980,0,0,0,0,0) if name == "external_attr": return 0 return object.__getattribute__(self, name) ... myzipfile.writestr(ZeroedZipInfo(info), data) -- components: +IO type: enhancement -> versions: +Python 3.6, Python 3.7, Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue43547> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43547] support ZIP files with zeroed out fields (e.g. for reproducible builds)
Change by Felix C. Stegerman : -- components: -IO, Library (Lib) versions: -Python 3.6, Python 3.7, Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue43547> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43547] support ZIP files with zeroed out fields (e.g. for reproducible builds)
Change by Felix C. Stegerman : -- components: +Library (Lib) ___ Python tracker <https://bugs.python.org/issue43547> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43547] support ZIP files with zeroed out fields (e.g. for reproducible builds)
Change by Felix C. Stegerman : -- type: -> enhancement ___ Python tracker <https://bugs.python.org/issue43547> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43547] support ZIP files with zeroed out fields (e.g. for reproducible builds)
Felix C. Stegerman added the comment: > external_attr == 0 may cause issues with permissions. That may be true in some scenarios, but not being able to set it to 0 means you can't create identical files to those produced by other tools -- like those used to generate APKs -- which do in fact set it to 0. -- ___ Python tracker <https://bugs.python.org/issue43547> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43547] support ZIP files with zeroed out fields (e.g. for reproducible builds)
Felix C. Stegerman added the comment: > The __getattr__ hack is not needed. You can reset the flags in a different, > more straight forward way As mentioned, ZipFile._open_to_write() will modify the ZipInfo's .external_attr when it is set to 0. > I just found another specific example in _open_to_write(). 0 is a valid > value for zinfo.external_attr. But this code always forces 0 to something > else: > > if not zinfo.external_attr: > zinfo.external_attr = 0o600 << 16 # permissions: ?rw--- Your alternative doesn't seem to take that subsequent modification into account. -- ___ Python tracker <https://bugs.python.org/issue43547> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2824] zipfile to handle duplicate files in archive
Change by Felix C. Stegerman : -- nosy: +obfusk ___ Python tracker <https://bugs.python.org/issue2824> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29708] support reproducible Python builds
Change by Felix C. Stegerman : -- nosy: +obfusk ___ Python tracker <https://bugs.python.org/issue29708> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29708] support reproducible Python builds
Felix C. Stegerman added the comment: Hi! I've been working on reproducible builds for python-for-android [1,2,3]. Current issues with .pyc files are: * .pyc files differ depending on whether Python was compiled w/ liblzma-dev installed or not; * many .pyc files include build paths; * some .pyc files include paths to system utilities, like `/bin/mkdir` or `/usr/bin/install`, which can differ between systems (e.g. on Debian w/ merged /usr). [1] https://github.com/kivy/python-for-android/pull/2390 [2] https://lists.reproducible-builds.org/pipermail/rb-general/2021-January/002132.html [3] https://lists.reproducible-builds.org/pipermail/rb-general/2021-March/002207.html -- ___ Python tracker <https://bugs.python.org/issue29708> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43547] support ZIP files with zeroed out fields (e.g. for reproducible builds)
Felix C. Stegerman added the comment:
https://github.com/obfusk/apksigcopier currently produces reproducible ZIP
files identical to those produced by apksigner using this code:
DATETIMEZERO = (1980, 0, 0, 0, 0, 0)
class ReproducibleZipInfo(zipfile.ZipInfo):
"""Reproducible ZipInfo hack."""
_override = {} # type: Dict[str, Any]
def __init__(self, zinfo, **override):
if override:
self._override = {**self._override, **override}
for k in self.__slots__:
if hasattr(zinfo, k):
setattr(self, k, getattr(zinfo, k))
def __getattribute__(self, name):
if name != "_override":
try:
return self._override[name]
except KeyError:
pass
return object.__getattribute__(self, name)
class APKZipInfo(ReproducibleZipInfo):
"""Reproducible ZipInfo for APK files."""
_override = dict(
compress_type=8,
create_system=0,
create_version=20,
date_time=DATETIMEZERO,
external_attr=0,
extract_version=20,
flag_bits=0x800,
)
def patch_meta(...):
...
with zipfile.ZipFile(output_apk, "a") as zf_out:
info_data = [(APKZipInfo(info, date_time=date_time), data)
for info, data in extracted_meta]
_write_to_zip(info_data, zf_out)
if sys.version_info >= (3, 7):
def _write_to_zip(info_data, zf_out):
for info, data in info_data:
zf_out.writestr(info, data, compresslevel=9)
else:
def _write_to_zip(info_data, zf_out):
old = zipfile._get_compressor
zipfile._get_compressor = lambda _: zlib.compressobj(9, 8, -15)
try:
for info, data in info_data:
zf_out.writestr(info, data)
finally:
zipfile._get_compressor = old
--
___
Python tracker
<https://bugs.python.org/issue43547>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37596] Reproducible pyc: frozenset is not serialized in a deterministic order
Change by Felix C. Stegerman : -- nosy: +obfusk ___ Python tracker <https://bugs.python.org/issue37596> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42151] Pure Python xml.etree.ElementTree is missing default attribute values
New submission from Felix C. Stegerman : I originally reported this as a bug in PyPy, but it turns out that CPython's C implementation (_elementtree) behaves differently than the pure Python version (b/c it sets specified_attributes = 1). PyPy issue with example code: https://foss.heptapod.net/pypy/pypy/-/issues/ -- components: Library (Lib) messages: 379637 nosy: obfusk priority: normal severity: normal status: open title: Pure Python xml.etree.ElementTree is missing default attribute values type: behavior versions: Python 3.10, Python 3.6, Python 3.7, Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue42151> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42151] Pure Python xml.etree.ElementTree is missing default attribute values
Change by Felix C. Stegerman : -- keywords: +patch pull_requests: +21901 stage: -> patch review pull_request: https://github.com/python/cpython/pull/22987 ___ Python tracker <https://bugs.python.org/issue42151> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42151] Pure Python xml.etree.ElementTree is missing default attribute values
Felix C. Stegerman added the comment: > specified_attributes = True is also set in xml.dom.expatbuilder. That's good to know and should perhaps be addressed as well. > Should not it be set to true in the C implementation of ElementTree? That would break existing code. Including mine. I also think the current behaviour of the C implementation makes a lot more sense, especially as there is currently no way to request the alternative. I think using specified_attributes=False as the default behaviour for both implementations is the best solution. But I certainly would not oppose adding e.g. a keyword argument to override the default behaviour for those who would prefer the alternative. -- ___ Python tracker <https://bugs.python.org/issue42151> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
