Re: [Python-Dev] Deterministic builds of the interpreter

2017-02-19 Thread Freddy Rietdijk
Hi Armin, Thank you for your explanation. I've now managed to build 2.7 and 3.5 deterministic by recompiling the bytecode at the end of the build (and excluding 2to3). Freddy On Sun, Feb 19, 2017 at 9:30 AM, Armin Rigo wrote: > Hi Freddy, > > On 16 February 2017 at 18:03, Freddy Rietdijk >

Re: [Python-Dev] Deterministic builds of the interpreter

2017-02-19 Thread Armin Rigo
Hi Freddy, On 16 February 2017 at 18:03, Freddy Rietdijk wrote: > As I mentioned, it seems only sets cause unreproducible > bytecode. Sets have no order. But when generating the bytecode, I would > expect there would still be an order since the code isn't actually executed, > right? No, the sets

Re: [Python-Dev] Deterministic builds of the interpreter

2017-02-16 Thread Freddy Rietdijk
Hi, Are there anymore suggestions how to improve the determinism of the Python 3 interpreter? As I mentioned, it seems only sets cause unreproducible bytecode. Sets have no order. But when generating the bytecode, I would expect there would still be an order since the code isn't actually executed,

Re: [Python-Dev] Deterministic builds of the interpreter

2017-02-10 Thread Freddy Rietdijk
That should have been `PYTHONHASHSEED=0 $out/bin/python -m compileall -f $out`. On Fri, Feb 10, 2017 at 11:58 AM, Freddy Rietdijk wrote: > For Python 3.5 PYTHONHASHSEED doesn't seem to be sufficient, these items > still seem indeterministic. > To be sure, I ran `PYTHONHASHSEED=1 $out/bin/python

Re: [Python-Dev] Deterministic builds of the interpreter

2017-02-10 Thread Freddy Rietdijk
For Python 3.5 PYTHONHASHSEED doesn't seem to be sufficient, these items still seem indeterministic. To be sure, I ran `PYTHONHASHSEED=1 $out/bin/python -m compileall -f $out` where $out is the path where I installed Python. Do you have an idea why in [3], this is Python 2.7, the timestamps are st

Re: [Python-Dev] Deterministic builds of the interpreter

2017-02-10 Thread INADA Naoki
On Fri, Feb 10, 2017 at 7:58 PM, Freddy Rietdijk wrote: > For Python 3.5 PYTHONHASHSEED doesn't seem to be sufficient, these items > still seem indeterministic. > To be sure, I ran `PYTHONHASHSEED=1 $out/bin/python -m compileall -f $out` > where $out is the path where I installed Python. > > Do yo

Re: [Python-Dev] Deterministic builds of the interpreter

2017-02-09 Thread INADA Naoki
On Fri, Feb 10, 2017 at 2:45 AM, Freddy Rietdijk wrote: > Correct, that was an older version from before I patched > `_bootstrap_external.py`. A more recent diff can be found at > > https://gist.github.com/anonymous/d40f24fd6b636ba40d345ff3f12a0aaa > > These all seem to be sets. Maybe, PYTHONHASH

Re: [Python-Dev] Deterministic builds of the interpreter

2017-02-09 Thread Freddy Rietdijk
Correct, that was an older version from before I patched `_bootstrap_external.py`. A more recent diff can be found at https://gist.github.com/anonymous/d40f24fd6b636ba40d345ff3f12a0aaa These all seem to be sets. On Thu, Feb 9, 2017 at 6:04 PM, INADA Naoki wrote: > As reading [4], mtime is not

Re: [Python-Dev] Deterministic builds of the interpreter

2017-02-09 Thread INADA Naoki
As reading [4], mtime is not 0. data = bytearray(MAGIC_NUMBER) data.extend(_w_long(mtime)) data.extend(_w_long(source_size)) data.extend(marshal.dumps(code)) First 4 bytes are magic. Next 4 bytes are mtime. │ │ │ │ -: 160d 0d0a 6b2e 9c58 6c21 e300 k..Xl!

[Python-Dev] Deterministic builds of the interpreter

2017-02-09 Thread Freddy Rietdijk
Hi, I'm attempting to make the builds of the Python interpreters for Nixpkgs [1] deterministic. In the case of Python 2.7 we have a patch [2] that fixes the timestamp used in .pyc files in case the env var `DETERMINISTIC_BUILD` is set. We also remove `wininst*.exe`. This works fine, although ther