On 2024-11-01 06:08, Alexander Kanavin wrote:
On Wed, 30 Oct 2024 at 15:23, Mathieu Dubois-Briand via
lists.openembedded.org
<[email protected]> wrote:
Differences can be seen here:
- https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20241030-a5e77d8y/
-
https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20241030-a5e77d8y/packages/diff-html/
Can you have a look please ?
I believe there are two separate reproducibility issues there:
1. usr/lib/python3.13/site-packages/numpy/__config__.py contains
numerous references to compiler and linker flags which contains build
paths. This is not uncommon, and we generally handle it by patching
these out after the fact with sed in do_install:append() or similar.
Plenty of examples all over oe-core.
2. The second issue is slightly trickier. There's a number of
generated c files produced by cython and they contain lines like:
static const char __pyx_k_srv_storage_alex_yocto_build_64[] =
"/srv/storage/alex/yocto/build-64/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/numpy-2.1.2/numpy/random/bit_generator.pyx";
PyObject *__pyx_kp_s_srv_storage_alex_yocto_build_64;
etc.
This needs to be fixed by tracking down where in cython this stuff is
generated, and replacing real paths (or parts of them) with something
reproducible, like 'reproducible_build_path' or similar. Then this
should be submitted or raised with upstream.
Yeah, I've gotten this far, but I think the real issue is that these
should be configurable via some (possibly-nonexistent) bitbake variable
in the recipe. I haven't yet determined how to add this stuff yet, though.
In the python3-pandas recipe in meta-python, there's a
fix_cythonized_sources() function that seems to be of the right spirit:
https://git.openembedded.org/meta-openembedded/tree/meta-python/recipes-devtools/python/python3-pandas_2.2.2.bb#n26
I don't think this is quite the right fix, though - it seems like these
adjustments should be happening during do_configure if possible, so that
the host paths aren't being propagated through all of the different
copies of files like __config__.py...
There are quite a lot of hits under the numpy workdir for "Generated by
Cython":
tgamblin@megalith ~/workspace/yocto/poky (master)$ grep -r "Generated by
Cython" build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/temp/run.fix_cythonized_sources.581946:
for f in `grep -l -r -e '\/* Generated by Cython.*/$'
/home/tgamblin/workspace/yocto/poky/build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2`;
do
grep:
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/recipe-sysroot-native/usr/lib/python3.13/site-packages/Cython/Utils.cpython-313-x86_64-linux-gnu.so:
binary file matches
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/recipe-sysroot-native/usr/lib/python3.13/site-packages/Cython/Utils.py:GENERATED_BY_MARKER
= "/* Generated by Cython %s */" % cython_version
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/recipe-sysroot-native/usr/lib/python3.13/site-packages/Cython/Utils.py:
file_content.startswith(b"/* Generated by Cython ") or
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/recipe-sysroot-native/usr/lib/python3.13/site-packages/Cython/Compiler/Annotate.py:
<!-- Generated by Cython {watermark} -->
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/recipe-sysroot-native/usr/lib/python3.13/site-packages/Cython/Compiler/Annotate.py:
<p><span style="border-bottom: solid 1px grey;">Generated by Cython
{watermark}</span>{more_info}</p>
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/recipe-sysroot-native/usr/lib/python3.13/site-packages/Cython/Compiler/Main.py:
if u'<!-- Generated by Cython' in html_file.read(100):
grep:
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/recipe-sysroot-native/usr/lib/python3.13/site-packages/Cython/Compiler/__pycache__/Annotate.cpython-313.pyc:
binary file matches
grep:
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/recipe-sysroot-native/usr/lib/python3.13/site-packages/Cython/Compiler/__pycache__/Main.cpython-313.pyc:
binary file matches
grep:
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/recipe-sysroot-native/usr/lib/python3.13/site-packages/Cython/__pycache__/Utils.cpython-313.pyc:
binary file matches
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/build/numpy/random/_mt19937.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_mt19937.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/build/numpy/random/_common.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_common.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/build/numpy/random/_generator.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_generator.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/build/numpy/random/_bounded_integers.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_bounded_integers.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/build/numpy/random/_pcg64.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_pcg64.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/build/numpy/random/_sfc64.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_sfc64.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/build/numpy/random/bit_generator.cpython-313-x86_64-linux-gnu.so.p/numpy/random/bit_generator.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/build/numpy/random/mtrand.cpython-313-x86_64-linux-gnu.so.p/numpy/random/mtrand.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/build/numpy/random/_philox.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_philox.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/package/usr/src/debug/python3-numpy/2.1.2/numpy/random/_mt19937.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_mt19937.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/package/usr/src/debug/python3-numpy/2.1.2/numpy/random/_common.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_common.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/package/usr/src/debug/python3-numpy/2.1.2/numpy/random/_generator.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_generator.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/package/usr/src/debug/python3-numpy/2.1.2/numpy/random/_bounded_integers.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_bounded_integers.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/package/usr/src/debug/python3-numpy/2.1.2/numpy/random/_pcg64.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_pcg64.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/package/usr/src/debug/python3-numpy/2.1.2/numpy/random/_sfc64.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_sfc64.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/package/usr/src/debug/python3-numpy/2.1.2/numpy/random/bit_generator.cpython-313-x86_64-linux-gnu.so.p/numpy/random/bit_generator.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/package/usr/src/debug/python3-numpy/2.1.2/numpy/random/mtrand.cpython-313-x86_64-linux-gnu.so.p/numpy/random/mtrand.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/package/usr/src/debug/python3-numpy/2.1.2/numpy/random/_philox.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_philox.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/packages-split/python3-numpy-src/usr/src/debug/python3-numpy/2.1.2/numpy/random/_mt19937.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_mt19937.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/packages-split/python3-numpy-src/usr/src/debug/python3-numpy/2.1.2/numpy/random/_common.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_common.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/packages-split/python3-numpy-src/usr/src/debug/python3-numpy/2.1.2/numpy/random/_generator.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_generator.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/packages-split/python3-numpy-src/usr/src/debug/python3-numpy/2.1.2/numpy/random/_bounded_integers.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_bounded_integers.pyx.c:/*
Generated by Cython 3.0.11/
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/packages-split/python3-numpy-src/usr/src/debug/python3-numpy/2.1.2/numpy/random/_pcg64.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_pcg64.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/packages-split/python3-numpy-src/usr/src/debug/python3-numpy/2.1.2/numpy/random/_sfc64.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_sfc64.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/packages-split/python3-numpy-src/usr/src/debug/python3-numpy/2.1.2/numpy/random/bit_generator.cpython-313-x86_64-linux-gnu.so.p/numpy/random/bit_generator.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/packages-split/python3-numpy-src/usr/src/debug/python3-numpy/2.1.2/numpy/random/mtrand.cpython-313-x86_64-linux-gnu.so.p/numpy/random/mtrand.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/packages-split/python3-numpy-src/usr/src/debug/python3-numpy/2.1.2/numpy/random/_philox.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_philox.pyx.c:/*
Generated by Cython 3.0.11 */
This makes me think that the Cython recipe is also missing a few cases
in cython_fix_sources() (namely, ../Cython/Compiler/Annotate.py,
../Cython/Compiler/Main.py, and ../Cython/Utils.py, but locally adding
those and rebuild doesn't seem to fix the numpy issue.
Still looking.
3. We should also drop
# Python pyx -> c -> so build leaves absolute build paths in the code
INSANE_SKIP:${PN} += "buildpaths"
INSANE_SKIP:${PN}-src += "buildpaths"
from meta/classes-recipe/python_mesonpy.bbclass when point two is addressed.
Completely agree, it's on my radar. Apparently I need to learn a lot
more about how all of this is actually happening underneath...
Hope this helps,
Alex
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#206621):
https://lists.openembedded.org/g/openembedded-core/message/206621
Mute This Topic: https://lists.openembedded.org/mt/109284027/21656
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
