Control: clone -1 -2 -3 Control: reassign -2 apt: -Wp,-D_GLIBCXX_ASSERTION defines new symbols Control: severity -2 important Control: reassign -3 g++-14 Control: retitle -3 g++-14: Linking picks custom library symbols over stdc++ Control: severity -3 normal
On Mon, Nov 25, 2024 at 06:59:49PM -0800, Otto Kekäläinen wrote: > Source: python-apt > Version: 2.9.1 > Severity: serious > > Seems the latest version of python-apt has some serious regressions as > https://tracker.debian.org/pkg/python-apt shows wide-spread > autopkgtest failures. > > In a clean Debian unstable container a simple installation is failing with: > > Hit:1 http://deb.debian.org/debian sid InRelease > Get:2 http://deb.debian.org/debian sid/main all Contents (deb) [43.5 MB] > Get:3 http://deb.debian.org/debian sid/main amd64 Contents (deb) [15.8 MB] > Traceback (most recent call last): > File "/usr/lib/cnf-update-db", line 3, in <module> > import apt_pkg > ImportError: > /usr/lib/python3/dist-packages/apt_pkg.cpython-312-x86_64-linux-gnu.so: > undefined symbol: > _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE10_M_replaceEmmPKcm, > version APTPKG_6.0 > Fetched 59.3 MB in 13s (4620 kB/s) > Reading package lists... > E: Problem executing scripts APT::Update::Post-Invoke-Success 'if > /usr/bin/test -w /var/lib/command-not-found/ -a -e > /usr/lib/cnf-update-db; then /usr/lib/cnf-update-db > /dev/null; fi' > E: Sub-process returned an error code > Error: error building at STEP "RUN apt-get install -q --yes > command-not-found && apt-get update -q": error while running > runtime: exit status 100 > > Please take a look what is going on. For the record, this is std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_replace(unsigned long, unsigned long, char const*, unsigned long)@@APTPKG_6.0 aka std::string::replace(). In APT 2.9.4 that is being tested, here, the symbol is undefined and referencing the version shipped in libstdc++: root@84857ed287c5:/# grep _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE10_M_replaceEmmPKcm <(nm -D /usr/lib/x86_64-linux-gnu/libapt-pkg.so.6.0) U _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE10_M_replaceEmmPKcm@GLIBCXX_3.4.21 However in 2.9.14, the symbol is suddenly defined and hence emitted using the APTPKG_6.0 versioning per our APTPKG_6.0 { global: *; }; version script: # grep _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE10_M_replaceEmmPKcm <(nm -D /usr/lib/x86_64-linux-gnu/libapt-pkg.so.6.0) 000000000009b1f0 W _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE10_M_replaceEmmPKcm@@APTPKG_6.0 This now caused python-apt to favour that version of the symbol over the C++ one when it called std::string::replace() itself. Normally if we did not have that version script, all callers of an inline function should receive a weak symbol and one is resolved at runtime - however, here the linker decided to instead drop the weak symbol in favor of using the export from APTPKG_6.0; the inverse of what happened in libapt-pkg6.0t64 itself. This is all not correct: The expectation for the symbols provided by GLIBCXX_ versions is to have them be satisfied by libstdc++, so you don't need to keep around multiple copies of common things like std::string::replace(). Rebuilding apt 2.9.14 in testing does not restore the symbol to undefined, so it must be down to a change in apt. And there is one: In the second object that is linked in, a call to std::string::replace has been dropped. Is it that? So now we build 2.9.4 and 2.9.14 and get: root@2540807186ac:/build/apt# nm -D obj-x86_64-linux-gnu/apt-pkg/libapt-pkg.so | grep _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE10_M_replaceEmmPKcm U _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE10_M_replaceEmmPKcm@GLIBCXX_3.4.21 root@2540807186ac:/build/apt# cp ^C root@2540807186ac:/build/apt# cp -a obj-x86_64-linux-gnu/ 2.9.4 root@2540807186ac:/build/apt# git checkout 2.9.14 Previous HEAD position was de7c78680 Release 2.9.4 HEAD is now at 1a03ebc11 Release 2.9.14 root@2540807186ac:/build/apt# ninja -C obj-x86_64-linux-gnu/ apt-pkg ninja: Entering directory `obj-x86_64-linux-gnu/' [0/1] Re-running CMake... -- Performing Test have-compiler-flag:-Wp,-D_GLIBCXX_ASSERTIONS -- Performing Test have-compiler-flag:-Wp,-D_GLIBCXX_ASSERTIONS - Success -- Detected vendor: debian (cached) -- Building libapt-pkg 6.0 (release 0) -- Could NOT find GTest (missing: GTEST_LIBRARY GTEST_INCLUDE_DIR GTEST_MAIN_LIBRARY) -- Found GTest at /usr/src/googletest/googletest, headers at /usr/src/googletest/googletest/include -- Configuring done (0.4s) -- Generating done (0.1s) -- Build files have been written to: /build/apt/obj-x86_64-linux-gnu [67/68] Creating library symlink apt-pkg/libapt-pkg.so.6.0 apt-pkg/libapt-pkg.so root@2540807186ac:/build/apt# nm -D obj-x86_64-linux-gnu/apt-pkg/libapt-pkg.so | grep _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE10_M_replaceEmmPKcm 0000000000089f60 W _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE10_M_replaceEmmPKcm@@APTPKG_6.0 Hmm but oh Let's revert -Wp,-D_GLIBCXX_ASSERTIONS and we get: root@2540807186ac:/build/apt# ninja -C obj-x86_64-linux-gnu/ apt-pkg ninja: Entering directory `obj-x86_64-linux-gnu/' [0/1] Re-running CMake... -- Detected vendor: debian (cached) -- Building libapt-pkg 6.0 (release 0) -- Could NOT find GTest (missing: GTEST_LIBRARY GTEST_INCLUDE_DIR GTEST_MAIN_LIBRARY) -- Found GTest at /usr/src/googletest/googletest, headers at /usr/src/googletest/googletest/include -- Configuring done (0.3s) -- Generating done (0.1s) -- Build files have been written to: /build/apt/obj-x86_64-linux-gnu [67/68] Creating library symlink apt-pkg/libapt-pkg.so.6.0 apt-pkg/libapt-pkg.so root@2540807186ac:/build/apt# nm -D obj-x86_64-linux-gnu/apt-pkg/libapt-pkg.so | grep _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE10_M_replaceEmmPKcm U _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE10_M_replaceEmmPKcm@GLIBCXX_3.4.21 So what happens is that the linker is deduplicating the symbol, preferring the @GLIBCXX_3.4.21 version when it's the same, but now we built with assertions, so it's not the same anymore, so it's not dedupped. It's unclear why when python-apt is built the linker prefers libapt-pkg's version over libstdc++ And of course we used to have these symbols pop up before we used -fvisibility=hidden, so it turns out that they had an entry in the symbols file using a regex to match, that was set to 0.8.0, and hence python-apt got the wrong version. But what is going on in python-apt itself now? It is confronted with the symbol from libstdc++ and libapt-pkg. Building the normal way we get the APTPKG_6.0 symbol: x86_64-linux-gnu-g++ -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O2 -Wall -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 build/temp.linux-x86_64-cpython-312/python/acquire-item.o build/temp.linux-x86_64-cpython-312/python/acquire.o build/temp.linux-x86_64-cpython-312/python/apt_pkgmodule.o build/temp.linux-x86_64-cpython-312/python/cache.o build/temp.linux-x86_64-cpython-312/python/cachegroup.o build/temp.linux-x86_64-cpython-312/python/cdrom.o build/temp.linux-x86_64-cpython-312/python/configuration.o build/temp.linux-x86_64-cpython-312/python/depcache.o build/temp.linux-x86_64-cpython-312/python/generic.o build/temp.linux-x86_64-cpython-312/python/hashes.o build/temp.linux-x86_64-cpython-312/python/hashstring.o build/temp.linux-x86_64-cpython-312/python/hashstringlist.o build/temp.linux-x86_64-cpython-312/python/indexfile.o build/temp.linux-x86_64-cpython-312/python/lock.o build/temp.linux-x86_64-cpython-312/python/metaindex.o build/temp.linux-x86_64-cpython-312/python/orderlist.o build/temp.linux-x86_64-cpython-312/python/pkgmanager.o build/temp.linux-x86_64-cpython-312/python/pkgmanagerprogress.o build/temp.linux-x86_64-cpython-312/python/pkgrecords.o build/temp.linux-x86_64-cpython-312/python/pkgsrcrecords.o build/temp.linux-x86_64-cpython-312/python/policy.o build/temp.linux-x86_64-cpython-312/python/progress.o build/temp.linux-x86_64-cpython-312/python/python-apt-helpers.o build/temp.linux-x86_64-cpython-312/python/sourcelist.o build/temp.linux-x86_64-cpython-312/python/string.o build/temp.linux-x86_64-cpython-312/python/tag.o -L/usr/lib/x86_64-linux-gnu -lapt-pkg -o build/lib.linux-x86_64-cpython-312/apt_pkg.cpython-312-x86_64-linux-gnu.so root@79f30361847b:/build/python-apt# nm -D build/lib.linux-x86_64-cpython-312/apt_pkg.cpython-312-x86_64-linux-gnu.so | grep '_ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE10_M_replaceEmmPKcm' U _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE10_M_replaceEmmPKcm@APTPKG_6.0 If we pass -lstdc++ -lapt-pkg we get the GLIBCXX one: x86_64-linux-gnu-g++ -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O2 -Wall -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 build/temp.linux-x86_64-cpython-312/python/acquire-item.o build/temp.linux-x86_64-cpython-312/python/acquire.o build/temp.linux-x86_64-cpython-312/python/apt_pkgmodule.o build/temp.linux-x86_64-cpython-312/python/cache.o build/temp.linux-x86_64-cpython-312/python/cachegroup.o build/temp.linux-x86_64-cpython-312/python/cdrom.o build/temp.linux-x86_64-cpython-312/python/configuration.o build/temp.linux-x86_64-cpython-312/python/depcache.o build/temp.linux-x86_64-cpython-312/python/generic.o build/temp.linux-x86_64-cpython-312/python/hashes.o build/temp.linux-x86_64-cpython-312/python/hashstring.o build/temp.linux-x86_64-cpython-312/python/hashstringlist.o build/temp.linux-x86_64-cpython-312/python/indexfile.o build/temp.linux-x86_64-cpython-312/python/lock.o build/temp.linux-x86_64-cpython-312/python/metaindex.o build/temp.linux-x86_64-cpython-312/python/orderlist.o build/temp.linux-x86_64-cpython-312/python/pkgmanager.o build/temp.linux-x86_64-cpython-312/python/pkgmanagerprogress.o build/temp.linux-x86_64-cpython-312/python/pkgrecords.o build/temp.linux-x86_64-cpython-312/python/pkgsrcrecords.o build/temp.linux-x86_64-cpython-312/python/policy.o build/temp.linux-x86_64-cpython-312/python/progress.o build/temp.linux-x86_64-cpython-312/python/python-apt-helpers.o build/temp.linux-x86_64-cpython-312/python/sourcelist.o build/temp.linux-x86_64-cpython-312/python/string.o build/temp.linux-x86_64-cpython-312/python/tag.o -L/usr/lib/x86_64-linux-gnu -lstdc++ -lapt-pkg -o build/lib.linux-x86_64-cpython-312/apt_pkg.cpython-312-x86_64-linux-gnu.so root@79f30361847b:/build/python-apt# nm -D build/lib.linux-x86_64-cpython-312/apt_pkg.cpython-312-x86_64-linux-gnu.so | grep '_ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE10_M_replaceEmmPKcm' U _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE10_M_replaceEmmPKcm@GLIBCXX_3.4.21 My expectation would have been for ld as invoked by g++ to always pass -lstdc++ first such that the symbol resolves to the libstdc++ one. -- debian developer - deb.li/jak | jak-linux.org - free software dev ubuntu core developer i speak de, en
