Your message dated Mon, 16 Oct 2023 20:51:58 +0000 with message-id <e1qsuzc-007mrp...@fasolo.debian.org> and subject line Bug#1028212: fixed in prometheus-node-exporter-collectors 0.0~git20231016.66010f0-1 has caused the Debian Bug report #1028212, regarding prometheus-node-exporter-collectors: APT update deadlock - prevents unattended security upgrades to be marked as done.
This means that you claim that the problem has been dealt with. If this is not the case it is now your responsibility to reopen the Bug report if necessary, and/or fix the problem forthwith. (NB: If you are a system administrator and have no idea what this message is talking about, this may indicate a serious mail system misconfiguration somewhere. Please contact ow...@bugs.debian.org immediately.) -- 1028212: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1028212 Debian Bug Tracking System Contact ow...@bugs.debian.org with problems
--- Begin Message ---Package: prometheus-node-exporter-collectors Version: 0.0~git20221011.8f6be63-1 Severity: important Tags: security X-Debbugs-Cc: Salvatore Bonaccorso <car...@debian.org>, steve+...@tecwec.eu, Debian Security Team <t...@security.debian.org> As requested, by Salvatore lowering prio and avoiding embargo. ----- Hello, happy new year, and thanks. This looks like an apt deadlock, which prevents updates, unattended upgrades, and so critical security updates for systems where they are enabled. (Yes, we can just manually kill the offending apt_info.py process to temporarily solve the issue - but this is not the good solution). As it prevents security updates, and despite it unlikely to happen massively, and be practically exploited, I feel this requires real attention. Symptoms: Persistent apt update locking error: # apt update Reading package lists... Done E: Could not get lock /var/lib/apt/lists/lock. It is held by process 65553 (python3) N: Be aware that removing the lock file is not a solution and may break your system. E: Unable to lock directory /var/lib/apt/lists/ # 1 hour later, same issue, same holding PID 65553 # Concerned processes: # ps aux |grep pyth root 1259 0.0 0.1 121076 27528 ? Ssl Jan06 0:00 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgra> root 65553 0.0 0.4 89640 76908 ? S 12:09 0:03 python3 /usr/share/prometheus-node-exporter-collectors/apt_info.py ee 70395 0.0 0.2 124164 42844 ? Sl 12:35 0:00 /bin/python3.11 /home/ee/.vscode-oss/extensions/ms-python.python> (not suspected) # ps aux |grep apt root 65551 0.0 0.0 9552 4252 ? Ss 12:09 0:00 /bin/bash -c /usr/share/prometheus-node-exporter-collectors/apt_> root 65553 0.0 0.4 89640 76908 ? S 12:09 0:03 python3 /usr/share/prometheus-node-exporter-collectors/apt_info.> root 65554 0.0 0.0 2464 884 ? S 12:09 0:00 sponge /var/lib/prometheus/node-exporter/apt.prom _apt 65814 0.0 0.0 27192 13204 ? S 12:09 0:00 /usr/lib/apt/methods/https _apt 65815 0.0 0.0 24420 10236 ? S 12:09 0:00 /usr/lib/apt/methods/http _apt 65816 0.0 0.0 27192 13204 ? S 12:09 0:00 /usr/lib/apt/methods/https _apt 65817 0.0 0.0 24420 10272 ? S 12:09 0:00 /usr/lib/apt/methods/http _apt 65819 0.0 0.0 17572 7624 ? S 12:09 0:00 /usr/lib/apt/methods/gpgv _apt 65826 0.0 0.0 27192 13464 ? S 12:09 0:00 /usr/lib/apt/methods/https _apt 65829 0.0 0.0 24420 10292 ? S 12:09 0:00 /usr/lib/apt/methods/http _apt 66110 0.0 0.0 17528 7500 ? S 12:10 0:00 /usr/lib/apt/methods/store _apt 66112 0.0 0.0 18436 8636 ? S 12:10 0:00 /usr/lib/apt/methods/rred _apt 66113 0.0 0.0 18576 8860 ? S 12:10 0:00 /usr/lib/apt/methods/rred The deadlock is obviously between the unattended-upgrade proc (1259), and the prometheus tryptic: 65551/53/54. # 65553 seems to be the culprit - as apt update tells us # strace -p 65553 strace: Process 65553 attached pselect6(29, [12 13 14 16 18 20 22 24 26 28], [], NULL, {tv_sec=0, tv_nsec=499419645}, NULL) = 0 (Timeout) pselect6(29, [12 13 14 16 18 20 22 24 26 28], [], NULL, {tv_sec=0, tv_nsec=500000000}, NULL) = 0 (Timeout) ... repeats 'forever' .... All fds are pipes, I could not get more info until the processed crashed due to my diagnostic atttempts. An apt/python/prom collector specialist should instantly identify these pipes and make more deductions, from the following state: # gdb -p 65553 and bt: #0 0x00007fa4bf65f794 in __GI___select (nfds=29, readfds=0x7ffc24f8e7c0, writefds=0x7ffc24f8e840, exceptfds=0x0, timeout=0x7ffc24f8e750) at ../sysdeps/unix/sysv/linux/select.c:69 #1 0x00007fa4bebad338 in pkgAcquire::Run(int) () from /lib/x86_64-linux-gnu/libapt-pkg.so.6.0 #2 0x00007fa4becb1485 in AcquireUpdate(pkgAcquire&, int, bool, bool) () from /lib/x86_64-linux-gnu/libapt-pkg.so.6.0 #3 0x00007fa4becb1976 in ListUpdate(pkgAcquireStatus&, pkgSourceList&, int) () from /lib/x86_64-linux-gnu/libapt-pkg.so.6.0 #4 0x00007fa4bed32fe1 in ?? () from /usr/lib/python3/dist-packages/apt_pkg.cpython-311-x86_64-linux-gnu.so #5 0x0000000000521cf0 in ?? () #6 0x000000000053983c in PyObject_Vectorcall () #7 0x000000000052a570 in _PyEval_EvalFrameDefault () #8 0x000000000052222b in PyEval_EvalCode () #9 0x0000000000647f07 in ?? () #10 0x00000000006457cf in ?? () #11 0x0000000000651920 in ?? () #12 0x000000000065166b in _PyRun_SimpleFileObject () #13 0x0000000000651494 in _PyRun_AnyFileObject () #14 0x000000000065022f in Py_RunMain () #15 0x00000000006248b7 in Py_BytesMain () #16 0x00007fa4bf58818a in __libc_start_call_main (main=main@entry=0x624820, argc=argc@entry=2, argv=argv@entry=0x7ffc24f8f298) at ../sysdeps/nptl/libc_start_call_main.h:58 #17 0x00007fa4bf588245 in __libc_start_main_impl (main=0x624820, argc=2, argv=0x7ffc24f8f298, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffc24f8f288) at ../csu/libc-start.c:381 #18 0x0000000000624751 in _start () This seems to suggest that the location of the deadlock, for 65553, is: (apt_info.py) def _main(): cache = apt.cache.Cache() # First of all, attempt to update the index. If we don't have permission # to do so (or it fails for some reason), it's not the end of the world, # we'll operate on the old index. with contextlib.suppress(apt.cache.LockFailedException, apt.cache.FetchFailedException): cache.update() <<<<<<<<<<<< VERY LIKELY I could not confirm the precise location, as trying to get a python backtrace from the process generated a SEGV: (gdb) call PyRun_SimpleString("print('toto\n')") # to test 'PyRun_SimpleString' has unknown return type; cast the call to its declared return type (gdb) call (void*)PyRun_SimpleString("print('toto\n')") Program received signal SIGSEGV, Segmentation fault. # Oops... will not get a python trace now. Hopefully, I collected the core (~27MB) - if interested, tell me - keeping it for a few weeks: #0 0x000000000063187a in ?? () #1 0x00000000006349b2 in PyImport_AddModuleObject () #2 0x0000000000634688 in PyImport_AddModule () #3 0x000000000063e323 in PyRun_SimpleStringFlags () (but I feel it unrelated, and not so usefull - but I may be wrong) I feel I can't help more now, so throwing the potato 😉 Best, Eric 'Steve' Estievenart -- System Information: Debian Release: bookworm/sid APT prefers unstable APT policy: (990, 'unstable') Architecture: amd64 (x86_64) Kernel: Linux 6.0.0-6-amd64 (SMP w/4 CPU threads; PREEMPT) Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US:en Shell: /bin/sh linked to /usr/bin/dash Init: systemd (via /run/systemd/system) LSM: AppArmor: enabled Versions of packages prometheus-node-exporter-collectors depends on: ii moreutils 0.67-1 ii prometheus-node-exporter 1.5.0-1+b1 ii python3-apt 2.5.0 ii systemd-sysv 252.4-1 Versions of packages prometheus-node-exporter-collectors recommends: ii ipmitool 1.8.19-4 ii jq 1.6-2.1 ii nvme-cli 2.2.1-3 ii python3 3.11.1-1 ii smartmontools 7.3-1+b1 prometheus-node-exporter-collectors suggests no packages. -- no debconf information
--- End Message ---
--- Begin Message ---Source: prometheus-node-exporter-collectors Source-Version: 0.0~git20231016.66010f0-1 Done: Antoine Beaupré <anar...@debian.org> We believe that the bug you reported is fixed in the latest version of prometheus-node-exporter-collectors, which is due to be installed in the Debian FTP archive. A summary of the changes between this version and the previous one is attached. Thank you for reporting the bug, which will now be closed. If you have further comments please address them to 1028...@bugs.debian.org, and the maintainer will reopen the bug report if appropriate. Debian distribution maintenance software pp. Antoine Beaupré <anar...@debian.org> (supplier of updated prometheus-node-exporter-collectors package) (This message was generated automatically at their request; if you believe that there is a problem with it please contact the archive administrators by mailing ftpmas...@ftp-master.debian.org) -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Format: 1.8 Date: Mon, 16 Oct 2023 15:27:25 -0400 Source: prometheus-node-exporter-collectors Architecture: source Version: 0.0~git20231016.66010f0-1 Distribution: unstable Urgency: medium Maintainer: Debian Go Packaging Team <team+pkg...@tracker.debian.org> Changed-By: Antoine Beaupré <anar...@debian.org> Closes: 1028212 Changes: prometheus-node-exporter-collectors (0.0~git20231016.66010f0-1) unstable; urgency=medium . * Team upload. * New upstream snapshot (Closes: #1028212) Checksums-Sha1: 05cee0333e123a60ab3d7bc1df991681ec9cb90f 1813 prometheus-node-exporter-collectors_0.0~git20231016.66010f0-1.dsc ac23e023af54b14b8138386c9a2d78955cc2376d 36320 prometheus-node-exporter-collectors_0.0~git20231016.66010f0.orig.tar.xz 05da95387d7871700277c43d943b997a12736ef1 4256 prometheus-node-exporter-collectors_0.0~git20231016.66010f0-1.debian.tar.xz c838c61da4f9f9f47421a8a402aa39412216ebbf 5574 prometheus-node-exporter-collectors_0.0~git20231016.66010f0-1_amd64.buildinfo Checksums-Sha256: a2660611404f8a84e82dd22ea9b8b5647bcc729b6c078d1fb5bda8c19b7b66f1 1813 prometheus-node-exporter-collectors_0.0~git20231016.66010f0-1.dsc 5dadaedf43d624f1cac377c2bf6f9d45f9b5062fe5faa1d63671c5c10ef0093c 36320 prometheus-node-exporter-collectors_0.0~git20231016.66010f0.orig.tar.xz 9e8a7377f75207ef416e047c082ffae774e6c98b7ebe1bf94b2971d03479800e 4256 prometheus-node-exporter-collectors_0.0~git20231016.66010f0-1.debian.tar.xz 46d1e3838b09cbcaa5f004e2f96548b59c5ad5ac51859296d51342c723ac48d0 5574 prometheus-node-exporter-collectors_0.0~git20231016.66010f0-1_amd64.buildinfo Files: 89ac836a71819591964302ec6a67aa4d 1813 net optional prometheus-node-exporter-collectors_0.0~git20231016.66010f0-1.dsc db8f40b2085d802792af641895935931 36320 net optional prometheus-node-exporter-collectors_0.0~git20231016.66010f0.orig.tar.xz 9ceee4919d1aff30ac7fdf2364699e8d 4256 net optional prometheus-node-exporter-collectors_0.0~git20231016.66010f0-1.debian.tar.xz 38023b63e214ea478a2d4a17d77fe9a4 5574 net optional prometheus-node-exporter-collectors_0.0~git20231016.66010f0-1_amd64.buildinfo -----BEGIN PGP SIGNATURE----- iHUEARYIAB0WIQS7ts1MmNdOE1inUqYCKTpvpOU0cwUCZS2TUgAKCRACKTpvpOU0 c+yoAQC2UiiFqcz6ZGTmQQYLNFgnWu9EIPFsZkpMD66Qu5XVaQD+M011/jQpLMvW 5fi9nRadE2WuwVfRhyS/gdcR9BrP2g8= =3pKR -----END PGP SIGNATURE-----
--- End Message ---
