Occurred again. This time, I got it right and got a trace on everything:
Output is for "thread apply all bt" in gdb for each pid (other than systemd) in "pstree -sp 42017". It looks like it's definitely stuck inside a network call. PROCESS 42017 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 42017 0.0 0.0 2800 1956 ? Ss Jun02 0:00 /bin/sh /usr/lib/apt/apt.systemd.daily update [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Download failed: Invalid argument. Continuing without source file ./posix/../sysdeps/unix/sysv/linux/wait4.c. 0x00007d95a7f107a7 in __GI___wait4 (pid=-1, stat_loc=0x7ffce02c6f6c, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30 Thread 1 (Thread 0x7d95a81ec740 (LWP 42017) "apt.systemd.dai"): #0 0x00007d95a7f107a7 in __GI___wait4 (pid=-1, stat_loc=0x7ffce02c6f6c, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30 #1 0x00005ec3c029621d in ?? () #2 0x00005ec3c02964a9 in ?? () #3 0x00005ec3c02a25e9 in ?? () #4 0x00005ec3c02991b0 in ?? () #5 0x00005ec3c02991b0 in ?? () #6 0x00005ec3c0299173 in ?? () #7 0x00005ec3c02991b0 in ?? () #8 0x00005ec3c02a163d in ?? () #9 0x00005ec3c0294d7f in ?? () #10 0x00007d95a7e2a1ca in __libc_start_call_main (main=main@entry=0x5ec3c0294b20, argc=argc@entry=3, argv=argv@entry=0x7ffce02c7678) at ../sysdeps/nptl/libc_start_call_main.h:58 #11 0x00007d95a7e2a28b in __libc_start_main_impl (main=0x5ec3c0294b20, argc=3, argv=0x7ffce02c7678, init=<optimised out>, fini=<optimised out>, rtld_fini=<optimised out>, stack_end=0x7ffce02c7668) at ../csu/libc-start.c:360 #12 0x00005ec3c0294f35 in ?? () [Inferior 1 (process 42017) detached] PROCESS 42022 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 42022 0.0 0.0 2800 1972 ? S Jun02 0:00 /bin/sh /usr/lib/apt/apt.systemd.daily lock_is_held update [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Download failed: Invalid argument. Continuing without source file ./posix/../sysdeps/unix/sysv/linux/wait4.c. 0x00007a64ed3107a7 in __GI___wait4 (pid=-1, stat_loc=0x7ffd573d4fcc, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30 Thread 1 (Thread 0x7a64ed602740 (LWP 42022) "apt.systemd.dai"): #0 0x00007a64ed3107a7 in __GI___wait4 (pid=-1, stat_loc=0x7ffd573d4fcc, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30 #1 0x00005e13a9a5421d in ?? () #2 0x00005e13a9a544a9 in ?? () #3 0x00005e13a9a605e9 in ?? () #4 0x00005e13a9a571b0 in ?? () #5 0x00005e13a9a5da49 in ?? () #6 0x00005e13a9a5dddf in ?? () #7 0x00005e13a9a6074b in ?? () #8 0x00005e13a9a571b0 in ?? () #9 0x00005e13a9a5736e in ?? () #10 0x00005e13a9a571b0 in ?? () #11 0x00005e13a9a571b0 in ?? () #12 0x00005e13a9a57173 in ?? () #13 0x00005e13a9a57173 in ?? () #14 0x00005e13a9a57173 in ?? () #15 0x00005e13a9a571b0 in ?? () #16 0x00005e13a9a5f63d in ?? () #17 0x00005e13a9a52d7f in ?? () #18 0x00007a64ed22a1ca in __libc_start_call_main (main=main@entry=0x5e13a9a52b20, argc=argc@entry=4, argv=argv@entry=0x7ffd573d5bc8) at ../sysdeps/nptl/libc_start_call_main.h:58 #19 0x00007a64ed22a28b in __libc_start_main_impl (main=0x5e13a9a52b20, argc=4, argv=0x7ffd573d5bc8, init=<optimised out>, fini=<optimised out>, rtld_fini=<optimised out>, stack_end=0x7ffd573d5bb8) at ../csu/libc-start.c:360 #20 0x00005e13a9a52f35 in ?? () [Inferior 1 (process 42022) detached] PROCESS 42067 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 42067 0.0 0.0 21364 11324 ? S Jun02 0:00 apt-get -qq -y update [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Download failed: Invalid argument. Continuing without source file ./misc/../sysdeps/unix/sysv/linux/select.c. 0x000074010cb26c3e in __GI___select (nfds=12, readfds=0x7ffc979403f0, writefds=0x7ffc97940370, exceptfds=0x0, timeout=0x7ffc97940300) at ../sysdeps/unix/sysv/linux/select.c:69 Thread 1 (Thread 0x74010d08c880 (LWP 42067) "apt-get"): #0 0x000074010cb26c3e in __GI___select (nfds=12, readfds=0x7ffc979403f0, writefds=0x7ffc97940370, exceptfds=0x0, timeout=0x7ffc97940300) at ../sysdeps/unix/sysv/linux/select.c:69 #1 0x000074010d233f27 in pkgAcquire::Run(int) () from /lib/x86_64-linux-gnu/libapt-pkg.so.6.0 #2 0x000074010d320a55 in AcquireUpdate(pkgAcquire&, int, bool, bool) () from /lib/x86_64-linux-gnu/libapt-pkg.so.6.0 #3 0x000074010d320e76 in ListUpdate(pkgAcquireStatus&, pkgSourceList&, int) () from /lib/x86_64-linux-gnu/libapt-pkg.so.6.0 #4 0x000074010d3d8385 in DoUpdate() () from /lib/x86_64-linux-gnu/libapt-private.so.0.0 #5 0x000074010d26626f in CommandLine::DispatchArg(CommandLine::Dispatch const*, bool) () from /lib/x86_64-linux-gnu/libapt-pkg.so.6.0 #6 0x000074010d3a5396 in DispatchCommandLine(CommandLine&, std::vector<CommandLine::Dispatch, std::allocator<CommandLine::Dispatch> > const&) () from /lib/x86_64-linux-gnu/libapt-private.so.0.0 #7 0x000058456934a13e in ?? () #8 0x000074010ca2a1ca in __libc_start_call_main (main=main@entry=0x58456934a0a0, argc=argc@entry=4, argv=argv@entry=0x7ffc97940e78) at ../sysdeps/nptl/libc_start_call_main.h:58 #9 0x000074010ca2a28b in __libc_start_main_impl (main=0x58456934a0a0, argc=4, argv=0x7ffc97940e78, init=<optimised out>, fini=<optimised out>, rtld_fini=<optimised out>, stack_end=0x7ffc97940e68) at ../csu/libc-start.c:360 #10 0x000058456934a1d5 in ?? () [Inferior 1 (process 42067) detached] PROCESS 42117 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND _apt 42117 0.0 0.0 18292 7184 ? S Jun02 0:00 /usr/lib/apt/methods/gpgv [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Download failed: Invalid argument. Continuing without source file ./misc/../sysdeps/unix/sysv/linux/select.c. 0x0000798dc8926c3e in __GI___select (nfds=1, readfds=0x7ffc549f01e0, writefds=0x0, exceptfds=0x0, timeout=0x0) at ../sysdeps/unix/sysv/linux/select.c:69 Thread 1 (Thread 0x798dc8e81880 (LWP 42117) "gpgv"): #0 0x0000798dc8926c3e in __GI___select (nfds=1, readfds=0x7ffc549f01e0, writefds=0x0, exceptfds=0x0, timeout=0x0) at ../sysdeps/unix/sysv/linux/select.c:69 #1 0x0000798dc8fae38c in WaitFd(int, bool, unsigned long) () from /lib/x86_64-linux-gnu/libapt-pkg.so.6.0 #2 0x0000798dc8f630f3 in pkgAcqMethod::Run(bool) () from /lib/x86_64-linux-gnu/libapt-pkg.so.6.0 #3 0x0000578bf5ecc9fe in ?? () #4 0x0000798dc882a1ca in __libc_start_call_main (main=main@entry=0x578bf5ecc8b0, argc=argc@entry=1, argv=argv@entry=0x7ffc549f06f8) at ../sysdeps/nptl/libc_start_call_main.h:58 #5 0x0000798dc882a28b in __libc_start_main_impl (main=0x578bf5ecc8b0, argc=1, argv=0x7ffc549f06f8, init=<optimised out>, fini=<optimised out>, rtld_fini=<optimised out>, stack_end=0x7ffc549f06e8) at ../csu/libc-start.c:360 #6 0x0000578bf5eccb85 in ?? () [Inferior 1 (process 42117) detached] PROCESS 42077 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND _apt 42077 0.0 0.0 25460 10960 ? S Jun02 0:00 /usr/lib/apt/methods/http [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Download failed: Invalid argument. Continuing without source file ./misc/../sysdeps/unix/sysv/linux/select.c. 0x000074c196326c3e in __GI___select (nfds=1, readfds=0x7ffe58e6c540, writefds=0x0, exceptfds=0x0, timeout=0x0) at ../sysdeps/unix/sysv/linux/select.c:69 Thread 1 (Thread 0x74c19687fa80 (LWP 42077) "http"): #0 0x000074c196326c3e in __GI___select (nfds=1, readfds=0x7ffe58e6c540, writefds=0x0, exceptfds=0x0, timeout=0x0) at ../sysdeps/unix/sysv/linux/select.c:69 #1 0x000074c196d6a38c in WaitFd(int, bool, unsigned long) () from /lib/x86_64-linux-gnu/libapt-pkg.so.6.0 #2 0x00005a376b2f5c97 in ?? () #3 0x000074c19622a1ca in __libc_start_call_main (main=main@entry=0x5a376b2f5880, argc=argc@entry=1, argv=argv@entry=0x7ffe58e6cd08) at ../sysdeps/nptl/libc_start_call_main.h:58 #4 0x000074c19622a28b in __libc_start_main_impl (main=0x5a376b2f5880, argc=1, argv=0x7ffe58e6cd08, init=<optimised out>, fini=<optimised out>, rtld_fini=<optimised out>, stack_end=0x7ffe58e6ccf8) at ../csu/libc-start.c:360 #5 0x00005a376b2f8485 in ?? () [Inferior 1 (process 42077) detached] PROCESS 42078 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND _apt 42078 0.0 0.0 25460 10964 ? S Jun02 0:00 /usr/lib/apt/methods/http [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Download failed: Invalid argument. Continuing without source file ./misc/../sysdeps/unix/sysv/linux/select.c. 0x0000762446d26c3e in __GI___select (nfds=1, readfds=0x7ffcf2cd0750, writefds=0x0, exceptfds=0x0, timeout=0x0) at ../sysdeps/unix/sysv/linux/select.c:69 Thread 1 (Thread 0x7624465afa80 (LWP 42078) "http"): #0 0x0000762446d26c3e in __GI___select (nfds=1, readfds=0x7ffcf2cd0750, writefds=0x0, exceptfds=0x0, timeout=0x0) at ../sysdeps/unix/sysv/linux/select.c:69 #1 0x00007624476e938c in WaitFd(int, bool, unsigned long) () from /lib/x86_64-linux-gnu/libapt-pkg.so.6.0 #2 0x000055a03c457c97 in ?? () #3 0x0000762446c2a1ca in __libc_start_call_main (main=main@entry=0x55a03c457880, argc=argc@entry=1, argv=argv@entry=0x7ffcf2cd0f18) at ../sysdeps/nptl/libc_start_call_main.h:58 #4 0x0000762446c2a28b in __libc_start_main_impl (main=0x55a03c457880, argc=1, argv=0x7ffcf2cd0f18, init=<optimised out>, fini=<optimised out>, rtld_fini=<optimised out>, stack_end=0x7ffcf2cd0f08) at ../csu/libc-start.c:360 #5 0x000055a03c45a485 in ?? () [Inferior 1 (process 42078) detached] PROCESS 42079 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND _apt 42079 0.0 0.1 29496 16348 ? S Jun02 0:00 /usr/lib/apt/methods/https [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Download failed: Invalid argument. Continuing without source file ./misc/../sysdeps/unix/sysv/linux/select.c. 0x0000706503d26c3e in __GI___select (nfds=1, readfds=0x7ffd2b268600, writefds=0x0, exceptfds=0x0, timeout=0x0) at ../sysdeps/unix/sysv/linux/select.c:69 Thread 1 (Thread 0x706503e19a80 (LWP 42079) "https"): #0 0x0000706503d26c3e in __GI___select (nfds=1, readfds=0x7ffd2b268600, writefds=0x0, exceptfds=0x0, timeout=0x0) at ../sysdeps/unix/sysv/linux/select.c:69 #1 0x000070650476138c in WaitFd(int, bool, unsigned long) () from /lib/x86_64-linux-gnu/libapt-pkg.so.6.0 #2 0x000059cb8a464c97 in ?? () #3 0x0000706503c2a1ca in __libc_start_call_main (main=main@entry=0x59cb8a464880, argc=argc@entry=1, argv=argv@entry=0x7ffd2b268dc8) at ../sysdeps/nptl/libc_start_call_main.h:58 #4 0x0000706503c2a28b in __libc_start_main_impl (main=0x59cb8a464880, argc=1, argv=0x7ffd2b268dc8, init=<optimised out>, fini=<optimised out>, rtld_fini=<optimised out>, stack_end=0x7ffd2b268db8) at ../csu/libc-start.c:360 #5 0x000059cb8a467485 in ?? () [Inferior 1 (process 42079) detached] PROCESS 42782 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND _apt 42782 0.0 0.0 26792 15120 ? S Jun02 0:00 /usr/lib/apt/methods/store [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Download failed: Invalid argument. Continuing without source file ./misc/../sysdeps/unix/sysv/linux/select.c. 0x00007cfffa926c3e in __GI___select (nfds=1, readfds=0x7ffdc639fa00, writefds=0x0, exceptfds=0x0, timeout=0x0) at ../sysdeps/unix/sysv/linux/select.c:69 Thread 1 (Thread 0x7cfffaa32880 (LWP 42782) "store"): #0 0x00007cfffa926c3e in __GI___select (nfds=1, readfds=0x7ffdc639fa00, writefds=0x0, exceptfds=0x0, timeout=0x0) at ../sysdeps/unix/sysv/linux/select.c:69 #1 0x00007cfffb15d38c in WaitFd(int, bool, unsigned long) () from /lib/x86_64-linux-gnu/libapt-pkg.so.6.0 #2 0x00007cfffb1120f3 in pkgAcqMethod::Run(bool) () from /lib/x86_64-linux-gnu/libapt-pkg.so.6.0 #3 0x000057952d30a296 in ?? () #4 0x00007cfffa82a1ca in __libc_start_call_main (main=main@entry=0x57952d30a010, argc=argc@entry=1, argv=argv@entry=0x7ffdc639ff58) at ../sysdeps/nptl/libc_start_call_main.h:58 #5 0x00007cfffa82a28b in __libc_start_main_impl (main=0x57952d30a010, argc=1, argv=0x7ffdc639ff58, init=<optimised out>, fini=<optimised out>, rtld_fini=<optimised out>, stack_end=0x7ffdc639ff48) at ../csu/libc-start.c:360 #6 0x000057952d30a705 in ?? () [Inferior 1 (process 42782) detached] -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to apt in Ubuntu. https://bugs.launchpad.net/bugs/2109495 Title: unattended-upgrades can (and regularly does) pause mid-update indefinitely with no error or feedback Status in apt package in Ubuntu: New Bug description: unattended-upgrades can pause, probably forever, mid-upgrade. This prevents any further upgrades until the service is restarted. The direct cause is probably inside apt rather than the upgrade process itself, but the upgrade process has apparently no defence against apt hanging, either. An interactive user would see their command hang forever and would likely interrupt it, so the issue in apt is not a security problem (probably -- if it were missed, I guess it would hold the lock and prevent upgrades, too). I've observed this on both Debian and Ubuntu, on systems which suspend often, so I assume the issue is an infinite timeout somewhere in the networking code being hit because of a suspend operation occurring mid-update. It's hard to know how common it is, as it is not directly visible (see below). In this instance, the observed state was like this: $ ps aux|grep apt root 158770 0.0 0.0 2800 1536 ? Ss Mar24 0:00 /bin/sh /usr/lib/apt/apt.systemd.daily update root 158812 0.0 0.0 2800 1584 ? S Mar24 0:00 /bin/sh /usr/lib/apt/apt.systemd.daily lock_is_held update root 158857 0.0 0.0 22232 9876 ? S Mar24 2:00 apt-get -qq -y update _apt 158867 0.0 0.0 25440 7980 ? S Mar24 0:00 /usr/lib/apt/methods/http _apt 158868 0.0 0.0 25440 8072 ? S Mar24 0:00 /usr/lib/apt/methods/http _apt 158869 0.0 0.0 29468 12396 ? S Mar24 0:00 /usr/lib/apt/methods/https _apt 158870 0.0 0.0 29408 12424 ? S Mar24 0:00 /usr/lib/apt/methods/https _apt 159069 0.0 0.0 18280 5564 ? S Mar24 0:00 /usr/lib/apt/methods/gpgv _apt 159622 0.0 0.0 26792 6456 ? S Mar24 0:00 /usr/lib/apt/methods/store Note the date: I'd not had updates performed for over a month. I use the machine daily, but mostly suspend it overnight, so it's had multiple days of awake time since then, too. I recognise "apt/methods/http", so I think this is the common failure mode. There's no clues in the log files (they appear to most recently show a normal no-upgrades cycle on the 22nd). There are no errors reported to me as a user. Note that though I'm not on the default desktop environment, so it's possible the error reporting is broken due to me not starting a service which would have reported it, I do get popups telling me to restart applications when updates do run. The only feedback I have from the system is the suspicious absence of these requests to restart, and inability to run apt manually if I try. Error in this case: $ sudo apt update && sudo apt full-upgrade Reading package lists... Done E: Could not get lock /var/lib/apt/lists/lock. It is held by process 158857 (apt-get) N: Be aware that removing the lock file is not a solution and may break your system. E: Unable to lock directory /var/lib/apt/lists/ Killing some part of the tree usually wakes it back up again. I usually find the issue because I want to change packages or force an upgrade pre-reboot; this also means "fixing" the upgrade task prevents what I want to do as it the immediately starts doing upgrades, so I usually just stop the service fully and start again either manually or after reboot. * Probably the service ought to set a robust, real time, timeout on the entire upgrade process, or at least on the networking part of it. Even a 24h timeout would be sufficient to mitigate most of the damage. * Having a timeout in the apt code would probably be useful, too. * Having some feedback from unattended-upgrades when it has failed to run due to lock contention many times in a row would be good, too. ProblemType: Bug DistroRelease: Ubuntu 24.04 Package: unattended-upgrades 2.9.1+nmu4ubuntu1 ProcVersionSignature: Ubuntu 6.8.0-52.53-generic 6.8.12 Uname: Linux 6.8.0-52-generic x86_64 NonfreeKernelModules: zfs ApportVersion: 2.28.1-0ubuntu3.5 Architecture: amd64 CasperMD5CheckResult: pass Date: Mon Apr 28 11:00:20 2025 InstallationDate: Installed on 2025-01-17 (101 days ago) InstallationMedia: Ubuntu 24.04.1 LTS "Noble Numbat" - Release amd64 (20240827.1) PackageArchitecture: all RebootRequiredPkgs: Error: path contained symlinks. SourcePackage: unattended-upgrades UpgradeStatus: No upgrade log present (probably fresh install) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/2109495/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
