Hi everyone, it has been one month since the last crash. I wanted to check if everything is ok today and right in to moment i tried to access systemd, it crashed. Unfortunately, it received an ABRT signal right after the SEGV, which had overwritten the core dump. So i couldn't figure out where the SEGV was comming from. I compared the ps faux of the crashed system with the rebooted one and there was no difference except PID 1 (which is usually /sbin/init). Since systemd was crashing right in the moment of the "systemctl" command, the origin of the SEGV should be systemd, right?
I don't know if the problem of these SEGV is still the same with version 221-1. At least the system is somewhat stable if i don't touch anything. Have a nice weekend Tino
login as: draghi
draghi@Storage-Blue's password:
Welcome to Storage-Blue!
Last login: Thu Aug 13 13:24:58 2015 from *******
draghi@Storage-Blue:~$ sudo systemctl
[sudo] password for draghi:
Broadcast message from systemd-journald@Storage-Blue (Sat 2015-08-22 12:15:20
CEST):
systemd[1]: Caught <SEGV>, dumped core as pid 7388.
Broadcast message from systemd-journald@Storage-Blue (Sat 2015-08-22 12:15:20
CEST):
systemd[1]: Freezing execution.
Broadcast message from systemd-journald@Storage-Blue (Sat 2015-08-22 12:15:20
CEST):
systemd[1]: Caught <ABRT>, dumped core as pid 7389.
Broadcast message from systemd-journald@Storage-Blue (Sat 2015-08-22 12:15:20
CEST):
systemd[1]: Freezing execution.
Message from syslogd@Storage-Blue at Aug 22 12:15:20 ...
systemd[1]: Caught <SEGV>, dumped core as pid 7388.
Message from syslogd@Storage-Blue at Aug 22 12:15:20 ...
systemd[1]: Freezing execution.
Message from syslogd@Storage-Blue at Aug 22 12:15:20 ...
systemd[1]: Caught <ABRT>, dumped core as pid 7389.
Message from syslogd@Storage-Blue at Aug 22 12:15:20 ...
systemd[1]: Freezing execution.
lines 1--1...skipping...
Failed to list units: Connection timed out
draghi@Storage-Blue:~$ uptime
12:26:15 up 33 days, 58 min, 1 user, load average: 0.11, 0.50, 0.35
draghi@Storage-Blue:~$ ls -la /core
-rw------- 1 root root 1359872 Aug 22 12:15 /core
$ sudo gdb /lib/systemd/systemd /core
[...]
[New LWP 7389]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabi/libthread_db.so.1".
Core was generated by `/lib/systemd/systemd --system --deserialize 18'.
Program terminated with signal SIGABRT, Aborted.
#0 0xb6ba0d2c in kill () at ../sysdeps/unix/syscall-template.S:81
81 ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) bt
#0 0xb6ba0d2c in kill () at ../sysdeps/unix/syscall-template.S:81
#1 0xb6f07740 in crash.lto_priv.246 (sig=6) at ../src/core/main.c:167
#2 <signal handler called>
#3 0xb6ba0aa4 in __GI_raise (sig=sig@entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:56
#4 0xb6ba1f30 in __GI_abort () at abort.c:89
#5 0xb6be4088 in __malloc_assert (assertion=<optimized out>, file=<optimized
out>, line=line@entry=2372, function=<optimized out>) at malloc.c:293
#6 0xb6be79b4 in sysmalloc (av=0xb6ca44d4 <main_arena>, nb=3066712212) at
malloc.c:2369
#7 _int_malloc (av=av@entry=0xb6ca44d4 <main_arena>, bytes=bytes@entry=32800)
at malloc.c:3800
#8 0xb6be9140 in __GI___libc_malloc (bytes=32800) at malloc.c:2891
#9 0xb6c0cb2c in __alloc_dir (fd=19, close_fd=<optimized out>,
flags=<optimized out>, statp=<optimized out>) at ../sysdeps/posix/opendir.c:207
#10 0xb6ef094c in close_all_fds (except=except@entry=0x0,
n_except=n_except@entry=0) at ../src/basic/util.c:1411
#11 0xb6ef0b38 in freeze () at ../src/basic/util.c:2173
#12 0xb6f074a8 in crash.lto_priv.246 (sig=<optimized out>) at
../src/core/main.c:222
#13 <signal handler called>
#14 0xb6f10274 in unit_following (u=0xb8f53b98) at ../src/core/unit.c:2981
#15 0xb6f183fc in list_units_filtered (message=message@entry=0xb8fb0450,
userdata=userdata@entry=0xb8f1a308, error=error@entry=0xbe93d0c4, states=0x0)
at ../src/core/dbus-manager.c:881
#16 0xb6f18764 in method_list_units_filtered (message=0xb8fb0450,
userdata=0xb8f1a308, error=0xbe93d0c4) at ../src/core/dbus-manager.c:934
#17 0xb6eeaa78 in method_callbacks_run (found_object=0xbe93d1ac,
require_fallback=false, c=0xb8faf5c8, m=0xb8fb0450, bus=0xb8f2a938)
at ../src/libsystemd/sd-bus/bus-objects.c:398
#18 object_find_and_run.lto_priv.270 (bus=0xb8f2a938, m=0xb8fb0450,
p=0xbe93d0c4 "", require_fallback=<optimized out>, found_object=0xbe93d1ac)
at ../src/libsystemd/sd-bus/bus-objects.c:1232
#19 0xb6ed0984 in bus_process_object (m=0xb8fb0450, bus=0xb8f2a938) at
../src/libsystemd/sd-bus/bus-objects.c:1348
#20 process_message (m=0xb8fb0450, bus=0xb8f2a938) at
../src/libsystemd/sd-bus/sd-bus.c:2513
#21 process_running (hint_priority=false, priority=0, ret=0x0, bus=0xb8f2a938)
at ../src/libsystemd/sd-bus/sd-bus.c:2555
#22 bus_process_internal.constprop.170 (bus=0xb8f2a938, ret=0x0, priority=0,
hint_priority=false) at ../src/libsystemd/sd-bus/sd-bus.c:2744
#23 0xb6ec7cb0 in sd_bus_process (ret=0x0, bus=<optimized out>) at
../src/libsystemd/sd-bus/sd-bus.c:2763
#24 io_callback (s=<optimized out>, fd=<optimized out>, revents=<optimized
out>, userdata=<optimized out>) at ../src/libsystemd/sd-bus/sd-bus.c:3033
#25 0xb6ecbee0 in source_dispatch.lto_priv.935 (s=0xb8fad860) at
../src/libsystemd/sd-event/sd-event.c:2116
#26 0xb6f5728c in sd_event_dispatch (e=0xb8f1a760) at
../src/libsystemd/sd-event/sd-event.c:2472
#27 sd_event_run (timeout=<optimized out>, e=0xb8f1a760) at
../src/libsystemd/sd-event/sd-event.c:2500
#28 manager_loop (m=0xb8f1a308) at ../src/core/manager.c:2046
#29 0xb6e9955c in main (argc=<optimized out>, argv=<optimized out>) at
../src/core/main.c:1750
$ dpkg -l | grep systemd
ii libsystemd0:armel 221-1+deb9u2 armel
systemd utility library
ii systemd 221-1+deb9u2 armel
system and service manager
ii systemd-dbg 221-1+deb9u2 armel
system and service manager (debug symbols)
ii systemd-sysv 221-1 armel
system and service manager - SysV links
$ sudo journalctl
[...]
Aug 22 11:50:18 Storage-Blue systemd[1]: Starting Cleanup of Temporary
Directories...
Aug 22 11:50:18 Storage-Blue systemd[1]: Started Cleanup of Temporary
Directories.
Aug 22 12:15:20 Storage-Blue systemd[1]: Caught <SEGV>, dumped core as pid 7388.
Aug 22 12:15:20 Storage-Blue systemd[1]: Freezing execution.
Aug 22 12:15:20 Storage-Blue systemd[1]: Caught <ABRT>, dumped core as pid 7389.
Aug 22 12:15:20 Storage-Blue systemd[1]: Freezing execution.
[...]
$ sudo ps faux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 2 0.0 0.0 0 0 ? S Jul20 0:02 [kthreadd]
root 3 0.0 0.0 0 0 ? S Jul20 6:55 \_
[ksoftirqd/0]
root 5 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[kworker/0:0H]
root 7 0.0 0.0 0 0 ? S Jul20 0:14 \_
[watchdog/0]
root 8 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [khelper]
root 9 0.0 0.0 0 0 ? S Jul20 0:00 \_ [kdevtmpfs]
root 10 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [netns]
root 11 0.0 0.0 0 0 ? S Jul20 0:00 \_
[khungtaskd]
root 12 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [writeback]
root 13 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [crypto]
root 14 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[kintegrityd]
root 15 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [bioset]
root 16 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [kblockd]
root 18 0.0 0.0 0 0 ? S Jul20 6:17 \_ [kswapd0]
root 19 0.0 0.0 0 0 ? S Jul20 0:00 \_
[fsnotify_mark]
root 25 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [kthrotld]
root 26 0.0 0.0 0 0 ? S Jul20 4:33 \_ [spi0]
root 27 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [deferwq]
root 64 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [ata_sff]
root 65 0.0 0.0 0 0 ? S Jul20 0:00 \_ [scsi_eh_0]
root 66 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[scsi_tmf_0]
root 67 0.0 0.0 0 0 ? S Jul20 0:00 \_ [scsi_eh_1]
root 68 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[scsi_tmf_1]
root 69 0.0 0.0 0 0 ? S Jul20 0:00 \_ [scsi_eh_2]
root 70 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[scsi_tmf_2]
root 71 0.0 0.0 0 0 ? S Jul20 0:00 \_ [scsi_eh_3]
root 72 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[scsi_tmf_3]
root 73 0.0 0.0 0 0 ? S Jul20 0:00 \_ [scsi_eh_4]
root 75 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[scsi_tmf_4]
root 77 0.0 0.0 0 0 ? S Jul20 0:00 \_ [scsi_eh_5]
root 79 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[scsi_tmf_5]
root 88 0.0 0.0 0 0 ? S< Jul20 40:11 \_
[kworker/0:1H]
root 94 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [md]
root 98 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [bioset]
root 102 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [bioset]
root 132 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [bioset]
root 150 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[btrfs-worker]
root 152 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[btrfs-worker-hi]
root 153 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[btrfs-delalloc]
root 154 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[btrfs-flush_del]
root 155 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[btrfs-cache]
root 156 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[btrfs-submit]
root 157 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[btrfs-fixup]
root 158 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[btrfs-endio]
root 159 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[btrfs-endio-met]
root 160 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[btrfs-endio-met]
root 161 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[btrfs-endio-rai]
root 162 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [btrfs-rmw]
root 163 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[btrfs-endio-wri]
root 164 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[btrfs-freespace]
root 165 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[btrfs-delayed-m]
root 166 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[btrfs-readahead]
root 167 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[btrfs-qgroup-re]
root 168 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[btrfs-extent-re]
root 169 0.0 0.0 0 0 ? S Jul20 0:00 \_
[btrfs-cleaner]
root 170 0.0 0.0 0 0 ? S Jul20 0:41 \_
[btrfs-transacti]
root 190 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[ipv6_addrconf]
root 212 0.0 0.0 0 0 ? S Jul20 0:00 \_ [kauditd]
root 387 0.0 0.0 0 0 ? S Jul20 0:00 \_ [khubd]
root 410 0.0 0.0 0 0 ? S Jul20 0:02 \_ [mv_crypto]
root 539 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [kdmflush]
root 575 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [bioset]
root 576 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[kcryptd_io]
root 578 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [kcryptd]
root 579 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [bioset]
root 604 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [kdmflush]
root 607 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [bioset]
root 609 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[kcryptd_io]
root 612 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [kcryptd]
root 613 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [bioset]
root 614 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [kdmflush]
root 615 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [bioset]
root 616 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[kcryptd_io]
root 618 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [kcryptd]
root 620 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [bioset]
root 650 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [kdmflush]
root 651 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [bioset]
root 652 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[kcryptd_io]
root 653 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [kcryptd]
root 654 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [bioset]
root 911 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [rpciod]
root 913 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [nfsiod]
root 24638 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [kdmflush]
root 24639 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [bioset]
root 24640 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[kcryptd_io]
root 24641 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [kcryptd]
root 24642 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [bioset]
root 24652 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [kdmflush]
root 24653 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [bioset]
root 24654 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[kcryptd_io]
root 24655 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [kcryptd]
root 24656 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [bioset]
root 24669 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[btrfs-worker]
root 24670 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[btrfs-worker-hi]
root 24671 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[btrfs-delalloc]
root 24672 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[btrfs-flush_del]
root 24673 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[btrfs-cache]
root 24674 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[btrfs-submit]
root 24675 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[btrfs-fixup]
root 24676 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[btrfs-endio]
root 24677 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[btrfs-endio-met]
root 24678 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[btrfs-endio-met]
root 24679 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[btrfs-endio-rai]
root 24680 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [btrfs-rmw]
root 24681 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[btrfs-endio-wri]
root 24682 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[btrfs-freespace]
root 24683 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[btrfs-delayed-m]
root 24684 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[btrfs-readahead]
root 24685 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[btrfs-qgroup-re]
root 24686 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[btrfs-extent-re]
root 24687 0.0 0.0 0 0 ? S Jul20 0:01 \_
[btrfs-cleaner]
root 24688 0.1 0.0 0 0 ? S Jul20 71:10 \_
[btrfs-transacti]
root 24725 0.0 0.0 0 0 ? S< Jul20 0:00 \_ [nfsd4]
root 24726 0.0 0.0 0 0 ? S< Jul20 0:00 \_
[nfsd4_callbacks]
root 24727 0.0 0.0 0 0 ? S Jul20 0:00 \_ [lockd]
root 24730 0.0 0.0 0 0 ? S Jul20 39:39 \_ [nfsd]
root 15694 0.0 0.0 0 0 ? S< Aug11 0:00 \_
[kworker/u3:0]
root 6246 0.0 0.0 0 0 ? S< 01:20 0:13 \_
[kworker/u3:2]
root 6948 0.2 0.0 0 0 ? S 06:50 0:48 \_
[kworker/0:1]
root 7131 0.2 0.0 0 0 ? S 09:11 0:28 \_
[kworker/0:0]
root 7281 0.2 0.0 0 0 ? S 11:11 0:10 \_
[kworker/0:2]
root 7322 0.0 0.0 0 0 ? S 11:38 0:00 \_
[kworker/u2:4]
root 7334 0.0 0.0 0 0 ? S 11:47 0:00 \_
[kworker/u2:1]
root 7356 0.0 0.0 0 0 ? S 12:02 0:00 \_
[kworker/u2:2]
root 7362 0.3 0.0 0 0 ? S 12:08 0:01 \_
[kworker/0:3]
root 7364 0.0 0.0 0 0 ? S 12:08 0:00 \_
[kworker/u2:0]
root 7369 0.0 0.0 0 0 ? S 12:13 0:00 \_
[kworker/0:4]
root 7373 0.0 0.0 0 0 ? S 12:14 0:00 \_
[kworker/u2:3]
root 1 0.0 0.5 6296 2932 ? Ss Jul20 1:23
/lib/systemd/systemd --system --deserialize 18
root 225 0.0 0.5 8408 3048 ? Ss Jul20 0:30
/lib/systemd/systemd-journald
root 775 0.0 0.7 35376 3940 ? Ss Jul20 1:58 /usr/sbin/smbd
-D
root 788 0.0 0.1 8596 968 ? Ss Jul20 0:00 dhclient -v
-pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
root 833 0.0 0.1 8592 700 ? Ss Jul20 0:00 dhclient -v
-pf /run/dhclient.eth1.pid -lf /var/lib/dhcp/dhclient.eth1.leases eth1
root 896 0.0 0.2 4356 1064 ? Ss Jul20 0:04 /sbin/rpcbind
-w
statd 905 0.0 0.2 4544 1048 ? Ss Jul20 0:00 /sbin/rpc.statd
root 920 0.0 0.1 2812 956 ? Ss Jul20 0:00
/usr/sbin/rpc.idmapd
root 922 0.0 0.3 5340 1736 ? Ss Jul20 0:00
/usr/sbin/smartd -n --interval=31536000
message+ 925 0.0 0.3 6768 1876 ? Ss Jul20 0:03
/usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile
--systemd-activation
root 928 0.0 0.2 4044 1052 ? Ss Jul20 0:00 /usr/bin/rsync
--daemon --no-detach
root 929 0.0 0.4 31276 2216 ? Ssl Jul20 0:06
/usr/sbin/rsyslogd -n
root 934 0.0 0.2 4980 1220 ? Ss Jul20 0:04 /usr/sbin/cron
-f
root 947 0.0 0.3 7832 1996 ? Ss Jul20 0:00 /usr/sbin/sshd
-D
root 7372 0.1 0.9 11568 4996 ? Ss 12:14 0:00 \_ sshd:
draghi [priv]
draghi 7375 0.0 0.8 11568 4204 ? S 12:14 0:00 \_ sshd:
draghi@pts/0
draghi 7376 0.2 0.8 6100 4168 pts/0 Ss 12:14 0:00 \_
-bash
root 7396 0.0 0.6 6316 3096 pts/0 S+ 12:17 0:00
\_ sudo ps faux
root 7397 0.0 0.4 4760 2272 pts/0 R+ 12:17 0:00
\_ ps faux
mail 955 0.0 0.2 3488 1064 ? S Jul20 0:00
/usr/sbin/nullmailer-send -s
root 970 0.0 0.2 28452 1104 ? Sl Jul20 1:33 qcontrol -f
root 992 0.0 0.3 22556 1968 ? Ss Jul20 0:47 /usr/sbin/nmbd
-D
root 21056 0.0 0.2 12292 1140 ? Ss Jul20 0:09
/lib/systemd/systemd-udevd
root 24531 0.0 0.2 3572 1424 ? Ss Jul20 0:10
/lib/systemd/systemd-logind
root 24705 0.0 0.1 2468 640 ? Ss Jul20 0:00
/usr/sbin/in.tftpd --listen --user tftp --address 0.0.0.0:69 --secure /srv/tftp
root 24733 0.0 0.2 5376 1372 ? Ss Jul20 0:16
/usr/sbin/rpc.mountd --manage-gids
signature.asc
Description: OpenPGP digital signature
