This seems to be a timing issue between network coming up and kdump-tools service trying to access the remote target with ssh because when I get into emergency shell after failure, this is what I get to see..
--- [FAILED] Failed to start Kernel crash dump capture service. See 'systemctl status kdump-tools.service' for details. Welcome to emergency mode! After logging in, type "journalctl -xb" to view system logs, "systemctl reboot" to reboot, "systemctl default" or ^D to try again to boot into default mode. Give root password for maintenance (or press Control-D to continue): root@pinelp2:~# ip addr show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 56:2d:60:47:c4:02 brd ff:ff:ff:ff:ff:ff 3: enP257p80s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq portid 40f2e93188c0 state UP group default qlen 1000 link/ether 40:f2:e9:31:88:c0 brd ff:ff:ff:ff:ff:ff inet 10.33.11.58/16 brd 10.33.255.255 scope global enP257p80s0f0 valid_lft forever preferred_lft forever inet6 2002:903:15f:1130:89c:e5b:6d8c:753f/64 scope global temporary dynamic valid_lft 604746sec preferred_lft 85746sec inet6 2002:903:15f:1130:42f2:e9ff:fe31:88c0/64 scope global mngtmpaddr dynamic valid_lft 2591946sec preferred_lft 604746sec inet6 2002:926:3e2:1130:89c:e5b:6d8c:753f/64 scope global temporary dynamic valid_lft 604746sec preferred_lft 85746sec inet6 2002:926:3e2:1130:42f2:e9ff:fe31:88c0/64 scope global mngtmpaddr dynamic valid_lft 2591946sec preferred_lft 604746sec inet6 fe80::42f2:e9ff:fe31:88c0/64 scope link valid_lft forever preferred_lft forever 4: enP257p80s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop portid 40f2e93188c2 state DOWN group default qlen 1000 link/ether 40:f2:e9:31:88:c2 brd ff:ff:ff:ff:ff:ff root@pinelp2:~# hostname -I 10.33.11.58 2002:903:15f:1130:89c:e5b:6d8c:753f 2002:903:15f:1130:42f2:e9ff:fe31:88c0 2002:926:3e2:1130:89c:e5b:6d8c:753f 2002:926:3e2:1130:42f2:e9ff:fe31:88c0 root@pinelp2:~# ping 10.33.31.113 PING 10.33.31.113 (10.33.31.113) 56(84) bytes of data. 64 bytes from 10.33.31.113: icmp_seq=1 ttl=64 time=0.399 ms 64 bytes from 10.33.31.113: icmp_seq=2 ttl=64 time=0.239 ms 64 bytes from 10.33.31.113: icmp_seq=3 ttl=64 time=0.234 ms --- 10.33.31.113 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 1998ms rtt min/avg/max/mdev = 0.234/0.290/0.399/0.079 ms root@pinelp2:~# root@pinelp2:~# root@pinelp2:~# root@pinelp2:~# ssh -i /root/.ssh/kdump_id_rsa root@10.33.31.113 mkdir -p /var/crash/10.33.11.58-test root@pinelp2:~# --- Most importantly, ssh command to create dir on remote target succeeds while that is what has failed leading to the below error message in the first place "Starting kdump-tools: ssh: connect to host 10.33.31.113 port 22: Network is unreachable" Looking into ways to fix that up. Thanks Hari -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to makedumpfile in Ubuntu. https://bugs.launchpad.net/bugs/1571590 Title: pVM:pinelp2:ubuntu 16.04: Network is unreachable when using ssh in kdump Status in makedumpfile package in Ubuntu: New Bug description: Problem Description ============================ When dumping to ssh target , kdump failed: Starting Kernel crash dump capture service... [ 18.542292] kdump-tools[2748]: Starting kdump-tools: ssh: connect to host 10.33.31.113 port 22: Network is unreachable [ 18.542894] kdump-tools[2748]: * kdump-tools: Unable to reach remote server root@10.33.31.113. No reason to continue [ 18.613376] kdump-tools[2748]: Mon, 11 Apr 2016 01:23:55 -0500 [ 19.310893] kdump-tools[2748]: Rebooting. [ 19.652714] reboot: Restarting system But the weired thing is that nfs target works on the same system. ---uname output--- Linux pinelp2 4.4.0-18-generic #34-Ubuntu SMP Wed Apr 6 14:00:30 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux Steps to Reproduce =================================== 1. install latest ubuntu 16.04 on pinelp2 2. config kdump to use ssh: SSH="root@10.33.31.113" 3. run "kdump-config propagate" as root 4. trigger kdump Userspace tool common name: kdump-tools The userspace tool has the following bit modes: 64-bit Userspace rpm: version 1:1.5.9-5 == Comment: #8 - Ping Tian Han <pt...@cn.ibm.com> - 2016-04-17 21:58:52 == (In reply to comment #6) > I am able to reproduce this issue as well. > Ping, what is you try to dump on some other machine with ssh ? Does it still > complain the same ? Is the ssh failing to connect to the network ? I saw the same problem when dumping to medlp4 with ssh: Starting Kernel crash dump capture service... [ 17.293028] kdump-tools[2733]: Starting kdump-tools: ssh: connect to host 10.33.7.181 port 22: Network is unreachable [ 17.293692] kdump-tools[2733]: * kdump-tools: Unable to reach remote server root@10.33.7.181. No reason to continue [ 17.405333] kdump-tools[2733]: Sun, 17 Apr 2016 20:56:59 -0500 [ 18.536587] kdump-tools[2733]: Rebooting. [ 18.903280] reboot: Restarting system but I can ssh log noto medlp4 before triggering kdump. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/makedumpfile/+bug/1571590/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
