how do I debug this? if i manually run 'ifup -a --read-environment' or 'systemctl start networking' it brings all stuff up cleanly. by 'debug' i mean more than just looking at journalctl -u networking
below contents of network/interfaces, journalctl -u networking and contents of netwroking unit --------- [Unit] Description=Raise network interfaces Documentation=man:interfaces(5) DefaultDependencies=no Wants=network.target After=local-fs.target network-pre.target apparmor.service systemd-sysctl.service systemd-modules-load.service Before=network.target shutdown.target network-online.target Conflicts=shutdown.target [Install] WantedBy=multi-user.target WantedBy=network-online.target [Service] Type=oneshot EnvironmentFile=-/etc/default/networking ExecStartPre=-/bin/sh -c '[ "$CONFIGURE_INTERFACES" != "no" ] && [ -n "$(ifquery --read-environment --list --exclude=lo)" ] && udevadm settle' ExecStart=/sbin/ifup -a --read-environment ExecStop=/sbin/ifdown -a --read-environment --exclude=lo RemainAfterExit=true TimeoutStartSec=3sec #modified from default which just made the machine hang on boot until timeout ------ # This file describes the network interfaces available on your system # and how to activate them. For more information, see interfaces(5). source /etc/network/interfaces.d/* # The loopback network interface auto lo iface lo inet loopback # The primary network interface #allow-hotplug eno1 #allow-hotplug ipsec0 auto eno1 auto ipsec0 iface eno1 inet manual up ip link set eno1 up iface ipsec0 inet static bridge_ports eno1 address 192.168.15.113 netmask 255.255.255.0 gateway 192.168.15.254 # dns-* options are implemented by the resolvconf package, if installed dns-nameservers 8.8.8.8 #link-speed 1000 #link-duplex full #ethernet-autoneg off #dns-search project.local #up dnsmasq -s lan -S //192.168.15.113 -u root --strict-order --pid-file=/run/dnsmasq-lan.pid --dhcp-no-override --except-interface=lo --interface=ipsec0 --dhcp-leasefile=/var/lib/dnsmasq.lan.leases --dhcp-authoritative --listen-address 192.168.15.113 --dhcp-range 192.168.15.101,192.168.15.199 --dhcp-lease-max=252 --bind-interfaces --dhcp-option-force=option:router,192.168.15.254 --dhcp-option-force=option:ntp-server,192.168.15.113 up dnsmasq -s project -S //192.168.15.113 -u root --strict-order --pid-file=/run/dnsmasq-project.pid --dhcp-no-override --except-interface=lo --interface=ipsec0 --dhcp-leasefile=/var/lib/dnsmasq.project.leases --dhcp-authoritative --listen-address 192.168.15.113 --dhcp-range 192.168.15.101,192.168.15.199 --dhcp-lease-max=252 --bind-interfaces --dhcp-option-force=option:router,192.168.15.254 --dhcp-option-force=option:ntp-server,192.168.15.113 down kill $(cat /run/dnsmasq-project.pid) #allow-hotplug eno2 auto eno2 iface eno2 inet static address 192.168.100.100 netmask 255.255.255.0 #link-speed 100 #link-duplex full #ethernet-autoneg off #gateway # dns-* options are implemented by the resolvconf package, if installed #dns-nameservers 10.10.200.10 #dns-search project.local ------------- -- Logs begin at Thu 2017-02-02 17:34:56 EET, end at Thu 2017-02-02 18:25:01 EET. -- Feb 02 17:35:13 linux systemd[1]: Starting Raise network interfaces... Feb 02 17:35:13 linux ifup[6077]: /bin/run-parts --exit-on-error --verbose /etc/network/if-pre-up.d Feb 02 17:35:13 linux ifup[6077]: run-parts: executing /etc/network/if-pre-up.d/bridge Feb 02 17:35:13 linux ifup[6077]: run-parts: executing /etc/network/if-pre-up.d/ethtool Feb 02 17:35:13 linux ifup[6077]: run-parts: executing /etc/network/if-pre-up.d/hostapd Feb 02 17:35:13 linux ifup[6077]: run-parts: executing /etc/network/if-pre-up.d/wireless-tools Feb 02 17:35:13 linux ifup[6077]: run-parts: executing /etc/network/if-pre-up.d/wpasupplicant Feb 02 17:35:13 linux ifup[6077]: Configuring interface lo=lo (inet) Feb 02 17:35:13 linux ifup[6077]: /bin/run-parts --exit-on-error --verbose /etc/network/if-pre-up.d Feb 02 17:35:13 linux ifup[6077]: run-parts: executing /etc/network/if-pre-up.d/bridge Feb 02 17:35:13 linux ifup[6077]: run-parts: executing /etc/network/if-pre-up.d/ethtool Feb 02 17:35:13 linux ifup[6077]: run-parts: executing /etc/network/if-pre-up.d/hostapd Feb 02 17:35:13 linux ifup[6077]: run-parts: executing /etc/network/if-pre-up.d/wireless-tools Feb 02 17:35:13 linux ifup[6077]: run-parts: executing /etc/network/if-pre-up.d/wpasupplicant Feb 02 17:35:13 linux ifup[6077]: /bin/ip link set dev lo up Feb 02 17:35:13 linux ifup[6077]: /bin/run-parts --exit-on-error --verbose /etc/network/if-up.d Feb 02 17:35:13 linux ifup[6077]: run-parts: executing /etc/network/if-up.d/000resolvconf Feb 02 17:35:13 linux ifup[6077]: run-parts: executing /etc/network/if-up.d/arping Feb 02 17:35:13 linux ifup[6077]: run-parts: executing /etc/network/if-up.d/ethtool Feb 02 17:35:13 linux ifup[6077]: run-parts: executing /etc/network/if-up.d/ntpdate Feb 02 17:35:13 linux ifup[6077]: run-parts: executing /etc/network/if-up.d/openssh-server Feb 02 17:35:13 linux ifup[6077]: run-parts: executing /etc/network/if-up.d/upstart Feb 02 17:35:13 linux ifup[6077]: run-parts: executing /etc/network/if-up.d/wpasupplicant Feb 02 17:35:13 linux ifup[6077]: /sbin/ifup: waiting for lock on /run/network/ifstate.eno1 Feb 02 17:35:13 linux ifup[6077]: Configuring interface ipsec0=ipsec0 (inet) Feb 02 17:35:13 linux ifup[6077]: /bin/run-parts --exit-on-error --verbose /etc/network/if-pre-up.d Feb 02 17:35:13 linux ifup[6077]: run-parts: executing /etc/network/if-pre-up.d/bridge Feb 02 17:35:14 linux ifup[6077]: Waiting for ipsec0 to get ready (MAXWAIT is 32 seconds). Feb 02 17:35:17 linux ifup[6077]: run-parts: executing /etc/network/if-pre-up.d/ethtool Feb 02 17:35:17 linux ifup[6077]: run-parts: executing /etc/network/if-pre-up.d/hostapd Feb 02 17:35:17 linux ifup[6077]: run-parts: executing /etc/network/if-pre-up.d/wireless-tools Feb 02 17:35:17 linux ifup[6077]: run-parts: executing /etc/network/if-pre-up.d/wpasupplicant Feb 02 17:35:17 linux ifup[6077]: /bin/ip addr add 192.168.15.113/255.255.255.0 broadcast 192.168.15.255 dev ipsec0 label ipsec0 Feb 02 17:35:17 linux ifup[6077]: /bin/ip link set dev ipsec0 up Feb 02 17:35:17 linux ifup[6077]: /bin/ip route add default via 192.168.15.254 dev ipsec0 onlink Feb 02 17:35:17 linux ifup[6077]: dnsmasq -s cdsl -S //192.168.15.113 -u root --strict-order --pid-file=/run/dnsmasq-cdsl.pid --dhcp-no-override --except-interface=lo --interface=ipsec0 --dhcp-leasefile=/var/lib/dnsmasq.cdsl.leases --dhcp-authoritative --listen-address 192.168.15.113 --dhcp-range 192.168.15.101,192.168.15.199 --dhcp-lease-max=252 --bind-interfaces --dhcp-option-force=option:router,192.168.15.254 --dhcp-option-force=option:ntp-server,192.168.15.113 & Feb 02 17:35:17 linux ifup[6077]: /bin/run-parts --exit-on-error --verbose /etc/network/if-up.d Feb 02 17:35:17 linux ifup[6077]: run-parts: executing /etc/network/if-up.d/000resolvconf Feb 02 17:35:17 linux dnsmasq[6679]: started, version 2.75 cachesize 150 Feb 02 17:35:17 linux dnsmasq[6679]: compile time options: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset auth DNSSEC loop-detect inotify Feb 02 17:35:17 linux dnsmasq-dhcp[6679]: DHCP, IP range 192.168.15.101 -- 192.168.15.199, lease time 1h Feb 02 17:35:17 linux dnsmasq-dhcp[6679]: DHCP, sockets bound exclusively to interface ipsec0 Feb 02 17:35:17 linux dnsmasq[6679]: ignoring nameserver 192.168.15.113 - local interface Feb 02 17:35:17 linux dnsmasq[6679]: reading /etc/resolv.conf Feb 02 17:35:17 linux dnsmasq[6679]: using nameserver 127.0.0.1#53 Feb 02 17:35:17 linux dnsmasq[6679]: read /etc/hosts - 5 addresses Feb 02 17:35:23 linux systemd[1]: networking.service: Start operation timed out. Terminating. Feb 02 17:35:23 linux systemd[1]: Failed to start Raise network interfaces. Feb 02 17:35:23 linux ifup[6077]: Terminated Feb 02 17:35:23 linux dnsmasq[6679]: exiting on receipt of SIGTERM Feb 02 17:35:23 linux systemd[1]: networking.service: Unit entered failed state. Feb 02 17:35:23 linux systemd[1]: networking.service: Failed with result 'timeout'. then it is started immediately from rc.local: "systemctl start networking" Feb 02 17:35:24 linux systemd[1]: Starting Raise network interfaces... Feb 02 17:35:24 linux ifup[6916]: /bin/run-parts --exit-on-error --verbose /etc/network/if-pre-up.d Feb 02 17:35:24 linux ifup[6916]: run-parts: executing /etc/network/if-pre-up.d/bridge Feb 02 17:35:24 linux ifup[6916]: run-parts: executing /etc/network/if-pre-up.d/ethtool Feb 02 17:35:24 linux ifup[6916]: run-parts: executing /etc/network/if-pre-up.d/hostapd Feb 02 17:35:24 linux ifup[6916]: run-parts: executing /etc/network/if-pre-up.d/wireless-tools Feb 02 17:35:24 linux ifup[6916]: run-parts: executing /etc/network/if-pre-up.d/wpasupplicant Feb 02 17:35:24 linux ifup[6916]: /bin/run-parts --exit-on-error --verbose /etc/network/if-up.d Feb 02 17:35:24 linux ifup[6916]: run-parts: executing /etc/network/if-up.d/000resolvconf Feb 02 17:35:24 linux ifup[6916]: run-parts: executing /etc/network/if-up.d/arping Feb 02 17:35:24 linux ifup[6916]: run-parts: executing /etc/network/if-up.d/ethtool Feb 02 17:35:24 linux ifup[6916]: run-parts: executing /etc/network/if-up.d/ntpdate Feb 02 17:35:24 linux ifup[6916]: run-parts: executing /etc/network/if-up.d/openssh-server Feb 02 17:35:24 linux ifup[6916]: run-parts: executing /etc/network/if-up.d/upstart Feb 02 17:35:24 linux ifup[6916]: run-parts: executing /etc/network/if-up.d/wpasupplicant Feb 02 17:35:24 linux systemd[1]: Started Raise network interfaces. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to ifupdown in Ubuntu. https://bugs.launchpad.net/bugs/1551415 Title: systemctl stop networking hang / timeout Status in ifupdown package in Ubuntu: Confirmed Bug description: I noticed today on lxd that the latest xenial image is failing to bring down networking $ cat /etc/cloud/build.info build_name: server serial: 20160227-141431 to reproduce: $ lxd-images import ubuntu --alias xenial xenial --stream=daily $ lxc launch xenial xtest # wait a bit $ lxc exec xtest /bin/bash % systemctl stop networking ... long time passes .... % systemctl status networking ● networking.service - Raise network interfaces Loaded: loaded (/lib/systemd/system/networking.service; enabled; vendor preset: enabled) Drop-In: /run/systemd/generator/networking.service.d └─50-insserv.conf-$network.conf Active: failed (Result: timeout) since Mon 2016-02-29 20:28:14 UTC; 5s ago Docs: man:interfaces(5) Process: 1241 ExecStop=/sbin/ifdown -a --read-environment (code=exited, status=0/SUCCESS) Process: 951 ExecStart=/sbin/ifup -a --read-environment (code=exited, status=0/SUCCESS) Process: 948 ExecStartPre=/bin/sh -c [ "$CONFIGURE_INTERFACES" != "no" ] && [ -n "$(ifquery --read-environme Main PID: 951 (code=exited, status=0/SUCCESS) Feb 29 20:26:35 xtest dhclient[992]: DHCPOFFER of 10.0.3.238 from 10.0.3.1 Feb 29 20:26:35 xtest dhclient[992]: DHCPACK of 10.0.3.238 from 10.0.3.1 Feb 29 20:26:35 xtest ifup[951]: DHCPACK of 10.0.3.238 from 10.0.3.1 Feb 29 20:26:35 xtest ifup[951]: bound to 10.0.3.238 -- renewal in 1669 seconds. Feb 29 20:26:35 xtest systemd[1]: Started Raise network interfaces. Feb 29 20:26:44 xtest systemd[1]: Stopping Raise network interfaces... Feb 29 20:28:14 xtest systemd[1]: networking.service: State 'stop-sigterm' timed out. Killing. Feb 29 20:28:14 xtest systemd[1]: Stopped Raise network interfaces. Feb 29 20:28:14 xtest systemd[1]: networking.service: Unit entered failed state. Feb 29 20:28:14 xtest systemd[1]: networking.service: Failed with result 'timeout'. root@xtest:~# ubuntu-bug /lib/systemd/system/networking.service Related bugs: * bug 1551351: dhclient does not renew leases ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: ifupdown 0.8.10ubuntu1 ProcVersionSignature: Ubuntu 4.4.0-8.23-generic 4.4.2 Uname: Linux 4.4.0-8-generic x86_64 ApportVersion: 2.20-0ubuntu3 Architecture: amd64 Date: Mon Feb 29 20:28:28 2016 ProcEnviron: TERM=xterm-256color PATH=(custom, no user) SourcePackage: ifupdown UpgradeStatus: No upgrade log present (probably fresh install) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ifupdown/+bug/1551415/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp