On Tue, 03 Nov 2015, Tianon Gravi wrote: > On 9 October 2015 at 21:11, Tianon Gravi <admwig...@gmail.com> wrote: > > Can you give the output of "docker info" ? My guess is an accidental switch > > of graph drivers (since that's the usual culprit for images disappearing as > > you describe).
> Yo Yoh, any luck figuring out what happened, especially whether my > hunch was correct? :) > #799386 is probably also of interest. :) Hi Tianon, Sorry that I have missed your original request for more info. uptime of that beast is 25 days so I guess may be it is not too late to provide docker info: $> docker info Containers: 5 Images: 240 Storage Driver: btrfs Build Version: Btrfs v3.17 Library Version: 101 Execution Driver: native-0.2 Kernel Version: 4.1.0-0.bpo.2-amd64 Operating System: Debian GNU/Linux 8 (jessie) CPUs: 16 Total Memory: 125.8 GiB Name: smaug ID: 3WTM:DRZA:PELN:YWYN:Y4DG:CV2Z:AIFY:OTIK:CAOG:AHSP:WSRX:X6EF WARNING: No memory limit support WARNING: No swap limit support I never figured out what has happened and meanwhile anyways switched to use btrfs as the storage driver in a different location (under btrfs volume) -- thus above info now reflects that :-/ Thanks for the pointer to possibly related #799386 which inspired me to do some archaeological exploration of the logs: reboot into 4.1.0-0.bpo... happend on Oct 09 22:49:14 and I don't see any "driver not supported" messages. But before that reboot I see a set of Oct 09 22:47:41 smaug systemd[1]: docker-22ba315ad486df48a588c9cfe3b37845109944f6e34f2cb65059ddbaa40a38e2.scope stopping timed out. Killing. and here is more of the log concerning docker around point of reboot: Oct 09 22:46:11 smaug docker[4058]: time="2015-10-09T22:46:11-04:00" level=info msg="+job release_interface(60e42bcbeb66b75b985b77edd2bf852fe9be5ec64160d75b545f4e5df31eb28b)" Oct 09 22:46:11 smaug docker[4058]: time="2015-10-09T22:46:11-04:00" level=info msg="-job release_interface(60e42bcbeb66b75b985b77edd2bf852fe9be5ec64160d75b545f4e5df31eb28b) = OK (0)" Oct 09 22:46:11 smaug docker[4058]: time="2015-10-09T22:46:11-04:00" level=info msg="+job release_interface(22ba315ad486df48a588c9cfe3b37845109944f6e34f2cb65059ddbaa40a38e2)" Oct 09 22:46:11 smaug docker[4058]: time="2015-10-09T22:46:11-04:00" level=info msg="-job release_interface(22ba315ad486df48a588c9cfe3b37845109944f6e34f2cb65059ddbaa40a38e2) = OK (0)" Oct 09 22:46:11 smaug docker[4058]: time="2015-10-09T22:46:11-04:00" level=info msg="+job release_interface(ae6eeb4963fff91dbb79dc5fc43e3d1f38113da8fad264474c5d3a5fa5b94ac9)" Oct 09 22:46:11 smaug docker[4058]: time="2015-10-09T22:46:11-04:00" level=info msg="-job release_interface(ae6eeb4963fff91dbb79dc5fc43e3d1f38113da8fad264474c5d3a5fa5b94ac9) = OK (0)" Oct 09 22:46:11 smaug docker[4058]: time="2015-10-09T22:46:11-04:00" level=info msg="+job release_interface(54ab9b8ed38d5dde513e3b09c5df3f71aa3842acfb93fcfe4c64f9b37cd2afe7)" Oct 09 22:46:11 smaug docker[4058]: time="2015-10-09T22:46:11-04:00" level=info msg="-job release_interface(54ab9b8ed38d5dde513e3b09c5df3f71aa3842acfb93fcfe4c64f9b37cd2afe7) = OK (0)" Oct 09 22:46:12 smaug kernel: docker0: port 4(veth78d7aab) entered disabled state Oct 09 22:46:12 smaug kernel: docker0: port 5(veth0d4d907) entered disabled state Oct 09 22:46:12 smaug kernel: docker0: port 3(veth939e845) entered disabled state Oct 09 22:46:12 smaug kernel: docker0: port 1(veth88edd9c) entered disabled state Oct 09 22:46:12 smaug kernel: docker0: port 1(veth88edd9c) entered disabled state Oct 09 22:46:12 smaug kernel: docker0: port 2(veth44cc24b) entered disabled state Oct 09 22:47:41 smaug systemd[1]: docker-22ba315ad486df48a588c9cfe3b37845109944f6e34f2cb65059ddbaa40a38e2.scope stopping timed out. Killing. Oct 09 22:47:41 smaug systemd[1]: Unit docker-22ba315ad486df48a588c9cfe3b37845109944f6e34f2cb65059ddbaa40a38e2.scope entered failed state. Oct 09 22:47:41 smaug systemd[1]: docker-ae6eeb4963fff91dbb79dc5fc43e3d1f38113da8fad264474c5d3a5fa5b94ac9.scope stopping timed out. Killing. Oct 09 22:47:41 smaug systemd[1]: Unit docker-ae6eeb4963fff91dbb79dc5fc43e3d1f38113da8fad264474c5d3a5fa5b94ac9.scope entered failed state. Oct 09 22:47:41 smaug systemd[1]: docker-60e42bcbeb66b75b985b77edd2bf852fe9be5ec64160d75b545f4e5df31eb28b.scope stopping timed out. Killing. Oct 09 22:47:41 smaug systemd[1]: Unit docker-60e42bcbeb66b75b985b77edd2bf852fe9be5ec64160d75b545f4e5df31eb28b.scope entered failed state. Oct 09 22:49:14 smaug kernel: Linux version 4.1.0-0.bpo.2-amd64 (debian-ker...@lists.debian.org) (gcc version 4.9.2 (Debian 4.9.2-10) ) #1 SMP Debian 4.1.6-1~bpo8+1 (2015-09-09) Oct 09 22:49:14 smaug kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-4.1.0-0.bpo.2-amd64 root=UUID=c7000eed-a2f2-47c4-8f9d-4adf61626173 ro console=ttyS1,115200n8 console=tty1 quiet Oct 09 22:49:14 smaug kernel: Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.1.0-0.bpo.2-amd64 root=UUID=c7000eed-a2f2-47c4-8f9d-4adf61626173 ro console=ttyS1,115200n8 console=tty1 quiet Oct 09 22:49:14 smaug kernel: usb usb1: Manufacturer: Linux 4.1.0-0.bpo.2-amd64 ehci_hcd Oct 09 22:49:14 smaug kernel: usb usb2: Manufacturer: Linux 4.1.0-0.bpo.2-amd64 ehci_hcd Oct 09 22:49:14 smaug kernel: usb usb3: Manufacturer: Linux 4.1.0-0.bpo.2-amd64 xhci-hcd Oct 09 22:49:14 smaug kernel: usb usb4: Manufacturer: Linux 4.1.0-0.bpo.2-amd64 xhci-hcd Oct 09 22:49:14 smaug kernel: CPU: 0 PID: 481 Comm: systemd-udevd Not tainted 4.1.0-0.bpo.2-amd64 #1 Debian 4.1.6-1~bpo8+1 Oct 09 22:49:39 smaug arno-iptables-firewall[1314]: Starting Arno's Iptables Firewall: arno-iptables-firewallWARNING: Internal interface docker0 does NOT exist (yet?) Oct 09 22:49:39 smaug arno-iptables-firewall[1314]: WARNING: Internal interface docker1 does NOT exist (yet?) Oct 09 22:49:39 smaug arno-iptables-firewall[1314]: WARNING: Internal interface docker2 does NOT exist (yet?) Oct 09 22:49:42 smaug smartd[2152]: smartd 6.4 2014-10-07 r4002 [x86_64-linux-4.1.0-0.bpo.2-amd64] (local build) Oct 09 22:49:43 smaug dnsmasq[2485]: warning: interface docker0 does not currently exist Oct 09 22:49:44 smaug docker[3981]: time="2015-10-09T22:49:44-04:00" level=info msg="+job serveapi(fd://, tcp://0.0.0.0:4243, unix:///var/run/docker.sock)" Oct 09 22:49:44 smaug docker[3981]: time="2015-10-09T22:49:44-04:00" level=info msg="Listening for HTTP on fd ()" Oct 09 22:49:44 smaug docker[3981]: time="2015-10-09T22:49:44-04:00" level=info msg="Listening for HTTP on tcp (0.0.0.0:4243)" Oct 09 22:49:44 smaug docker[3981]: time="2015-10-09T22:49:44-04:00" level=info msg="/!\\ DON'T BIND ON ANY IP ADDRESS WITHOUT setting -tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOING /!\\" Oct 09 22:49:44 smaug docker[3981]: time="2015-10-09T22:49:44-04:00" level=info msg="Listening for HTTP on unix (/var/run/docker.sock)" Oct 09 22:49:46 smaug docker[3981]: time="2015-10-09T22:49:46-04:00" level=warning msg="Graphdriver devicemapper selected. Your graphdriver directory /var/lib/docker already contains data managed by other graphdrivers: aufs" Oct 09 22:49:46 smaug docker[3981]: time="2015-10-09T22:49:46-04:00" level=info msg="+job init_networkdriver()" Oct 09 22:49:46 smaug avahi-daemon[2324]: Joining mDNS multicast group on interface docker0.IPv4 with address 10.0.42.1. Oct 09 22:49:46 smaug avahi-daemon[2324]: New relevant interface docker0.IPv4 for mDNS. Oct 09 22:49:46 smaug avahi-daemon[2324]: Registering new address record for 10.0.42.1 on docker0.IPv4. Oct 09 22:49:46 smaug docker[3981]: time="2015-10-09T22:49:46-04:00" level=info msg="-job init_networkdriver() = OK (0)" Oct 09 22:49:46 smaug docker[3981]: time="2015-10-09T22:49:46-04:00" level=warning msg="mountpoint for memory not found" Oct 09 22:49:46 smaug docker[3981]: time="2015-10-09T22:49:46-04:00" level=info msg="Loading containers: start." Oct 09 22:49:46 smaug docker[3981]: ...............................................................time="2015-10-09T22:49:46-04:00" level=error msg="Failed to load container c01b548d724e41d44edc70895f78aee1259674d749f63153e98bca54ce5f9777: open /var/lib/docker/containers/c01b548d724e41d44edc70895f78aee1259674d749f63153e98bca54ce5f9777/config.json: no such file or directory" Oct 09 22:49:46 smaug docker[3981]: ............................................................................................................. Oct 09 22:49:46 smaug docker[3981]: time="2015-10-09T22:49:46-04:00" level=info msg="Loading containers: done." Oct 09 22:49:46 smaug docker[3981]: time="2015-10-09T22:49:46-04:00" level=info msg="docker daemon: 1.6.2 7c8fca2; execdriver: native-0.2; graphdriver: devicemapper" Oct 09 22:49:46 smaug docker[3981]: time="2015-10-09T22:49:46-04:00" level=info msg="+job acceptconnections()" Oct 09 22:49:46 smaug docker[3981]: time="2015-10-09T22:49:46-04:00" level=info msg="-job acceptconnections() = OK (0)" Oct 09 22:49:46 smaug docker[3981]: time="2015-10-09T22:49:46-04:00" level=info msg="Daemon has completed initialization" Oct 09 22:49:46 smaug docker[3981]: time="2015-10-09T22:49:46-04:00" level=info msg="GET /v1.18/containers/json" Oct 09 22:49:46 smaug docker[3981]: time="2015-10-09T22:49:46-04:00" level=info msg="+job containers()" Oct 09 22:49:46 smaug docker[3981]: time="2015-10-09T22:49:46-04:00" level=info msg="-job containers() = OK (0)" Oct 09 22:49:46 smaug docker[3981]: time="2015-10-09T22:49:46-04:00" level=info msg="POST /v1.18/containers/create" Oct 09 22:49:46 smaug docker[3981]: time="2015-10-09T22:49:46-04:00" level=info msg="+job create()" Oct 09 22:49:46 smaug docker[3981]: No such image: datalad_pymvpa/buildbot:slave-docker-dl-nd70 (tag: slave-docker- complete output of journalctl --since "2015-10-09" --until "2015-10-10" | grep -e '4\.1\.0' -e docker is available from http://www.onerussian.com/tmp/journal-docker-20151009.txt if there would be more of relevant information of interest -- Yaroslav O. Halchenko Center for Open Neuroscience http://centerforopenneuroscience.org Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik
