Public bug reported:

This was observed while testing landscape autopilot to deploy an
openstack cluster. This uses juju deployed lxd containers to host
various openstack services. Creation of a container failed, causing the
entire openstack deployment to fail.

lxd: 2.0.9 (xenial)
juju: 1:2.1~rc2-0ubuntu1~16.04.1~juju1
landscape: 17.01~bzr10906+jenkins3476-2

I don't know if this is an issue with lxd, juju, or something else. From
what I can see, the container requested by juju failed to start due to
err="Container is already running a start operation". Juju then deleted
the container and tried again about a minute later. The retry was
successful.

Landscape errors out on the first failure and isn't aware that things
worked after the retry.

[from landscape-0-inner-logs/base-machine-2/var/log/lxd/lxd.log]
ephemeral=false lvl=info msg="Creating container" name=juju-2c21a7-1-lxd-4 
t=2017-02-18T14:04:57+0000
ephemeral=false lvl=info msg="Created container" name=juju-2c21a7-1-lxd-4 
t=2017-02-18T14:04:57+0000
action=start creation date=2017-02-18T14:04:57+0000 ephemeral=false lvl=info 
msg="Starting container" name=juju-2c21a7-1-lxd-4 stateful=false 
t=2017-02-18T14:05:27+0000
container=juju-2c21a7-1-lxd-4 err="Container is already running a start 
operation" lvl=eror msg="stop hook failed" t=2017-02-18T14:05:28+0000
action=start creation date=2017-02-18T14:04:57+0000 ephemeral=false lvl=eror 
msg="Failed starting container" name=juju-2c21a7-1-lxd-4 stateful=false 
t=2017-02-18T14:05:33+0000
creation date=2017-02-18T14:04:57+0000 ephemeral=false lvl=info msg="Deleting 
container" name=juju-2c21a7-1-lxd-4 t=2017-02-18T14:05:34+0000
creation date=2017-02-18T14:04:57+0000 ephemeral=false lvl=info msg="Deleted 
container" name=juju-2c21a7-1-lxd-4 t=2017-02-18T14:05:34+0000
ephemeral=false lvl=info msg="Creating container" name=juju-2c21a7-1-lxd-4 
t=2017-02-18T14:05:46+0000
ephemeral=false lvl=info msg="Created container" name=juju-2c21a7-1-lxd-4 
t=2017-02-18T14:05:46+0000
action=start creation date=2017-02-18T14:05:46+0000 ephemeral=false lvl=info 
msg="Starting container" name=juju-2c21a7-1-lxd-4 stateful=false 
t=2017-02-18T14:06:11+0000
action=start creation date=2017-02-18T14:05:46+0000 ephemeral=false lvl=info 
msg="Started container" name=juju-2c21a7-1-lxd-4 stateful=false 
t=2017-02-18T14:06:11+0000


The failure and retry is visible in the juju logsink.log:

[from landscape-0-inner-logs/bootstrap/var/log/juju/logsink.log]
 failed to start instance (Error calling 'lxd forkstart juju-2c21a7-1-lxd-4 
/var/lib/lxd/containers /var/log/lxd/juju-2c21a7-1-lxd-4/lxc.conf': err='exit 
status 1'
  lxc 20160218140528.391 ERROR lxc_apparmor - 
lsm/apparmor.c:apparmor_process_label_set:234 - No such file or directory - 
failed to change apparmor profile to 
lxd-juju-2c21a7-1-lxd-4_</var/lib/lxd>//&:lxd-juju-2c21a7-1-lxd-4_<var-lib-lxd>:
  lxc 20160218140528.391 ERROR lxc_sync - sync.c:__sync_wait:57 - An error 
occurred in another process (expected sequence number 5)
  lxc 20160218140528.391 ERROR lxc_start - start.c:__lxc_start:1346 - Failed to 
spawn container "juju-2c21a7-1-lxd-4".
  lxc 20160218140528.986 ERROR lxc_conf - conf.c:run_buffer:405 - Script exited 
with status 1.
  lxc 20160218140528.986 ERROR lxc_start - start.c:lxc_fini:546 - Failed to run 
lxc.hook.post-stop for container "juju-2c21a7-1-lxd-4".
), retrying in 10s (3 more attempts)
...
e331b8e0-d8ca-4007-8573-d9dd932c21a7: machine-1-lxd-4 2017-02-18 14:07:02 INFO 
juju.cmd supercommand.go:63 running jujud [2.1-rc2 gc go1.6]
e331b8e0-d8ca-4007-8573-d9dd932c21a7: machine-1-lxd-4 2017-02-18 14:07:02 DEBUG 
juju.cmd supercommand.go:64   args: 
[]string{"/var/lib/juju/tools/machine-1-lxd-4/jujud", "machine", "--data-dir", 
"/var/lib/juju", "--machine-id", "1/lxd/4", "--debug"}


This was found by an automated test [1]. We run 8-12 or so of these per day and 
this is the first time I've seen this particular error.

[1] - https://ci.lscape.net/job/landscape-system-tests/5330/

** Affects: lxd (Ubuntu)
     Importance: Undecided
         Status: New

** Attachment added: "all-logs.tar.gz"
   
https://bugs.launchpad.net/bugs/1667444/+attachment/4825387/+files/all-logs.tar.gz

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1667444

Title:
  Failed to start lxd container because it is already running a start
  operation

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/lxd/+bug/1667444/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

Reply via email to