Public bug reported:
This was observed while testing landscape autopilot to deploy an
openstack cluster. This uses juju deployed lxd containers to host
various openstack services. Creation of a container failed, causing the
entire openstack deployment to fail.
lxd: 2.0.9 (xenial)
juju: 1:2.1~rc2-0ubuntu1~16.04.1~juju1
landscape: 17.01~bzr10906+jenkins3476-2
I don't know if this is an issue with lxd, juju, or something else. From
what I can see, the container requested by juju failed to start due to
err="Container is already running a start operation". Juju then deleted
the container and tried again about a minute later. The retry was
successful.
Landscape errors out on the first failure and isn't aware that things
worked after the retry.
[from landscape-0-inner-logs/base-machine-2/var/log/lxd/lxd.log]
ephemeral=false lvl=info msg="Creating container" name=juju-2c21a7-1-lxd-4
t=2017-02-18T14:04:57+0000
ephemeral=false lvl=info msg="Created container" name=juju-2c21a7-1-lxd-4
t=2017-02-18T14:04:57+0000
action=start creation date=2017-02-18T14:04:57+0000 ephemeral=false lvl=info
msg="Starting container" name=juju-2c21a7-1-lxd-4 stateful=false
t=2017-02-18T14:05:27+0000
container=juju-2c21a7-1-lxd-4 err="Container is already running a start
operation" lvl=eror msg="stop hook failed" t=2017-02-18T14:05:28+0000
action=start creation date=2017-02-18T14:04:57+0000 ephemeral=false lvl=eror
msg="Failed starting container" name=juju-2c21a7-1-lxd-4 stateful=false
t=2017-02-18T14:05:33+0000
creation date=2017-02-18T14:04:57+0000 ephemeral=false lvl=info msg="Deleting
container" name=juju-2c21a7-1-lxd-4 t=2017-02-18T14:05:34+0000
creation date=2017-02-18T14:04:57+0000 ephemeral=false lvl=info msg="Deleted
container" name=juju-2c21a7-1-lxd-4 t=2017-02-18T14:05:34+0000
ephemeral=false lvl=info msg="Creating container" name=juju-2c21a7-1-lxd-4
t=2017-02-18T14:05:46+0000
ephemeral=false lvl=info msg="Created container" name=juju-2c21a7-1-lxd-4
t=2017-02-18T14:05:46+0000
action=start creation date=2017-02-18T14:05:46+0000 ephemeral=false lvl=info
msg="Starting container" name=juju-2c21a7-1-lxd-4 stateful=false
t=2017-02-18T14:06:11+0000
action=start creation date=2017-02-18T14:05:46+0000 ephemeral=false lvl=info
msg="Started container" name=juju-2c21a7-1-lxd-4 stateful=false
t=2017-02-18T14:06:11+0000
The failure and retry is visible in the juju logsink.log:
[from landscape-0-inner-logs/bootstrap/var/log/juju/logsink.log]
failed to start instance (Error calling 'lxd forkstart juju-2c21a7-1-lxd-4
/var/lib/lxd/containers /var/log/lxd/juju-2c21a7-1-lxd-4/lxc.conf': err='exit
status 1'
lxc 20160218140528.391 ERROR lxc_apparmor -
lsm/apparmor.c:apparmor_process_label_set:234 - No such file or directory -
failed to change apparmor profile to
lxd-juju-2c21a7-1-lxd-4_</var/lib/lxd>//&:lxd-juju-2c21a7-1-lxd-4_<var-lib-lxd>:
lxc 20160218140528.391 ERROR lxc_sync - sync.c:__sync_wait:57 - An error
occurred in another process (expected sequence number 5)
lxc 20160218140528.391 ERROR lxc_start - start.c:__lxc_start:1346 - Failed to
spawn container "juju-2c21a7-1-lxd-4".
lxc 20160218140528.986 ERROR lxc_conf - conf.c:run_buffer:405 - Script exited
with status 1.
lxc 20160218140528.986 ERROR lxc_start - start.c:lxc_fini:546 - Failed to run
lxc.hook.post-stop for container "juju-2c21a7-1-lxd-4".
), retrying in 10s (3 more attempts)
...
e331b8e0-d8ca-4007-8573-d9dd932c21a7: machine-1-lxd-4 2017-02-18 14:07:02 INFO
juju.cmd supercommand.go:63 running jujud [2.1-rc2 gc go1.6]
e331b8e0-d8ca-4007-8573-d9dd932c21a7: machine-1-lxd-4 2017-02-18 14:07:02 DEBUG
juju.cmd supercommand.go:64 args:
[]string{"/var/lib/juju/tools/machine-1-lxd-4/jujud", "machine", "--data-dir",
"/var/lib/juju", "--machine-id", "1/lxd/4", "--debug"}
This was found by an automated test [1]. We run 8-12 or so of these per day and
this is the first time I've seen this particular error.
[1] - https://ci.lscape.net/job/landscape-system-tests/5330/
** Affects: lxd (Ubuntu)
Importance: Undecided
Status: New
** Attachment added: "all-logs.tar.gz"
https://bugs.launchpad.net/bugs/1667444/+attachment/4825387/+files/all-logs.tar.gz
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1667444
Title:
Failed to start lxd container because it is already running a start
operation
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/lxd/+bug/1667444/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
