I was unable to reproduce this in a Disco VM that I manually configured
to mount an ext4 virtio-blk device at boot:
$ dmesg | grep vdb
[ 2.352490] virtio_blk virtio4: [vdb] 20971520 512-byte logical blocks (10.7
GB/10.0 GiB)
[ 6.898149] EXT4-fs (vdb): mounted filesystem with ordered data mode. Opts:
(null)
tyhicks@sec-disco-amd64:~$ uname -a
Linux sec-disco-amd64 5.0.0-27-generic #28-Ubuntu SMP Tue Aug 20 19:53:07 UTC
2019 x86_64 x86_64 x86_64 GNU/Linux
tyhicks@sec-disco-amd64:~$ lsb_release -c
Codename: disco
tyhicks@sec-disco-amd64:~$ sudo umount /mnt
[sudo] password for tyhicks:
tyhicks@sec-disco-amd64:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sr0 11:0 1 1024M 0 rom
vda 252:0 0 16G 0 disk
└─vda1 252:1 0 16G 0 part
├─vmtools--vg-root 253:0 0 15G 0 lvm /
└─vmtools--vg-swap_1 253:1 0 980M 0 lvm [SWAP]
vdb 252:16 0 10G 0 disk
tyhicks@sec-disco-amd64:~$ sudo mkfs.xfs -f -i size=1024 /dev/vdb
meta-data=/dev/vdb isize=1024 agcount=4, agsize=655360 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=0, rmapbt=0,
reflink=0
data = bsize=4096 blocks=2621440, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
I manually ran those commands after the VM came up. This doesn't really
rule anything out but might indicate that the more complex
juju/OpenStack environment is needed to reproduce.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1842751
Title:
[disco] [eoan] After unmount, cannot open /dev/vdb: Device or resource
busy
Status in OpenStack cinder charm:
New
Status in OpenStack nova-compute charm:
New
Status in linux package in Ubuntu:
Confirmed
Bug description:
Disco and Eoan device is busy after unmounting an ephemeral disk,
cannot format the device until rebooting.
This is blocking all of OpenStack Charms which interact with block
devices (Nova Compute, Ceph, Swift, Cinder), on the Disco and Eoan
series. As we are nearing LTS-1, this will become urgent pretty
quickly.
Reproducer, on an OpenStack cloud:
juju deploy cs:ubuntu ubuntu-bionic --series bionic
juju deploy cs:ubuntu ubuntu-disco --series disco
juju deploy cs:ubuntu ubuntu-eoan --series eoan
#### Succeeds on Bionic:
ubuntu@juju-8d01b7-foo-14:~$ uname -a
Linux juju-8d01b7-foo-14 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30
UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@juju-8d01b7-foo-14:~$ lsb_release -c
Codename: bionic
ubuntu@juju-8d01b7-foo-14:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 252:0 0 20G 0 disk
├─vda1 252:1 0 19.9G 0 part /
├─vda14 252:14 0 4M 0 part
└─vda15 252:15 0 106M 0 part /boot/efi
vdb 252:16 0 40G 0 disk /mnt
ubuntu@juju-8d01b7-foo-14:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 985M 0 985M 0% /dev
tmpfs 200M 712K 199M 1% /run
/dev/vda1 20G 1.7G 18G 9% /
tmpfs 997M 0 997M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 997M 0 997M 0% /sys/fs/cgroup
/dev/vda15 105M 3.6M 101M 4% /boot/efi
/dev/vdb 40G 49M 38G 1% /mnt
tmpfs 100K 0 100K 0% /var/lib/lxd/shmounts
tmpfs 100K 0 100K 0% /var/lib/lxd/devlxd
tmpfs 200M 0 200M 0% /run/user/1000
ubuntu@juju-8d01b7-foo-14:~$ sudo umount /dev/vdb
ubuntu@juju-8d01b7-foo-14:~$ sudo mkfs.xfs -f -i size=1024 /dev/vdb
meta-data=/dev/vdb isize=1024 agcount=4, agsize=2621440 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=0, rmapbt=0,
reflink=0
data = bsize=4096 blocks=10485760, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=5120, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
#### Fails on Disco:
ubuntu@juju-8d01b7-foo-12:~$ uname -a
Linux juju-8d01b7-foo-12 5.0.0-27-generic #28-Ubuntu SMP Tue Aug 20 19:53:07
UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@juju-8d01b7-foo-12:~$ lsb_release -c
Codename: disco
ubuntu@juju-8d01b7-foo-12:~$ sudo umount /mnt
ubuntu@juju-8d01b7-foo-12:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 88.7M 1 loop /snap/core/7396
loop1 7:1 0 54.5M 1 loop /snap/lxd/11727
vda 252:0 0 20G 0 disk
├─vda1 252:1 0 19.9G 0 part /
├─vda14 252:14 0 4M 0 part
└─vda15 252:15 0 106M 0 part /boot/efi
vdb 252:16 0 40G 0 disk
ubuntu@juju-8d01b7-foo-12:~$ sudo mkfs.xfs -f -i size=1024 /dev/vdb
mkfs.xfs: cannot open /dev/vdb: Device or resource busy
#### Fails on Eoan:
ubuntu@juju-8d01b7-foo-13:~$ sudo umount /mnt
ubuntu@juju-8d01b7-foo-13:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 88.7M 1 loop /snap/core/7396
loop1 7:1 0 54.5M 1 loop /snap/lxd/11727
vda 252:0 0 20G 0 disk
├─vda1 252:1 0 19.9G 0 part /
├─vda14 252:14 0 4M 0 part
└─vda15 252:15 0 106M 0 part /boot/efi
vdb 252:16 0 40G 0 disk
ubuntu@juju-8d01b7-foo-13:~$ sudo mkfs.xfs -f -i size=1024 /dev/vdb
mkfs.xfs: cannot open /dev/vdb: Device or resource busy
..
Original bug description:
On disco-stein, hook failed: "config-changed" with mkfs.xfs: cannot
open /dev/vdb: Device or resource busy
https://openstack-ci-
reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack
/charm-cinder/678676/3/3803/index.html
https://openstack-ci-
reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack
/charm-cinder/678676/3/3803/test_charm_func_full_7062/juju-status-
zaza-5b39f0208674.txt
https://openstack-ci-
reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack
/charm-cinder/678676/3/3803/test_charm_func_full_7062/juju-crashdump-
7d3902a2-4fdf-435a-82c4-5d2ad9af4cb5.tar.xz
2019-09-03 21:33:27 DEBUG config-changed Unpacking apparmor-utils
(2.13.2-9ubuntu6.1) ...
2019-09-03 21:33:27 DEBUG config-changed Setting up python3-libapparmor
(2.13.2-9ubuntu6.1) ...
2019-09-03 21:33:27 DEBUG config-changed Setting up python3-apparmor
(2.13.2-9ubuntu6.1) ...
2019-09-03 21:33:27 DEBUG config-changed Setting up apparmor-utils
(2.13.2-9ubuntu6.1) ...
2019-09-03 21:33:27 DEBUG config-changed Processing triggers for man-db
(2.8.5-2) ...
2019-09-03 21:33:30 INFO juju-log Setting up the apparmor profile for
usr.bin.nova-compute in enforce mode.
2019-09-03 21:33:30 DEBUG config-changed Setting
/etc/apparmor.d/usr.bin.nova-compute to enforce mode.
2019-09-03 21:33:30 DEBUG config-changed mkfs.xfs: cannot open /dev/vdb:
Device or resource busy
2019-09-03 21:33:30 DEBUG config-changed Traceback (most recent call last):
2019-09-03 21:33:30 DEBUG config-changed File
"/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/config-changed", line
751, in <module>
2019-09-03 21:33:30 DEBUG config-changed main()
2019-09-03 21:33:30 DEBUG config-changed File
"/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/config-changed", line
744, in main
2019-09-03 21:33:30 DEBUG config-changed hooks.execute(sys.argv)
2019-09-03 21:33:30 DEBUG config-changed File
"/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/core/hookenv.py",
line 914, in execute
2019-09-03 21:33:30 DEBUG config-changed self._hooks[hook_name]()
2019-09-03 21:33:30 DEBUG config-changed File
"/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/contrib/openstack/utils.py",
line 1560, in wrapped_f
2019-09-03 21:33:30 DEBUG config-changed stopstart, restart_functions)
2019-09-03 21:33:30 DEBUG config-changed File
"/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/core/host.py",
line 741, in restart_on_change_helper
2019-09-03 21:33:30 DEBUG config-changed r = lambda_f()
2019-09-03 21:33:30 DEBUG config-changed File
"/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/contrib/openstack/utils.py",
line 1559, in <lambda>
2019-09-03 21:33:30 DEBUG config-changed (lambda: f(*args, **kwargs)),
__restart_map_cache['cache'],
2019-09-03 21:33:30 DEBUG config-changed File
"/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/contrib/hardening/harden.py",
line 93, in _harden_inner2
2019-09-03 21:33:30 DEBUG config-changed return f(*args, **kwargs)
2019-09-03 21:33:30 DEBUG config-changed File
"/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/config-changed", line
264, in config_changed
2019-09-03 21:33:30 DEBUG config-changed
configure_local_ephemeral_storage()
2019-09-03 21:33:30 DEBUG config-changed File
"/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/nova_compute_utils.py",
line 1015, in configure_local_ephemeral_storage
2019-09-03 21:33:30 DEBUG config-changed mkfs_xfs(dev, force=True)
2019-09-03 21:33:30 DEBUG config-changed File
"/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/contrib/storage/linux/utils.py",
line 128, in mkfs_xfs
2019-09-03 21:33:30 DEBUG config-changed check_call(cmd)
2019-09-03 21:33:30 DEBUG config-changed File
"/usr/lib/python3.7/subprocess.py", line 347, in check_call
2019-09-03 21:33:30 DEBUG config-changed raise
CalledProcessError(retcode, cmd)
2019-09-03 21:33:30 DEBUG config-changed subprocess.CalledProcessError:
Command '['mkfs.xfs', '-f', '-i', 'size=1024', '/dev/vdb']' returned non-zero
exit status 1.
2019-09-03 21:33:30 ERROR juju.worker.uniter.operation runhook.go:132 hook
"config-changed" failed: exit status 1
---
ProblemType: Bug
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Sep 5 20:47 seq
crw-rw---- 1 root audio 116, 33 Sep 5 20:47 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay'
ApportVersion: 2.20.10-0ubuntu27.1
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord':
'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq',
'/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
DistroRelease: Ubuntu 19.04
Ec2AMI: ami-0000078c
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.small
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig'
Lsusb:
Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd
Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: OpenStack Foundation OpenStack Nova
Package: linux (not installed)
PciMultimedia:
ProcEnviron:
TERM=screen-256color
PATH=(custom, no user)
XDG_RUNTIME_DIR=<set>
LANG=C.UTF-8
SHELL=/bin/bash
ProcFB:
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.0.0-27-generic
root=PARTUUID=c6e26b6d-1cc5-47e0-8025-e7a9c79727cc ro console=tty1 console=ttyS0
ProcVersionSignature: Ubuntu 5.0.0-27.28-generic 5.0.21
RelatedPackageVersions:
linux-restricted-modules-5.0.0-27-generic N/A
linux-backports-modules-5.0.0-27-generic N/A
linux-firmware N/A
RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill'
Tags: disco ec2-images
Uname: Linux 5.0.0-27-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video
_MarkForUpload: True
dmi.bios.date: 04/01/2014
dmi.bios.vendor: SeaBIOS
dmi.bios.version: 1.10.2-1ubuntu1
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.version: pc-i440fx-bionic
dmi.modalias:
dmi:bvnSeaBIOS:bvr1.10.2-1ubuntu1:bd04/01/2014:svnOpenStackFoundation:pnOpenStackNova:pvr17.0.10:cvnQEMU:ct1:cvrpc-i440fx-bionic:
dmi.product.family: Virtual Machine
dmi.product.name: OpenStack Nova
dmi.product.version: 17.0.10
dmi.sys.vendor: OpenStack Foundation
---
ProblemType: Bug
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Sep 5 20:48 seq
crw-rw---- 1 root audio 116, 33 Sep 5 20:48 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay'
ApportVersion: 2.20.11-0ubuntu7
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord':
'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq',
'/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
DistroRelease: Ubuntu 19.10
Ec2AMI: ami-0000078d
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.small
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig'
Lsusb:
Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd QEMU USB Tablet
Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: OpenStack Foundation OpenStack Nova
Package: linux (not installed)
PciMultimedia:
ProcEnviron:
TERM=screen-256color
PATH=(custom, no user)
XDG_RUNTIME_DIR=<set>
LANG=C.UTF-8
SHELL=/bin/bash
ProcFB:
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.2.0-15-generic
root=PARTUUID=c92b681b-892a-4294-a588-50850030af8a ro console=tty1 console=ttyS0
ProcVersionSignature: Ubuntu 5.2.0-15.16-generic 5.2.9
RelatedPackageVersions:
linux-restricted-modules-5.2.0-15-generic N/A
linux-backports-modules-5.2.0-15-generic N/A
linux-firmware N/A
RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill'
Tags: eoan ec2-images
Uname: Linux 5.2.0-15-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video
_MarkForUpload: True
dmi.bios.date: 04/01/2014
dmi.bios.vendor: SeaBIOS
dmi.bios.version: 1.10.2-1ubuntu1
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.version: pc-i440fx-bionic
dmi.modalias:
dmi:bvnSeaBIOS:bvr1.10.2-1ubuntu1:bd04/01/2014:svnOpenStackFoundation:pnOpenStackNova:pvr17.0.10:cvnQEMU:ct1:cvrpc-i440fx-bionic:
dmi.product.family: Virtual Machine
dmi.product.name: OpenStack Nova
dmi.product.version: 17.0.10
dmi.sys.vendor: OpenStack Foundation
---
ProblemType: Bug
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Sep 5 20:48 seq
crw-rw---- 1 root audio 116, 33 Sep 5 20:48 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay'
ApportVersion: 2.20.9-0ubuntu7.7
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord':
'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq',
'/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
DistroRelease: Ubuntu 18.04
Ec2AMI: ami-00000789
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.small
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig'
Lsusb:
Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd
Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: OpenStack Foundation OpenStack Nova
Package: linux (not installed)
PciMultimedia:
ProcEnviron:
TERM=screen-256color
PATH=(custom, no user)
XDG_RUNTIME_DIR=<set>
LANG=C.UTF-8
SHELL=/bin/bash
ProcFB:
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-60-generic
root=LABEL=cloudimg-rootfs ro console=tty1 console=ttyS0
ProcVersionSignature: Ubuntu 4.15.0-60.67-generic 4.15.18
RelatedPackageVersions:
linux-restricted-modules-4.15.0-60-generic N/A
linux-backports-modules-4.15.0-60-generic N/A
linux-firmware N/A
RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill'
Tags: bionic ec2-images
Uname: Linux 4.15.0-60-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video
_MarkForUpload: True
dmi.bios.date: 04/01/2014
dmi.bios.vendor: SeaBIOS
dmi.bios.version: 1.10.2-1ubuntu1
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.version: pc-i440fx-bionic
dmi.modalias:
dmi:bvnSeaBIOS:bvr1.10.2-1ubuntu1:bd04/01/2014:svnOpenStackFoundation:pnOpenStackNova:pvr17.0.10:cvnQEMU:ct1:cvrpc-i440fx-bionic:
dmi.product.family: Virtual Machine
dmi.product.name: OpenStack Nova
dmi.product.version: 17.0.10
dmi.sys.vendor: OpenStack Foundation
To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-cinder/+bug/1842751/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
