As a hint on reproducing it, it may be a problem on xenial only (which we used for that deployment) for clean devices.
On a bionic VM it seems like the symlink is created but it's not clear if luksFormat returns before that symlink gets created or after - this is the important part because the automation tries to use that symlink right after `cryptsetup luksFormat <dev>` exits. tree /sys/class/block https://paste.ubuntu.com/p/vhjjzdytH7/ uname -a Linux maas-vhost6 4.15.0-34-generic #37-Ubuntu SMP Mon Aug 27 15:21:48 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux ubuntu@maas-vhost6:~$ tree /dev/disk/by-uuid/ /dev/disk/by-uuid/ └── d26a75c9-15f7-41de-8c0e-20f795ed5729 -> ../../sda1 0 directories, 1 file ubuntu@maas-vhost6:~$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 64G 0 disk └─sda1 8:1 0 64G 0 part / sdb 8:16 0 8G 0 disk sdc 8:32 0 102.4M 0 disk sdd 8:48 0 102.4M 0 disk sde 8:64 0 102.4M 0 disk vda 252:0 0 102.4M 0 disk vdb 252:16 0 102.4M 0 disk nvme0n1 259:0 0 20G 0 disk nvme1n1 259:1 0 20G 0 disk ubuntu@maas-vhost6:~$ sudo cryptsetup luksFormat /dev/sdb WARNING! ======== This will overwrite data on /dev/sdb irrevocably. Are you sure? (Type uppercase yes): YES Enter passphrase for /dev/sdb: Verify passphrase: ubuntu@maas-vhost6:~$ tree /dev/disk/by-uuid/ /dev/disk/by-uuid/ ├── a82ddfe0-7de8-4c6c-aaca-a074f000b746 -> ../../sdb └── d26a75c9-15f7-41de-8c0e-20f795ed5729 -> ../../sda1 0 directories, 2 files sudo cryptsetup luksFormat /dev/sdb WARNING! ======== This will overwrite data on /dev/sdb irrevocably. Are you sure? (Type uppercase yes): YES Enter passphrase for /dev/sdb: Verify passphrase: ubuntu@maas-vhost6:~$ sudo cryptsetup luksDump /dev/sdb | grep UUID UUID: 21bacaf0-9eea-4809-9b1c-6f4d7e614f5b ubuntu@maas-vhost6:~$ tree /dev/disk/by-uuid/ /dev/disk/by-uuid/ ├── 21bacaf0-9eea-4809-9b1c-6f4d7e614f5b -> ../../sdb └── d26a75c9-15f7-41de-8c0e-20f795ed5729 -> ../../sda1 0 directories, 2 files -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1780332 Title: vaultlocker does not ensure that udev is triggered to create /dev/disk /by-uuid/<uuid-in-luks-header> symlink and fails Status in vaultlocker: Fix Released Status in cryptsetup package in Ubuntu: New Status in systemd package in Ubuntu: Incomplete Bug description: When an encrypted device is setup up a UUID (osd_fsid) is passed from the charm to be used in the cryptsetup command which accepts a UUID to place into the LUKS header (shown in cryptsetup luksDump <path-to- block-device>). https://github.com/openstack/charm-ceph-osd/blob/stable/18.05/lib/ceph/utils.py#L1788-L1804 UUID comes from osd_fsid https://github.com/openstack-charmers/vaultlocker/blob/8c9cb85dc3ed5dbf18c66a810d189a5230d85c34/vaultlocker/shell.py#L69-L80 # else statement is used here block_uuid = str(uuid.uuid4()) if not args.uuid else args.uuid dmcrypt.luks_format(key, block_device, block_uuid) # creates a LUKS header # ... dmcrypt.luks_open(key, block_uuid) # sets up a device with device mapper decrypting it via dmcrypt https://github.com/openstack- charmers/vaultlocker/blob/d813233179bdf2eec8ed101c702a8e552a966f44/vaultlocker/dmcrypt.py#L44-L56 This UUID is visible in blkid output /dev/sdc: UUID="<luks-header-uuid>" TYPE="crypto_LUKS" and a udev rule exists to create a /dev/disk/by-uuid/<luks-header- uuid> symlink (which is normally used for filesystem -> block device resolution) https://git.launchpad.net/~usd-import-team/ubuntu/+source/lvm2/tree/udev/13-dm-disk.rules.in#n25 ENV{ID_FS_USAGE}=="filesystem|other|crypto", ENV{ID_FS_UUID_ENC}=="?*", SYMLINK+="disk/by-uuid/$env{ID_FS_UUID_ENC}" Where vaultlocker fails is in luks_open command (right after luks_format) # cryptsetup --batch-mode --key-file - open UUID=<luks-header-uuid> crypt-<luks-header-uuid> --type luks because it tries to access /dev/disk/by-uuid/<luks-header-uuid> which does not exist. This happens since udev rules are not re-triggered to create this symlink after a LUKS device is created. Solution: call the command below after luks_format before luks_open udevadm settle --exit-if-exists=/dev/disk/by-uuid/<luks-header-uuid- equal-to-osd-fsid> To manage notifications about this bug go to: https://bugs.launchpad.net/vaultlocker/+bug/1780332/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
