** Description changed: + ********** SRU TEMPLATE AT THE BOTTOM ********* + CentOS 7.6 Kolla Ansible 7.0.2 deployment Steps to reproduce: 1) Add "enable_force_upload = True" to [DEFAULT] section of cinder.conf to allow clone of in-use volume to an image, restarting all Cinder containers. 2) Create an image from an in-use volume: openstack image create --disk-format qcow2 --force --volume 7b434a13-025c-4e8a-9718-bcb5d75cae18 TestImage In our case, the VM is powered down, but the volume is attached to this VM, so the volume status is "in-use". 3) Deploy image to new volume with: openstack volume create --image TestImage --size 50 TestVolume 4) Status shows as "creating" for a while, then "downloading" for a bit. Note that we are not using Ceph storage for this test. So, when the "creating" status is shown, the image is being copied to the compute node temporarily. The compute node has enough disk space for this process. Then, the download occurs, where the image is decompressed and written to the volume, after which the cinder-volume log indicates this was successful and that the "temporary image is deleted". 5) The metadata error occurs and the volume status is set to "error": MetadataCopyFailure: Failed to copy metadata to volume: Glance metadata cannot be updated, key signature_verified exists for volume id 5846f529-ff3a-4d5f-bd01-0860dbb9bcf6 - - The respective cinder-volume log entries are at the end of this message (no lines were removed - just spaced out for clarity) starting at the point in time where image->volume copy process succeeded. - + The respective cinder-volume log entries are at the end of this message + (no lines were removed - just spaced out for clarity) starting at the + point in time where image->volume copy process succeeded. The volume image metadata property of the original "in-use" volume (from Step 2 above) that was imported to an image is: - | id | 7b434a13-025c-4e8a-9718-bcb5d75cae18 + | id | 7b434a13-025c-4e8a-9718-bcb5d75cae18 | volume_image_metadata | {u'checksum': u'2a291147d0e8157d0a06097f46b32791', u'min_ram': u'0', u'disk_format': u'qcow2', u'image_name': u'TestImage', u'image_id': u'9eb38b7e-35e0-4465-b074-fb2de78990dd', u'signature_verified': u'False', u'container_format': u'bare', u'min_disk': u'0', u'size': u'9635037184'} The properties of the resulting image (from Step 2 above) are: - | id | 3b56e3c1-02c3-4713-9248-d1581f1f7221 + | id | 3b56e3c1-02c3-4713-9248-d1581f1f7221 | properties | locations='[{u'url': u'rbd://e7b5693b-800d-49fd-a483-d184e1ebb376/images/3b56e3c1-02c3-4713-9248-d1581f1f7221/snap', u'metadata': {}}]', os_hash_algo='sha512', os_hash_value='ee09c60599dbc00f44409bd7f15ae5b720bf23e8d9c48c8d19028cf947a8e10a4f3d750815e78dfe0fb15395b6d7c9a874981b62233840971073ff6268c7eda7', os_hidden='False', signature_verified='False' | The volume image metadata property of the newly created, error'd, image is: | id | 5846f529-ff3a-4d5f-bd01-0860dbb9bcf6 | | volume_image_metadata | {u'signature_verified': u'False'} | - - Since it appears that the signature_verified property already exists in the target volume, I thought this might cause the error since the source image has this property also. + Since it appears that the signature_verified property already exists in + the target volume, I thought this might cause the error since the source + image has this property also. So, I unset this property on the image using: openstack image unset --property signature_verified 3b56e3c1-02c3-4713-9248-d1581f1f7221 and the process succeeded! This seems like a bug somewhere in the metadata property cloning process. I would expect all volume_image_metadata to be over-written without error during a clone. Eric - - 2019-04-06 00:21:28.734 137 DEBUG cinder.volume.flows.manager.create_volume [req-2b67aef2-a543-48b4-b5f8-b19ecef9fe0d f6f81f8e87994ca785b746af4f39f6ee 6baee5fa402b4850a313b7d55d7b1aba - edcea72f06da4d5a879427b35649ffa2 edcea72f06da4d5a879427b35649ffa2] Downloaded image 3b56e3c1-02c3-4713-9248-d1581f1f7221 ((None, [{u'url': u'rbd://e7b5693b-800d-49fd-a483-d184e1ebb376/images/3b56e3c1-02c3-4713-9248-d1581f1f7221/snap', u'metadata': {}}])) to volume 5846f529-ff3a-4d5f-bd01-0860dbb9bcf6 successfully. _copy_image_to_volume /var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py:557 + 2019-04-06 00:21:28.734 137 DEBUG + cinder.volume.flows.manager.create_volume + [req-2b67aef2-a543-48b4-b5f8-b19ecef9fe0d + f6f81f8e87994ca785b746af4f39f6ee 6baee5fa402b4850a313b7d55d7b1aba - + edcea72f06da4d5a879427b35649ffa2 edcea72f06da4d5a879427b35649ffa2] + Downloaded image 3b56e3c1-02c3-4713-9248-d1581f1f7221 ((None, [{u'url': + u'rbd://e7b5693b-800d-49fd-a483-d184e1ebb376/images/3b56e3c1-02c3-4713-9248-d1581f1f7221/snap', + u'metadata': {}}])) to volume 5846f529-ff3a-4d5f-bd01-0860dbb9bcf6 + successfully. _copy_image_to_volume + /var/lib/kolla/venv/lib/python2.7/site- + packages/cinder/volume/flows/manager/create_volume.py:557 2019-04-06 00:21:29.769 137 DEBUG cinder.image.image_utils [req-2b67aef2-a543-48b4-b5f8-b19ecef9fe0d f6f81f8e87994ca785b746af4f39f6ee 6baee5fa402b4850a313b7d55d7b1aba - edcea72f06da4d5a879427b35649ffa2 edcea72f06da4d5a879427b35649ffa2] Temporary image 3b56e3c1-02c3-4713-9248-d1581f1f7221 for user f6f81f8e87994ca785b746af4f39f6ee is deleted. fetch /var/lib/kolla/venv/lib/python2.7/site- packages/cinder/image/image_utils.py:824 2019-04-06 00:21:29.769 137 DEBUG cinder.volume.flows.manager.create_volume [req-2b67aef2-a543-48b4-b5f8-b19ecef9fe0d f6f81f8e87994ca785b746af4f39f6ee 6baee5fa402b4850a313b7d55d7b1aba - edcea72f06da4d5a879427b35649ffa2 edcea72f06da4d5a879427b35649ffa2] Marking volume 5846f529-ff3a-4d5f-bd01-0860dbb9bcf6 as bootable. _enable_bootable_flag /var/lib/kolla/venv/lib/python2.7/site- packages/cinder/volume/flows/manager/create_volume.py:476 2019-04-06 00:21:29.775 137 DEBUG cinder.volume.flows.manager.create_volume [req-2b67aef2-a543-48b4-b5f8-b19ecef9fe0d f6f81f8e87994ca785b746af4f39f6ee 6baee5fa402b4850a313b7d55d7b1aba - edcea72f06da4d5a879427b35649ffa2 edcea72f06da4d5a879427b35649ffa2] Copying metadata from image 3b56e3c1-02c3-4713-9248-d1581f1f7221 to 5846f529-ff3a-4d5f-bd01-0860dbb9bcf6. _handle_bootable_volume_glance_meta /var/lib/kolla/venv/lib/python2.7/site- packages/cinder/volume/flows/manager/create_volume.py:432 2019-04-06 00:21:29.776 137 DEBUG cinder.volume.flows.manager.create_volume [req-2b67aef2-a543-48b4-b5f8-b19ecef9fe0d f6f81f8e87994ca785b746af4f39f6ee 6baee5fa402b4850a313b7d55d7b1aba - edcea72f06da4d5a879427b35649ffa2 edcea72f06da4d5a879427b35649ffa2] Creating volume glance metadata for volume 5846f529-ff3a-4d5f-bd01-0860dbb9bcf6 backed by image 3b56e3c1-02c3-4713-9248-d1581f1f7221 with: {'container_format': u'bare', 'min_ram': 0, 'disk_format': u'qcow2', 'image_name': u'Win2019_20190405', 'image_id': u'3b56e3c1-02c3-4713-9248-d1581f1f7221', u'signature_verified': u'False', 'checksum': u'29eebf5c7f1bad94c751ebdb8e9d615e', 'min_disk': 0, 'size': 13862240256}. _capture_volume_image_metadata /var/lib/kolla/venv/lib/python2.7/site- packages/cinder/volume/flows/manager/create_volume.py:591 2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume [req-2b67aef2-a543-48b4-b5f8-b19ecef9fe0d f6f81f8e87994ca785b746af4f39f6ee 6baee5fa402b4850a313b7d55d7b1aba - edcea72f06da4d5a879427b35649ffa2 edcea72f06da4d5a879427b35649ffa2] Failed updating volume 5846f529-ff3a-4d5f-bd01-0860dbb9bcf6 metadata using the provided image 3b56e3c1-02c3-4713-9248-d1581f1f7221 metadata: GlanceMetadataExists: Glance metadata cannot be updated, key signature_verified exists for volume id 5846f529-ff3a-4d5f-bd01-0860dbb9bcf6 2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume Traceback (most recent call last): 2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 434, in _handle_bootable_volume_glance_meta 2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume image_id, image_meta) 2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 593, in _capture_volume_image_metadata 2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume volume_metadata) 2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/api.py", line 971, in volume_glance_metadata_bulk_create 2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume metadata) 2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 183, in wrapper 2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume return f(*args, **kwargs) 2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 198, in wrapper 2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume return f(context, volume_id, *args, **kwargs) 2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 5042, in volume_glance_metadata_bulk_create 2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume volume_id=volume_id) 2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume GlanceMetadataExists: Glance metadata cannot be updated, key signature_verified exists for volume id 5846f529-ff3a-4d5f-bd01-0860dbb9bcf6 2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume 2019-04-06 00:21:29.796 137 WARNING cinder.volume.manager [req-2b67aef2-a543-48b4-b5f8-b19ecef9fe0d f6f81f8e87994ca785b746af4f39f6ee 6baee5fa402b4850a313b7d55d7b1aba - edcea72f06da4d5a879427b35649ffa2 edcea72f06da4d5a879427b35649ffa2] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (e01844d0-4736-4762-89ff-93338d70f38b) transitioned into state 'FAILURE' from state 'RUNNING' + + ============ + SRU TEMPLATE + ============ + + =============== + SRU DESCRIPTION + =============== + + [Impact] + + This issue has already been addressed by another LP (LP: #1945500) but + it hadn't included all possible string properties affected to be + filtered. We stumbled across another one that hits the same issue so we + need to add the property to the list so filtered properties. The impact + is very annoying to customers because their users unknownly hit this + issue by creating a volume from an image, then creating an image from + the volume, and then attempting to create a new volume from the derived + image. + + [Test case] + + 1) Deploy OpenStack with Cinder and have a test image in glance + + openstack image create --file ~/images/cirros-0.4.0-x86_64-disk.img + cirros --disk-format qcow2 + + 2) Create a volume from the test image + + openstack volume create --image cirros --size 1 v1 + + 3) Create a new image from the volume + + openstack image create --volume v1 v1 + + 4) Attempt to create a new volume from the derived image + + openstack volume create --image v1 --size 1 v2 + + 5) Verify in logs the error message in logs or volume show: + + GlanceMetadataExists: Glance metadata cannot be updated, key + signature_verified exists + + 6) install updated package + + 7) repeat steps 2-4 + + 8) Verify volume created successfully this time + + + [Regression Potential] + + Given the simplicity of this particular fix, and that it completements + an existing fix (by just adding an extra filter property) that has been + in production for quite a while, we can estimate the regression + potential to be very close to zero. The "signature_verified" property + workflow that is disrupted here by being filtered is actually intended + behavior, so we can be comfortable of not breaking something else that + expects the property to be there. + + [Other Info] + + None.
** Also affects: cinder (Ubuntu) Importance: Undecided Status: New ** Also affects: cinder (Ubuntu Oracular) Importance: Undecided Status: New ** Also affects: cinder (Ubuntu Noble) Importance: Undecided Status: New ** Also affects: cinder (Ubuntu Jammy) Importance: Undecided Status: New ** Also affects: cloud-archive Importance: Undecided Status: New ** Also affects: cloud-archive/yoga Importance: Undecided Status: New ** Also affects: cloud-archive/caracal Importance: Undecided Status: New ** Also affects: cloud-archive/bobcat Importance: Undecided Status: New ** Also affects: cloud-archive/dalmation Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1823445 Title: [SRU] MetadataCopyFailure: Failed to copy metadata to volume: Glance metadata cannot be updated, key signature_verified exists for volume id <volume id> To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1823445/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs