** Description changed:
+ ********** SRU TEMPLATE AT THE BOTTOM *********
+
CentOS 7.6
Kolla Ansible 7.0.2 deployment
Steps to reproduce:
1) Add "enable_force_upload = True" to [DEFAULT] section of cinder.conf to
allow clone of in-use volume to an image, restarting all Cinder containers.
2) Create an image from an in-use volume: openstack image create
--disk-format qcow2 --force --volume
7b434a13-025c-4e8a-9718-bcb5d75cae18 TestImage
In our case, the VM is powered down, but the volume is attached to this
VM, so the volume status is "in-use".
3) Deploy image to new volume with: openstack volume create --image
TestImage --size 50 TestVolume
4) Status shows as "creating" for a while, then "downloading" for a bit.
Note that we are not using Ceph storage for this test. So, when the
"creating" status is shown, the image is being copied to the compute
node temporarily. The compute node has enough disk space for this
process. Then, the download occurs, where the image is decompressed and
written to the volume, after which the cinder-volume log indicates this
was successful and that the "temporary image is deleted".
5) The metadata error occurs and the volume status is set to "error":
MetadataCopyFailure: Failed to copy metadata to volume: Glance metadata
cannot be updated, key signature_verified exists for volume id
5846f529-ff3a-4d5f-bd01-0860dbb9bcf6
-
- The respective cinder-volume log entries are at the end of this message (no
lines were removed - just spaced out for clarity) starting at the point in time
where image->volume copy process succeeded.
-
+ The respective cinder-volume log entries are at the end of this message
+ (no lines were removed - just spaced out for clarity) starting at the
+ point in time where image->volume copy process succeeded.
The volume image metadata property of the original "in-use" volume (from Step
2 above) that was imported to an image is:
- | id | 7b434a13-025c-4e8a-9718-bcb5d75cae18
+ | id | 7b434a13-025c-4e8a-9718-bcb5d75cae18
| volume_image_metadata | {u'checksum':
u'2a291147d0e8157d0a06097f46b32791', u'min_ram': u'0', u'disk_format':
u'qcow2', u'image_name': u'TestImage', u'image_id':
u'9eb38b7e-35e0-4465-b074-fb2de78990dd', u'signature_verified': u'False',
u'container_format': u'bare', u'min_disk': u'0', u'size': u'9635037184'}
The properties of the resulting image (from Step 2 above) are:
- | id | 3b56e3c1-02c3-4713-9248-d1581f1f7221
+ | id | 3b56e3c1-02c3-4713-9248-d1581f1f7221
| properties | locations='[{u'url':
u'rbd://e7b5693b-800d-49fd-a483-d184e1ebb376/images/3b56e3c1-02c3-4713-9248-d1581f1f7221/snap',
u'metadata': {}}]', os_hash_algo='sha512',
os_hash_value='ee09c60599dbc00f44409bd7f15ae5b720bf23e8d9c48c8d19028cf947a8e10a4f3d750815e78dfe0fb15395b6d7c9a874981b62233840971073ff6268c7eda7',
os_hidden='False', signature_verified='False' |
The volume image metadata property of the newly created, error'd, image is:
| id | 5846f529-ff3a-4d5f-bd01-0860dbb9bcf6 |
| volume_image_metadata | {u'signature_verified': u'False'} |
-
- Since it appears that the signature_verified property already exists in the
target volume, I thought this might cause the error since the source image has
this property also.
+ Since it appears that the signature_verified property already exists in
+ the target volume, I thought this might cause the error since the source
+ image has this property also.
So, I unset this property on the image using:
openstack image unset --property signature_verified
3b56e3c1-02c3-4713-9248-d1581f1f7221
and the process succeeded!
This seems like a bug somewhere in the metadata property cloning
process. I would expect all volume_image_metadata to be over-written
without error during a clone.
Eric
-
- 2019-04-06 00:21:28.734 137 DEBUG cinder.volume.flows.manager.create_volume
[req-2b67aef2-a543-48b4-b5f8-b19ecef9fe0d f6f81f8e87994ca785b746af4f39f6ee
6baee5fa402b4850a313b7d55d7b1aba - edcea72f06da4d5a879427b35649ffa2
edcea72f06da4d5a879427b35649ffa2] Downloaded image
3b56e3c1-02c3-4713-9248-d1581f1f7221 ((None, [{u'url':
u'rbd://e7b5693b-800d-49fd-a483-d184e1ebb376/images/3b56e3c1-02c3-4713-9248-d1581f1f7221/snap',
u'metadata': {}}])) to volume 5846f529-ff3a-4d5f-bd01-0860dbb9bcf6
successfully. _copy_image_to_volume
/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py:557
+ 2019-04-06 00:21:28.734 137 DEBUG
+ cinder.volume.flows.manager.create_volume
+ [req-2b67aef2-a543-48b4-b5f8-b19ecef9fe0d
+ f6f81f8e87994ca785b746af4f39f6ee 6baee5fa402b4850a313b7d55d7b1aba -
+ edcea72f06da4d5a879427b35649ffa2 edcea72f06da4d5a879427b35649ffa2]
+ Downloaded image 3b56e3c1-02c3-4713-9248-d1581f1f7221 ((None, [{u'url':
+
u'rbd://e7b5693b-800d-49fd-a483-d184e1ebb376/images/3b56e3c1-02c3-4713-9248-d1581f1f7221/snap',
+ u'metadata': {}}])) to volume 5846f529-ff3a-4d5f-bd01-0860dbb9bcf6
+ successfully. _copy_image_to_volume
+ /var/lib/kolla/venv/lib/python2.7/site-
+ packages/cinder/volume/flows/manager/create_volume.py:557
2019-04-06 00:21:29.769 137 DEBUG cinder.image.image_utils
[req-2b67aef2-a543-48b4-b5f8-b19ecef9fe0d
f6f81f8e87994ca785b746af4f39f6ee 6baee5fa402b4850a313b7d55d7b1aba -
edcea72f06da4d5a879427b35649ffa2 edcea72f06da4d5a879427b35649ffa2]
Temporary image 3b56e3c1-02c3-4713-9248-d1581f1f7221 for user
f6f81f8e87994ca785b746af4f39f6ee is deleted. fetch
/var/lib/kolla/venv/lib/python2.7/site-
packages/cinder/image/image_utils.py:824
2019-04-06 00:21:29.769 137 DEBUG
cinder.volume.flows.manager.create_volume
[req-2b67aef2-a543-48b4-b5f8-b19ecef9fe0d
f6f81f8e87994ca785b746af4f39f6ee 6baee5fa402b4850a313b7d55d7b1aba -
edcea72f06da4d5a879427b35649ffa2 edcea72f06da4d5a879427b35649ffa2]
Marking volume 5846f529-ff3a-4d5f-bd01-0860dbb9bcf6 as bootable.
_enable_bootable_flag /var/lib/kolla/venv/lib/python2.7/site-
packages/cinder/volume/flows/manager/create_volume.py:476
2019-04-06 00:21:29.775 137 DEBUG
cinder.volume.flows.manager.create_volume
[req-2b67aef2-a543-48b4-b5f8-b19ecef9fe0d
f6f81f8e87994ca785b746af4f39f6ee 6baee5fa402b4850a313b7d55d7b1aba -
edcea72f06da4d5a879427b35649ffa2 edcea72f06da4d5a879427b35649ffa2]
Copying metadata from image 3b56e3c1-02c3-4713-9248-d1581f1f7221 to
5846f529-ff3a-4d5f-bd01-0860dbb9bcf6.
_handle_bootable_volume_glance_meta
/var/lib/kolla/venv/lib/python2.7/site-
packages/cinder/volume/flows/manager/create_volume.py:432
2019-04-06 00:21:29.776 137 DEBUG
cinder.volume.flows.manager.create_volume
[req-2b67aef2-a543-48b4-b5f8-b19ecef9fe0d
f6f81f8e87994ca785b746af4f39f6ee 6baee5fa402b4850a313b7d55d7b1aba -
edcea72f06da4d5a879427b35649ffa2 edcea72f06da4d5a879427b35649ffa2]
Creating volume glance metadata for volume
5846f529-ff3a-4d5f-bd01-0860dbb9bcf6 backed by image
3b56e3c1-02c3-4713-9248-d1581f1f7221 with: {'container_format': u'bare',
'min_ram': 0, 'disk_format': u'qcow2', 'image_name':
u'Win2019_20190405', 'image_id':
u'3b56e3c1-02c3-4713-9248-d1581f1f7221', u'signature_verified':
u'False', 'checksum': u'29eebf5c7f1bad94c751ebdb8e9d615e', 'min_disk':
0, 'size': 13862240256}. _capture_volume_image_metadata
/var/lib/kolla/venv/lib/python2.7/site-
packages/cinder/volume/flows/manager/create_volume.py:591
2019-04-06 00:21:29.792 137 ERROR
cinder.volume.flows.manager.create_volume
[req-2b67aef2-a543-48b4-b5f8-b19ecef9fe0d
f6f81f8e87994ca785b746af4f39f6ee 6baee5fa402b4850a313b7d55d7b1aba -
edcea72f06da4d5a879427b35649ffa2 edcea72f06da4d5a879427b35649ffa2]
Failed updating volume 5846f529-ff3a-4d5f-bd01-0860dbb9bcf6 metadata
using the provided image 3b56e3c1-02c3-4713-9248-d1581f1f7221 metadata:
GlanceMetadataExists: Glance metadata cannot be updated, key
signature_verified exists for volume id
5846f529-ff3a-4d5f-bd01-0860dbb9bcf6
2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume
Traceback (most recent call last):
2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume
File
"/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py",
line 434, in _handle_bootable_volume_glance_meta
2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume
image_id, image_meta)
2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume
File
"/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py",
line 593, in _capture_volume_image_metadata
2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume
volume_metadata)
2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume
File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/api.py", line
971, in volume_glance_metadata_bulk_create
2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume
metadata)
2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume
File
"/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py",
line 183, in wrapper
2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume
return f(*args, **kwargs)
2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume
File
"/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py",
line 198, in wrapper
2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume
return f(context, volume_id, *args, **kwargs)
2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume
File
"/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py",
line 5042, in volume_glance_metadata_bulk_create
2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume
volume_id=volume_id)
2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume
GlanceMetadataExists: Glance metadata cannot be updated, key signature_verified
exists for volume id 5846f529-ff3a-4d5f-bd01-0860dbb9bcf6
2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume
2019-04-06 00:21:29.796 137 WARNING cinder.volume.manager
[req-2b67aef2-a543-48b4-b5f8-b19ecef9fe0d f6f81f8e87994ca785b746af4f39f6ee
6baee5fa402b4850a313b7d55d7b1aba - edcea72f06da4d5a879427b35649ffa2
edcea72f06da4d5a879427b35649ffa2] Task
'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create'
(e01844d0-4736-4762-89ff-93338d70f38b) transitioned into state 'FAILURE' from
state 'RUNNING'
+
+ ============
+ SRU TEMPLATE
+ ============
+
+ ===============
+ SRU DESCRIPTION
+ ===============
+
+ [Impact]
+
+ This issue has already been addressed by another LP (LP: #1945500) but
+ it hadn't included all possible string properties affected to be
+ filtered. We stumbled across another one that hits the same issue so we
+ need to add the property to the list so filtered properties. The impact
+ is very annoying to customers because their users unknownly hit this
+ issue by creating a volume from an image, then creating an image from
+ the volume, and then attempting to create a new volume from the derived
+ image.
+
+ [Test case]
+
+ 1) Deploy OpenStack with Cinder and have a test image in glance
+
+ openstack image create --file ~/images/cirros-0.4.0-x86_64-disk.img
+ cirros --disk-format qcow2
+
+ 2) Create a volume from the test image
+
+ openstack volume create --image cirros --size 1 v1
+
+ 3) Create a new image from the volume
+
+ openstack image create --volume v1 v1
+
+ 4) Attempt to create a new volume from the derived image
+
+ openstack volume create --image v1 --size 1 v2
+
+ 5) Verify in logs the error message in logs or volume show:
+
+ GlanceMetadataExists: Glance metadata cannot be updated, key
+ signature_verified exists
+
+ 6) install updated package
+
+ 7) repeat steps 2-4
+
+ 8) Verify volume created successfully this time
+
+
+ [Regression Potential]
+
+ Given the simplicity of this particular fix, and that it completements
+ an existing fix (by just adding an extra filter property) that has been
+ in production for quite a while, we can estimate the regression
+ potential to be very close to zero. The "signature_verified" property
+ workflow that is disrupted here by being filtered is actually intended
+ behavior, so we can be comfortable of not breaking something else that
+ expects the property to be there.
+
+ [Other Info]
+
+ None.
** Also affects: cinder (Ubuntu)
Importance: Undecided
Status: New
** Also affects: cinder (Ubuntu Oracular)
Importance: Undecided
Status: New
** Also affects: cinder (Ubuntu Noble)
Importance: Undecided
Status: New
** Also affects: cinder (Ubuntu Jammy)
Importance: Undecided
Status: New
** Also affects: cloud-archive
Importance: Undecided
Status: New
** Also affects: cloud-archive/yoga
Importance: Undecided
Status: New
** Also affects: cloud-archive/caracal
Importance: Undecided
Status: New
** Also affects: cloud-archive/bobcat
Importance: Undecided
Status: New
** Also affects: cloud-archive/dalmation
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1823445
Title:
[SRU] MetadataCopyFailure: Failed to copy metadata to volume: Glance
metadata cannot be updated, key signature_verified exists for volume
id <volume id>
To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1823445/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
