I suspect the reason this was not picked up in the SRU test, is possibly related to code from the Squid charm being used in the test instead of the Reef Charm.
The squid charm merged a "tactical fix" to manually install python3-packaging in this change: https://review.opendev.org/c/openstack/charm-ceph-osd/+/918992 But not for Reef, it was originally proposed but abandoned when we thought it wasn't needed for Reef: https://review.opendev.org/c/openstack/charm-ceph-osd/+/919794 The Squid charm supports running/installing reef because you're expected to upgrade the charm before Ceph itself, to orchestrate an upgrade. So both the Reef and Squid charm branches have a test for Reef (tests/bundles/jammy-bobcat). IMHO merging this charm change was a bad idea and it should be reverted once all the packages are fixed. The package should simply have been fixed immediately in the first instance. While I can appreciate this might have been done as a stop-gap to get the charm CI working while the issue was not yet fixed in an SRU, the problem is that we are using the charm tests to verify the SRU of the Ubuntu package which is potentially (and actually, even in the cloud- archive) used by people without the charms, so this is likely to hide such an issue as it did here. It also means we don't have a functional test to actually test that the issue is fixed, both in the Reef and Squid SRUs. I can't quite figure out exactly how this test was done though. The original message said it was tested with the ceph-osd charm tests, but the zaza.openstack.charm_tests.ceph.tests.CephPrometheusTest test listed in the output only exists in charm-ceph-mon. Then those tests all use the reef branch of the charm.. I am guessing maybe since we had to test with bobcat-proposed that the squid branch was used by with openstack- origin overriden to bobcat-proposed or something? Luciano: Would be great if you can clarify/reverse engineer exactly how you managed to do this so we can learn for next time. I also wonder if we'd be better using charmed-openstack-tester or something like that, instead of purely the charm-ceph-mon tests, for validating SRUs? A few possible lessons for future SRU verificaiton: - We need to ensure we verify SRUs with all GIT/charmhub branches of the charm that support a release. So generally that would be both the matching and newer version. It's not sufficient to check with only one of those. - Thinking more about the charm users that are the majority, I think ideally we also need to run both the charmhub stable AND candidate branches for both of those releases. Currently the test bundles use the '/edge' channel (which maps to candidate) and would only test the candidate charm, and won't show up if we're about to release a package that is broken wtith the stable charms. Especially for the latest release of Ceph, due to the Solutions QA process, sometimes the stable channel is lagging the edge channel by weeks or even months. So this is not unlikely. - Using the charm tests to verify the Ubuntu package in general has some limits, in that it may miss scenarios that would still effect non-charm users. I am not proposing we stop using it, but we should be aware of that. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2064717 Title: ceph-volume needs "packaging" and "ceph" modules To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ceph-osd/+bug/2064717/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
