This is an automated email from the ASF dual-hosted git repository.
davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel.git
The following commit(s) were added to refs/heads/main by this push:
new 56237588cb0b CAMEL-26492: check multi-arch manifest before proposing
container upgrades (#29149)
56237588cb0b is described below
commit 56237588cb0b30684388f1944335bcd8f4d4c58c
Author: Guillaume Nodet <[email protected]>
AuthorDate: Fri Nov 6 10:28:07 2026 +0100
CAMEL-26492: check multi-arch manifest before proposing container upgrades
(#29149)
* CAMEL-26492: check multi-arch manifest before proposing container upgrades
The container version checker proposed upgrades even when the new image
tag lacked linux/s390x or linux/ppc64le support, breaking CI on those
architectures.
Changes:
- Add a module-level REQUIRED_ARCHITECTURES constant ['linux/s390x',
'linux/ppc64le']
- Add ContainerRegistryAPI.check_architectures(image, tag) that calls the
Docker Registry v2 manifest list endpoint (GET /v2/<repo>/manifests/<tag>
with Accept: application/vnd.docker.distribution.manifest.list.v2+json)
and returns the required architectures absent from the manifest.
- DockerHubAPI overrides check_architectures to obtain an anonymous Bearer
token from auth.docker.io before fetching the manifest.
- QuayAPI overrides check_architectures to use the quay.io/v2/... endpoint.
- ContainerImage gains a has_arch_override flag that is set when the
properties file already declares an arch-specific sibling (e.g.
keycloak.container.ppc64le). Such images are exempt from the check.
- VersionCheckResult gains an arch_missing field. When the top candidate
is missing required architectures, newer_versions is left empty so no
PR is opened for that image; arch_missing is populated for reporting.
- print_results shows a dedicated 'ARCH-INCOMPLETE IMAGES' section.
- output_json includes arch_missing in the JSON payload.
- Add --skip-arch-check CLI flag to bypass the verification.
- GHA workflow step summary reports the number of arch-blocked images.
- Add test_check_container_versions.py with 19 unit tests covering all
new code paths.
* CAMEL-26492: address review issues
* CAMEL-26492: Address review issues in multi-arch container version checker
- Fix mirror.gcr.io manifest checks: add GCRAPI.check_architectures override
that uses the image's original registry (mirror.gcr.io) instead of
normalising
to gcr.io, which returns 401 for public images
- Fix single-arch manifests: a response with no 'manifests' key (schema1 or
single-arch) now correctly reports all REQUIRED_ARCHITECTURES as missing,
rather than silently allowing the upgrade
- Detect skipITs.<arch> from component poms: add
_build_component_skip_its_map
that pre-scans all pom.xml files in the scan path at startup, building a
map
from camel-test-infra-X artifact IDs to their consumer component's skipped
arches. parse_properties_file merges these with the local pom walk-up
result.
- Skip arch-override sibling keys (e.g. keycloak.container.ppc64le) from
being
parsed as standalone images; they are arch-specific registry references,
not
independent images to be version-checked
- Remove duplicate 'import re as _re' inside functions; use the top-level
're'
- Tests: rename test_returns_empty_on_single_arch_manifest to
test_returns_missing_arches_on_single_arch_manifest and update assertion;
add GCRAPICheckArchitecturesTest (mirror.gcr.io, single-arch);
add ComponentSkipItsMapTest (3 cases for _build_component_skip_its_map)
Total: 28 → 33 unit tests
* CAMEL-26492: Address remaining review issues (skipITs intersection,
repo-root scan, unverified warning)
- Fix Issue 1: Build the component skipITs map from the repository root
(not just scan_path) so that component poms under components/ are found
even when scanning test-infra/. Added _find_repo_root() helper that walks
up to the directory containing components/ or .git. Also filter out
test-infra module poms that declare their own artifactId (not consumers).
- Fix Issue 2: Change _build_component_skip_its_map to use intersection
instead of union across consumer modules. An arch is now only exempted
when EVERY consumer pom that references a test-infra module skips it.
This prevents camel-test-infra-postgres from losing s390x just because
camel-pgvector skips it while camel-sql, camel-jta etc. still need it.
- Fix Issue 3: Add arch_unverified field to VersionCheckResult. When
check_architectures cannot verify (network error, auth failure, unknown
format), it now returns None instead of []. The upgrade is still proposed
but arch_unverified=true is set, which triggers a warning in the step
summary and human-readable output. Also added to JSON output.
Co-authored-by: Claude <[email protected]>
* CAMEL-26492: Address remaining review issues (camel-test-infra-all
exemption, offline tests, main guard)
* CAMEL-26492: use relative path parts to avoid false positives in checkout
dirs
* CAMEL-26492: fix RunCheckRepoRootTest docstring and remove tautological
assertion
The class docstring now says "via _find_repo_root" to match what it actually
tests (repo-root discovery when scan_path is test-infra). The description in
davsclaus's review pointed out it claimed to cover the camel-test-infra-all
exemption, which is covered by
ComponentSkipItsMapTest.test_test_infra_all_pom_is_ignored.
The final assertIn("linux/s390x", es_result.image.skipped_arches) only
verified the mocked return value (which always has s390x in skipped_arches),
so it could never catch a regression. The _component_skip_its_map assertions
above it are the meaningful checks.
---------
Co-authored-by: Claude <[email protected]>
---
.../check-container-versions.py | 537 +++++++++++++-
.../test_check_container_versions.py | 779 +++++++++++++++++++++
.github/workflows/check-container-versions.yml | 13 +
.github/workflows/pr-ci-scripts-validation.yml | 5 +
4 files changed, 1327 insertions(+), 7 deletions(-)
diff --git
a/.github/actions/check-container-upgrade/check-container-versions.py
b/.github/actions/check-container-upgrade/check-container-versions.py
index 9716313c92d7..1889df67c8eb 100755
--- a/.github/actions/check-container-upgrade/check-container-versions.py
+++ b/.github/actions/check-container-upgrade/check-container-versions.py
@@ -21,7 +21,10 @@
Container Version Checker for Apache Camel Test Infrastructure
This script scans all container.properties files in the test-infra modules and
checks
-for newer versions of container images in their respective registries.
+for newer versions of container images in their respective registries. For
images
+without architecture-specific overrides, it also verifies that the new version
is
+available as a multi-arch manifest for the required architectures (linux/s390x
and
+linux/ppc64le) before proposing an upgrade.
Supported Registries:
- Docker Hub (docker.io, registry-1.docker.io)
@@ -87,6 +90,7 @@ Options:
--json Output results in JSON format
--check-prereleases Include pre-release versions in checks
--registry-timeout Registry API timeout in seconds (default: 30)
+ --skip-arch-check Skip multi-arch manifest verification
--help, -h Show this help message
Dependencies:
@@ -204,6 +208,9 @@ class ContainerImage:
version_include: List[str] = None # Whitelist: version must contain one
of these words
version_exclude: List[str] = None # Blacklist: version must not contain
any of these words
version_freeze_major: bool = False # Lock to same major version when True
+ has_arch_override: bool = False # True when arch-specific properties
exist for this image
+ arch_overrides: List[str] = None # Specific arches that have override
properties (e.g. ['linux/ppc64le'])
+ skipped_arches: List[str] = None # Arches skipped by skipITs.* in the
module pom.xml
def __post_init__(self):
"""Initialize default values for optional fields."""
@@ -211,6 +218,10 @@ class ContainerImage:
self.version_include = []
if self.version_exclude is None:
self.version_exclude = []
+ if self.arch_overrides is None:
+ self.arch_overrides = []
+ if self.skipped_arches is None:
+ self.skipped_arches = []
# Infer structural pattern from current version (not a dataclass
field, excluded from asdict)
self._version_pattern = infer_version_pattern(self.current_version)
@@ -274,6 +285,16 @@ class VersionCheckResult:
newer_versions: List[str]
is_latest: bool
error: Optional[str] = None
+ arch_missing: Optional[List[str]] = None # Architectures absent from the
manifest list of latest_version
+ arch_unverified: bool = False # True when the multi-arch check could not
be performed (network error, unknown format)
+
+ def __post_init__(self):
+ if self.arch_missing is None:
+ self.arch_missing = []
+
+# Architectures that must be present in a multi-arch manifest before proposing
an upgrade.
+# Images with explicit arch-specific property overrides are exempt from this
check.
+REQUIRED_ARCHITECTURES = ['linux/s390x', 'linux/ppc64le']
class ContainerRegistryAPI:
"""Base class for container registry API interactions."""
@@ -297,9 +318,150 @@ class ContainerRegistryAPI:
return "registry-1.docker.io"
return registry
+ def _repo_name(self, image: ContainerImage) -> str:
+ """Build the repository path (namespace/name or just name)."""
+ if image.namespace:
+ return f"{image.namespace}/{image.name}"
+ return image.name
+
+ @staticmethod
+ def _parse_manifest_list(data: dict) -> Optional[List[str]]:
+ """Return present os/arch strings from a manifest-list payload, or
None for single-arch.
+
+ Returns a list when the payload is a manifest list (OCI image index or
+ Docker manifest list), or None when it is a single-arch manifest (no
+ ``manifests`` key, or a schema1 manifest with a top-level
``architecture``
+ field). The caller treats None as "single-arch / unverifiable".
+ """
+ manifests = data.get('manifests', [])
+ if not manifests:
+ return None
+ return [
+ f"{m.get('platform', {}).get('os', '')}/{m.get('platform',
{}).get('architecture', '')}"
+ for m in manifests
+ ]
+
+ def _fetch_manifest(self, url: str, token: str = '') -> Optional[dict]:
+ """GET a manifest URL with the multi-arch Accept header and optional
Bearer token.
+
+ Returns the parsed JSON on HTTP 200, None otherwise (non-200 or
network error).
+ """
+ headers = {
+ 'Accept': (
+ 'application/vnd.docker.distribution.manifest.list.v2+json,'
+ 'application/vnd.oci.image.index.v1+json'
+ )
+ }
+ if token:
+ headers['Authorization'] = f"Bearer {token}"
+ try:
+ response = self.session.get(url, headers=headers,
timeout=self.timeout)
+ if response.status_code == 200:
+ return response.json()
+ # Some registries (ghcr.io, docker.elastic.co) return 401 and a
+ # Www-Authenticate challenge even for public images. Attempt a
+ # token exchange using the challenge header before giving up.
+ if response.status_code == 401:
+ www_auth = response.headers.get('Www-Authenticate', '')
+ bearer_token = self._bearer_token_from_challenge(www_auth)
+ if bearer_token:
+ headers['Authorization'] = f"Bearer {bearer_token}"
+ retry = self.session.get(url, headers=headers,
timeout=self.timeout)
+ if retry.status_code == 200:
+ return retry.json()
+ return None
+ except Exception:
+ return None
+
+ def _bearer_token_from_challenge(self, www_auth: str) -> str:
+ """Parse a Bearer Www-Authenticate challenge and fetch an anonymous
token.
+
+ Handles the form:
+ Bearer realm="https://...",service="...",scope="...";
+
+ Returns the token string, or '' on failure.
+ """
+ if not www_auth.lower().startswith('bearer '):
+ return ''
+ try:
+ realm = re.search(r'realm="([^"]+)"', www_auth)
+ service = re.search(r'service="([^"]+)"', www_auth)
+ scope = re.search(r'scope="([^"]+)"', www_auth)
+ if not realm:
+ return ''
+ params = {}
+ if service:
+ params['service'] = service.group(1)
+ if scope:
+ params['scope'] = scope.group(1)
+ token_resp = self.session.get(realm.group(1), params=params,
timeout=self.timeout)
+ if token_resp.status_code == 200:
+ return token_resp.json().get('token', '')
+ except Exception:
+ pass
+ return ''
+
+ def check_architectures(self, image: ContainerImage, tag: str) ->
Optional[List[str]]:
+ """Return the list of REQUIRED_ARCHITECTURES that are absent from the
manifest for *tag*.
+
+ Uses the Docker Registry v2 manifest list API
+ (``GET /v2/<repo>/manifests/<tag>`` with
+ ``Accept: application/vnd.docker.distribution.manifest.list.v2+json``
+ or its OCI equivalent). Returns an empty list when all required
+ architectures are present. Returns ``None`` when the check cannot be
+ performed (network error, unrecognised format, non-200 after token
+ retry). Returns the full set of required arches when the response is
+ a single-arch manifest (no ``manifests`` key), because a single-arch
+ image is by definition not multi-arch.
+ """
+ try:
+ registry = self.normalize_registry_url(image.registry)
+ repo = self._repo_name(image)
+ url = f"https://{registry}/v2/{repo}/manifests/{tag}";
+ data = self._fetch_manifest(url)
+ if data is None:
+ # Cannot verify (network error, 401 without challenge, …).
+ return None
+ present = self._parse_manifest_list(data)
+ if present is None:
+ # Single-arch manifest: the image is definitely not multi-arch.
+ return list(REQUIRED_ARCHITECTURES)
+ return [arch for arch in REQUIRED_ARCHITECTURES if arch not in
present]
+ except Exception:
+ # Unexpected format or error — cannot verify.
+ return None
+
class DockerHubAPI(ContainerRegistryAPI):
"""Docker Hub registry API implementation."""
+ def check_architectures(self, image: ContainerImage, tag: str) ->
Optional[List[str]]:
+ """Check architectures via registry-1.docker.io, obtaining a Bearer
token first."""
+ try:
+ if not image.namespace or image.namespace == "library":
+ repo_name = f"library/{image.name}"
+ else:
+ repo_name = f"{image.namespace}/{image.name}"
+
+ # Obtain an anonymous Bearer token for the manifest scope
+ token_url = (
+ f"https://auth.docker.io/token";
+
f"?service=registry.docker.io&scope=repository:{repo_name}:pull"
+ )
+ token_resp = self.session.get(token_url, timeout=self.timeout)
+ token = token_resp.json().get('token', '') if
token_resp.status_code == 200 else ''
+
+ url =
f"https://registry-1.docker.io/v2/{repo_name}/manifests/{tag}";
+ data = self._fetch_manifest(url, token=token)
+ if data is None:
+ return None
+ present = self._parse_manifest_list(data)
+ if present is None:
+ # Single-arch manifest: the image is not multi-arch.
+ return list(REQUIRED_ARCHITECTURES)
+ return [arch for arch in REQUIRED_ARCHITECTURES if arch not in
present]
+ except Exception:
+ return None
+
def get_available_versions(self, image: ContainerImage) -> List[str]:
"""Get available versions from Docker Hub."""
try:
@@ -345,6 +507,22 @@ class DockerHubAPI(ContainerRegistryAPI):
class QuayAPI(ContainerRegistryAPI):
"""Quay.io registry API implementation."""
+ def check_architectures(self, image: ContainerImage, tag: str) ->
Optional[List[str]]:
+ """Check architectures using the Quay.io distribution-spec manifest
endpoint."""
+ try:
+ repo_name = self._repo_name(image)
+ url = f"https://quay.io/v2/{repo_name}/manifests/{tag}";
+ data = self._fetch_manifest(url)
+ if data is None:
+ return None
+ present = self._parse_manifest_list(data)
+ if present is None:
+ # Single-arch manifest: the image is not multi-arch.
+ return list(REQUIRED_ARCHITECTURES)
+ return [arch for arch in REQUIRED_ARCHITECTURES if arch not in
present]
+ except Exception:
+ return None
+
def get_available_versions(self, image: ContainerImage) -> List[str]:
"""Get available versions from Quay.io."""
try:
@@ -385,6 +563,26 @@ class QuayAPI(ContainerRegistryAPI):
class GCRAPI(ContainerRegistryAPI):
"""Google Container Registry API implementation."""
+ def check_architectures(self, image: ContainerImage, tag: str) ->
Optional[List[str]]:
+ """Check architectures using the original registry URL.
+
+ ``mirror.gcr.io`` answers manifest requests anonymously (200) while
+ ``gcr.io`` returns 401 for public images. Use the image's registry
as-is
+ (i.e. do NOT normalise) so ``mirror.gcr.io`` manifests are reachable.
+ """
+ try:
+ repo = self._repo_name(image)
+ url = f"https://{image.registry}/v2/{repo}/manifests/{tag}";
+ data = self._fetch_manifest(url)
+ if data is None:
+ return None
+ present = self._parse_manifest_list(data)
+ if present is None:
+ return list(REQUIRED_ARCHITECTURES)
+ return [arch for arch in REQUIRED_ARCHITECTURES if arch not in
present]
+ except Exception:
+ return None
+
def get_available_versions(self, image: ContainerImage) -> List[str]:
"""Get available versions from GCR."""
try:
@@ -576,9 +774,14 @@ class ContainerVersionChecker:
def __init__(self,
include_prereleases: bool = False,
registry_timeout: int = 30,
- verbose: bool = False):
+ verbose: bool = False,
+ skip_arch_check: bool = False):
self.include_prereleases = include_prereleases
self.verbose = verbose
+ self.skip_arch_check = skip_arch_check
+ # Populated by run_check() before processing files; maps
+ # camel-test-infra-X → [skipped linux/arches] from component poms.
+ self._component_skip_its_map: Dict[str, List[str]] = {}
# Initialize registry APIs
self.registry_apis = {
@@ -732,6 +935,136 @@ class ContainerVersionChecker:
return sorted(properties_files)
+ @staticmethod
+ def _read_skip_its_from_pom(container_properties_path: str) -> List[str]:
+ """Return 'linux/<arch>' entries for arches that have
skipITs.<arch>=true in a pom.xml.
+
+ Walks up the directory tree from the container.properties file, finds
the
+ nearest pom.xml, and reads the ``skipITs.ppc64le`` / ``skipITs.s390x``
+ Maven properties. Returns a list of 'linux/ppc64le' and/or
'linux/s390x'
+ for each arch that is set to true. An empty list means nothing is
skipped
+ or the pom.xml could not be found/parsed.
+ """
+ SKIP_ITS_PLATFORMS = {
+ 'skipITs.ppc64le': 'linux/ppc64le',
+ 'skipITs.s390x': 'linux/s390x',
+ 'skipITs.aarch64': 'linux/arm64',
+ }
+ try:
+ path = Path(container_properties_path).resolve().parent
+ # Walk up at most 10 levels to find a pom.xml
+ for _ in range(10):
+ pom = path / 'pom.xml'
+ if pom.exists():
+ content = pom.read_text(encoding='utf-8', errors='replace')
+ skipped = []
+ for prop, platform in SKIP_ITS_PLATFORMS.items():
+ # Match <skipITs.ppc64le>true</skipITs.ppc64le> (with
any whitespace)
+ tag = prop.replace('.', r'\.')
+ m = re.search(rf'<{tag}>\s*(true)\s*</{tag}>',
content, re.IGNORECASE)
+ if m:
+ skipped.append(platform)
+ return skipped
+ parent = path.parent
+ if parent == path:
+ break
+ path = parent
+ except Exception:
+ pass
+ return []
+
+ @staticmethod
+ def _build_component_skip_its_map(scan_path: str) -> Dict[str, List[str]]:
+ """Build a mapping from test-infra artifact IDs to skipped
architectures.
+
+ Scans all pom.xml files in the scan_path that reference a
+ ``camel-test-infra-*`` dependency and reads their ``skipITs.ppc64le``
+ / ``skipITs.s390x`` properties. The result is a dict from artifact ID
+ (e.g. ``camel-test-infra-elasticsearch``) to a list of platform strings
+ (e.g. ``['linux/s390x']``).
+
+ The logic uses **intersection** across all consumer modules: an arch is
+ only considered skipped for a test-infra module when **every** consumer
+ pom that references that module sets ``skipITs.<arch>=true``. This
+ prevents exempting an arch just because one consumer skips it while
+ others still need it (e.g. ``camel-pgvector`` skips s390x but
+ ``camel-sql`` does not, so ``camel-test-infra-postgres`` should still
+ be checked for s390x).
+
+ This handles the case where the skipITs property lives in a *component*
+ pom (e.g. ``components/camel-elasticsearch-rest-client/pom.xml``)
rather
+ than in the test-infra module pom itself.
+ """
+ SKIP_ITS_PLATFORMS = {
+ 'skipITs.ppc64le': 'linux/ppc64le',
+ 'skipITs.s390x': 'linux/s390x',
+ 'skipITs.aarch64': 'linux/arm64',
+ }
+ # First pass: collect per-consumer skipped arches for each test-infra
module.
+ # consumers_map: {artifact_id: [set_of_skipped_arches_for_consumer_1,
…]}
+ consumers_map: Dict[str, List[set]] = {}
+ try:
+ base = Path(scan_path).resolve()
+ exclude_dirs = {'target', 'build', '.git', '__pycache__'}
+ for pom_file in base.rglob('pom.xml'):
+ rel_parts = pom_file.relative_to(base).parts
+ if any(e in rel_parts for e in exclude_dirs):
+ continue
+ # Skip test-infra module poms — they are NOT consumer poms.
+ # e.g. test-infra/camel-test-infra-all/pom.xml declares every
+ # camel-test-infra-* artifact but has no skipITs, which would
+ # veto every exemption via the intersection logic.
+ if any(part.startswith('camel-test-infra-') for part in
rel_parts):
+ continue
+ try:
+ content = pom_file.read_text(encoding='utf-8',
errors='replace')
+ except Exception:
+ continue
+ # Find all camel-test-infra-* artifact references in this pom
+ deps =
re.findall(r'<artifactId>(camel-test-infra-[^<]+)</artifactId>', content)
+ if not deps:
+ continue
+ # Identify the pom's own artifactId to filter it out: a
test-infra
+ # module pom declares its own artifactId which is NOT a
dependency
+ # reference. We detect this by looking for an <artifactId>
that
+ # appears directly after <project> or inside the top-level
block.
+ # As a heuristic, exclude artifacts whose name matches a
directory
+ # component of the pom's path (the module directory is usually
named
+ # after the artifactId).
+ pom_dir_name = pom_file.parent.name
+ # Filter: only keep deps that are actual dependency references,
+ # not the pom's own artifactId declaration.
+ consumer_deps = [d.strip() for d in deps if d.strip() !=
pom_dir_name]
+ if not consumer_deps:
+ continue
+ # Read skipped arches for this pom
+ skipped: set = set()
+ for prop, platform in SKIP_ITS_PLATFORMS.items():
+ tag = prop.replace('.', r'\.')
+ m = re.search(rf'<{tag}>\s*(true)\s*</{tag}>', content,
re.IGNORECASE)
+ if m:
+ skipped.add(platform)
+ for dep in consumer_deps:
+ dep = dep.strip()
+ if dep not in consumers_map:
+ consumers_map[dep] = []
+ consumers_map[dep].append(skipped)
+ except Exception:
+ pass
+
+ # Second pass: intersect — an arch is only skipped when ALL consumers
skip it.
+ result_map: Dict[str, List[str]] = {}
+ for artifact_id, consumer_sets in consumers_map.items():
+ if not consumer_sets:
+ continue
+ # Start with the first consumer's skipped set, intersect with the
rest
+ common = consumer_sets[0]
+ for s in consumer_sets[1:]:
+ common = common & s
+ if common:
+ result_map[artifact_id] = sorted(common)
+ return result_map
+
def parse_properties_file(self, file_path: str) -> List[ContainerImage]:
"""Parse a container.properties file and extract container images."""
images = []
@@ -761,6 +1094,51 @@ class ContainerVersionChecker:
properties[key] = value
+ # Determine which base properties have arch-specific overrides.
+ # An arch override is a property whose name matches
+ # <base>.{ppc64le,s390x,aarch64,amd64,...} (i.e. a known arch
suffix
+ # that makes the update-metadata-version.py skip it via
property_prefix()).
+ # Maps the arch suffix (e.g. 'ppc64le') to 'linux/ppc64le'.
+ ARCH_SUFFIX_TO_PLATFORM = {
+ 'ppc64le': 'linux/ppc64le',
+ 's390x': 'linux/s390x',
+ 'aarch64': 'linux/arm64',
+ 'arm64': 'linux/arm64',
+ 'amd64': 'linux/amd64',
+ 'arm': 'linux/arm',
+ }
+ # Collect base names → set of overridden linux platforms
+ keys_arch_override_platforms: Dict[str, List[str]] = {}
+ for key in properties:
+ for arch, platform in ARCH_SUFFIX_TO_PLATFORM.items():
+ if key.endswith(f'.{arch}'):
+ base = key[:-(len(arch) + 1)]
+ if base not in keys_arch_override_platforms:
+ keys_arch_override_platforms[base] = []
+ if platform not in keys_arch_override_platforms[base]:
+ keys_arch_override_platforms[base].append(platform)
+ break
+
+ # Read skipITs.<arch> from the nearest pom.xml in the same module
tree.
+ # e.g. test-infra/camel-test-infra-observability/pom.xml may set
+ # skipITs.ppc64le=true and skipITs.s390x=true to indicate those
arches
+ # are already known to be unsupported for this module's tests.
+ module_skipped_arches = self._read_skip_its_from_pom(file_path)
+
+ # Also check the component-level skipITs map, which captures cases
where
+ # skipITs.<arch> is defined in a component pom (e.g.
components/camel-elasticsearch-rest-client/pom.xml)
+ # rather than in the test-infra module pom itself. We identify the
+ # test-infra module by extracting its artifact ID from the path.
+ test_infra_module = None
+ for part in Path(file_path).parts:
+ if part.startswith('camel-test-infra-'):
+ test_infra_module = part
+ break
+ if test_infra_module and self._component_skip_its_map:
+ for arch in
self._component_skip_its_map.get(test_infra_module, []):
+ if arch not in module_skipped_arches:
+ module_skipped_arches.append(arch)
+
# Second pass: build ContainerImage objects
processed_keys = set()
@@ -769,6 +1147,18 @@ class ContainerVersionChecker:
if key in processed_keys or '.version.include' in key or
'.version.exclude' in key or '.version.freeze' in key:
continue
+ # Skip arch-override sibling keys (e.g.
keycloak.container.ppc64le):
+ # they point to arch-specific registries that are not
standalone images.
+ is_arch_override_key = False
+ for arch in ARCH_SUFFIX_TO_PLATFORM:
+ if key.endswith(f'.{arch}'):
+ base = key[:-(len(arch) + 1)]
+ if base in properties:
+ is_arch_override_key = True
+ break
+ if is_arch_override_key:
+ continue
+
try:
registry, namespace, name, current_version =
self.parse_container_reference(value)
@@ -792,6 +1182,10 @@ class ContainerVersionChecker:
if freeze_major_key in properties:
version_freeze_major =
properties[freeze_major_key].lower() in ['true', 'yes', '1']
+ # Detect whether this property has arch-specific override
siblings
+ arch_overrides = keys_arch_override_platforms.get(key, [])
+ has_arch_override = bool(arch_overrides)
+
image = ContainerImage(
registry=registry,
namespace=namespace,
@@ -801,7 +1195,10 @@ class ContainerVersionChecker:
file_path=file_path,
version_include=version_include,
version_exclude=version_exclude,
- version_freeze_major=version_freeze_major
+ version_freeze_major=version_freeze_major,
+ has_arch_override=has_arch_override,
+ arch_overrides=arch_overrides,
+ skipped_arches=module_skipped_arches,
)
images.append(image)
processed_keys.add(key)
@@ -814,6 +1211,10 @@ class ContainerVersionChecker:
print(f" Exclude filter: {',
'.join(version_exclude)}")
if version_freeze_major:
print(f" Major version freeze: enabled")
+ if arch_overrides:
+ print(f" Arch-specific overrides present for:
{', '.join(arch_overrides)}")
+ if module_skipped_arches:
+ print(f" skipITs arches (pom.xml): {',
'.join(module_skipped_arches)}")
except ValueError as e:
if self.verbose:
@@ -906,12 +1307,58 @@ class ContainerVersionChecker:
latest_version = sorted_versions[0] if sorted_versions else None
is_latest = current_ver == latest_version or len(newer_versions)
== 0
+ # Multi-arch check: verify the version the workflow will actually
bump to
+ # (latest_version = sorted_versions[0]) exposes the required
architectures
+ # in its manifest list.
+ #
+ # Architectures that are already covered by an explicit
arch-override
+ # property (image.arch_overrides) or skipped for this module's
tests
+ # (image.skipped_arches) are removed from the required set before
checking,
+ # so the check is precise rather than all-or-nothing.
+ arch_missing: List[str] = []
+ arch_unverified = False
+ if (
+ not self.skip_arch_check
+ and newer_versions
+ and latest_version is not None
+ and api is not None
+ ):
+ # Build the set of arches we actually need to check for this
image.
+ exempted = set(image.arch_overrides or []) |
set(image.skipped_arches or [])
+ arches_to_check = [a for a in REQUIRED_ARCHITECTURES if a not
in exempted]
+
+ if arches_to_check:
+ # The workflow bumps to latest_version, so that is what we
verify.
+ candidate = latest_version
+ if self.verbose:
+ print(f" Checking multi-arch manifest for
{image.full_name}:{candidate}...")
+ raw_missing = api.check_architectures(image, candidate)
+ if raw_missing is None:
+ # The check could not be performed (network error,
auth failure, etc.)
+ arch_unverified = True
+ if self.verbose:
+ print(f" ⚠ Multi-arch check could not be
performed for {candidate}")
+ else:
+ # Only report arches we still need to check (not
exempted ones).
+ arch_missing = [a for a in raw_missing if a in
arches_to_check]
+ if arch_missing:
+ if self.verbose:
+ print(f" ⚠ Missing architectures in
{candidate}: {', '.join(arch_missing)}")
+ # Remove versions that lack the required
architectures from the
+ # candidate list so the bot does not propose them
as upgrades.
+ # We only check the top candidate; if it fails we
conservatively
+ # treat the whole newer_versions list as
unavailable.
+ newer_versions = []
+ is_latest = True # Suppress the "outdated"
classification
+
return VersionCheckResult(
image=image,
available_versions=sorted_versions[:10], # Limit to top 10
latest_version=latest_version,
newer_versions=newer_versions[:5], # Limit to top 5 newer
- is_latest=is_latest
+ is_latest=is_latest,
+ arch_missing=arch_missing,
+ arch_unverified=arch_unverified,
)
except Exception as e:
@@ -924,6 +1371,24 @@ class ContainerVersionChecker:
error=str(e)
)
+ @staticmethod
+ def _find_repo_root(scan_path: str) -> str:
+ """Walk up from *scan_path* to find the repository root.
+
+ The root is the first ancestor directory that contains a
``components/``
+ sub-directory (which is where the component poms live) **or** a
``.git``
+ marker. Falls back to *scan_path* if neither is found.
+ """
+ current = Path(scan_path).resolve()
+ for _ in range(20): # safety bound
+ if (current / 'components').is_dir() or (current /
'.git').exists():
+ return str(current)
+ parent = current.parent
+ if parent == current:
+ break
+ current = parent
+ return scan_path
+
def run_check(self, scan_path: str, quiet: bool = False) ->
List[VersionCheckResult]:
"""Run the version check on all container.properties files."""
if not quiet:
@@ -939,6 +1404,17 @@ class ContainerVersionChecker:
if not quiet:
print(f"📁 Found {len(properties_files)} container.properties
files")
+ # Pre-build the component skipITs map so parse_properties_file can use
it.
+ # This handles the common case where skipITs.ppc64le/s390x lives in a
component
+ # pom rather than in the test-infra module pom.
+ # We scan from the repository root (not just scan_path) because the
component
+ # poms that set skipITs.* are typically under components/, which may
be outside
+ # the scan_path (e.g. when scan_path is 'test-infra').
+ repo_root = self._find_repo_root(scan_path)
+ self._component_skip_its_map =
self._build_component_skip_its_map(repo_root)
+ if self.verbose and self._component_skip_its_map:
+ print(f" Built skipITs map for
{len(self._component_skip_its_map)} test-infra modules from component poms
(scanned {repo_root})")
+
all_images = []
# Parse all properties files
@@ -990,7 +1466,9 @@ def print_results(results: List[VersionCheckResult],
verbose: bool = False):
# Separate results into categories
outdated = [r for r in results if not r.is_latest and not r.error and
r.newer_versions]
- up_to_date = [r for r in results if r.is_latest and not r.error]
+ arch_blocked = [r for r in results if r.arch_missing and not r.error]
+ arch_unverified = [r for r in results if r.arch_unverified and not r.error]
+ up_to_date = [r for r in results if r.is_latest and not r.error and not
r.arch_missing]
errors = [r for r in results if r.error]
print(f"\n{Style.BRIGHT}📊 VERSION CHECK SUMMARY{Style.RESET_ALL}")
@@ -999,6 +1477,10 @@ def print_results(results: List[VersionCheckResult],
verbose: bool = False):
print(f"Total images checked: {len(results)}")
print(f"{Fore.GREEN}✅ Up to date: {len(up_to_date)}{Style.RESET_ALL}")
print(f"{Fore.YELLOW}⚠️ Outdated: {len(outdated)}{Style.RESET_ALL}")
+ if arch_blocked:
+ print(f"{Fore.MAGENTA}🏗️ Arch-incomplete (upgrade blocked):
{len(arch_blocked)}{Style.RESET_ALL}")
+ if arch_unverified:
+ print(f"{Fore.YELLOW}⚠️ Multi-arch not verified:
{len(arch_unverified)}{Style.RESET_ALL}")
print(f"{Fore.RED}❌ Errors: {len(errors)}{Style.RESET_ALL}")
if outdated:
@@ -1018,6 +1500,38 @@ def print_results(results: List[VersionCheckResult],
verbose: bool = False):
newer_display.append(f"... (+{len(result.newer_versions) -
3} more)")
print(f" Newer: {Fore.GREEN}{',
'.join(newer_display)}{Style.RESET_ALL}")
+ if arch_blocked:
+ print(f"\n{Style.BRIGHT}{Fore.MAGENTA}🏗️ ARCH-INCOMPLETE IMAGES
(upgrade blocked){Style.RESET_ALL}")
+ print("-" * 50)
+ print(f" These images have a newer version available but it is
missing")
+ print(f" required architectures in its manifest list.")
+ print(f" Upgrade skipped — ensure the image is published for all")
+ print(f" required platforms ({', '.join(REQUIRED_ARCHITECTURES)})
before upgrading,")
+ print(f" or add an arch-specific property override in
container.properties.")
+
+ for result in arch_blocked:
+ image = result.image
+ print(f"\n{Fore.CYAN}{image.property_name}{Style.RESET_ALL}")
+ print(f" File: {os.path.relpath(image.file_path)}")
+ print(f" Current:
{Fore.YELLOW}{image.current_version}{Style.RESET_ALL}")
+ print(f" Latest:
{Fore.MAGENTA}{result.latest_version}{Style.RESET_ALL}")
+ print(f" Missing: {Fore.MAGENTA}{', '.join(result.arch_missing or
[])}{Style.RESET_ALL}")
+
+ if arch_unverified:
+ print(f"\n{Style.BRIGHT}{Fore.YELLOW}⚠️ MULTI-ARCH NOT
VERIFIED{Style.RESET_ALL}")
+ print("-" * 50)
+ print(f" The multi-arch manifest check could not be performed for
these images")
+ print(f" (network error, authentication failure, or unsupported
registry format).")
+ print(f" Upgrades were still proposed but multi-arch availability is
unconfirmed.")
+
+ for result in arch_unverified:
+ image = result.image
+ print(f"\n{Fore.CYAN}{image.property_name}{Style.RESET_ALL}")
+ print(f" File: {os.path.relpath(image.file_path)}")
+ print(f" Current:
{Fore.YELLOW}{image.current_version}{Style.RESET_ALL}")
+ if result.latest_version:
+ print(f" Latest:
{Fore.YELLOW}{result.latest_version}{Style.RESET_ALL}")
+
if errors:
print(f"\n{Style.BRIGHT}{Fore.RED}❌ ERRORS{Style.RESET_ALL}")
print("-" * 15)
@@ -1047,7 +1561,9 @@ def output_json(results: List[VersionCheckResult]) -> str:
'latest_version': result.latest_version,
'newer_versions': result.newer_versions,
'is_latest': result.is_latest,
- 'error': result.error
+ 'error': result.error,
+ 'arch_missing': result.arch_missing or [],
+ 'arch_unverified': result.arch_unverified,
}
json_results.append(json_result)
@@ -1098,6 +1614,12 @@ Examples:
help='Registry API timeout in seconds (default: 30)'
)
+ parser.add_argument(
+ '--skip-arch-check',
+ action='store_true',
+ help='Skip multi-arch manifest verification (default: check
linux/s390x and linux/ppc64le)'
+ )
+
parser.add_argument(
'--test-infra-path',
type=str,
@@ -1141,7 +1663,8 @@ Examples:
checker = ContainerVersionChecker(
include_prereleases=args.check_prereleases,
registry_timeout=args.registry_timeout,
- verbose=args.verbose
+ verbose=args.verbose,
+ skip_arch_check=args.skip_arch_check
)
try:
diff --git
a/.github/actions/check-container-upgrade/test_check_container_versions.py
b/.github/actions/check-container-upgrade/test_check_container_versions.py
new file mode 100644
index 000000000000..10fc546ce0f6
--- /dev/null
+++ b/.github/actions/check-container-upgrade/test_check_container_versions.py
@@ -0,0 +1,779 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Tests for the multi-arch manifest verification added by CAMEL-26492.
+
+Run with: python3 -m unittest test_check_container_versions
+"""
+
+import importlib.util
+import json
+import os
+import tempfile
+import unittest
+from pathlib import Path
+from unittest.mock import MagicMock, patch
+
+HERE = Path(__file__).parent
+
+_spec = importlib.util.spec_from_file_location(
+ "check_container_versions", HERE / "check-container-versions.py"
+)
+checker_mod = importlib.util.module_from_spec(_spec)
+_spec.loader.exec_module(checker_mod)
+
+ContainerImage = checker_mod.ContainerImage
+ContainerVersionChecker = checker_mod.ContainerVersionChecker
+ContainerRegistryAPI = checker_mod.ContainerRegistryAPI
+DockerHubAPI = checker_mod.DockerHubAPI
+QuayAPI = checker_mod.QuayAPI
+REQUIRED_ARCHITECTURES = checker_mod.REQUIRED_ARCHITECTURES
+VersionCheckResult = checker_mod.VersionCheckResult
+
+
+def make_image(registry="mirror.gcr.io", namespace="library", name="postgres",
+ current_version="17.0", property_name="postgres.container",
+ has_arch_override=False, arch_overrides=None,
skipped_arches=None):
+ return ContainerImage(
+ registry=registry,
+ namespace=namespace,
+ name=name,
+ current_version=current_version,
+ property_name=property_name,
+ file_path="/fake/container.properties",
+ has_arch_override=has_arch_override,
+ arch_overrides=arch_overrides if arch_overrides is not None else [],
+ skipped_arches=skipped_arches if skipped_arches is not None else [],
+ )
+
+
+def manifest_response(architectures):
+ """Build a fake manifest list JSON payload for the given os/arch
strings."""
+ manifests = [
+ {"platform": {"os": os_, "architecture": arch_}}
+ for os_, arch_ in (a.split("/") for a in architectures)
+ ]
+ return {"manifests": manifests, "mediaType":
"application/vnd.docker.distribution.manifest.list.v2+json"}
+
+
+class RequiredArchitecturesTest(unittest.TestCase):
+
+ def test_required_architectures_includes_s390x_and_ppc64le(self):
+ self.assertIn("linux/s390x", REQUIRED_ARCHITECTURES)
+ self.assertIn("linux/ppc64le", REQUIRED_ARCHITECTURES)
+
+
+class ContainerRegistryAPICheckArchitecturesTest(unittest.TestCase):
+ """Unit tests for ContainerRegistryAPI.check_architectures()."""
+
+ def _api(self):
+ api = checker_mod.GCRAPI(timeout=5)
+ return api
+
+ def _mock_response(self, payload, status=200):
+ resp = MagicMock()
+ resp.status_code = status
+ resp.json.return_value = payload
+ return resp
+
+ def test_returns_empty_when_all_arches_present(self):
+ api = self._api()
+ image = make_image()
+ with patch.object(api.session, "get",
+ return_value=self._mock_response(
+ manifest_response(["linux/amd64", "linux/s390x",
"linux/ppc64le", "linux/arm64"]))):
+ missing = api.check_architectures(image, "17.1")
+ self.assertEqual([], missing)
+
+ def test_returns_missing_ppc64le(self):
+ api = self._api()
+ image = make_image()
+ with patch.object(api.session, "get",
+ return_value=self._mock_response(
+ manifest_response(["linux/amd64", "linux/s390x",
"linux/arm64"]))):
+ missing = api.check_architectures(image, "17.1")
+ self.assertEqual(["linux/ppc64le"], missing)
+
+ def test_returns_both_missing(self):
+ api = self._api()
+ image = make_image()
+ with patch.object(api.session, "get",
+ return_value=self._mock_response(
+ manifest_response(["linux/amd64",
"linux/arm64"]))):
+ missing = api.check_architectures(image, "17.1")
+ self.assertIn("linux/s390x", missing)
+ self.assertIn("linux/ppc64le", missing)
+
+ def test_returns_none_on_non_200(self):
+ api = self._api()
+ image = make_image()
+ with patch.object(api.session, "get",
+ return_value=self._mock_response({}, status=404)):
+ missing = api.check_architectures(image, "17.1")
+ self.assertIsNone(missing)
+
+ def test_returns_missing_arches_on_single_arch_manifest(self):
+ """A single-arch manifest (no 'manifests' key) must count as
arch-missing, not pass."""
+ api = self._api()
+ image = make_image()
+ # Single-arch manifest has no 'manifests' key
+ with patch.object(api.session, "get",
+ return_value=self._mock_response({"schemaVersion":
2, "layers": []})):
+ missing = api.check_architectures(image, "17.1")
+ # Both required arches must be reported as missing
+ self.assertIn("linux/s390x", missing)
+ self.assertIn("linux/ppc64le", missing)
+
+ def test_returns_none_on_exception(self):
+ api = self._api()
+ image = make_image()
+ with patch.object(api.session, "get", side_effect=Exception("network
error")):
+ missing = api.check_architectures(image, "17.1")
+ self.assertIsNone(missing)
+
+
+class DockerHubAPICheckArchitecturesTest(unittest.TestCase):
+ """DockerHub override: obtains a Bearer token before fetching the
manifest."""
+
+ def _mock_response(self, payload, status=200):
+ resp = MagicMock()
+ resp.status_code = status
+ resp.json.return_value = payload
+ return resp
+
+ def test_uses_token_for_manifest_fetch(self):
+ api = DockerHubAPI(timeout=5)
+ image = make_image(registry="docker.io", namespace="library",
name="postgres")
+ token_resp = self._mock_response({"token": "mytoken"})
+ manifest_resp = self._mock_response(
+ manifest_response(["linux/amd64", "linux/s390x", "linux/ppc64le"])
+ )
+ with patch.object(api.session, "get", side_effect=[token_resp,
manifest_resp]) as mock_get:
+ missing = api.check_architectures(image, "17.1")
+ self.assertEqual([], missing)
+ # Second call must carry the Authorization header
+ _, kwargs = mock_get.call_args_list[1]
+ headers = kwargs.get("headers", {})
+ self.assertIn("Authorization", headers)
+ self.assertIn("mytoken", headers["Authorization"])
+
+ def test_official_image_uses_library_prefix(self):
+ api = DockerHubAPI(timeout=5)
+ image = make_image(registry="docker.io", namespace="", name="postgres")
+ with patch.object(api.session, "get",
+ side_effect=[self._mock_response({"token": "t"}),
+
self._mock_response(manifest_response(["linux/amd64"]))]) as mock_get:
+ api.check_architectures(image, "17.1")
+ manifest_url = mock_get.call_args_list[1][0][0]
+ self.assertIn("library/postgres", manifest_url)
+
+
+class QuayAPICheckArchitecturesTest(unittest.TestCase):
+ """QuayAPI override uses quay.io/v2/<repo>/manifests/<tag>."""
+
+ def _mock_response(self, payload, status=200):
+ resp = MagicMock()
+ resp.status_code = status
+ resp.json.return_value = payload
+ return resp
+
+ def test_uses_v2_manifest_endpoint(self):
+ api = QuayAPI(timeout=5)
+ image = make_image(registry="quay.io", namespace="strimzi",
name="kafka")
+ with patch.object(api.session, "get",
+ return_value=self._mock_response(
+ manifest_response(["linux/amd64", "linux/s390x",
"linux/ppc64le"])
+ )) as mock_get:
+ missing = api.check_architectures(image, "latest-kafka-3.9.1")
+ self.assertEqual([], missing)
+ url = mock_get.call_args[0][0]
+ self.assertIn("quay.io/v2/strimzi/kafka/manifests/latest-kafka-3.9.1",
url)
+
+
+class ParsePropertiesArchOverrideTest(unittest.TestCase):
+ """parse_properties_file must set has_arch_override for images that have
arch siblings."""
+
+ def setUp(self):
+ self.tmp = tempfile.TemporaryDirectory()
+ self.addCleanup(self.tmp.cleanup)
+ self.checker = ContainerVersionChecker()
+
+ def _write(self, content):
+ path = os.path.join(self.tmp.name, "container.properties")
+ with open(path, "w") as f:
+ f.write(content)
+ return path
+
+ def test_has_arch_override_set_when_ppc64le_sibling_present(self):
+ path = self._write(
+ "keycloak.container=mirror.gcr.io/keycloak/keycloak:26.0.0\n"
+
"keycloak.container.ppc64le=icr.io/ppc64le-oss/keycloak-ppc64le:v26.0.0-bv\n"
+ )
+ images = self.checker.parse_properties_file(path)
+ main = next((i for i in images if i.property_name ==
"keycloak.container"), None)
+ self.assertIsNotNone(main)
+ self.assertTrue(main.has_arch_override,
+ "keycloak.container should be flagged because
keycloak.container.ppc64le exists")
+
+ def test_has_arch_override_set_when_s390x_sibling_present(self):
+ path = self._write(
+ "cassandra.container=mirror.gcr.io/cassandra:4.0.0\n"
+ "cassandra.container.s390x=mirror.gcr.io/s390x/cassandra:4.0.0\n"
+ )
+ images = self.checker.parse_properties_file(path)
+ main = next((i for i in images if i.property_name ==
"cassandra.container"), None)
+ self.assertIsNotNone(main)
+ self.assertTrue(main.has_arch_override)
+
+ def test_no_arch_override_when_no_arch_sibling(self):
+ path = self._write(
+ "kafka.container=quay.io/strimzi/kafka:latest-kafka-3.9.1\n"
+ )
+ images = self.checker.parse_properties_file(path)
+ self.assertEqual(1, len(images))
+ self.assertFalse(images[0].has_arch_override)
+
+ def test_arch_override_property_itself_not_in_returned_images(self):
+ """Arch-specific override properties (e.g. keycloak.container.ppc64le)
contain
+ platform-specific registry refs that cannot be parsed as a standard
container
+ ref. Whether they parse or not, the caller only cares about the base
image."""
+ path = self._write(
+ "kafka.container=quay.io/strimzi/kafka:3.9.1\n"
+ "kafka.container.ppc64le=icr.io/ppc64le-oss/kafka-ppc64le:3.9.1\n"
+ )
+ images = self.checker.parse_properties_file(path)
+ names = {i.property_name for i in images}
+ # The base property must be present
+ self.assertIn("kafka.container", names)
+
+
+class CheckImageVersionsArchGatingTest(unittest.TestCase):
+ """check_image_versions must gate upgrades on multi-arch availability."""
+
+ def setUp(self):
+ self.checker = ContainerVersionChecker(skip_arch_check=False)
+ # Replace the GCR API with a mock
+ self.mock_api = MagicMock()
+ self.checker.registry_apis["gcr.io"] = self.mock_api
+ self.checker.registry_apis["mirror.gcr.io"] = self.mock_api
+
+ def test_upgrade_proposed_when_all_arches_present(self):
+ image = make_image(current_version="17.0", has_arch_override=False)
+ self.mock_api.get_available_versions.return_value = ["17.1", "17.0"]
+ self.mock_api.check_architectures.return_value = [] # all good
+
+ result = self.checker.check_image_versions(image)
+
+ self.assertEqual(["17.1"], result.newer_versions)
+ self.assertFalse(result.is_latest)
+ self.assertEqual([], result.arch_missing)
+
+ def test_upgrade_blocked_when_arch_missing(self):
+ image = make_image(current_version="17.0", has_arch_override=False)
+ self.mock_api.get_available_versions.return_value = ["17.1", "17.0"]
+ self.mock_api.check_architectures.return_value = ["linux/ppc64le"]
+
+ result = self.checker.check_image_versions(image)
+
+ self.assertEqual([], result.newer_versions,
+ "No upgrade should be proposed when ppc64le is
missing")
+ self.assertTrue(result.is_latest)
+ self.assertIn("linux/ppc64le", result.arch_missing)
+
+ def test_arch_check_targets_latest_version_not_just_newer(self):
+ """The arch check must verify latest_version (what the workflow bumps
to)."""
+ image = make_image(current_version="17.0", has_arch_override=False)
+ # latest_version = "18.0" (sorted_versions[0]); newer_versions[0]
would be "17.1"
+ self.mock_api.get_available_versions.return_value = ["18.0", "17.1",
"17.0"]
+ self.mock_api.check_architectures.return_value = ["linux/ppc64le"]
+
+ result = self.checker.check_image_versions(image)
+
+ # check_architectures should have been called with "18.0"
(latest_version), not "17.1"
+ called_with_tag = self.mock_api.check_architectures.call_args[0][1]
+ self.assertEqual("18.0", called_with_tag,
+ "Arch check must target latest_version, not
newer_versions[0]")
+
+ def test_partial_arch_override_still_checks_unoverridden_arch(self):
+ """Only ppc64le is overridden → s390x must still be checked and can
block an upgrade."""
+ image = make_image(current_version="17.0", has_arch_override=True,
+ arch_overrides=["linux/ppc64le"])
+ self.mock_api.get_available_versions.return_value = ["17.1", "17.0"]
+ # s390x is missing; ppc64le is covered by arch_override
+ self.mock_api.check_architectures.return_value = ["linux/s390x",
"linux/ppc64le"]
+
+ result = self.checker.check_image_versions(image)
+
+ # Only linux/s390x should block (linux/ppc64le is exempted)
+ self.assertEqual([], result.newer_versions)
+ self.assertIn("linux/s390x", result.arch_missing)
+ self.assertNotIn("linux/ppc64le", result.arch_missing)
+
+ def test_skipped_arches_exempted_from_check(self):
+ """Arches marked skipITs.* in pom.xml must be exempt from the manifest
check."""
+ image = make_image(current_version="17.0", has_arch_override=False,
+ skipped_arches=["linux/s390x", "linux/ppc64le"])
+ self.mock_api.get_available_versions.return_value = ["17.1", "17.0"]
+ self.mock_api.check_architectures.return_value = ["linux/s390x",
"linux/ppc64le"]
+
+ result = self.checker.check_image_versions(image)
+
+ # Both arches are skipped → upgrade should be proposed
+ self.mock_api.check_architectures.assert_not_called()
+ self.assertEqual(["17.1"], result.newer_versions)
+
+ def test_upgrade_proposed_when_has_arch_override(self):
+ """Images with explicit arch overrides for all required arches skip
the multi-arch check."""
+ image = make_image(current_version="17.0", has_arch_override=True,
+ arch_overrides=REQUIRED_ARCHITECTURES)
+ self.mock_api.get_available_versions.return_value = ["17.1", "17.0"]
+ # check_architectures should NOT be called when all required arches
are overridden
+ self.mock_api.check_architectures.return_value = ["linux/s390x",
"linux/ppc64le"]
+
+ result = self.checker.check_image_versions(image)
+
+ self.mock_api.check_architectures.assert_not_called()
+ self.assertEqual(["17.1"], result.newer_versions)
+ self.assertEqual([], result.arch_missing)
+
+ def test_upgrade_proposed_when_skip_arch_check(self):
+ """--skip-arch-check bypasses the manifest verification."""
+ self.checker.skip_arch_check = True
+ image = make_image(current_version="17.0", has_arch_override=False)
+ self.mock_api.get_available_versions.return_value = ["17.1", "17.0"]
+ self.mock_api.check_architectures.return_value = ["linux/s390x"]
+
+ result = self.checker.check_image_versions(image)
+
+ self.mock_api.check_architectures.assert_not_called()
+ self.assertEqual(["17.1"], result.newer_versions)
+
+ def test_arch_missing_included_in_json_output(self):
+ result = VersionCheckResult(
+ image=make_image(),
+ available_versions=["17.1", "17.0"],
+ latest_version="17.1",
+ newer_versions=[],
+ is_latest=True,
+ arch_missing=["linux/s390x"],
+ )
+ json_str = checker_mod.output_json([result])
+ data = json.loads(json_str)
+ self.assertEqual(["linux/s390x"], data[0]["arch_missing"])
+
+
+class GCRAPICheckArchitecturesTest(unittest.TestCase):
+ """GCRAPI.check_architectures must use the original registry
(mirror.gcr.io) not normalise it."""
+
+ def _mock_response(self, payload, status=200):
+ resp = MagicMock()
+ resp.status_code = status
+ resp.json.return_value = payload
+ return resp
+
+ def test_uses_mirror_gcr_io_directly_for_manifest(self):
+ """mirror.gcr.io must be used as-is; normalising to gcr.io causes
401."""
+ GCRAPI = checker_mod.GCRAPI
+ api = GCRAPI(timeout=5)
+ image = make_image(registry="mirror.gcr.io", namespace="apache",
name="kafka")
+ with patch.object(api.session, "get",
+ return_value=self._mock_response(
+ manifest_response(["linux/amd64", "linux/s390x",
"linux/ppc64le"])
+ )) as mock_get:
+ missing = api.check_architectures(image, "3.9.1")
+ self.assertEqual([], missing)
+ url = mock_get.call_args[0][0]
+ self.assertIn("mirror.gcr.io", url, "Manifest URL must use
mirror.gcr.io, not gcr.io")
+ self.assertNotIn("//gcr.io", url)
+
+ def test_single_arch_counts_as_missing(self):
+ """A single-arch manifest (no 'manifests' key) must block the
upgrade."""
+ GCRAPI = checker_mod.GCRAPI
+ api = GCRAPI(timeout=5)
+ image = make_image(registry="mirror.gcr.io", namespace="library",
name="postgres")
+ with patch.object(api.session, "get",
+ return_value=self._mock_response({"schemaVersion":
2, "layers": []})):
+ missing = api.check_architectures(image, "17.1")
+ self.assertIn("linux/s390x", missing)
+ self.assertIn("linux/ppc64le", missing)
+
+
+class ComponentSkipItsMapTest(unittest.TestCase):
+ """_build_component_skip_its_map must find skipITs in component poms that
depend on test-infra."""
+
+ def setUp(self):
+ self.tmp = tempfile.TemporaryDirectory()
+ self.addCleanup(self.tmp.cleanup)
+ self.base = Path(self.tmp.name)
+
+ def _write_pom(self, path: Path, content: str):
+ path.parent.mkdir(parents=True, exist_ok=True)
+ path.write_text(content)
+
+ def test_discovers_skipped_arches_from_component_pom(self):
+ # Component pom references camel-test-infra-elasticsearch and sets
skipITs.s390x
+ self._write_pom(
+ self.base / "components" / "camel-elasticsearch" / "pom.xml",
+ """<project><properties>
+ <skipITs.s390x>true</skipITs.s390x>
+ </properties>
+ <dependencies><dependency>
+ <artifactId>camel-test-infra-elasticsearch</artifactId>
+ </dependency></dependencies></project>"""
+ )
+ result =
ContainerVersionChecker._build_component_skip_its_map(str(self.base))
+ self.assertIn("camel-test-infra-elasticsearch", result)
+ self.assertIn("linux/s390x", result["camel-test-infra-elasticsearch"])
+
+ def test_no_entry_when_no_skip_its(self):
+ self._write_pom(
+ self.base / "components" / "camel-kafka" / "pom.xml",
+ """<project><dependencies><dependency>
+ <artifactId>camel-test-infra-kafka</artifactId>
+ </dependency></dependencies></project>"""
+ )
+ result =
ContainerVersionChecker._build_component_skip_its_map(str(self.base))
+ self.assertNotIn("camel-test-infra-kafka", result)
+
+ def test_intersection_across_consumers_not_union(self):
+ """When multiple consumers reference the same test-infra module, an
arch is only
+ exempt when ALL consumers skip it (intersection), not when any one
does (union)."""
+ # Consumer A skips s390x only; consumer B skips ppc64le only.
+ # Intersection is empty → nothing should be exempted.
+ self._write_pom(
+ self.base / "components" / "camel-es-a" / "pom.xml",
+
"""<project><properties><skipITs.s390x>true</skipITs.s390x></properties>
+ <dependencies><dependency>
+ <artifactId>camel-test-infra-es</artifactId>
+ </dependency></dependencies></project>"""
+ )
+ self._write_pom(
+ self.base / "components" / "camel-es-b" / "pom.xml",
+
"""<project><properties><skipITs.ppc64le>true</skipITs.ppc64le></properties>
+ <dependencies><dependency>
+ <artifactId>camel-test-infra-es</artifactId>
+ </dependency></dependencies></project>"""
+ )
+ result =
ContainerVersionChecker._build_component_skip_its_map(str(self.base))
+ # Intersection of {s390x} and {ppc64le} is empty
+ self.assertEqual(result.get("camel-test-infra-es", []), [])
+
+ def test_intersection_all_consumers_skip_same_arch(self):
+ """When ALL consumers skip the same arch, it appears in the result."""
+ self._write_pom(
+ self.base / "components" / "camel-es-a" / "pom.xml",
+ """<project><properties>
+ <skipITs.s390x>true</skipITs.s390x>
+ <skipITs.ppc64le>true</skipITs.ppc64le>
+ </properties>
+ <dependencies><dependency>
+ <artifactId>camel-test-infra-es</artifactId>
+ </dependency></dependencies></project>"""
+ )
+ self._write_pom(
+ self.base / "components" / "camel-es-b" / "pom.xml",
+
"""<project><properties><skipITs.s390x>true</skipITs.s390x></properties>
+ <dependencies><dependency>
+ <artifactId>camel-test-infra-es</artifactId>
+ </dependency></dependencies></project>"""
+ )
+ result =
ContainerVersionChecker._build_component_skip_its_map(str(self.base))
+ # Intersection: both skip s390x, only one skips ppc64le
+ self.assertIn("linux/s390x", result.get("camel-test-infra-es", []))
+ self.assertNotIn("linux/ppc64le", result.get("camel-test-infra-es",
[]))
+
+ def test_test_infra_all_pom_is_ignored(self):
+ """test-infra/camel-test-infra-all/pom.xml references every module but
must be skipped."""
+ # Component pom that legitimately skips s390x
+ self._write_pom(
+ self.base / "components" / "camel-elasticsearch" / "pom.xml",
+ """<project><properties>
+ <skipITs.s390x>true</skipITs.s390x>
+ </properties>
+ <dependencies><dependency>
+ <artifactId>camel-test-infra-elasticsearch</artifactId>
+ </dependency></dependencies></project>"""
+ )
+ # test-infra-all aggregator pom — references every module, no skipITs
+ self._write_pom(
+ self.base / "test-infra" / "camel-test-infra-all" / "pom.xml",
+ """<project><modules>
+ <module>../camel-test-infra-elasticsearch</module>
+ </modules>
+ <dependencies><dependency>
+ <artifactId>camel-test-infra-elasticsearch</artifactId>
+ </dependency></dependencies></project>"""
+ )
+ result =
ContainerVersionChecker._build_component_skip_its_map(str(self.base))
+ # The -all pom should be ignored; only the component pom counts
+ self.assertIn("camel-test-infra-elasticsearch", result)
+ self.assertIn("linux/s390x", result["camel-test-infra-elasticsearch"])
+
+ def test_consumer_without_skip_its_clears_exemption(self):
+ """A consumer pom with no skipITs.* should clear all exemptions via
intersection."""
+ # Consumer A skips both arches; consumer B has no skipITs at all
+ self._write_pom(
+ self.base / "components" / "camel-pg-a" / "pom.xml",
+ """<project><properties>
+ <skipITs.s390x>true</skipITs.s390x>
+ <skipITs.ppc64le>true</skipITs.ppc64le>
+ </properties>
+ <dependencies><dependency>
+ <artifactId>camel-test-infra-postgres</artifactId>
+ </dependency></dependencies></project>"""
+ )
+ self._write_pom(
+ self.base / "components" / "camel-pg-b" / "pom.xml",
+ """<project>
+ <dependencies><dependency>
+ <artifactId>camel-test-infra-postgres</artifactId>
+ </dependency></dependencies></project>"""
+ )
+ result =
ContainerVersionChecker._build_component_skip_its_map(str(self.base))
+ # Intersection of {s390x, ppc64le} and {} is empty
+ self.assertEqual(result.get("camel-test-infra-postgres", []), [])
+
+
+
+class ReadSkipItsPomTest(unittest.TestCase):
+ """Unit tests for ContainerVersionChecker._read_skip_its_from_pom."""
+
+ def setUp(self):
+ self.tmp = tempfile.TemporaryDirectory()
+ self.addCleanup(self.tmp.cleanup)
+
+ def _write_pom(self, path, content):
+ pom = Path(path) / 'pom.xml'
+ pom.parent.mkdir(parents=True, exist_ok=True)
+ pom.write_text(content)
+ return str(pom.parent)
+
+ def test_ppc64le_and_s390x_set(self):
+ pom_content = """<project><properties>
+ <skipITs.ppc64le>true</skipITs.ppc64le>
+ <skipITs.s390x>true</skipITs.s390x>
+ </properties></project>"""
+ module_dir = self._write_pom(Path(self.tmp.name) / 'module',
pom_content)
+ # container.properties lives inside the module directory
+ props_path = str(Path(module_dir) / 'src' / 'container.properties')
+ result = ContainerVersionChecker._read_skip_its_from_pom(props_path)
+ self.assertIn('linux/ppc64le', result)
+ self.assertIn('linux/s390x', result)
+
+ def test_no_skip_its(self):
+ pom_content = "<project><properties></properties></project>"
+ module_dir = self._write_pom(Path(self.tmp.name) / 'module2',
pom_content)
+ props_path = str(Path(module_dir) / 'container.properties')
+ result = ContainerVersionChecker._read_skip_its_from_pom(props_path)
+ self.assertEqual([], result)
+
+ def test_only_ppc64le_skipped(self):
+ pom_content = """<project><properties>
+ <skipITs.ppc64le>true</skipITs.ppc64le>
+ </properties></project>"""
+ module_dir = self._write_pom(Path(self.tmp.name) / 'module3',
pom_content)
+ props_path = str(Path(module_dir) / 'container.properties')
+ result = ContainerVersionChecker._read_skip_its_from_pom(props_path)
+ self.assertIn('linux/ppc64le', result)
+ self.assertNotIn('linux/s390x', result)
+
+
+class BearerTokenChallengeTest(unittest.TestCase):
+ """Unit tests for ContainerRegistryAPI._bearer_token_from_challenge."""
+
+ def setUp(self):
+ self.api = checker_mod.GCRAPI(timeout=5)
+
+ def test_extracts_token_from_challenge(self):
+ challenge = 'Bearer
realm="https://auth.example.com/token",service="example",scope="pull";'
+ mock_resp = MagicMock()
+ mock_resp.status_code = 200
+ mock_resp.json.return_value = {'token': 'abc123'}
+ with patch.object(self.api.session, 'get', return_value=mock_resp):
+ token = self.api._bearer_token_from_challenge(challenge)
+ self.assertEqual('abc123', token)
+
+ def test_returns_empty_for_non_bearer(self):
+ token = self.api._bearer_token_from_challenge('Basic realm="example"')
+ self.assertEqual('', token)
+
+ def test_returns_empty_when_no_realm(self):
+ token = self.api._bearer_token_from_challenge('Bearer
service="example"')
+ self.assertEqual('', token)
+
+
+class FindRepoRootTest(unittest.TestCase):
+ """_find_repo_root must walk up to the directory containing components/ or
.git."""
+
+ def setUp(self):
+ self.tmp = tempfile.TemporaryDirectory()
+ self.addCleanup(self.tmp.cleanup)
+ self.root = Path(self.tmp.name) / "repo"
+ self.root.mkdir()
+
+ def test_finds_root_by_components_dir(self):
+ (self.root / "components").mkdir()
+ test_infra = self.root / "test-infra"
+ test_infra.mkdir()
+ result = ContainerVersionChecker._find_repo_root(str(test_infra))
+ self.assertEqual(str(self.root), result)
+
+ def test_finds_root_by_git_marker(self):
+ (self.root / ".git").mkdir()
+ test_infra = self.root / "test-infra" / "sub"
+ test_infra.mkdir(parents=True)
+ result = ContainerVersionChecker._find_repo_root(str(test_infra))
+ self.assertEqual(str(self.root), result)
+
+ def test_falls_back_to_scan_path(self):
+ # No components/ or .git above — falls back to scan_path itself
+ deep = self.root / "a" / "b" / "c"
+ deep.mkdir(parents=True)
+ result = ContainerVersionChecker._find_repo_root(str(deep))
+ self.assertEqual(str(deep), result)
+
+
+class RunCheckRepoRootTest(unittest.TestCase):
+ """run_check with scan_path=test-infra must still find skipITs in
components/ via _find_repo_root."""
+
+ def setUp(self):
+ self.tmp = tempfile.TemporaryDirectory()
+ self.addCleanup(self.tmp.cleanup)
+ self.root = Path(self.tmp.name)
+
+ def _write(self, relpath, content):
+ p = self.root / relpath
+ p.parent.mkdir(parents=True, exist_ok=True)
+ p.write_text(content)
+
+ def test_skip_its_found_from_components_when_scanning_test_infra(self):
+ """When scan_path is test-infra, skipITs in components/ must be
discovered."""
+ # Create the repo structure with components/ at root
+ (self.root / "components").mkdir()
+ # Component pom that skips s390x and references
camel-test-infra-elasticsearch
+ self._write(
+ "components/camel-elasticsearch/pom.xml",
+ """<project><properties>
+ <skipITs.s390x>true</skipITs.s390x>
+ <skipITs.ppc64le>true</skipITs.ppc64le>
+ </properties>
+ <dependencies><dependency>
+ <artifactId>camel-test-infra-elasticsearch</artifactId>
+ </dependency></dependencies></project>"""
+ )
+ # Create test-infra module with container.properties
+ self._write(
+
"test-infra/camel-test-infra-elasticsearch/src/main/resources/container.properties",
+
"elasticsearch.container=docker.elastic.co/elasticsearch/elasticsearch:8.17.3\n"
+ )
+ # Create a pom.xml in the test-infra module
+ self._write(
+ "test-infra/camel-test-infra-elasticsearch/pom.xml",
+
"<project><artifactId>camel-test-infra-elasticsearch</artifactId></project>"
+ )
+
+ checker = ContainerVersionChecker(skip_arch_check=True)
+ # Patch check_image_versions to keep the test offline
+ fake_result = VersionCheckResult(
+ image=make_image(
+ registry="docker.elastic.co",
+ namespace="elasticsearch",
+ name="elasticsearch",
+ current_version="8.17.3",
+ property_name="elasticsearch.container",
+ skipped_arches=["linux/s390x", "linux/ppc64le"],
+ ),
+ available_versions=["8.17.3"],
+ latest_version="8.17.3",
+ newer_versions=[],
+ is_latest=True,
+ )
+ with patch.object(checker, 'check_image_versions',
return_value=fake_result):
+ # Scan only test-infra — _find_repo_root should walk up to
self.root
+ scan_path = str(self.root / "test-infra")
+ results = checker.run_check(scan_path, quiet=True)
+
+ # The skipITs map should have been built from components/
+ self.assertIn("camel-test-infra-elasticsearch",
checker._component_skip_its_map)
+ skipped =
checker._component_skip_its_map["camel-test-infra-elasticsearch"]
+ self.assertIn("linux/s390x", skipped)
+ self.assertIn("linux/ppc64le", skipped)
+
+ # The mock result should be returned
+ self.assertEqual(len(results), 1)
+
+
+class ArchUnverifiedTest(unittest.TestCase):
+ """check_image_versions must set arch_unverified when check_architectures
returns None."""
+
+ def setUp(self):
+ self.checker = ContainerVersionChecker(skip_arch_check=False)
+ self.mock_api = MagicMock()
+ self.checker.registry_apis["gcr.io"] = self.mock_api
+ self.checker.registry_apis["mirror.gcr.io"] = self.mock_api
+
+ def test_arch_unverified_set_when_check_returns_none(self):
+ image = make_image(current_version="17.0", has_arch_override=False)
+ self.mock_api.get_available_versions.return_value = ["17.1", "17.0"]
+ self.mock_api.check_architectures.return_value = None # could not
verify
+
+ result = self.checker.check_image_versions(image)
+
+ self.assertTrue(result.arch_unverified,
+ "arch_unverified should be True when
check_architectures returns None")
+ # Upgrade should still be proposed (not blocked)
+ self.assertEqual(["17.1"], result.newer_versions)
+ self.assertFalse(result.is_latest)
+
+ def test_arch_unverified_false_when_check_succeeds(self):
+ image = make_image(current_version="17.0", has_arch_override=False)
+ self.mock_api.get_available_versions.return_value = ["17.1", "17.0"]
+ self.mock_api.check_architectures.return_value = [] # all arches
present
+
+ result = self.checker.check_image_versions(image)
+
+ self.assertFalse(result.arch_unverified)
+
+ def test_arch_unverified_in_json_output(self):
+ result = VersionCheckResult(
+ image=make_image(),
+ available_versions=["17.1", "17.0"],
+ latest_version="17.1",
+ newer_versions=["17.1"],
+ is_latest=False,
+ arch_unverified=True,
+ )
+ json_str = checker_mod.output_json([result])
+ data = json.loads(json_str)
+ self.assertTrue(data[0]["arch_unverified"])
+
+ def test_arch_unverified_false_in_json_by_default(self):
+ result = VersionCheckResult(
+ image=make_image(),
+ available_versions=["17.1"],
+ latest_version="17.1",
+ newer_versions=[],
+ is_latest=True,
+ )
+ json_str = checker_mod.output_json([result])
+ data = json.loads(json_str)
+ self.assertFalse(data[0]["arch_unverified"])
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/.github/workflows/check-container-versions.yml
b/.github/workflows/check-container-versions.yml
index 10aac581495c..476b4c2bdc32 100644
--- a/.github/workflows/check-container-versions.yml
+++ b/.github/workflows/check-container-versions.yml
@@ -401,3 +401,16 @@ jobs:
echo "⚠️ No Pull Requests were created" >> $GITHUB_STEP_SUMMARY
fi
fi
+
+ ARCH_BLOCKED=$(jq '[.[] | select(.arch_missing != null and
(.arch_missing | length) > 0)] | length' versions.json 2>/dev/null || echo "0")
+ if [[ "$ARCH_BLOCKED" -gt 0 ]]; then
+ echo "" >> $GITHUB_STEP_SUMMARY
+ echo "🏗️ **$ARCH_BLOCKED** image(s) have a newer version available
but it lacks linux/s390x or linux/ppc64le support — upgrade skipped." >>
$GITHUB_STEP_SUMMARY
+ echo "To investigate, run the checker locally with \`--verbose\`."
>> $GITHUB_STEP_SUMMARY
+ fi
+
+ ARCH_UNVERIFIED=$(jq '[.[] | select(.arch_unverified == true)] |
length' versions.json 2>/dev/null || echo "0")
+ if [[ "$ARCH_UNVERIFIED" -gt 0 ]]; then
+ echo "" >> $GITHUB_STEP_SUMMARY
+ echo "⚠️ **$ARCH_UNVERIFIED** image(s) could not be verified for
multi-arch availability (network error or unsupported registry). Upgrades were
proposed but linux/s390x and linux/ppc64le support is unconfirmed." >>
$GITHUB_STEP_SUMMARY
+ fi
diff --git a/.github/workflows/pr-ci-scripts-validation.yml
b/.github/workflows/pr-ci-scripts-validation.yml
index 982b1a00560e..1d1f9c8bdd95 100644
--- a/.github/workflows/pr-ci-scripts-validation.yml
+++ b/.github/workflows/pr-ci-scripts-validation.yml
@@ -41,6 +41,11 @@ jobs:
with:
persist-credentials: false
+ - name: Install container upgrade script dependencies
+ shell: bash
+ run: |
+ pip install --only-binary :all: --require-hashes -r
.github/actions/check-container-upgrade/requirements.txt
+
- name: Test the container upgrade scripts
shell: bash
run: |
