GitHub user tuhaihe created a discussion: [Proposal] Consolidate CI Workflows
Using Matrix Strategy
### Proposers
@tuhaihe
### Proposal Status
Under Discussion
### Abstract
This proposal suggests consolidating our current four separate CI workflow
files (Rocky 8, Rocky 9, Ubuntu 22.04, Ubuntu 24.04 - `coming Rocky 10`) into
two unified workflows using GitHub Actions matrix strategy - one for RPM-based
distributions and one for DEB-based distributions.
### Motivation
This consolidation aligns with GitHub Actions best practices and will make our
CI infrastructure more maintainable and scalable without increasing PR
validation time. The matrix strategy is specifically designed for this use case.
This consolidation is now feasible because our project has evolved to use
unified build scripts (`devops/build/automation/cloudberry/scripts/`) that work
consistently across different OS versions, rather than requiring OS-specific
configurations. The workflows primarily differ only in the container images
used, making them ideal candidates for matrix strategy.
### Implementation
## Current State
We currently maintain four separate workflow files:
| Workflow File | OS | Trigger on PR | Trigger on Push | Notes |
|--------------|-----|---------------|-----------------|-------|
| `build-cloudberry.yml` | Rocky 9 | Yes | Yes | Primary RPM workflow |
| `build-cloudberry-rocky8.yml` | Rocky 8 | No (only on workflow file changes)
| Yes | Secondary RPM workflow |
| `build-deb-cloudberry.yml` | Ubuntu 22.04 | Yes | Yes | Primary DEB workflow |
| `build-deb-cloudberry-ubuntu24.04.yml` | Ubuntu 24.04 | No (only on workflow
file changes) | Yes | Secondary DEB workflow |
### Issues with Current Approach
1. **Code Duplication**: The Rocky 8/9 workflows are nearly identical (~1,900
lines each), as are the Ubuntu 22.04/24.04 workflows (~1,900 lines each). This
leads to:
- Maintenance burden when updating test configurations
- Risk of inconsistencies between versions
- Copy-paste errors
2. **Inconsistent PR Coverage**: Rocky 8 and Ubuntu 24.04 are not tested during
PR reviews, potentially missing compatibility issues until after merge.
3. **Scalability**: Adding more newer OS version support would require creating
yet another ~1,000 line workflow file.
## Proposed Solution
Consolidate into two matrix-based workflows:
### 1. `build-cloudberry-rpm.yml` (RPM-based distributions)
```yaml
name: Apache Cloudberry RPM Build
on:
push:
branches: [main, REL_2_STABLE]
pull_request:
branches: [main, REL_2_STABLE]
types: [opened, synchronize, reopened, edited]
workflow_dispatch:
jobs:
build:
strategy:
matrix:
os_version: [rocky8, rocky9]
# Future: rocky10 can be added here
container:
image: apache/incubator-cloudberry:cbdb-build-${{ matrix.os_version
}}-latest
```
### 2. `build-cloudberry-deb.yml` (DEB-based distributions)
```yaml
name: Apache Cloudberry DEB Build
on:
push:
branches: [main, REL_2_STABLE]
pull_request:
branches: [main, REL_2_STABLE]
types: [opened, synchronize, reopened, edited]
workflow_dispatch:
jobs:
build:
strategy:
matrix:
os_version: [ubuntu22.04, ubuntu24.04]
container:
image: apache/incubator-cloudberry:cbdb-build-${{ matrix.os_version
}}-latest
```
## Benefits
**1. No Increase in PR CI Time.** GitHub Actions matrix jobs run **in
parallel**, not sequentially. Same total time, more coverage
**2. Reduced Maintenance Burden.** When updating test configurations, changes
only need to be made once per package type (RPM/DEB) instead of twice.
**3. Better Test Coverage.** All supported OS versions will be tested on every
PR, catching compatibility issues earlier in the development cycle.
**4. Easier to Scale.**
Adding Rocky 10 support becomes trivial:
```yaml
matrix:
os_version: [rocky8, rocky9, rocky10] # Just add one line
```
**5. Consistency Guaranteed.** Since all OS versions use the same workflow
logic, they're guaranteed to run identical tests with identical configurations
(except for OS-specific differences).
### Rollout/Adoption Plan
Love to hear your thoughts and feedback!
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
GitHub link: https://github.com/apache/cloudberry/discussions/1696
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
