Hi Cloudberry Dev Team,
I wanted to provide some additional important details about the CI
implementation that I forgot to mention in my previous email.
The CI build process will also:
- Create a development RPM package for el9 (Rocky Linux 9) that will be
available for download from the workflow page
- Use this RPM in the testing phase
- Utilize a minimal testing container to validate that:
- The RPM correctly pulls in all required runtime dependencies
- Package installation works as expected
- installcheck can execute successfully in a clean environment
This minimal testing container approach helps ensure that we're properly
capturing all runtime dependencies and that the package will work correctly
in production environments.
A significant infrastructure change is that we plan to execute these
workflows on GitHub-hosted runners instead of the previously used
self-hosted AWS VMs. This shift will simplify our infrastructure
requirements and maintenance overhead.
Regarding test suite execution, there are some important considerations:
Currently Disabled Tests:
- cbdb_parallel
- instr_in_shmem_verify
These test suites are currently commented out as they're experiencing
execution issues in the GitHub-hosted runner environment. We'll need to
review their implementation to determine what modifications are needed to
make them compatible.
As we move forward with the CI implementation, we anticipate we may
encounter flaky tests. To maintain development velocity, we'll need to
prioritize addressing any test stability issues quickly when they arise.
This will be crucial for maintaining an efficient development workflow for
our community.
Best regards,
-=e
On Wed, Nov 13, 2024 at 11:43 PM Ed Espino <esp...@apache.org> wrote:
> Hi Cloudberry Dev Team,
>
> I'd like to discuss a series of planned Pull Requests aimed at
> implementing Continuous Integration (CI) support for Apache Cloudberry. The
> changes will be introduced incrementally across multiple repositories to
> establish a robust CI pipeline.
>
> The implementation will come in two main phases:
>
> 1. Docker Container Infrastructure (cloudberry-devops-release)
> - Introducing Docker containers for building and testing Cloudberry
> on Rocky Linux 9
> - Publishing build artifacts to DockerHub
> - Adding GitHub workflows to automate container building process
> - Repository: https://github.com/apache/cloudberry-devops-release
> 2. CI Pipeline Implementation (cloudberry main repository)
> - Implementing CI workflows for building and testing on Rocky Linux
> 9
> - Initial focus on unittest-check and installcheck-good (with
> Optimizer disabled)
> - Adding new GitHub workflow configurations
> - Introducing supporting CI scripts from cloudberry-devops-release
> - Repository: https://github.com/apache/cloudberry
>
> As part of this effort, we will also:
>
> - Clean up legacy build scripts to reduce technical debt
> - Evaluate moving the build scripts back to the main cloudberry
> repository
>
> The CI implementation will be actively developed over the next few months
> to support our initial release processes. We'll be using simplified CI
> scripts initially, with plans to expand functionality as we progress.
>
> The current separation between devops-release and main repository allows
> us to:
>
> - Maintain clear separation of concerns
> - Iterate on CI infrastructure independently
> - Reduce complexity in the main repository
> - Enable easier maintenance and updates of CI components
>
> I welcome any feedback or suggestions on this approach before I begin
> submitting the PRs.
>
> Best regards,
> -=e
> --
> Ed Espino
> Apache Cloudberry (incubating) & MADlib
>
--
Ed Espino
Apache Cloudberry (incubating) & MADlib
