Hi Leonid,
Thank you for your interest in contributing to Apache Cloudberry
(incubating) and for offering both implementation support and compute
resources.
Regarding Ubuntu 22.04 Support: We welcome the addition of Ubuntu 22.04
build support. Your offer to provide the necessary Docker files and testing
scripts would be valuable. We're currently in the process of establishing
our CI pipeline with Rocky Linux 8 and 9, so this would be a good time to
incorporate Ubuntu support.
To proceed, we suggest:
1. Create a PR in the cloudberry-devops-release repository with:
- Docker files for Ubuntu 22.04 build and test environments
- Necessary CI workflow modifications to support Ubuntu builds
- Any required testing script adaptations
Regarding Cloud Resources: Thank you for the generous offer of Yandex Cloud
compute resources. However, we're actually moving in the opposite direction
- transitioning from self-hosted AWS VMs to GitHub-hosted runners to
simplify our infrastructure requirements. This change aims to reduce
maintenance overhead and infrastructure dependencies.
While we appreciate the offer, maintaining our focus on GitHub-hosted
runners will help us establish a more standardized and easily maintainable
CI environment.
Best regards,
-=e
On Fri, Nov 15, 2024 at 2:06 AM Leonid Borchuk <le.borc...@gmail.com> wrote:
> Hi Cloudberry Dev Team,
>
> We currently use Ubuntu 22.04 (Jammy Jellyfish) as the host operating
> system for our Cloudberry instances.
>
> We would like to request to add a build step for Jammy, so that we can
> ensure that it builds successfully on our platform. We can provide the
> necessary Docker files and testing scripts ourselves.
>
> Additionally, we have some free compute resources available in our cloud
> https://yandex.cloud/en/services/compute , which could be used for your
> build and testing process. This would be at no cost. We use Terraform for
> our infrastructure, although there may be some differences between it and
> AWS. However, I believe that it would not be difficult to migrate from AWS
> to YC.
>
> On Thu, Nov 14, 2024 at 12:09 PM Ed Espino <esp...@apache.org> wrote:
>
> > Hi Cloudberry Dev Team,
> >
> > I wanted to provide some additional important details about the CI
> > implementation that I forgot to mention in my previous email.
> >
> > The CI build process will also:
> >
> > - Create a development RPM package for el9 (Rocky Linux 9) that will
> be
> > available for download from the workflow page
> > - Use this RPM in the testing phase
> > - Utilize a minimal testing container to validate that:
> > - The RPM correctly pulls in all required runtime dependencies
> > - Package installation works as expected
> > - installcheck can execute successfully in a clean environment
> >
> > This minimal testing container approach helps ensure that we're properly
> > capturing all runtime dependencies and that the package will work
> correctly
> > in production environments.
> >
> > A significant infrastructure change is that we plan to execute these
> > workflows on GitHub-hosted runners instead of the previously used
> > self-hosted AWS VMs. This shift will simplify our infrastructure
> > requirements and maintenance overhead.
> >
> > Regarding test suite execution, there are some important considerations:
> >
> > Currently Disabled Tests:
> >
> > - cbdb_parallel
> > - instr_in_shmem_verify
> >
> > These test suites are currently commented out as they're experiencing
> > execution issues in the GitHub-hosted runner environment. We'll need to
> > review their implementation to determine what modifications are needed to
> > make them compatible.
> >
> > As we move forward with the CI implementation, we anticipate we may
> > encounter flaky tests. To maintain development velocity, we'll need to
> > prioritize addressing any test stability issues quickly when they arise.
> > This will be crucial for maintaining an efficient development workflow
> for
> > our community.
> >
> > Best regards,
> > -=e
> >
> > On Wed, Nov 13, 2024 at 11:43 PM Ed Espino <esp...@apache.org> wrote:
> >
> > > Hi Cloudberry Dev Team,
> > >
> > > I'd like to discuss a series of planned Pull Requests aimed at
> > > implementing Continuous Integration (CI) support for Apache Cloudberry.
> > The
> > > changes will be introduced incrementally across multiple repositories
> to
> > > establish a robust CI pipeline.
> > >
> > > The implementation will come in two main phases:
> > >
> > > 1. Docker Container Infrastructure (cloudberry-devops-release)
> > > - Introducing Docker containers for building and testing
> Cloudberry
> > > on Rocky Linux 9
> > > - Publishing build artifacts to DockerHub
> > > - Adding GitHub workflows to automate container building process
> > > - Repository:
> https://github.com/apache/cloudberry-devops-release
> > > 2. CI Pipeline Implementation (cloudberry main repository)
> > > - Implementing CI workflows for building and testing on Rocky
> Linux
> > > 9
> > > - Initial focus on unittest-check and installcheck-good (with
> > > Optimizer disabled)
> > > - Adding new GitHub workflow configurations
> > > - Introducing supporting CI scripts from
> cloudberry-devops-release
> > > - Repository: https://github.com/apache/cloudberry
> > >
> > > As part of this effort, we will also:
> > >
> > > - Clean up legacy build scripts to reduce technical debt
> > > - Evaluate moving the build scripts back to the main cloudberry
> > > repository
> > >
> > > The CI implementation will be actively developed over the next few
> months
> > > to support our initial release processes. We'll be using simplified CI
> > > scripts initially, with plans to expand functionality as we progress.
> > >
> > > The current separation between devops-release and main repository
> allows
> > > us to:
> > >
> > > - Maintain clear separation of concerns
> > > - Iterate on CI infrastructure independently
> > > - Reduce complexity in the main repository
> > > - Enable easier maintenance and updates of CI components
> > >
> > > I welcome any feedback or suggestions on this approach before I begin
> > > submitting the PRs.
> > >
> > > Best regards,
> > > -=e
> > > --
> > > Ed Espino
> > > Apache Cloudberry (incubating) & MADlib
> > >
> >
> >
> > --
> > Ed Espino
> > Apache Cloudberry (incubating) & MADlib
> >
>
--
Ed Espino
Apache Cloudberry (incubating) & MADlib
