Hi everyone,
As the Gravitino community grows, we are seeing more requests to
support various new catalogs. While this is great news, adding
every new catalog directly to the core module brings some
challenges: it increases the maintenance burden, makes our CI slower,
and
makes the release package excessively large.
However, we don't want to turn away contributors or stop supporting
new data sources.
To solve this, I propose introducing a new module called
catalogs-contrib. The main idea is to separate "core" catalogs from
"community/vendor-specific" ones.
Here is a quick summary of the proposal:
- New Directory: We will create a catalogs-contrib/ folder in the
main repo. Core catalogs (like Hive, Iceberg) stay in catalogs/,
while others (like OceanBase or experimental ones) go into
catalogs-contrib/.
- Simplified CI: To save resources and reduce flakiness, catalogs
in catalogs-contrib will only run unit tests in CI, not the full
integration tests.
- Distribution Package: We will build & generate two binary
packages and only release one: (To be discussed)
- Standard package: Contains only core catalogs (smaller size).
This is the final release package.
- Extended package: Contains everything, including
catalogs-contrib (batteries included). This one will not be released,
and users can build it if needed.
This way, we can welcome more contributions without slowing down
the core development or breaking the binary size limit.
You can read the full proposal here:
https://docs.google.com/document/d/159jBRfujZ4CVE6jZi6FguBoBAMfRU4ORAYMoyIsKFlU/edit?tab=t.0
Looking forward to your feedback!
Best regards, Qi Yu
