iting0321 opened a new pull request, #3723: URL: https://github.com/apache/polaris/pull/3723
## Description ### Summary This PR adds first-class support for Apache Paimon tables in the Polaris Spark Catalog plugin, enabling users to manage Paimon tables alongside Iceberg, Delta Lake, and Hudi tables through a unified catalog interface. ### How It Works Paimon tables are stored using the existing **Generic Table API** with `format: "paimon"`. The implementation follows the same pattern as Delta and Hudi support: 1. Tables created with `USING paimon` are routed through `PaimonHelper` 2. `PaimonHelper` dynamically loads the Paimon Spark Catalog (`org.apache.paimon.spark.SparkCatalog`) 3. Paimon catalog delegates to `PolarisSparkCatalog` for metadata management 4. Paimon handles format-specific operations (snapshots, manifests, etc.) ### Usage Example ```sql -- Configure Spark with Polaris and Paimon spark.sql.catalog.polaris = org.apache.polaris.spark.SparkCatalog spark.sql.catalog.polaris.uri = http://localhost:8181/api/catalog spark.jars.packages = org.apache.paimon:paimon-spark-3.5:1.0.0 -- Create a Paimon table through Polaris CREATE TABLE polaris.my_namespace.my_paimon_table ( id INT, name STRING ) USING paimon LOCATION 's3://bucket/path/to/table'; -- Query works seamlessly SELECT * FROM polaris.my_namespace.my_paimon_table; ``` --- ## Checklist - [x] ๐ก๏ธ **Don't disclose security issues!** - N/A, this is a feature addition - [x] ๐ **Clearly explained why the changes are needed** - Enables multi-format catalog support as requested by users who want unified catalog management - [x] ๐งช **Added/updated tests with good coverage** - Added `paimon` to parameterized test `testCreateAndLoadGenericTable`, created `NoopPaimonCatalog` mock, added Paimon format detection mock in test setup - [x] ๐ก **Added comments for complex logic** - Added Javadoc comments explaining PaimonHelper's purpose and the delegation pattern - [ ] ๐งพ **Updated CHANGELOG.md** - *To be added if required* - [ ] ๐ **Updated documentation** - *Documentation update for site/content may be needed* --- ## Testing ### Unit Tests ```bash ./gradlew :polaris-spark-3.5_2.12:test --tests "org.apache.polaris.spark.SparkCatalogTest" ``` --- ## Related Issues #3343 - Support Apache Paimon as a first-class citizen in the catalog -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
