rahil-c commented on code in PR #1862:
URL: https://github.com/apache/polaris/pull/1862#discussion_r2180581211
##########
plugins/spark/v3.5/spark/build.gradle.kts:
##########
@@ -46,6 +46,47 @@ dependencies {
// TODO: extract a polaris-rest module as a thin layer for
// client to depends on.
implementation(project(":polaris-core")) { isTransitive = false }
+ implementation(project(":polaris-api-iceberg-service")) {
+ // exclude the iceberg dependencies, use the ones pulled
+ // by iceberg-core
+ exclude("org.apache.iceberg", "*")
+ // exclude all cloud and quarkus specific dependencies to avoid
+ // running into problems with signature files.
+ exclude("com.azure", "*")
+ exclude("software.amazon.awssdk", "*")
+ exclude("com.google.cloud", "*")
+ exclude("io.airlift", "*")
+ exclude("io.smallrye", "*")
+ exclude("io.smallrye.common", "*")
+ exclude("io.swagger", "*")
+ exclude("org.apache.commons", "*")
+ }
+ implementation(project(":polaris-api-catalog-service")) {
+ exclude("org.apache.iceberg", "*")
+ exclude("com.azure", "*")
+ exclude("software.amazon.awssdk", "*")
+ exclude("com.google.cloud", "*")
+ exclude("io.airlift", "*")
+ exclude("io.smallrye", "*")
+ exclude("io.smallrye.common", "*")
+ exclude("io.swagger", "*")
+ exclude("org.apache.commons", "*")
+ }
+ implementation(project(":polaris-core")) {
+ exclude("org.apache.iceberg", "*")
+ exclude("com.azure", "*")
+ exclude("software.amazon.awssdk", "*")
+ exclude("com.google.cloud", "*")
+ exclude("io.airlift", "*")
+ exclude("io.smallrye", "*")
+ exclude("io.smallrye.common", "*")
+ exclude("io.swagger", "*")
+ exclude("org.apache.commons", "*")
+ }
+
+ implementation("org.apache.iceberg:iceberg-core:${icebergVersion}")
+ compileOnly("org.apache.hudi:hudi-spark3.5-bundle_${scalaVersion}:0.15.0")
Review Comment:
Currently I am directly using this one hudi class
`import org.apache.spark.sql.hudi.catalog.HoodieInternalV2Table;` within the
`PolarisCatalogUtils`
https://github.com/apache/polaris/pull/1862/files#diff-f351cfc050ac63c907c7f35f8052eb88ee6241c39f95a8b81df3d895349bafa0R139
So we will need to keep a provided/compileOnly dependency, in order to
ensure that compilation succeeds, as the user will anyway have to provide the
hudi spark bundle directly like this
https://hudi.apache.org/docs/0.15.0/quick-start-guide
If we are still concerned of having the bundle as provided/compileOnly
dependency, I can try to narrow to just the exact hudi spark dep that contains
this class which is hudi-spark3.2plus-common based on what I see in intellij
and try declaring this instead.
<img width="537" alt="Screenshot 2025-07-02 at 10 06 46 AM"
src="https://github.com/user-attachments/assets/168879af-9257-4006-9c47-971ad207b8fe";
/>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
