rdblue commented on code in PR #12095:
URL: https://github.com/apache/iceberg/pull/12095#discussion_r1936437048
##########
spark/v3.5/spark-runtime/NOTICE:
##########
@@ -244,8 +372,63 @@ This binary artifact includes Apache Arrow with the
following in its NOTICE file
--------------------------------------------------------------------------------
-This binary artifact includes Netty buffers with the following in its NOTICE
-file:
+This binary artifact includes Apache Spark 3.5.4 with the following in its
NOTICE file:
Review Comment:
@jbonofre, Spark is not included in the runtime bundle -- I just double
checked the Jar. It is provided by the Spark installation where the runtime
bundle is used. I don't think this is accurate.
For the Spark runtime bundle, we shade the `runtimeClasspath` dependency
configuration and excluded dependencies that are coming from the Spark
installation. Dependencies are excluded from the `implementation` configuration:
```
configurations {
implementation {
exclude group: 'org.apache.spark'
// included in Spark
exclude group: 'org.slf4j'
exclude group: 'org.apache.commons'
exclude group: 'commons-pool'
exclude group: 'commons-codec'
exclude group: 'org.xerial.snappy'
exclude group: 'javax.xml.bind'
exclude group: 'javax.annotation'
exclude group: 'com.github.luben'
exclude group: 'com.ibm.icu'
exclude group: 'org.glassfish'
exclude group: 'org.abego.treelayout'
exclude group: 'org.antlr'
exclude group: 'org.scala-lang'
exclude group: 'org.scala-lang.modules'
}
}
```
Here's the expected set of dependencies from the `runtimeClasspath`
configuration:
```
runtimeClasspath - Runtime classpath of source set 'main'.
+--- project :iceberg-api
| \--- project :iceberg-bundled-guava
+--- project :iceberg-spark:iceberg-spark-3.5_2.12
| +--- project :iceberg-api (*)
| +--- project :iceberg-bundled-guava
| +--- project :iceberg-common
| | \--- project :iceberg-bundled-guava
| +--- project :iceberg-core
| | +--- project :iceberg-api (*)
| | +--- org.apache.avro:avro:1.12.0
| | | +--- com.fasterxml.jackson.core:jackson-core:2.17.2 -> 2.15.2
| | | | \--- com.fasterxml.jackson:jackson-bom:2.15.2 -> 2.18.2
| | | | +---
com.fasterxml.jackson.core:jackson-annotations:2.18.2 (c)
| | | | +--- com.fasterxml.jackson.core:jackson-core:2.18.2
-> 2.15.2 (c)
| | | | +---
com.fasterxml.jackson.core:jackson-databind:2.18.2 -> 2.15.2 (c)
| | | | \---
com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.18.2 (c)
| | | \--- com.fasterxml.jackson.core:jackson-databind:2.17.2 ->
2.15.2
| | | +---
com.fasterxml.jackson.core:jackson-annotations:2.15.2 -> 2.18.2
| | | | \--- com.fasterxml.jackson:jackson-bom:2.18.2 (*)
| | | +--- com.fasterxml.jackson.core:jackson-core:2.15.2 (*)
| | | \--- com.fasterxml.jackson:jackson-bom:2.15.2 -> 2.18.2
(*)
| | +--- project :iceberg-common (*)
| | +--- project :iceberg-bundled-guava
| | +--- io.airlift:aircompressor:0.27 -> 2.0.2
| | +--- org.apache.httpcomponents.client5:httpclient5:5.4.1
| | | +--- org.apache.httpcomponents.core5:httpcore5:5.3.1
| | | \--- org.apache.httpcomponents.core5:httpcore5-h2:5.3.1
| | | \--- org.apache.httpcomponents.core5:httpcore5:5.3.1
| | +--- com.fasterxml.jackson:jackson-bom:2.18.2 (*)
| | +--- com.fasterxml.jackson.core:jackson-core:2.18.2 -> 2.15.2 (*)
| | +--- com.fasterxml.jackson.core:jackson-databind:2.18.2 -> 2.15.2
(*)
| | +--- com.github.ben-manes.caffeine:caffeine:2.9.3
| | | +--- org.checkerframework:checker-qual:3.19.0
| | | \--- com.google.errorprone:error_prone_annotations:2.10.0
| | \--- org.roaringbitmap:RoaringBitmap:1.3.0
| +--- project :iceberg-data
| | +--- project :iceberg-api (*)
| | +--- project :iceberg-bundled-guava
| | +--- project :iceberg-core (*)
| | +--- org.apache.orc:orc-core:1.9.5
| | | +--- org.apache.orc:orc-shims:1.9.5
| | | +--- io.airlift:aircompressor:0.27 -> 2.0.2
| | | +--- org.jetbrains:annotations:17.0.0
| | | \--- org.threeten:threeten-extra:1.7.1
| | \--- org.apache.parquet:parquet-avro:1.15.0
| | +--- org.apache.parquet:parquet-column:1.15.0
| | | +--- org.apache.parquet:parquet-common:1.15.0
| | | | \---
org.apache.parquet:parquet-format-structures:1.15.0
| | | \--- org.apache.parquet:parquet-encoding:1.15.0
| | | \--- org.apache.parquet:parquet-common:1.15.0 (*)
| | +--- org.apache.parquet:parquet-hadoop:1.15.0
| | | +--- org.apache.parquet:parquet-column:1.15.0 (*)
| | | +--- org.apache.parquet:parquet-format-structures:1.15.0
| | | +--- org.apache.parquet:parquet-common:1.15.0 (*)
| | | +--- org.apache.parquet:parquet-jackson:1.15.0
| | | \--- io.airlift:aircompressor:2.0.2
| | \--- org.apache.parquet:parquet-common:1.15.0 (*)
| +--- project :iceberg-orc
| | +--- project :iceberg-api (*)
| | +--- project :iceberg-bundled-guava
| | +--- project :iceberg-common (*)
| | +--- project :iceberg-core (*)
| | +--- org.apache.avro:avro:1.12.0 (*)
| | \--- org.apache.orc:orc-core:1.9.5 (*)
| +--- project :iceberg-parquet
| | +--- project :iceberg-api (*)
| | +--- project :iceberg-bundled-guava
| | +--- project :iceberg-core (*)
| | +--- project :iceberg-common (*)
| | \--- org.apache.parquet:parquet-avro:1.15.0 (*)
| +--- project :iceberg-arrow
| | +--- project :iceberg-api (*)
| | +--- project :iceberg-bundled-guava
| | +--- project :iceberg-core (*)
| | +--- project :iceberg-parquet (*)
| | +--- org.apache.arrow:arrow-vector:15.0.2
| | | +--- org.apache.arrow:arrow-format:15.0.2
| | | | \--- com.google.flatbuffers:flatbuffers-java:23.5.26
| | | +--- org.apache.arrow:arrow-memory-core:15.0.2
| | | +--- com.fasterxml.jackson.core:jackson-core:2.16.0 -> 2.15.2
(*)
| | | +--- com.fasterxml.jackson.core:jackson-annotations:2.16.0 ->
2.18.2 (*)
| | | +--- com.fasterxml.jackson.core:jackson-databind:2.16.0 ->
2.15.2 (*)
| | | +---
com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.16.0 -> 2.18.2
| | | | +---
com.fasterxml.jackson.core:jackson-annotations:2.18.2 (*)
| | | | +--- com.fasterxml.jackson.core:jackson-core:2.18.2 ->
2.15.2 (*)
| | | | +--- com.fasterxml.jackson.core:jackson-databind:2.18.2
-> 2.15.2 (*)
| | | | \--- com.fasterxml.jackson:jackson-bom:2.18.2 (*)
| | | +--- com.google.flatbuffers:flatbuffers-java:23.5.26
| | | \--- org.eclipse.collections:eclipse-collections:11.1.0
| | | \---
org.eclipse.collections:eclipse-collections-api:11.1.0
| | +--- org.apache.arrow:arrow-memory-netty:15.0.2
| | | \--- org.apache.arrow:arrow-memory-core:15.0.2
| | +--- org.apache.parquet:parquet-avro:1.15.0 (*)
| | \--- io.netty:netty-buffer:4.1.117.Final
| | \--- io.netty:netty-common:4.1.117.Final
| +--- org.apache.datasketches:datasketches-java:6.2.0
| | \--- org.apache.datasketches:datasketches-memory:3.0.2
| +--- org.apache.parquet:parquet-column:1.15.0 (*)
| +--- org.apache.parquet:parquet-hadoop:1.15.0 (*)
| +--- org.apache.orc:orc-core:1.9.5 (*)
| +--- org.apache.arrow:arrow-vector:15.0.2 (*)
| \--- com.github.ben-manes.caffeine:caffeine:2.9.3 (*)
+--- project :iceberg-spark:iceberg-spark-extensions-3.5_2.12
| \--- org.roaringbitmap:RoaringBitmap:1.3.0
+--- project :iceberg-aws
| +--- project :iceberg-api (*)
| +--- project :iceberg-bundled-guava
| +--- project :iceberg-common (*)
| +--- project :iceberg-core (*)
| +--- com.github.ben-manes.caffeine:caffeine:2.9.3 (*)
| +--- dev.failsafe:failsafe:3.3.2
| +--- com.fasterxml.jackson:jackson-bom:2.18.2 (*)
| +--- com.fasterxml.jackson.core:jackson-core:2.18.2 -> 2.15.2 (*)
| \--- com.fasterxml.jackson.core:jackson-databind:2.18.2 -> 2.15.2 (*)
+--- project :iceberg-azure
| +--- project :iceberg-api (*)
| +--- project :iceberg-bundled-guava
| +--- project :iceberg-common (*)
| \--- project :iceberg-core (*)
+--- project :iceberg-aliyun
| +--- project :iceberg-api (*)
| +--- project :iceberg-bundled-guava
| +--- project :iceberg-core (*)
| \--- project :iceberg-common (*)
+--- project :iceberg-gcp
| +--- project :iceberg-api (*)
| +--- project :iceberg-bundled-guava
| +--- project :iceberg-common (*)
| \--- project :iceberg-core (*)
+--- project :iceberg-hive-metastore
| +--- project :iceberg-api (*)
| +--- project :iceberg-bundled-guava
| +--- project :iceberg-core (*)
| +--- project :iceberg-common (*)
| \--- com.github.ben-manes.caffeine:caffeine:2.9.3 (*)
+--- project :iceberg-nessie
| +--- project :iceberg-api (*)
| +--- project :iceberg-common (*)
| +--- project :iceberg-core (*)
| +--- project :iceberg-bundled-guava
| +--- org.projectnessie.nessie:nessie-client:0.102.2
| | +--- com.fasterxml.jackson.core:jackson-core -> 2.15.2 (*)
| | +--- com.fasterxml.jackson.core:jackson-databind -> 2.15.2 (*)
| | +--- com.fasterxml.jackson.core:jackson-annotations -> 2.18.2 (*)
| | +---
org.eclipse.microprofile.openapi:microprofile-openapi-api:4.0.2
| | \--- org.projectnessie.nessie:nessie-model:0.102.2
| | +--- com.fasterxml.jackson.core:jackson-databind -> 2.15.2 (*)
| | \--- com.fasterxml.jackson.core:jackson-annotations -> 2.18.2
(*)
| +--- com.fasterxml.jackson:jackson-bom:2.18.2 (*)
| +--- com.fasterxml.jackson.core:jackson-core:2.18.2 -> 2.15.2 (*)
| \--- com.fasterxml.jackson.core:jackson-databind:2.18.2 -> 2.15.2 (*)
\--- project :iceberg-snowflake
+--- project :iceberg-core (*)
+--- project :iceberg-common (*)
+--- project :iceberg-bundled-guava
+--- com.fasterxml.jackson:jackson-bom:2.18.2 (*)
+--- com.fasterxml.jackson.core:jackson-core:2.18.2 -> 2.15.2 (*)
\--- com.fasterxml.jackson.core:jackson-databind:2.18.2 -> 2.15.2 (*)
```
The bundled dependencies that can be are also relocated:
```
// Relocate dependencies to avoid conflicts
relocate 'com.google.errorprone',
'org.apache.iceberg.shaded.com.google.errorprone'
relocate 'com.google.flatbuffers',
'org.apache.iceberg.shaded.com.google.flatbuffers'
relocate 'com.fasterxml', 'org.apache.iceberg.shaded.com.fasterxml'
relocate 'com.github.benmanes',
'org.apache.iceberg.shaded.com.github.benmanes'
relocate 'org.checkerframework',
'org.apache.iceberg.shaded.org.checkerframework'
relocate 'org.apache.avro', 'org.apache.iceberg.shaded.org.apache.avro'
relocate 'avro.shaded',
'org.apache.iceberg.shaded.org.apache.avro.shaded'
relocate 'com.thoughtworks.paranamer',
'org.apache.iceberg.shaded.com.thoughtworks.paranamer'
relocate 'org.apache.parquet',
'org.apache.iceberg.shaded.org.apache.parquet'
relocate 'shaded.parquet',
'org.apache.iceberg.shaded.org.apache.parquet.shaded'
relocate 'org.apache.orc', 'org.apache.iceberg.shaded.org.apache.orc'
relocate 'io.airlift', 'org.apache.iceberg.shaded.io.airlift'
relocate 'org.apache.hc.client5',
'org.apache.iceberg.shaded.org.apache.hc.client5'
relocate 'org.apache.hc.core5',
'org.apache.iceberg.shaded.org.apache.hc.core5'
// relocate Arrow and related deps to shade Iceberg specific version
relocate 'io.netty', 'org.apache.iceberg.shaded.io.netty'
relocate 'org.apache.arrow', 'org.apache.iceberg.shaded.org.apache.arrow'
relocate 'com.carrotsearch', 'org.apache.iceberg.shaded.com.carrotsearch'
relocate 'org.threeten.extra',
'org.apache.iceberg.shaded.org.threeten.extra'
relocate 'org.roaringbitmap',
'org.apache.iceberg.shaded.org.roaringbitmap'
relocate 'org.apache.datasketches',
'org.apache.iceberg.shaded.org.apache.datasketches'
```
The result is that the Jar should have very few classes outside of the
`org.apache.iceberg` namespace. When building a LICENSE and NOTICE, we need to
use the classes actually in the Jar file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
