rdblue opened a new pull request, #16147:
URL: https://github.com/apache/iceberg/pull/16147
This is a WIP PR to update the LICENSE and NOTICE in Kafka Connect
distribution zip files.
This makes a few changes to make the process easier:
1. This removes the `hive` distribution in order to focus on the non-Hive
distribution. Non-Hive contains 248 Jars, while the Hive distribution contains
nearly 100 more (343 Jars). That includes basically all of Hadoop, like
`hadoop-mapreduce-client-{core,common,jobclient}` that are probably
unnecessary. The non-Hive distribution is also covered by the new
`runtime-deps.txt` file that tracks bundled dependencies as they change. (To
add Hive back, a follow up will need to add a secondary `runtime-deps.txt` file
for it.)
2. This updates the distribution to depend on the cloud provider bundles,
rather than bundling in AWS, GCP, and Azure dependencies manually. This should
be better for downstream users because all features should be available without
needing to change this distribution directly. For LICENSE changes, it means
that we can copy the LICENSE and NOTICE content from the cloud provider bundles
to stay in sync.
#16081 added `runtime-deps.txt` and I've verified that the non-Hive
distribution it file closely. After accounting for Iceberg Jars, LICENSE, and
NOTICE:
```diff
180c164
< netty-resolver-dns-native-macos-4.1.128.Final-osx-x86_64.jar
---
> netty-resolver-dns-native-macos-4.1.128.Final.jar
187,188c171
< netty-transport-native-epoll-4.1.130.Final-linux-x86_64.jar
< netty-transport-native-kqueue-4.1.130.Final-osx-x86_64.jar
---
> netty-transport-native-kqueue-4.1.130.Final.jar
203c185
< orc-core-1.9.8-nohive.jar
---
> orc-core-1.9.8.jar
```
I think that means we can get the LICENSE for this done fairly quickly. The
remaining dependencies on top of the cloud provider bundles are not a huge
list, and many of these are included in runtime Jars so we can copy the LICENSE
content. These remaining Jars need to be checked to see whether they actually
need to be included:
```
com.github.pjfanning:jersey-json
com.jcraft:jsch
commons-cli:commons-cli
commons-io:commons-io
commons-net:commons-net
commons-pool:commons-pool
org.apache.commons:commons-collections4
org.apache.commons:commons-compress
org.apache.commons:commons-math3
org.apache.commons:commons-text
com.sun.xml.bind:jaxb-impl
dnsjava:dnsjava
jakarta.activation:jakarta.activation-api
javax.servlet:javax.servlet-api
javax.servlet.jsp:jsp-api
javax.xml.bind:jaxb-api
javax.xml.stream:stax-api
org.apache.commons:commons-collections4
org.apache.commons:commons-compress
org.apache.commons:commons-math3
org.apache.commons:commons-text
org.apache.hadoop:hadoop-annotations
org.apache.hadoop:hadoop-common
org.apache.hadoop.thirdparty:hadoop-shaded-guava
org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_25
org.bouncycastle:bcprov-jdk18on
org.codehaus.jettison:jettison
org.mongodb:bson
```
And we will also need to make sure the LICENSE is up to date for Jars that
are included in other runtimes:
```
com.github.luben:zstd-jni
dev.failsafe:failsafe
io.airlift:aircompressor
io.dropwizard.metrics:metrics-core
org.apache.avro:avro
org.apache.httpcomponents.client5:httpclient5
org.apache.httpcomponents.core5:httpcore5
org.apache.httpcomponents.core5:httpcore5-h2
org.apache.orc:orc-core
org.apache.orc:orc-shims
org.apache.parquet:parquet-avro
org.apache.parquet:parquet-column
org.apache.parquet:parquet-common
org.apache.parquet:parquet-encoding
org.apache.parquet:parquet-format-structures
org.apache.parquet:parquet-hadoop
org.apache.parquet:parquet-jackson
org.apache.parquet:parquet-variant
org.locationtech.jts:jts-core
org.roaringbitmap:RoaringBitmap
org.xerial.snappy:snappy-java
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
