amaechler opened a new pull request, #20652: URL: https://github.com/apache/druid/pull/20652
### Description Follow-up to #20528. `check-licenses.py` only compares compile-scope dependencies against `licenses.yaml`, but runtime-scope dependencies ship in the binary distribution too: the `lib/` dependency set in `distribution/src/assembly/assembly.xml` uses the default (runtime) scope, and `pull-deps` resolves extensions with `JavaScopes.RUNTIME`. ASF policy requires the binary `LICENSE` and `NOTICE` to [account for the package's exact content](https://www.apache.org/legal/release-policy.html#licensing-documentation), including [transitive](https://infra.apache.org/licensing-howto.html#deps-of-deps) dependencies [bundled only in the binary](https://infra.apache.org/licensing-howto.html#binary). Runtime-scope dependencies have gone unchecked since the check was introduced in #7998. Every entry below was confirmed against a `-Pdist` tarball built from this branch and against the jar's pom and `META-INF` license and notice files. <details> #### Parse runtime-scope tables `DependencyReportParser` now also matches the `Project_Dependencies_runtime` and `Project_Transitive_Dependencies_runtime` anchors. It also maps the `Eclipse Public License v. 2.0` spelling used by `jakarta.el-api` and `expressly`, which are runtime dependencies whose license was otherwise not compared. #### Register the 27 runtime dependencies the check now reports - Bumped entries whose old version no longer ships anywhere in the tarball: `janino` and `commons-compiler` 3.0.11 to 3.1.12, `jboss-logging` 3.2.1.Final to 3.6.3.Final, `jcl-over-slf4j` 1.7.36 to 2.0.20. `log4j-slf4j-impl` no longer ships, so the Log4j entry now lists `log4j-slf4j2-impl` with its notice. - `jaxb-runtime` and `txw2` 2.3.9 and `istack-commons-runtime` 3.0.12 replace 2.3.1 and 3.0.7. These versions are under EDL 1.0 rather than CDDL 1.1, so the entries now point to `licenses/bin/jakarta.EDL1`. - `lib/` ships older Google client versions than `extensions/druid-google-extensions/`, pulled in through `druid-gcp-common`: `google-auth-library-*` 1.30.0 vs 1.53.0, `grpc-api`/`grpc-context` 1.70.0 vs 1.84.0, `google-oauth-client` 1.36.0 vs 1.39.0, `google-http-client-apache-v2`/`-gson` 1.45.2 vs 2.2.0. Both sets are now registered. The existing `google-oauth-client` 1.39.0 entry was tagged `java-core`; it is now 1.36.0, and a separate `druid-google-extensions` entry covers 1.39.0. `google-http-client` and `-jackson2` 2.2.0 ship in both places and are listed under both modules. - `detector-resources-support` ships at 0.31.0, not at 0.33.0 with the other Google Cloud OpenTelemetry jars, so it has its own entry. - New entries: `hibernate-validator`, `classmate` (with notice) and `uzaygezen-core` in core; `perfmark-api` and `com.google.android:annotations` in `druid-google-extensions`; `hadoop-aws` and `hadoop-client-runtime` (with the Hadoop notice), `wildfly-openssl` and `analyticsaccelerator-s3` (with notice) in `druid-hdfs-storage`. #### List third-party code shaded into `hadoop-client-runtime` `hadoop-client-runtime` relocates about 70 third-party artifacts into one jar, and Hadoop's shade configuration replaces their `LICENSE` and `NOTICE` files with Hadoop's own (HADOOP-14018). The jar's `LICENSE.txt` is Hadoop's source `LICENSE`, so it does not cover the non-Apache code inside. The ASF licensing guide only allows leaving `LICENSE` unchanged for a bundled Apache-2.0 jar when it has no sub-components under other licenses, so the binary `LICENSE` now has an entry per non-Apache component, without `libraries` (the same pattern as the existing "Public Suffix List bundled by AsyncHttpClient" entry, which the check skips): - BSD: JSch 0.1.55, dnsjava 3.6.1, JLine 3.9.0, Protocol Buffers 3.25.5, Stax2 API 4.2.1 - Go license: RE2/J 1.1 - MIT: Checker Framework qualifiers 3.51.1 - EDL 1.0: Jakarta Activation API 1.2.1, Jakarta XML Binding API 2.3.3 - EPL 2.0, chosen over GPLv2 with Classpath Exception as Hadoop does: Jakarta Servlet API 4.0.4, Server Pages API 2.3.6, RESTful Web Services API 2.1.6 - MPL 2.0: the Public Suffix List shipped with Apache HttpClient Versions come from the poms embedded in the jar, and licenses from Hadoop 3.5.0's `LICENSE-binary`, except dnsjava, which Hadoop lists as BSD-2 but whose `LICENSE` has three clauses. New license texts were fetched from each project at the matching tag: `dnsjava.BSD3`, `jline3.BSD3` (the existing `jline.BSD3` is JLine 2's), `re2j.GO` and `jakarta.EPL2`. The existing RE2/J and Expressly entries pointed at `re2j.GO` and `jakarta.el.EPL2`, which did not exist; both now resolve. Other ASF projects differ here: Spark, Kyuubi and Pinot list `hadoop-client-runtime` as a single Apache-2.0 line, while Flink and Ozone enumerate the contents of other ASF projects' pre-shaded jars. This follows the latter. #### Not in this PR - 174 registered entries still produce "unchecked" warnings because their version is not in any report (for example Guava 32.1.3-jre, Avro 1.11.5, Confluent 6.2.15, `stax-ex`, `FastInfoset`, `hadoop-client-api` 3.3.6 under `druid-hdfs-storage`). These only warn, and each needs a tarball comparison before removal. - `hadoop-client-runtime` also ships in `druid-orc-extensions` and `druid-parquet-extensions`; its entries are attributed to `druid-hdfs-storage` only. - The JDBI entry's notice text is ClassMate's notice, so that text now appears twice in the generated NOTICE. </details> #### Release note The binary distribution's `LICENSE` and `NOTICE` now cover runtime-scope third-party dependencies, and the third-party code shaded into the Hadoop client jar, which were previously missing. <hr> This PR has: - [x] been self-reviewed. - [x] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
