This is an automated email from the ASF dual-hosted git repository.
yangjie01 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 0ad2bf6b8ec2 [SPARK-46483][INFRA] Exclude `apache-rat.jar` from
`ScalaUnidoc/unidoc/unidocAllClasspaths`
0ad2bf6b8ec2 is described below
commit 0ad2bf6b8ec2f965c87f98d6ef01ef294d7b28e9
Author: yangjie01 <[email protected]>
AuthorDate: Sat Dec 23 13:47:07 2023 +0800
[SPARK-46483][INFRA] Exclude `apache-rat.jar` from
`ScalaUnidoc/unidoc/unidocAllClasspaths`
### What changes were proposed in this pull request?
This pr aims to exclude `lib/apache-rat-*.jar` from `ScalaUnidoc / unidoc /
unidocAllClasspaths` to avoid accidental contamination of the classpath when
executing `sbt unidoc`.
### Why are the changes needed?
Exclude unexpected dependencies from `ScalaUnidoc / unidoc /
unidocAllClasspaths`:
`lib/apache-rat-*.jar` is downloaded by `check-license` for license
checking. It is not a direct dependency of Apache Spark, but it is accidentally
added to `ScalaUnidoc / unidoc / unidocAllClasspaths` when the `sbt unidoc`
command is executed.
And the following contents have been shaded in `lib/apache-rat-*.jar`, but
none of them have been relocated:
```
org.apache.commons:commons-collections4:jar:4.4
org.apache.commons:commons-lang3:jar:3.12.0
commons-io:commons-io:jar:2.11.0
org.apache.commons:commons-compress:jar:1.25.0
commons-cli:commons-cli:jar:1.5.0
```
On the other hand, the aforementioned fact could lead to a build failure
when executing `sbt unidoc` if Apache Spark uses a higher version of the
related dependency API in the code. For instance,
https://github.com/apache/spark/pull/44444 encountered the following error
after using `JavaVersion.JAVA_21`, an enumeration value that is only supported
in versions higher than commons-lang3 3.13.0:
-
https://github.com/LuciferYang/spark/actions/runs/7287625573/job/19859603486
```
[error]
/__w/spark/spark/core/src/main/scala/org/apache/spark/util/Utils.scala:1777:77:
value JAVA_21 is not a member of object org.apache.commons.lang3.JavaVersion
[error] did you mean JAVA_11? or perhaps JAVA_10, JAVA_12, or JAVA_13?
or...?
[error] val isJavaVersionAtLeast21 =
SystemUtils.isJavaVersionAtLeast(JavaVersion.JAVA_21)
[error]
^
```
Additionally, since `lib/apache-rat-*.jar` is not a direct dependency of
Apache Spark, it cannot be excluded through the configuration of
`ExcludedDependencies`. Nor can the other dependencies shaded by
`lib/apache-rat-*.jar` be version-pin through the configuration of
`DependencyOverrides`. So this pr adopts the approach of excluding it from
`ScalaUnidoc / unidoc / unidocAllClasspaths` to solve the problem.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Pass GitHub Actions
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #44453 from LuciferYang/SPARK-46483.
Lead-authored-by: yangjie01 <[email protected]>
Co-authored-by: YangJie <[email protected]>
Signed-off-by: yangjie01 <[email protected]>
---
project/SparkBuild.scala | 1 +
1 file changed, 1 insertion(+)
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index bcfac15ffdde..75d3a2cf18df 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -1373,6 +1373,7 @@ object Unidoc {
classpaths
.map(_.filterNot(_.data.getCanonicalPath.matches(""".*kafka-clients-0\.10.*""")))
.map(_.filterNot(_.data.getCanonicalPath.matches(""".*kafka_2\..*-0\.10.*""")))
+ .map(_.filterNot(_.data.getCanonicalPath.contains("apache-rat")))
}
val unidocSourceBase = settingKey[String]("Base URL of source links in
Scaladoc.")
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
