This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 1dad0fd792a [SPARK-42029][CONNECT] Add Guava Shading rules to
`connect-common` to avoid startup failure
1dad0fd792a is described below
commit 1dad0fd792a8b03fca8f5cce769913bc9532ea32
Author: Martin Grund <[email protected]>
AuthorDate: Thu Jan 12 19:25:12 2023 +0900
[SPARK-42029][CONNECT] Add Guava Shading rules to `connect-common` to avoid
startup failure
### What changes were proposed in this pull request?
When building a release package for PySpark and manually adding the
`spark-connect_2.12-3.4.0-SNAPSHOT.jar` jar to the distribution. The PySpark
shell will not start when started using the following option:
```
> psypark --remote local
py4j.protocol.Py4JJavaError: An error occurred while calling
None.org.apache.spark.api.java.JavaSparkContext.
: java.lang.NoClassDefFoundError:
org/sparkproject/guava/util/concurrent/internal/InternalFutureFailureAccess
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:757)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:473)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
```
The reason is that the `connect-common` package applies the default shading
rules of Spark and thus we dont pickup the right dependency for the connect
package. This patch fixes this issue by applying the same shading rules for
`guava` to `connect-common`.
### Why are the changes needed?
Release distribution
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Manual testing
Closes #39531 from grundprinzip/build_for_release.
Authored-by: Martin Grund <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
connector/connect/common/pom.xml | 37 +++++++++++++++++++++++++++++++++++++
1 file changed, 37 insertions(+)
diff --git a/connector/connect/common/pom.xml b/connector/connect/common/pom.xml
index 2d80d8215be..bb43f7d7585 100644
--- a/connector/connect/common/pom.xml
+++ b/connector/connect/common/pom.xml
@@ -156,6 +156,43 @@
</execution>
</executions>
</plugin>
+ <!-- Shade all GRPC / Guava / Protobuf dependencies of this build
-->
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-shade-plugin</artifactId>
+ <configuration>
+ <shadedArtifactAttached>false</shadedArtifactAttached>
+ <artifactSet>
+ <includes>
+ <include>com.google.guava:*</include>
+ </includes>
+ </artifactSet>
+ <relocations>
+ <relocation>
+ <pattern>com.google.common</pattern>
+
<shadedPattern>${spark.shade.packageName}.connect.guava</shadedPattern>
+ <includes>
+ <include>com.google.common.**</include>
+ </includes>
+ </relocation>
+ <relocation>
+ <pattern>com.google.thirdparty</pattern>
+
<shadedPattern>${spark.shade.packageName}.connect.guava</shadedPattern>
+ <includes>
+ <include>com.google.thirdparty.**</include>
+ </includes>
+ </relocation>
+ <relocation>
+ <pattern>android.annotation</pattern>
+
<shadedPattern>${spark.shade.packageName}.connect.android_annotation</shadedPattern>
+ </relocation>
+ <relocation>
+
<pattern>com.google.errorprone.annotations</pattern>
+
<shadedPattern>${spark.shade.packageName}.connect.errorprone_annotations</shadedPattern>
+ </relocation>
+ </relocations>
+ </configuration>
+ </plugin>
</plugins>
</build>
<profiles>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
