This is an automated email from the ASF dual-hosted git repository.
yangjie01 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new b3d2f28962db [SPARK-50671][BUILD] Error compiling spark-protobuf
module using user-defined protoc
b3d2f28962db is described below
commit b3d2f28962dbabfe6632a3c9f945bd20770184ba
Author: morvenhuang <[email protected]>
AuthorDate: Mon Jan 6 16:24:11 2025 +0800
[SPARK-50671][BUILD] Error compiling spark-protobuf module using
user-defined protoc
### What changes were proposed in this pull request?
People may use user-defined protoc(as below) for various reasons, for
example, if they're on macOS 11, they have to use it since the default version
of protoc and protoc-gen-grpc-java plugin do not work on macOS 11.
```
$ export SPARK_PROTOC_EXEC_PATH=/Users/foobar/dev/protoc-4.29.1/protoc
$ export
CONNECT_PLUGIN_EXEC_PATH=/Users/foobar/dev/protoc-gen-grpc-java-1.59.1/protoc-gen-grpc-java
$ ./build/mvn -DskipTests clean package -Puser-defined-protoc
-DskipDefaultProtoc
```
But this results in compilation failure:
```
[ERROR] Failed to execute goal
com.github.os72:protoc-jar-maven-plugin:3.11.4:run (default) on project
spark-protobuf_2.13: Execution default of goal
com.github.os72:protoc-jar-maven-plugin:3.11.4:run failed: Cannot read the
array length because "<local6>" is null
```
This PR aims to fix this compilation failure, the error is caused by
`protocol-jar-maven-plugin` bug:
- os72/protoc-jar-maven-plugin#104
### Why are the changes needed?
Before:
```
$ export SPARK_PROTOC_EXEC_PATH=/Users/foobar/dev/protoc-4.29.1/protoc
$ export
CONNECT_PLUGIN_EXEC_PATH=/Users/foobar/dev/protoc-gen-grpc-java-1.59.1/protoc-gen-grpc-java
$ ./build/mvn -DskipTests clean package -Puser-defined-protoc
-DskipDefaultProtoc
...
[ERROR] Failed to execute goal
com.github.os72:protoc-jar-maven-plugin:3.11.4:run (default) on project
spark-protobuf_2.13: Execution default of goal
com.github.os72:protoc-jar-maven-plugin:3.11.4:run failed: Cannot read the
array length because "<local6>" is null
```
After:
```
$ export SPARK_PROTOC_EXEC_PATH=/Users/foobar/dev/protoc-4.29.1/protoc
$ export
CONNECT_PLUGIN_EXEC_PATH=/Users/foobar/dev/protoc-gen-grpc-java-1.59.1/protoc-gen-grpc-java
$ ./build/mvn -DskipTests clean package -Puser-defined-protoc
-DskipDefaultProtoc
...
[INFO]
------------------------------------------------------------------------
[INFO] Reactor Summary for Spark Project Parent POM 4.0.0-SNAPSHOT:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [
7.539 s]
[INFO] Spark Project Tags ................................. SUCCESS [
7.527 s]
[INFO] Spark Project Sketch ............................... SUCCESS [
8.413 s]
[INFO] Spark Project Common Utils ......................... SUCCESS [
29.923 s]
[INFO] Spark Project Local DB ............................. SUCCESS [
10.732 s]
[INFO] Spark Project Networking ........................... SUCCESS [
16.691 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [
12.653 s]
[INFO] Spark Project Variant .............................. SUCCESS [
5.262 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [
16.246 s]
[INFO] Spark Project Connect Shims ........................ SUCCESS [
4.356 s]
[INFO] Spark Project Launcher ............................. SUCCESS [
8.666 s]
[INFO] Spark Project Core ................................. SUCCESS [02:56
min]
[INFO] Spark Project ML Local Library ..................... SUCCESS [
30.063 s]
[INFO] Spark Project GraphX ............................... SUCCESS [
33.989 s]
[INFO] Spark Project Streaming ............................ SUCCESS [
51.897 s]
[INFO] Spark Project SQL API .............................. SUCCESS [
39.508 s]
[INFO] Spark Project Catalyst ............................. SUCCESS [03:15
min]
[INFO] Spark Project SQL .................................. SUCCESS [04:56
min]
[INFO] Spark Project ML Library ........................... SUCCESS [02:30
min]
[INFO] Spark Project Tools ................................ SUCCESS [
4.463 s]
[INFO] Spark Project Hive ................................. SUCCESS [01:33
min]
[INFO] Spark Project Connect Common ....................... SUCCESS [
55.308 s]
[INFO] Spark Avro ......................................... SUCCESS [
33.350 s]
[INFO] Spark Protobuf ..................................... SUCCESS [
38.981 s]
[INFO] Spark Project REPL ................................. SUCCESS [
21.165 s]
[INFO] Spark Project Connect Server ....................... SUCCESS [
58.033 s]
[INFO] Spark Project Connect Client ....................... SUCCESS [
59.078 s]
[INFO] Spark Project Assembly ............................. SUCCESS [
6.703 s]
[INFO] Kafka 0.10+ Token Provider for Streaming ........... SUCCESS [
20.047 s]
[INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [
26.882 s]
[INFO] Kafka 0.10+ Source for Structured Streaming ........ SUCCESS [
41.397 s]
[INFO] Spark Project Examples ............................. SUCCESS [
41.308 s]
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [
11.300 s]
[INFO]
------------------------------------------------------------------------
[INFO] BUILD SUCCESS
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Run command:
```
$ export SPARK_PROTOC_EXEC_PATH=/Users/foobar/dev/protoc-4.29.1/protoc
$ export
CONNECT_PLUGIN_EXEC_PATH=/Users/foobar/dev/protoc-gen-grpc-java-1.59.1/protoc-gen-grpc-java
$ ./build/mvn -DskipTests clean package -Puser-defined-protoc
-DskipDefaultProtoc
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #49296 from morvenhuang/SPARK-50671.
Authored-by: morvenhuang <[email protected]>
Signed-off-by: yangjie01 <[email protected]>
---
connector/protobuf/pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/connector/protobuf/pom.xml b/connector/protobuf/pom.xml
index e85481ef9e1c..22d24a7cdb62 100644
--- a/connector/protobuf/pom.xml
+++ b/connector/protobuf/pom.xml
@@ -189,7 +189,7 @@
<inputDirectories>
<include>src/test/resources/protobuf</include>
</inputDirectories>
- <includeStdTypes>true</includeStdTypes>
+ <includeMavenTypes>direct</includeMavenTypes>
<outputTargets>
<outputTarget>
<type>java</type>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
