This is an automated email from the ASF dual-hosted git repository.
cloud-fan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 13b526d534e3 [SPARK-56413][SPARK-56661][UDF][BUILD] Confine gRPC to a
dedicated udf-worker-grpc module
13b526d534e3 is described below
commit 13b526d534e3b6115be4769d39d861b321bb3a02
Author: Haiyang Sun <[email protected]>
AuthorDate: Wed Jun 3 22:10:24 2026 +0800
[SPARK-56413][SPARK-56661][UDF][BUILD] Confine gRPC to a dedicated
udf-worker-grpc module
### What changes were proposed in this pull request?
This PR extracts the gRPC-based UDF worker transport into a new
`udf/worker/grpc` Maven/SBT module, sibling to the existing `udf/worker/proto`
and `udf/worker/core` modules, so that gRPC is no longer pulled onto the shared
Spark classpath.
Concretely:
- **New module `spark-udf-worker-grpc`** — generates the gRPC service stubs
(`UdfWorkerGrpc`) from the `.proto` definitions in `udf-worker-proto`
(`compile-custom` / grpc-java only), and owns the gRPC runtime dependencies
(`grpc-api`, `grpc-protobuf`, `grpc-stub`, plus `grpc-inprocess` for tests).
- **`udf-worker-proto`** now generates only protobuf-java message classes
(dropped the grpc-java codegen goal and the `grpc-*` dependencies).
- **`udf-worker-core`** no longer depends on gRPC (the `grpc-inprocess`
test dependency was removed).
- **`EchoProtocolSuite`** (the gRPC protocol test) moved from
`udf-worker-core` to the new `udf-worker-grpc` module and re-packaged to
`org.apache.spark.udf.worker.grpc`.
- Registered the module in the root `pom.xml` and in
`project/SparkBuild.scala` (new `udfWorkerGrpc` project, `UDFWorkerGrpc`
settings for grpc-stub-only codegen, and `UDFWorkerProto` restricted to
message-only codegen).
- Regenerated `dev/deps/spark-deps-hadoop-3-hive-2.3`, which drops
`grpc-api`, `grpc-protobuf`, `grpc-protobuf-lite`, `grpc-stub`,
`proto-google-common-protos`, `animal-sniffer-annotations`, and
`error_prone_annotations` from the assembly classpath.
Module dependency shape after this change:
```
udf-worker-proto (protobuf-java messages only)
^ ^
| |
core/catalyst/sql-core -- use message types + worker abstractions (NO gRPC)
|
udf-worker-core (worker abstractions, no gRPC)
^
|
udf-worker-grpc (gRPC service stubs + gRPC runtime -- confined here)
```
### Why are the changes needed?
Introducing the language-agnostic UDF worker framework made
`spark-udf-worker-proto`/`-core` compile dependencies of `core`, `catalyst`,
and `sql/core`. Because the proto module carried the gRPC stack as
compile-scope dependencies (needed to compile its generated gRPC service
stubs), this dragged `grpc-api`, `grpc-protobuf{,-lite}`, `grpc-stub`, and
`proto-google-common-protos` transitively onto the widely-shared Spark
core/assembly classpath. Spark has historically kept gRPC isolated [...]
No code on the runtime classpath actually uses the gRPC stubs yet (only
`EchoProtocolSuite` did, a test). Confining gRPC to its own module removes the
unnecessary footprint from `core`/`catalyst`/`sql-core` while keeping the
framework's message types and worker abstractions available to them.
### Does this PR introduce _any_ user-facing change?
No. This is a build/module reorganization; the affected UDF worker
framework is experimental and not yet consumed at runtime.
### How was this patch tested?
- Existing tests, relocated: `EchoProtocolSuite` now runs under
`udf-worker-grpc`.
- Verified with SBT that `udf-worker-grpc/Test`, `udf-worker-core/Test`,
`catalyst`, `core`, and `sql` compile, and confirmed the codegen split on disk
(proto -> `generated-sources/protobuf/java` messages only; grpc ->
`generated-sources/protobuf/grpc-java/UdfWorkerGrpc.java`).
- Regenerated and validated the dependency manifest via
`./dev/test-dependencies.sh --replace-manifest`.
### Was this patch authored or co-authored using generative AI tooling?
Yes
Closes #56273 from haiyangsun-db/SPARK-56661.
Authored-by: Haiyang Sun <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
---
dev/deps/spark-deps-hadoop-3-hive-2.3 | 7 --
pom.xml | 1 +
project/SparkBuild.scala | 71 ++++++++++++--
udf/worker/README.md | 36 ++++---
udf/worker/core/pom.xml | 6 --
udf/worker/{proto => grpc}/pom.xml | 105 +++++++++------------
.../spark/udf/worker/grpc}/EchoProtocolSuite.scala | 2 +-
udf/worker/proto/pom.xml | 24 ++---
8 files changed, 140 insertions(+), 112 deletions(-)
diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3
b/dev/deps/spark-deps-hadoop-3-hive-2.3
index 6e1458207b65..82562bbb0371 100644
--- a/dev/deps/spark-deps-hadoop-3-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-3-hive-2.3
@@ -12,7 +12,6 @@ aliyun-java-sdk-kms/2.11.0//aliyun-java-sdk-kms-2.11.0.jar
aliyun-java-sdk-ram/3.1.0//aliyun-java-sdk-ram-3.1.0.jar
aliyun-sdk-oss/3.18.1//aliyun-sdk-oss-3.18.1.jar
analyticsaccelerator-s3/1.3.1//analyticsaccelerator-s3-1.3.1.jar
-animal-sniffer-annotations/1.24//animal-sniffer-annotations-1.24.jar
antlr-runtime/3.5.2//antlr-runtime-3.5.2.jar
antlr4-runtime/4.13.1//antlr4-runtime-4.13.1.jar
aopalliance-repackaged/3.0.6//aopalliance-repackaged-3.0.6.jar
@@ -65,15 +64,10 @@ derbyshared/10.16.1.1//derbyshared-10.16.1.1.jar
derbytools/10.16.1.1//derbytools-10.16.1.1.jar
dom4j/2.1.4//dom4j-2.1.4.jar
dropwizard-metrics-hadoop-metrics2-reporter/0.1.2//dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar
-error_prone_annotations/2.18.0//error_prone_annotations-2.18.0.jar
esdk-obs-java/3.20.4.2//esdk-obs-java-3.20.4.2.jar
failureaccess/1.0.3//failureaccess-1.0.3.jar
flatbuffers-java/25.2.10//flatbuffers-java-25.2.10.jar
gmetric4j/1.0.10//gmetric4j-1.0.10.jar
-grpc-api/1.76.0//grpc-api-1.76.0.jar
-grpc-protobuf-lite/1.76.0//grpc-protobuf-lite-1.76.0.jar
-grpc-protobuf/1.76.0//grpc-protobuf-1.76.0.jar
-grpc-stub/1.76.0//grpc-stub-1.76.0.jar
gson/2.14.0//gson-2.14.0.jar
guava/33.6.0-jre//guava-33.6.0-jre.jar
hadoop-aliyun/3.5.0//hadoop-aliyun-3.5.0.jar
@@ -254,7 +248,6 @@
parquet-format-structures/1.17.1//parquet-format-structures-1.17.1.jar
parquet-hadoop/1.17.1//parquet-hadoop-1.17.1.jar
parquet-jackson/1.17.1//parquet-jackson-1.17.1.jar
pickle/1.5//pickle-1.5.jar
-proto-google-common-protos/2.59.2//proto-google-common-protos-2.59.2.jar
py4j/0.10.9.9//py4j-0.10.9.9.jar
reactive-streams/1.0.3//reactive-streams-1.0.3.jar
remotetea-oncrpc/1.1.2//remotetea-oncrpc-1.1.2.jar
diff --git a/pom.xml b/pom.xml
index f985ace87ee3..a501bbdeb014 100644
--- a/pom.xml
+++ b/pom.xml
@@ -113,6 +113,7 @@
<module>connector/protobuf</module>
<module>udf/worker/proto</module>
<module>udf/worker/core</module>
+ <module>udf/worker/grpc</module>
<!-- See additional modules enabled by profiles below -->
</modules>
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index 7696e09865f8..2b74f45b37d6 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -59,8 +59,8 @@ object BuildCommons {
Seq("connect-common", "connect", "connect-client-jdbc",
"connect-client-jvm", "connect-shims")
.map(ProjectRef(buildLocation, _))
- val udfWorkerProjects@Seq(udfWorkerProto, udfWorkerCore) =
- Seq("udf-worker-proto", "udf-worker-core").map(ProjectRef(buildLocation,
_))
+ val udfWorkerProjects@Seq(udfWorkerProto, udfWorkerCore, udfWorkerGrpc) =
+ Seq("udf-worker-proto", "udf-worker-core",
"udf-worker-grpc").map(ProjectRef(buildLocation, _))
val allProjects@Seq(
core, graphx, mllib, mllibLocal, repl, networkCommon, networkShuffle,
launcher, unsafe, tags, sketch, kvstore,
@@ -416,7 +416,7 @@ object SparkBuild extends PomBuild {
spark, hive, hiveThriftServer, repl, networkCommon, networkShuffle,
networkYarn,
unsafe, tags, tokenProviderKafka010, sqlKafka010, pipelines,
connectCommon, connect,
connectJdbc, connectClient, variant, connectShims, profiler,
commonUtilsJava,
- udfWorkerProto, udfWorkerCore
+ udfWorkerProto, udfWorkerCore, udfWorkerGrpc
).contains(x)
}
@@ -472,6 +472,9 @@ object SparkBuild extends PomBuild {
/* UDF Worker Proto settings */
enable(UDFWorkerProto.settings)(udfWorkerProto)
+ /* UDF Worker gRPC settings */
+ enable(UDFWorkerGrpc.settings)(udfWorkerGrpc)
+
enable(DockerIntegrationTests.settings)(dockerIntegrationTests)
enable(KubernetesIntegrationTests.settings)(kubernetesIntegrationTests)
@@ -1092,8 +1095,11 @@ object SparkProtobuf {
}
object UDFWorkerProto {
- // Reuses SparkConnectCommon for proto + gRPC codegen wiring; overrides
- // only the assembly fields that need the UDF-worker namespace.
+ // Reuses SparkConnectCommon for proto codegen wiring; overrides the assembly
+ // fields that need the UDF-worker namespace and restricts code generation to
+ // protobuf-java messages only. The gRPC service stubs are generated in the
+ // separate udf-worker-grpc module, so this module (and its consumers) carry
+ // no gRPC dependency.
lazy val settings = SparkConnectCommon.settings ++ Seq(
// Include only this module's jar and protobuf-java in the assembly.
(assembly / assemblyExcludedJars) := {
@@ -1107,10 +1113,62 @@ object UDFWorkerProto {
(assembly / assemblyShadeRules) := Seq(
ShadeRule.rename("com.google.protobuf.**" ->
"org.sparkproject.spark_udf_worker.protobuf.@1").inAll
+ ),
+
+ // Generate only protobuf-java messages here (overrides the java +
grpc-java
+ // targets inherited from SparkConnectCommon).
+ (Compile / PB.targets) := Seq(
+ PB.gens.java -> target.value / "generated-sources" / "protobuf" / "java"
)
)
}
+object UDFWorkerGrpc {
+ import BuildCommons.protoVersion
+
+ // Generates only the gRPC service stubs, reading the .proto definitions from
+ // the udf-worker-proto module (whose protobuf-java messages the stubs
compile
+ // against). Confines the gRPC runtime to this module.
+ lazy val settings = Seq(
+ PB.protocVersion := protoVersion,
+
+ libraryDependencies ++= {
+ val grpcVersion =
+ SbtPomKeys.effectivePom.value.getProperties.get(
+ "io.grpc.version").asInstanceOf[String]
+ Seq(
+ "io.grpc" % "protoc-gen-grpc-java" % grpcVersion asProtocPlugin(),
+ "com.google.protobuf" % "protobuf-java" % protoVersion % "protobuf"
+ )
+ },
+
+ dependencyOverrides += "com.google.protobuf" % "protobuf-java" %
protoVersion,
+
+ // Read the shared .proto sources from the proto module.
+ (Compile / PB.protoSources) := Seq(
+ baseDirectory.value / ".." / "proto" / "src" / "main" / "protobuf")
+ ) ++ {
+ val sparkProtocExecPath = sys.props.get("spark.protoc.executable.path")
+ val connectPluginExecPath = sys.props.get("connect.plugin.executable.path")
+ if (sparkProtocExecPath.isDefined && connectPluginExecPath.isDefined) {
+ Seq(
+ (Compile / PB.targets) := Seq(
+ PB.gens.plugin(name = "grpc-java", path = connectPluginExecPath.get)
->
+ target.value / "generated-sources" / "protobuf" / "grpc-java"
+ ),
+ PB.protocExecutable := file(sparkProtocExecPath.get)
+ )
+ } else {
+ Seq(
+ (Compile / PB.targets) := Seq(
+ PB.gens.plugin("grpc-java") ->
+ target.value / "generated-sources" / "protobuf" / "grpc-java"
+ )
+ )
+ }
+ }
+}
+
object Unsafe {
lazy val settings = Seq()
}
@@ -1700,7 +1758,8 @@ object Unidoc {
(JavaUnidoc / unidoc / unidocProjectFilter) :=
inAnyProject -- inProjects(OldDeps.project, repl, examples, tools,
kubernetes,
yarn, tags, streamingKafka010, sqlKafka010, connectCommon, connect,
connectJdbc,
- connectClient, connectShims, protobuf, profiler, udfWorkerProto,
udfWorkerCore),
+ connectClient, connectShims, protobuf, profiler, udfWorkerProto,
udfWorkerCore,
+ udfWorkerGrpc),
)
}
diff --git a/udf/worker/README.md b/udf/worker/README.md
index 21846218d3f9..cf3253fdaa89 100644
--- a/udf/worker/README.md
+++ b/udf/worker/README.md
@@ -33,21 +33,24 @@ provisioning service or daemon).
```
udf/worker/
-├── proto/
+├── proto/ -- protobuf message classes only
(protobuf-java)
│ worker_spec.proto -- UDFWorkerSpecification protobuf
│ udf_protocol.proto -- UDF execution protocol (Init, UdfPayload,
...)
│ common.proto -- shared enums (UDFWorkerDataFormat, etc.)
│
-└── core/ -- abstract interfaces
- WorkerDispatcher.scala -- creates sessions, manages worker lifecycle
- WorkerSession.scala -- per-UDF init/process/cancel/close
- WorkerConnection.scala -- transport channel abstraction
- WorkerSecurityScope.scala -- security boundary for worker pooling
- │
- └── direct/ -- "direct" creation: local OS processes
- DirectWorkerDispatcher.scala -- spawns processes, env lifecycle
- DirectWorkerProcess.scala -- OS process + connection + UDS
socket
- DirectWorkerSession.scala -- session backed by a direct process
+├── core/ -- abstract interfaces
+│ WorkerDispatcher.scala -- creates sessions, manages worker lifecycle
+│ WorkerSession.scala -- per-UDF init/process/cancel/close
+│ WorkerConnection.scala -- transport channel abstraction
+│ WorkerSecurityScope.scala -- security boundary for worker pooling
+│ │
+│ └── direct/ -- "direct" creation: local OS processes
+│ DirectWorkerDispatcher.scala -- spawns processes, env lifecycle
+│ DirectWorkerProcess.scala -- OS process + connection + UDS
socket
+│ DirectWorkerSession.scala -- session backed by a direct process
+│
+└── grpc/ -- gRPC transport (gRPC runtime confined
here)
+ (generated) -- UdfWorkerGrpc service stubs from
proto/udf_protocol.proto
```
The `core/` package defines abstract interfaces that are independent of how
@@ -56,6 +59,11 @@ worker creation where Spark spawns local OS processes.
Future packages
(e.g., `core/indirect/`) can implement alternative creation modes such as
obtaining workers from a provisioning service or daemon.
+The `grpc/` module owns the gRPC service-stub generation (from
+`proto/`'s `udf_protocol.proto`) and the gRPC runtime dependencies. Keeping
+gRPC here means `proto/`, `core/`, and their consumers (`core`, `catalyst`,
+`sql/core`) carry no gRPC dependency on their classpath.
+
## Wire protocol
Each UDF execution uses a single bidirectional `Execute` gRPC stream.
@@ -157,19 +165,19 @@ dispatcher.close()
SBT:
```
-build/sbt "udf-worker-proto/compile" "udf-worker-core/compile"
+build/sbt "udf-worker-proto/compile" "udf-worker-core/compile"
"udf-worker-grpc/compile"
```
Maven:
```
-build/mvn compile -pl udf/worker/proto,udf/worker/core -am
+build/mvn compile -pl udf/worker/proto,udf/worker/core,udf/worker/grpc -am
```
## Test
SBT:
```
-build/sbt "udf-worker-core/test"
+build/sbt "udf-worker-core/test" "udf-worker-grpc/test"
```
## Current status
diff --git a/udf/worker/core/pom.xml b/udf/worker/core/pom.xml
index 3cd281cc1a14..7847410ec9bb 100644
--- a/udf/worker/core/pom.xml
+++ b/udf/worker/core/pom.xml
@@ -56,12 +56,6 @@
<artifactId>mockito-core</artifactId>
<scope>test</scope>
</dependency>
- <dependency>
- <groupId>io.grpc</groupId>
- <artifactId>grpc-inprocess</artifactId>
- <version>${io.grpc.version}</version>
- <scope>test</scope>
- </dependency>
</dependencies>
<build>
diff --git a/udf/worker/proto/pom.xml b/udf/worker/grpc/pom.xml
similarity index 63%
copy from udf/worker/proto/pom.xml
copy to udf/worker/grpc/pom.xml
index 8868a50707cb..9250260032a1 100644
--- a/udf/worker/proto/pom.xml
+++ b/udf/worker/grpc/pom.xml
@@ -28,16 +28,47 @@
<relativePath>../../../pom.xml</relativePath>
</parent>
- <artifactId>spark-udf-worker-proto_2.13</artifactId>
+ <artifactId>spark-udf-worker-grpc_2.13</artifactId>
<packaging>jar</packaging>
- <name>Spark Project UDF Worker Proto</name>
+ <name>Spark Project UDF Worker gRPC</name>
<url>https://spark.apache.org/</url>
<properties>
- <sbt.project.name>udf-worker-proto</sbt.project.name>
+ <sbt.project.name>udf-worker-grpc</sbt.project.name>
</properties>
+ <!--
+ Home for the gRPC-based UDF worker transport. The gRPC service stubs are
+ generated here (from the proto module's service definitions) and the gRPC
+ runtime is confined to this module, so spark-udf-worker-proto / -core and
+ their consumers (core, catalyst, sql/core) stay free of any gRPC
+ transitive dependency. (protobuf-java remains a transitive dependency of
+ those modules via the proto messages, as it was before; only gRPC is
+ removed from their classpath.)
+
+ Note: gRPC is kept off every shared classpath by the dependency graph
+ alone, since nothing on the runtime/assembly classpath depends on this
+ module yet. The gRPC runtime here is NOT shaded. Once the gRPC transport
+ is wired up and a runtime module depends on this one, the io.grpc and
+ protobuf classes should be relocated (mirroring the
+ org.sparkproject.spark_udf_worker.protobuf relocation in the proto module)
+ to avoid version clashes with Spark Connect's gRPC. That shading is
+ deferred to a follow-up.
+ -->
<dependencies>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-tags_${scala.binary.version}</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-udf-worker-proto_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.scala-lang</groupId>
+ <artifactId>scala-library</artifactId>
+ </dependency>
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
@@ -59,12 +90,10 @@
<version>${io.grpc.version}</version>
</dependency>
<dependency>
- <groupId>org.apache.spark</groupId>
- <artifactId>spark-tags_${scala.binary.version}</artifactId>
- </dependency>
- <dependency>
- <groupId>org.scala-lang</groupId>
- <artifactId>scala-library</artifactId>
+ <groupId>io.grpc</groupId>
+ <artifactId>grpc-inprocess</artifactId>
+ <version>${io.grpc.version}</version>
+ <scope>test</scope>
</dependency>
</dependencies>
@@ -90,70 +119,26 @@
<groupId>org.xolstice.maven.plugins</groupId>
<artifactId>protobuf-maven-plugin</artifactId>
<version>0.6.1</version>
+ <!--
+ Generate only the gRPC service stubs (compile-custom). The protobuf
+ message classes are generated by and consumed from
+ spark-udf-worker-proto; the .proto source root points at that module
+ so protoc can resolve imports while emitting only the grpc-java
output.
+ -->
<configuration>
<protocArtifact>com.google.protobuf:protoc:${protobuf.version}:exe:${os.detected.classifier}</protocArtifact>
<pluginId>grpc-java</pluginId>
<pluginArtifact>io.grpc:protoc-gen-grpc-java:${io.grpc.version}:exe:${os.detected.classifier}</pluginArtifact>
- <protoSourceRoot>src/main/protobuf</protoSourceRoot>
+
<protoSourceRoot>${project.basedir}/../proto/src/main/protobuf</protoSourceRoot>
</configuration>
<executions>
<execution>
<goals>
- <goal>compile</goal>
<goal>compile-custom</goal>
</goals>
</execution>
</executions>
</plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-shade-plugin</artifactId>
- <!--
- Override the parent pom's shade configuration completely (rather than
- merging) so this module's shade output contains only the proto module
- plus its relocated protobuf-java classes. The shaded fat jar is
published
- as an attached `-shaded` classifier artifact, keeping the main
artifact
- unshaded so consumers (e.g. spark-udf-worker-core) and their tests
can
- continue to compile against the unrelocated
com.google.protobuf.ByteString.
- This mirrors the SBT setup where `udf-worker-proto/assembly`
produces a
- shaded fat jar separately from the regular `package` output.
- -->
- <configuration combine.self="override">
- <shadedArtifactAttached>true</shadedArtifactAttached>
- <shadedClassifierName>shaded</shadedClassifierName>
- <shadeTestJar>false</shadeTestJar>
- <artifactSet>
- <includes>
- <include>com.google.protobuf:*</include>
- </includes>
- </artifactSet>
- <relocations>
- <relocation>
- <pattern>com.google.protobuf</pattern>
-
<shadedPattern>${spark.shade.packageName}.spark_udf_worker.protobuf</shadedPattern>
- <includes>
- <include>com.google.protobuf.**</include>
- </includes>
- </relocation>
- </relocations>
- <filters>
- <filter>
- <artifact>*:*</artifact>
- <excludes>
- <exclude>google/protobuf/**</exclude>
- </excludes>
- </filter>
- </filters>
- </configuration>
- <executions>
- <execution>
- <phase>package</phase>
- <goals>
- <goal>shade</goal>
- </goals>
- </execution>
- </executions>
- </plugin>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
diff --git
a/udf/worker/core/src/test/scala/org/apache/spark/udf/worker/core/EchoProtocolSuite.scala
b/udf/worker/grpc/src/test/scala/org/apache/spark/udf/worker/grpc/EchoProtocolSuite.scala
similarity index 99%
rename from
udf/worker/core/src/test/scala/org/apache/spark/udf/worker/core/EchoProtocolSuite.scala
rename to
udf/worker/grpc/src/test/scala/org/apache/spark/udf/worker/grpc/EchoProtocolSuite.scala
index 80802dd082f6..eae6ef1a639a 100644
---
a/udf/worker/core/src/test/scala/org/apache/spark/udf/worker/core/EchoProtocolSuite.scala
+++
b/udf/worker/grpc/src/test/scala/org/apache/spark/udf/worker/grpc/EchoProtocolSuite.scala
@@ -14,7 +14,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.spark.udf.worker.core
+package org.apache.spark.udf.worker.grpc
import java.util.concurrent.{CountDownLatch, LinkedBlockingQueue, TimeUnit}
import java.util.concurrent.atomic.AtomicBoolean
diff --git a/udf/worker/proto/pom.xml b/udf/worker/proto/pom.xml
index 8868a50707cb..603da4c28ff2 100644
--- a/udf/worker/proto/pom.xml
+++ b/udf/worker/proto/pom.xml
@@ -43,21 +43,12 @@
<artifactId>protobuf-java</artifactId>
<scope>compile</scope>
</dependency>
- <dependency>
- <groupId>io.grpc</groupId>
- <artifactId>grpc-api</artifactId>
- <version>${io.grpc.version}</version>
- </dependency>
- <dependency>
- <groupId>io.grpc</groupId>
- <artifactId>grpc-protobuf</artifactId>
- <version>${io.grpc.version}</version>
- </dependency>
- <dependency>
- <groupId>io.grpc</groupId>
- <artifactId>grpc-stub</artifactId>
- <version>${io.grpc.version}</version>
- </dependency>
+ <!--
+ gRPC service stubs are generated in (and the gRPC runtime is confined to)
+ spark-udf-worker-grpc, so this module produces only protobuf-java message
+ classes and carries no gRPC dependency. This keeps gRPC off the classpath
+ of every consumer of this module (core, catalyst, sql/core,
udf-worker-core).
+ -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-tags_${scala.binary.version}</artifactId>
@@ -92,15 +83,12 @@
<version>0.6.1</version>
<configuration>
<protocArtifact>com.google.protobuf:protoc:${protobuf.version}:exe:${os.detected.classifier}</protocArtifact>
- <pluginId>grpc-java</pluginId>
-
<pluginArtifact>io.grpc:protoc-gen-grpc-java:${io.grpc.version}:exe:${os.detected.classifier}</pluginArtifact>
<protoSourceRoot>src/main/protobuf</protoSourceRoot>
</configuration>
<executions>
<execution>
<goals>
<goal>compile</goal>
- <goal>compile-custom</goal>
</goals>
</execution>
</executions>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
