This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new e113afcfb881 [SPARK-57286][BUILD] Add `wildfly-openssl-macosx-aarch64`
dependency to support Apple Silicon
e113afcfb881 is described below
commit e113afcfb8812050123bde2acb8630ddf25f051b
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri Jun 5 16:20:59 2026 -0700
[SPARK-57286][BUILD] Add `wildfly-openssl-macosx-aarch64` dependency to
support Apple Silicon
### What changes were proposed in this pull request?
For Apache Spark 5.0.0, this PR aims to add the
`wildfly-openssl-macosx-aarch64` artifact as a dependency of the `hadoop-cloud`
module to support `Apple Silicon` when `hadoop-cloud` profile is enabled.
This artifact is available from `wildly-openssl` `2.3.0.Final` and now
Apache Spark uses it since SPARK-57283.
-
https://repo1.maven.org/maven2/org/wildfly/openssl/wildfly-openssl-macosx-aarch64/2.3.0.Final/
- #56347
### Why are the changes needed?
The main `wildfly-openssl` artifact does not ship the native OpenSSL
binding for macOS on Apple Silicon (`macosx-aarch64`). Without this
platform-specific artifact, `wildfly-openssl` fails to load the native OpenSSL
library on Apple Silicon and falls back to the JSSE implementation. Bundling
the `macosx-aarch64` variant enables native OpenSSL acceleration (e.g., for S3A
via `hadoop-aws`) on Apple Silicon machines.
**BEFORE (Apple Silicon Mac)**
```
$ build/sbt package -Phadoop-cloud
$ bin/spark-shell -c
spark.hadoop.fs.s3a.aws.credentials.provider=software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider
-c spark.hadoop.fs.s3a.ssl.channel.mode=openssl --driver-java-options
"-Dorg.wildfly.openssl.path=/opt/homebrew/opt/openssl3/lib"
...
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 5.0.0-SNAPSHOT
/_/
Using Scala version 2.13.18 (OpenJDK 64-Bit Server VM, Java 21.0.11)
...
scala> spark.read.text("s3a://.../README.md")
26/06/05 14:41:23 WARN FileSystem: Failed to initialize filesystem
s3a://.../README.md: java.io.IOException:
java.security.NoSuchAlgorithmException: Error constructing implementation
(algorithm: openssl.TLS, provider: openssl, class:
org.wildfly.openssl.OpenSSLContextSPI$OpenSSLTLSContextSpi)
26/06/05 14:41:23 WARN FileStreamSink: Assume no metadata directory. Error
while looking for metadata directory in the path: s3a://dongjoon/README.md.
java.io.IOException: java.security.NoSuchAlgorithmException: Error
constructing implementation (algorithm: openssl.TLS, provider: openssl, class:
org.wildfly.openssl.OpenSSLContextSPI$OpenSSLTLSContextSpi)
...
```
**AFTER (Apple Silicon Mac)**
```
$ build/sbt package -Phadoop-cloud
$ bin/spark-shell -c
spark.hadoop.fs.s3a.aws.credentials.provider=software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider
-c spark.hadoop.fs.s3a.ssl.channel.mode=openssl --driver-java-options
"-Dorg.wildfly.openssl.path=/opt/homebrew/opt/openssl3/lib"
...
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 5.0.0-SNAPSHOT
/_/
Using Scala version 2.13.18 (OpenJDK 64-Bit Server VM, Java 21.0.11)
...
scala> spark.read.text("s3a://.../README.md")
val res0: org.apache.spark.sql.DataFrame = [value: string]
scala>
```
### Does this PR introduce _any_ user-facing change?
No behavior change. This is just additional small jar file with 23552 bytes.
```
$ ls -al *jar
-rw-r--r-- 1 dongjoon staff 23552 Mar 13 07:06
wildfly-openssl-macosx-aarch64-2.3.0.Final.jar
```
### How was this patch tested?
Pass the CIs.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8
Closes #56349 from dongjoon-hyun/SPARK-57286.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
dev/deps/spark-deps-hadoop-3-hive-2.3 | 1 +
hadoop-cloud/pom.xml | 5 +++++
2 files changed, 6 insertions(+)
diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3
b/dev/deps/spark-deps-hadoop-3-hive-2.3
index 795e284690fa..f273539d4197 100644
--- a/dev/deps/spark-deps-hadoop-3-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-3-hive-2.3
@@ -280,6 +280,7 @@ vertx-web-client/4.5.26//vertx-web-client-4.5.26.jar
vertx-web-common/4.5.26//vertx-web-common-4.5.26.jar
volcano-client/7.7.0//volcano-client-7.7.0.jar
volcano-model/7.7.0//volcano-model-7.7.0.jar
+wildfly-openssl-macosx-aarch64/2.3.0.Final//wildfly-openssl-macosx-aarch64-2.3.0.Final.jar
wildfly-openssl/2.3.0.Final//wildfly-openssl-2.3.0.Final.jar
xbean-asm9-shaded/4.30//xbean-asm9-shaded-4.30.jar
xmlschema-core/2.3.1//xmlschema-core-2.3.1.jar
diff --git a/hadoop-cloud/pom.xml b/hadoop-cloud/pom.xml
index cf0a3bf85d3e..10408ba66f98 100644
--- a/hadoop-cloud/pom.xml
+++ b/hadoop-cloud/pom.xml
@@ -93,6 +93,11 @@
<artifactId>wildfly-openssl</artifactId>
<version>${wildfly-openssl.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.wildfly.openssl</groupId>
+ <artifactId>wildfly-openssl-macosx-aarch64</artifactId>
+ <version>${wildfly-openssl.version}</version>
+ </dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>bundle</artifactId>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
