This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 4cc7bd1c74c [SPARK-43842][BUILD] Upgrade `gcs-connector` to 2.2.14
4cc7bd1c74c is described below
commit 4cc7bd1c74c0b9512ddcc7ce4f1d2818cab43ac7
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Mon May 29 09:19:24 2023 +0900
[SPARK-43842][BUILD] Upgrade `gcs-connector` to 2.2.14
### What changes were proposed in this pull request?
This PR aims to upgrade `gcs-connector` to 2.2.14.
### Why are the changes needed?
-
https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/tag/v2.2.14
-
https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/tag/v2.2.13
-
https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/tag/v2.2.12
### Does this PR introduce _any_ user-facing change?
Although this is a dependency change, there is no user-facing change.
### How was this patch tested?
Pass the CIs and manual verification.
**BUILD**
```
$ dev/make-distribution.sh -Phadoop-cloud
```
**RUN**
```
$ export KEYFILE=YOUR-credentials.json
$ export EMAIL=$(jq -r '.client_email' < $KEYFILE)
$ export PRIVATE_KEY_ID=$(jq -r '.private_key_id' < $KEYFILE)
$ export PRIVATE_KEY="$(jq -r '.private_key' < $KEYFILE)"
$ bin/spark-shell \
-c spark.hadoop.fs.gs.auth.service.account.email=$EMAIL \
-c
spark.hadoop.fs.gs.auth.service.account.private.key.id=$PRIVATE_KEY_ID \
-c spark.hadoop.fs.gs.auth.service.account.private.key="$PRIVATE_KEY"
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
setLogLevel(newLevel).
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.5.0-SNAPSHOT
/_/
Using Scala version 2.13.8 (OpenJDK 64-Bit Server VM, Java 17.0.7)
Type in expressions to have them evaluated.
Type :help for more information.
23/05/28 14:43:31 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
Spark context Web UI available at http://localhost:4040
Spark context available as 'sc' (master = local[*], app id =
local-1685310212206).
Spark session available as 'spark'.
scala> spark.read.text("gs://apache-spark-bucket/README.md").count()
val res0: Long = 124
scala>
spark.read.orc("examples/src/main/resources/users.orc").write.mode("overwrite").orc("gs://apache-spark-bucket/users.orc")
scala> spark.read.orc("gs://apache-spark-bucket/users.orc").show()
+------+--------------+----------------+
| name|favorite_color|favorite_numbers|
+------+--------------+----------------+
|Alyssa| NULL| [3, 9, 15, 20]|
| Ben| red| []|
+------+--------------+----------------+
```
Closes #41352 from dongjoon-hyun/SPARK-43842.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
dev/deps/spark-deps-hadoop-3-hive-2.3 | 2 +-
pom.xml | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3
b/dev/deps/spark-deps-hadoop-3-hive-2.3
index bed3be0b086..99b627af340 100644
--- a/dev/deps/spark-deps-hadoop-3-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-3-hive-2.3
@@ -62,7 +62,7 @@ datasketches-memory/2.1.0//datasketches-memory-2.1.0.jar
derby/10.14.2.0//derby-10.14.2.0.jar
dropwizard-metrics-hadoop-metrics2-reporter/0.1.2//dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar
flatbuffers-java/1.12.0//flatbuffers-java-1.12.0.jar
-gcs-connector/hadoop3-2.2.11/shaded/gcs-connector-hadoop3-2.2.11-shaded.jar
+gcs-connector/hadoop3-2.2.14/shaded/gcs-connector-hadoop3-2.2.14-shaded.jar
gmetric4j/1.0.10//gmetric4j-1.0.10.jar
gson/2.2.4//gson-2.2.4.jar
guava/14.0.1//guava-14.0.1.jar
diff --git a/pom.xml b/pom.xml
index cc61c72ed75..8e91e9f9cbe 100644
--- a/pom.xml
+++ b/pom.xml
@@ -159,7 +159,7 @@
<aws.java.sdk.version>1.11.655</aws.java.sdk.version>
<!-- the producer is used in tests -->
<aws.kinesis.producer.version>0.12.8</aws.kinesis.producer.version>
- <gcs-connector.version>hadoop3-2.2.11</gcs-connector.version>
+ <gcs-connector.version>hadoop3-2.2.14</gcs-connector.version>
<!-- org.apache.httpcomponents/httpclient-->
<commons.httpclient.version>4.5.14</commons.httpclient.version>
<commons.httpcore.version>4.4.16</commons.httpcore.version>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
