This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-swift.git
The following commit(s) were added to refs/heads/main by this push:
new 4185eb9 [SPARK-51763] Support `struct` type in `ArrowReader`
4185eb9 is described below
commit 4185eb9197eec6dee28d85d7e5dec46ddf7d2f17
Author: Dongjoon Hyun <dongj...@apache.org>
AuthorDate: Fri Apr 11 06:56:23 2025 +0900
[SPARK-51763] Support `struct` type in `ArrowReader`
### What changes were proposed in this pull request?
This PR aims to support `struct` type in `ArrowReader`.
### Why are the changes needed?
`ArrowReader` depends on `ArrowReaderHelper` which depends on
`ProtoUtil.fromProto` function. This PR extends `fromProto` to handle `struct`
type in order to avoid unknown type error in `ArrowReader` . The following is
the result of `Scala` client.
```
$ bin/spark-connect-shell --remote sc://localhost:15002
...
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 4.0.0-preview2
/_/
Type in expressions to have them evaluated.
Spark session available as 'spark'.
scala> sql("SELECT struct(1), struct(2, struct(3))").show()
+---------+--------------------+
|struct(1)|struct(2, struct(3))|
+---------+--------------------+
| {1}| {2, {3}}|
+---------+--------------------+
```
### Does this PR introduce _any_ user-facing change?
No. There is no behavior change.
### How was this patch tested?
Pass the CIs with the newly added test cases.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #50 from dongjoon-hyun/SPARK-51763.
Authored-by: Dongjoon Hyun <dongj...@apache.org>
Signed-off-by: Dongjoon Hyun <dongj...@apache.org>
---
Sources/SparkConnect/ProtoUtil.swift | 2 ++
Tests/SparkConnectTests/Resources/queries/struct.sql | 1 +
Tests/SparkConnectTests/Resources/queries/struct.sql.json | 1 +
3 files changed, 4 insertions(+)
diff --git a/Sources/SparkConnect/ProtoUtil.swift
b/Sources/SparkConnect/ProtoUtil.swift
index ace45a9..5a16269 100644
--- a/Sources/SparkConnect/ProtoUtil.swift
+++ b/Sources/SparkConnect/ProtoUtil.swift
@@ -64,6 +64,8 @@ func fromProto( // swiftlint:disable:this
cyclomatic_complexity
let arrowUnit: ArrowTime64Unit = timeType.unit == .microsecond ?
.microseconds : .nanoseconds
arrowType = ArrowTypeTime64(arrowUnit)
}
+ case .struct_:
+ arrowType = ArrowType(ArrowType.ArrowStruct)
default:
arrowType = ArrowType(ArrowType.ArrowUnknown)
}
diff --git a/Tests/SparkConnectTests/Resources/queries/struct.sql
b/Tests/SparkConnectTests/Resources/queries/struct.sql
new file mode 100644
index 0000000..3ec2472
--- /dev/null
+++ b/Tests/SparkConnectTests/Resources/queries/struct.sql
@@ -0,0 +1 @@
+SELECT struct(1), struct(2, struct(3))
diff --git a/Tests/SparkConnectTests/Resources/queries/struct.sql.json
b/Tests/SparkConnectTests/Resources/queries/struct.sql.json
new file mode 100644
index 0000000..08ef088
--- /dev/null
+++ b/Tests/SparkConnectTests/Resources/queries/struct.sql.json
@@ -0,0 +1 @@
+[["{1}","{2,{3}}"]]
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
