This is an automated email from the ASF dual-hosted git repository.
xinrong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 84858f3d8be0 [SPARK-52215][PYTHON][TESTS][FOLLOW-UP] Fix
`test_arrow_udf_output_nested_arrays`
84858f3d8be0 is described below
commit 84858f3d8be029dcfd3410d1edce9117969557bb
Author: Ruifeng Zheng <ruife...@apache.org>
AuthorDate: Thu Aug 7 11:44:14 2025 -0700
[SPARK-52215][PYTHON][TESTS][FOLLOW-UP] Fix
`test_arrow_udf_output_nested_arrays`
### What changes were proposed in this pull request?
this test is incorrect, but it happened to pass with special partitioning
### Why are the changes needed?
to correct a test, it fails with different envs
### Does this PR introduce _any_ user-facing change?
no, test-only
### How was this patch tested?
ci and manually check with pyspark shell
### Was this patch authored or co-authored using generative AI tooling?
no
Closes #51898 from zhengruifeng/fix_nest_array_test.
Authored-by: Ruifeng Zheng <ruife...@apache.org>
Signed-off-by: Xinrong Meng <xinr...@apache.org>
---
python/pyspark/sql/tests/arrow/test_arrow_udf_scalar.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/python/pyspark/sql/tests/arrow/test_arrow_udf_scalar.py
b/python/pyspark/sql/tests/arrow/test_arrow_udf_scalar.py
index 97db29ae6590..15e5123b65ed 100644
--- a/python/pyspark/sql/tests/arrow/test_arrow_udf_scalar.py
+++ b/python/pyspark/sql/tests/arrow/test_arrow_udf_scalar.py
@@ -97,7 +97,7 @@ class ScalarArrowUDFTestsMixin:
df = self.spark.createDataFrame([("hi boo",), ("bye boo",)], ["vals"])
tokenize = arrow_udf(
- lambda s:
pa.array([pa.compute.ascii_split_whitespace(s).to_pylist()]),
+ lambda s: pa.array([[v] for v in
pa.compute.ascii_split_whitespace(s).to_pylist()]),
ArrayType(ArrayType(StringType())),
)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
