This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 64b0cb6a4654 [SPARK-53372][INFRA][FOLLOWUP] Synchronize the
installation of `pyyaml` into `maven_test.yml`
64b0cb6a4654 is described below
commit 64b0cb6a4654bb260401a6c0ec847578f0fa8021
Author: yangjie01 <[email protected]>
AuthorDate: Thu Sep 25 09:10:09 2025 -0700
[SPARK-53372][INFRA][FOLLOWUP] Synchronize the installation of `pyyaml`
into `maven_test.yml`
### What changes were proposed in this pull request?
This pr aims to synchronize the installation of `pyyaml` into
`maven_test.yml`
### Why are the changes needed?
Restore maven daily tests:
- https://github.com/apache/spark/actions/runs/17977674122/job/51135114818
```
- SQL Pipeline with mv, st, and flows *** FAILED ***
java.lang.RuntimeException: Pipeline update process failed with exit code
1.
Output:
Error: Traceback (most recent call last):
File "/home/runner/work/spark/spark/python/pyspark/pipelines/cli.py",
line 29, in <module>
import yaml
ModuleNotFoundError: No module named 'yaml'
at
org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.awaitPipelineTermination(EndToEndAPISuite.scala:124)
at
org.apache.spark.sql.pipelines.utils.APITest.$anonfun$$init$$1(APITest.scala:114)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
at
org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$2(EndToEndAPISuite.scala:47)
at
org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$2$adapted(EndToEndAPISuite.scala:45)
at
org.apache.spark.sql.test.SQLTestUtils.$anonfun$withTempDir$1(SQLTestUtils.scala:81)
at
org.apache.spark.sql.test.SQLTestUtils.$anonfun$withTempDir$1$adapted(SQLTestUtils.scala:80)
at org.apache.spark.SparkFunSuite.withTempDir(SparkFunSuite.scala:258)
at
org.apache.spark.sql.pipelines.utils.PipelineTest.org$apache$spark$sql$test$SQLTestUtils$$super$withTempDir(PipelineTest.scala:40)
at
org.apache.spark.sql.test.SQLTestUtils.withTempDir(SQLTestUtils.scala:80)
...
- SQL Pipeline with CTE *** FAILED ***
java.lang.RuntimeException: Pipeline update process failed with exit code
1.
Output:
Error: Traceback (most recent call last):
File "/home/runner/work/spark/spark/python/pyspark/pipelines/cli.py",
line 29, in <module>
import yaml
ModuleNotFoundError: No module named 'yaml'
at
org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.awaitPipelineTermination(EndToEndAPISuite.scala:124)
at
org.apache.spark.sql.pipelines.utils.APITest.$anonfun$$init$$2(APITest.scala:139)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
at
org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$2(EndToEndAPISuite.scala:47)
at
org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$2$adapted(EndToEndAPISuite.scala:45)
at
org.apache.spark.sql.test.SQLTestUtils.$anonfun$withTempDir$1(SQLTestUtils.scala:81)
at
org.apache.spark.sql.test.SQLTestUtils.$anonfun$withTempDir$1$adapted(SQLTestUtils.scala:80)
at org.apache.spark.SparkFunSuite.withTempDir(SparkFunSuite.scala:258)
at
org.apache.spark.sql.pipelines.utils.PipelineTest.org$apache$spark$sql$test$SQLTestUtils$$super$withTempDir(PipelineTest.scala:40)
at
org.apache.spark.sql.test.SQLTestUtils.withTempDir(SQLTestUtils.scala:80)
...
```
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Pass Github Actions
- Test with maven:
https://github.com/LuciferYang/spark/actions/runs/18000700692/job/51209057251
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #52450 from LuciferYang/SPARK-53372-FOLLOWUP.
Authored-by: yangjie01 <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.github/workflows/maven_test.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.github/workflows/maven_test.yml b/.github/workflows/maven_test.yml
index 93f6e38a8ee1..914fe2e6f53a 100644
--- a/.github/workflows/maven_test.yml
+++ b/.github/workflows/maven_test.yml
@@ -175,7 +175,7 @@ jobs:
- name: Install Python packages (Python 3.11)
if: contains(matrix.modules, 'resource-managers#yarn') ||
(contains(matrix.modules, 'sql#core')) || contains(matrix.modules, 'connect')
run: |
- python3.11 -m pip install 'numpy>=1.22' pyarrow pandas scipy
unittest-xml-reporting 'grpcio==1.67.0' 'grpcio-status==1.67.0'
'protobuf==5.29.5'
+ python3.11 -m pip install 'numpy>=1.22' pyarrow pandas pyyaml scipy
unittest-xml-reporting 'grpcio==1.67.0' 'grpcio-status==1.67.0'
'protobuf==5.29.5'
python3.11 -m pip list
# Run the tests using script command.
# BSD's script command doesn't support -c option, and the usage is
different from Linux's one.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
