This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 544eccc6a7f6 [SPARK-54143][INFRA] Add `branch-4.1` Daily GitHub Action
jobs
544eccc6a7f6 is described below
commit 544eccc6a7f69d1a16ee979a209203727ddacad1
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sun Nov 2 18:56:58 2025 -0800
[SPARK-54143][INFRA] Add `branch-4.1` Daily GitHub Action jobs
### What changes were proposed in this pull request?
This PR aims to add `branch-4.1` Daily GitHub Action jobs.
### Why are the changes needed?
Like `4.0.x`, we need to maintain the full test coverage for Apache Spark
4.1.0 preparation. The following jobs will be added.
-
https://github.com/apache/spark/actions/workflows/build_branch41_java21.yml
-
https://github.com/apache/spark/actions/workflows/build_branch41_non_ansi.yml
- https://github.com/apache/spark/actions/workflows/build_branch41_maven.yml
-
https://github.com/apache/spark/actions/workflows/build_branch41_maven_java21.yml
-
https://github.com/apache/spark/actions/workflows/build_branch41_python.yml
-
https://github.com/apache/spark/actions/workflows/build_branch41_python_pypy3.10.yml
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manual review.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #52842 from dongjoon-hyun/SPARK-54143.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.github/workflows/build_branch41.yml | 53 ++++++++++++++++++++
.github/workflows/build_branch41_java21.yml | 57 ++++++++++++++++++++++
.github/workflows/build_branch41_maven.yml | 35 +++++++++++++
.github/workflows/build_branch41_maven_java21.yml | 36 ++++++++++++++
.github/workflows/build_branch41_non_ansi.yml | 53 ++++++++++++++++++++
.github/workflows/build_branch41_python.yml | 47 ++++++++++++++++++
.../workflows/build_branch41_python_pypy3.10.yml | 47 ++++++++++++++++++
README.md | 7 +++
8 files changed, 335 insertions(+)
diff --git a/.github/workflows/build_branch41.yml
b/.github/workflows/build_branch41.yml
new file mode 100644
index 000000000000..a9ee7057cd53
--- /dev/null
+++ b/.github/workflows/build_branch41.yml
@@ -0,0 +1,53 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: "Build (branch-4.1, Scala 2.13, Hadoop 3, JDK 17)"
+
+on:
+ schedule:
+ - cron: '0 12 * * *'
+ workflow_dispatch:
+
+jobs:
+ run-build:
+ permissions:
+ packages: write
+ name: Run
+ uses: ./.github/workflows/build_and_test.yml
+ if: github.repository == 'apache/spark'
+ with:
+ java: 17
+ branch: branch-4.1
+ hadoop: hadoop3
+ envs: >-
+ {
+ "SCALA_PROFILE": "scala2.13",
+ "PYSPARK_IMAGE_TO_TEST": "",
+ "PYTHON_TO_TEST": "",
+ "ORACLE_DOCKER_IMAGE_NAME": "gvenzl/oracle-free:23.7-slim"
+ }
+ jobs: >-
+ {
+ "build": "true",
+ "sparkr": "true",
+ "tpcds-1g": "true",
+ "docker-integration-tests": "true",
+ "k8s-integration-tests": "true",
+ "lint" : "true"
+ }
diff --git a/.github/workflows/build_branch41_java21.yml
b/.github/workflows/build_branch41_java21.yml
new file mode 100644
index 000000000000..4df4cfd9796d
--- /dev/null
+++ b/.github/workflows/build_branch41_java21.yml
@@ -0,0 +1,57 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: "Build (branch-4.1, Scala 2.13, Hadoop 3, JDK 21)"
+
+on:
+ schedule:
+ - cron: '0 5 * * *'
+ workflow_dispatch:
+
+jobs:
+ run-build:
+ permissions:
+ packages: write
+ name: Run
+ uses: ./.github/workflows/build_and_test.yml
+ if: github.repository == 'apache/spark'
+ with:
+ java: 21
+ branch: branch-4.1
+ hadoop: hadoop3
+ envs: >-
+ {
+ "PYSPARK_IMAGE_TO_TEST": "python-311",
+ "PYTHON_TO_TEST": "python3.11",
+ "SKIP_MIMA": "true",
+ "SKIP_UNIDOC": "true",
+ "DEDICATED_JVM_SBT_TESTS":
"org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormatV1Suite,org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormatV2Suite,org.apache.spark.sql.execution.datasources.orc.OrcSourceV1Suite,org.apache.spark.sql.execution.datasources.orc.OrcSourceV2Suite"
+ }
+ jobs: >-
+ {
+ "build": "true",
+ "pyspark": "true",
+ "sparkr": "true",
+ "tpcds-1g": "true",
+ "docker-integration-tests": "true",
+ "yarn": "true",
+ "k8s-integration-tests": "true",
+ "buf": "true",
+ "ui": "true"
+ }
diff --git a/.github/workflows/build_branch41_maven.yml
b/.github/workflows/build_branch41_maven.yml
new file mode 100644
index 000000000000..0cb38cbb067e
--- /dev/null
+++ b/.github/workflows/build_branch41_maven.yml
@@ -0,0 +1,35 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: "Build / Maven (branch-4.1, Scala 2.13, Hadoop 3, JDK 17)"
+
+on:
+ schedule:
+ - cron: '0 14 * * *'
+ workflow_dispatch:
+
+jobs:
+ run-build:
+ permissions:
+ packages: write
+ name: Run
+ uses: ./.github/workflows/maven_test.yml
+ if: github.repository == 'apache/spark'
+ with:
+ branch: branch-4.1
diff --git a/.github/workflows/build_branch41_maven_java21.yml
b/.github/workflows/build_branch41_maven_java21.yml
new file mode 100644
index 000000000000..42bc5f03fd89
--- /dev/null
+++ b/.github/workflows/build_branch41_maven_java21.yml
@@ -0,0 +1,36 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: "Build / Maven (branch-4.1, Scala 2.13, Hadoop 3, JDK 21)"
+
+on:
+ schedule:
+ - cron: '0 14 * * *'
+ workflow_dispatch:
+
+jobs:
+ run-build:
+ permissions:
+ packages: write
+ name: Run
+ uses: ./.github/workflows/maven_test.yml
+ if: github.repository == 'apache/spark'
+ with:
+ branch: branch-4.1
+ java: 21
diff --git a/.github/workflows/build_branch41_non_ansi.yml
b/.github/workflows/build_branch41_non_ansi.yml
new file mode 100644
index 000000000000..b2fc650022b8
--- /dev/null
+++ b/.github/workflows/build_branch41_non_ansi.yml
@@ -0,0 +1,53 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: "Build / Non-ANSI (branch-4.1, Hadoop 3, JDK 17, Scala 2.13)"
+
+on:
+ schedule:
+ - cron: '0 2 * * *'
+ workflow_dispatch:
+
+jobs:
+ run-build:
+ permissions:
+ packages: write
+ name: Run
+ uses: ./.github/workflows/build_and_test.yml
+ if: github.repository == 'apache/spark'
+ with:
+ java: 17
+ branch: branch-4.1
+ hadoop: hadoop3
+ envs: >-
+ {
+ "PYSPARK_IMAGE_TO_TEST": "python-311",
+ "PYTHON_TO_TEST": "python3.11",
+ "SPARK_ANSI_SQL_MODE": "false",
+ }
+ jobs: >-
+ {
+ "build": "true",
+ "docs": "true",
+ "pyspark": "true",
+ "sparkr": "true",
+ "tpcds-1g": "true",
+ "docker-integration-tests": "true",
+ "yarn": "true"
+ }
diff --git a/.github/workflows/build_branch41_python.yml
b/.github/workflows/build_branch41_python.yml
new file mode 100644
index 000000000000..2e45c49b0d8c
--- /dev/null
+++ b/.github/workflows/build_branch41_python.yml
@@ -0,0 +1,47 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: "Build / Python-only (branch-4.1)"
+
+on:
+ schedule:
+ - cron: '0 12 * * *'
+ workflow_dispatch:
+
+jobs:
+ run-build:
+ permissions:
+ packages: write
+ name: Run
+ uses: ./.github/workflows/build_and_test.yml
+ if: github.repository == 'apache/spark'
+ with:
+ java: 17
+ branch: branch-4.1
+ hadoop: hadoop3
+ envs: >-
+ {
+ "PYSPARK_IMAGE_TO_TEST": "python-311",
+ "PYTHON_TO_TEST": "python3.11"
+ }
+ jobs: >-
+ {
+ "pyspark": "true",
+ "pyspark-pandas": "true"
+ }
diff --git a/.github/workflows/build_branch41_python_pypy3.10.yml
b/.github/workflows/build_branch41_python_pypy3.10.yml
new file mode 100644
index 000000000000..8aa0e97a9ffd
--- /dev/null
+++ b/.github/workflows/build_branch41_python_pypy3.10.yml
@@ -0,0 +1,47 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: "Build / Python-only (branch-4.1, PyPy 3.10)"
+
+on:
+ schedule:
+ - cron: '0 16 * * *'
+ workflow_dispatch:
+
+jobs:
+ run-build:
+ permissions:
+ packages: write
+ name: Run
+ uses: ./.github/workflows/build_and_test.yml
+ if: github.repository == 'apache/spark'
+ with:
+ java: 17
+ branch: branch-4.1
+ hadoop: hadoop3
+ envs: >-
+ {
+ "PYSPARK_IMAGE_TO_TEST": "pypy-310",
+ "PYTHON_TO_TEST": "pypy3"
+ }
+ jobs: >-
+ {
+ "pyspark": "true",
+ "pyspark-pandas": "true"
+ }
diff --git a/README.md b/README.md
index f3435178c68e..1e88ee6e077e 100644
--- a/README.md
+++ b/README.md
@@ -52,6 +52,13 @@ This README file only contains basic setup instructions.
| | [](https://github.com/apache/spark/actions/workflows/build_python_connect.yml)
|
| | [](https://github.com/apache/spark/actions/workflows/build_sparkr_window.yml)
|
| | [](https://github.com/apache/spark/actions/workflows/publish_snapshot.yml)
|
+| branch-4.1 | [](https://github.com/apache/spark/actions/workflows/build_branch41.yml)
|
+| | [](https://github.com/apache/spark/actions/workflows/build_branch41_java21.yml)
|
+| | [](https://github.com/apache/spark/actions/workflows/build_branch41_non_ansi.yml)
|
+| | [](https://github.com/apache/spark/actions/workflows/build_branch41_maven.yml)
|
+| | [](https://github.com/apache/spark/actions/workflows/build_branch41_maven_java21.yml)
|
+| | [](https://github.com/apache/spark/actions/workflows/build_branch41_python.yml)
|
+| | [](https://github.com/apache/spark/actions/workflows/build_branch41_python_pypy3.10.yml)
|
| branch-4.0 | [](https://github.com/apache/spark/actions/workflows/build_branch40.yml)
|
| | [](https://github.com/apache/spark/actions/workflows/build_branch40_java21.yml)
|
| | [](https://github.com/apache/spark/actions/workflows/build_branch40_non_ansi.yml)
|
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
