This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 8b0d04eee390 [SPARK-56889][INFRA][FOLLOWUP] Restore python3.10 in
dev/infra/Dockerfile
8b0d04eee390 is described below
commit 8b0d04eee390e0992f0122581186aaf6d0398c68
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Mon May 18 09:58:37 2026 -0700
[SPARK-56889][INFRA][FOLLOWUP] Restore python3.10 in dev/infra/Dockerfile
### What changes were proposed in this pull request?
This is a partial revert of #55914 (SPARK-56889) restricted to
`dev/infra/Dockerfile`. It restores the file to its state at `0a0d31bea00~1` —
switching the system Python used in the base CI image from `python3.11` back to
`python3.10`. All other 20 files changed by #55914 are kept as-is.
### Why are the changes needed?
The `Build / Cache base image` workflow has been failing on every branch
since #55914 was merged (2026-05-16):
https://github.com/apache/spark/actions/runs/26010261434/job/76449371588
The failure aborts at the first `Build and push` step (the base
`./dev/infra/` image), with:
```
RUN add-apt-repository ppa:deadsnakes/ppa
...
ModuleNotFoundError: No module named 'pyparsing'
ERROR: process "/bin/sh -c add-apt-repository ppa:deadsnakes/ppa" did not
complete successfully: exit code: 1
```
This breaks the cache build for all downstream image jobs, since the base
layer is shared. Restoring the previous Dockerfile is the smallest change that
unblocks the cache workflow while a forward fix is investigated.
### Does this PR introduce _any_ user-facing change?
No. CI-only change. PySpark itself is unaffected — only the system Python
inside the test image.
### How was this patch tested?
This PR re-runs `Build / Cache base image` against the restored Dockerfile.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (model: claude-opus-4-7)
Closes #55946 from zhengruifeng/restore-infra-dockerfile-python310.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 3608538009238741812ec6c8f482eec1c4553bd8)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
dev/infra/Dockerfile | 18 +++++++++---------
1 file changed, 9 insertions(+), 9 deletions(-)
diff --git a/dev/infra/Dockerfile b/dev/infra/Dockerfile
index 9c0584f6b02d..0fc11e1021a8 100644
--- a/dev/infra/Dockerfile
+++ b/dev/infra/Dockerfile
@@ -61,7 +61,7 @@ RUN apt-get update && apt-get install -y \
openjdk-17-jdk-headless \
pandoc \
pkg-config \
- python3.11 \
+ python3.10 \
python3-psutil \
python3-pyparsing \
qpdf \
@@ -95,14 +95,14 @@ ARG BASIC_PIP_PKGS="numpy pyarrow>=18.0.0 six==1.16.0
pandas==2.3.3 scipy plotly
# Python deps for Spark Connect
ARG CONNECT_PIP_PKGS="grpcio==1.76.0 grpcio-status==1.76.0 protobuf==6.33.5
googleapis-common-protos==1.71.0 graphviz==0.20.3"
-# Install Python 3.11 packages
-RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.11
-RUN python3.11 -m pip install --ignore-installed 'blinker>=1.6.2' # mlflow
needs this
-RUN python3.11 -m pip install --ignore-installed 'six==1.16.0' # Avoid
`python3-six` installation
-RUN python3.11 -m pip install $BASIC_PIP_PKGS unittest-xml-reporting
$CONNECT_PIP_PKGS && \
- python3.11 -m pip install torch torchvision --index-url
https://download.pytorch.org/whl/cpu && \
- python3.11 -m pip install deepspeed torcheval && \
- python3.11 -m pip cache purge
+# Install Python 3.10 packages
+RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.10
+RUN python3.10 -m pip install --ignore-installed 'blinker>=1.6.2' # mlflow
needs this
+RUN python3.10 -m pip install --ignore-installed 'six==1.16.0' # Avoid
`python3-six` installation
+RUN python3.10 -m pip install $BASIC_PIP_PKGS unittest-xml-reporting
$CONNECT_PIP_PKGS && \
+ python3.10 -m pip install torch torchvision --index-url
https://download.pytorch.org/whl/cpu && \
+ python3.10 -m pip install deepspeed torcheval && \
+ python3.10 -m pip cache purge
# Install Python 3.9
RUN add-apt-repository ppa:deadsnakes/ppa
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
