This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 2b8d5af0bd87 [SPARK-57760][SS][PYTHON][FOLLOWUP] Fix mypy union-attr
error in _normalize_state_value
2b8d5af0bd87 is described below
commit 2b8d5af0bd8713485615b4ba32ef1f017a14bcb2
Author: Kousuke Saruta <[email protected]>
AuthorDate: Fri Jul 10 07:57:50 2026 +0900
[SPARK-57760][SS][PYTHON][FOLLOWUP] Fix mypy union-attr error in
_normalize_state_value
### What changes were proposed in this pull request?
This PR adds `np is not None` guard before `isinstance(v, np.generic)` in
`_normalize_state_value` to fix a mypy `union-attr` error.
### Why are the changes needed?
To recover CI.
https://github.com/apache/spark/actions/runs/29053540012/job/86240085317
```
annotations failed mypy checks:
Error: python/pyspark/sql/streaming/stateful_processor_api_client.py:64:
error: Item "None" of Module | None has no attribute "generic" [union-attr]
python/pyspark/worker.py:2114: note: By default the bodies of untyped
functions are not checked, consider using --check-untyped-defs
[annotation-unchecked]
Found 1 error in 1 file (checked 1243 source files)
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- `python3 -m mypy --config-file python/mypy.ini
python/pyspark/sql/streaming/stateful_processor_api_client.py` passes.
-
`pyspark.tests.test_import_spark.ImportSparkTest.test_import_spark_libraries`
still passes.
### Was this patch authored or co-authored using generative AI tooling?
Kiro CLI / Claude
Closes #57173 from sarutak/fix-mypy-stateful-processor-api-client.
Authored-by: Kousuke Saruta <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
(cherry picked from commit 13fbf14363da905963f6f441376f62d053ab8ad2)
Signed-off-by: Hyukjin Kwon <[email protected]>
---
python/pyspark/sql/streaming/stateful_processor_api_client.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/python/pyspark/sql/streaming/stateful_processor_api_client.py
b/python/pyspark/sql/streaming/stateful_processor_api_client.py
index a679c499dc08..4be92ff38dad 100644
--- a/python/pyspark/sql/streaming/stateful_processor_api_client.py
+++ b/python/pyspark/sql/streaming/stateful_processor_api_client.py
@@ -61,7 +61,7 @@ def _normalize_state_value(v: Any) -> Any:
if type(v) in SCALAR_TYPES: # Fast path for common scalar values.
return v
# Convert NumPy scalar values to Python primitive values.
- if isinstance(v, np.generic):
+ if np is not None and isinstance(v, np.generic):
return v.tolist()
# Named tuples (collections.namedtuple or typing.NamedTuple) and Row both
# require positional arguments and cannot be instantiated with a generator
expression.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
