This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 8d1f88ff3026 [SPARK-56760][PYTHON] Remove dead numpy 1.21 version
check in pandas typehints
8d1f88ff3026 is described below
commit 8d1f88ff3026992cb0656d11bfb86f533fa8a3ff
Author: Yicong Huang <[email protected]>
AuthorDate: Tue May 12 09:53:34 2026 +0900
[SPARK-56760][PYTHON] Remove dead numpy 1.21 version check in pandas
typehints
### What changes were proposed in this pull request?
Removes the dead `if LooseVersion(np.__version__) >= LooseVersion("1.21"):`
guard in:
- `python/pyspark/pandas/typedef/typehints.py` (`as_spark_type`) -- inner
block handling `numpy.typing.NDArray` is dedented one level.
- `python/pyspark/pandas/tests/test_typedef.py`
(`test_as_spark_type_pandas_on_spark_dtype`) -- inner assertions on
`ntp.NDArray[...]` are dedented one level.
Accompanying `# For NumPy typing, NumPy version should be 1.21+` comments
are removed as well.
### Why are the changes needed?
[SPARK-45179](https://issues.apache.org/jira/browse/SPARK-45179)
(2023-09-15) bumped `_minimum_numpy_version` to `"1.21"` in `python/setup.py`,
so this version guard is always true and is dead code.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing CI.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #55746 from Yicong-Huang/SPARK-56760.
Authored-by: Yicong Huang <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
(cherry picked from commit 8a05ae713833ea98a49c34a740cf2c6f502d9286)
Signed-off-by: Hyukjin Kwon <[email protected]>
---
python/pyspark/pandas/tests/test_typedef.py | 20 +++++++++-----------
python/pyspark/pandas/typedef/typehints.py | 20 ++++++++------------
2 files changed, 17 insertions(+), 23 deletions(-)
diff --git a/python/pyspark/pandas/tests/test_typedef.py
b/python/pyspark/pandas/tests/test_typedef.py
index cb07b36a1c99..1b9db0a6a29a 100644
--- a/python/pyspark/pandas/tests/test_typedef.py
+++ b/python/pyspark/pandas/tests/test_typedef.py
@@ -407,17 +407,15 @@ class TypeHintTestsMixin:
(np.dtype("object"), ArrayType(spark_type)),
)
- # For NumPy typing, NumPy version should be 1.21+
- if LooseVersion(np.__version__) >= LooseVersion("1.21"):
- import numpy.typing as ntp
-
- self.assertEqual(
- as_spark_type(ntp.NDArray[numpy_or_python_type]),
ArrayType(spark_type)
- )
- self.assertEqual(
- pandas_on_spark_type(ntp.NDArray[numpy_or_python_type]),
- (np.dtype("object"), ArrayType(spark_type)),
- )
+ import numpy.typing as ntp
+
+ self.assertEqual(
+ as_spark_type(ntp.NDArray[numpy_or_python_type]),
ArrayType(spark_type)
+ )
+ self.assertEqual(
+ pandas_on_spark_type(ntp.NDArray[numpy_or_python_type]),
+ (np.dtype("object"), ArrayType(spark_type)),
+ )
with self.assertRaisesRegex(TypeError, "Type uint64 was not
understood."):
as_spark_type(np.dtype("uint64"))
diff --git a/python/pyspark/pandas/typedef/typehints.py
b/python/pyspark/pandas/typedef/typehints.py
index a98691efcb0d..5c3ea90f34ee 100644
--- a/python/pyspark/pandas/typedef/typehints.py
+++ b/python/pyspark/pandas/typedef/typehints.py
@@ -150,18 +150,14 @@ def as_spark_type(
- dictionaries of field_name -> type
- Python3's typing system
"""
- # For NumPy typing, NumPy version should be 1.21+
- if LooseVersion(np.__version__) >= LooseVersion("1.21"):
- if (
- hasattr(tpe, "__origin__")
- and tpe.__origin__ is np.ndarray
- and hasattr(tpe, "__args__")
- and len(tpe.__args__) > 1
- ):
- # numpy.typing.NDArray
- return types.ArrayType(
- as_spark_type(tpe.__args__[1].__args__[0],
raise_error=raise_error)
- )
+ if (
+ hasattr(tpe, "__origin__")
+ and tpe.__origin__ is np.ndarray
+ and hasattr(tpe, "__args__")
+ and len(tpe.__args__) > 1
+ ):
+ # numpy.typing.NDArray
+ return types.ArrayType(as_spark_type(tpe.__args__[1].__args__[0],
raise_error=raise_error))
if isinstance(tpe, np.dtype) and tpe == np.dtype("object"):
pass
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
