This is an automated email from the ASF dual-hosted git repository.
ueshin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 02c016fe4911 [SPARK-54762][PYTHON] Fix `_create_converter` and
`covert` overload signature
02c016fe4911 is described below
commit 02c016fe4911a18b53212b25bc25f62dd5db3a06
Author: Tian Gao <[email protected]>
AuthorDate: Thu Dec 18 11:46:35 2025 -0800
[SPARK-54762][PYTHON] Fix `_create_converter` and `covert` overload
signature
### What changes were proposed in this pull request?
Fix the overload signature of `_create_converter`. The default value should
be `False`.
Also fixed the overload signature of `convert`. The return value should be
List[Row | tuple] if `return_as_tuples` is passed - because an explicit `True`
can be passed.
### Why are the changes needed?
I would guess that the original author tried to express that "if
`none_on_identity` is `True`, then it could return `None`", but this is the
wrong expression. You can't express such a concept in the current Python typing
system. (You can use `Literal` but that requires the users to pass the literal
`True` or `False`, not a variable).
The current signature means the default value of `none_on_identity` is
`True` which is incorrect. The default value of a overload signature is not
used by the type checker, but it will be shown in IDE and this is misleading.
Same thing for `convert`.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
No code is changed.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #53467 from gaogaotiantian/conversion-overload.
Authored-by: Tian Gao <[email protected]>
Signed-off-by: Takuya Ueshin <[email protected]>
---
python/pyspark/sql/conversion.py | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/python/pyspark/sql/conversion.py b/python/pyspark/sql/conversion.py
index f73727d1d534..359dd32cc889 100644
--- a/python/pyspark/sql/conversion.py
+++ b/python/pyspark/sql/conversion.py
@@ -113,7 +113,7 @@ class LocalDataToArrowConversion:
dataType: DataType,
nullable: bool = True,
*,
- none_on_identity: bool = True,
+ none_on_identity: bool = False,
int_to_decimal_coercion_enabled: bool = False,
) -> Optional[Callable]:
pass
@@ -811,7 +811,9 @@ class ArrowTableToRowsConversion:
@overload
@staticmethod
- def convert(table: "pa.Table", schema: StructType, *, return_as_tuples:
bool) -> List[tuple]:
+ def convert(
+ table: "pa.Table", schema: StructType, *, return_as_tuples: bool
+ ) -> List[Row | tuple]:
pass
@staticmethod # type: ignore[misc]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
