This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.5
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.5 by this push:
new bd076eb2364 [SPARK-44822][PYTHON][FOLLOW-UP] Make Python UDTFs by
default non-deterministic
bd076eb2364 is described below
commit bd076eb2364a27d4e22c6767aff2e43c093bacd1
Author: allisonwang-db <[email protected]>
AuthorDate: Thu Aug 24 19:31:48 2023 -0700
[SPARK-44822][PYTHON][FOLLOW-UP] Make Python UDTFs by default
non-deterministic
### What changes were proposed in this pull request?
This PR is a follow up for SPARK-44822. It modifies one more default value
for Python UDTF to make it by default non-deterministic.
### Why are the changes needed?
To prevent future issues.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Existing tests
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #42665 from allisonwang-db/spark-44822-follow-up.
Authored-by: allisonwang-db <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 492064dd95eaa50bd30c363c97d3a703fd39c872)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
python/pyspark/sql/udtf.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/python/pyspark/sql/udtf.py b/python/pyspark/sql/udtf.py
index c345df1aaab..7e056589b7d 100644
--- a/python/pyspark/sql/udtf.py
+++ b/python/pyspark/sql/udtf.py
@@ -44,7 +44,7 @@ def _create_udtf(
returnType: Union[StructType, str],
name: Optional[str] = None,
evalType: int = PythonEvalType.SQL_TABLE_UDF,
- deterministic: bool = True,
+ deterministic: bool = False,
) -> "UserDefinedTableFunction":
"""Create a Python UDTF with the given eval type."""
udtf_obj = UserDefinedTableFunction(
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
