This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.5
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.5 by this push:
new daa3281e6a6 [SPARK-45538][PYTHON][CONNECT] pyspark connect
overwrite_partitions bug
daa3281e6a6 is described below
commit daa3281e6a68845943fcf61ba7ad1d2d3c8be28f
Author: xieshuaihu <[email protected]>
AuthorDate: Mon Oct 16 17:01:18 2023 +0900
[SPARK-45538][PYTHON][CONNECT] pyspark connect overwrite_partitions bug
Fix a bug in pyspark connect.
DataFrameWriterV2.overwritePartitions set mode as overwrite_partitions
[pyspark/sql/connect/readwriter.py, line 825], but WirteOperationV2 take it as
overwrite_partition [pyspark/sql/connect/plan.py, line 1660]
make dataframe.writeTo(table).overwritePartitions() work
No
No test. This bug is very obvious.
No
Closes #43367 from xieshuaihu/python_connect_overwrite.
Authored-by: xieshuaihu <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
(cherry picked from commit 9bdad31039134b492caeeba430120d5978a085ee)
Signed-off-by: Hyukjin Kwon <[email protected]>
---
python/pyspark/sql/connect/plan.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/python/pyspark/sql/connect/plan.py
b/python/pyspark/sql/connect/plan.py
index b7ea1f94993..9af5823dd8b 100644
--- a/python/pyspark/sql/connect/plan.py
+++ b/python/pyspark/sql/connect/plan.py
@@ -1655,7 +1655,7 @@ class WriteOperationV2(LogicalPlan):
plan.write_operation_v2.mode =
proto.WriteOperationV2.Mode.MODE_CREATE
elif wm == "overwrite":
plan.write_operation_v2.mode =
proto.WriteOperationV2.Mode.MODE_OVERWRITE
- elif wm == "overwrite_partition":
+ elif wm == "overwrite_partitions":
plan.write_operation_v2.mode =
proto.WriteOperationV2.Mode.MODE_OVERWRITE_PARTITIONS
elif wm == "append":
plan.write_operation_v2.mode =
proto.WriteOperationV2.Mode.MODE_APPEND
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
