szehon-ho commented on code in PR #7499:
URL: https://github.com/apache/iceberg/pull/7499#discussion_r1183092869
##########
docs/spark-writes.md:
##########
@@ -312,20 +312,12 @@ data.writeTo("prod.db.table")
.createOrReplace()
```
-## Writing to partitioned tables
+## Writing Distribution Modes
-Iceberg requires the data to be sorted according to the partition spec per
task (Spark partition) in prior to write
-against partitioned table. This applies both Writing with SQL and Writing with
DataFrames.
-
-{{< hint info >}}
-Explicit sort is necessary because Spark doesn't allow Iceberg to request a
sort before writing as of Spark 3.0.
-[SPARK-23889](https://issues.apache.org/jira/browse/SPARK-23889) is filed to
enable Iceberg to require specific
-distribution & sort order to Spark.
-{{< /hint >}}
-
-{{< hint info >}}
-Both global sort (`orderBy`/`sort`) and local sort (`sortWithinPartitions`)
work for the requirement.
-{{< /hint >}}
+Iceberg's default Spark writers require that the data in each spark task is
clustered by partition values. This
+distribution is required to minimize the number of file handles that are held
open while writing. By default, starting
+in Iceberg 1.2.0, Iceberg now also requests that Spark pre-sort data to be
written to fit this distribution. The
Review Comment:
Nit: 'now' may be a bit redundant with 'starting in Iceberg 1.2.0'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
