nastra commented on code in PR #9439:
URL: https://github.com/apache/iceberg/pull/9439#discussion_r1455411644
##########
docs/spark-writes.md:
##########
@@ -343,7 +343,8 @@
data.writeTo("prod.db.sample").option("mergeSchema","true").append()
Iceberg's default Spark writers require that the data in each spark task is
clustered by partition values. This
distribution is required to minimize the number of file handles that are held
open while writing. By default, starting
in Iceberg 1.2.0, Iceberg also requests that Spark pre-sort data to be written
to fit this distribution. The
-request to Spark is done through the table property `write.distribution-mode`
with the value `hash`.
+request to Spark is done through the table property `write.distribution-mode`
with the value `hash`. Spark doesn't respect
+distribution mode in CTAS/RTAS before 3.5.0.
Review Comment:
do we have tests that check the behavior in Spark 3.4 + Spark 3.5 on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
