aokolnychyi commented on code in PR #8660:
URL: https://github.com/apache/iceberg/pull/8660#discussion_r1338931458
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkWriteRequirements.java:
##########
@@ -20,20 +20,24 @@
import org.apache.spark.sql.connector.distributions.Distribution;
import org.apache.spark.sql.connector.distributions.Distributions;
+import org.apache.spark.sql.connector.distributions.UnspecifiedDistribution;
import org.apache.spark.sql.connector.expressions.SortOrder;
/** A set of requirements such as distribution and ordering reported to Spark
during writes. */
public class SparkWriteRequirements {
public static final SparkWriteRequirements EMPTY =
- new SparkWriteRequirements(Distributions.unspecified(), new
SortOrder[0]);
+ new SparkWriteRequirements(Distributions.unspecified(), new
SortOrder[0], 0);
Review Comment:
Yes, it matches the default value in Spark and means no preference. See
`RequiresDistributionAndOrdering`.
```
/**
* Returns the advisory (not guaranteed) shuffle partition size in bytes
for this write.
* <p>
* Implementations may override this to indicate the preferable partition
size in shuffles
* performed to satisfy the requested distribution. Note that Spark
doesn't support setting
* the advisory partition size for {@link UnspecifiedDistribution}, the
query will fail if
* the advisory partition size is set but the distribution is unspecified.
Data sources may
* either request a particular number of partitions via {@link
#requiredNumPartitions()} or
* a preferred partition size, not both.
* <p>
* Data sources should be careful with large advisory sizes as it will
impact the writing
* parallelism and may degrade the overall job performance.
* <p>
* Note this value only acts like a guidance and Spark does not guarantee
the actual and advisory
* shuffle partition sizes will match. Ignored if the adaptive execution
is disabled.
*
* @return the advisory partition size, any value less than 1 means no
preference.
*/
default long advisoryPartitionSizeInBytes() { return 0; }
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
