rdblue commented on code in PR #7714:
URL: https://github.com/apache/iceberg/pull/7714#discussion_r1278598322
##########
spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/source/SparkPartitioningAwareScan.java:
##########
@@ -232,6 +232,16 @@ protected synchronized List<ScanTaskGroup<T>> taskGroups()
{
return taskGroups;
}
+ private long targetSplitSize() {
+ if (readConf().adaptiveSplitSizeEnabled()) {
+ long scanSize = tasks().stream().mapToLong(ScanTask::sizeBytes).sum();
+ int parallelism = sparkContext().defaultParallelism();
Review Comment:
> The core count is being updated each time an executor is added/dropped so
dynamic allocation should work.
I don't think it would because the job may be planned before the initial
stage is submitted and the cluster scales up. I think shuffle parallelism is
the most reliable way to know how big to go.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
