dramaticlly commented on code in PR #16750:
URL: https://github.com/apache/iceberg/pull/16750#discussion_r4383033158
##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/SparkBatch.java:
##########
@@ -64,6 +64,8 @@ class SparkBatch implements Batch {
private final boolean executorCacheLocalityEnabled;
private final int scanHashCode;
private final boolean cacheDeleteFilesOnExecutors;
+ private final boolean orderingEnabled;
+ private final boolean anyTaskNeedsMerging;
Review Comment:
I think there's a lot of plumbing involved to pass down the ordering flag
which essentially depends on the state of ScanTaskGroup, I am wondering if
worth a new `org.apache.iceberg.OrderedTaskGroup` in core module ? then
needsMerging is implied to instance check of OrderedTaskGroup and at least 2
tasks?
it shall
- reduce the new flag here
- drop the change in SparkChangelogScan
- drop the change in SparkScan
- new constructor in SparkRowReaderFactory and remain stateless
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
