gerashegalov commented on code in PR #18709:
URL: https://github.com/apache/iceberg/pull/18709#discussion_r4520630446
##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java:
##########
@@ -134,6 +137,7 @@ abstract class SparkScan implements Scan,
SupportsReportStatistics {
this.projection = projection;
this.filters = filters != null ? filters : Collections.emptyList();
this.scanReportSupplier = scanReportSupplier;
+ this.effectiveSplitTargetSize = readConf.splitSize();
Review Comment:
Good catch, fixed in 2e3c2179e. The value now starts at 0 and is set only
where task groups are planned, so a scan without a snapshot reports 0 (covered
by `effectiveSplitTargetSizeWithoutSnapshot`). This brings back the one-line
`setEffectiveSplitTargetSize(splitSize)` in `SparkStagedScan.taskGroups()`,
which @liurenjie1024 suggested removing when the initial value was the
configured split size. The field is still a plain `long`.
##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java:
##########
@@ -301,13 +305,14 @@ public String toString() {
@Override
public CustomTaskMetric[] reportDriverMetrics() {
ScanReport scanReport = scanReportSupplier != null ?
scanReportSupplier.get() : null;
+ List<CustomTaskMetric> driverMetrics = Lists.newArrayList();
+
+ driverMetrics.add(new
TaskEffectiveSplitTargetSize(effectiveSplitTargetSize));
if (scanReport == null) {
- return new CustomTaskMetric[0];
+ return driverMetrics.toArray(new CustomTaskMetric[0]);
Review Comment:
Done in 2e3c2179e: the scan-report metrics are wrapped in `if (scanReport !=
null)` with a single return.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
