rdblue commented on code in PR #18884:
URL: https://github.com/apache/iceberg/pull/18884#discussion_r4566941365
##########
core/src/main/java/org/apache/iceberg/SnapshotScan.java:
##########
@@ -185,6 +186,37 @@ public Snapshot snapshot() {
return snapshotId() != null ? table().snapshot(snapshotId()) :
table().currentSnapshot();
}
+ /** Plans file scan tasks for a snapshot that references a v4 root manifest.
*/
+ CloseableIterable<FileScanTask> planV4RootManifest(Snapshot snapshot) {
+ ManifestFile root = RootManifestFile.from(snapshot, table().io(),
table().location());
+ FilePlanner planner =
+ new FilePlanner(table().io(), root, tableSchema(), specs())
+ .filterData(filter())
+ .caseSensitive(isCaseSensitive())
+ .scanMetrics(scanMetrics())
+ .tableLocation(table().location())
+ .planWith(shouldPlanWithExecutor() ? planExecutor() : null);
+
+ if (shouldIgnoreResiduals()) {
+ planner.ignoreResiduals();
+ }
+
+ if (shouldReturnColumnStats()) {
+ planner.projectStats(statsColumnIds());
+ }
+
+ return planner.planFiles();
+ }
+
+ private int[] statsColumnIds() {
+ Set<Integer> columns = columnsToKeepStats();
+ if (columns == null) {
+ columns = TypeUtil.getProjectedIds(tableSchema());
Review Comment:
This logic is only correct if `shouldReturnColumnStats` is true.
By default, stats are not copied. There are two ways to produce stats from
the scan:
- `Scan.includeColumnStats()` returns all stats
- `Scan.includeColumnStats(Collection<Integer>)` returns stats for specific
field IDs
Both of those methods set `shouldReturnColumnStats(true)` on the context,
but the second one also sets `columnsToKeepStats(Set<Integer>)`. That means
`columnsToKeepStats()` is more specific than `returnColumnStats()`.
The correct logic is:
- If `columnsToKeepStats()` is non-null, use that set
- Otherwise, if `returnColumnStats()` is true, pass
`TypeUtil.getProjectedIds(tableSchema())`
- Otherwise, do not call `projectStats`
That means the correctness of this method is dependent on how it is called.
A better solution is to use the `Set<Integer>` argument as @amogh-jahagirdar
suggests and embed this logic above:
```java
if (shouldReturnColumnStats()) {
if (columnsToKeepStats() != null) {
planner.projectStats(columnsToKeepStats());
} else {
planner.projectStats(TypeUtil.getProjectedIds(tableSchema()));
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]