alexanderbianchi commented on code in PR #3667:
URL: https://github.com/apache/iceberg-rust/pull/3667#discussion_r4607054870
##########
crates/iceberg/src/scan/context.rs:
##########
@@ -354,21 +395,29 @@ impl PlanContext {
)
.eval(manifest_file)?
{
- continue;
+ skipped = true;
}
Some(partition_bound_predicate)
} else {
None
};
+ if skipped {
Review Comment:
The description says manifests containing only deleted entries are skipped,
but `skipped` is only set by predicate evaluation here.
I reproduced this with one data manifest containing only `DELETED` entries,
with both `added_files_count` and `existing_files_count` equal to `Some(0)`.
The scan returns no tasks, but reports:
```text
scanned_data_manifests = 1
skipped_data_manifests = 0
```
[Java excludes these manifests before incrementing the scanned
counter](https://github.com/apache/iceberg/blob/2517ec2a7f70721166bd2b2ca0061984e4eb2bdc/core/src/main/java/org/apache/iceberg/ManifestGroup.java#L287-L308),
producing scanned=0 and skipped=1.
Should this also check `!manifest_file.has_added_files() &&
!manifest_file.has_existing_files()`? The existing helpers conservatively scan
when either count is unknown.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]