raghav-reglobe opened a new pull request, #67406:
URL: https://github.com/apache/doris/pull/67406
### What problem does this PR solve?
Issue Number: N/A
Related PR: #64304 (where the plugin-world port of this path landed; the
data-manifest side got the projection, the delete-manifest side did not)
Problem Summary:
`cacheBackedFileScanTasks` — the manifest-cache planning path shared by the
synchronous, streaming, and COUNT(*) plans — prunes **delete** manifests with:
```java
ManifestEvaluator.forPartitionFilter(filterExpr, spec,
caseSensitive).eval(manifest)
```
passing the **raw row filter**. `forPartitionFilter` binds against the
partition struct, so this works by accident on identity-only specs (the
partition field keeps the source column name) and throws on any spec with a
transform — e.g. `(identity(flag), month(ts))` stores the field as `ts_month:
int` — whenever the filter references the transform's source column. The catch
in `planFileScanTask` then aborts the whole cached plan into the SDK fallback,
once per query:
```
[IcebergScanPlanProvider.planFileScanTask():2468] Iceberg plan with manifest
cache failed, falling back to SDK scan: Cannot find field 'ts' in struct:
struct<1000: flag: optional boolean, 1001: ts_month: optional int>
org.apache.iceberg.exceptions.ValidationException: Cannot find field 'ts' in
struct: ...
at
org.apache.iceberg.expressions.NamedReference.bind(NamedReference.java:45)
...
```
Net effect on v2 tables with delete files and a time-transform partition
spec: **every filtered query silently loses the manifest cache** (planning
latency + repeated catalog/storage manifest reads) and logs a WARN with a full
stack trace. On one production FE where most queries filter on the transform's
source column we measured ~1,100 such stacks per hour; the fallback also means
the manifest cache is effectively dead for the hottest query shape on those
tables.
The **data-manifest** side of the very same method (`getMatchingManifest`)
already projects the filter into partition space before building its evaluator.
This PR does the same for delete manifests — `Projections.inclusive(spec,
caseSensitive).project(rowFilter)` — via a small package-private helper so the
behavior is unit-testable. An inclusive projection maps predicates on
non-partition columns to `alwaysTrue()`, so pruning semantics are unchanged;
the projected transform predicates now actually prune delete manifests instead
of failing to bind.
### Release note
Fix Iceberg manifest-cache planning falling back to the SDK scan (with a
per-query WARN + stack trace) for every filtered query on tables whose
partition spec contains a transform (e.g. `month(ts)`) and that carry delete
files.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason.
`IcebergScanPlanProviderDeleteManifestPruneTest` (real `InMemoryCatalog`
v2 table, `(identity(flag), month(ts))` spec, one position-delete file): pins
that the raw row filter still fails to bind (the projection stays
load-bearing), the projected evaluator keeps an overlapping month and prunes a
far month, the identity leg still prunes, and a residual-only filter keeps the
manifest.
- Behavior changed:
- [x] No.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merged this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
