ion-elgreco commented on issue #25775:
URL: https://github.com/apache/datafusion/issues/25775#issuecomment-5868531913
@xudong963 @zhuqi-lucas I think we are already aligned here, this is what
we do at the moment, and it's effective! We pass a static file list (from log
replay) into our Parquet `DataSourceExec`, together with a
`DynamicFilterPhysicalExpr` on the file id column.
The problem is we have to do log replay first so that we can pass this
static list, to then later prune it inside DeltaScanExec. This means we have to
hold a materialized snapshot to run a second replay once `SourceStatsExec`
sends the predicates that can be used to prune the materialized file view.
Regarding the timing: the target scan has not started yet, because the hash
join reads the whole source first to build its hash table. `DeltaScanExec`
executes its DataSourceExec only at its first poll, and after it has set the
filter and marked it complete. So the filter is final before the first target
file is opened.
Maybe the simplified flow below helps to follow how it currently works in
delta-rs@main:
```
PLANNING
┌───────────────────────┐
│ DeltaScan │
│ + runtime_file_filter │
└───────────┬───────────┘
│ TableProvider::scan
┌───────────▼───────────┐
│ scan::execution_plan ├──► RuntimeScanFilePruner
└───────────┬───────────┘ filter: DynamicFilterPhysicalExpr, true for now
┌───────────▼───────────┐
│ get_data_scan_plan ├──► DeltaScanExec { file_pruner }
└───────────────────────┘ └─ DataSourceExec { f0..f3, predicate:
filter }
EXECUTION (source = build side, target = probe side)
┌──────────────┐
│ HashJoinExec │ reads the whole build side,
└──▲────────▲──┘ then polls the probe side
┌─────────────┘ └──────────────┐
┌────────┴────────┐ ┌────────┴────────┐
│ SourceStatsExec │─ (1) predicates ─►│ DeltaScanExec │ (2) first poll:
the pruner
└────────▲────────┘ at source end └────────▲────────┘ sets filter
to keep f0
│ │ (3) then
executes its input
┌────────┴────────┐ ┌────────┴────────┐
│ source │ │ DataSourceExec │ (4) filter skips
f1 f2 f3
└─────────────────┘ └─────────────────┘ before it
reads them
```
My question was more about option 2: streaming the file list itself into the
scan, instead of using a fixed list provided at plan time, so that we can make
the snapshot completely lazy and part of the whole plan:
```
SourceStatsExec ┌───────────────┐ file f0 ┌────────────────┐ batches
┌───────────────┐
─ ─ ─ ─ ─ ─ ─ ─ ─►│ LogReplayExec ├────────►│ DataSourceExec ├────────►│
DeltaScanExec │
└───────────────┘ └────────────────┘
└───────────────┘
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
