viirya opened a new issue, #26034:
URL: https://github.com/apache/datafusion/issues/26034
### Describe the bug
Substrait roundtripping can change query results by dropping
`TableScan::skip` and `TableScan::fetch`.
The producer converts a table scan into a `ReadRel`, which cannot represent
either field. `from_limit` compensates for the scan offset only when its
immediate input is a `TableScan`. A standalone scan, or a scan separated from
its parent limit by a projection, loses the pushed-down offset.
### To Reproduce
Register a single CSV file as `t`, with one integer column `x` containing
`0, 1, 2, 3, 4, 5`. Build and roundtrip a plan using the public APIs:
```rust
let source = provider_as_source(ctx.table_provider("t").await?);
let scan = TableScanBuilder::new("t", source)
.with_skip(Some(2))
.with_fetch(Some(2))
.build()?;
let plan = LogicalPlan::TableScan(scan);
let proto = to_substrait_plan(&plan, &ctx.state())?;
let restored = from_substrait_plan(&ctx.state(), &proto).await?;
let before = DataFrame::new(ctx.state(), plan).collect().await?;
let after = DataFrame::new(ctx.state(), restored).collect().await?;
```
The original plan returns `[2, 3]`; the restored standalone scan returns all
six rows. With `Limit(0, 2) -> Projection(x) -> TableScan(skip=2, fetch=2)`,
the restored plan returns `[0, 1]`. The direct `Limit(0, 2) -> TableScan` case
happens to retain `[2, 3]` because of the parent-limit compensation.
An independently verified offset-only reproduction on
`60477917bd9eadd1a299185db6a8a08ccae7a4c7` returns `[2, 3, 4, 5]` before
roundtripping and `[0, 1, 2, 3, 4, 5]` afterward.
### Expected behavior
Roundtripping preserves the scan's offset and row limit regardless of
whether a parent limit exists or a projection separates the nodes.
### Additional context
A fix can encode the scan's skip/fetch as a `FetchRel` wrapping its
`ReadRel`, and remove the parent-limit offset compensation to avoid applying
the scan offset twice. Regression coverage should compare executed results,
including offset-only scans, offset plus fetch, projections, nested limits, and
scans without an offset.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
