goutamadwant opened a new pull request, #24353:
URL: https://github.com/apache/datafusion/pull/24353
## Which issue does this PR close?
- Part of #9016.
## Rationale for this change
While prototyping the minimal push-based streaming example proposed in
#9016, I found that `FilterExec` could buffer small filtered results
indefinitely when its input was unbounded. Because an unbounded source may
never finish, consumers could not observe those results while continuing to
push input batches.
This PR extracts the smallest prerequisite fix so the streaming example and
user-guide documentation can follow as separate, focused PRs.
This is complementary to #23856: that PR changes how filtered batches are
supplied to the coalescer, while this PR controls when buffered rows become
observable for an unbounded input.
## What changes are included in this PR?
- Add a non-final `LimitedBatchCoalescer` flush operation that emits
buffered rows while still allowing subsequent input.
- Preserve idempotent coalescer finalization.
- Flush `FilterExec` output after each input batch when its input is
unbounded.
- Preserve the existing coalescing behavior for bounded inputs.
- Preserve existing fetch-limit behavior.
- Add regression coverage for:
- emitting results while an unbounded input remains open;
- multiple input pushes;
- a fully filtered batch followed by a matching batch;
- bounded-input coalescing;
- flushing without preventing subsequent input.
Planned follow-up PRs:
1. Add the simplest possible push-based streaming example under
`datafusion-examples`, showing how to feed batches into a running query and
consume results incrementally.
2. Add a “Using DataFusion for streaming” section to the library user guide,
based on that example and documenting relevant boundedness and execution
considerations.
## Are these changes tested?
Yes.
- Verified that the new unbounded-input regression test fails before the
implementation and passes afterward.
- `cargo test -p datafusion-physical-plan`
- `cargo test -p datafusion`
- `cargo test -p datafusion-cli`
- `cargo test --profile=ci --test sqllogictests`
- Contributor-guide extended workspace test suite with
`avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption`
- `cargo clippy --all-targets --all-features -- -D warnings`
- `cargo fmt --all -- --check`
- Rust documentation, license-header, and typo checks
## Are there any user-facing changes?
Yes. `FilterExec` now emits filtered results incrementally for unbounded
inputs instead of waiting for the target batch size or input completion.
Bounded-input coalescing remains unchanged. There are no public API or
breaking changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
