QinXi-ai opened a new pull request, #25765: URL: https://github.com/apache/datafusion/pull/25765
## Which issue does this PR close? Part of #25395. This is the wire-compatible generator-hook step; it does not close the multi-generator serialization gap. ## Rationale for this change Serializing a built-in `LazyMemoryExec` currently requires `datafusion-proto` to recognize each concrete generator type. A generator cannot supply its own encoding even when it can produce the existing generate-series representation. This change lets the generator provide that representation and keeps decoding alongside the concrete generator types. Existing single-generator plans retain their protobuf bytes and execution behavior. ## What changes are included in this PR? - Add an optional `LazyBatchGenerator::try_to_proto` hook and implement `LazyMemoryExec::try_to_proto` using it. - Implement the hook for empty, integer and timestamp/date series, behind a new `datafusion-functions-table/proto` feature. Generic series values can supply their argument encoding; the default remains extension-codec fallback. - Move built-in decoding into `datafusion-functions-table` and remove the generator downcasts from the central serializer. - Preserve extension fallback for unsupported, empty and multiple-generator plans, and propagate real encoding errors. - Fix two pre-existing Windows Clippy blockers with the repository's Rust 1.98.1 toolchain: remove a redundant semicolon in conservative floating-point rounding, and document the intentional async API of the Windows-only stub for the FIFO example. - Mark the literal `file:///` example in the 14.0.0 changelog as code so the link checker does not treat it as a local filesystem link on Windows. The fixed built-in decoder still requires the `datafusion-functions-table` dependency. Removing it would require a separate injection/registration design. The existing wire message has a single argument payload, so multiple generators also need a separate protocol change. ## What is the testing strategy for this PR? The new integration tests construct legacy wire messages independently of the hooks and check exact re-encoding, output values, batch boundaries and repeatable execution. They cover `generate_series` and `range`, empty/integer/timestamp/date payloads, a generator absent from the old downcast chain, extension fallback, malformed input and batch-size overflow. - `cargo check --locked -p datafusion-functions-table --no-default-features`: passed. - `cargo test --locked -p datafusion-proto -p datafusion-functions-table --all-features`: 301 tests passed, including 276 integration tests and 4 doctests; no failures. - `cargo fmt --all` and `git diff --check`: passed. - `cargo clippy --locked --all-targets --all-features -- -D warnings`: passed. - All checks from `dev/rust_lint.sh` passed individually, including its Clippy feature combination, formatting, licenses, spelling, generated docs, links, security audit, dependency checks, Rust docs and Sphinx docs. On native Windows, the Sphinx build used `uv run --package datafusion-docs bash ./docs/build.sh` to launch the shell script explicitly. - The physical-plan crate passed all 2,390 tests in the extended-workspace configuration. The extended Windows workspace run is not fully green. Four datasource URL/cache tests and four floating-point statistics/interval tests failed; all eight reproduce on an unmodified source export of base `6a792c6`, with the same assertions. A repartition spill timeout test failed in the concurrent run but passed its exact isolated rerun (12 internal attempts, 104.23 seconds). Remaining long-running integration suites are still in progress. This PR stays draft pending completion of platform/CI validation. ## Are there any user-facing changes? New defaulted Rust trait methods and an opt-in `proto` feature. Existing downstream generators need no changes. Generate-series helper methods are removed from `PhysicalPlanNodeExt`; direct callers of its decoder can use `datafusion_functions_table::generate_series::proto::try_from_proto`. The built-in single-generator wire format is unchanged; multi-generator serialization remains outside this first step. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
