Dandandan opened a new issue, #23304: URL: https://github.com/apache/datafusion/issues/23304
### Is your feature request related to a problem or challenge? Follow-up to #23303 (prototype in #23282). The in-process distributed execution example validates the stage/serialize/isolate contract with a handful of handwritten integration tests (joins, sorts, multi-stage aggregates). That is enough to demonstrate the model, but the coverage is only as broad as the queries someone thought to write. DataFusion already has a large, curated corpus of queries with expected results: the sqllogictest suite (`datafusion/sqllogictest`). None of it currently exercises the distributed execution path, so an operator change that breaks isolated/serialized execution for some query shape (a window function, a particular join order, a subquery pattern, …) can pass both the sqllogictest suite and the distributed example's handwritten tests. ### Describe the solution you'd like Run a subset of the sqllogictests through the distributed execution example, so each query is planned normally, split into stages, serialized via `datafusion-proto`, executed as isolated tasks over the in-memory exchange, and its results compared against the expected output. Sketch: - Add a runner mode (or a separate binary/test in the example crate) that plugs the distributed harness in as the query execution path for the sqllogictest engine, reusing the existing `.slt` files and expected results. - Start with an explicit **subset**: files that are deterministic, don't depend on session-local state or DDL side effects that don't survive stage serialization, and don't rely on partition-count-sensitive output. A skip/allow list keeps it maintainable while coverage grows. - Queries that legitimately can't run distributed (e.g. `EXPLAIN`-output tests, settings-dependent tests) are skipped explicitly rather than silently. - Once stable, run the subset in CI (an optional/nightly job is fine initially) so the class of regression described in #23303 is caught before it reaches Ballista / datafusion-distributed. An added benefit beyond regression coverage: the differential comparison (distributed vs. expected results, and by extension vs. single-process execution) turns every sqllogictest into a correctness test for plan serialization in `datafusion-proto` — historically a common source of downstream breakage on its own. ### Describe alternatives you've considered - **Keep only handwritten integration tests** in the example crate: coverage stays narrow and grows only when someone remembers to add a case. - **Fuzz-generated query differential testing**: complementary and valuable, but harder to maintain and debug than a curated corpus with expected results that already exists in-tree. - **Run the full sqllogictest suite distributed from day one**: many tests are not meaningful or not stable under distributed execution; starting with a curated subset keeps signal high and flakiness low. ### Additional context - Depends on #23303; working prototype of the execution harness in #23282. - Prior art: [datafusion-distributed](https://github.com/datafusion-contrib/datafusion-distributed) and Ballista both maintain their own differential testing against DataFusion; running the in-tree sqllogictests through an in-tree harness shortens that feedback loop to a single repo. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
