andygrove opened a new issue, #23887: URL: https://github.com/apache/datafusion/issues/23887
### Is your feature request related to a problem or challenge? The sqllogictest files under `datafusion/sqllogictest/test_files/spark/` hard-code their expected results, so each file encodes the behavior of exactly one Spark version. There is currently no way to express that a function's behavior changed between Spark versions. This matters because Spark's built-in expressions do change across releases. A few of the axes that move: ANSI mode became the default in Spark 4.0, collation support was added to many string functions in 4.0, and individual expressions have picked up new accepted input types or altered edge-case handling between 3.5 and 4.2. Today the only options when we discover such a divergence are: 1. Assert one version's behavior and silently be wrong about the others. 2. Leave the case untested entirely. Neither leaves a durable record, and neither lets CI catch a regression on the version we did not pick. Concretely, a test author who finds that `f(x)` returns `NULL` on Spark 3.5 and throws on Spark 4.2 has nowhere to put that knowledge except a comment. ### Describe the solution you'd like A way to scope a `.slt` file, or an individual query within one, to a Spark version or version range. There is prior art worth borrowing from. Apache DataFusion Comet solves the analogous problem for its SQL file tests with a file-level directive: ```sql -- MinSparkVersion: 3.5 ``` which skips the whole file when running against an older Spark. See `docs/source/contributor-guide/sql-file-tests.md` in apache/datafusion-comet. Two shapes seem plausible here, and I do not have a strong preference yet: **Option A: file-level directive.** Mirror Comet directly, e.g. a `# MinSparkVersion: 4.0` comment directive honored by the sqllogictest runner. Simple, but coarse: a single divergent query forces the whole file to be scoped, or the divergent cases to be split into a separate file. **Option B: per-query labels.** sqllogictest already supports `skipif <label>` / `onlyif <label>`. Exposing the target Spark version as a label would allow: ```sql onlyif spark-4.2 query I SELECT ... ---- 42 onlyif spark-3.5 query I SELECT ... ---- NULL ``` This is finer-grained and keeps both behaviors visible side by side at the test site, which is where a reader wants them. It needs a decision on what the label namespace looks like and how a range (rather than an exact version) is expressed. Either option needs a notion of "which Spark version is this test suite currently targeting", since there is no Spark process in the loop on the DataFusion side. That target is a declared baseline rather than something detected at runtime. ### Describe alternatives you've considered Keeping the status quo and recording divergences only in comments. This is what we do implicitly today. It loses the ability to test the non-baseline behavior at all, and the comments drift out of date with no signal. ### Additional context This came up while designing an agent skill for auditing `datafusion-spark` expression implementations against the Spark source. The audit compares a function's behavior across several Spark releases, and cross-version divergences are a routine finding. With no mechanism to express them, the audit can only leave a comment and link here. The related epic is #15914. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
