andygrove opened a new issue, #6088: URL: https://github.com/apache/datafusion-comet/issues/6088
Comet's [versioning policy](https://datafusion.apache.org/comet/about/versioning_policy.html) currently puts every `spark.comet.*` key inside the compatibility surface: > **Configuration keys under `spark.comet.*`**: their names, types, accepted values, default values, and semantics. That is too broad. Configs declared with `.category(CATEGORY_TESTING)` are not a deployment surface — they exist so Comet's own suites, and contributors chasing a bug, can reach a state the rest of the code is not built to support. `spark.comet.scan.enabled` disables native scans to isolate a planner problem. `spark.comet.explain.fallback.strict.enabled` makes a declined operator throw instead of quietly reporting itself. `spark.comet.testing.aggregate.partialMode.enabled` runs a partial aggregate without its final counterpart. The on-heap family (`spark.comet.exec.onHeap.enabled`, `spark.comet.exec.onHeap.memoryPool`, `spark.comet.memoryOverhead`, `spark.comet.shuffle.jvm.memoryFactor`) exists because Spark's own SQL suite needs on-heap mode; the tuning guide documents off-heap only. Taken literally, the policy as written means renaming one of these needs a `withAlternative` alias, changing a default needs a `spark.comet.legacy.*` escape hatch and an upgrade guide entry, and removing one needs a deprecation cycle and a major release. None of that is worth doing for a knob whose only callers are in `spark/src/test`, and it would pin the internals the knob exists to expose — the same internals the policy otherwise leaves explicitly free to change. Proposal: state that the `testing` category is exempt. Names, types, accepted values, defaults, and semantics of those keys may change in any release, including a patch release, with no alias, no legacy config, no deprecation cycle, and no upgrade guide entry. The corollary is a rule for contributors: a `testing` key must not be the only way to reach a behavior production users need, so choosing a category becomes a policy decision rather than a routing detail. Documentation to update: - `docs/source/about/versioning_policy.md` — the carve-out itself, plus the sections that currently imply the opposite (patch releases, behavior changes, renames). - `docs/source/contributor-guide/config_conventions.md` — what the choice of category commits the project to. - `docs/source/user-guide/latest/configs.md` — a note on the Development & Testing Settings table so users see it where they would set one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
