alamb commented on issue #21301: URL: https://github.com/apache/datafusion/issues/21301#issuecomment-5639260115
> @sdf-jkl also posted to the mailing list here: https://lists.apache.org/thread/bg71srlcd84y4gv42cqzdo3j6mj3t93f Relevant content : ---------- Forwarded message --------- From: Kosta Tarasov <[email protected]> Date: Thu, Sep 10, 2026 at 6:42 PM Subject: [DISCUSS] Moving Variant support forward in DataFusion To: <[email protected]> Hi all, I'd like to bring the discussion around PR #22908[0] to the mailing list and get broader input on how we should move Variant support forward in DataFusion. PR #22908 proposes bringing datafusion-variant into the DataFusion repository as an extension crate, following a model similar to datafusion-functions-spark. More recently, Andrew suggested another option: keeping the crate external and making it easier to integrate with DataFusion through cookbook examples and documentation.[1] Either way, I think the goal is the same: make Variant straightforward to use with DataFusion. I'd appreciate input on a few areas: - Integration and maintenance: Should we bring the crate into the main DataFusion repository, or focus on making the external crate easier to integrate? If we bring it in, are there concerns about the additional maintenance commitment? - SQL API and semantics: Spark compatibility was the original target for datafusion-variant, with some additional functions influenced by datafusion-functions-json[2] and DuckDB. Should Spark semantics and naming be the compatibility baseline? How should we handle useful functions that do not have Spark equivalents? I also think it would be useful to distinguish the user-facing SQL API from functions that primarily exist to enable optimizer rewrites.[3] - Comet coordination: Comet now has an active native Variant support epic[4]. There appears to be significant overlap around Variant semantics, expression kernels, and representation boundaries. I’d like to coordinate where possible rather than build parallel implementations. Input from Comet contributors would be especially useful here. If we decide to bring the crate into the DataFusion repository, I think the incremental approach discussed on the PR makes sense: start with the crate scaffolding and a basic JSON-to-Variant function, then add and review the remaining functionality in smaller PRs. We should also make sure we document both how to integrate Variant support and how users are expected to query Variant values. I’m happy to help move this forward whichever integration approach we choose. Feedback on the overall direction, use cases we should prioritize, and offers to help with implementation or review would be very welcome. Thanks, Kosta [0] https://github.com/apache/datafusion/pull/22908 [1] https://github.com/apache/datafusion/pull/22908#issuecomment-5603803800 [2] https://github.com/datafusion-contrib/datafusion-functions-json [3] https://github.com/apache/datafusion/pull/22908#issuecomment-5469828372 [4] https://github.com/apache/datafusion-comet/issues/5438 --------------------------------------------------------------------- To unsubscribe, e-mail: [[email protected]](mailto:[email protected]) For additional commands, e-mail: [[email protected]](mailto:[email protected]) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
