andygrove opened a new pull request, #5009: URL: https://github.com/apache/datafusion-comet/pull/5009
## Which issue does this PR close? N/A. This adds a contributor tooling skill alongside the existing `.claude/skills/` for Comet expression work (`implement-comet-expression`, `wire-datafusion-function`, `audit-comet-expression`, `optimize-comet-expression`). ## Rationale for this change Comet maintains a growing set of hand-rolled native Spark expressions in `native/spark-expr/`. Many of these have Spark-compatible semantics that belong in the upstream `datafusion-spark` crate, where they can be maintained once and consumed by Comet (and other DataFusion-based engines) rather than carried as a native port. Today there is no documented workflow for moving an expression in that direction. This skill fills that gap, mirroring the existing `implement-comet-expression` (which pulls upstream functions in) and complementing `wire-datafusion-function` (which consumes a released upstream function). ## What changes are included in this PR? Adds `.claude/skills/upstream-comet-expression/SKILL.md`, an 8-step workflow for upstreaming a Comet-native expression into `apache/datafusion`: - Confirm the function is genuinely missing and not already in flight (grep, tracking epic #15914, open PRs). - Study Comet's native impl for semantics only, then adopt current upstream idioms rather than copying the older port (the `ScalarUDFImpl` trait shape drifts; prefer `simplify()`-to-casts over hand-rolled kernels where possible; pick the signature style by input type; use direct `datafusion_common`/`datafusion_expr` paths). - Implement and register the function (four touch points in the category `mod.rs`) and add sqllogictest coverage per the datafusion spark test guide. - Pre-PR checks, then a human-confirmation gate before filing the upstream PR. - The Comet re-wire (via `wire-datafusion-function` once released) is documented as an explicitly deferred step. The workflow was validated end to end by upstreaming `date_from_unix_date` (apache/datafusion#23852), which the skill references as a worked example. ## How are these changes tested? This is documentation (a contributor skill), so there is no automated test. The workflow it describes was exercised in full against the live `apache/datafusion` checkout while producing apache/datafusion#23852, and every command, file path, and registration step in the skill was verified against that repo. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
