AnuragRaut08 opened a new pull request, #2390: URL: https://github.com/apache/datafusion-ballista/pull/2390
# Which issue does this PR close? Closes #2377. # Rationale for this change `dev/update_datafusion_proto.py` was silently passing CI without actually checking `datafusion.proto`. In DataFusion 55 the logical-plan proto moved from the `datafusion-proto` crate into a new `datafusion-proto-models` crate. The script still targeted `datafusion-proto`, hit a broad "crate doesn't ship this file" escape hatch, printed a `note:` to stderr, and **exited 0** — so the CI job *"Check vendored DataFusion proto is in sync"* was permanently green while validating nothing for `datafusion.proto`. Reproducer (before this fix): ```text $ python3 dev/update_datafusion_proto.py --check note: [email protected] does not ship proto/datafusion.proto; leaving vendored datafusion.proto unchanged $ echo $? 0 ``` # What changes are included in this PR? **`dev/update_datafusion_proto.py`** * `FILES` values are now `([ordered crate list], filename)`. For `datafusion.proto` the list is `["datafusion-proto-models", "datafusion-proto"]` — the script tries each in order and uses the first one that actually ships the file. This keeps the script working on both v55+ (`datafusion-proto-models`) and v54 (`datafusion-proto`) without a separate code path. * `_ALL_CRATES` is derived automatically from `FILES`, so `crate_source_dirs()` always looks up every candidate without a separate maintenance step. * The silent-skip path (`if not src.exists(): continue`) is **removed**. If none of the candidate crates ships the expected file it is now a hard `sys.exit` with a diagnostic listing which crates were tried, so a future rename fails loudly instead of silently passing CI. **`ballista/core/proto/datafusion.proto`** Re-vendored from `[email protected]` (211 insertions, 39 deletions vs the previously stale copy). Key additions: * `reserved 8/9` annotations on `ListingTableScanNode` (was `collect_stat` / `target_partitions`) * `RangeRepartition` variant on `RepartitionNode` + new `RangeSplitPoint` message * `repeated string locations = 16` on `CreateExternalTableNode` * `AnalyzeNode` gains `analyze_level`, `analyze_categories`, `format` fields * `ExplainNode` gains `show_statistics` field * `MERGE_INTO` added to `DmlNode.Type` enum # Are there any user-facing changes? No. The vendored `.proto` files are build-time stubs only — `ballista/core/build.rs` maps their packages to the real DataFusion crates via `extern_path`, so no Rust code is generated from them in Ballista. This change only makes the CI guard actually guard. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
