dwsmith1983 opened a new pull request, #5365: URL: https://github.com/apache/datafusion-comet/pull/5365
## Which issue does this PR close? Part of #174 (Explore integration with Delta Lake). It does not close #174, that issue also tracks writes, CDF, and broader integration; this PR delivers the native read path. **Supersedes** two earlier efforts, and deliberately builds on both (both given co-authored by since ideas were learned and borrowed): - #4366 (@schenksj) — the delta-kernel-rs-based contrib. This PR keeps its contrib module shape, build gating, decline-gate philosophy, and test scenario catalog, but replaces the kernel data-read path: kernel's parquet reader has no page-index pruning, and its planning duplicates work delta-spark has already done by the time the DSv1 scan reaches Comet. - #4669 (@adityavaish) — the minimal plain-table native scan. This PR takes the same core insight (reuse Comet's existing native parquet scan; no kernel needed) and extends it to deletion vectors, column mapping, DPP, and the full Spark matrix. ## Rationale for this change Comet currently falls back to Spark's reader for all Delta tables (`isFileFormatSupported` requires exact `ParquetFileFormat`, and `DeltaParquetFileFormat` is a subclass). That forfeits native execution and all of Comet's parquet pruning on one of the most common table formats. Key observation: delta-spark has already done log replay, snapshot resolution, time travel, and partition pruning by the time `CometScanRule` sees the `FileSourceScanExec`. So no Delta planning is needed on the native side at all, the scan can route through the exact same DataFusion `ParquetSource` path as `CometNativeScanExec`, inheriting row-group stats pruning, page-index pruning (#5142), and filter pushdown (#4722) for free. The only genuinely Delta-specific native code is deletion-vector decoding: DV bitmaps are decoded into per-file `ParquetAccessPlan`s, which DataFusion *intersects* with page-index pruning, so deleted rows are skipped in-scan and DV skips compose with page skips. Local benchmark (20M rows, selective predicate): 1.44x faster than stock Spark 9.4% of bytes read; DV tables at time parity with in-scan DV application. ## What changes are included in this PR? - **`contrib/delta/`** new Maven module behind a `-Pdelta` profile: scan rule, decline gates, serde, `CometDeltaNativeScanExec` (split-mode partition serialization, DPP via derived scan helper), ServiceLoader registrations, differential test suites, Delta own-suite regression harness, benchmark script. - **Core (Delta-free, generic)** new `CometScanRuleExtension` SPI + ServiceLoader hook at the top of `transformV1Scan`; `CometNativeScan.convert body extracted into reusable `buildNativeScanCommon`. - **Native** `delta` cargo feature: `DeltaScan` proto + planner arm delegati the shared parquet scan builder; `delta_dv.rs` for DV blob unframing (CRC verified), roaring decode (portable + native magic), and access-plan construction. - **Supported** plain/partitioned tables, deletion vectors (inline + on-disk), column mapping name mode, time travel, checkpoints, OPTIMIZE'd tables, schema evolution + defaults, DPP, INT96, special-character paths. - **Declines (fall back to Spark, tagged for EXPLAIN)** CDC reads, column mapping id mode, row-index-consuming plans, unknown reader features, generated columns, encryption, `input_file_name()`. - **Spark matrix**: 3.5 / 4.0 / 4.1 (Delta 3.3.2 / 4.0.1 / 4.3.1 — Delta 4.1.0/4.2.0 are binary-incompatible with Spark 4.1.3). Spark 3.4 declines (Delta 2.4 needs shims); spark-4.2 profile is dormant until Delta ships support. ## How are these changes tested? - 36-test differential suite (comet-on, comet-off) covering the DV, column mapping, DPP, schema evolution, and decline-gate matrices green on Spark 3.5/4.0/4.1 cells. Pruning is asserted as hard metrics checks (`page_index_rows_pruned > 0`, `row_groups_pruned_statistics > 0`), not benchmark notes. - Delta's own test suites run with Comet injected (`dev/run-delta-regression.s DeletionVectorsSuite 29/29, TimeTravel/ColumnMapping/DeleteSQL/UpdateSQL 197/197, MergeIntoSQLSuite 664/665 (the one failure is a scan-telemetry count assertio plan-shape artifact; data assertions pass). - Rust unit tests for DV unframing/decode/access-plan edge cases and the feature-off error path. - A DML repro suite proving DELETE writes DVs (not rewrites) under the claimed scan, in both `useMetadataRowIndex` modes. --- Co-authored-by: Scott Schenkein <[email protected]> Co-authored-by: Aditya Vaish <[email protected]> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
