andygrove opened a new issue, #6892: URL: https://github.com/apache/datafusion-comet/issues/6892
Triage pass over the open `requires-triage` queue, per the project [Bug Triage Guide](https://github.com/apache/datafusion-comet/blob/main/docs/source/contributor-guide/bug_triage.md). - Date: 2026-10-12 - Total issues processed: 89 (84 triaged, 5 skipped, 0 failed) - Type counts: 49 bugs, 35 enhancements - Priority counts applied: `priority:critical` 18, `priority:high` 0, `priority:medium` 25, `priority:low` 6 - Guide: [docs/source/contributor-guide/bug_triage.md](https://github.com/apache/datafusion-comet/blob/main/docs/source/contributor-guide/bug_triage.md) Labels have already been applied. A reviewer should spot-check the calls below and close this issue when satisfied. Corrections should be made directly on the affected issue. Notes on this pass: - **Reviewer corrections since the last pass.** No type or priority label from the 2026-10-05 pass has changed since. `correctness` was added to five of its critical bugs (#6476, #6477, #6519, #6570 and #6573). The label isn't in the guide, so this pass doesn't add it. The critical bugs below that don't carry it are #6839, #6800, #6787, #6779, #6768, #6762, #6759, #6758, #6747, #6746, #6742, #6718 and #6711. - **Two type reclassifications.** #6832 arrived as `bug` and is now `enhancement`: its outer joins run on Spark, as they would without the opt-in feature, and the issue says correctness is unaffected. #6771 arrived as `enhancement` and is now a `priority:medium` bug, because the native fanout write fails its task where iceberg-java's writer succeeds. Either can be flipped back with one edit. The 29 issues that arrived without a type label were classified from their bodies. - **One author priority was changed.** #6816 went from `priority:low` to `priority:medium` (see the escalations). The author's `priority:low` was confirmed on #6890. Three enhancements arrived with an author's priority: #6886 (`priority:medium`), #6849 and #6717 (`priority:low`). As in earlier passes, these were left in place, and the reviewer may want to remove them. - **Spark SQL test findings from draft #6780.** #6795 to #6803 come from turning on `spark.comet.exec.localTableScan.enabled`, which is off by default. The three whose failures are only in the tests (#6795, #6796 and #6803) are low. The others are labeled by the product behavior they report: #6799 and #6800 critical, and #6797, #6798, #6801 and #6802 medium. - **Likely duplicate.** #6718 and #6779 are very likely one defect. Both are from unikdahal, both drop rows from a broadcast join whose build side is a `DISTINCT` over string keys, and #6779 traces it to sliced string vectors in `Utils.serializeBatches` and `coalesceBroadcastBatches`. Both are labeled, and the reviewer may want to close #6718 in favor of #6779. - The guide's area table still lacks `area:joins`, `area:Iceberg`, `area:udf` and `area:memory`, so this pass added none of them. Issues that would have taken one and don't already carry it: `area:joins` for #6718, #6779, #6787 and #6832, `area:Iceberg` for #6719 and #6737, `area:udf` for #6711 and #6811, and `area:memory` for #6880. - The guide lists `spark 4` as an area indicator, but the repository only has `spark 3.x`, `spark 4.0`, `spark 4.1` and `spark 4.2`, so nothing was applied to #6768 and #6769, which affect only Spark 4 releases. - `regression` is not in the guide, so this pass didn't add it. It fits the `IF` and `nullif` shapes of #6762, which come from the eager CASE path in #6350, merged after 1.1.0. - #6667 was closed while this pass was running and isn't included. #6891 was opened during the pass and is included. ## Bugs ### priority:critical - [EPIC] In-memory cache audit: correctness and performance follow-ups for InMemoryTableScanExec support ([#6834](https://github.com/apache/datafusion-comet/issues/6834)) - Area labels: `area:scan`; also carries `performance`, `EPIC`, `correctness` - Rationale: Its first open item returns wrong results with default settings on Spark 3.5+, where the cache is on by default: a sorted cache under a native hash join loses its order (the root cause of #6787, fix in #6835). An EPIC of bugs takes the priority of its worst open item, as #6335 and #6385 did, so step 1 (see escalations). - Native string to float/double and decimal casts accept strings Spark rejects ([#6839](https://github.com/apache/datafusion-comet/issues/6839)) - Area labels: `area:expressions` - Rationale: `CometCast` reports string to `FLOAT`, `DOUBLE` and `DECIMAL` as `Compatible`, so with default configs `'NaNF'`, `'-nan'` and `'1e-2147483648'` cast to NaN, infinity or `0.00` where Spark returns NULL, or raises `CAST_INVALID_INPUT` under ANSI. Incorrect cast output is the guide's own step 1 example. Fix in #6826. - Native Iceberg scan returns wrong values for a column renamed after a VERSION AS OF snapshot ([#6822](https://github.com/apache/datafusion-comet/issues/6822)) - Area labels: `area:scan`; also carries `correctness`, `area:Iceberg` - Rationale: A `VERSION AS OF` read of a snapshot from before a rename returns NULL, or the other column's values after a swap, from the native Iceberg scan, which is on by default. Silent wrong results at step 1, on `branch-1.1` too. - from_csv: wrong results in permissive modes and failure with a variant schema ([#6800](https://github.com/apache/datafusion-comet/issues/6800)) - Area labels: `area:expressions`, `spark sql tests`; also carries `expression` - Rationale: The report says results differ from Spark in permissive modes, which is step 1. It was seen only with `localTableScan` on, but `from_csv` goes through the codegen dispatcher, which is on by default, so a native projection over another source may reach it too (see escalations). - percentile of -0.0 returns 0.0 ([#6799](https://github.com/apache/datafusion-comet/issues/6799)) - Area labels: `area:aggregation`, `spark sql tests`; also carries `correctness` - Rationale: `CometPercentile` reports `Compatible` for numeric input, so the native `percentile` that returns `0.0` where Spark returns `-0.0` runs by default. Step 1, the same class as #6519 and #6549 under #6385. - Native Iceberg scan returns nested values that Spark reads as NULL in files without field ids or a name mapping ([#6794](https://github.com/apache/datafusion-comet/issues/6794)) - Area labels: `area:scan`; also carries `correctness`, `area:Iceberg` - Rationale: For data files with no field ids and no name mapping, the native Iceberg scan returns the stored nested values where Spark returns NULL, so filters and counts differ (`count(*) WHERE s.a IS NULL` is 50 in Spark and 0 in Comet). Step 1 on the default native scan (see escalations). - Native Iceberg scan returns NULL for nested fields of migrated tables that use a name mapping ([#6790](https://github.com/apache/datafusion-comet/issues/6790)) - Area labels: `area:scan`; also carries `correctness`, `area:Iceberg` - Rationale: Tables migrated with `migrate`, `snapshot` or `add_files` get a name mapping, and the native Iceberg scan, on by default, returns NULL for their nested fields whenever the query also reads a top-level primitive. Silent wrong results at step 1, and 1.1.0 has it too. - Native hash joins lose the streamed side's order when it arrives sorted from outside the native plan ([#6787](https://github.com/apache/datafusion-comet/issues/6787)) - Area labels: none - Rationale: Spark drops a sort above the join as redundant, and the native join appends the unmatched probe rows after the matched ones, so `sortWithinPartitions` returns unsorted partitions with no error. #6834 reproduces it with default settings through the native cache scan, and #6770, the same class, is critical. Fix in #6835. - Broadcast hash join produces incorrect results with sliced Arrow string/binary batches ([#6779](https://github.com/apache/datafusion-comet/issues/6779)) - Area labels: `area:ffi` - Rationale: A broadcast build side made of sliced string or binary vectors is serialized with its unused buffer prefix, so join keys are corrupted and matching rows are dropped with no error. That matches the guide's step 1 example of FFI data corruption from arrays with a non-zero offset. - arrays_overlap keeps -0.0 and 0.0 apart on Spark 4.0.5, 4.1.4 and 4.2.1, which merge them (SPARK-59602) ([#6768](https://github.com/apache/datafusion-comet/issues/6768)) - Area labels: `area:expressions` - Rationale: On those Spark releases `arrays_overlap` returns `false` where Spark returns `true` for signed zeros, with default configs. Step 1, under #6385 (see escalations). - IN treats a column-dependent CASE WHEN / IF candidate as the constant NULL ([#6762](https://github.com/apache/datafusion-comet/issues/6762)) - Area labels: `area:expressions` - Rationale: `IN` returns NULL for every row, with no error, when a candidate is an `IF`, `CASE WHEN` or `nullif` that reads a column. The reproduction runs over `range`, but the constant check is in the native `IN` itself, so a native projection over a Parquet scan takes the same path by default. Step 1. Fix in #6763. - least orders a null inside a struct or array last, where Spark orders it first ([#6759](https://github.com/apache/datafusion-comet/issues/6759)) - Area labels: `area:expressions` - Rationale: `least` over structs or arrays returns a different value from Spark when a nested value is null, because the DataFusion comparator it uses puts nulls last. The plan is native and raises nothing, so step 1. Fix in #6761. - greatest/least compare struct arguments by field name instead of position when names differ only in case ([#6758](https://github.com/apache/datafusion-comet/issues/6758)) - Area labels: `area:expressions` - Rationale: With the default `spark.sql.caseSensitive=false`, struct arguments whose field names differ only in case are matched by name instead of by position, so `greatest` and `least` return a different struct from Spark. Step 1. Fix in #6760. - Native CSV scan reads every partition through the first partition's object store ([#6747](https://github.com/apache/datafusion-comet/issues/6747)) - Area labels: `area:scan` - Rationale: A native CSV scan over two buckets returns the first bucket's rows twice, with no error, when the keys match. That is step 1 on the opt-in native CSV scan, as #6573 was on its opt-in path (see escalations). Fix in #6753. - Native Parquet scan reads files from another bucket when one partition spans buckets ([#6746](https://github.com/apache/datafusion-comet/issues/6746)) - Area labels: `area:scan` - Rationale: When Spark packs small files from two buckets into one partition, the default native Parquet scan fetches the second bucket's keys from the first and returns duplicated rows with no error. Step 1. Fix in #6753. - arrays_overlap returns wrong answers or panics with a constant array beside a column ([#6742](https://github.com/apache/datafusion-comet/issues/6742)) - Area labels: `area:expressions` - Rationale: With default configs, a constant first argument compares only row 0 and uses that answer for the whole batch, and a constant second argument panics. The wrong answers make it step 1. Fix in #6743. - Broadcast hash join on string keys drops rows when the build side is a large DISTINCT ([#6718](https://github.com/apache/datafusion-comet/issues/6718)) - Area labels: `area:ffi` - Rationale: About 0.012% of the matching rows go missing with no error, which is step 1. It is very likely the defect #6779 traces to sliced string vectors in the broadcast path (see the notes). - Identical nondeterministic expressions share one codegen dispatcher kernel and continue each other's state ([#6711](https://github.com/apache/datafusion-comet/issues/6711)) - Area labels: `area:expressions` - Rationale: Two identical dispatched subtrees share one kernel and one counter, so the second `monotonically_increasing_id()` continues the first one's sequence. The dispatcher is on by default, so this is silent wrong results at step 1, like #6570. Fix in #6712. ### priority:medium - Native PartialMerge over input sorted on a prefix of its keys fails to spill on DataFusion 55.2 ([#6880](https://github.com/apache/datafusion-comet/issues/6880)) - Area labels: `area:aggregation` - Rationale: Since #6867, this shape fails the task under memory pressure instead of returning the wrong count from #6857. The failure is visible and turning off the native aggregate avoids it, so step 3, like #6363. - Native date functions and date-to-string cast return NULL or fail for dates outside chrono's NaiveDate range ([#6843](https://github.com/apache/datafusion-comet/issues/6843)) - Area labels: `area:expressions` - Rationale: `year`, `month`, `trunc`, `next_day` and others return NULL, and `cast(d AS string)` and `last_day` fail, for dates beyond year 262142. Held at medium like #5208 and #6145, since real data doesn't reach those years (see escalations). - Native cast string to date returns NULL for years outside chrono's NaiveDate range ([#6842](https://github.com/apache/datafusion-comet/issues/6842)) - Area labels: `area:expressions` - Rationale: `CAST`, `TRY_CAST` and `to_date` return NULL for years beyond chrono's range, even under ANSI. The cast's support note documents the range, and it is held at medium like #5208 and #6145 (see escalations). Fix in #6844. - CometScanRule reads fs.comet.libhdfs.schemes from SQLConf, but native reads it from the Hadoop conf ([#6837](https://github.com/apache/datafusion-comet/issues/6837)) - Area labels: `area:scan` - Rationale: A list set with `SET spark.hadoop.fs.comet.libhdfs.schemes` makes Comet claim a scan that then fails at execution, and one set only in `core-site.xml` makes the scan fall back. A visible failure in an uncommon setup, with a workaround (set it at startup), so step 3. Fix in #6838. - Cast of a literal that fails under ANSI fails the query at planning time, even in a branch no row reaches ([#6825](https://github.com/apache/datafusion-comet/issues/6825)) - Area labels: `area:expressions` - Rationale: Under ANSI, the default on Spark 4, `CometCast` evaluates a literal cast while planning and fails a query that Spark completes because no row reaches that branch. A visible failure with workarounds, step 3, like #6613. Fix in #6826. - IN evaluates later candidates over the whole batch, so an ANSI error can fire for a row that already matched ([#6823](https://github.com/apache/datafusion-comet/issues/6823)) - Area labels: `area:expressions` - Rationale: Under ANSI, Comet raises for a row that an earlier `IN` candidate already matched, where Spark returns a result. A visible failure, step 3, like #6613, and one instance of #6006. - Native timezone conversions stop applying DST after 2099, while Spark keeps applying the zone's rules ([#6816](https://github.com/apache/datafusion-comet/issues/6816)) - Area labels: `area:expressions`; also carries `correctness` - Rationale: After about 2100, `hour`, casts to string or date, and `date_trunc` in a DST zone can be off by the DST offset, and the first two run natively by default. That is silent wrong results, documented in the compatibility guide. Raised from the author's `priority:low` to medium, where this pass holds the other far-from-present date issues (see escalations). - Non-AQE DPP subquery doesn't reuse the Comet broadcast when the dimension is a UNION ([#6815](https://github.com/apache/datafusion-comet/issues/6815)) - Area labels: none; also carries `performance`, `area:joins` - Rationale: Results are correct, but with AQE off the DPP subquery computes and broadcasts a `UNION ALL` dimension a second time instead of reusing the join's broadcast. Lost exchange reuse is step 3, as it was for #6645. - ShuffleScan metrics are dropped under AQE: shuffle read and decode time missing on the direct-read path ([#6804](https://github.com/apache/datafusion-comet/issues/6804)) - Area labels: `area:shuffle` - Rationale: With AQE and direct read, both on by default, the shuffle read and decode time never reach any SQL metric. Results are unaffected, but the metrics are broken, so step 3 (see escalations). - Null-named attribute fails Arrow export with "field name cannot be null" ([#6802](https://github.com/apache/datafusion-comet/issues/6802)) - Area labels: `area:ffi`, `spark sql tests` - Rationale: An NPE in Arrow export fails the query, but it needs the opt-in `localTableScan` and a null-named attribute (SPARK-57725). A visible failure on an opt-in path, step 3 (see escalations). - GROUP BY on a struct containing a map fails with CometNativeException ([#6801](https://github.com/apache/datafusion-comet/issues/6801)) - Area labels: `spark sql tests` - Rationale: The query fails with `CometNativeException` where Spark succeeds. It was seen only with the opt-in `localTableScan` and isn't diagnosed yet, so step 3 for a visible failure (see escalations). - Native errors surface as SparkException where Spark raises a specific error class ([#6798](https://github.com/apache/datafusion-comet/issues/6798)) - Area labels: `area:expressions`, `spark sql tests`; also carries `correctness` - Rationale: Comet raises where Spark raises, but as a generic `SparkException` instead of Spark's error class. Only the error differs, the tier of #6661 and #6217, so step 3. - Math functions differ from Spark in the last digits ([#6797](https://github.com/apache/datafusion-comet/issues/6797)) - Area labels: `area:expressions`, `spark sql tests`; also carries `expression`, `correctness` - Rationale: Native `asinh`, `cosh`, `tan`, `cot`, `atanh`, `cbrt`, `exp`, `pow` and `atan2` run by default and return values that differ from Spark in the last digits. Held at medium because the values differ only in floating-point precision (see escalations). - Native Iceberg scan fails with `field not found` when an equality delete is keyed on a nested field ([#6782](https://github.com/apache/datafusion-comet/issues/6782)) - Area labels: `area:scan`; also carries `area:Iceberg` - Rationale: Every native Iceberg query on such a table fails where Spark answers. A visible failure with a workaround (turning off the native Iceberg scan), so step 3. - Native Parquet writer starts a new row group for every batch when writing to HDFS ([#6774](https://github.com/apache/datafusion-comet/issues/6774)) - Area labels: `area:writer`; also carries `performance` - Rationale: Files written natively to HDFS get a row group every 8,192 rows instead of up to 1,048,576, so readers do more work. The data is right, so this is performance degradation on the off-by-default writer, step 3. - Native fanout Iceberg writes fail when their open partitions outgrow the memory pool ([#6771](https://github.com/apache/datafusion-comet/issues/6771)) - Area labels: `area:writer`; also carries `area:Iceberg` - Rationale: The task fails with `Additional allocation failed for IcebergWriteExec` where iceberg-java's fanout writer succeeds, and the retry fails the same way. A failure is a bug under the guide's type table, even on the off-by-default writer, so this is reclassified from the author's `enhancement` to a step 3 bug. Fix in #6773. - Native sequence raises "Unreachable code reached." where Spark 4.0.5, 4.1.4 and 4.2.1 return the sequence (SPARK-58821) ([#6769](https://github.com/apache/datafusion-comet/issues/6769)) - Area labels: `area:expressions` - Rationale: On Spark 4.0.5, 4.1.4 and 4.2.1, which include SPARK-58821, Comet fails a query that Spark answers. A visible failure on edge-case inputs, step 3. - spark.comet.expression.<name>.enabled has no effect on expressions inside a codegen-dispatched tree ([#6756](https://github.com/apache/datafusion-comet/issues/6756)) - Area labels: `area:expressions`; also carries `area:udf` - Rationale: The flag the user guide documents for forcing a Spark fallback has no effect inside a dispatched tree. Results match Spark, so this is a broken feature with workarounds, step 3. - Native Parquet writes on Spark 3.4/3.5 ignore maxRecordsPerFile ([#6751](https://github.com/apache/datafusion-comet/issues/6751)) - Area labels: `area:writer` - Rationale: With the native writer opted in on Spark 3.4 and 3.5, Comet writes one file per task where Spark rolls a new file every N rows. The rows are right and the writer needs `allowIncompatible`, so step 3, as for #6315. - [Bug] Native make_date mishandles Date32 overflow and skipped evaluation ([#6741](https://github.com/apache/datafusion-comet/issues/6741)) - Area labels: `area:expressions` - Rationale: For a year past Date32 (6,000,000 in the report), `make_date` returns NULL where Spark raises `ARITHMETIC_OVERFLOW`, held at medium like #5208. The NULL-skip and LIMIT items are errors where Spark returns a result, step 3 like #6613, and come from reading the code (see escalations). - Native `mode` treats all NaNs as one value on Spark 3.4, where Spark counts each NaN separately ([#6738](https://github.com/apache/datafusion-comet/issues/6738)) - Area labels: `area:aggregation` - Rationale: Only with `spark.comet.expression.Mode.allowIncompatible=true`, Comet returns a different mode from Spark 3.4, 3.5.0 and 3.5.1. Wrong results behind an `Incompatible` opt-in follow the #5131 rule at step 3 (see escalations). - Iceberg data tables whose names end with a metadata-table name fall back to Spark ([#6719](https://github.com/apache/datafusion-comet/issues/6719)) - Area labels: `area:scan` - Rationale: An ordinary Iceberg table named, say, `user_files` or `order_history` falls back with a misleading reason, because the metadata-table check matches the end of the name. Results are correct, but those tables lose the native scan, so step 3. - Cached AQE queries fail after transition reversion on Spark 4.0+ ([#6687](https://github.com/apache/datafusion-comet/issues/6687)) - Area labels: none - Rationale: A cached AQE query fails with `FilterExec has column support mismatch`, but only with the opt-in `spark.comet.exec.transitionRevert.enabled=true`. A visible failure with a workaround, step 3. Fix in #6852. - TakeOrderedAndProject fails with "Subquery not found" when the sort order has a scalar subquery and the input has several partitions ([#6676](https://github.com/apache/datafusion-comet/issues/6676)) - Area labels: none - Rationale: With default configs, a scalar subquery in the sort keys over a multi-partition input fails the query where Spark returns rows. A visible failure with a workaround, step 3. - Codegen dispatcher fails with "Subquery has not finished" when a scalar subquery is inside the dispatched expression ([#6674](https://github.com/apache/datafusion-comet/issues/6674)) - Area labels: `area:expressions`; also carries `area:udf` - Rationale: With the dispatcher on by default, a scalar subquery inside a Scala UDF's argument, or inside `to_json` on Spark 4, fails the task where Spark succeeds, on 1.0 and 1.1 as well. A visible failure with a workaround, step 3. ### priority:low - SQLAppStatusListenerMemoryLeakSuite "no memory leak" is flaky in the Spark SQL tests ([#6890](https://github.com/apache/datafusion-comet/issues/6890)) - Area labels: `area:ci`, `spark sql tests`; also carries `test` - Rationale: A CI flake caused by a race in the Spark test itself (2 of 252 runs), which upstream fixed in SPARK-57711. Step 4, confirming the author's label. It hasn't evicted a PR from the merge queue, so the escalation trigger doesn't apply yet. - S3 test suites can no longer pull the minio/minio image ([#6884](https://github.com/apache/datafusion-comet/issues/6884)) - Area labels: `area:ci` - Rationale: Docker Hub now refuses `minio/minio`, so every suite on `CometS3TestBase` fails in `beforeAll`. Test-only, step 4, and the PR workflows don't run these suites. Fix in #6887. - CometIcebergWriteDetectionSuite assumes a split Iceberg write plan ([#6803](https://github.com/apache/datafusion-comet/issues/6803)) - Area labels: `area:writer`, `spark sql tests`; also carries `area:Iceberg` - Rationale: Five tests fail only once `localTableScan` is on by default, as in draft #6780, because the write becomes fully native. Test-only, step 4. - collect_set / collect_list return elements in a different order than Spark ([#6796](https://github.com/apache/datafusion-comet/issues/6796)) - Area labels: `area:aggregation`, `spark sql tests` - Rationale: Spark documents the order of `collect_list` and `collect_set` results as non-deterministic, so the failing tests depend on an order Spark doesn't promise. Test-only, step 4. - test: Spark SQL tests that inspect the physical plan fail when LocalTableScanExec is native ([#6795](https://github.com/apache/datafusion-comet/issues/6795)) - Area labels: `spark sql tests`; also carries `test` - Rationale: About 40 upstream tests assert on Spark nodes that become Comet nodes once `localTableScan` is on by default. Test-only, step 4. - Spark SQL Writer Tests (manual) never runs Comet's native writer ([#6749](https://github.com/apache/datafusion-comet/issues/6749)) - Area labels: `area:writer`, `area:ci` - Rationale: The manual workflow never turns on the native writer, so a green run tests Spark's writer. CI tooling, step 4. Its second finding, that the documented environment variable for `allowIncompatible` is ignored, is a minor config defect with an easy workaround. ## Enhancements - Rename Cast.is_spark4_plus and get_json_object_spark34 to describe the behavior they select ([#6891](https://github.com/apache/datafusion-comet/issues/6891)) - Area labels: `area:expressions` - Rationale: A rename with no behavior change, a refactor under the guide's type table. Confirms the author's label. - Write blog post for 1.2.0 release ([#6889](https://github.com/apache/datafusion-comet/issues/6889)) - Area labels: none - Rationale: Release work under #6550, like the 1.1.0 post (#6120). Confirms the author's label. - Add a GroupsAccumulator for the native regr_* aggregates ([#6886](https://github.com/apache/datafusion-comet/issues/6886)) - Area labels: `area:aggregation`; also carries `performance` - Rationale: A performance optimization. The slowdown it warns about would only follow if #6076 lands first, so nothing is broken on `main`. The author's `priority:medium` was left in place. - Native Iceberg writes: count the DataFiles a task keeps in its memory reservation ([#6885](https://github.com/apache/datafusion-comet/issues/6885)) - Area labels: `area:writer`; also carries `area:Iceberg`, `area:memory` - Rationale: A memory accounting improvement for the native Iceberg writer, with nothing failing. Confirms the author's label. - [EPIC] Nested-type conversion performance: C2R, R2C and JVM shuffle slower than Spark ([#6855](https://github.com/apache/datafusion-comet/issues/6855)) - Area labels: `area:shuffle`, `area:scan`; also carries `performance`, `EPIC` - Rationale: A performance EPIC. Every item is a gap against Spark or a benchmark follow-up, with correct results. - Push Iceberg residual predicates on decimal columns into the native scan ([#6854](https://github.com/apache/datafusion-comet/issues/6854)) - Area labels: `area:scan`; also carries `performance`, `area:Iceberg` - Rationale: Results are correct, and this adds row group and page pruning on decimal columns. - Follow-ups to native NULL short-circuiting of array and map functions ([#6850](https://github.com/apache/datafusion-comet/issues/6850)) - Area labels: `area:expressions` - Rationale: Follow-up cleanups to #6716. The one behavior difference it lists needs `spark.sql.subexpressionElimination.enabled=false`. Confirms the author's label. - Check the timezone label of every native expression that returns a timestamp ([#6849](https://github.com/apache/datafusion-comet/issues/6849)) - Area labels: `area:expressions`; also carries `test` - Rationale: New test coverage. #6868, merged today, checks labels where a native plan hands batches to the JVM, and these tests would also catch a label consumed inside one plan. The author's `priority:low` was left in place. - [followup] Broadcast build-side auto-bridge only accelerates the innermost join when a fact has multiple unsupported-format dimensions ([#6832](https://github.com/apache/datafusion-comet/issues/6832)) - Area labels: none - Rationale: The outer joins run on Spark, as they do without the feature, and the issue says correctness is unaffected, so this is a missed optimization in the opt-in bridge from #6786. Reclassified from the author's `bug` (see the notes). - Native fanout Iceberg writes: sort the remaining rows by partition when the memory pool runs short, instead of writing small files ([#6818](https://github.com/apache/datafusion-comet/issues/6818)) - Area labels: `area:writer`; also carries `area:Iceberg`, `area:memory` - Rationale: An improvement on the early closes in #6773, which already let the write finish. Confirms the author's label. - Interrupt a native scalar Arrow UDF when its task is killed ([#6811](https://github.com/apache/datafusion-comet/issues/6811)) - Area labels: none - Rationale: A follow-up to the unmerged #6130, for a path that isn't on `main` yet. Confirms the author's label. - perf: without AQE, native operators read Comet shuffle through a JVM round trip per block, making small reduce tasks up to 4x slower than Spark ([#6807](https://github.com/apache/datafusion-comet/issues/6807)) - Area labels: `area:shuffle`, `area:ffi`; also carries `performance` - Rationale: A performance gap against Spark with correct results and no regression identified, like #6528. Confirms the author's label. - perf: every Comet broadcast hash join probe task re-decodes the whole broadcast in the JVM ([#6806](https://github.com/apache/datafusion-comet/issues/6806)) - Area labels: none; also carries `performance`, `area:joins` - Rationale: A performance optimization of the broadcast decode path, related to #6013. Confirms the author's label. - Draft the Apache Comet proposal and board resolution ([#6791](https://github.com/apache/datafusion-comet/issues/6791)) - Area labels: none - Rationale: Project governance work under #6793, labeled like the planning trackers in earlier passes. Confirms the author's label. - Agree on the scope and repositories for Apache Comet ([#6789](https://github.com/apache/datafusion-comet/issues/6789)) - Area labels: none - Rationale: Project governance work under #6793. Confirms the author's label. - Agree on the initial PMC, committers, and chair for Apache Comet ([#6788](https://github.com/apache/datafusion-comet/issues/6788)) - Area labels: none - Rationale: Project governance work under #6793. Confirms the author's label. - perf: divide once per row in `pmod` and `div_round_half_up` ([#6781](https://github.com/apache/datafusion-comet/issues/6781)) - Area labels: `area:shuffle`, `area:expressions`; also carries `performance` - Rationale: A performance optimization of `pmod` in shuffle partitioning and of wide decimal rescaling. - Hand natively written Iceberg files' Parquet footers to the JVM instead of reading them back ([#6772](https://github.com/apache/datafusion-comet/issues/6772)) - Area labels: `area:writer`; also carries `area:Iceberg` - Rationale: A performance optimization that saves two storage reads for each natively written file. Confirms the author's label. - Warn when unrecognized spark.comet.* configuration keys are set ([#6767](https://github.com/apache/datafusion-comet/issues/6767)) - Area labels: none - Rationale: A new warning for mistyped configuration keys. Confirms the author's label. - perf: reuse the nested unsafe views in ArrowWriter's row path, detaching them once per row ([#6765](https://github.com/apache/datafusion-comet/issues/6765)) - Area labels: `area:scan`; also carries `performance` - Rationale: A performance optimization of Spark-to-Arrow conversion, under #6565. - perf: converting unsafe arrays of maps to Arrow is up to 1.7x slower when C2 inlines MapWriter.writeEntries ([#6757](https://github.com/apache/datafusion-comet/issues/6757)) - Area labels: `area:scan`; also carries `performance` - Rationale: A JIT tuning question in the #6730 writer, which is still faster than before. Not a regression, under #6565. - Codegen dispatcher: run more Scala UDF null-guard shapes inside the kernel ([#6755](https://github.com/apache/datafusion-comet/issues/6755)) - Area labels: `area:expressions`; also carries `performance`, `area:udf` - Rationale: A performance follow-up to #6715 that doesn't change results. Confirms the author's label. - Stop ending Spark-to-Arrow batches at input file boundaries ([#6752](https://github.com/apache/datafusion-comet/issues/6752)) - Area labels: `area:scan`; also carries `performance` - Rationale: A batch size optimization that depended on #6703, now merged. Confirms the author's label. - Add an end-to-end HDFS write test for both native Parquet writers ([#6750](https://github.com/apache/datafusion-comet/issues/6750)) - Area labels: `area:writer` - Rationale: New test coverage. Confirms the author's label. - Extend per-scan HTTP attempt and retry metrics to native Azure and GCS ([#6748](https://github.com/apache/datafusion-comet/issues/6748)) - Area labels: `area:scan` - Rationale: Extends the scan HTTP metrics from the unmerged #6429 to Azure and GCS. - Design question: long-term, should native Iceberg scans run through DataFusion's ParquetSource instead of iceberg-rust's ArrowReader? ([#6737](https://github.com/apache/datafusion-comet/issues/6737)) - Area labels: `area:scan` - Rationale: A design discussion about the long-term read path for native Iceberg scans, with nothing broken. - Explore Kubernetes in-place pod memory resize (Spark 4.2 ExecutorResizePlugin) for Comet's untracked native memory ([#6727](https://github.com/apache/datafusion-comet/issues/6727)) - Area labels: none; also carries `area:memory` - Rationale: An exploration of Spark 4.2's `ExecutorResizePlugin` for Comet's untracked native memory. Confirms the author's label. - Native Iceberg scan reads every field of a nested column even when Spark prunes it ([#6724](https://github.com/apache/datafusion-comet/issues/6724)) - Area labels: `area:scan`; also carries `performance`, `area:Iceberg` - Rationale: Results are correct, and this removes the I/O and decode cost of nested fields Spark has pruned. Fix in #6725. - Row-input conversionTime metric includes the time the child spends producing rows ([#6723](https://github.com/apache/datafusion-comet/issues/6723)) - Area labels: `area:scan` - Rationale: A metric that counts more than its description says. The issue proposes relabeling or dropping it, since timing each row would cost more than the conversion. Confirms the author's label. - Admit all array and map types in CometSparkToColumnarExec ([#6722](https://github.com/apache/datafusion-comet/issues/6722)) - Area labels: `area:scan`; also carries `performance` - Rationale: These types fall back today, so admitting them is new coverage. Fix in #6726. Confirms the author's label. - Reuse Arrow buffers across batches in Spark-to-Arrow conversion instead of reallocating and zeroing them ([#6720](https://github.com/apache/datafusion-comet/issues/6720)) - Area labels: `area:scan`; also carries `performance` - Rationale: A performance optimization under #6565. Confirms the author's label. - Vectorize the lower-bound check in native date_trunc for fine units ([#6717](https://github.com/apache/datafusion-comet/issues/6717)) - Area labels: `area:expressions`; also carries `performance` - Rationale: A performance optimization of a check #5956 added. The author's `priority:low` was left in place. - Native Parquet scan: FLOAT and DOUBLE comparisons in data filters don't prune with row-level pushdown ([#6702](https://github.com/apache/datafusion-comet/issues/6702)) - Area labels: `area:scan`; also carries `performance` - Rationale: Lost pruning in the opt-in row-level pushdown mode, with correct results. Confirms the author's label. - Support custom vectorized Java UDFs that operate on Arrow vectors ([#6694](https://github.com/apache/datafusion-comet/issues/6694)) - Area labels: none; also carries `area:udf` - Rationale: A new UDF API. Its prototype, #6697, was closed. Confirms the author's label. - Native Iceberg writes: write row lineage for copy-on-write DML and compaction on format-version 3 tables ([#6669](https://github.com/apache/datafusion-comet/issues/6669)) - Area labels: `area:writer`; also carries `area:Iceberg` - Rationale: These writes fall back to iceberg-java today, so native support is new functionality. Confirms the author's label. ## Escalations to consider - Native timezone conversions stop applying DST after 2099, while Spark keeps applying the zone's rules ([#6816](https://github.com/apache/datafusion-comet/issues/6816)) - Raised from the author's `priority:low` to medium, the level this pass and #5208 hold far-from-present dates at. Step 1 reads critical, and there is a realistic trigger: in a southern-hemisphere DST zone such as `Australia/Sydney`, a `9999-12-31` sentinel timestamp falls in summer, so `hour()` and casts to string come out an hour off from Spark. The reviewer may restore low, since the compatibility guide documents the difference, or raise it. - [EPIC] In-memory cache audit: correctness and performance follow-ups for InMemoryTableScanExec support ([#6834](https://github.com/apache/datafusion-comet/issues/6834)) - Labeled critical for its open wrong-results item, #6787, whose fix is #6835. Once that lands, the open items are performance work and the opt-in #6687, so medium would fit. - from_csv: wrong results in permissive modes and failure with a variant schema ([#6800](https://github.com/apache/datafusion-comet/issues/6800)) - Labeled critical from the report alone. It isn't diagnosed and was seen only with `localTableScan` on. `from_csv` goes through the codegen dispatcher, which runs Spark's own code and is on by default, so a reproduction on `main` over a Parquet source would settle whether a default path reaches it. - arrays_overlap keeps -0.0 and 0.0 apart on Spark 4.0.5, 4.1.4 and 4.2.1, which merge them (SPARK-59602) ([#6768](https://github.com/apache/datafusion-comet/issues/6768)) - Labeled critical, but only Spark 4.0.5, 4.1.4 and 4.2.1 are affected, and they were release candidates when the issue was filed. Spark 4.0.4, 4.1.3 and 4.2.0 aren't affected. - Native CSV scan reads every partition through the first partition's object store ([#6747](https://github.com/apache/datafusion-comet/issues/6747)) - Labeled critical on the opt-in native CSV scan (`spark.comet.scan.csv.v2.enabled`), as #6573 was for `spark.comet.convert.parquet.enabled`. The reviewer may prefer `priority:high` under the "core path over experimental" principle. - Native Iceberg scan returns nested values that Spark reads as NULL in files without field ids or a name mapping ([#6794](https://github.com/apache/datafusion-comet/issues/6794)) - Labeled critical because Comet's answer differs from Spark's. Here Comet returns the values stored in the file and Spark returns NULL, as the Iceberg spec requires. It needs data files added without field ids and without a name mapping, which the migration procedures don't produce. - Math functions differ from Spark in the last digits ([#6797](https://github.com/apache/datafusion-comet/issues/6797)) - Held at medium. Step 1 reads critical for any default-config difference from Spark, but these values differ only in the last digits. If any of the functions turns out to differ by more than floating-point precision, critical fits. - Native date functions and date-to-string cast return NULL or fail for dates outside chrono's NaiveDate range ([#6843](https://github.com/apache/datafusion-comet/issues/6843)) - Held at medium, with #6842 and the Date32 overflow in #6741, following #5208 and #6145. Step 1 reads critical, but they need years beyond 262142 (6,000,000 for #6741), which real data doesn't reach. #5456, a wide-date cast that overflows into wrong values, is critical, so the reviewer may want to compare. - ShuffleScan metrics are dropped under AQE: shuffle read and decode time missing on the direct-read path ([#6804](https://github.com/apache/datafusion-comet/issues/6804)) - Held at medium. Metrics-only defects have gone both ways: #6442 (lost fallback reasons) is low, and a reviewer raised #5879 (overwritten scan metrics) to high. - GROUP BY on a struct containing a map fails with CometNativeException ([#6801](https://github.com/apache/datafusion-comet/issues/6801)) - Held at medium with #6802, because both were seen only with the opt-in `localTableScan`. If either is a native panic or an NPE that a default path can reach, `priority:high` fits. - Native `mode` treats all NaNs as one value on Spark 3.4, where Spark counts each NaN separately ([#6738](https://github.com/apache/datafusion-comet/issues/6738)) - Held at medium under the #5131 rule for `Incompatible` opt-ins. Spark 3.4's behavior here is the bug SPARK-45599 fixed, which is why a reviewer lowered #5701 from critical, so low is also defensible. ## Skipped (needs more info) #6576 has no reproduction on a supported path. The others are tracking or record issues rather than bug reports or feature requests. `requires-triage` was left in place on all five, so they reappear in the next pass until they are closed or classified. - Arrow struct writer should respect projected schema width ([#6576](https://github.com/apache/datafusion-comet/issues/6576)) - There is still no reproduction on a supported path. The #6578 review on 2026-10-07 asked which producer hands Comet a struct column wider than the plan's attribute, and that is unanswered. If one exists, `main` converts a mismatched prefix silently, which is step 1. If none does, this is hardening for #6240. - Audit the PRs in 1.1.0 for regressions since 1.0.0 ([#6399](https://github.com/apache/datafusion-comet/issues/6399)) - An audit task, not a bug report or a feature request. Every phase is done and the results are in #6402, so it can be closed. - Bug triage results: 2026-10-05 ([#6665](https://github.com/apache/datafusion-comet/issues/6665)) - The summary of the 2026-10-05 pass. No type or priority label it applied has changed since, so it can be closed once reviewed. - Bug triage results: 2026-09-28 ([#6321](https://github.com/apache/datafusion-comet/issues/6321)) - The summary of the 2026-09-28 pass, with nothing outstanding, so it can be closed. - Bug triage results: 2026-08-24 ([#5454](https://github.com/apache/datafusion-comet/issues/5454)) - The summary of the 2026-08-24 pass. dwsmith1983 suggested closing it on 2026-10-04, as the other summaries from that period were. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
