andygrove opened a new issue, #7403: URL: https://github.com/apache/datafusion-comet/issues/7403
## Summary - Date: 2026-11-02 - `requires-triage` issues found: 65. Labeled: 63 (21 bugs, 42 enhancements). Skipped: 2. Failed to label: 0. - Bugs by priority: `priority:critical` 2, `priority:high` 7, `priority:medium` 10, `priority:low` 2 - Guide: [docs/source/contributor-guide/bug_triage.md](https://github.com/apache/datafusion-comet/blob/main/docs/source/contributor-guide/bug_triage.md) Labels have already been applied and `requires-triage` removed from every labeled issue. Please spot-check the calls and close this issue when satisfied. Many issues already carried a type label, a priority, or both from their author; those were kept when they matched the guide. ## Bugs ### priority:critical - Native parse_url accepts URLs and registry-based hosts that java.net.URI treats differently ([#7330](https://github.com/apache/datafusion-comet/issues/7330)) - Area labels: `area:expressions` - Rationale: Native parse_url returns components or hosts where Spark returns NULL/INVALID_URL, a silent wrong result on a default-on expression. - Native parse_url returns NULL for relative URLs (no scheme), Spark parses them ([#7385](https://github.com/apache/datafusion-comet/issues/7385)) - Area labels: `area:expressions` - Rationale: Native parse_url silently returns NULL for scheme-less URLs where Spark returns values, a silent wrong result. ### priority:high - Native Iceberg write fails with "no more field nodes" on a struct column when its input is a Spark-to-Arrow conversion ([#7239](https://github.com/apache/datafusion-comet/issues/7239)) - Area labels: `area:writer`, `area:Iceberg` - Rationale: Native Iceberg write task fails with a native exception on a struct column instead of falling back. - Native execution fails with offset overflow when a large scalar string is expanded to the batch length ([#7247](https://github.com/apache/datafusion-comet/issues/7247)) - Area labels: `area:shuffle`, `area:aggregation`, `area:expressions` - Rationale: An offset overflow fails the query on a native path; the author set priority:high and it matches the guide. - Nested dictionary-encoded BINARY read as string overflows Utf8 offsets when flattened per batch ([#7254](https://github.com/apache/datafusion-comet/issues/7254)) - Area labels: `area:scan` - Rationale: Native scan fails with an offset overflow on valid data that Spark reads, the same class as #7247. - Executor JVM abort (NMT footer canary broken) when the in-memory cache is enabled on Spark 3.4 ([#7305](https://github.com/apache/datafusion-comet/issues/7305)) - Area labels: `area:scan` - Rationale: Executor JVM aborts on native memory corruption (NMT footer canary), a crash. - Native Parquet scan builds an invalid StringArray for BINARY map keys read as STRING ([#7360](https://github.com/apache/datafusion-comet/issues/7360)) - Area labels: `area:scan` - Rationale: StringArray::new_unchecked is given invalid UTF-8, a memory-safety violation in native code (not a regression). - GetStructField fails on a scalar struct beside a column ([#7380](https://github.com/apache/datafusion-comet/issues/7380)) - Area labels: `area:expressions` - Rationale: CometNativeException on a scalar struct field beside a column fails the query on a native path. - Comet 1.1.0: FileSourceScanExec.copy NoSuchMethodError on EMR Spark 4.1.1 ([#7393](https://github.com/apache/datafusion-comet/issues/7393)) - Area labels: `area:scan`, `spark 4.1` - Rationale: NoSuchMethodError fails the cached query on a vendor Spark build, an unhandled exception on a supported code path. ### priority:medium - Native Iceberg write silently falls back on Iceberg 1.12: ManifestFiles.read(ManifestFile, FileIO) was removed ([#7240](https://github.com/apache/datafusion-comet/issues/7240)) - Area labels: `area:writer`, `area:Iceberg` - Rationale: Native Iceberg writes silently fall back on Iceberg 1.12; the query still runs on Spark. - Native memory requests treat an error from Spark as a refusal ([#7244](https://github.com/apache/datafusion-comet/issues/7244)) - Area labels: `area:shuffle` - Rationale: An error from Spark during a memory request is treated as a refusal, so a task spills and continues instead of failing. - Native lpad and rpad fail when the string argument evaluates to a scalar, such as a scalar subquery inside IF ([#7245](https://github.com/apache/datafusion-comet/issues/7245)) - Area labels: `area:expressions` - Rationale: lpad/rpad fail on scalar string input, a broken feature with a fallback to Spark; the author-set priority:medium is kept. - Native OR, AND, IN, concat, NULL operands and later top-K sort keys evaluate operands Spark skips, raising ANSI errors ([#7246](https://github.com/apache/datafusion-comet/issues/7246)) - Area labels: `area:expressions` - Rationale: Native operators evaluate operands Spark skips and raise ANSI errors; the failure is visible, not silent, so the author-set priority:medium is kept. - make_date under a codegen-dispatched parent falls back to Spark since #7261 ([#7284](https://github.com/apache/datafusion-comet/issues/7284)) - Area labels: `area:expressions` - Rationale: Since #7261 make_date under a codegen-dispatched parent moves the whole operator back to Spark, a lost-native-coverage regression with identical results. - JVM columnar shuffle writers use Comet defaults in map stages of jobs outside a SQL execution ([#7303](https://github.com/apache/datafusion-comet/issues/7303)) - Area labels: `area:shuffle` - Rationale: JVM columnar shuffle writers ignore session spark.comet.* settings outside a SQL execution, a functional bug with a workaround. - CapturedSQLConf applies a stale capture inside a SQL execution when every captured key was unset ([#7312](https://github.com/apache/datafusion-comet/issues/7312)) - Area labels: `area:shuffle` - Rationale: A stale config capture overrides the session configs in a narrow case, a functional bug that needs a rerun map stage. - Native Iceberg scan fails with ExpiredToken once REST-vended credentials expire ([#7387](https://github.com/apache/datafusion-comet/issues/7387)) - Area labels: `area:scan`, `area:Iceberg` - Rationale: Long native Iceberg scans fail with ExpiredToken; a credential provider class is the workaround. - The documented IcebergRESTVendedS3Provider never refreshes against a REST catalog ([#7388](https://github.com/apache/datafusion-comet/issues/7388)) - Area labels: `area:scan`, `area:Iceberg` - Rationale: The documented reference provider never refreshes credentials against a REST catalog; a working provider is the workaround. - S3 credential providers are retained until JVM shutdown when a catalog re-vends credentials ([#7390](https://github.com/apache/datafusion-comet/issues/7390)) - Area labels: `area:scan`, `area:Iceberg` - Rationale: Superseded credential providers (HTTP client, auth session) are retained until JVM shutdown, a resource leak on long-lived executors. ### priority:low - ci: concurrency group is keyed on the branch name, so same-named fork branches can cancel each other's CI ([#7315](https://github.com/apache/datafusion-comet/issues/7315)) - Area labels: `area:ci` - Rationale: CI concurrency key can cancel a same-named fork branch, a tooling problem not seen in practice. - Delta contrib: a split classloader is reported as a Comet version mismatch ([#7394](https://github.com/apache/datafusion-comet/issues/7394)) - Area labels: `area:scan`, `area:Delta` - Rationale: The scan falls back as designed but the log blames a version mismatch; the defect is a misleading message with a classpath workaround. ## Enhancements - Audit the PRs in 1.1.0 for regressions since 1.0.0 ([#6399](https://github.com/apache/datafusion-comet/issues/6399)) - Area labels: none - Rationale: An audit task over the 1.1.0 PRs rather than a reported defect. - Move the version checks for Spark patch-release fixes into SparkFixes ([#7241](https://github.com/apache/datafusion-comet/issues/7241)) - Area labels: none - Rationale: Refactor that moves Spark patch-release version checks into SparkFixes. - Run user-defined generators (Hive UDTFs, custom Generators) in the Comet pipeline ([#7252](https://github.com/apache/datafusion-comet/issues/7252)) - Area labels: `area:udf` - Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept. - Offset overflow- Native shuffle writer: retry with fewer rows on offset overflow ([#7269](https://github.com/apache/datafusion-comet/issues/7269)) - Area labels: `area:shuffle` - Rationale: Proposes retrying with fewer rows on offset overflow, a new capability tracked under #7267. - [EPIC] Comet Native Executors: run Spark queries on native executors built on DataFusion Ballista ([#7288](https://github.com/apache/datafusion-comet/issues/7288)) - Area labels: none - Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept. - [EPIC] Iceberg work for 1.3.0: native writes on by default and scan correctness ([#7293](https://github.com/apache/datafusion-comet/issues/7293)) - Area labels: `area:Iceberg` - Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept. - perf: decode cached batches in native code instead of on the JVM ([#7296](https://github.com/apache/datafusion-comet/issues/7296)) - Area labels: `area:scan` - Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept. - perf: store dictionary-encoded columns as dictionaries in the in-memory cache ([#7297](https://github.com/apache/datafusion-comet/issues/7297)) - Area labels: `area:scan` - Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept. - perf: report vectorTypes from the Arrow cached batch serializer ([#7298](https://github.com/apache/datafusion-comet/issues/7298)) - Area labels: `area:scan` - Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept. - [EPIC] Native Spark expressions: run as many Spark built-in expressions as possible natively, by default ([#7300](https://github.com/apache/datafusion-comet/issues/7300)) - Area labels: `area:expressions` - Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept. - Extend sort-merge membership key reduction to anti joins and eligible conditions ([#7317](https://github.com/apache/datafusion-comet/issues/7317)) - Area labels: none - Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept. - Evaluate membership key reduction across small shuffle batches ([#7318](https://github.com/apache/datafusion-comet/issues/7318)) - Area labels: none - Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept. - ci: a publish_snapshot.yml dry run still passes the Nexus secrets to the build ([#7319](https://github.com/apache/datafusion-comet/issues/7319)) - Area labels: `area:ci` - Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept. - ci: the docs deploy builds with its dependencies in a job that can write to the repository ([#7320](https://github.com/apache/datafusion-comet/issues/7320)) - Area labels: `area:ci` - Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept. - Run filter lambdas natively with DataFusion's array_filter ([#7321](https://github.com/apache/datafusion-comet/issues/7321)) - Area labels: none - Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept. - perf: hll_sketch_agg merge builds a new union for every merged row ([#7341](https://github.com/apache/datafusion-comet/issues/7341)) - Area labels: `area:aggregation` - Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept. - Evaluate making jemalloc the default allocator for the release build ([#7344](https://github.com/apache/datafusion-comet/issues/7344)) - Area labels: none - Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept. - docs: move existing pages to the style guide's "native" rules ([#7358](https://github.com/apache/datafusion-comet/issues/7358)) - Area labels: none - Rationale: Documentation migration to the style guide. - [EPIC] Track arrow-java bugs and limitations that Comet works around ([#7362](https://github.com/apache/datafusion-comet/issues/7362)) - Area labels: `area:ffi` - Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept. - Add workflow to reject AI spam PRs ([#7364](https://github.com/apache/datafusion-comet/issues/7364)) - Area labels: none - Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept. - Comet Native Executors: run LIMIT and top-K queries with a per-partition limit before the exchange ([#7365](https://github.com/apache/datafusion-comet/issues/7365)) - Area labels: none - Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept. - Comet Native Executors: range-partition sorts on float, nested and expression keys ([#7366](https://github.com/apache/datafusion-comet/issues/7366)) - Area labels: none - Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept. - Comet Native Executors: partition hash exchanges with Spark's Murmur3 hash ([#7368](https://github.com/apache/datafusion-comet/issues/7368)) - Area labels: none - Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept. - Comet Native Executors: run a small broadcast side once rather than in every task that reads it ([#7369](https://github.com/apache/datafusion-comet/issues/7369)) - Area labels: none - Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept. - Comet Native Executors: support dynamic partition pruning ([#7370](https://github.com/apache/datafusion-comet/issues/7370)) - Area labels: none - Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept. - Comet Native Executors: support writes ([#7371](https://github.com/apache/datafusion-comet/issues/7371)) - Area labels: none - Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept. - Comet Native Executors: cancel a query on the cluster when Spark cancels it ([#7372](https://github.com/apache/datafusion-comet/issues/7372)) - Area labels: none - Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept. - Comet Native Executors: report a query's metrics in the Spark UI ([#7373](https://github.com/apache/datafusion-comet/issues/7373)) - Area labels: none - Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept. - Comet Native Executors: pass scalar subquery results to the cluster without rewriting the plan ([#7374](https://github.com/apache/datafusion-comet/issues/7374)) - Area labels: none - Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept. - Comet Native Executors: let the native planner reject every JVM callback, and drop the driver's list ([#7375](https://github.com/apache/datafusion-comet/issues/7375)) - Area labels: none - Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept. - Comet Native Executors: reuse Flight connections in the driver's client ([#7376](https://github.com/apache/datafusion-comet/issues/7376)) - Area labels: none - Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept. - Audit license and copyright texts for all native dependencies bundled in libcomet ([#7383](https://github.com/apache/datafusion-comet/issues/7383)) - Area labels: none - Rationale: An audit of bundled license texts and tooling to generate them, a follow-up to #7382. - Comet Native Executors: experiment with a native client and server for Spark's shuffle block transfer protocol ([#7384](https://github.com/apache/datafusion-comet/issues/7384)) - Area labels: none - Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept. - No refreshing Iceberg credential provider on Spark 3.x builds ([#7389](https://github.com/apache/datafusion-comet/issues/7389)) - Area labels: `area:scan`, `area:Iceberg`, `spark 3.x` - Rationale: Explicit request to ship a refreshing credential provider in Spark 3.x builds. - Comet Native Executors: use Ballista's adaptive planning for joins inside Comet fragments ([#7391](https://github.com/apache/datafusion-comet/issues/7391)) - Area labels: none - Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept. - Upgrade the datasketches crate to 0.6.0 and drop the HLL decode workarounds ([#7395](https://github.com/apache/datafusion-comet/issues/7395)) - Area labels: `area:expressions` - Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept. - Audit the HLL expressions against Spark on datasketches 0.6.0 and revisit the allowIncompatible opt-in ([#7396](https://github.com/apache/datafusion-comet/issues/7396)) - Area labels: `area:aggregation`, `area:expressions` - Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept. - [EPIC] Native DataSketches: theta, KLL, frequent items and tuple sketches ([#7397](https://github.com/apache/datafusion-comet/issues/7397)) - Area labels: `area:aggregation`, `area:expressions` - Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept. - Native theta sketch functions (theta_*): audit the crate against Java and implement ([#7398](https://github.com/apache/datafusion-comet/issues/7398)) - Area labels: `area:aggregation`, `area:expressions` - Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept. - Native KLL sketch functions (kll_*): audit the crate against Java and implement ([#7399](https://github.com/apache/datafusion-comet/issues/7399)) - Area labels: `area:aggregation`, `area:expressions` - Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept. - Native approx_top_k functions: audit the frequent-items sketch against Java and implement ([#7400](https://github.com/apache/datafusion-comet/issues/7400)) - Area labels: `area:aggregation`, `area:expressions` - Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept. - Native tuple sketch functions (tuple_*, Spark 4.2): audit the crate against Java and implement ([#7401](https://github.com/apache/datafusion-comet/issues/7401)) - Area labels: `area:aggregation`, `area:expressions` - Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept. ## Escalations to consider - Executor JVM abort (NMT footer canary broken) when the in-memory cache is enabled on Spark 3.4 ([#7305](https://github.com/apache/datafusion-comet/issues/7305)) - A `priority:high` crash from memory corruption (a write past the end of a native block) may also produce wrong results silently before it is detected; consider `priority:critical` if the cause turns out to be corruption of cached data. - Native Parquet scan builds an invalid StringArray for BINARY map keys read as STRING ([#7360](https://github.com/apache/datafusion-comet/issues/7360)) - A `priority:high` memory-safety violation (invalid UTF-8 in a `StringArray`) could produce wrong results silently rather than a crash; consider `priority:critical` if so. ## Skipped — needs more info - Bug triage results: 2026-10-26 ([#7230](https://github.com/apache/datafusion-comet/issues/7230)) - This is the previous bug triage summary issue, not a bug or enhancement; the reviewer closes it. - TPC-DS q54/q68: local SF100 results and distributed reproduction limits ([#6946](https://github.com/apache/datafusion-comet/issues/6946)) - A benchmark reproduction report for #6418 that says it is not a claim of a bug; it is unclear whether it is a bug, an enhancement or a record, and no defect is stated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
