andygrove opened a new issue, #6889: URL: https://github.com/apache/datafusion-comet/issues/6889
### What is the problem the feature request solves? The Comet 1.2.0 release needs a blog post. The [release process](https://github.com/apache/datafusion-comet/blob/main/docs/source/contributor-guide/release_process.md#write-a-blog-post) is to draft it in a shared document, agree on the content, and then open a PR against [datafusion-site](https://github.com/apache/datafusion-site). Release tracking: #6550. The 1.1.0 post is tracked in #6120. ### Describe the potential solution Draft the post from the summary below, then open the `datafusion-site` PR so it can go out with the 1.2.0 announcement. The summary is a snapshot of `main` at 019f3be9d1 (2026-10-12). Counts, defaults and the list of open PRs will move before the release branch is cut, so refresh them then. Performance figures are quoted from the PR descriptions and were not re-measured, so re-verify any number before it goes in the post. ### Additional context #### What is new on `main` since 1.1.0 The `1.1.0` tag is on `branch-1.1`, which was cut from `main` on 2026-09-25, so `git log 1.1.0..main` overcounts. After removing the commits that shipped in 1.1.0 as backports (matched by PR number and by title), `main` has **247 new commits** from 2026-09-26 to 2026-10-12, from 26 authors plus Dependabot: 98 fixes, 33 features, 27 performance changes, 26 docs changes, 25 test changes and 38 chore, refactor, CI and build changes. For scale, 1.1.0 contains 418 commits that 1.0.0 does not. **1. Spark-exact floats and timezones.** A systematic sweep instead of per-expression fixes. - The timezone EPIC #6335 is closed (8 of 8). It covers correct timezone labels on native timestamp results (#6341, #6345, #6347), session timezone ID normalization (#6351), a warning when the native and JVM timezone databases differ (#6353), per-row `date_trunc` formats (#6354), and a test check that fails when a native plan mislabels a timestamp (#6868). - The float EPIC #6385 is 16 of 20 (open: #6549 for map keys on Spark 4.0+, #6614, #6636, #6738). Signed zero and NaN now follow Spark in hashing (#6413), `min`/`max`/`greatest`/`least` (#6457), nested sort and window keys (#6475), `array_remove` and `sort_array` (#6518), `signum` (#6556) and `percentile` (#6829). - Collation: unsupported collated expressions now fall back to Spark or route through the codegen dispatcher (#5302, #6206, #6471, #6480). **2. Comet's in-memory cache is on by default (Spark 3.5+).** #5634 makes `spark.comet.exec.inMemoryCache.enabled` default to `true` there, and Spark 3.4 stays off. Hardening around it: serializer and Kryo gating (#6360, #6537), EXPLAIN and SQL-tab fidelity (#6453, #6574, #6577), planner statistics (#6412), fused vector reads (#5859) and an opt-in lz4 codec (#6859). Audit follow-ups are tracked in #6834. **3. Iceberg native writes and row-level operations.** - Writes to format-version 3 tables (#6672), per-location credentials for reads and writes (#6478), dictionary encoding that matches iceberg-java (#6241), writer buffers charged to the task's memory pool (#6247), and eligibility gates so unsupported writes fall back to iceberg-java (#6441, #6502, #6531, #6677). - Row-level operations: native `MergeRowsExec` for `MERGE INTO` (#5318, `spark.comet.exec.mergeRows.enabled`, off by default), `MergeSummary` on Spark 4.1+ (#6640) and Spark 4.2 write transactions (#6587). - Native Iceberg writes are still off by default. #6664 flips that and is planned for 1.2.0 (open). - The write settings were renamed (#6494): `spark.comet.iceberg.write.enabled` is now `spark.comet.write.iceberg.enabled`, and `spark.comet.parquet.write.enabled` is now `spark.comet.write.parquet.enabled`. **4. Memory and spill.** - Native plan memory is returned before the Spark task ends (#6261), acquires are retried after Spark drops the task's entry (#6310, #6403), a spilled final aggregate reads its spill files back past its memory share (#6544), and the native memory limit follows YARN and Kubernetes container sizing (#6375). - Spill files are compressed with lz4 by default (#6698, `spark.comet.exec.spill.compression.codec`). The PR reports TPC-H q21 going from 380 s to 250 s, against 367 s for Spark. - Executor memory usage is written to the Spark event log (#6401, #6250, #6271). **5. More of the plan runs natively.** - On by default: existence joins (#4587, `spark.comet.exec.existenceJoin.enabled`) and conversion of `OneRowRelation` (#6666). - Supported: `SortAggregateExec` (#4565), distinct-combined `collect_list` (#4727), HLL sketch functions (#4802), and interval multiply and divide through codegen dispatch (#4715, #4901). - Opt-in: native `RangeExec` (#6553, `spark.comet.exec.range.enabled`), conversion of typed Dataset operations (#6564), native shuffle over row input (#6607), and RDD, range, cache and Data Source V1 conversions, each with its own `spark.comet.convert.*` setting (#6602). - Experimental: custom Rust UDFs through Arrow FFI (#4459). **6. Performance.** - Native `CASE WHEN` and `IF` up to 12x faster (#6350), string-to-timestamp up to 5.6x faster (#5130). - Spark-to-Arrow conversion: bulk column writes (#6566), unsafe rows converted straight from their memory (#6730), and JVM columnar shuffle for strings, maps and arrays (#6604). - Joins, scans and shuffle: filtering decoded probe batches before intermediate joins (#6433), opt-in TopK threshold pushdown into Parquet readers (#6181, #6263) and shuffle blocks read in 64 KiB pieces (#6805). - Benchmarks for the remaining native kernels (#6541, #6647, #6648). **7. Platform and project.** - Spark 4.2.0 is a supported version, and the Spark SQL tests run against the released 4.2.0 (#4950, #6417). - DataFusion 55.2.0 (#6766). iceberg-rust is still a pinned git revision, as in 1.1.0. - The native code is split into focused crates (#6435, #6450, #6503, #6534). - Release hygiene: a license check on the release source tarball (#6639) and `NOTICE` shipped in the jars (#6657). - Docs: an FAQ and a refreshed Gluten comparison (#6778, #6783), a Celeborn guide (#6373), a timezone page (#6337, #6686) and TPC-DS results for 1.1.0 on Spark 4.2 (#6308). **Not worth a headline:** Spark 4.1 `TIME` type support through codegen dispatch (#4821). Spark keeps the type behind an internal flag, `spark.sql.timeType.enabled`, that defaults to off outside tests in 4.1.1, 4.1.3 and 4.2.0. #### Open PRs that would be headline items if they land before the cut (as of 2026-10-12) - #6664: native Iceberg writes on by default (blocked on #6773). - #5365: native Delta Lake scan contrib module (approved, not yet merged). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
