mbutrovich commented on code in PR #201:
URL: https://github.com/apache/datafusion-site/pull/201#discussion_r3730779060
##########
content/blog/2026-08-07-datafusion-comet-1.0.0.md:
##########
@@ -0,0 +1,188 @@
+---
+layout: post
+title: Apache DataFusion Comet 1.0.0 Release
+date: 2026-08-07
+author: pmc
+categories: [subprojects]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+The Apache DataFusion PMC is pleased to announce version 1.0.0 of the
[Comet](https://datafusion.apache.org/comet/) subproject.
+
+Comet is an accelerator for Apache Spark that translates Spark physical plans
to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.
+
+This release covers roughly six weeks of development since 0.17.0 and consists
of 244 commits from 23
+contributors. See the [change log] for the full list of changes.
+
+[change log]:
https://github.com/apache/datafusion-comet/blob/main/docs/source/changelog/1.0.0.md
+
+## The Road to 1.0
+
+Comet was [donated] to the Apache DataFusion project in March 2024 and cut its
first release, 0.1.0, five
+months later with support for 13 operators and 106 expressions. Since then,
the project has shipped 20
+releases and drawn contributions from more than 120 developers, and the
codebase now recognizes over 400
+Spark expressions.
+
+[donated]: https://datafusion.apache.org/blog/2024/03/06/comet-donation/
+
+The 1.0 release marks the point at which Comet begins following semantic
versioning. Users upgrading within
+the 1.x line can expect backward-compatible changes only; features slated for
removal will be deprecated in
+a minor release before being dropped in the next major version. This is why
the deprecations of JDK 11 and
+Spark 3.4 announced below are scheduled for 1.1 rather than landing in 1.0
itself.
+
+The rest of this section is a recap of the main advances since donation.
+
+### Support for Spark 4.0+ with ANSI mode
+
+Comet 1.0.0 supports Spark versions 3.4 though 4.1, with experimental support
for 4.2. Comet fully supports Spark's ANSI mode, which is enabled by default
starting with Spark 4.0.
+
+### Correctness Testing
+
+It is important that queries accelerated by Comet produce the same results as
Spark. Correctness checking has always been a large effort in Comet
development, but the approach has evolved over time.
+
+- Comet has always run Spark's own test suite with Comet enabled, providing
more than 24,000 unit tests effectively for free. These tests run in Comet's CI
for all supported Spark versions.
+- Scala tests: Comet has Scala tests that run queries end to end with Comet
enabled vs disabled and ensure that the results match
+- Fuzz testing: Many of the scala tests use a fuzz testing approach to
generate randomized data that queries run against, helping to catch regressions
around edge cases such as nulls, NaN, Infinity, and timezone issues
+- Comet SQL Tests: In an effort to make it easier to write tests, Comet now
provides a SQL-based testing approach that is inspired by sqllogictest
+- Generative AI: More recently, Comet has taken advantage of agentic skills to
perform audit sweeps of all expressions, comparing the implementation to
Spark's source code and ensuring that Comet has tests covering all important
edge cases
+
+### Performance
+
+The early Comet releases provided a very modest speedup and the published
benchmark results were based on running TPC workloads at small scale factors on
a single node. There are now independent benchmark results published by AWS
Labs that show significant speedups for [TPC-DS @ 3TB running in
EKS](https://awslabs.github.io/data-on-eks/docs/benchmarks/spark-datafusion-comet-benchmark).
+
+### Codegen Dispatch
+
+Comet 0.17.0 introduced a new approach to filling gaps in expression coverage.
In earlier releases,
+whenever Comet's planner encountered an expression that lacked a native Rust
implementation, it fell back
+to executing an entire subtree of the plan in Spark. That required converting
Arrow columns back to Spark
+rows before the expression ran and back to Arrow after, and the cost was often
enough to erase the speedup
+Comet had bought elsewhere in the plan.
+
+Codegen dispatch scopes that fallback down to the expression itself: the batch
stays in the Comet pipeline
+and Comet invokes Spark's own generated code for just the missing expression,
leaving the rest of the query
+running natively. Three consequences are worth calling out.
Review Comment:
I would add a fourth: expression fusion. Because we compile these
expressions in the same dispatch, the compile is free to run suitable
optimizations and it avoids intermittent materialization of entire Arrow
RecordBatches between each expression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
