andygrove opened a new pull request, #6644:
URL: https://github.com/apache/datafusion-comet/pull/6644
## Which issue does this PR close?
No issue. Found while updating the native Iceberg writes epic, #5649.
## Rationale for this change
#6494 renamed `spark.comet.iceberg.write.enabled` to
`spark.comet.write.iceberg.enabled`. #6415 merged after it and added a Comet
session to `TestPartitionedWritesToWapBranch` in the Iceberg 1.11.0 diff that
still sets the old key. Comet does not read that key, so the suite has been
running Iceberg's JVM writer under Comet's split operator instead of the native
writer. The other 31 sessions in the 1.11.0 diff, and every session in the
1.8.1, 1.9.1 and 1.10.0 diffs, use the new key.
The CI write report from #6155 did record these writes as JVM writes, but
under the reason `IcebergWrite is not supported`, which does not point at the
config.
## What changes are included in this PR?
`dev/diffs/iceberg/1.11.0.diff` now sets `spark.comet.write.iceberg.enabled`
in `TestPartitionedWritesToWapBranch`. The diff was regenerated as the
contributor guide describes: apply it to `apache-iceberg-1.11.0`, edit the
test, `git add -A`, then `git diff apache-iceberg-1.11.0`. The only other
change is that file's `index` line.
## How are these changes tested?
- Regenerating the diff without the edit reproduces the committed diff byte
for byte, and the new diff applies cleanly to a fresh `apache-iceberg-1.11.0`
checkout.
- `TestPartitionedWritesToWapBranch` was run locally with Iceberg 1.11.0 on
Spark 4.1, against Comet built from `main` at bdbc4605d, with the environment
variables the CI job sets. All 36 tests pass with either key.
`dev/ci/summarize-iceberg-writes.py` reports:
| Key | Comet native writer | Iceberg JVM writer under Comet's split
operator |
| --- | ---: | ---: |
| `spark.comet.iceberg.write.enabled` (before) | 0 | 76 |
| `spark.comet.write.iceberg.enabled` (after) | 76 | 0 |
- The Iceberg 1.11.0 jobs run in the merge queue and cover the rest of the
extensions suite.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
