andygrove opened a new issue, #4929:
URL: https://github.com/apache/datafusion-comet/issues/4929
## What is the problem the feature request solves?
Comet's native Parquet writer supports `none`, `snappy`, `lz4`, and `zstd`.
A write configured with `spark.sql.parquet.compression.codec=gzip` (or
`.option("compression", "gzip")`) falls back to Spark's writer, since
`CometDataWritingCommand.getSupportLevel` reports `Unsupported("Unsupported
compression codec: gzip")`.
gzip is a valid Spark Parquet codec and is widely used, so this is an
avoidable loss of native write coverage.
While looking at this, two related gaps in the same code path:
- `parseCompressionCodec` does not honor Spark's codec precedence. Spark's
`ParquetOptions` resolves `compression` > `parquet.compression`
(`ParquetOutputFormat.COMPRESSION`) > `spark.sql.parquet.compression.codec`,
but Comet checks only the first and third. A write with
`.option("parquet.compression", "gzip")` and the default SQLConf is therefore
natively written as snappy, silently substituting the user's requested codec.
- Spark accepts `uncompressed` as a spelling of `none`, but Comet's
supported-codec set only lists `none`, so `uncompressed` falls back
unnecessarily.
## Describe the potential solution
Add a `Gzip` value to the protobuf `CompressionCodec` enum and map it in the
native Parquet writer to `Compression::GZIP(GzipLevel::default())` (level 6,
matching parquet-mr's zlib default). Fix the codec precedence and accept the
`uncompressed` alias.
Part of #1625.
## Additional context
Compression levels remain non-configurable (gzip and zstd use fixed
defaults), so parquet-mr's `parquet.compression.codec.{gzip,zstd}.level`
configs are still ignored. That is out of scope here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
