matanper opened a new issue, #1368: URL: https://github.com/apache/arrow-go/issues/1368
### Describe the enhancement requested `ipc.WithZstd()` turns on ZSTD buffer compression for IPC writers but gives no way to choose the level. `getCompressor` in `arrow/ipc/compression.go` always builds the encoder with `zstd.NewWriter(nil)`, so every IPC writer compresses at klauspost/compress's `SpeedDefault` (roughly zstd level 3). The compressor is created inside the record encoder, so a caller can't supply its own either. For a producer that streams large compressed record batches to a consumer on the same network (e.g. Arrow IPC over HTTP to an ingestion endpoint), compression is a noticeable share of producer CPU. Since decompression speed barely depends on the level, level 1 (`SpeedFastest`) would cut producer CPU with almost no effect on the reader. Today the only way to get there is to patch arrow-go. Proposal: an option that sets the ZSTD level, mirroring what Parquet's writer properties already allow for column compression, e.g. ```go // WithZstdLevel enables ZSTD compression of record batch bodies at the given // compression level (as in compress.Zstd levels; zstd.EncoderLevelFromZstd maps it). func WithZstdLevel(level int) Option ``` `getCompressor` would then pass `zstd.WithEncoderLevel(zstd.EncoderLevelFromZstd(level))` to `zstd.NewWriter`, while `WithZstd()` keeps today's default. The C++ implementation already lets callers set this via `IpcWriteOptions::codec` built from `util::Codec::Create(Compression::ZSTD, level)`, so this would also bring Go in line with C++. I'm happy to send a PR if this direction sounds good. ### Component(s) Other -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
