1fanwang opened a new pull request, #2438:
URL: https://github.com/apache/iceberg-rust/pull/2438
The PuffinWriter exposed a `compress_footer: bool` flag that set the
FooterPayloadCompressed bit in the file footer and recorded LZ4 as the codec,
but `CompressionCodec::Lz4.compress` returned `FeatureUnsupported`, so calling
`PuffinWriter::new(_, _, true)` always failed at close time:
thread '...' panicked at crates/iceberg/src/puffin/writer.rs:350:
called `Result::unwrap()` on an `Err` value:
FeatureUnsupported => LZ4 compression is not supported currently
The reader side had the same hole: a Puffin file written by another
implementation with FooterPayloadCompressed=1 was unreadable.
This change wires LZ4 through `CompressionCodec::{compress,decompress}`
using `lz4_flex`'s frame encoder/decoder, matching the Puffin spec requirement
of "LZ4 single compression frame with content size present" by setting
`FrameInfo::content_size(Some(len))` on the encoder. The reader path is
symmetric — `FrameDecoder::read_to_end` consumes the same frame. `lz4_flex` is
already in the lock file as a transitive dep of `parquet`, so no new dependency
tree is introduced.
## Are these changes tested?
Yes:
- `test_compress_empty_footer_lz4_succeeds` — direct adaptation of the
reproducer in the issue (close() with no blobs, compress_footer=true).
- `test_compress_footer_lz4_round_trips` — full encode + decode loop with
one blob and non-empty file properties.
- `test_write_lz4_compressed_metric_data` — previously asserted the error
string; now asserts a two-blob round-trip.
- `test_lz4_compressed_footer_is_decoded` (metadata) — verifies the reader
honors the FooterPayloadCompressed flag.
- `test_compression_codec_lz4_roundtrip` — round-trip on empty and mixed
payloads, plus a magic-number check that the frame begins with `0x184D2204`.
```
running 6 tests (compression::)
test result: ok. 6 passed; 0 failed
running 38 tests (puffin::)
test result: ok. 38 passed; 0 failed
Full iceberg lib suite: 1245 passed; 0 failed
cargo clippy -p iceberg --lib --tests: clean
cargo fmt --check: clean
```
Closes #2419
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
