LuciferYang opened a new issue, #1013:
URL: https://github.com/apache/iceberg-cpp/issues/1013
## Summary
With `write.metadata.compression-codec=gzip`, iceberg-cpp names new metadata
files `<version>-<uuid>.gz.metadata.json`, the same layout Iceberg Java uses,
but the round trip is broken on both sides:
- `TableMetadataUtil::Write` never compresses. It writes plain JSON into the
`.gz.metadata.json` file, so Java (or any reader that trusts the name) fails to
inflate it.
- `Codec::FromFileName` misdetects the codec. It calls
`find_last_of(".metadata.json")`, which searches for any single character of
that set, not the suffix. For `00001-abc.gz.metadata.json` it lands on the
final `n`, the `.gz` check fails, and the file is read as uncompressed. A gzip
file written by Java under that name therefore fails to parse.
The two bugs mask each other inside iceberg-cpp: a file it writes is plain
JSON and is read back as plain JSON, so its own round trip happens to work.
## Root cause
`src/iceberg/table_metadata.cc`:
```cpp
auto pos = file_name.find_last_of(kTableMetadataFileSuffix); //
character-set search
```
and `TableMetadataUtil::Write(io, location, metadata)` serializes and calls
`io.WriteFile` with no codec step. Only a `GZipDecompressor` exists in
`util/gzip_internal.h`.
Java's `TableMetadataParser.Codec.fromFileName` uses
`lastIndexOf(".metadata.json")`, and `internalWrite` wraps the stream in
`GZIPOutputStream` when that codec is gzip.
## Impact
A table configured with gzip metadata compression is not readable across
engines in either direction: Java cannot read what iceberg-cpp writes, and
iceberg-cpp cannot read what Java writes. The default codec is `none`, so
tables without that property are unaffected.
The `find_last_of` bug also means `FromFileName` accepts names with no
`.metadata.json` at all (e.g. `foo.txt` is classified as uncompressed), while
Java rejects them.
## Proposed fix
Use `rfind` in `FromFileName`, and compress in `Write` when the location's
codec is gzip, with a small `GZipCompressor` next to the existing
`GZipDecompressor`. Both halves have to land together: fixing only the read
side makes iceberg-cpp try to inflate the plain JSON it writes today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
