mavam opened a new issue, #1016: URL: https://github.com/apache/iceberg-cpp/issues/1016
## Summary Manifest Avro headers written by iceberg-cpp do not conform to the Iceberg manifest metadata contract: 1. The `partition-spec` header contains the full partition-spec object instead of only its partition-fields array. 2. The separate `schema-id` header is omitted, although it is required for v2 manifests. The first defect prevents ClickHouse from reading the resulting table, including queries that select only scalar fields. Both defects occur at the manifest metadata adapter boundary, independently of the data schema or Parquet contents. ## Versions and scope - Observed in generated v2 manifests with iceberg-cpp commit `131f9763ecca8b5985e48bb06a957b4bebc6779f`, used by Tenzir's `to_iceberg` writer. - Confirmed by source inspection to remain in main at `57ec6b8101952a47916ab9cd4328a17319b40195`. - The completed interoperability test used an unpartitioned v2 table created through `apache/iceberg-rest-fixture:1.10.1`, followed by four sequential appends through `NewFastAppend()`. All four original manifests had both header defects. - The incorrect partition-spec serialization also appears in the v1 and v3 adapter implementations. These versions were inspected, not exercised in the interoperability test. ## Expected and actual metadata The [Iceberg 1.10.1 manifest specification](https://github.com/apache/iceberg/blob/apache-iceberg-1.10.1/format/spec.md#manifests) requires: - `partition-spec`: "JSON representation of only the partition fields array of the partition spec used to write the manifest." - `schema-id`: "ID of the schema used to write the manifest as a string." This header is optional in v1 and required in v2. For an unpartitioned manifest written with schema ID 0, the expected headers are: ```text partition-spec = [] schema-id = 0 ``` The actual headers are: ```text partition-spec = {"fields":[],"spec-id":0} schema-id = <absent> ``` The JSON `schema` header contains its own `schema-id`, but this does not supply the separately required Avro key-value metadata entry. In our four manifests, the embedded schema IDs were 0, 0, 1, and 2; the separate header was absent in every case. ## Inspect an affected manifest After creating an unpartitioned v2 table and appending a data file through iceberg-cpp, inspect the generated data manifest, not its manifest list: ```sh uv run --no-project --with fastavro python - /path/to/data-manifest.avro <<'PY' import json import sys from fastavro import reader with open(sys.argv[1], "rb") as stream: metadata = reader(stream).metadata print("partition-spec:", metadata["partition-spec"]) print("schema-id:", metadata.get("schema-id")) print("embedded schema-id:", json.loads(metadata["schema"])["schema-id"]) PY ``` Observed output for the first original manifest: ```text partition-spec: {"fields":[],"spec-id":0} schema-id: None embedded schema-id: 0 ``` ## Source of the defects The [v2 adapter's `Init()` implementation](https://github.com/apache/iceberg-cpp/blob/57ec6b8101952a47916ab9cd4328a17319b40195/src/iceberg/manifest/v2_metadata.cc#L76-L86) calls the general serializer for the entire `PartitionSpec`: ```cpp ICEBERG_ASSIGN_OR_RAISE(metadata_["schema"], ToJsonString(*current_schema_)) ICEBERG_ASSIGN_OR_RAISE(metadata_["partition-spec"], ToJsonString(*partition_spec_)); metadata_["partition-spec-id"] = std::to_string(partition_spec_->spec_id()); metadata_["format-version"] = "2"; ``` There is no assignment for `metadata_["schema-id"]`. The same serialization pattern appears in [`ManifestEntryAdapterV1::Init()`](https://github.com/apache/iceberg-cpp/blob/57ec6b8101952a47916ab9cd4328a17319b40195/src/iceberg/manifest/v1_metadata.cc) and [`ManifestEntryAdapterV3::Init()`](https://github.com/apache/iceberg-cpp/blob/57ec6b8101952a47916ab9cd4328a17319b40195/src/iceberg/manifest/v3_metadata.cc). ## Interoperability impact ClickHouse-local 26.7.1.1315 and 26.9.17.8 failed actual data scans with `Poco::Bad cast exception` while parsing the object-valued `partition-spec` header as a JSON array. This affected scalar projections such as `class_uid`, so it was not a nested-Parquet limitation. Optimized `count()` queries could still return metadata-derived counts and did not establish data readability. For diagnosis only, replacing the header with `[]` in disposable manifests and updating their recorded `manifest_length` in the manifest lists removed this parsing failure. Parsed manifest records and schemas remained unchanged. This was not a production repair or a source-code fix. PyIceberg tolerated the original headers and read the expected data. Its acceptance does not establish specification conformance. We did not isolate a reader failure attributable specifically to the missing `schema-id` header. ## Suggested fix and regression coverage Use manifest-specific serialization for the partition-fields array; retain the general partition-spec object serializer for REST and table metadata contexts where that shape is correct. Emit the separate schema ID from `current_schema_->schema_id()`. Add assertions against the actual written Avro headers for partitioned and unpartitioned manifests across supported format versions. Cover nonzero schema IDs as well as the initial schema, and check that v2/v3 manifests include the separately required `schema-id` entry. An independent-reader interoperability regression would help prevent permissive readers or writer/reader round trips from hiding these defects. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
