moomindani commented on PR #6: URL: https://github.com/apache/iceberg-verification/pull/6#issuecomment-5737120640
Exercised all three tables from the consumer side with PyIceberg `main` (`0d584073`), via the symlink route. **All three load and scan correctly.** `v1/partitioned-parquet-data` decodes its spec as `identity(2)` + `day(3)` and returns 3 rows; `v2/nested-types-parquet-data` decodes the struct's child ids 5 and 6 with the list and map nulls intact; `v2/simple-append-parquet-data` returns 4 rows with the one null. Sizes measured 16161, 25967 and 28047 bytes, matching the 16–28 KB in the README. `index.json` resolved all three metadata paths as documented. **One correction to the Paths section.** The substitution route does not work for PyIceberg as described, because the absolute prefix is not only in `metadata.json`: the manifest lists carry `manifest_path` and the manifests carry `file_path`, both absolute (`file:/tmp/iceberg-verification/...`). Measured: - No symlink, no substitution: `StaticTable.from_metadata` **succeeds** and the scan then fails on the manifest list. - Prefix substituted in `metadata.json` only: the load succeeds, `location` is the new prefix, and the scan fails one level deeper, on the `manifest_path` inside the manifest-list Avro. PyIceberg has no load-time path-rewriting hook — `StaticTable.from_metadata` takes a metadata path and properties and nothing else — so on Windows the honest recipe is "copy the corpus and rewrite the prefix in the metadata JSON *and* in the manifest lists and manifests", which is a different operation from substituting while loading. Worth saying that explicitly, since the two routes are currently presented as equally cheap. **Related, and worth a line in the procedure:** a broken path setup surfaces at scan, not at load. A surface that asserts only on decoded metadata passes with no symlink at all, so a consumer can report green on a corpus it never actually opened. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
