technicolorbeat opened a new issue, #1979: URL: https://github.com/apache/iceberg-go/issues/1979
### Description `table.ParseMetadataBytes` currently makes multiple complete passes over the metadata JSON before returning a concrete metadata object: 1. `assignMissingPartitionFieldIDs` decodes the full document into raw-message maps to normalize legacy partition field IDs. 2. `requirePartitionSpecIDs` decodes the full document again to validate partition spec IDs. 3. Format-version detection via `json.Unmarshal` scans the full input even though it needs one top-level field. 4. The version-specific metadata decoder scans the full input again. The deferred-snapshot work discussed in #1946 additionally captures and validates raw snapshots. Avoiding full historical snapshot materialization reduces allocations, but repeated full-document scans and copies can still consume substantial CPU for large metadata. This issue proposes a general `ParseMetadataBytes` optimization, separate from #1946, because it should benefit eager parsing and all callers rather than only REST commit responses. ### Proposed direction Introduce a single top-level structural scan that can: - identify and validate `format-version`; - establish that required fields such as `last-updated-ms` are present with the expected JSON shape; - record byte ranges for `partition-spec`, `partition-specs`, `snapshots`, and `snapshot-log`; - detect the partition-spec fields that require legacy field-ID normalization or spec-ID validation; - allow the ordinary version-specific decoder to process a document with a large snapshot array removed or replaced by an empty array when deferred snapshot parsing is requested; - validate and normalize only the relevant partition-spec subdocuments instead of decoding the complete metadata document into raw-message maps repeatedly; and - retain raw ranges safely by copying owned bytes rather than keeping aliases into caller-owned input. The exact implementation does not need to be a hand-written general JSON decoder. A small structural scanner combined with the standard library decoder may be sufficient and easier to maintain. ### Correctness requirements - Preserve support and validation semantics for metadata format versions v1, v2, and v3. - Preserve legacy missing partition field-ID assignment and required partition spec-ID behavior. - Preserve precise, synchronous errors for malformed JSON, missing required fields, invalid field types, overflow, duplicates, and version-specific constraints. - Preserve unknown-field compatibility and current `null`/absent-field behavior. - Preserve input ownership: any JSON retained after parsing must be copied before `ParseMetadataBytes` returns. - Preserve serialization, equality, builders, version upgrades, and concurrent-read behavior. - Do not change public API behavior or move existing parse-time errors to later accessors. - Prefer the Go standard library unless a dependency has a clearly demonstrated, compatible, and safe advantage. ### Benchmarks and profiling Add benchmarks organized by actual serialized metadata byte size, not only snapshot count. Suggested size points include approximately 100 KiB, 500 KiB, 1 MiB, 2 MiB, and 5 MiB, with the exact generated byte count reported. For each relevant case, compare: - current eager parsing; - optimized eager parsing; - deferred-snapshot parsing from #1946, when available; and - deferred parsing plus current-snapshot access versus a historical lookup/full materialization where useful. Report `ns/op`, `B/op`, and `allocs/op`, and collect CPU/allocation profiles to establish which repeated passes materially affect runtime. Fixtures should distinguish snapshot-heavy metadata from metadata whose size comes from schemas, partition specs, properties, or snapshot logs. PR #1654 added a reusable load-table response JSON benchmark and evaluated `goccy/go-json`. Its reported results were mixed across snapshot counts, and the proposed replacement also raised encoder compatibility and concurrency-safety concerns, so a decoder replacement was not merged. A future standard-library JSON implementation or safe decoder evaluation is orthogonal to this issue: a faster decoder may reduce the cost of each pass, while this issue aims to eliminate unnecessary passes altogether. Related: #1946 Related benchmark work: #1654 ### Are you willing to contribute this feature? - [x] Yes, I am willing to contribute this feature. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
