technicolorbeat opened a new pull request, #1981: URL: https://github.com/apache/iceberg-go/pull/1981
Closes #1979. ## Motivation `table.ParseMetadataBytes` is on the critical path for table loads, refreshes, and metadata returned by catalog operations. Today it performs four complete passes over every metadata document: 1. decode `format-version`; 2. decode the full document into raw-message maps to assign legacy partition field IDs; 3. decode the full document again to require partition spec IDs; and 4. decode the concrete v1/v2/v3 metadata object. The raw-message map passes also copy top-level values. For tables with long snapshot histories or otherwise large metadata, this means repeatedly scanning and transiently copying several MiB before callers can use the table. That adds load/refresh latency, reduces catalog-client throughput, increases peak memory, and creates avoidable GC work. The cost is paid by every caller even when the document has no legacy partition fields to normalize. This change reduces the pipeline to one structural scan followed by the existing version-specific `encoding/json` decode. Users with large metadata should see lower table-load latency and substantially lower transient allocation, without an API or serialized-format change. ## Design The new scanner is deliberately narrow: it understands JSON structure but only records byte spans for fields needed before the concrete metadata decode: - `format-version` - `last-updated-ms` - `last-partition-id` - `partition-spec` - `partition-specs` During the structural pass it validates objects, arrays, strings and escapes, literals, JSON number grammar, trailing input, and nesting depth. Top-level keys are decoded with `encoding/json`, so escaped field names and duplicate-key last-value behavior match the standard decoder. After the scan: - `format-version` is decoded from its recorded span and selects v1/v2/v3; - required `last-updated-ms` presence/null behavior is checked synchronously; - only the selected partition-spec subdocument is decoded to validate required spec IDs and find legacy fields missing `field-id`; - when normalization is unnecessary, the original byte slice goes directly to the concrete decoder; - when normalization is necessary, only the partition-spec value and `last-partition-id` spans are replaced. Unrelated JSON is neither decoded into raw-message maps nor re-marshaled. The final metadata construction and all version-specific validation remain with the existing `encoding/json` implementations. The scanner does not retain aliases into caller-owned input and does not replace the standard library as the semantic decoder. This also establishes reusable top-level range discovery for future deferred-snapshot work, while keeping this PR independent of that work. ## Benchmarks The new benchmark is organized by actual serialized metadata size and reports that byte count in both the benchmark name and a `metadata-bytes` metric. It covers two document shapes: - `snapshot-history`: size comes from many nested snapshot objects; - `properties`: size comes from non-snapshot top-level metadata. Command: ```console go test ./table -run '^$' -bench '^BenchmarkParseMetadataBytes$' -benchmem -benchtime=3x ``` Apple M1 Max, darwin/arm64; three iterations per case. These short runs are directional, but the improvement is consistent across every size/profile. | Profile | Serialized bytes | Before ns/op | After ns/op | Time | Before B/op | After B/op | Bytes | |---|---:|---:|---:|---:|---:|---:|---:| | snapshot-history | 102,133 | 4,905,375 | 2,963,153 | -39.6% | 793,170 | 573,136 | -27.7% | | snapshot-history | 511,718 | 25,274,111 | 14,896,514 | -41.1% | 3,760,160 | 2,720,962 | -27.6% | | snapshot-history | 1,048,451 | 49,959,236 | 30,726,986 | -38.5% | 7,702,213 | 5,598,098 | -27.3% | | snapshot-history | 2,096,888 | 99,760,819 | 61,165,819 | -38.7% | 16,127,200 | 11,925,328 | -26.1% | | snapshot-history | 5,242,839 | 249,408,167 | 157,828,625 | -36.7% | 39,873,570 | 29,380,248 | -26.3% | | properties | 102,249 | 3,109,222 | 1,445,083 | -53.5% | 389,528 | 168,560 | -56.7% | | properties | 511,932 | 15,133,486 | 6,665,986 | -56.0% | 1,894,344 | 881,424 | -53.5% | | properties | 1,048,481 | 31,195,236 | 13,700,958 | -56.1% | 3,845,632 | 1,767,360 | -54.0% | | properties | 2,096,926 | 61,931,180 | 27,713,708 | -55.3% | 7,716,272 | 3,486,669 | -54.8% | | properties | 5,242,815 | 153,879,056 | 68,523,875 | -55.5% | 18,591,498 | 8,096,813 | -56.4% | Allocations/op also decrease slightly for snapshot-heavy documents and by roughly one allocation per property entry for property-heavy documents because the full-document raw-message maps are gone. ## Correctness coverage Tests cover: - v1, v2, and v3 through the existing metadata suite; - legacy missing partition field-ID assignment across specs; - missing/null partition spec IDs; - malformed nested JSON, invalid escapes and number grammar, and trailing input; - escaped top-level keys; - duplicate selected fields using the last value; - preservation of unrelated JSON during localized normalization; - top-level `null` format-version error compatibility; - unknown-field compatibility and existing version-specific constraints through the full repository suite. ## Validation ```console go test -p 2 ./... golangci-lint run --timeout=10m ./table/... ``` Both pass locally. (`-p 2` was used only to limit peak linker disk usage.) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
