technicolorbeat opened a new issue, #1946: URL: https://github.com/apache/iceberg-go/issues/1946
### Proposed Change When a client commits changes via `Transaction.Commit()`, the REST server returns a `commitTableResponse` containing the full table metadata, including all snapshots. Today, iceberg-go eagerly decodes all snapshots into memory even though most callers only need the `CurrentSnapshot`. For tables with long snapshot histories (1000+ snapshots), this adds 50-100ms of latency to `AddFiles` operations—the decode cost is paid even when the returned `*Table` is only used to verify the commit succeeded. **Proposed change:** - When `snapshot-loading-mode=refs` is set on the REST catalog, defer snapshot decoding in `commitTableResponse` - Store the raw snapshots JSON (`json.RawMessage`) without parsing - Only eagerly decode the snapshot matching `current-snapshot-id` - Lazily decode remaining snapshots on first access to `Snapshots()` or `SnapshotByID()` using `sync.Once` - Use standard Go patterns for thread-safe lazy initialization **Result:** - Most callers (commit-and-exit pattern): snapshot decode is skipped entirely → ~50-100ms latency improvement - Callers needing snapshots: cost is deferred to first access, paid transparently with no API changes - No breaking changes; backward compatible; can be feature-gated **Backward compatibility:** - Default behavior unchanged (no lazy decode unless `snapshot-loading-mode=refs`) - API surface unchanged (`*Table` interface identical) - Full snapshot list returned on access (no silent truncation like rejected Phase 1 proposal) **Related:** - #1748 — added `snapshot-loading-mode` catalog property - #1749 — implemented `?snapshots=refs` on `LoadTable` (merged) - #1792 — lazy-load unreferenced snapshots on `LoadTable` (orthogonal, independent) **Spec reference:** No spec change needed. This is a client-side optimization (how iceberg-go decodes the response), not a protocol change. **Willingness to contribute** - [x] I can contribute this improvement/feature independently - [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community - [ ] I cannot contribute this improvement/feature at this time ### Proposal document _Not applicable — client-side optimization._ ### Specifications - [x] Table - [ ] View - [x] REST - [ ] Puffin - [ ] Encryption - [ ] Other -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
