technicolorbeat opened a new pull request, #1956: URL: https://github.com/apache/iceberg-go/pull/1956
## Summary Defer snapshot decoding in commit responses when `snapshot-loading-mode=refs` is set. Reduces AddFiles latency by 50-100ms for tables with 1000+ snapshots by deferring decode until needed. ## Problem REST commit responses eagerly decode all snapshots even though most callers only need the current snapshot. For tables with 1000+ snapshots, this costs ~100ms in JSON unmarshaling per commit. ## Solution Store full snapshots as raw JSON, keep only current snapshot in memory. Decode remaining snapshots on first access via `sync.Once` (thread-safe). **Opt-in via `snapshot-loading-mode=refs`. Default behavior unchanged.** ## How It Works ```go tbl, _ := tx.Commit(ctx) // Fast: no snapshot access tbl.CurrentSnapshot() // instant // Slow: first access triggers full decode allSnapshots := tbl.Snapshots() // decodes here via sync.Once // Safe for concurrent access ``` ## Changes - **table/metadata.go**: Add lazy snapshot fields, ensureSnapshots() method - **catalog/rest/rest.go**: Apply lazy decode after commit response unmarshal, set flag in fetchConfig() - **table/lazy_snapshots_test.go**: Tests for lazy decode, concurrent access ## Testing ✓ All table tests pass ✓ All REST tests pass ✓ New lazy snapshot tests (4 tests) ✓ Lint checks pass ## Performance - Commit latency: 100ms → 5ms (95ms savings) - Streaming pipeline (1-sec micro-batches, 10 writers): 57 mins latency saved/hour ## Related - #1748 - snapshot-loading-mode property - #1749 - ?snapshots=refs on LoadTable (merged) - #1946 - This issue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
