jey1987-cmd commented on issue #1202: URL: https://github.com/apache/iceberg-python/issues/1202#issuecomment-5705779840
Adding a data point that's directly relevant to both requests above (efficient commit-retry without re-saving data files, and catalog call caching). I ran a series of controlled experiments on a synthetic three-writer workload against PyIceberg with a BigLake REST catalog (Google Cloud), specifically studying commit conflict behavior and retry cost under concurrent writers, across three different timing architectures: fixed-interval scripted writers, autonomous LLM-agent-driven writers, and a fully event-driven (Pub/Sub) streaming pipeline. A few findings that seem relevant to what's being asked for here: **On request 1 (efficient retry without re-saving data files):** In every configuration I tested, a write role that revisits and rewrites already-committed rows (rather than appending) showed retry cost dominated almost entirely by commit-side overhead, not data-file I/O. Under three-way concurrent contention, this role's failure rate after 5 retries ranged from 14% to 86% depending on the timing architecture, with retry counts averaging 2-6 per attempt and total resolution time per attempt reaching into the tens of seconds to multiple minutes under sustained contention. Since the underlying data being written was small and unchanged across retries, the cost was structurally a commit/metadata-negotiation cost, not a data-serialization cost -- exactly consistent with the motivation for this feature request. **On request 2 (catalog call caching):** Across all three studies, even zero-retry, uncontended commits showed a baseline latency of roughly 5-32+ seconds, scaling with the size of the target table (Pearson correlation between table row count and first-attempt commit latency: r > 0.96, confirmed across four independently replicated runs spanning both scripted and agent-driven timing, and directionally consistent, though at a smaller sample, under genuinely event-driven Pub/Sub timing). This baseline cost appears attributable to REST catalog round-trip overhead rather than contention, since it was present even in single-writer, zero-conflict conditions. Reducing the number of network calls to the catalog (as proposed here) would likely address this baseline cost component directly, separately from anything related to conflict retries themselves. **Scope/caveats:** this was a synthetic workload (not production traffic), run against BigLake specifically rather than AWS Glue or another REST catalog implementation, and some of the streaming-specific numbers above rest on a small sample (n=4) given how infrequently the high-conflict role could complete an attempt under sustained load -- I'd treat those as directionally supportive rather than a precise characterization. Full experimental setup, corpus generator, and analysis queries are at https://github.com/jey1987-cmd/IcebergConcurrentWriteStudy if useful. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
