koodin9 commented on PR #18003: URL: https://github.com/apache/iceberg/pull/18003#issuecomment-6599727212
@pvary > What are the expectation of the time for the commits? Well below the commit interval. Our target is about 3 minutes from a change to its visibility. With a commit interval of 60 to 90 s, the conversion should take at most about 80 s, or about 40 s so that one retry still fits. > Is a worst case full read acceptable? Not on large tables, I think. When the changes are spread over the key space, every commit reads the key column of the whole table. Our two largest CDC tables are unpartitioned, with about 150 to 160 data files and 1.7 and 2.4 billion rows. Amoro already reads all of these files after each commit to turn the equality deletes into position deletes. It runs one task per data file in parallel, and still takes 14 s and 43 s. I haven't measured the conversion on our storage yet, but doing that in one coordinator on every commit is too expensive. So I'd document in part 2 that the conversion suits tables where the key filter or the partition spec prunes most data files, such as increasing keys with changes on recent rows. Tables with changes spread over many rows need an index behind `KeyPositionResolver`, as a follow-up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
