gaborkaszab commented on PR #16207: URL: https://github.com/apache/iceberg/pull/16207#issuecomment-4397888445
Hey @grantatspothero , I had a discussion with @nastra about the original intention of the REFS snapshot mode. It is slightly ambiguous in the REST spec for me (but whoever I asked understood it clearly :) ) that the intention is to have effect on the list of snapshots returned by the REST catalog and not on the snapshot log. This means kind of bad news for your approach to reduce memory footprint by keeping a reduced snapshot log, I'm afraid, because all the REST catalog servers that comply with the spec have to return a full snapshot log regardless of the snapshot mode. Now, this PR might still makes sense, IMO, if we want to be foolproof on the client side and not break internal state of the TableMetadata if a catalog returns a reduced snapshot log for REFS mode for some reason (maybe misinterpreting the spec), and then we load the snapshots lazily (but not the log). But then, we kind of guard against some behavior of another projects that in fact breaks the contract of the REST spec. I'm somewhat hesitant to do that, but lean to have this PR anyway. I'm curious of others opinions here. Note, we also discussed with @nastra that we probably want to reach out to Polaris to also fix this on their side, to return full snapshot log. Taking one step back, what I still don't entirely get is this: @grantatspothero you mentioned that you'd like to reduce memory pressure by not keeping entire snapshot log in REFS mode. Could you help me understand the full picture how you try to achieve that? This PR doesn't help you with that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
