adutra commented on PR #1844: URL: https://github.com/apache/polaris/pull/1844#issuecomment-2983941280
> From yesterday's discussion: We should go with the minimal viable change. There are strong concerns about the "buffering/spilling" introduced here. > > Query patterns & event payload, considering the Iceberg proposal, are still not set in stone - I have mentioned concerns about the IRC proposal as it stands. Considering especially the event payload size issues in that proposal, I think it is way too early to push this one. +1 to the above. Also I would like to stress that the need for a commitlog-like structure on disk stems from the design choice to isolate events persistence from catalog persistence. If all writes (catalog and events) were done within the same transaction, we would be leveraging the transactional backend to do the heavy lifting of maintaining those two consistent with each other. And we would get "exactly once" semantics for free. For this reason, I think we need to take a step back and rethink the persistence model. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
