avshenuk commented on issue #16359: URL: https://github.com/apache/pinot/issues/16359#issuecomment-3081407062
> Upsert metadata can only be applied to real-time table, but not offline table. Is the idea to only move the real-time segment to offline when it is beyond the upsert metadata TTL, which indicates that there is no more change allowed to the segment? Yes, conceptually, the idea is to have the real-time table allowing upsets within a certain time window. So that when the segments are outside that window, you know they’re finalized and ready for the offline table. I feel like it matches exactly the idea of such hybrid setups - your real-time tables cover a certain time period allowing any kind of updates on your data. Then once the threshold is crossed - your data becomes «historical» and can now be sealed. Haven’t thought about using upset metadata TTL exactly, as it can work without it: as long as the user aligns the buffer time window with their own data upsert flows. But yes, it’s good point, we could use it in place of the buffer for maximum consistency? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
