mayankshriv commented on pull request #5745: URL: https://github.com/apache/incubator-pinot/pull/5745#issuecomment-663326776
> @mayankshriv, thanks for reviewing. See answers inline below. > > > 1. The first offline segment should immediately push the high-water-mark to (maxTime-1). Are you referring to a case with single segment push per day? > > Yes, this is for cases where there is a single segment push per day. > > > 1. Your requirement seems contrary to the other requests we are getting, where the high-water-mark being pushed up pre-maturely (before all segments from the new date are ONLINE) causes inconsistent results during the push time. And we are getting multiple requests to address the inconsistency by deferring the push of high-water-mark until the entire push is complete. > > This sounds like the current supported case. In our case, we have a Hybrid table where we serve current day metrics from realtime and once per day upload an offline segment. The issue is, since we only upload one segment per day, the latest offline segment isn't used until the next day's data is pushed. > > Today we get around this by creating "dummy" segments with empty rows and +1 day as a way of forcing the time value to increment further and serve the latest actual offline segment. Unfortunately, this won't work for us any longer and we need a better long-term solution. > > I hope this clarifies, but let me know if you have any further questions. I see, the single segment per day clarifies the confusion. There's also a new api being added in the controller that makes it a protocol. Perhaps you could piggy-back on it to update time-boundary? @snleee What do you think? https://github.com/apache/incubator-pinot/pull/5712 ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
