klsince opened a new pull request, #13992: URL: https://github.com/apache/pinot/pull/13992
The segments uploaded to table directly are not ONLINE in EV, so brokers can't include them in query routing table, but those segments are already processed by servers, thus invaliding docs in existing segments. If we don't let query access them, the query would see an incomplete upsert view. So we can mark them as optional (as we have done for consuming segment, which have similar race condition between server and broker), then those segments can be accessed by queries for a complete upsert view, even though broker doesn't add them into the list of selected segments for a query. This PR adds two new configs for the upsert view feature, `optionalSegmentsTTLMs` and `optionalSegmentCleanInternalMs` to allow one to add some delay when to unmask the segment. In theory, we can unmark a segment immediately if we know brokers have included it in the routing table. But there is no efficient and easy way for the server to know if brokers have included the optional segments in their routing table, so we use this TTL config to delay the unmarking as needed. It's a best effort mechanism here. The default TTL as 5min should be enough for brokers to include a segment into their routing table after the segment gets ONLINE in EV. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
