klsince opened a new pull request, #11978: URL: https://github.com/apache/pinot/pull/11978
This PR tries to address issue: https://github.com/apache/pinot/issues/11965, basically by extending the NewSegment concept added for instanceSelector early on. The NewSegment concept was added to handle the case when new consuming segments are added in IdealState but not online in ExternalView yet, as servers may take time to load new consuming segments and report the status back to Helix. Instead of reporting such new segments as unavailable segments, broker simply skips them. But simply skipping them at broker side could cause the wrong query results as detailed in the new issue. e.g. for upsert table, the server could start to ingest new records into the new segments and invalidate the records in existing segments, even before the broker could mark those new consuming segments as online, thus queries could not see the new records in new consuming segments during a short window. This issue could cause inconsistent query results, particularly for upsert table. So in this PR, we track those new segments that not online in ExternalView, and pass them to servers, instead of skipping them at broker side. The servers can skip them if they are not available, otherwise, process they for correct query results, particular for upsert table. TODO: only support for SingleConnectionBrokerRequestHandler in this PR. Probably add support for GrpcServerRequest (based on protobuf) and multi-stage engine in a separate PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
