yupeng9 commented on issue #6302: URL: https://github.com/apache/pinot/issues/6302#issuecomment-893694952
> > e ingestion. Otherwise the upsert metadata will hit a race condition. Though it's possible to implement this pause internally, > > Thanks, yupeng. I need to understand this better. Is this a race condition in pinot? If so, we should fix it regardless? > > Let us discuss a little bit on this. I will reach out. and then we can summarize our discussion here. You can think of it this way: When the segment in the history is modified, we need to scan the history to derive the latest upsert state per primary key, especially for the partial upsert. This scan cannot proceed when the real-time consumption is on, as we dont have the latest metadata info to handle the real-time update. The way to solve it is to put a lock and block the real-time ingestion, until the load/scan is completed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
