snleee opened a new pull request, #12346: URL: https://github.com/apache/pinot/pull/12346
The current code can face the race condition when the following condition is met: 1. Segment upload is complete for segment replacement and segment zk metadata is changed to point the new segment. 2. Server is still loading the segment (e.g. if offheap upsert is enabled, segment reload can take a while) 3. If minion picks up the compaction task at the above step, minion will download the new segment (replaced) while it will fetch the validDocId bitmap for old segment. To avoid this scenario, we added the extra crc check on the minion side. Minion will check if the crc from downloaded segment and crc from server api for valid doc id bitmap are the same. Changes from the PR: 1. Add the new API for fetching validDocId bitmap. the response is changed to JSON to include extra information like crc. 2. Wired the new API & added tests 3. Added the crc check on the minion -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
