jtao15 opened a new issue #7413: URL: https://github.com/apache/pinot/issues/7413
There's a potential race condition issue when we run merge/roll-up task & segment upload tasks (e.g. purge, convertToRawIndex, etc) on the same table (purge task for example): 1. Segment A is picked both by merge/roll-up and purge tasks 2. Merge/roll-up task is finished, and the retention manager tries to clean up segment A and the lineage entry before purge task uploading the purged segment. 3. Purge task uploaded the segment, and we have duplicate data for the table. There are two potential improvements: 1. Honor segment lineage when scheduling purge tasks to avoid picking segments that will be removed. 2. Existence check + ZK version check when uploading segments by purge tasks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
