jtao15 commented on issue #7413: URL: https://github.com/apache/pinot/issues/7413#issuecomment-917294620
> I think the key to solve this problem is to keep the lineage entry longer and only remove it when the segment is already removed in the previous round of cleanup. Combine this with segment existence check should be good enough to solve this race condition In extreme case: 1. Segment A is picked for merge/roll-up 2. Before the merge/roll-up task is done, A is picked for purging 3. merge task is finished 4. retention manager removed segment A 5. after 6 hours, retention manager removed the lineage 6. Purge task is done, the purged segment is uploaded and we have duplicate data So if the purge task run longer than 6 hour, we still have the issue? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
