jtao15 commented on issue #7413: URL: https://github.com/apache/pinot/issues/7413#issuecomment-917290051
> Does it make sense to have some sort of segment level locks so that the same set of segments are not picked up by multiple tasks? If we can come up with a lock-free algorithms on this, that will be even better. > > Alternatively, some config value of not doing merge/rollup if the retention is just N hours/days away (if this is the only race condition). The segment zk metadata version will achieve the segment level lock? If we always check the version when replacing/deleting the segments, only one task will succeed. The other replacing/deleting request can retry later. I think scheduling part will not cause any issue, but the clean up part of merge/roll-up does. By adding merge/roll-up task, the retention manager now need to clean up merged segment to free the disk space. Say we merge A and B to C. After merging is done, the retention manager will remove A and B (even if they didn't pass the retention threshold, they should be removed since they are not serving queries anymore). If A and B are scheduled with purge task, the race condition can happen. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
