jtao15 opened a new issue #8423: URL: https://github.com/apache/pinot/issues/8423
Current code does the followings: 1. `tableDataManager` uses `segmentDataManagerMap` to track the segments. `tableDataManager` is created and initialized when adding the first segment to the server, and shutdown if the last segment is removed. 2. The `tmp` directory for segment download will be removed and recreated as part of `tableDataManager` initialization. 3. When adding a new segment to server, the server will download the segment first and update the segmentDataManagerMap after. This can cause two issues: 1. If the last segment is removed from `segmentDataManagerMap` during new segment download time and before updating `segmentDataManagerMap`, the `tableDataManager` will be shutdown, and the new segment will become unreachable. We will have partial query results with `SERVER_SEGMENT_MISSING_ERROR`. 2. If `tableDataManager` is initialized during the segment download in scenario 1, the downloaded tar file will be removed. The server will take it as download failure, and the segment will stay in `ERROR` state until fixed manually by restarting the server. We have observed this issue recently after #7961 was merged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
