sajjad-moradi opened a new issue #6966: URL: https://github.com/apache/incubator-pinot/issues/6966
The changes introduced in this [PR](https://github.com/apache/incubator-pinot/pull/6567/files#diff-d0a60155908b7919915de944c49dfad57e2e5e1cc7ad2988176ac090e57567d7R299) caused the following problem for Refresh use cases in LinkedIn: - The segment directory in deep store is changed from `tableName/segmentName` to `tableName_OFFLINE/segmentName`. This means that for refresh use cases, the new segments will go to `tableName_OFFLINE` directory, but the older version of the same segment will remain undeleted in `tableName` directory. This is problematic because the older versions need to be manually deleted and now they need extra storage. Also by just looking at the code, we believe SegmentDeletionManager cannot function properly as it uses rawTableName to construct the uri for the segments to be deleted (details [here](https://github.com/apache/incubator-pinot/blob/master/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/SegmentDeletionManager.java#L179)). Please note that with current code, here's the state of the deep store: - the realtime segments pushed from RT servers use a separate API and then finally reside in `tableName/segmentName` directory. - the realtime segments for upsert use cases will land in `tableName_REALTIME/segmentName` directory. - the offline segments will land in `tableName_OFFLINE/segmentName` directory. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
