amarnathkarthik commented on issue #6349: URL: https://github.com/apache/incubator-pinot/issues/6349#issuecomment-749209495
Here is my analysis, Looked at `LaunchDataIngestionJobCommand.java` and it's all job runner (Segment Generator and Push job). Did not see implementation to clean up `outputDirURI` after the push but do see tempDir cleanup after copying the generated segments to outputDirURI. @kishoreg @plaisted We have 2 option, let me know which one would be appropriate: 1. **Backward compatible** - Introduce new property in `SegmentGenerationJobSpec._cleanupOutputAfterPush` to remove segments from `outputDirURI` once after push call finishes successfully. CleanupOutputAfterPush can be configurable using yaml and default it to `false` 2. **Cleanup by default** - Enhance the push job to remove the segments from the outputDirURI once after push call finishes successfully. Note: Both Spark and Hadoop push jobs does not support cleanup today. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
