fx19880617 commented on a change in pull request #6385:
URL: https://github.com/apache/incubator-pinot/pull/6385#discussion_r548912564
##########
File path:
pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-common/src/main/java/org/apache/pinot/plugin/ingestion/batch/common/SegmentPushUtils.java
##########
@@ -134,6 +135,8 @@ public static void pushSegments(SegmentGenerationJobSpec
spec, PinotFS fileSyste
segmentName, controllerURI, e);
throw e;
}
+ } finally {
+ FileUtils.deleteQuietly(tarFile);
Review comment:
this one assumes the tar file is on local disk, which may not be true
always. E.g. the tar file could be on S3.
The path is `s3://<my-bucket>/<my-tar>.tar.gz`
In this case, you need to use PinotFs to delete the tar file.
##########
File path:
pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-common/src/main/java/org/apache/pinot/plugin/ingestion/batch/common/SegmentPushUtils.java
##########
@@ -183,6 +187,8 @@ public static void sendSegmentUris(SegmentGenerationJobSpec
spec, List<String> s
tableName, segmentUri, controllerURI, e);
throw e;
}
+ } finally {
+ FileUtils.deleteQuietly(segmentPath);
Review comment:
Same above.
##########
File path:
pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-common/src/main/java/org/apache/pinot/plugin/ingestion/batch/common/SegmentPushUtils.java
##########
@@ -134,6 +135,8 @@ public static void pushSegments(SegmentGenerationJobSpec
spec, PinotFS fileSyste
segmentName, controllerURI, e);
throw e;
}
+ } finally {
+ FileUtils.deleteQuietly(tarFile);
Review comment:
Please use flag to turn on/off this deletion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
