mcvsubbu commented on a change in pull request #8176:
URL: https://github.com/apache/pinot/pull/8176#discussion_r806249898
##########
File path:
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java
##########
@@ -703,7 +703,9 @@ public synchronized PinotResourceManagerResponse
deleteSegments(String tableName
Preconditions.checkArgument(TableNameBuilder.isTableResource(tableNameWithType),
"Table name: %s is not a valid table name with type suffix",
tableNameWithType);
HelixHelper.removeSegmentsFromIdealState(_helixZkManager,
tableNameWithType, segmentNames);
- _segmentDeletionManager.deleteSegments(tableNameWithType, segmentNames);
+ TableConfig tableConfig =
ZKMetadataProvider.getTableConfig(_propertyStore, tableNameWithType);
+ long retentionMs =
_segmentDeletionManager.getRetentionMsFromTableConfig(tableConfig);
Review comment:
I have a question here. why call an API in `SegmentDeletionManager` to
get the retention period, and then use that same value to call back another API
in `SegmentDeletionManager`? Why not call the deletion API with the (posilby
null) table config and let SegmentDeletionManager figure out whatever is
needed from it?
##########
File path:
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/SegmentDeletionManager.java
##########
@@ -53,22 +59,34 @@
private static final Logger LOGGER =
LoggerFactory.getLogger(SegmentDeletionManager.class);
private static final long MAX_DELETION_DELAY_SECONDS = 300L; // Maximum of
5 minutes back-off to retry the deletion
private static final long DEFAULT_DELETION_DELAY_SECONDS = 2L;
+
+ // Retention date format will be written as suffix to deleted segments under
`Deleted_Segments` folder. for example:
+ //
`Deleted_Segments/myTable/myTable_mySegment_0__RETENTION_UNTIL__20220202_120000`
to indicate that this segment
+ // file will be permanently deleted after Feb 2nd 2022 12PM.
private static final String DELETED_SEGMENTS = "Deleted_Segments";
+ private static final String RETENTION_UNTIL_SEPARATOR =
"__RETENTION_UNTIL__";
+ private static final String RETENTION_DATE_FORMAT_STR = "yyyyMMdd_HHmmss";
Review comment:
Do we want up to seconds here? I suggest using the same time format as
we did for LLC segments: `yyyyMMddHHmmZ`
##########
File path:
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/SegmentDeletionManager.java
##########
@@ -53,22 +59,34 @@
private static final Logger LOGGER =
LoggerFactory.getLogger(SegmentDeletionManager.class);
private static final long MAX_DELETION_DELAY_SECONDS = 300L; // Maximum of
5 minutes back-off to retry the deletion
private static final long DEFAULT_DELETION_DELAY_SECONDS = 2L;
+
+ // Retention date format will be written as suffix to deleted segments under
`Deleted_Segments` folder. for example:
+ //
`Deleted_Segments/myTable/myTable_mySegment_0__RETENTION_UNTIL__20220202_120000`
to indicate that this segment
+ // file will be permanently deleted after Feb 2nd 2022 12PM.
private static final String DELETED_SEGMENTS = "Deleted_Segments";
+ private static final String RETENTION_UNTIL_SEPARATOR =
"__RETENTION_UNTIL__";
+ private static final String RETENTION_DATE_FORMAT_STR = "yyyyMMdd_HHmmss";
+ private static final SimpleDateFormat RETENTION_DATE_FORMAT;
+
+ static {
+ RETENTION_DATE_FORMAT = new SimpleDateFormat(RETENTION_DATE_FORMAT_STR,
Locale.getDefault());
Review comment:
Change this to UTC. We don't want locale based time stamps floating
around.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
