swaminathanmanish commented on code in PR #16719:
URL: https://github.com/apache/pinot/pull/16719#discussion_r2329316789
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/retention/RetentionManager.java:
##########
@@ -483,4 +495,21 @@ private void
manageSegmentLineageCleanupForTable(TableConfig tableConfig) {
}
LOGGER.info("Segment lineage metadata clean-up is successfully processed
for table: {}", tableNameWithType);
}
+
+ private RetentionStrategy createUntrackedSegmentsRetentionStrategy(
+ SegmentsValidationAndRetentionConfig validationConfig, String
tableNameWithType) {
+ if (validationConfig.getUntrackedSegmentsRetentionTimeUnit() != null
+ && validationConfig.getUntrackedSegmentsRetentionTimeValue() != null) {
+ try {
+ return new TimeRetentionStrategy(
+
TimeUnit.valueOf(validationConfig.getUntrackedSegmentsRetentionTimeUnit().toUpperCase()),
+
Long.parseLong(validationConfig.getUntrackedSegmentsRetentionTimeValue()));
+ } catch (Exception e) {
+ LOGGER.warn("Invalid untracked segments retention time: {} {} for
table: {}, using default 3 days",
+ validationConfig.getUntrackedSegmentsRetentionTimeUnit(),
+ validationConfig.getUntrackedSegmentsRetentionTimeValue(),
tableNameWithType, e);
+ }
+ }
+ return new TimeRetentionStrategy(TimeUnit.DAYS, 3);
Review Comment:
Also please add comments on why are doing this and reason for 3d as default
for orphan segments.
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/retention/RetentionManager.java:
##########
@@ -483,4 +495,21 @@ private void
manageSegmentLineageCleanupForTable(TableConfig tableConfig) {
}
LOGGER.info("Segment lineage metadata clean-up is successfully processed
for table: {}", tableNameWithType);
}
+
+ private RetentionStrategy createUntrackedSegmentsRetentionStrategy(
+ SegmentsValidationAndRetentionConfig validationConfig, String
tableNameWithType) {
+ if (validationConfig.getUntrackedSegmentsRetentionTimeUnit() != null
+ && validationConfig.getUntrackedSegmentsRetentionTimeValue() != null) {
+ try {
+ return new TimeRetentionStrategy(
+
TimeUnit.valueOf(validationConfig.getUntrackedSegmentsRetentionTimeUnit().toUpperCase()),
+
Long.parseLong(validationConfig.getUntrackedSegmentsRetentionTimeValue()));
+ } catch (Exception e) {
+ LOGGER.warn("Invalid untracked segments retention time: {} {} for
table: {}, using default 3 days",
+ validationConfig.getUntrackedSegmentsRetentionTimeUnit(),
+ validationConfig.getUntrackedSegmentsRetentionTimeValue(),
tableNameWithType, e);
+ }
+ }
+ return new TimeRetentionStrategy(TimeUnit.DAYS, 3);
Review Comment:
Can you add this constant to the same place where we have other constants
for retention manager. It'll be easier to spot.
Thanks for adding validation for configs to address Abhishek's concern
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
