sajjad-moradi commented on code in PR #10754:
URL: https://github.com/apache/pinot/pull/10754#discussion_r1195998765
##########
pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/retention/RetentionManagerTest.java:
##########
@@ -233,6 +233,39 @@ public void testRealtimeLLCCleanup()
verify(pinotHelixResourceManager, times(1)).deleteSegments(anyString(),
anyList());
}
+ // This test makes sure that we do not clean up last llc completed segments
+ @Test
+ public void testRealtimeLastLLCCleanup()
+ throws Exception {
+ final long now = System.currentTimeMillis();
+ final int replicaCount = 1;
+
+ TableConfig tableConfig = createRealtimeTableConfig1(replicaCount);
+ List<String> removedSegments = new ArrayList<>();
+ LeadControllerManager leadControllerManager =
mock(LeadControllerManager.class);
+ when(leadControllerManager.isLeaderForTable(anyString())).thenReturn(true);
+ PinotHelixResourceManager pinotHelixResourceManager =
+ setupSegmentMetadataForPausedTable(tableConfig, now, removedSegments);
+ setupPinotHelixResourceManager(tableConfig, removedSegments,
pinotHelixResourceManager, leadControllerManager);
+
+ ControllerConf conf = new ControllerConf();
+ ControllerMetrics controllerMetrics = new
ControllerMetrics(PinotMetricUtils.getPinotMetricsRegistry());
+ conf.setRetentionControllerFrequencyInSeconds(0);
+ conf.setDeletedSegmentsRetentionInDays(0);
+ RetentionManager retentionManager =
+ new RetentionManager(pinotHelixResourceManager, leadControllerManager,
conf, controllerMetrics);
+ retentionManager.start();
+ retentionManager.run();
+
+ SegmentDeletionManager deletionManager =
pinotHelixResourceManager.getSegmentDeletionManager();
+
+ // Verify that the removeAgedDeletedSegments() method in deletion manager
is actually called.
+ verify(deletionManager,
times(1)).removeAgedDeletedSegments(leadControllerManager);
+
+ // Verify that the deleteSegments method is actually called.
+ verify(pinotHelixResourceManager, times(1)).deleteSegments(anyString(),
anyList());
Review Comment:
```suggestion
String tableNameWithType = tableConfig.getTableName();
String deletedSegmentName = new LLCSegmentName("testTable", 5, 0,
now).getSegmentName();
verify(pinotHelixResourceManager,
times(1)).deleteSegments(tableNameWithType,
ImmutableList.of(deletedSegmentName));
```
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java:
##########
@@ -872,6 +874,24 @@ public List<SegmentZKMetadata>
getSegmentsZKMetadata(String tableNameWithType) {
return ZKMetadataProvider.getSegmentsZKMetadata(_propertyStore,
tableNameWithType);
}
+ public List<String> getLastLLCCompletedSegments(String tableNameWithType) {
+ Map<Integer, String> partitionIdToLastLLCCompletedSegmentMap = new
HashMap<>();
+ List<SegmentZKMetadata> realtimeSegments =
getSegmentsZKMetadata(tableNameWithType);
+ for (SegmentZKMetadata segMetadata : realtimeSegments) {
+ if
(SegmentName.isLowLevelConsumerSegmentName(segMetadata.getSegmentName())
+ && segMetadata.getStatus() ==
CommonConstants.Segment.Realtime.Status.DONE) {
+ LLCSegmentName llcName =
LLCSegmentName.of(segMetadata.getSegmentName());
+ int partitionGroupId = llcName.getPartitionGroupId();
+ int sequenceNumber = llcName.getSequenceNumber();
+ String curSegName =
partitionIdToLastLLCCompletedSegmentMap.get(partitionGroupId);
Review Comment:
Maybe `s/curSegName/lastCompletedSegName/`
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java:
##########
@@ -872,6 +874,24 @@ public List<SegmentZKMetadata>
getSegmentsZKMetadata(String tableNameWithType) {
return ZKMetadataProvider.getSegmentsZKMetadata(_propertyStore,
tableNameWithType);
}
+ public List<String> getLastLLCCompletedSegments(String tableNameWithType) {
+ Map<Integer, String> partitionIdToLastLLCCompletedSegmentMap = new
HashMap<>();
+ List<SegmentZKMetadata> realtimeSegments =
getSegmentsZKMetadata(tableNameWithType);
+ for (SegmentZKMetadata segMetadata : realtimeSegments) {
Review Comment:
```suggestion
for (SegmentZKMetadata segMetadata :
getSegmentsZKMetadata(tableNameWithType)) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
