xiangfu0 commented on code in PR #16361:
URL: https://github.com/apache/pinot/pull/16361#discussion_r2442590135
##########
pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/realtimetoofflinesegments/RealtimeToOfflineSegmentsTaskGenerator.java:
##########
@@ -354,4 +399,80 @@ public void validateTaskConfigs(TableConfig tableConfig,
Schema schema, Map<Stri
}
}
}
+
+ private Map<String, List<ValidDocIdsMetadataInfo>> fetchValidDocIds(String
realtimeTableName,
+ Map<String, String> taskConfigs) {
+ // Get server to segment mappings
+ PinotHelixResourceManager pinotHelixResourceManager =
_clusterInfoAccessor.getPinotHelixResourceManager();
+ Map<String, List<String>> serverToSegments =
+
pinotHelixResourceManager.getServerToOnlineSegmentsMapFromEV(realtimeTableName,
true);
+ BiMap<String, String> serverToEndpoints;
+ try {
+ serverToEndpoints =
pinotHelixResourceManager.getDataInstanceAdminEndpoints(serverToSegments.keySet());
+ } catch (InvalidConfigException e) {
+ throw new RuntimeException(e);
+ }
+
+ ServerSegmentMetadataReader serverSegmentMetadataReader =
+ new ServerSegmentMetadataReader(_clusterInfoAccessor.getExecutor(),
+ _clusterInfoAccessor.getConnectionManager());
+
+ // Number of segments to query per server request
+ int numSegmentsBatchPerServerRequest = Integer.parseInt(
+
taskConfigs.getOrDefault(RealtimeToOfflineSegmentsTask.NUM_SEGMENTS_BATCH_PER_SERVER_REQUEST,
+ String.valueOf(DEFAULT_NUM_SEGMENTS_BATCH_PER_SERVER_REQUEST)));
+
Review Comment:
validate numSegmentsBatchPerServerRequest is > 0, fallback to default if
invalid
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
