Jackie-Jiang commented on code in PR #11576:
URL: https://github.com/apache/pinot/pull/11576#discussion_r1328136076
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/TablesResource.java:
##########
@@ -285,6 +289,51 @@ public String getSegmentMetadata(
return ResourceUtils.convertToJsonString(tableMetadataInfo);
}
+ @GET
+ @Encoded
+ @Path("/tables/{tableName}/indexes")
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation(value = "Provide index metadata", notes = "Provide index
details for the table")
+ @ApiResponses(value = {
+ @ApiResponse(code = 200, message = "Success"), @ApiResponse(code = 500,
message = "Internal server error",
+ response = ErrorInfo.class), @ApiResponse(code = 404, message = "Table
or segment not found", response =
+ ErrorInfo.class)
+ })
+ public String getTableIndexes(
+ @ApiParam(value = "Table name including type", required = true, example
= "myTable_OFFLINE")
+ @PathParam("tableName") String tableName)
+ throws Exception {
+ TableDataManager tableDataManager =
ServerResourceUtils.checkGetTableDataManager(_serverInstance, tableName);
+ List<SegmentDataManager> allSegments =
tableDataManager.acquireAllSegments();
+ try {
+ long totalSegmentCount = 0;
+ Map<String, Map<String, Long>> columnToIndexesCount = new HashMap<>();
+ for (SegmentDataManager segmentDataManager : allSegments) {
+ if (segmentDataManager instanceof RealtimeSegmentDataManager) {
Review Comment:
In the documentation, let's mention that consuming segment is not counted to
avoid confusion
##########
pinot-server/src/main/java/org/apache/pinot/server/api/resources/TablesResource.java:
##########
@@ -285,6 +289,51 @@ public String getSegmentMetadata(
return ResourceUtils.convertToJsonString(tableMetadataInfo);
}
+ @GET
+ @Encoded
+ @Path("/tables/{tableName}/indexes")
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation(value = "Provide index metadata", notes = "Provide index
details for the table")
+ @ApiResponses(value = {
+ @ApiResponse(code = 200, message = "Success"), @ApiResponse(code = 500,
message = "Internal server error",
+ response = ErrorInfo.class), @ApiResponse(code = 404, message = "Table
or segment not found", response =
+ ErrorInfo.class)
+ })
+ public String getTableIndexes(
+ @ApiParam(value = "Table name including type", required = true, example
= "myTable_OFFLINE")
+ @PathParam("tableName") String tableName)
+ throws Exception {
+ TableDataManager tableDataManager =
ServerResourceUtils.checkGetTableDataManager(_serverInstance, tableName);
+ List<SegmentDataManager> allSegments =
tableDataManager.acquireAllSegments();
+ try {
+ long totalSegmentCount = 0;
Review Comment:
(minor) `long` seems not necessary, same for the index count
##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotTableRestletResource.java:
##########
@@ -964,6 +965,77 @@ public String getTableAggregateMetadata(
return segmentsMetadata;
}
+ @GET
+ @Path("tables/{tableName}/indexes")
+ @Authorize(targetType = TargetType.TABLE, paramName = "tableName", action =
Actions.Table.GET_METADATA)
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation(value = "Get the aggregate index details of all segments for a
table", notes = "Get the aggregate "
+ + "index details of all segments for a table")
+ public String getTableIndexes(
+ @ApiParam(value = "Name of the table", required = true)
@PathParam("tableName") String tableName,
+ @ApiParam(value = "OFFLINE|REALTIME") @QueryParam("type") String
tableTypeStr) {
+ LOGGER.info("Received a request to fetch aggregate metadata for a table
{}", tableName);
+ TableType tableType = Constants.validateTableType(tableTypeStr);
+ String tableNameWithType =
+
ResourceUtils.getExistingTableNamesWithType(_pinotHelixResourceManager,
tableName, tableType, LOGGER).get(0);
+
+ String tableIndexMetadata;
+ try {
+ JsonNode segmentsMetadataJson =
getAggregateIndexMetadataFromServer(tableNameWithType);
+ tableIndexMetadata =
JsonUtils.objectToPrettyString(segmentsMetadataJson);
+ } catch (InvalidConfigException e) {
+ throw new ControllerApplicationException(LOGGER, e.getMessage(),
Response.Status.BAD_REQUEST);
+ } catch (IOException ioe) {
+ throw new ControllerApplicationException(LOGGER, "Error parsing Pinot
server response: " + ioe.getMessage(),
+ Response.Status.INTERNAL_SERVER_ERROR, ioe);
+ }
+ return tableIndexMetadata;
+ }
+
+ private JsonNode getAggregateIndexMetadataFromServer(String
tableNameWithType)
+ throws InvalidConfigException, JsonProcessingException {
+ final Map<String, List<String>> serverToSegments =
+ _pinotHelixResourceManager.getServerToSegmentsMap(tableNameWithType);
+ BiMap<String, String> endpoints =
+
_pinotHelixResourceManager.getDataInstanceAdminEndpoints(serverToSegments.keySet());
+
+ BiMap<String, String> serverEndPoints =
+
_pinotHelixResourceManager.getDataInstanceAdminEndpoints(serverToSegments.keySet());
+ CompletionServiceHelper completionServiceHelper =
+ new CompletionServiceHelper(_executor, _connectionManager,
serverEndPoints);
+
+ List<String> serverUrls = new ArrayList<>();
+ BiMap<String, String> endpointsToServers = serverEndPoints.inverse();
+ for (String endpoint : endpointsToServers.keySet()) {
+ String reloadTaskStatusEndpoint = endpoint +
String.format("/tables/%s/indexes", tableNameWithType);
+ serverUrls.add(reloadTaskStatusEndpoint);
+ }
+
+ CompletionServiceHelper.CompletionServiceResponse serviceResponse =
+ completionServiceHelper.doMultiGetRequest(serverUrls, null, true,
10000);
+
+ long totalSegments = 0;
+ Map<String, Map<String, Long>> columnToIndexCountMap = new HashMap<>();
+ for (Map.Entry<String, String> streamResponse :
serviceResponse._httpResponses.entrySet()) {
+ String responseString = streamResponse.getValue();
+ TableIndexMetadataResponse response =
JsonUtils.stringToObject(responseString, TableIndexMetadataResponse.class);
+ totalSegments += response.getTotalOnlineSegments();
+ response.getColumnToIndexesCount().forEach((col, indexToCount) -> {
+ columnToIndexCountMap.putIfAbsent(col, new HashMap<>());
+
+ indexToCount.forEach((indexName, count) -> {
+ columnToIndexCountMap.get(col)
+ .put(indexName,
columnToIndexCountMap.get(col).getOrDefault(indexName, 0L) + count);
+ });
Review Comment:
Reduce map lookup
```suggestion
Map<String, Integer> indexCountMap =
columnToIndexCountMap.computeIfAbsent(col, new HashMap<>());
indexToCount.forEach((indexName, count) -> {
indexCountMap.compute(col, (k, v) -> v != null ? v + count :
count);
});
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
