xiangfu0 opened a new pull request, #16620: URL: https://github.com/apache/pinot/pull/16620
## Summary This PR optimizes Kafka admin client usage in the stream ingestion modules by implementing connection reuse instead of creating new clients for each operation. ## Changes - Add reusable admin client instance in `KafkaPartitionLevelConnectionHandler` - Implement `getOrCreateAdminClient()` with thread-safe lazy initialization - Update `KafkaStreamMetadataProvider.getTopics()` to reuse admin client - Ensure proper cleanup of admin client in `close()` method - Apply changes to both Kafka 2.0 and 3.0 modules ## Benefits - **Performance**: Eliminates overhead of creating new admin client connections for each `getTopics()` call - **Resource Efficiency**: Reduces connection pooling overhead and network setup/teardown - **Thread Safety**: Uses double-checked locking for safe lazy initialization - **Backward Compatibility**: Preserves existing `createAdminClient()` method ## Files Modified - `pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/src/main/java/org/apache/pinot/plugin/stream/kafka20/KafkaPartitionLevelConnectionHandler.java` - `pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/src/main/java/org/apache/pinot/plugin/stream/kafka20/KafkaStreamMetadataProvider.java` - `pinot-plugins/pinot-stream-ingestion/pinot-kafka-3.0/src/main/java/org/apache/pinot/plugin/stream/kafka30/KafkaPartitionLevelConnectionHandler.java` - `pinot-plugins/pinot-stream-ingestion/pinot-kafka-3.0/src/main/java/org/apache/pinot/plugin/stream/kafka30/KafkaStreamMetadataProvider.java` This optimization improves performance in production environments where `getTopics()` is called frequently by reusing admin client connections instead of creating new ones for each call. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
