gharris1727 commented on code in PR #16053:
URL: https://github.com/apache/kafka/pull/16053#discussion_r1626593367
##########
connect/runtime/src/main/java/org/apache/kafka/connect/storage/KafkaConfigBackingStore.java:
##########
@@ -1123,6 +1125,24 @@ private void processTasksCommitRecord(String
connectorName, SchemaAndValue value
connectorTaskConfigGenerations.compute(connectorName,
(ignored, generation) -> generation != null ? generation + 1 : 0);
}
inconsistent.remove(connectorName);
+
+ Map<String, String> rawConnectorConfig =
connectorConfigs.get(connectorName);
+ Map<String, String> appliedConnectorConfig;
+ if (configTransformer != null) {
+ try {
+ appliedConnectorConfig =
configTransformer.transform(rawConnectorConfig);
Review Comment:
Let's say that there's 1000 connector configs in the config topic. Today I
think the configTransformer only transforms that config when it's in-use (like
a connector being started, fenced, updated, etc). That should only be a handful
of times after the startup completes.
With the code as-written, I think this would make 1000 configTransformer
requests, including ones on out-of-date configurations. If the ConfigProvider
is making a network request, this has the potential to be slow/expensive, and
generate a spike of load that wasn't there before.
Do you think there's a simple way to avoid this burst of ConfigProvider
calls during startup? The built-in AK providers can handle the burst safely,
but I wonder if there are third-party providers that could have bad
side-effects.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
