k-apol commented on code in PR #20326:
URL: https://github.com/apache/kafka/pull/20326#discussion_r2280926879
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/InternalTopicManager.java:
##########
@@ -461,120 +466,119 @@ public Set<String> makeReady(final Map<String,
InternalTopicConfig> topics) {
// have existed with the expected number of partitions, or some create
topic returns fatal errors.
log.debug("Starting to validate internal topics {} in partition
assignor.", topics);
- long currentWallClockMs = time.milliseconds();
+ final long currentWallClockMs = time.milliseconds();
final long deadlineMs = currentWallClockMs + retryTimeoutMs;
- Set<String> topicsNotReady = new HashSet<>(topics.keySet());
- final Set<String> newlyCreatedTopics = new HashSet<>();
+ final Set<String> topicsNotReady = new HashSet<>(topics.keySet());
+ final Set<String> newTopics = new HashSet<>();
while (!topicsNotReady.isEmpty()) {
- final Set<String> tempUnknownTopics = new HashSet<>();
- topicsNotReady = validateTopics(topicsNotReady, topics,
tempUnknownTopics);
- newlyCreatedTopics.addAll(topicsNotReady);
-
+ final Set<NewTopic> topicsToCreate = computeTopicsToCreate(topics,
topicsNotReady, newTopics);
+ if (!topicsToCreate.isEmpty()) {
+ readyTopics(topicsToCreate, topicsNotReady);
+ }
if (!topicsNotReady.isEmpty()) {
- final Set<NewTopic> newTopics = new HashSet<>();
+ maybeThrowTimeout(new TimeoutContext(
+ Collections.singleton("makeReadyCheck"), // dummy
collection just to trigger if `topicsNotReady` is non-empty
+ deadlineMs,
+ "MakeReady timeout",
+ String.format("Could not create topics within %d
milliseconds. This can happen if the Kafka cluster is temporarily not
available.", retryTimeoutMs),
+ null
+ ));
- for (final String topicName : topicsNotReady) {
- if (tempUnknownTopics.contains(topicName)) {
- // for the tempUnknownTopics, don't create topic for
them
- // we'll check again later if remaining retries > 0
- continue;
- }
- final InternalTopicConfig internalTopicConfig =
Objects.requireNonNull(topics.get(topicName));
- final Map<String, String> topicConfig =
internalTopicConfig.properties(defaultTopicConfigs,
windowChangeLogAdditionalRetention);
+ }
+ }
+ log.debug("Completed validating internal topics and created {}",
newTopics);
- log.debug("Going to create topic {} with {} partitions and
config {}.",
- internalTopicConfig.name(),
- internalTopicConfig.numberOfPartitions(),
- topicConfig);
+ return newTopics;
+ }
+
+ private Set<NewTopic> computeTopicsToCreate(final Map<String,
InternalTopicConfig> topics,
+ final Set<String>
topicsNotReady,
+ final Set<String>
newTopics) {
+ final Set<String> tempUnknownTopics = new HashSet<>();
+ final Set<String> validatedTopics = validateTopics(topicsNotReady,
topics, tempUnknownTopics);
Review Comment:
The return value represents topics that are configured correctly, and that
also need to get created
The method call here iterates the provided topics, and validates three
things:
1. The `topicsMap `provided in the config knows about all of the topics we
are trying to validate
2. The number of partitions for each internal topic is defined
3. If the internal topic already exists, we decide **not** to add it to the
return set but validate the number of partitions it has matches what the
`InternalTopicManager` has
If all 1 and 2 pass, we add to the return set, and repeat for all topics
passed into the function
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
