[
https://issues.apache.org/jira/browse/KAFKA-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16705465#comment-16705465
]
Chris Egerton edited comment on KAFKA-6833 at 11/30/18 11:43 PM:
-----------------------------------------------------------------
[~bob-barrett] another possible approach could be to alter the behavior of the
producer while it waits for metadata on a topic-partition.
Currently, when a record is given to the producer to send, it checks its
metadata cache and, if the topic-partition exists in the cache, [the cached
data is
used|https://github.com/apache/kafka/blob/9b476bc5f4a2fdbd62ad84e50e65331c21b321d0/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L967-L968].
If the topic doesn't exist or that partition doesn't exist in the topic, the
producer requests a metadata update and then waits until either
{{max.block.ms}} has been exceeded or until an update has occurred that
guarantees that the [topic
exists|https://github.com/apache/kafka/blob/9b476bc5f4a2fdbd62ad84e50e65331c21b321d0/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L998].
We could tweak the logic so that the producer waits until either
{{max.block.ms}} has been exceeded or until the topic-partition exists.
This would relieve users of the KafkaProducer from having to handle retry logic
for the suggested {{PartitionOutOfRangeException}}, although it could also
potentially lead to slower failures when trying to send a record to an invalid
partition.
was (Author: chrisegerton):
[~bob-barrett] another possible approach could be to alter the behavior of the
producer while it waits for metadata on a topic-partition.
Currently, when a record is given to the producer to send, it checks its
metadata cache and, if the topic-partition exists in the cache, [the cached
data is
used|https://github.com/apache/kafka/blob/9b476bc5f4a2fdbd62ad84e50e65331c21b321d0/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L967-L968].
If the topic doesn't exist or that partition doesn't exist in the topic, the
producer requests a metadata update and then waits until either
{{max.block.ms}} has been exceeded or until an update has occurred that results
in the [topic being
created|https://github.com/apache/kafka/blob/9b476bc5f4a2fdbd62ad84e50e65331c21b321d0/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L998].
We could tweak the logic so that the producer waits until either
{{max.block.ms}} has been exceeded or until the topic-partition exists.
This would relieve users of the KafkaProducer from having to handle retry logic
for the suggested {{PartitionOutOfRangeException}}, although it could also
potentially lead to slower failures when trying to send a record to an invalid
partition.
> KafkaProducer throws "Invalid partition given with record" exception
> --------------------------------------------------------------------
>
> Key: KAFKA-6833
> URL: https://issues.apache.org/jira/browse/KAFKA-6833
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Arjun Satish
> Assignee: Bob Barrett
> Priority: Minor
>
> Currently, when creating topics via ZooKeeper, there is a small but definite
> delay between creating the nodes in ZK, and having the topics created in the
> brokers. the KafkaProducer maintains a metadata cache about topics which get
> updated after the broker metadata is updated. If an application adds
> partitions to a topic, and immediately tries to produce records to a new
> partition, a KafkaException is throw with a message similar to the following:
> {code:java}
> Caused by: org.apache.kafka.common.KafkaException: Invalid partition given
> with record: 12 is not in the range [0...1).
> {code}
> In this case, since the application has context that it created the topics,
> it might be worthwhile to consider if a more specific exception can be thrown
> instead of KafkaException. For example:
> {code:java}
> public class PartitionNotFoundException extends KafkaException {...}{code}
> This could allow the application to be able to interpret such an error, and
> act accordingly.
> EDIT: Correct "create topics" to "adds partitions to a topic".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
