kamalcph commented on PR #16602: URL: https://github.com/apache/kafka/pull/16602#issuecomment-2235785232
Need some clarifications on defining the timeout for the delayed remote LIST_OFFSETS request. Currently, the LIST_OFFSETS API is invoked by: 1. [KafkaConsumer#offsetForTimes(Map<TopicPartition, Long> timestampsToSearch)](https://javadoc.io/doc/org.apache.kafka/kafka-clients/latest/org/apache/kafka/clients/consumer/KafkaConsumer.html#offsetsForTimes-java.util.Map-) 2. [KafkaConsumer#offsetForTimes(Map<TopicPartition, Long> timestampsToSearch, Duration timeout)](https://javadoc.io/doc/org.apache.kafka/kafka-clients/latest/org/apache/kafka/clients/consumer/KafkaConsumer.html#offsetsForTimes-java.util.Map-java.time.Duration-) and 3. [Admin#listOffsets(Map<TopicPartition, OffsetSpec> topicPartitionOffsets)](https://javadoc.io/static/org.apache.kafka/kafka-clients/3.7.1/org/apache/kafka/clients/admin/Admin.html#listOffsets-java.util.Map-) Both KafkaConsumer and Admin defines the default api-timeout to 60 seconds and default request-timeout to 30 seconds. There is no upper timeout boundary for handling the LIST_OFFSETS request on the server side and the client does not convey any timeout in the [ListOffsetsRequest.json](https://github.com/apache/kafka/blob/trunk/clients/src/main/resources/common/message/ListOffsetsRequest.json) request. **Current behavior** `KafkaConsumer` sends one or more requests internally depends on the configured request-timeout and default-api-timeout. (eg) If the consumer is configured with default-api-timeout as 5 mins and request-timeout as 30 seconds. And, the server takes 50 seconds to process the LIST_OFFSETS request, then the consumer internally sends 10 LIST_OFFSETS requests before failing with TimeoutException after 5 minutes. `Admin` sends only one request and wait for upto default-api-timeout. (eg) If the admin is configured with default-api-timeout as 5 mins and request-timeout as 30 seconds. And, the server takes 50 seconds to process the LIST_OFFSETS request, then the admin sends only one LIST_OFFSETS request, then receives the request from server after 50 seconds. **Options** 1. If we add a new timeout field in the [ListOffsetsRequest](https://github.com/apache/kafka/blob/trunk/clients/src/main/resources/common/message/ListOffsetsRequest.json), then only new clients can able to specify the timeout. What timeout should we set for the requests from older clients? 2. If we propose a new dynamic `remote.list.offsets.timeout.ms` timeout on the server end, then only the operator of the cluster can be able to change the timeout. Most of the clients won't be having the cluster level permissions to update the timeout value on the server. cc @satishd @showuon @chia7712 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
