[
https://issues.apache.org/jira/browse/KAFKA-19236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17949302#comment-17949302
]
Matthias J. Sax edited comment on KAFKA-19236 at 5/4/25 5:24 PM:
-----------------------------------------------------------------
Don't think it's very common. Still we should provide reasonable behavior.
{quote}If not, maybe we can throw exception due to "no offset"?
{quote}
Maybe. But I believe the promise/idea of `auto.offset.reset` is to not throw an
exception, as long as used don't specify "none"? Of course, we don't
necessarily need to hold this up with "by-duration".
To me, to problem actually larger and it's about how to interpret `ListOffset`,
and how `offsetsForTimestamps()` should work. Assume, we have a topic with 1
day retention, and nothing was written for 1 hour. If you do `ListOffsets(
minus two days )` you get the very first offset back (ie, same as "earliest"
reset). Given the contract to return all data after the specified timestamp,
and older data than 1 day got deletes, this make sense.
On the other hand if you do a `ListOffsets (minus 10 minutes)` you get `null`
back. I believe we don't want to return `endOffset` because it won't be a
"sharp" bound – endOffset could be too far in the past (this was added a long
time ago and I was not part of these discussions – so no idea; it's gust a
guess – would need to dig into the history). However, I am really not sure if
the contract of `ListOffsets / offsetsForTimestamps` is strict anyway – if you
seek to a specific ts, the guarantee is that you don't miss anything data with
equal or larger ts – but you might still get data with lower timestamps due to
out-of-order data – so I believe, returning "endOffset" could actually work,
too?
Of course, changing the established behavior to return `null` would be a
semantics / breaking change, so it's not something we could just do...
was (Author: mjsax):
Don't think it's very common. Still we should provide reasonable behavior.
{quote}If not, maybe we can throw exception due to "no offset"?
{quote}
Maybe. – To me, to problem is really, how to interpret `ListOffsetRequest`, and
how `offsetsForTimestamps()` should work. Assume, we have a topic with 1 day
retention, and nothing was written for 1 hour. If you so `ListOffsets( minus
two day )` you get the very first offset back. Given the contract to return all
data after the specified timestamp, and older data than 1 day got deletes, this
make sense. – On the other hand if you
> Auto.offset.reset "by-duration" should reset a single time
> ----------------------------------------------------------
>
> Key: KAFKA-19236
> URL: https://issues.apache.org/jira/browse/KAFKA-19236
> Project: Kafka
> Issue Type: Improvement
> Components: clients, consumer, streams
> Affects Versions: 4.0.0
> Reporter: Matthias J. Sax
> Assignee: Chia-Ping Tsai
> Priority: Major
>
> KIP-1106 introduced a new option "by-duration" for config `auto.offset.reset`
>
> ([https://cwiki.apache.org/confluence/display/KAFKA/KIP-1106%3A+Add+duration+based+offset+reset+option+for+consumer+clients)]
> If a consumer tries to reset to a "future" time, the observed behavior is
> somewhat odd, and we should change it:
> Assume there is a topic for which no new data was written for the last hour.
> A new consumer starts up at 1pm, and tries to reset by 10 minutes. There is
> no data for 12:50pm, so the consumer won't complete the reset, but will keep
> retrying (every 30 seconds by default) until it can resolve offsets – the
> issue is, that the "seek ts" (ie 12:50pm) is recomputed on every retry and
> thus move while the consumer wait.
> Because the consumer could not resolve offset, but still has `offsets=null`,
> it keep re-executing the reset logic.
> Hence, if there is not data for another 30 minutes, the consumer would now
> retry to find offset at 1:20pm. This is rather unexpected, as if one starts
> the consumer at 1:00pm and resets by 10 minutes, it's reasonable to assume
> that data would start flowing when the topic reaches 12:50pm, even if the
> consumer was idling for 30 minutes.
> Thus, instead of executing the reset logic every 30 seconds, the reset logic
> should be executed once, to compute 12:50pm as "seek ts", and if the request
> returns `null` (ie, no offset found), the same request should be resent every
> 30 second, w/o re-triggering the rest logic itself, to keep the "seek ts" at
> 12:50pm.
> Kafka Streams, which re-implement the by-duration reset logic by itself, has
> the same behavior as the consumer, and should be updated, as well.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
