kamalcph commented on code in PR #15634:
URL: https://github.com/apache/kafka/pull/15634#discussion_r1569328393
##########
core/src/main/scala/kafka/log/UnifiedLog.scala:
##########
@@ -282,15 +282,15 @@ class UnifiedLog(@volatile var logStartOffset: Long,
/**
* Update high watermark with offset metadata. The new high watermark will
be lower
- * bounded by the log start offset and upper bounded by the log end offset.
+ * bounded by the local-log-start-offset and upper bounded by the
log-end-offset.
*
* @param highWatermarkMetadata the suggested high watermark with offset
metadata
* @return the updated high watermark offset
*/
def updateHighWatermark(highWatermarkMetadata: LogOffsetMetadata): Long = {
val endOffsetMetadata = localLog.logEndOffsetMetadata
- val newHighWatermarkMetadata = if (highWatermarkMetadata.messageOffset <
logStartOffset) {
- new LogOffsetMetadata(logStartOffset)
+ val newHighWatermarkMetadata = if (highWatermarkMetadata.messageOffset <
_localLogStartOffset) {
Review Comment:
> For the makeLeaders path, it will call
UnifiedLog.convertToOffsetMetadataOrThrow. Within it,
checkLogStartOffset(offset) shouldn't throw OFFSET_OUT_OF_RANGE since we are
comparing the offset with logStartOffset. Do you know which part throws
OFFSET_OUT_OF_RANGE error?
The next line `localLog.convertToOffsetMetadataOrThrow` in
[convertToOffsetMetadataOrThrow](https://sourcegraph.com/github.com/apache/kafka@a3dcbd4e28a35f79f75ec1bf316ef0b39c0df164/-/blob/core/src/main/scala/kafka/log/UnifiedLog.scala?L1429)
method reads the segment from disk, there it throws the error. The call
[segments.floorSegment(startOffset)](https://sourcegraph.com/github.com/apache/kafka@a3dcbd4e28a35f79f75ec1bf316ef0b39c0df164/-/blob/core/src/main/scala/kafka/log/LocalLog.scala?L366)
in LocalLog fails to find the segment with log-start-offset, then
OffsetOutOfRangeException is thrown.
> For the follower fetch path, it's bounded by LogEndOffset. So it shouldn't
need to call UnifiedLog.fetchHighWatermarkMetadata, right? The regular consumer
will call UnifiedLog.fetchHighWatermarkMetadata.
yes, you're right. I attached the wrong call stack for handling the follower
request. Please find the updated call stack below:
Leader with invalid high-watermark handles the FETCH requests from follower
and throws OFFSET_OUT_OF_RANGE error:
```
KafkaApis.handleFetchRequest
ReplicaManager.fetchMessages
ReplicaManager.readFromLog
Partition.fetchRecords
# readFromLocalLog
Partition.updateFollowerFetchState
Partition.maybeIncrementLeaderHW
LeaderLog.maybeIncrementHighWatermark
UnifiedLog.fetchHighWatermarkMetadata
UnifiedLog.convertToOffsetMetadataOrThrow
LocalLog.convertToOffsetMetadataOrThrow
LocalLog.read
# OffsetOutOfRangeException
exception
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
