[
https://issues.apache.org/jira/browse/KAFKA-13251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18001354#comment-18001354
]
wangqingyu commented on KAFKA-13251:
------------------------------------
I’d like to work on this
> ISR shrink to an error broker
> -----------------------------
>
> Key: KAFKA-13251
> URL: https://issues.apache.org/jira/browse/KAFKA-13251
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 2.4.0
> Environment: linux 4.1.0
> Reporter: chaos
> Priority: Major
>
> Disk error occurred in broker(=42),and then Shrinking ISR to itself.
> so why Shrinking ISR to an error broker?
> i.e. why not "Shrinking ISR from 55,42 to 55" but "Shrinking ISR from 55,42
> to 42".
> note:
> other partition(110) shrink correctly.
>
> kafka logs:
> broker42:
> [2021-08-26 20:20:55,640] ERROR [ReplicaManager broker=42] Error processing
> fetch with max size 1048576 from consumer on partition topic_xx-123:
> (fetchOffset=11061228956, logStartOffset=-1, maxBytes=1048576,
> currentLeaderEpoch=Optional.empty) (kafka.server.ReplicaManager)
> org.apache.kafka.common.errors.CorruptRecordException: Found record size 0
> smaller than minimum record overhead (14) in file
> /data4/kafka-logs/topic_xx-123/00000000011060934646.log.
> [2021-08-26 20:20:55,640] ERROR Error while appending records to
> topic_xx-123 in dir /data4/kafka-logs (kafka.server.LogDirFailureChannel)
> [2021-08-26 20:20:55,645] ERROR Error while deleting segments for
> topic_xx-123 in dir /data4/kafka-logs (kafka.server.LogDirFailureChannel)
> java.nio.file.FileSystemException:
> /data4/kafka-logs/topic_xx-123/00000000011040402299.log ->
> /data4/kafka-logs/topic_xx-123/00000000011040402299.log.deleted: Read-only
> file system
> Suppressed: java.nio.file.FileSystemException:
> /data4/kafka-logs/topic_xx-123/00000000011040402299.log ->
> /data4/kafka-logs/topic_xx-123/00000000011040402299.log.deleted: Read-only
> file system
> [2021-08-26 20:20:55,644] ERROR Error while appending records to
> topic_xx-123 in dir /data4/kafka-logs (kafka.server.LogDirFailureChannel)
> [2021-08-26 20:20:55,652] INFO [Partition topic_xx-123 broker=42] Shrinking
> ISR from 55,42 to 42. Leader: (highWatermark: 11061228956, endOffset:
> 11061228965). Out of sync replicas: (brokerId: 55, endOffset: 11061228956).
> (kafka.cluster.Partition)
>
> broker55:
> [2021-08-26 20:20:32,456] WARN [ReplicaFetcher replicaId=55, leaderId=42,
> fetcherId=0] Error in response for fetch request (type=FetchRequest,
> replicaId=55, maxWait=500, minBytes=1, maxBytes=10485760, fetchData={},
> isolationLevel=READ_UNCOMMITTED, toForget=, metadata=(sessionId=830774713,
> epoch=1562806014), rackId=) (kafka.server.ReplicaFetcherThread)
> [2021-08-26 20:20:43,503] INFO [Partition topic_xxx-110 broker=55] Shrinking
> ISR from 55,42 to 55. Leader: (highWatermark: 11061384367, endOffset:
> 11061388788). Out of sync replicas: (brokerId: 42, endOffset: 11061384367).
> (kafka.cluster.Partition)
>
> disk error on broker42 is:
> Aug 26 20:20:55 kernel: sd 0:2:5:0: [sdf] tag#33 FAILED Result:
> hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
