Luke Chen created KAFKA-16709:
---------------------------------
Summary: move logDir within broker might cause log cleanup hanging
Key: KAFKA-16709
URL: https://issues.apache.org/jira/browse/KAFKA-16709
Project: Kafka
Issue Type: Bug
Affects Versions: 3.7.0
Reporter: Luke Chen
Assignee: Luke Chen
When doing alter replica logDirs, we'll create a future log and pause log
cleaning for the partition(
[here|https://github.com/apache/kafka/blob/643db430a707479c9e87eec1ad67e1d4f43c9268/core/src/main/scala/kafka/server/ReplicaManager.scala#L1200]).
And this log cleaning pausing will resume after alter replica logDirs
completes
([here|https://github.com/apache/kafka/blob/643db430a707479c9e87eec1ad67e1d4f43c9268/core/src/main/scala/kafka/log/LogManager.scala#L1254]).
And when in the resuming log cleaning, we'll decrement 1 for the
LogCleaningPaused count. Once the count reached 0, the cleaning pause is really
resuming.
([here|https://github.com/apache/kafka/blob/643db430a707479c9e87eec1ad67e1d4f43c9268/core/src/main/scala/kafka/log/LogCleanerManager.scala#L310]).
For more explanation about the logCleaningPaused state can check
[here|https://github.com/apache/kafka/blob/643db430a707479c9e87eec1ad67e1d4f43c9268/core/src/main/scala/kafka/log/LogCleanerManager.scala#L55].
But, there's still one factor that could increase the LogCleaningPaused count:
leadership change
([here|https://github.com/apache/kafka/blob/643db430a707479c9e87eec1ad67e1d4f43c9268/core/src/main/scala/kafka/server/ReplicaManager.scala#L2126]).
When there's a leadership change, we'll check if there's a future log in this
partition, if so, we'll create future log and pauseCleaning (LogCleaningPaused
count + 1). So, if during the alter replica logDirs:
# alter replica logDirs for tp0 triggered (LogCleaningPaused count = 1)
# tp0 leadership changed (LogCleaningPaused count = 2)
# alter replica logDirs completes, resuming logCleaning (LogCleaningPaused
count = 1)
# LogCleaning keeps paused because the count is always > 0
The log cleaning is not just related to compacting logs, but also affecting the
normal log retention processing, which means, the log retention for these
paused partitions will be pending. This issue can be fixed when broker
restarted.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
