[
https://issues.apache.org/jira/browse/HDFS-17805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18063870#comment-18063870
]
ASF GitHub Bot commented on HDFS-17805:
---------------------------------------
balodesecurity opened a new pull request, #8315:
URL: https://github.com/apache/hadoop/pull/8315
## Problem
In `BlockReceiver.flushOrSync()`, flush and sync durations are accumulated
into a single `flushTotalNanos` counter. When the total duration exceeds the
slow-IO threshold, the WARN log only reports the combined value:
```
Slow flushOrSync took 120ms ..., flushTotalNanos=120000000ns
```
This makes it impossible to tell whether the latency originates from the
flush step or the fsync step, hindering production diagnosis.
## Fix
Track flush and sync durations in separate counters (`flushTotalNanos`,
`syncTotalNanos`). The slow-IO WARN log now reports them independently:
```
Slow flushOrSync took 120ms ..., flushNanos=5000000ns, syncNanos=115000000ns
```
This lets operators immediately determine whether a bottleneck is in the
page-cache flush or the disk fsync.
## Testing
- Added
`TestBlockReceiverSlowLog#testFlushOrSyncSlowLogContainsSeparateFlushAndSyncNanos`:
starts a single-DN MiniDFSCluster with slow-IO threshold set to 0 ms (triggers
the log on every call), writes a file and calls `hsync()`, captures the WARN
log output, and asserts both `flushNanos=` and `syncNanos=` are present.
- Test passes locally.
> A more detailed flushOrSync Warn Log
> ------------------------------------
>
> Key: HDFS-17805
> URL: https://issues.apache.org/jira/browse/HDFS-17805
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
> Reporter: Jiandan Yang
> Priority: Major
>
> When DataNode receives a request to flush or sync a block being written by a
> client , the method flushOrSync (boolean isSync) will be invoked.
> In this method, the duration of the whole process will be recorded, also the
> flush and sync time of streams will be summed and then marked as
> ‘flushTotalNanos’.
> If the duration of this ‘flush or sync’ process exceeds a certain threshold,
> a warn log would appear and tells the value of the duration and the
> ‘flushTotalNanos’, enabling us to pinpoint the slower part of the execution
> within the flushOrSync(boolean isSync) methods.
> The current log records the time for both flush and sync operations as a
> single metric called flushTotalNanos. However, if flushTotalNanos takes too
> long, users will not be able to further diagnose the issue in detail based on
> the log.
> I propose that the log separately display the durations of flush and sync
> operations, instead of combining them into flushTotalNanos. This would allow
> users to pinpoint issues more accurately.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
