Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21309 )
Change subject: IMPALA-12046: Add profile counter for scan range queueing time on disk queues ...................................................................... Patch Set 7: (1 comment) http://gerrit.cloudera.org:8080/#/c/21309/7//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21309/7//COMMIT_MSG@16 PS7, Line 16: Each range starts the timer when it is enqueued and stops it when : dequeued The ConcurrentStopWatch is owned by HdfsScanNode and shared by all the corresponding scanner threads. I think it's started when the first scanner adds a request into the queue and stops when there are no concurrent scanner threads waiting for the queue. During this time, some scanner threads might already get the data and are materializing tuples. It's hard to compare this with existing counters like ScannerIoWaitTime, TotalRawHdfsOpenFileTime and TotalRawHdfsReadTime. https://github.com/apache/impala/blob/de7f610abc4fe7bb74c3ffae5b5369158f4b5eba/be/src/util/stopwatch.h#L207-L222 It'd be more valuable if TotalDiskIoQueueTime is part of ScannerIoWaitTime, i.e., it measures the time when the scanner thread is waiting for data of a ScanRange and the ScanRange is still in the disk queue (HDFS open/read not start yet). And it's the sum of such time in all scanner threads. I haven't thought about how to implement this yet. But it's more helpful if we can break down ScannerIoWaitTime into TotalRawHdfsOpenFileTime, TotalRawHdfsReadTime, this counter and other time (probably negligible). -- To view, visit http://gerrit.cloudera.org:8080/21309 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia0cb24db36cd89933c150b88e166e68a3abc7a60 Gerrit-Change-Number: 21309 Gerrit-PatchSet: 7 Gerrit-Owner: Yida Wu <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Surya Hebbar <[email protected]> Gerrit-Reviewer: Yida Wu <[email protected]> Gerrit-Comment-Date: Sun, 12 Apr 2026 14:20:08 +0000 Gerrit-HasComments: Yes
