This is an automated email from the ASF dual-hosted git repository.
sarutak pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new 8f544a50359b [SPARK-54794][CORE] Suppress verbose
`FsHistoryProvider.checkForLogs` scanning logs
8f544a50359b is described below
commit 8f544a50359b837fd536e4170edd109ac99dad7b
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Mon Dec 22 12:05:52 2025 +0900
[SPARK-54794][CORE] Suppress verbose `FsHistoryProvider.checkForLogs`
scanning logs
### What changes were proposed in this pull request?
This PR aims to suppress verbose `FsHistoryProvider.checkForLogs` scanning
logs.
### Why are the changes needed?
This is a kind of regression at Apache Spark 4.1.0 switched from `logDebug`
to `logInfo` accidentally. We had better recover them back to `DEBUG` level
because it's excessively verbose currently.
- #52382
```
25/12/21 22:27:53 INFO FsHistoryProvider: Scanning s3a://spark-events with
lastScanTime=1766356063285
25/12/21 22:28:03 INFO FsHistoryProvider: Scanning s3a://spark-events with
lastScanTime=1766356073312
25/12/21 22:28:13 INFO FsHistoryProvider: Scanning s3a://spark-events with
lastScanTime=1766356083338
25/12/21 22:28:23 INFO FsHistoryProvider: Scanning s3a://spark-events with
lastScanTime=1766356093347
25/12/21 22:28:33 INFO FsHistoryProvider: Scanning s3a://spark-events with
lastScanTime=1766356103369
25/12/21 22:28:43 INFO FsHistoryProvider: Scanning s3a://spark-events with
lastScanTime=1766356113391
25/12/21 22:28:53 INFO FsHistoryProvider: Scanning s3a://spark-events with
lastScanTime=1766356123402
25/12/21 22:29:03 INFO FsHistoryProvider: Scanning s3a://spark-events with
lastScanTime=1766356133422
25/12/21 22:29:13 INFO FsHistoryProvider: Scanning s3a://spark-events with
lastScanTime=1766356143443
25/12/21 22:29:23 INFO FsHistoryProvider: Scanning s3a://spark-events with
lastScanTime=1766356153458
25/12/21 22:29:33 INFO FsHistoryProvider: Scanning s3a://spark-events with
lastScanTime=1766356163471
25/12/21 22:29:43 INFO FsHistoryProvider: Scanning s3a://spark-events with
lastScanTime=1766356173491
25/12/21 22:29:53 INFO FsHistoryProvider: Scanning s3a://spark-events with
lastScanTime=1766356183502
25/12/21 22:30:03 INFO FsHistoryProvider: Scanning s3a://spark-events with
lastScanTime=1766356193517
25/12/21 22:30:13 INFO FsHistoryProvider: Scanning s3a://spark-events with
lastScanTime=1766356203529
25/12/21 22:30:23 INFO FsHistoryProvider: Scanning s3a://spark-events with
lastScanTime=1766356213555
25/12/21 22:30:33 INFO FsHistoryProvider: Scanning s3a://spark-events with
lastScanTime=1766356223567
25/12/21 22:30:43 INFO FsHistoryProvider: Scanning s3a://spark-events with
lastScanTime=1766356233585
25/12/21 22:30:53 INFO FsHistoryProvider: Scanning s3a://spark-events with
lastScanTime=1766356243599
```
### Does this PR introduce _any_ user-facing change?
No behavior change except log message level.
### How was this patch tested?
Pass the CIs.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #53555 from dongjoon-hyun/SPARK-54794.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Kousuke Saruta <[email protected]>
(cherry picked from commit d0fbb15bbc98798473ba842dcfc2dec601a7a376)
Signed-off-by: Kousuke Saruta <[email protected]>
---
.../main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
b/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
index 4863291b529b..c723a8de8c44 100644
---
a/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
+++
b/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
@@ -502,7 +502,7 @@ private[history] class FsHistoryProvider(conf: SparkConf,
clock: Clock)
var count: Int = 0
try {
val newLastScanTime = clock.getTimeMillis()
- logInfo(log"Scanning ${MDC(HISTORY_DIR, logDir)} with " +
+ logDebug(log"Scanning ${MDC(HISTORY_DIR, logDir)} with " +
log"lastScanTime=${MDC(LAST_SCAN_TIME, lastScanTime)}")
// Mark entries that are processing as not stale. Such entries do not
have a chance to be
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
