[
https://issues.apache.org/jira/browse/HBASE-30291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18102356#comment-18102356
]
mazhengxuan commented on HBASE-30291:
-------------------------------------
I investigated this issue and would like to work on it.
The root cause is that {{BackupHFileCleaner#getDeletableFiles}} loads the fully
backed-up tables and bulk-loaded HFile references from the backup system tables
on every invocation. {{CleanerChore}} invokes this method concurrently for each
non-empty archive directory, while {{preClean()}} and {{postClean()}} are
called only once per cleaner cycle. As a result, the same data is scanned and
materialized thousands of times in a large cluster.
My proposed fix is:
# Load the referenced HFile names once in {{BackupHFileCleaner#preClean()}}
and publish them as a per-cleaning-cycle snapshot.
# Make {{getDeletableFiles()}} use only this in-memory snapshot for filtering.
# Preserve the existing {{previousCleaningCompletionTimestamp}} protection so
that HFiles archived after the snapshot are retained until the next cleaner
cycle.
# Fail closed if the system-table scan fails: mark the snapshot unavailable
and keep all files for that cycle instead of using an empty or stale snapshot.
# Add tests verifying that multiple {{getDeletableFiles()}} calls within one
cycle trigger only one system-table load, the snapshot is refreshed in the next
cycle, and scan failures do not make files deletable.
This should reduce the system-table scans from once per archive-directory batch
to once per cleaner cycle, without changing the general {{CleanerChore}}
framework or adding a cross-cycle cache.
> BackupHFileCleaner scans backups system table redundantly for every region,
> causing excessive I/O and archive buildup
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-30291
> URL: https://issues.apache.org/jira/browse/HBASE-30291
> Project: HBase
> Issue Type: Bug
> Components: backup&restore
> Reporter: Hernan Romer
> Priority: Major
>
> The BackupHFileCleaner scans the backups system table once per call to
> getDeletableFiles, which is invoked separately for every region in the
> cluster. On clusters with many regions (~40k), this results in tens of
> thousands of redundant full scans of the same table in a single cleaner
> cycle. This causes the cleaner to take so long that it misses its scheduled
> start time and runs continuously, generating sustained high read request
> volume against the host holding the system tables.
>
> Because the cleaner can't keep up, archived HFiles accumulate on disk,
> leading to unexpected spikes in disk utilization (40% increase observed in
> one case). The problem is exacerbated on clusters that do frequent bulk
> loads, since compactions triggered by those bulk loads produce a large volume
> of archived HFiles that the cleaner needs to process.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
