[
https://issues.apache.org/jira/browse/HBASE-30479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hernan Romer updated HBASE-30479:
---------------------------------
Description:
Since HBASE-29386, SnapshotProcedure takes an exclusive table lock for each
state, instead of a shared lock held for the whole procedure. While a state
runs, region assignment on that table is blocked. Some states do slow work
inline: consolidating the manifest, verifying regions when the table has fewer
than 10k regions, and adding split parent regions. On a table with tens of
thousands of regions these can take minutes, and regions moved during that time
stay in transition until the snapshot finishes. We hit this in production on a
cluster with ~64k regions.
The original design used a shared lock specifically to avoid this (see
HBASE-21480). I propose going back to the shared lock with holdLock=true, and
fixing the HBASE-29386 deadlock directly by failing the snapshot in
SNAPSHOT_WRITE_SNAPSHOT_INFO when the table is neither ENABLED nor DISABLED
was:
Since HBASE-29386, SnapshotProcedure takes an exclusive table lock for each
state, instead of a shared lock held for the whole procedure. While a state
runs, region assignment on that table is blocked. Some states do slow work
inline: consolidating the manifest, verifying regions when the table has fewer
than 10k regions, and adding split parent regions. On a table with tens of
thousands of regions these can take minutes, and regions moved during that time
stay in transition until the snapshot finishes. We hit this in production on a
cluster with ~64k regions.
The original design used a shared lock specifically to avoid this (see
HBASE-21480). I propose going back to the shared lock with holdLock=true, and
fixing the HBASE-29386 deadlock directly by failing the snapshot in
SNAPSHOT_WRITE_SNAPSHOT_INFO when the table is neither ENABLED nor DISABLED.
We should also improve the new backup system to be resilient of these failures
> SnapshotProcedure's exclusive table lock blocks region assignment on large
> tables
> ---------------------------------------------------------------------------------
>
> Key: HBASE-30479
> URL: https://issues.apache.org/jira/browse/HBASE-30479
> Project: HBase
> Issue Type: Improvement
> Components: Region Assignment, snapshots
> Affects Versions: 3.0.0, 2.6.3
> Reporter: Hernan Romer
> Priority: Major
>
> Since HBASE-29386, SnapshotProcedure takes an exclusive table lock for each
> state, instead of a shared lock held for the whole procedure. While a state
> runs, region assignment on that table is blocked. Some states do slow work
> inline: consolidating the manifest, verifying regions when the table has
> fewer than 10k regions, and adding split parent regions. On a table with tens
> of thousands of regions these can take minutes, and regions moved during that
> time stay in transition until the snapshot finishes. We hit this in
> production on a cluster with ~64k regions.
> The original design used a shared lock specifically to avoid this (see
> HBASE-21480). I propose going back to the shared lock with holdLock=true, and
> fixing the HBASE-29386 deadlock directly by failing the snapshot in
> SNAPSHOT_WRITE_SNAPSHOT_INFO when the table is neither ENABLED nor DISABLED
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
