yurongfu opened a new pull request, #8792:
URL: https://github.com/apache/hbase/pull/8792
JIRA: HBASE-30483
# Description
Truncating a table with `preserveSplits=false` replaces all of its regions
with a single new region, but the active Master's namespace quota cache keeps
the pre-truncate region count for that table. A later region split can then be
rejected even though the actual region count is within the namespace limit.
With `hbase.quota.enabled=true`, a namespace created with
`hbase.namespace.quota.maxregions=2`, and a table `T` pre-split at `m`, both
`hbase:meta` and the namespace quota cache record two regions for `T`:
```text
nsStateCache["ns"].tableAndRegionInfo[T] = 2
```
`TruncateTableProcedure` coordinates the filesystem layout, meta records,
assignment, and table state across separate states. With `preserveSplits=false`
it generates one new region in `TRUNCATE_TABLE_CLEAR_FS_LAYOUT`:
```java
if (!preserveSplits) {
// if we are not preserving splits, generate a new single region
regions =
Arrays.asList(ModifyRegionUtils.createRegionInfos(tableDescriptor, null));
}
```
`TRUNCATE_TABLE_REMOVE_FROM_META` uses
`DeleteTableProcedure.deleteFromMeta()` to remove the old regions from meta and
`DeleteTableProcedure.deleteAssignmentState()` to clear assignment and
table-state data, but does not touch namespace quota usage.
`TRUNCATE_TABLE_ADD_TO_META` uses `CreateTableProcedure.addTableToMeta()`,
which writes the replacement regions to meta but does not run the quota
accounting that normal table creation performs in `preCreate()`. The procedure
therefore completes with:
```text
hbase:meta : 1 region for T
quota cache : 2 regions for T
```
A subsequent split of the new region at `m` would take actual usage from one
region to two, exactly meeting the limit. But
`NamespaceStateManager.checkAndUpdateNamespaceRegionCount()` reads the stale
cached count of two and takes its rejection branch:
```java
int regionCount = currentStatus.getRegionCount();
long maxRegionCount = TableNamespaceManager.getMaxRegions(nspdesc);
if (incr > 0 && regionCount >= maxRegionCount) {
...
return false;
}
```
so `NamespaceAuditor.checkQuotaToSplitRegion()` fails the split with:
```text
org.apache.hadoop.hbase.quotas.QuotaExceededException:
Region split not possible for :<encoded name> as quota limits are exceeded
```
The valid split cannot proceed until the Master rebuilds its namespace state
from meta (for example on failover). Other quota decisions in the namespace can
also consume the stale count.
# Solution
After the replacement regions have been written to meta and before the table
is enabled, `TruncateTableProcedure` now sets the namespace quota cache's
absolute per-table region count to the number of default replicas in its new
region list:
```java
case TRUNCATE_TABLE_ADD_TO_META:
regions = CreateTableProcedure.addTableToMeta(env, tableDescriptor,
regions);
// Reconcile the namespace quota cache with the new region layout
(HBASE-30483).
// Replica regions do not count toward the namespace region quota. This
state has no
// rollback, so an IOException here makes the procedure retry the whole
state; both
// addTableToMeta and the absolute count update are idempotent, so
retrying is safe.
int defaultRegionCount =
(int)
regions.stream().filter(RegionReplicaUtil::isDefaultReplica).count();
ProcedureSyncWait.getMasterQuotaManager(env)
.setTableRegionCountForTruncate(getTableName(), defaultRegionCount);
setNextState(TruncateTableState.TRUNCATE_TABLE_ASSIGN_REGIONS);
break;
```
Non-default replicas are excluded because `addTableToMeta()` returns the
list expanded by `RegionReplicaUtil.addReplicas()`, while namespace region
quota accounting (table creation, split, merge) only counts default replicas.
The reconciliation flows through a new absolute setter,
`MasterQuotaManager.setTableRegionCountForTruncate()` ->
`NamespaceAuditor.updateQuotaForTableTruncate()` ->
`NamespaceStateManager.setRegionCountForTable()` ->
`NamespaceTableAndRegionInfo.setRegionCountForTable()`:
```java
synchronized void setRegionCountForTable(TableName tableName, int count) {
...
AtomicInteger regionCount = tableAndRegionInfo.get(tableName);
if (regionCount != null) {
regionCount.set(count);
} else {
// An existing table is expected to already be in the cache; recover
from the inconsistency
// but make it visible.
LOG.warn(...);
tableAndRegionInfo.put(tableName, new AtomicInteger(count));
}
}
```
An absolute setter is idempotent if the procedure retries the
`TRUNCATE_TABLE_ADD_TO_META` state, unlike a decrement based on the old count,
and the single `synchronized` method avoids the transient window of the
remove-then-add pattern used by
`NamespaceStateManager.checkAndUpdateNamespaceRegionCount(TableName, int)`. If
the table entry is unexpectedly missing from the cache, the setter recovers by
inserting it, with a warning.
The reconciliation deliberately performs no quota-limit check: truncate has
already replaced the regions, and a resource-reducing truncate must still
complete even if an administrator lowered the namespace limit below the table's
earlier usage. If the namespace auditor is not yet initialized, or the
namespace's state is missing from the cache, the setter throws `IOException` so
the procedure retries instead of silently completing with stale accounting.
System tables remain exempt from namespace quota accounting, and
`MasterQuotaManager.setTableRegionCountForTruncate()` is a no-op when quota
support is disabled.
# Tests
Added `TestTruncateTableNamespaceQuota` with four tests, running on a real
two-regionserver `HBaseTestingUtil` mini cluster with
`hbase.quota.enabled=true`:
- `testTruncateReconcilesQuotaAndAllowsSplit`
- `testTruncateSucceedsAfterQuotaLowered`
- `testTruncateCountsOnlyDefaultReplicas`
- `testTruncatePreservingSplitsKeepsQuotaCount`
`testTruncateReconcilesQuotaAndAllowsSplit` performs this sequence:
1. Creates a namespace with `hbase.namespace.quota.maxregions=2`.
2. Creates a table pre-split at `m`, producing two regions.
3. Confirms that both the actual region count and the cached count are 2.
4. Disables the table and runs `truncateTable(table, false)`.
5. Confirms that the table now has one region and that the cached count is 1.
6. Loads rows on both sides of the split key, flushes, and waits until the
region is splittable.
7. Splits the region at `m` through the public `splitRegionAsync()` API.
8. Confirms that both the actual region count and the cached count become 2.
`testTruncateSucceedsAfterQuotaLowered` lowers the namespace limit from 2 to
1 via `modifyNamespace()` before the truncate, pinning the design choice that
the reconciliation performs no quota-limit check: the resource-reducing
truncate still completes and reconciles both counts to 1.
`testTruncateCountsOnlyDefaultReplicas` uses a table with
`regionReplication=2` and verifies that after truncate the cached count is 1,
not 2, covering the default-replica filter.
`testTruncatePreservingSplitsKeepsQuotaCount` verifies that
`truncateTable(table, true)` keeps the cached count at 2.
Without the fix, the three reconciliation tests fail at the post-truncate
cache assertion:
```text
testTruncateReconcilesQuotaAndAllowsSplit: expected: <1> but was: <2>
testTruncateSucceedsAfterQuotaLowered: expected: <1> but was: <2>
testTruncateCountsOnlyDefaultReplicas: expected: <1> but was: <2>
```
`testTruncatePreservingSplitsKeepsQuotaCount` passes with or without the fix
and guards against regressions in the `preserveSplits=true` path. If the
post-truncate assertion is bypassed, the split in step 7 of
`testTruncateReconcilesQuotaAndAllowsSplit` is rejected with
`QuotaExceededException`.
With the fix, all four tests pass, and the existing
`TestTruncateTableProcedure` (10 tests) and `TestNamespaceAuditor` (11 tests)
suites pass with no failures. `spotless:check` passes for `hbase-server`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
