Umesh Kumar Kumawat created HBASE-30539:
-------------------------------------------
Summary: CreateTableProcedure ignores failed table descriptor
write and get stuck
Key: HBASE-30539
URL: https://issues.apache.org/jira/browse/HBASE-30539
Project: HBase
Issue Type: Bug
Components: master, proc-v2
Reporter: Umesh Kumar Kumawat
h3. Problem
CreateTableProcedure can complete CREATE_TABLE_WRITE_FS_LAYOUT successfully
even though the table descriptor file (.tableinfo) was never written to the
file system. The table is then added to hbase:meta and its regions are
assigned, but no RegionServer can open them.
h3. Root cause
Three behaviours combine:
# FSTableDescriptors.writeTableDescriptor() tries up to 10 sequence ids. Each
IOException is caught and logged at DEBUG only. When all attempts fail it
returns null instead of throwing.
# FSTableDescriptors.createTableDescriptorForTableDirectory() turns that null
into a boolean false. This is the same value it returns when an identical
descriptor already exists, so the caller cannot tell "already there" from
"write failed".
# CreateTableProcedure.createFsLayout() calls
createTableDescriptorForTableDirectory(tableDir, tableDescriptor, false) and
ignores the result. It then creates the region directories and the procedure
continues to CREATE_TABLE_ADD_TO_META and CREATE_TABLE_ASSIGN_REGIONS.
Note that FSTableDescriptors.updateTableDescriptor() already treats a null from
writeTableDescriptor() as an error (throws IOException("Failed update")). The
create path does not.
h3. Impact
* Master: CreateTableProcedure.CREATE_TABLE_WRITE_FS_LAYOUT updates the
Master's descriptor cache with cacheOnly=true, so the Master believes the table
is healthy.
* RegionServer: HRegionServer has no table descriptor cache, so
AssignRegionHandler reads the descriptor from the file system on every open and
fails with {{{}java.io.IOException: Missing table descriptor for <region>{}}}.
* TransitRegionStateProcedure keeps retrying with exponential backoff (capped
at 10 minutes). The region stays in OPENING with location=null,
CreateTableProcedure stays in CREATE_TABLE_ASSIGN_REGIONS, and the Master
reports a STUCK Region-In-Transition. Nothing fails, so there is no automatic
recovery.
* Manual recovery needs the HBCK2 command generateMissingTableDescriptorFile
(or writing the file by hand).
We saw this in production when HDFS refused all block allocations for a short
window during a table create. HDFS recovered immediately afterwards, so the
region directories were still created and the procedure carried on.
h3. Reproduction
A unit test using a DistributedFileSystem subclass (configured through
fs.hdfs.impl) that fails every create() under <table>/.tabledesc/ shows:
# all 10 writes fail, logged at DEBUG ("Failed write ...; retrying up to 10
times")
# CreateTableProcedure goes WRITE_FS_LAYOUT -> ADD_TO_META -> ASSIGN_REGIONS
without failing
# no .tableinfo exists, but the Master's cache has the descriptor
# the RegionServer logs "Missing table descriptor for ..." and
TransitRegionStateProcedure logs "Failed transition, suspend ..." with the
region in OPENING, location=null
h3. Proposed fix
* Make writeTableDescriptor() throw an IOException after the last attempt
instead of returning null. Keep returning false from
createTableDescriptorForTableDirectory() only for the "already exists and
equal" case.
* Because rollback is only supported at CREATE_TABLE_PRE_OPERATION, an
IOException in WRITE_FS_LAYOUT makes the procedure suspend and retry the state
with backoff. The pre-clean makes that retry safe, so the create succeeds once
the file system recovers instead of leaving an unopenable table.
* Log each failed attempt at WARN instead of DEBUG.
* Review the other callers of createTableDescriptorForTableDirectory
(CloneSnapshotProcedure, SnapshotManifest, MasterRegion) for the same pattern.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)