Umesh Kumar Kumawat created HBASE-30539:
-------------------------------------------

             Summary: CreateTableProcedure ignores failed table descriptor 
write and get stuck
                 Key: HBASE-30539
                 URL: https://issues.apache.org/jira/browse/HBASE-30539
             Project: HBase
          Issue Type: Bug
          Components: master, proc-v2
            Reporter: Umesh Kumar Kumawat


h3. Problem

CreateTableProcedure can complete CREATE_TABLE_WRITE_FS_LAYOUT successfully 
even though the table descriptor file (.tableinfo) was never written to the 
file system. The table is then added to hbase:meta and its regions are 
assigned, but no RegionServer can open them.
h3. Root cause

Three behaviours combine:
 # FSTableDescriptors.writeTableDescriptor() tries up to 10 sequence ids. Each 
IOException is caught and logged at DEBUG only. When all attempts fail it 
returns null instead of throwing.
 # FSTableDescriptors.createTableDescriptorForTableDirectory() turns that null 
into a boolean false. This is the same value it returns when an identical 
descriptor already exists, so the caller cannot tell "already there" from 
"write failed".
 # CreateTableProcedure.createFsLayout() calls 
createTableDescriptorForTableDirectory(tableDir, tableDescriptor, false) and 
ignores the result. It then creates the region directories and the procedure 
continues to CREATE_TABLE_ADD_TO_META and CREATE_TABLE_ASSIGN_REGIONS.

Note that FSTableDescriptors.updateTableDescriptor() already treats a null from 
writeTableDescriptor() as an error (throws IOException("Failed update")). The 
create path does not.
h3. Impact
 * Master: CreateTableProcedure.CREATE_TABLE_WRITE_FS_LAYOUT updates the 
Master's descriptor cache with cacheOnly=true, so the Master believes the table 
is healthy.
 * RegionServer: HRegionServer has no table descriptor cache, so 
AssignRegionHandler reads the descriptor from the file system on every open and 
fails with {{{}java.io.IOException: Missing table descriptor for <region>{}}}.
 * TransitRegionStateProcedure keeps retrying with exponential backoff (capped 
at 10 minutes). The region stays in OPENING with location=null, 
CreateTableProcedure stays in CREATE_TABLE_ASSIGN_REGIONS, and the Master 
reports a STUCK Region-In-Transition. Nothing fails, so there is no automatic 
recovery.
 * Manual recovery needs the HBCK2 command generateMissingTableDescriptorFile 
(or writing the file by hand).

We saw this in production when HDFS refused all block allocations for a short 
window during a table create. HDFS recovered immediately afterwards, so the 
region directories were still created and the procedure carried on.
h3. Reproduction

A unit test using a DistributedFileSystem subclass (configured through 
fs.hdfs.impl) that fails every create() under <table>/.tabledesc/ shows:
 # all 10 writes fail, logged at DEBUG ("Failed write ...; retrying up to 10 
times")
 # CreateTableProcedure goes WRITE_FS_LAYOUT -> ADD_TO_META -> ASSIGN_REGIONS 
without failing
 # no .tableinfo exists, but the Master's cache has the descriptor
 # the RegionServer logs "Missing table descriptor for ..." and 
TransitRegionStateProcedure logs "Failed transition, suspend ..." with the 
region in OPENING, location=null

h3. Proposed fix
 * Make writeTableDescriptor() throw an IOException after the last attempt 
instead of returning null. Keep returning false from 
createTableDescriptorForTableDirectory() only for the "already exists and 
equal" case.
 * Because rollback is only supported at CREATE_TABLE_PRE_OPERATION, an 
IOException in WRITE_FS_LAYOUT makes the procedure suspend and retry the state 
with backoff. The pre-clean makes that retry safe, so the create succeeds once 
the file system recovers instead of leaving an unopenable table.
 * Log each failed attempt at WARN instead of DEBUG.
 * Review the other callers of createTableDescriptorForTableDirectory 
(CloneSnapshotProcedure, SnapshotManifest, MasterRegion) for the same pattern.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to