[ 
https://issues.apache.org/jira/browse/HBASE-30539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Umesh Kumar Kumawat reassigned HBASE-30539:
-------------------------------------------

    Assignee: Umesh Kumar Kumawat

> CreateTableProcedure ignores failed table descriptor write and get stuck
> ------------------------------------------------------------------------
>
>                 Key: HBASE-30539
>                 URL: https://issues.apache.org/jira/browse/HBASE-30539
>             Project: HBase
>          Issue Type: Bug
>          Components: master, proc-v2
>            Reporter: Umesh Kumar Kumawat
>            Assignee: Umesh Kumar Kumawat
>            Priority: Major
>              Labels: pull-request-available
>
> h3. Problem
> CreateTableProcedure can complete CREATE_TABLE_WRITE_FS_LAYOUT successfully 
> even though the table descriptor file (.tableinfo) was never written to the 
> file system. The table is then added to hbase:meta and its regions are 
> assigned, but no RegionServer can open them.
> h3. Root cause
> Three behaviours combine:
>  # FSTableDescriptors.writeTableDescriptor() tries up to 10 sequence ids. 
> Each IOException is caught and logged at DEBUG only. When all attempts fail 
> it returns null instead of throwing.
>  # FSTableDescriptors.createTableDescriptorForTableDirectory() turns that 
> null into a boolean false. This is the same value it returns when an 
> identical descriptor already exists, so the caller cannot tell "already 
> there" from "write failed".
>  # CreateTableProcedure.createFsLayout() calls 
> createTableDescriptorForTableDirectory(tableDir, tableDescriptor, false) and 
> ignores the result. It then creates the region directories and the procedure 
> continues to CREATE_TABLE_ADD_TO_META and CREATE_TABLE_ASSIGN_REGIONS.
> Note that FSTableDescriptors.updateTableDescriptor() already treats a null 
> from writeTableDescriptor() as an error (throws IOException("Failed 
> update")). The create path does not.
> h3. Impact
>  * Master: CreateTableProcedure.CREATE_TABLE_WRITE_FS_LAYOUT updates the 
> Master's descriptor cache with cacheOnly=true, so the Master believes the 
> table is healthy.
>  * RegionServer: HRegionServer has no table descriptor cache, so 
> AssignRegionHandler reads the descriptor from the file system on every open 
> and fails with {{{}java.io.IOException: Missing table descriptor for 
> <region>{}}}.
>  * TransitRegionStateProcedure keeps retrying with exponential backoff 
> (capped at 10 minutes). The region stays in OPENING with location=null, 
> CreateTableProcedure stays in CREATE_TABLE_ASSIGN_REGIONS, and the Master 
> reports a STUCK Region-In-Transition. Nothing fails, so there is no automatic 
> recovery.
>  * Manual recovery needs the HBCK2 command generateMissingTableDescriptorFile 
> (or writing the file by hand).
> We saw this in production when HDFS refused all block allocations for a short 
> window during a table create. HDFS recovered immediately afterwards, so the 
> region directories were still created and the procedure carried on.
> h3. Reproduction
> A unit test using a DistributedFileSystem subclass (configured through 
> fs.hdfs.impl) that fails every create() under <table>/.tabledesc/ shows:
>  # all 10 writes fail, logged at DEBUG ("Failed write ...; retrying up to 10 
> times")
>  # CreateTableProcedure goes WRITE_FS_LAYOUT -> ADD_TO_META -> ASSIGN_REGIONS 
> without failing
>  # no .tableinfo exists, but the Master's cache has the descriptor
>  # the RegionServer logs "Missing table descriptor for ..." and 
> TransitRegionStateProcedure logs "Failed transition, suspend ..." with the 
> region in OPENING, location=null
> h3. Proposed fix
>  * Make writeTableDescriptor() throw an IOException after the last attempt 
> instead of returning null. Keep returning false from 
> createTableDescriptorForTableDirectory() only for the "already exists and 
> equal" case.
>  * Because rollback is only supported at CREATE_TABLE_PRE_OPERATION, an 
> IOException in WRITE_FS_LAYOUT makes the procedure suspend and retry the 
> state with backoff. The pre-clean makes that retry safe, so the create 
> succeeds once the file system recovers instead of leaving an unopenable table.
>  * Log each failed attempt at WARN instead of DEBUG.
>  * Review the other callers of createTableDescriptorForTableDirectory 
> (CloneSnapshotProcedure, SnapshotManifest, MasterRegion) for the same pattern.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to