[
https://issues.apache.org/jira/browse/HBASE-30539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Umesh Kumar Kumawat reassigned HBASE-30539:
-------------------------------------------
Assignee: Umesh Kumar Kumawat
> CreateTableProcedure ignores failed table descriptor write and get stuck
> ------------------------------------------------------------------------
>
> Key: HBASE-30539
> URL: https://issues.apache.org/jira/browse/HBASE-30539
> Project: HBase
> Issue Type: Bug
> Components: master, proc-v2
> Reporter: Umesh Kumar Kumawat
> Assignee: Umesh Kumar Kumawat
> Priority: Major
> Labels: pull-request-available
>
> h3. Problem
> CreateTableProcedure can complete CREATE_TABLE_WRITE_FS_LAYOUT successfully
> even though the table descriptor file (.tableinfo) was never written to the
> file system. The table is then added to hbase:meta and its regions are
> assigned, but no RegionServer can open them.
> h3. Root cause
> Three behaviours combine:
> # FSTableDescriptors.writeTableDescriptor() tries up to 10 sequence ids.
> Each IOException is caught and logged at DEBUG only. When all attempts fail
> it returns null instead of throwing.
> # FSTableDescriptors.createTableDescriptorForTableDirectory() turns that
> null into a boolean false. This is the same value it returns when an
> identical descriptor already exists, so the caller cannot tell "already
> there" from "write failed".
> # CreateTableProcedure.createFsLayout() calls
> createTableDescriptorForTableDirectory(tableDir, tableDescriptor, false) and
> ignores the result. It then creates the region directories and the procedure
> continues to CREATE_TABLE_ADD_TO_META and CREATE_TABLE_ASSIGN_REGIONS.
> Note that FSTableDescriptors.updateTableDescriptor() already treats a null
> from writeTableDescriptor() as an error (throws IOException("Failed
> update")). The create path does not.
> h3. Impact
> * Master: CreateTableProcedure.CREATE_TABLE_WRITE_FS_LAYOUT updates the
> Master's descriptor cache with cacheOnly=true, so the Master believes the
> table is healthy.
> * RegionServer: HRegionServer has no table descriptor cache, so
> AssignRegionHandler reads the descriptor from the file system on every open
> and fails with {{{}java.io.IOException: Missing table descriptor for
> <region>{}}}.
> * TransitRegionStateProcedure keeps retrying with exponential backoff
> (capped at 10 minutes). The region stays in OPENING with location=null,
> CreateTableProcedure stays in CREATE_TABLE_ASSIGN_REGIONS, and the Master
> reports a STUCK Region-In-Transition. Nothing fails, so there is no automatic
> recovery.
> * Manual recovery needs the HBCK2 command generateMissingTableDescriptorFile
> (or writing the file by hand).
> We saw this in production when HDFS refused all block allocations for a short
> window during a table create. HDFS recovered immediately afterwards, so the
> region directories were still created and the procedure carried on.
> h3. Reproduction
> A unit test using a DistributedFileSystem subclass (configured through
> fs.hdfs.impl) that fails every create() under <table>/.tabledesc/ shows:
> # all 10 writes fail, logged at DEBUG ("Failed write ...; retrying up to 10
> times")
> # CreateTableProcedure goes WRITE_FS_LAYOUT -> ADD_TO_META -> ASSIGN_REGIONS
> without failing
> # no .tableinfo exists, but the Master's cache has the descriptor
> # the RegionServer logs "Missing table descriptor for ..." and
> TransitRegionStateProcedure logs "Failed transition, suspend ..." with the
> region in OPENING, location=null
> h3. Proposed fix
> * Make writeTableDescriptor() throw an IOException after the last attempt
> instead of returning null. Keep returning false from
> createTableDescriptorForTableDirectory() only for the "already exists and
> equal" case.
> * Because rollback is only supported at CREATE_TABLE_PRE_OPERATION, an
> IOException in WRITE_FS_LAYOUT makes the procedure suspend and retry the
> state with backoff. The pre-clean makes that retry safe, so the create
> succeeds once the file system recovers instead of leaving an unopenable table.
> * Log each failed attempt at WARN instead of DEBUG.
> * Review the other callers of createTableDescriptorForTableDirectory
> (CloneSnapshotProcedure, SnapshotManifest, MasterRegion) for the same pattern.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)