[
https://issues.apache.org/jira/browse/HBASE-29554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Emil Kleszcz updated HBASE-29554:
---------------------------------
Issue Type: Bug (was: Task)
> Corrupted hbase:meta rowkeys cannot be deleted
> ----------------------------------------------
>
> Key: HBASE-29554
> URL: https://issues.apache.org/jira/browse/HBASE-29554
> Project: HBase
> Issue Type: Bug
> Affects Versions: 2.5.10
> Reporter: Emil Kleszcz
> Priority: Major
>
> *Context:*
> It is possible to corrupt the {{hbase:meta}} table by inserting rowkeys
> containing a single comma using commands like:
> {code:java}
> put 'hbase:meta', 'rowkey,', 'info:regioninfo', 'test'{code}
> Example of resulting rows in `{{{}hbase:meta`{}}}:
> {code:java}
> ROW COLUMN+CELL
> , column=info:regioninfo,
> timestamp=2025-08-26T21:49:33.427, value=
> rowkey, column=info:regioninfo,
> timestamp=2025-08-26T22:05:52.653, value=
> rowkey, column=info:regioninfo,
> timestamp=2025-08-26T21:36:56.560, value=
> rowkey,a column=info:regioninfo,
> timestamp=2025-08-26T22:29:06.576, value= {code}
> *Problem:*
> * When a rowkey in `{{{}hbase:meta`{}}} contains a single comma, scanners
> return the rowkeys correctly, but {{{}get{}}}, {{{}delete{}}}, and
> {{deleteall}} operations {*}do not work on these rowkeys{*}.
> * This behavior is specific to {{hbase:meta}} and does not occur in user
> tables.
> *Attempted Workarounds:*
> Tried deleting the corrupted row using the Java HBase client:
> {code:java}
> import org.apache.hadoop.hbase.client.ConnectionFactory
> import org.apache.hadoop.hbase.client.Table
> import org.apache.hadoop.hbase.TableName
> import org.apache.hadoop.hbase.client.Delete
> import org.apache.hadoop.hbase.util.Bytes
> connection = ConnectionFactory.createConnection
> tabletable = connection.getTable(TableName.valueOf("hbase:meta"))
> keyrowKey = Bytes.toBytes("rowkey,")
> delete = Delete.new(rowKey)
> delete.setTimestamp(Long::MAX_VALUE)
> delete.addColumn(Bytes.toBytes("info"), Bytes.toBytes("state"))
> table.delete(delete)
> table.close
> connection.close{code}
> *Result:* Unable to delete the corrupted row.
> It is possible to insert a properly serialized region info for such keys:
> {code:java}
> rowKey = Bytes.toBytes("rowkey,")
> tableName = TableName.valueOf("TEST:TEST")
> Infobuilder = RegionInfoBuilder.newBuilder(tableName)
> builder.setStartKey(Bytes.toBytes(""))
> builder.setEndKey(Bytes.toBytes("1"))
> builder.setRegionId(12345)
> fakeRegion = builder.build()
> serializedValue = RegionInfo.toByteArray(fakeRegion)
> put = Put.new(rowKey)put.addColumn(HConstants::CATALOG_FAMILY,
> HConstants::REGIONINFO_QUALIFIER, serializedValue) {code}
> Resulting row in {{{}hbase:meta{}}}:
> {code:java}
> rowkey, column=info:regioninfo,
> timestamp=2025-08-27T17:02:21.565, value={ENCODED =>
> a6839d33e016dc75cfb9ac9c74a576c8, NAME => 'TEST:TEST,,1.a6839d33e0
> 16dc75cfb9ac9c74a576c8.',
> STARTKEY => '', ENDKEY => '1'} {code}
>
> *Problems / Risks:*
> # Corrupted rowkeys with commas in {{hbase:meta}} cannot be fixed or removed
> on a running cluster.
> # Allowing insertion of such rowkeys is risky. If insertion is allowed,
> there must be a mechanism for administrators to clean them up.
> # {{HBCK2}} does not resolve the issue (e.g., {{fixMeta}} fails).
> *Request / Suggested Action:*
> * Investigate why rowkeys with a single comma break deletion in
> {{{}hbase:meta{}}}.
> * Consider adding safeguards to prevent insertion of rowkeys with wrong
> format into {{{}hbase:meta{}}}.
> * Provide an administrative method to safely remove or repair corrupted meta
> rowkeys.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
