[
https://issues.apache.org/jira/browse/HBASE-30014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Geiszler updated HBASE-30014:
-----------------------------------
Description:
There is an issue in our new HBASE-29081_rebased branch after rebasing with the
master branch. I am seeing the following being repeated in master log after
running {{refresh_meta}} on the replica cluster:
{code:java}
2026-03-20T22:13:37,693 WARN
[RpcServer.priority.RWQ.Fifo.write.handler=2,queue=0,port=16000]
assignment.AssignmentManager: Unable to acquire lock for regionNode state=OPEN,
location=hbase-docker-2,16020,1774044759623, table=test_table,
region=4f345a940e95b6eba65d80f1ec34d71c. It is likely that another thread is
currently holding the lock. To avoid deadlock, skip execution for now.{code}
I see this on the read-replica docker container setup. The steps to reproduce
are:
# Start the hbase-docker container as the active cluster and hbase-docker-2 as
the replica cluster
# On the active cluster, use the HBase shell to create a table
# On the replica cluster, run {{{}refresh_meta{}}}. You will then see the
"Unable to acquire lock for regionNode..." log message repeated in
{{{}/opt/hbase/logs/hbase-jboss-master-hbase-docker-2.log{}}}. Also, after
running {{{}refresh_meta{}}}, you will not see the table that was created on
the active cluster.
I have attached a log from both the working branch (HBASE-29081) and the branch
where it does not work (HBASE-29081_rebased). Search the logs for "Submitting
RefreshMetaProcedure" to see where the {{refresh_meta}} procedure began.
was:
There is an issue in our new HBASE-29081_rebased branch after rebasing with the
master branch. I am seeing the following being repeated in master log after
running {{refresh_meta}} on the replica cluster:
{code:java}
2026-03-20T22:13:37,693 WARN
[RpcServer.priority.RWQ.Fifo.write.handler=2,queue=0,port=16000]
assignment.AssignmentManager: Unable to acquire lock for regionNode state=OPEN,
location=hbase-docker-2,16020,1774044759623, table=test_table,
region=4f345a940e95b6eba65d80f1ec34d71c. It is likely that another thread is
currently holding the lock. To avoid deadlock, skip execution for now.{code}
I see this on the read-replica docker container setup. The steps to reproduce
are:
# Start the hbase-docker container as the active cluster and hbase-docker-2 as
the replica cluster
# On the active cluster, use the HBase shell to create a table
# On the replica cluster, run {{{}refresh_meta{}}}. You will then see the
"Unable to acquire lock for regionNode..." log message repeated in
{{{}/opt/hbase/logs/hbase-jboss-master-hbase-docker-2.log{}}}. Also, after
running {{{}refresh_meta{}}}, you will not see the table that was created on
the active cluster.
I have attached a log from both the working branch (HBASE-29081) and the branch
where it does not work (HBASE-29081_rebased). Search the logs for "Submitting
RefreshMetaProcedure" to see where the {{refresh_meta}} procedure began.
> refresh_meta not working due to regionNode lock
> -----------------------------------------------
>
> Key: HBASE-30014
> URL: https://issues.apache.org/jira/browse/HBASE-30014
> Project: HBase
> Issue Type: Sub-task
> Components: read replicas, Replication
> Reporter: Kevin Geiszler
> Priority: Major
> Attachments:
> unable-to-acquire-lock-for-regionMode-replica-master.log,
> working-refresh-meta-master-replica.log
>
>
> There is an issue in our new HBASE-29081_rebased branch after rebasing with
> the master branch. I am seeing the following being repeated in master log
> after running {{refresh_meta}} on the replica cluster:
> {code:java}
> 2026-03-20T22:13:37,693 WARN
> [RpcServer.priority.RWQ.Fifo.write.handler=2,queue=0,port=16000]
> assignment.AssignmentManager: Unable to acquire lock for regionNode
> state=OPEN, location=hbase-docker-2,16020,1774044759623, table=test_table,
> region=4f345a940e95b6eba65d80f1ec34d71c. It is likely that another thread is
> currently holding the lock. To avoid deadlock, skip execution for now.{code}
> I see this on the read-replica docker container setup. The steps to reproduce
> are:
> # Start the hbase-docker container as the active cluster and hbase-docker-2
> as the replica cluster
> # On the active cluster, use the HBase shell to create a table
> # On the replica cluster, run {{{}refresh_meta{}}}. You will then see the
> "Unable to acquire lock for regionNode..." log message repeated in
> {{{}/opt/hbase/logs/hbase-jboss-master-hbase-docker-2.log{}}}. Also, after
> running {{{}refresh_meta{}}}, you will not see the table that was created on
> the active cluster.
> I have attached a log from both the working branch (HBASE-29081) and the
> branch where it does not work (HBASE-29081_rebased). Search the logs for
> "Submitting RefreshMetaProcedure" to see where the {{refresh_meta}} procedure
> began.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
