[
https://issues.apache.org/jira/browse/HBASE-29390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17966040#comment-17966040
]
Duo Zhang commented on HBASE-29390:
-----------------------------------
I think the problem is here
https://github.com/apache/hbase/blob/54442b12e7ca583eeabb4d68174416964ac923b1/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncBatchRpcRetryingCaller.java#L336
For each action, will call the onComplete method, and in the onComplete method
https://github.com/apache/hbase/blob/54442b12e7ca583eeabb4d68174416964ac923b1/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncBatchRpcRetryingCaller.java#L310
We will call logException for every action.
The error in description is RegionTooBusyException, so all the actions should
fail with RegionTooBusyException, so after tries reaches the threshold, for
every action we will log the exception, which will flood the log output.
Let me think how to fix it properly...
> Too many logs in AsyncBatchRpcRetryingCaller when hitting
> RegionTooBusyException
> --------------------------------------------------------------------------------
>
> Key: HBASE-29390
> URL: https://issues.apache.org/jira/browse/HBASE-29390
> Project: HBase
> Issue Type: Improvement
> Components: asyncclient, Client
> Reporter: Duo Zhang
> Assignee: Duo Zhang
> Priority: Major
>
> When checking the iTBLL result, I saw bunch of logs in the link generator
> logs like this
> {noformat}
> 2025-06-10 06:52:30,800 WARN [RPCClient-NioEventLoopGroup-1-8]
> org.apache.hadoop.hbase.client.AsyncBatchRpcRetryingCaller: Process batch for
> ['IntegrationTestBigLinkedList,<"\x0F\x91\xF3\x07\x83\x07\xDBI\xF8\xB5\xCA\xD6\x19\xB8,1749399341229.6a9162
> 4d74729cdd03e61ee7500e6951.'] in IntegrationTestBigLinkedList from
> data01,16020,1749509062898 failed, tries=6
> org.apache.hadoop.hbase.RegionTooBusyException:
> org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 M,
> regionName=6a91624d74729cdd03e61ee7500e6951, server=data01,16020,1749509062898
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:5379)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4840)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4766)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1021)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:939)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:904)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2877)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45448)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:457)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
> at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
> at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
> at
> jdk.internal.reflect.GeneratedConstructorAccessor13.newInstance(Unknown
> Source)
> at
> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at
> java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
> at
> java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toException(ProtobufUtil.java:1725)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ResponseConverter.getResults(ResponseConverter.java:172)
> at
> org.apache.hadoop.hbase.client.AsyncBatchRpcRetryingCaller.lambda$sendToServer$11(AsyncBatchRpcRetryingCaller.java:409)
> at
> org.apache.hbase.thirdparty.com.google.protobuf.RpcUtil$1.run(RpcUtil.java:56)
> at
> org.apache.hbase.thirdparty.com.google.protobuf.RpcUtil$1.run(RpcUtil.java:47)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:400)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:430)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:425)
> at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:117)
> at org.apache.hadoop.hbase.ipc.Call.setResponse(Call.java:149)
> at
> org.apache.hadoop.hbase.ipc.RpcConnection.finishCall(RpcConnection.java:396)
> at
> org.apache.hadoop.hbase.ipc.RpcConnection.readResponse(RpcConnection.java:461)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:125)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:140)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
> at
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
> at
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
> at
> org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:289)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
> {noformat}
> The same output repeated many many times, which makes it really difficult to
> find useful logs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
