[
https://issues.apache.org/jira/browse/HBASE-29390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17985190#comment-17985190
]
Hudson commented on HBASE-29390:
--------------------------------
Results for branch branch-3
[build #431 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/431/]:
(/) *{color:green}+1 overall{color}*
----
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/431/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk17 hadoop3 checks{color}
-- For more information [see jdk17
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/431/JDK17_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk17 hadoop 3.3.5 backward compatibility checks{color}
-- For more information [see jdk17
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/431/JDK17_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk17 hadoop 3.3.6 backward compatibility checks{color}
-- For more information [see jdk17
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/431/JDK17_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk17 hadoop 3.4.0 backward compatibility checks{color}
-- For more information [see jdk17
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/431/JDK17_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test for 3.3.5 {color}
(/) {color:green}+1 client integration test for 3.3.6 {color}
(/) {color:green}+1 client integration test for 3.4.0 {color}
(/) {color:green}+1 client integration test for 3.4.1 {color}
> Too many logs in AsyncBatchRpcRetryingCaller when hitting
> RegionTooBusyException
> --------------------------------------------------------------------------------
>
> Key: HBASE-29390
> URL: https://issues.apache.org/jira/browse/HBASE-29390
> Project: HBase
> Issue Type: Improvement
> Components: asyncclient, Client
> Reporter: Duo Zhang
> Assignee: Duo Zhang
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.7.0, 3.0.0-beta-2, 2.6.3, 2.5.12
>
>
> When checking the iTBLL result, I saw bunch of logs in the link generator
> logs like this
> {noformat}
> 2025-06-10 06:52:30,800 WARN [RPCClient-NioEventLoopGroup-1-8]
> org.apache.hadoop.hbase.client.AsyncBatchRpcRetryingCaller: Process batch for
> ['IntegrationTestBigLinkedList,<"\x0F\x91\xF3\x07\x83\x07\xDBI\xF8\xB5\xCA\xD6\x19\xB8,1749399341229.6a9162
> 4d74729cdd03e61ee7500e6951.'] in IntegrationTestBigLinkedList from
> data01,16020,1749509062898 failed, tries=6
> org.apache.hadoop.hbase.RegionTooBusyException:
> org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 M,
> regionName=6a91624d74729cdd03e61ee7500e6951, server=data01,16020,1749509062898
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:5379)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4840)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4766)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1021)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:939)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:904)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2877)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45448)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:457)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
> at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
> at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
> at
> jdk.internal.reflect.GeneratedConstructorAccessor13.newInstance(Unknown
> Source)
> at
> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at
> java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
> at
> java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toException(ProtobufUtil.java:1725)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ResponseConverter.getResults(ResponseConverter.java:172)
> at
> org.apache.hadoop.hbase.client.AsyncBatchRpcRetryingCaller.lambda$sendToServer$11(AsyncBatchRpcRetryingCaller.java:409)
> at
> org.apache.hbase.thirdparty.com.google.protobuf.RpcUtil$1.run(RpcUtil.java:56)
> at
> org.apache.hbase.thirdparty.com.google.protobuf.RpcUtil$1.run(RpcUtil.java:47)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:400)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:430)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:425)
> at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:117)
> at org.apache.hadoop.hbase.ipc.Call.setResponse(Call.java:149)
> at
> org.apache.hadoop.hbase.ipc.RpcConnection.finishCall(RpcConnection.java:396)
> at
> org.apache.hadoop.hbase.ipc.RpcConnection.readResponse(RpcConnection.java:461)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:125)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:140)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
> at
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
> at
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
> at
> org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:289)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
> {noformat}
> The same output repeated many many times, which makes it really difficult to
> find useful logs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
