[
https://issues.apache.org/jira/browse/GEODE-8584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17210348#comment-17210348
]
Bruce J Schuchardt commented on GEODE-8584:
-------------------------------------------
Here is the crux of the problem:
{noformat}
[info 2020/10/07 15:54:20.931 PDT <vm_3_thr_11_client2_host1_10891> tid=0x9e]
BRUCE: acquired direct buffer 1a7301b8 of size 65536. requested size 65536
from: [info 2020/10/07 15:54:20.931 PDT <vm_3_thr_11_client2_host1_10891>
tid=0x9e] BRUCE: acquired direct buffer 1a7301b8 of size 65536. requested size
65536 from:
org.apache.geode.internal.net.BufferPool.acquireDirectSenderBuffer(BufferPool.java:92)
org.apache.geode.internal.net.BufferPool.acquireDirectBuffer(BufferPool.java:286)
org.apache.geode.internal.net.BufferPool.expandWriteBufferIfNeeded(BufferPool.java:270)
org.apache.geode.internal.net.NioSslEngine.expandWriteBuffer(NioSslEngine.java:212)
[info 2020/10/07 15:54:27.077 PDT <AsyncSocketCloser for
10.32.108.97(clientgemfire1_host1_10887:10887)<v1>:41002> tid=0x94]
NioSSLEngine releasing two buffers myNetData=1a7301b8(direct),
peerAppData=6fffd1b(direct)
[info 2020/10/07 15:54:27.077 PDT <AsyncSocketCloser for
10.32.108.97(clientgemfire1_host1_10887:10887)<v1>:41002> tid=0x94] BRUCE:
releasing direct buffer 1a7301b8
[info 2020/10/07 15:54:27.078 PDT <vm_3_thr_8_client2_host1_10891> tid=0xeb]
Exception occurred while processing
UpdateOperation(EntryEventImpl[op=UPDATE;region=/testRegion;key=85;callbackArg=null;originRemote=false;originMember=10.32.108.97(clientgemfire2_host1_10891:10891)<v1>:41001;callbacksInvoked;version={v42;
rv571; mbr=414d94d5780a4d72-b562d4269a59a785; ds=1;
time=1602111267048};id=EventID[id=44 bytes;threadID=37;sequenceID=10]])
org.apache.geode.ToDataException: toData failed on dsfid=71 msg:null
at
org.apache.geode.internal.InternalDataSerializer.writeDSFID(InternalDataSerializer.java:1509)
at
org.apache.geode.internal.tcp.MsgStreamer.writeMessage(MsgStreamer.java:249)
at
org.apache.geode.distributed.internal.direct.DirectChannel.sendToMany(DirectChannel.java:306)
{noformat}
The thread that's writing a serialized message to the socket is using the
NioSslEngine's "myNetData" buffer. Another thread is closing the connection
and has invoked the NioFilter's close() method. That method closes out the TLS
session and modifies the "myNetData" buffer in the process.
> Message transmission fails with IllegalStateException in socket i/o code
> ------------------------------------------------------------------------
>
> Key: GEODE-8584
> URL: https://issues.apache.org/jira/browse/GEODE-8584
> Project: Geode
> Issue Type: Bug
> Components: membership, messaging
> Affects Versions: 1.12.0, 1.13.0, 1.14.0
> Reporter: Bruce J Schuchardt
> Priority: Major
> Labels: pull-request-available
>
> Transmission of an update operation failed with a toData exception caused by
> an IllegalStateException in socket i/o code. It shouldn't be possible for
> this exception to happen unless multiple threads are using the buffer at the
> same time.
>
> {noformat}
> org.apache.geode.ToDataException: toData failed on dsfid=71 msg:null
> at
> org.apache.geode.internal.InternalDataSerializer.writeDSFID(InternalDataSerializer.java:1509)
> at
> org.apache.geode.internal.tcp.MsgStreamer.writeMessage(MsgStreamer.java:249)
> at
> org.apache.geode.distributed.internal.direct.DirectChannel.sendToMany(DirectChannel.java:306)
> at
> org.apache.geode.distributed.internal.direct.DirectChannel.sendToOne(DirectChannel.java:182)
> at
> org.apache.geode.distributed.internal.direct.DirectChannel.send(DirectChannel.java:511)
> at
> org.apache.geode.distributed.internal.DistributionImpl.directChannelSend(DistributionImpl.java:346)
> at
> org.apache.geode.distributed.internal.DistributionImpl.send(DistributionImpl.java:291)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.sendViaMembershipManager(ClusterDistributionManager.java:2053)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.sendOutgoing(ClusterDistributionManager.java:1981)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.sendMessage(ClusterDistributionManager.java:2018)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.putOutgoing(ClusterDistributionManager.java:1083)
> at
> org.apache.geode.internal.cache.DistributedCacheOperation._distribute(DistributedCacheOperation.java:572)
> at
> org.apache.geode.internal.cache.DistributedCacheOperation.startOperation(DistributedCacheOperation.java:277)
> at
> org.apache.geode.internal.cache.DistributedCacheOperation.distribute(DistributedCacheOperation.java:318)
> at
> org.apache.geode.internal.cache.DistributedRegion.distributeUpdate(DistributedRegion.java:520)
> at
> org.apache.geode.internal.cache.DistributedRegion.basicPutPart3(DistributedRegion.java:498)
> at
> org.apache.geode.internal.cache.map.RegionMapPut.doAfterCompletionActions(RegionMapPut.java:307)
> at
> org.apache.geode.internal.cache.map.AbstractRegionMapPut.doPut(AbstractRegionMapPut.java:185)
> at
> org.apache.geode.internal.cache.map.AbstractRegionMapPut.runWhileLockedForCacheModification(AbstractRegionMapPut.java:119)
> at
> org.apache.geode.internal.cache.map.RegionMapPut.runWhileLockedForCacheModification(RegionMapPut.java:161)
> at
> org.apache.geode.internal.cache.map.AbstractRegionMapPut.put(AbstractRegionMapPut.java:169)
> at
> org.apache.geode.internal.cache.AbstractRegionMap.basicPut(AbstractRegionMap.java:2044)
> at
> org.apache.geode.internal.cache.LocalRegion.virtualPut(LocalRegion.java:5599)
> at
> org.apache.geode.internal.cache.DistributedRegion.virtualPut(DistributedRegion.java:393)
> at
> org.apache.geode.internal.cache.LocalRegion.virtualPut(LocalRegion.java:5577)
> at
> org.apache.geode.internal.cache.LocalRegionDataView.putEntry(LocalRegionDataView.java:157)
> at
> org.apache.geode.internal.cache.LocalRegion.basicPut(LocalRegion.java:5035)
> at
> org.apache.geode.internal.cache.LocalRegion.replaceWithCallbackArgument(LocalRegion.java:10600)
> at
> org.apache.geode.internal.cache.LocalRegion.replace(LocalRegion.java:10566)
> [application code traces deleted]
> Caused by: java.lang.IllegalArgumentException
> at java.nio.Buffer.position(Buffer.java:244)
> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:96)
> at sun.nio.ch.IOUtil.write(IOUtil.java:51)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:469)
> at
> org.apache.geode.internal.tcp.Connection.writeFully(Connection.java:2596)
> at
> org.apache.geode.internal.tcp.Connection.sendPreserialized(Connection.java:1866)
> at
> org.apache.geode.internal.tcp.MsgStreamer.realFlush(MsgStreamer.java:330)
> at org.apache.geode.internal.tcp.MsgStreamer.write(MsgStreamer.java:390)
> at
> org.apache.geode.DataSerializer.writeByteArray(DataSerializer.java:1185)
> at
> org.apache.geode.DataSerializer.writeByteArray(DataSerializer.java:1153)
> at
> org.apache.geode.internal.cache.DistributedCacheOperation.writeValue(DistributedCacheOperation.java:137)
> at
> org.apache.geode.internal.cache.UpdateOperation$UpdateMessage.toData(UpdateOperation.java:417)
> at
> org.apache.geode.internal.serialization.internal.DSFIDSerializerImpl.invokeToData(DSFIDSerializerImpl.java:213)
> at
> org.apache.geode.internal.serialization.internal.DSFIDSerializerImpl.write(DSFIDSerializerImpl.java:137)
> at
> org.apache.geode.internal.InternalDataSerializer.writeDSFID(InternalDataSerializer.java:1484)
> ... 37 more {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
