Barry Oglesby created GEODE-2293:
------------------------------------
Summary: The AckReaderThread incorrectly shuts down when an
IllegalStateException is throw whiel releasing an off-heap object
Key: GEODE-2293
URL: https://issues.apache.org/jira/browse/GEODE-2293
Project: Geode
Issue Type: Bug
Components: wan
Reporter: Barry Oglesby
The regression test run showed the following severe message:
{noformat}
[severe 2017/01/07 09:14:40.789 UTC
bridgegemfire_1_1_rs-QueuesBTTest-2017-01-06-14-07-15-client-12_3912
<AckReaderThread for : Event Processor for
GatewaySender_sender_bridgeds_1_to_bridgeds_2_0> tid=0x97] Stopping the
processor because the following exception occurred while processing a batch:
java.lang.IllegalStateException: It looks like off heap memory @7f33a8000238
was already freed. rawBits=0 history=null
at
org.apache.geode.internal.offheap.OffHeapStoredObject.release(OffHeapStoredObject.java:675)
at
org.apache.geode.internal.offheap.OffHeapStoredObject.release(OffHeapStoredObject.java:659)
at
org.apache.geode.internal.offheap.OffHeapStoredObject.release(OffHeapStoredObject.java:373)
at
org.apache.geode.internal.offheap.OffHeapHelper.releaseAndTrackOwner(OffHeapHelper.java:138)
at
org.apache.geode.internal.cache.wan.GatewaySenderEventImpl.release(GatewaySenderEventImpl.java:1213)
at
org.apache.geode.internal.cache.wan.parallel.ParallelGatewaySenderQueue.remove(ParallelGatewaySenderQueue.java:1096)
at
org.apache.geode.internal.cache.wan.parallel.ParallelGatewaySenderQueue.remove(ParallelGatewaySenderQueue.java:1531)
at
org.apache.geode.internal.cache.wan.AbstractGatewaySenderEventProcessor.eventQueueRemove(AbstractGatewaySenderEventProcessor.java:231)
at
org.apache.geode.internal.cache.wan.AbstractGatewaySenderEventProcessor.handleSuccessBatchAck(AbstractGatewaySenderEventProcessor.java:981)
at
org.apache.geode.internal.cache.wan.GatewaySenderEventRemoteDispatcher$AckReaderThread.run(GatewaySenderEventRemoteDispatcher.java:636)
{noformat}
This exception shows that the {{AckReaderThread}} was processing a successful
batch acknowledgement, and an {{IllegalStateException}} was thrown while
releasing a {{GatewaySenderEventImpl}} from off-heap memory. This caused the
{{AckReaderThread}} to shut down. It looks like the {{GatewaySenderEventImpl}}
had already been released and is being released again. I'm not sure how the
{{GatewaySenderEventImpl}} got into this state, but the {{AckReaderThread}}
should not shut down because of this {{IllegalStateException}}.
The code in question is in the finally block of
{{ParallelGatewaySenderQueue.remove}}:
{noformat}
} finally {
event.release();
}
{noformat}
The test run is here:
/export/monaco1/users/lhughes/xfer/wanconflationPersist-0107-091328
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
