[
https://issues.apache.org/jira/browse/GEODE-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979207#comment-15979207
]
ASF GitHub Bot commented on GEODE-2808:
---------------------------------------
GitHub user upthewaterspout opened a pull request:
https://github.com/apache/geode/pull/472
GEODE-2808 - Fixing lock ordering issues in DeltaSession
Region expiration of sessions and explicit expiration of sessions had
lock ordering issues. Fixing the code so that expiration goes through
the region entry lock first, before getting the lock on StandardSession.
Adding a workaround for the fact that liferay calls removeAttribute
from within session expiration by ignoreing remoteAttribute calls during
expiration.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/upthewaterspout/incubator-geode
feature/GEODE-2808
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/geode/pull/472.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #472
----
commit 8667c805d4a7b77cca2c883e34200169beab585a
Author: Dan Smith <upthewatersp...@apache.org>
Date: 2017-04-21T18:36:24Z
GEODE-2808 - Fixing lock ordering issues in DeltaSession
Region expiration of sessions and explicit expiration of sessions had
lock ordering issues. Fixing the code so that expiration goes through
the region entry lock first, before getting the lock on StandardSession.
Adding a workaround for the fact that liferay calls removeAttribute
from within session expiration by ignoreing remoteAttribute calls during
expiration.
----
> Deadlock in tomcat session module during session expiration
> -----------------------------------------------------------
>
> Key: GEODE-2808
> URL: https://issues.apache.org/jira/browse/GEODE-2808
> Project: Geode
> Issue Type: Bug
> Components: http session
> Reporter: Dan Smith
> Assignee: Dan Smith
>
> We observed a deadlock in the tomcat session state replication module due to
> the way we do expiration. Here are the threads that are involved:
> {noformat}
> Found one Java-level deadlock:
> =============================
> "http-nio-21064-exec-2":
> waiting to lock monitor 0x00007f9a04bde298 (object 0x00000007947da1f0, a
> org.apache.geode.internal.cache.VersionedStatsLRURegionEntryHeapObjectKey),
> which is held by "Timer-1"
> "Timer-1":
> waiting to lock monitor 0x00007f99a4004e38 (object 0x00000007947da240, a
> org.apache.geode.modules.session.catalina.DeltaSession8),
> which is held by "http-nio-21064-exec-2"
> Java stack information for the threads listed above:
> ===================================================
> "http-nio-21064-exec-2":
> at
> org.apache.geode.internal.cache.AbstractRegionMap.destroy(AbstractRegionMap.java:1228)
> - waiting to lock <0x00000007947da1f0> (a
> org.apache.geode.internal.cache.VersionedStatsLRURegionEntryHeapObjectKey)
> at
> org.apache.geode.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:6785)
> at
> org.apache.geode.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:6762)
> at
> org.apache.geode.internal.cache.LocalRegionDataView.destroyExistingEntry(LocalRegionDataView.java:55)
> at
> org.apache.geode.internal.cache.LocalRegion.basicDestroy(LocalRegion.java:6724)
> at
> org.apache.geode.internal.cache.LocalRegion.validatedDestroy(LocalRegion.java:1110)
> at
> org.apache.geode.internal.cache.LocalRegion.destroy(LocalRegion.java:1095)
> at
> org.apache.geode.internal.cache.AbstractRegion.destroy(AbstractRegion.java:271)
> at
> org.apache.geode.modules.session.catalina.AbstractSessionCache.destroySession(AbstractSessionCache.java:72)
> at
> org.apache.geode.modules.session.catalina.DeltaSessionManager.remove(DeltaSessionManager.java:405)
> at
> org.apache.catalina.session.StandardSession.expire(StandardSession.java:850)
> - locked <0x00000007947da240> (a
> org.apache.geode.modules.session.catalina.DeltaSession8)
> at
> org.apache.catalina.session.StandardSession.isValid(StandardSession.java:682)
> at org.apache.catalina.connector.Request.doGetSession(Request.java:2917)
> at org.apache.catalina.connector.Request.getSession(Request.java:2367)
> at
> org.apache.geode.modules.session.catalina.CommitSessionValve.invoke(CommitSessionValve.java:50)
> at
> org.apache.geode.modules.session.catalina.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:46)
> at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:474)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
> at
> org.apache.geode.modules.session.catalina.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:46)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:349)
> at
> org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:783)
> at
> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
> at
> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:789)
> at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1437)
> at
> org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
> - locked <0x00000007948d6870> (a
> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> at java.lang.Thread.run(Thread.java:745)
> "Timer-1":
> at
> org.apache.catalina.session.StandardSession.expire(StandardSession.java:791)
> - waiting to lock <0x00000007947da240> (a
> org.apache.geode.modules.session.catalina.DeltaSession8)
> at
> org.apache.catalina.session.StandardSession.expire(StandardSession.java:766)
> at
> org.apache.geode.modules.session.catalina.DeltaSession8.processExpired(DeltaSession8.java:320)
> at
> org.apache.geode.modules.session.catalina.callback.SessionExpirationCacheListener.afterDestroy(SessionExpirationCacheListener.java:56)
> at
> org.apache.geode.internal.cache.EnumListenerEvent$AFTER_DESTROY.dispatchEvent(EnumListenerEvent.java:151)
> at
> org.apache.geode.internal.cache.LocalRegion.dispatchEvent(LocalRegion.java:8751)
> at
> org.apache.geode.internal.cache.LocalRegion.dispatchListenerEvent(LocalRegion.java:7245)
> at
> org.apache.geode.internal.cache.LocalRegion.invokeDestroyCallbacks(LocalRegion.java:7052)
> at
> org.apache.geode.internal.cache.EntryEventImpl.invokeCallbacks(EntryEventImpl.java:2293)
> at
> org.apache.geode.internal.cache.AbstractRegionEntry.dispatchListenerEvents(AbstractRegionEntry.java:140)
> at
> org.apache.geode.internal.cache.LocalRegion.basicDestroyPart2(LocalRegion.java:6992)
> at
> org.apache.geode.internal.cache.AbstractRegionMap.destroy(AbstractRegionMap.java:1419)
> - locked <0x00000007947da1f0> (a
> org.apache.geode.internal.cache.VersionedStatsLRURegionEntryHeapObjectKey)
> at
> org.apache.geode.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:6785)
> at
> org.apache.geode.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:6762)
> at
> org.apache.geode.internal.cache.LocalRegionDataView.destroyExistingEntry(LocalRegionDataView.java:55)
> at
> org.apache.geode.internal.cache.LocalRegion.basicDestroy(LocalRegion.java:6724)
> at
> org.apache.geode.internal.cache.LocalRegion.expireDestroy(LocalRegion.java:6864)
> at
> org.apache.geode.internal.cache.EntryExpiryTask.destroy(EntryExpiryTask.java:119)
> at
> org.apache.geode.internal.cache.ExpiryTask.expire(ExpiryTask.java:278)
> at
> org.apache.geode.internal.cache.ExpiryTask.expire(ExpiryTask.java:232)
> at
> org.apache.geode.internal.cache.EntryExpiryTask.basicPerformTimeout(EntryExpiryTask.java:305)
> at
> org.apache.geode.internal.cache.LocalRegion.performExpiryTimeout(LocalRegion.java:801)
> at
> org.apache.geode.internal.cache.EntryExpiryTask.performTimeout(EntryExpiryTask.java:219)
> at
> org.apache.geode.internal.cache.ExpiryTask.runInThreadPool(ExpiryTask.java:370)
> at org.apache.geode.internal.cache.ExpiryTask.run2(ExpiryTask.java:318)
> at
> org.apache.geode.internal.SystemTimer$SystemTimerTask.run(SystemTimer.java:444)
> at java.util.TimerThread.mainLoop(Timer.java:555)
> at java.util.TimerThread.run(Timer.java:505)
> Found 1 deadlock.
> {noformat}
> The basic issue is that we configure expiration on the geode region to expire
> sessions, and the tomcat container will also expire sessions when the get to
> old. But those two threads get locks in a different order:
> Tomcat expiration (http-nio-21064-exec-2)
> # sync session in the tomcat code (StandardSession.expire)
> # sync region entry in the geode code (AbstractRegionMap.destroy)
> region expiration
> # sync region entry in the geode code (AbstractRegionMap.destroy
> # sync session in geode code (StandardSession.expire)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
