[
https://issues.apache.org/jira/browse/GEODE-10308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17543359#comment-17543359
]
Jianxia Chen commented on GEODE-10308:
--------------------------------------
Looking the standard error and the test code, it is a test issue.
The root cause is at end of the test (tearDown), the client cache is closed too
early before stopping the Tomcat container. In
Tomcat8SessionsClientServerDUnitTest:
{code:java}
@After
public void tearDown() {
clientCache.close();
clientCache = null;
port = -1;
server.stopContainer();
server = null;
servlet = null;
sessionManager = null;
region = null;
} {code}
In the standard error:
{code:java}
SEVERE: A child container failed during stop
java.util.concurrent.ExecutionException:
org.apache.catalina.LifecycleException: A child container failed during stop
at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
at
org.apache.catalina.core.ContainerBase.stopInternal(ContainerBase.java:990)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
at
org.apache.catalina.core.StandardService.stopInternal(StandardService.java:486)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
at
org.apache.catalina.core.StandardServer.stopInternal(StandardServer.java:787)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
at org.apache.catalina.startup.Tomcat.stop(Tomcat.java:450)
at
org.apache.geode.modules.session.EmbeddedTomcat8.stopContainer(EmbeddedTomcat8.java:95)
at
org.apache.geode.modules.session.Tomcat8SessionsClientServerDUnitTest.tearDown(Tomcat8SessionsClientServerDUnitTest.java:111)
Caused by: org.apache.catalina.LifecycleException: A child container failed
during stop
at
org.apache.catalina.core.ContainerBase.stopInternal(ContainerBase.java:997)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
at
org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1428)
at
org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1417)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
... 1 more {code}
Because of CacheClosedException :
{code:java}
java.util.concurrent.ExecutionException:
org.apache.catalina.LifecycleException: Failed to stop component
[Tomcat8DeltaSessionManager[container=StandardEngine[localEngine].StandardHost[localhost].StandardContext[];
regionName=gemfire_modules_sessions; regionAttributesId=PARTITION_REDUNDANT]]
at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
at
org.apache.catalina.core.ContainerBase.stopInternal(ContainerBase.java:990)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
at
org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1428)
at
org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1417)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.catalina.LifecycleException: Failed to stop component
[Tomcat8DeltaSessionManager[container=StandardEngine[localEngine].StandardHost[localhost].StandardContext[];
regionName=gemfire_modules_sessions; regionAttributesId=PARTITION_REDUNDANT]]
at
org.apache.catalina.util.LifecycleBase.handleSubClassException(LifecycleBase.java:440)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:267)
at
org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5398)
... 7 more
Caused by: org.apache.geode.cache.CacheClosedException: The cache is closed.
at
org.apache.geode.internal.cache.GemFireCacheImpl$Stopper.generateCancelledException(GemFireCacheImpl.java:5207)
at
org.apache.geode.CancelCriterion.checkCancelInProgress(CancelCriterion.java:83)
at
org.apache.geode.internal.cache.LocalRegion.checkRegionDestroyed(LocalRegion.java:7382)
at
org.apache.geode.internal.cache.LocalRegion.checkReadiness(LocalRegion.java:2788)
at
org.apache.geode.internal.cache.AbstractRegion.localClear(AbstractRegion.java:514)
at
org.apache.geode.modules.session.catalina.DeltaSessionManager.clearLocalCache(DeltaSessionManager.java:686)
at
org.apache.geode.modules.session.catalina.DeltaSessionManager.unload(DeltaSessionManager.java:579)
at
org.apache.geode.modules.session.catalina.Tomcat8DeltaSessionManager.stopInternal(Tomcat8DeltaSessionManager.java:93)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
... 8 more{code}
Similarly, another CacheClosedException during function execution when stopping
a container at the end of the test, also because the client cache is closed,
but the timer thread is still running:
{code:java}
May 07, 2022 3:26:39 AM org.apache.geode.modules.session.EmbeddedTomcat8
stopContainer
WARNING: Cannot Stop TomcatA child container failed during stop
Exception in thread "Timer for
Tomcat8DeltaSessionManager[container=StandardEngine[localEngine].StandardHost[localhost].StandardContext[];
regionName=gemfire_modules_sessions; regionAttributesId=PARTITION_REDUNDANT]"
org.apache.geode.cache.CacheClosedException: The cache is closed.
at
org.apache.geode.internal.cache.GemFireCacheImpl$Stopper.generateCancelledException(GemFireCacheImpl.java:5207)
at
org.apache.geode.internal.cache.LocalRegion$Stopper.generateCancelledException(LocalRegion.java:11342)
at
org.apache.geode.CancelCriterion.checkCancelInProgress(CancelCriterion.java:83)
at
org.apache.geode.CancelCriterion.checkCancelInProgress(CancelCriterion.java:87)
at
org.apache.geode.internal.cache.execute.InternalFunctionExecutionServiceImpl.onRegion(InternalFunctionExecutionServiceImpl.java:122)
at
org.apache.geode.cache.execute.FunctionService.onRegion(FunctionService.java:79)
at
org.apache.geode.modules.session.catalina.ClientServerSessionCache.getExecutionForFunctionOnRegionWithFilter(ClientServerSessionCache.java:283)
at
org.apache.geode.modules.session.catalina.ClientServerSessionCache.touchSessions(ClientServerSessionCache.java:101)
at
org.apache.geode.modules.session.catalina.DeltaSessionManager$1.run(DeltaSessionManager.java:534)
at java.base/java.util.TimerThread.mainLoop(Timer.java:556)
at java.base/java.util.TimerThread.run(Timer.java:506) {code}
If we close the client cache after stopping the Tomcat container, there will be
no more CacheClosedException. The test then have warnings about memory leak,
for example:
{code:java}
May 27, 2022 5:28:14 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-nio-20012"]
May 27, 2022 5:28:14 PM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service [Tomcat]
May 27, 2022 5:28:14 PM org.apache.catalina.loader.WebappClassLoaderBase
clearReferencesThreads
WARNING: The web application [ROOT] appears to have started a thread named
[poolTimer-DEFAULT-1] but has failed to stop it. This is very likely to create
a memory leak. Stack trace of thread:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
May 27, 2022 5:28:14 PM org.apache.catalina.loader.WebappClassLoaderBase
clearReferencesThreads
WARNING: The web application [ROOT] appears to have started a thread named
[poolTimer-DEFAULT-2] but has failed to stop it. This is very likely to create
a memory leak. Stack trace of thread:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
May 27, 2022 5:28:14 PM org.apache.catalina.loader.WebappClassLoaderBase
clearReferencesThreads
WARNING: The web application [ROOT] appears to have started a thread named
[ClientMembership Event Invoker1] but has failed to stop it. This is very
likely to create a memory leak. Stack trace of thread:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
May 27, 2022 5:28:14 PM org.apache.catalina.loader.WebappClassLoaderBase
clearReferencesThreads
WARNING: The web application [ROOT] appears to have started a thread named
[poolLoadConditioningMonitor-DEFAULT1] but has failed to stop it. This is very
likely to create a memory leak. Stack trace of thread:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
May 27, 2022 5:28:14 PM org.apache.catalina.loader.WebappClassLoaderBase
clearReferencesThreads
WARNING: The web application [ROOT] appears to have started a thread named
[Cache Client Updater Thread on t2000-z4(server-1:77808)<v1>:41005 port 20010]
but has failed to stop it. This is very likely to create a memory leak. Stack
trace of thread:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
java.net.SocketInputStream.read(SocketInputStream.java:171)
java.net.SocketInputStream.read(SocketInputStream.java:141)
org.apache.geode.internal.cache.tier.sockets.Message.fetchHeader(Message.java:830)
org.apache.geode.internal.cache.tier.sockets.Message.readHeaderAndBody(Message.java:677)
org.apache.geode.internal.cache.tier.sockets.Message.receiveWithHeaderReadTimeout(Message.java:1141)
org.apache.geode.internal.cache.tier.sockets.CacheClientUpdater.processMessages(CacheClientUpdater.java:1595)
org.apache.geode.internal.cache.tier.sockets.CacheClientUpdater.run(CacheClientUpdater.java:486)
May 27, 2022 5:28:14 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-nio-20012"]
May 27, 2022 5:28:14 PM org.apache.geode.modules.session.EmbeddedTomcat8
stopContainer
INFO: Stopped container {code}
These warnings are reported when stopping the Tomcat container. At this time,
the client cache is not closed yet. Once the client cache is closed, it will
stop those threads. So I wouldn't worry about memory leak.
> CI Failure: Tomcat8SessionsClientServerDUnitTest.testSessionExpiration1 failed
> ------------------------------------------------------------------------------
>
> Key: GEODE-10308
> URL: https://issues.apache.org/jira/browse/GEODE-10308
> Project: Geode
> Issue Type: Bug
> Components: tests
> Affects Versions: 1.16.0
> Reporter: Eric Shu
> Assignee: Jianxia Chen
> Priority: Major
> Labels: needsTriage
>
> {noformat}
> org.apache.geode.cache.CacheClosedException: The cache is closed.
> at
> org.apache.geode.internal.cache.GemFireCacheImpl$Stopper.generateCancelledException(GemFireCacheImpl.java:5207)
> at
> org.apache.geode.internal.cache.LocalRegion$Stopper.generateCancelledException(LocalRegion.java:11342)
> at
> org.apache.geode.CancelCriterion.checkCancelInProgress(CancelCriterion.java:83)
> at
> org.apache.geode.CancelCriterion.checkCancelInProgress(CancelCriterion.java:87)
> at
> org.apache.geode.internal.cache.execute.InternalFunctionExecutionServiceImpl.onRegion(InternalFunctionExecutionServiceImpl.java:122)
> at
> org.apache.geode.cache.execute.FunctionService.onRegion(FunctionService.java:79)
> at
> org.apache.geode.modules.session.catalina.ClientServerSessionCache.getExecutionForFunctionOnRegionWithFilter(ClientServerSessionCache.java:283)
> at
> org.apache.geode.modules.session.catalina.ClientServerSessionCache.touchSessions(ClientServerSessionCache.java:101)
> at
> org.apache.geode.modules.session.catalina.DeltaSessionManager$1.run(DeltaSessionManager.java:534)
> at java.util.TimerThread.mainLoop(Timer.java:556)
> at java.util.TimerThread.run(Timer.java:506)
> {noformat}
> Artifacts can be found here:
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-main/jobs/distributed-test-openjdk11/builds/334
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
