[
https://issues.apache.org/jira/browse/GEODE-8990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17338653#comment-17338653
]
Eric Shu commented on GEODE-8990:
---------------------------------
After some debugging, I was wrong on the previous assessment that registering
interest was the cause of this flakiness. The reason is that value was modified
on the server only, but it is not entry operation that will be sent to client
cache.
serverVM.invoke("set remote session key", () -> {
final InternalCache cache = ClusterStartupRule.memberStarter.getCache();
Region<String, HttpSession> region =
cache.getRegion("gemfire_modules_sessions");
region.values().forEach(session -> session.setAttribute(key,
remoteValue));
});
With the new debugging, I found out the cause for this flakiness is caused by
the memory issue, the entry put was evicted immediately. This will cause client
gets cache miss and get from server of the newer value -- causing the test to
fail.
{noformat}
[info 2021/05/03 15:26:07.093 PDT <qtp2143437117-27> tid=0x1b] makeTombstone
stack for 15C3570A-13DD-45A0-932D-74C4CEB0958A-GF for entry
VersionedStatsLRURegionEntryHeapObjectKey@1d9493ec
(key=15C3570A-13DD-45A0-932D-74C4CEB0958A-GF;
rawValue=[id=15C3570A-13DD-45A0-932D-74C4CEB0958A-GF, isNew=false,
isValid=true, hasDelta=true, lastAccessedTime=1620080767081,
jvmOwnerId=default]; version={v3; rv3;
mbr=192.168.0.14(server-1:92400)<v1>:41001;
time=1620080767088};member=192.168.0.14(server-1:92400)<v1>:41001)
java.lang.Exception
at
org.apache.geode.internal.cache.entries.AbstractRegionEntry.makeTombstone(AbstractRegionEntry.java:258)
at
org.apache.geode.internal.cache.entries.AbstractRegionEntry.destroy(AbstractRegionEntry.java:880)
at
org.apache.geode.internal.cache.map.RegionMapDestroy.destroyEntry(RegionMapDestroy.java:740)
at
org.apache.geode.internal.cache.map.RegionMapDestroy.destroyExistingEntry(RegionMapDestroy.java:398)
at
org.apache.geode.internal.cache.map.RegionMapDestroy.handleExistingRegionEntry(RegionMapDestroy.java:244)
at
org.apache.geode.internal.cache.map.RegionMapDestroy.destroy(RegionMapDestroy.java:152)
at
org.apache.geode.internal.cache.AbstractRegionMap.destroy(AbstractRegionMap.java:972)
at
org.apache.geode.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:6514)
at
org.apache.geode.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:6488)
at
org.apache.geode.internal.cache.LocalRegion.evictDestroy(LocalRegion.java:6633)
at
org.apache.geode.internal.cache.VMLRURegionMap.evictEntry(VMLRURegionMap.java:240)
at
org.apache.geode.internal.cache.VMLRURegionMap.lruUpdateCallback(VMLRURegionMap.java:432)
at
org.apache.geode.internal.cache.map.RegionMapPut.lruUpdateCallbackIfNotCleared(RegionMapPut.java:330)
at
org.apache.geode.internal.cache.map.RegionMapPut.finishEviction(RegionMapPut.java:320)
at
org.apache.geode.internal.cache.map.RegionMapPut.doAfterCompletionActions(RegionMapPut.java:312)
at
org.apache.geode.internal.cache.map.AbstractRegionMapPut.doPut(AbstractRegionMapPut.java:185)
at
org.apache.geode.internal.cache.map.AbstractRegionMapPut.runWhileLockedForCacheModification(AbstractRegionMapPut.java:119)
at
org.apache.geode.internal.cache.map.RegionMapPut.runWhileLockedForCacheModification(RegionMapPut.java:161)
at
org.apache.geode.internal.cache.map.AbstractRegionMapPut.put(AbstractRegionMapPut.java:169)
at
org.apache.geode.internal.cache.AbstractRegionMap.basicPut(AbstractRegionMap.java:2040)
at
org.apache.geode.internal.cache.LocalRegion.virtualPut(LocalRegion.java:5619)
at
org.apache.geode.internal.cache.LocalRegion.virtualPut(LocalRegion.java:5597)
at
org.apache.geode.internal.cache.LocalRegionDataView.putEntry(LocalRegionDataView.java:157)
at
org.apache.geode.internal.cache.LocalRegion.basicPut(LocalRegion.java:5055)
at
org.apache.geode.internal.cache.LocalRegion.validatedPut(LocalRegion.java:1654)
at
org.apache.geode.internal.cache.LocalRegion.put(LocalRegion.java:1641)
at
org.apache.geode.internal.cache.AbstractRegion.put(AbstractRegion.java:445)
at
org.apache.geode.modules.session.internal.filter.GemfireSessionManager.wrapSession(GemfireSessionManager.java:238)
at
org.apache.geode.modules.session.filter.SessionCachingFilter$RequestWrapper.getSession(SessionCachingFilter.java:179)
at
org.apache.geode.modules.session.filter.SessionCachingFilter$RequestWrapper.getSession(SessionCachingFilter.java:148)
at
org.apache.geode.modules.session.CommandServlet.doGet(CommandServlet.java:63)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at
org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1452)
at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791)
at
org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
at
org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:228)
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at
org.apache.geode.modules.session.filter.SessionCachingFilter.doFilter(SessionCachingFilter.java:376)
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
[info 2021/05/03 15:26:07.293 PDT <qtp2143437117-18> tid=0x12]
getDeserializedValue gets regionEntry
VersionedStatsLRURegionEntryHeapObjectKey@1d9493ec
(key=15C3570A-13DD-45A0-932D-74C4CEB0958A-GF; rawValue=TOMBSTONE; version={v3;
rv3; mbr=192.168.0.14(server-1:92400)<v1>:41001;
time=1620080767088};member=192.168.0.14(server-1:92400)<v1>:41001)
java.lang.Exception
at
org.apache.geode.internal.cache.LocalRegion.getDeserializedValue(LocalRegion.java:1188)
at
org.apache.geode.internal.cache.LocalRegionDataView.getDeserializedValue(LocalRegionDataView.java:45)
at
org.apache.geode.internal.cache.LocalRegion.get(LocalRegion.java:1373)
at
org.apache.geode.internal.cache.LocalRegion.get(LocalRegion.java:1324)
at
org.apache.geode.internal.cache.LocalRegion.get(LocalRegion.java:1309)
at
org.apache.geode.internal.cache.AbstractRegion.get(AbstractRegion.java:439)
at
org.apache.geode.modules.session.internal.filter.GemfireSessionManager.getSession(GemfireSessionManager.java:192)
at
org.apache.geode.modules.session.filter.SessionCachingFilter$RequestWrapper.getSession(SessionCachingFilter.java:165)
at
org.apache.geode.modules.session.filter.SessionCachingFilter$RequestWrapper.getSession(SessionCachingFilter.java:148)
at
org.apache.geode.modules.session.CommandServlet.doGet(CommandServlet.java:71)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
{noformat}
This leads to a get from server:
vm1] [info 2021/05/03 15:26:07.309 PDT server-1 <ServerConnection on port 59596
Thread 2> tid=0x4a] Get 70 invoked stack
[vm1] java.lang.Exception
[vm1] at
org.apache.geode.internal.cache.tier.sockets.command.Get70.cmdExecute(Get70.java:65)
[vm1] at
org.apache.geode.internal.cache.tier.sockets.BaseCommand.execute(BaseCommand.java:185)
[vm1] at
org.apache.geode.internal.cache.tier.sockets.ServerConnection.doNormalMessage(ServerConnection.java:880)
[vm1] at
org.apache.geode.internal.cache.tier.sockets.ServerConnection.doOneMessage(ServerConnection.java:1022)
[vm1] at
org.apache.geode.internal.cache.tier.sockets.ServerConnection.run(ServerConnection.java:1275)
[vm1] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[vm1] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[vm1] at
org.apache.geode.internal.cache.tier.sockets.AcceptorImpl.lambda$initializeServerConnectionThreadPool$3(AcceptorImpl.java:690)
[vm1] at
org.apache.geode.logging.internal.executors.LoggingThreadFactory.lambda$newThread$0(LoggingThreadFactory.java:120)
[vm1] at java.lang.Thread.run(Thread.java:748)
Need to find out what causes the memory issue as there is only 1 entry in the
client cache. And see how to solve this test issue.
> CI Failure: Jetty9CachingClientServerTest.shouldCacheSessionOnClient
> --------------------------------------------------------------------
>
> Key: GEODE-8990
> URL: https://issues.apache.org/jira/browse/GEODE-8990
> Project: Geode
> Issue Type: Bug
> Affects Versions: 1.15.0
> Reporter: Donal Evans
> Assignee: Benjamin P Ross
> Priority: Major
> Labels: GeodeOperationAPI, pull-request-available
> Fix For: 1.15.0
>
>
> {noformat}
> org.apache.geode.session.tests.Jetty9CachingClientServerTest >
> shouldCacheSessionOnClient FAILED
> org.junit.ComparisonFailure: expected:<[Foo]> but was:<[bogus]>
> at org.junit.Assert.assertEquals(Assert.java:117)
> at org.junit.Assert.assertEquals(Assert.java:146)
> at
> org.apache.geode.session.tests.Jetty9CachingClientServerTest.shouldCacheSessionOnClient(Jetty9CachingClientServerTest.java:65)
> 371 tests completed, 1 failed
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-= Test Results URI
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> http://files.apachegeode-ci.info/builds/apache-develop-main/1.15.0-build.0023/test-results/distributedTest/1614646829/
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> Test report artifacts from this job are available at:
> http://files.apachegeode-ci.info/builds/apache-develop-main/1.15.0-build.0023/test-artifacts/1614646829/distributedtestfiles-OpenJDK8-1.15.0-build.0023.tgz
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
