Mark, Thanks for the other advice. I have recommended upgrading as a possible solution. I do see the inconsistencies you are referring to, and I will try to generate a thread dump with the issue happening using a different tool, but the issue is intermittent. Thank you much for the help, I'll msg when I know more.
-Andrew On Tue, Nov 27, 2018 at 1:24 PM Andrew Carr <andrewlanec...@gmail.com> wrote: > Tomcat 8.5.24 > Java - (1.8.0-zing_16.12.3.0-b2-product-azlinuxM-X86_64 mixed mode [Mon > Aug 13 05:18:16 PM UTC 2018]): > > On Wed, Nov 21, 2018 at 6:49 AM Mark Thomas <ma...@apache.org> wrote: > >> And which version of Java? >> >> See also comments below. >> >> On 20/11/2018 22:16, Mark Thomas wrote: >> > Which version of Tomcat are you using? >> > >> > Mark >> > >> > >> > On 20/11/2018 21:02, Andrew Carr wrote: >> >> Wait, I think I might see what I did, let me try again. >> >> >> >> Found one Java-level deadlock: >> >> ============================= >> >> "Thread-96": >> >> waiting to lock monitor 0x3a0d6938 (object 0x80048462700, a >> >> java.util.concurrent.ConcurrentHashMap), >> >> which is held by "Thread-29" >> >> The above thread can be ignored as it is blocked by the deadlock but >> isn't part of it. >> >> >> "Thread-29": >> >> waiting to lock monitor 0x3a0d58b0 (object 0x1000bbc1dd20, a >> >> java.lang.Object), >> >> which is held by "Event Manager Thread_QuartzSchedulerThread" >> >> I can see all of this in the provided stack traces. >> >> >> >> "Event Manager Thread_QuartzSchedulerThread": >> >> waiting to lock monitor 0x3803a2c0 (object 0x800674b7830, a >> >> org.apache.catalina.loader.ParallelWebappClassLoader), >> >> which is held by "Thread-29" >> >> There is no evidence in this stack trace that Thread 29 is holding that >> lock. >> >> There is no evidence in this stack trace that the Quartz thread is >> waiting for that lock. >> >> Hence my comment that the stack traces are not consistent with the >> reported deadlock. >> >> We really need to see where the lock on the ParallelWebappClassLoader >> object is being held. Maybe try getting a thread dump with a different >> tool? >> >> Until we can see where the locks are being held there isn't anything we >> can do. >> >> Generally, I'd recommend an update to the latest Tomcat point release >> and latest Java point release for whichever versions you are using just >> in case there is a JRE bug or similar contributing to this. I don't see >> anything related in the Tomcat changelog but I'd still always recommend >> updating. >> >> Mark >> >> >> >> >> >> >> "VM Thread" os_prio=0 tid=0x440000c00000 nid=0x1cbc >> >> >> >> "GC task thread#0 (GenPauselessGC)" os_prio=0 tid=0x440000200000 >> >> nid=0x1cb6 >> >> >> >> "GC task thread#1 (GenPauselessGC)" os_prio=0 tid=0x440000400000 >> >> nid=0x1cb7 >> >> >> >> "GC task thread#0 (GenPauselessGC)" os_prio=0 tid=0x440000600000 >> >> nid=0x1cb8 >> >> >> >> "VM Periodic Task Thread" os_prio=0 tid=0x440002e00000 nid=0x1ccd >> >> >> >> JNI global references: 98048 >> >> >> >> >> >> Java stack information for the threads listed above: >> >> =================================================== >> >> "Thread-96": >> >> at >> >> >> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:187) >> >> >> >> - waiting to lock <0x0000080048462700> (a >> >> java.util.concurrent.ConcurrentHashMap) >> >> at >> >> >> org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:486) >> >> >> >> at >> >> >> org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:432) >> >> >> >> at >> >> >> org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:403) >> >> >> >> at >> >> >> org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:389) >> >> >> >> at >> >> >> org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1002) >> >> >> >> at >> >> >> org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:345) >> >> >> >> at >> >> >> org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:340) >> >> >> >> at >> >> >> org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1093) >> >> >> >> at >> >> >> com.XXXXXXX.tenantprovisioning.component.tool.falcon.cluster.provisioning.ClusterProvisioningThread.launchProvisioning(ClusterProvisioningThread.java:75) >> >> >> >> at >> >> >> com.XXXXXXX.tenantprovisioning.component.tool.falcon.cluster.provisioning.ClusterProvisioningThread.run(ClusterProvisioningThread.java:63) >> >> >> >> at java.lang.Thread.run(Thread.java:745) >> >> "Thread-29": >> >> at >> >> >> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1152) >> >> >> >> - waiting to lock <0x00001000bbc1dd20> (a java.lang.Object) >> >> at >> >> >> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1119) >> >> >> >> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:488) >> >> at >> >> >> com.XXXXXXX.wfc.commonapp.eventmanager.business.tasksched.ScheduleEngine.generateJobDetail(ScheduleEngine.java:982) >> >> >> >> at >> >> >> com.XXXXXXX.wfc.commonapp.eventmanager.business.tasksched.ScheduleEngine.scheduleAtFixedRate(ScheduleEngine.java:107) >> >> >> >> at >> >> >> com.XXXXXXX.wfc.commonapp.eventmanager.business.tasksched.ScheduleEngine.scheduleMinutely(ScheduleEngine.java:1316) >> >> >> >> at >> >> >> com.XXXXXXX.wfc.commonapp.eventmanager.business.tasksched.ScheduleEngineService.scheduleMinutely(ScheduleEngineService.java:493) >> >> >> >> - locked <0x00001000aa2752d8> (a >> >> >> com.XXXXXXX.wfc.commonapp.eventmanager.business.tasksched.KQuartzEngineService) >> >> >> >> at >> >> >> com.XXXXXXX.wfc.commonapp.eventmanager.business.engine.EvtMgrSched.scheduleTask(EvtMgrSched.java:1057) >> >> >> >> at >> >> >> com.XXXXXXX.wfc.commonapp.eventmanager.business.engine.EvtMgrSched.scheduleTask(EvtMgrSched.java:876) >> >> >> >> at >> >> >> com.XXXXXXX.wfc.commonapp.eventmanager.business.startup.EvtMgrStartUp.tenantBasedSchedAndDeleteTask(EvtMgrStartUp.java:186) >> >> >> >> at >> >> >> com.XXXXXXX.wfc.commonapp.eventmanager.business.startup.EvtMgrStartUp.initialize(EvtMgrStartUp.java:138) >> >> >> >> - locked <0x0000080058b5c5e8> (a java.lang.Class) >> >> at >> >> >> com.XXXXXXX.wfc.commonapp.eventmanager.business.startup.EvtMgrStartUp.startup(EvtMgrStartUp.java:73) >> >> >> >> at sun.reflect.GeneratedMethodAccessor311.invoke(Unknown Source) >> >> at >> >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >> >> >> at java.lang.reflect.Method.invoke(Method.java:498) >> >> at >> >> >> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:366) >> >> >> >> at >> >> >> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:311) >> >> >> >> at >> >> >> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:134) >> >> >> >> at >> >> >> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:409) >> >> >> >> at >> >> >> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1620) >> >> >> >> at >> >> >> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555) >> >> >> >> at >> >> >> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483) >> >> >> >> at >> >> >> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) >> >> >> >> at >> >> >> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) >> >> >> >> - locked <0x0000080048462700> (a >> java.util.concurrent.ConcurrentHashMap) >> >> at >> >> >> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) >> >> >> >> at >> >> >> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:220) >> >> >> >> at >> >> >> org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1018) >> >> >> >> at >> >> >> org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:345) >> >> >> >> at >> >> >> org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:340) >> >> >> >> at >> >> >> org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1093) >> >> >> >> at >> >> >> com.XXXXXXX.container.api.access.SpringContext.getBean(SpringContext.java:62) >> >> >> >> at >> >> >> com.XXXXXXX.wfc.platform.startup.framework.DeferredSpringServicesStarter.startDeferredStartupServices(DeferredSpringServicesStarter.java:83) >> >> >> >> at >> >> >> com.XXXXXXX.wfc.platform.startup.framework.DeferredSpringServicesStarter.startup(DeferredSpringServicesStarter.java:64) >> >> >> >> at sun.reflect.GeneratedMethodAccessor311.invoke(Unknown Source) >> >> at >> >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >> >> >> at java.lang.reflect.Method.invoke(Method.java:498) >> >> at >> >> >> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:366) >> >> >> >> at >> >> >> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:311) >> >> >> >> at >> >> >> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:134) >> >> >> >> at >> >> >> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:409) >> >> >> >> at >> >> >> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1620) >> >> >> >> at >> >> >> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555) >> >> >> >> at >> >> >> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483) >> >> >> >> at >> >> >> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) >> >> >> >> at >> >> >> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) >> >> >> >> - locked <0x0000080048462700> (a >> java.util.concurrent.ConcurrentHashMap) >> >> at >> >> >> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) >> >> >> >> at >> >> >> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:220) >> >> >> >> at >> >> >> org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1018) >> >> >> >> at >> >> >> org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:345) >> >> >> >> at >> >> >> org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:340) >> >> >> >> at >> >> >> org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1093) >> >> >> >> at >> >> >> com.XXXXXXX.container.api.access.SpringContext.getBean(SpringContext.java:62) >> >> >> >> at >> >> >> com.XXXXXXX.wfc.platform.startup.framework.WFCStartUp.startDeferredServices(WFCStartUp.java:1264) >> >> >> >> at >> >> >> com.XXXXXXX.wfc.platform.startup.framework.WFCStartUp.WFCInit(WFCStartUp.java:1159) >> >> >> >> at >> >> >> com.XXXXXXX.wfc.platform.startup.framework.WFCStartUp.WFCInit(WFCStartUp.java:1277) >> >> >> >> at >> >> >> com.XXXXXXX.wfc.platform.startup.framework.WFCStartUp$WFCInitThread.run(WFCStartUp.java:1429) >> >> >> >> "Event Manager Thread_QuartzSchedulerThread": >> >> at java.lang.ClassLoader.defineClass1(Native Method) >> >> at java.lang.ClassLoader.defineClass(ClassLoader.java:769) >> >> at >> >> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) >> >> at >> >> >> org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:2283) >> >> >> >> - locked <0x00001000bbc1dd20> (a java.lang.Object) >> >> at >> >> >> org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:811) >> >> >> >> at >> >> >> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1260) >> >> >> >> - locked <0x00001000bbc1dd20> (a java.lang.Object) >> >> at >> >> >> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1119) >> >> >> >> at >> >> >> org.quartz.simpl.LoadingLoaderClassLoadHelper.loadClass(LoadingLoaderClassLoadHelper.java:59) >> >> >> >> at >> >> >> org.quartz.simpl.CascadingClassLoadHelper.loadClass(CascadingClassLoadHelper.java:99) >> >> >> >> at >> >> >> org.quartz.simpl.CascadingClassLoadHelper.loadClass(CascadingClassLoadHelper.java:138) >> >> >> >> at >> >> >> org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectJobDetail(StdJDBCDelegate.java:852) >> >> >> >> at >> >> >> org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveJob(JobStoreSupport.java:1385) >> >> >> >> at >> >> >> org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTrigger(JobStoreSupport.java:2818) >> >> >> >> at >> >> >> org.quartz.impl.jdbcjobstore.JobStoreSupport$40.execute(JobStoreSupport.java:2759) >> >> >> >> at >> >> >> org.quartz.impl.jdbcjobstore.JobStoreSupport$40.execute(JobStoreSupport.java:2757) >> >> >> >> at >> >> >> org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3799) >> >> >> >> at >> >> >> org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTriggers(JobStoreSupport.java:2756) >> >> >> >> at >> >> >> org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:272) >> >> >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: dev-h...@tomcat.apache.org >> >> > > -- > With Regards, > Andrew Carr > > e. andrewlanec...@gmail.com > w. andrew.c...@openlogic.com > c. 4239489206 > a. P.O. Box 1231, Greeneville, TN, 37744 > -- With Regards, Andrew Carr e. andrewlanec...@gmail.com w. andrew.c...@openlogic.com c. 4239489206 a. P.O. Box 1231, Greeneville, TN, 37744
