And which version of Java? See also comments below.
On 20/11/2018 22:16, Mark Thomas wrote: > Which version of Tomcat are you using? > > Mark > > > On 20/11/2018 21:02, Andrew Carr wrote: >> Wait, I think I might see what I did, let me try again. >> >> Found one Java-level deadlock: >> ============================= >> "Thread-96": >> waiting to lock monitor 0x3a0d6938 (object 0x80048462700, a >> java.util.concurrent.ConcurrentHashMap), >> which is held by "Thread-29" The above thread can be ignored as it is blocked by the deadlock but isn't part of it. >> "Thread-29": >> waiting to lock monitor 0x3a0d58b0 (object 0x1000bbc1dd20, a >> java.lang.Object), >> which is held by "Event Manager Thread_QuartzSchedulerThread" I can see all of this in the provided stack traces. >> "Event Manager Thread_QuartzSchedulerThread": >> waiting to lock monitor 0x3803a2c0 (object 0x800674b7830, a >> org.apache.catalina.loader.ParallelWebappClassLoader), >> which is held by "Thread-29" There is no evidence in this stack trace that Thread 29 is holding that lock. There is no evidence in this stack trace that the Quartz thread is waiting for that lock. Hence my comment that the stack traces are not consistent with the reported deadlock. We really need to see where the lock on the ParallelWebappClassLoader object is being held. Maybe try getting a thread dump with a different tool? Until we can see where the locks are being held there isn't anything we can do. Generally, I'd recommend an update to the latest Tomcat point release and latest Java point release for whichever versions you are using just in case there is a JRE bug or similar contributing to this. I don't see anything related in the Tomcat changelog but I'd still always recommend updating. Mark >> >> "VM Thread" os_prio=0 tid=0x440000c00000 nid=0x1cbc >> >> "GC task thread#0 (GenPauselessGC)" os_prio=0 tid=0x440000200000 >> nid=0x1cb6 >> >> "GC task thread#1 (GenPauselessGC)" os_prio=0 tid=0x440000400000 >> nid=0x1cb7 >> >> "GC task thread#0 (GenPauselessGC)" os_prio=0 tid=0x440000600000 >> nid=0x1cb8 >> >> "VM Periodic Task Thread" os_prio=0 tid=0x440002e00000 nid=0x1ccd >> >> JNI global references: 98048 >> >> >> Java stack information for the threads listed above: >> =================================================== >> "Thread-96": >> at >> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:187) >> >> - waiting to lock <0x0000080048462700> (a >> java.util.concurrent.ConcurrentHashMap) >> at >> org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:486) >> >> at >> org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:432) >> >> at >> org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:403) >> >> at >> org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:389) >> >> at >> org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1002) >> >> at >> org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:345) >> >> at >> org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:340) >> >> at >> org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1093) >> >> at >> com.XXXXXXX.tenantprovisioning.component.tool.falcon.cluster.provisioning.ClusterProvisioningThread.launchProvisioning(ClusterProvisioningThread.java:75) >> >> at >> com.XXXXXXX.tenantprovisioning.component.tool.falcon.cluster.provisioning.ClusterProvisioningThread.run(ClusterProvisioningThread.java:63) >> >> at java.lang.Thread.run(Thread.java:745) >> "Thread-29": >> at >> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1152) >> >> - waiting to lock <0x00001000bbc1dd20> (a java.lang.Object) >> at >> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1119) >> >> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:488) >> at >> com.XXXXXXX.wfc.commonapp.eventmanager.business.tasksched.ScheduleEngine.generateJobDetail(ScheduleEngine.java:982) >> >> at >> com.XXXXXXX.wfc.commonapp.eventmanager.business.tasksched.ScheduleEngine.scheduleAtFixedRate(ScheduleEngine.java:107) >> >> at >> com.XXXXXXX.wfc.commonapp.eventmanager.business.tasksched.ScheduleEngine.scheduleMinutely(ScheduleEngine.java:1316) >> >> at >> com.XXXXXXX.wfc.commonapp.eventmanager.business.tasksched.ScheduleEngineService.scheduleMinutely(ScheduleEngineService.java:493) >> >> - locked <0x00001000aa2752d8> (a >> com.XXXXXXX.wfc.commonapp.eventmanager.business.tasksched.KQuartzEngineService) >> >> at >> com.XXXXXXX.wfc.commonapp.eventmanager.business.engine.EvtMgrSched.scheduleTask(EvtMgrSched.java:1057) >> >> at >> com.XXXXXXX.wfc.commonapp.eventmanager.business.engine.EvtMgrSched.scheduleTask(EvtMgrSched.java:876) >> >> at >> com.XXXXXXX.wfc.commonapp.eventmanager.business.startup.EvtMgrStartUp.tenantBasedSchedAndDeleteTask(EvtMgrStartUp.java:186) >> >> at >> com.XXXXXXX.wfc.commonapp.eventmanager.business.startup.EvtMgrStartUp.initialize(EvtMgrStartUp.java:138) >> >> - locked <0x0000080058b5c5e8> (a java.lang.Class) >> at >> com.XXXXXXX.wfc.commonapp.eventmanager.business.startup.EvtMgrStartUp.startup(EvtMgrStartUp.java:73) >> >> at sun.reflect.GeneratedMethodAccessor311.invoke(Unknown Source) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >> at java.lang.reflect.Method.invoke(Method.java:498) >> at >> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:366) >> >> at >> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:311) >> >> at >> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:134) >> >> at >> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:409) >> >> at >> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1620) >> >> at >> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555) >> >> at >> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483) >> >> at >> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) >> >> at >> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) >> >> - locked <0x0000080048462700> (a java.util.concurrent.ConcurrentHashMap) >> at >> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) >> >> at >> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:220) >> >> at >> org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1018) >> >> at >> org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:345) >> >> at >> org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:340) >> >> at >> org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1093) >> >> at >> com.XXXXXXX.container.api.access.SpringContext.getBean(SpringContext.java:62) >> >> at >> com.XXXXXXX.wfc.platform.startup.framework.DeferredSpringServicesStarter.startDeferredStartupServices(DeferredSpringServicesStarter.java:83) >> >> at >> com.XXXXXXX.wfc.platform.startup.framework.DeferredSpringServicesStarter.startup(DeferredSpringServicesStarter.java:64) >> >> at sun.reflect.GeneratedMethodAccessor311.invoke(Unknown Source) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >> at java.lang.reflect.Method.invoke(Method.java:498) >> at >> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:366) >> >> at >> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:311) >> >> at >> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:134) >> >> at >> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:409) >> >> at >> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1620) >> >> at >> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555) >> >> at >> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483) >> >> at >> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) >> >> at >> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) >> >> - locked <0x0000080048462700> (a java.util.concurrent.ConcurrentHashMap) >> at >> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) >> >> at >> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:220) >> >> at >> org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1018) >> >> at >> org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:345) >> >> at >> org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:340) >> >> at >> org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1093) >> >> at >> com.XXXXXXX.container.api.access.SpringContext.getBean(SpringContext.java:62) >> >> at >> com.XXXXXXX.wfc.platform.startup.framework.WFCStartUp.startDeferredServices(WFCStartUp.java:1264) >> >> at >> com.XXXXXXX.wfc.platform.startup.framework.WFCStartUp.WFCInit(WFCStartUp.java:1159) >> >> at >> com.XXXXXXX.wfc.platform.startup.framework.WFCStartUp.WFCInit(WFCStartUp.java:1277) >> >> at >> com.XXXXXXX.wfc.platform.startup.framework.WFCStartUp$WFCInitThread.run(WFCStartUp.java:1429) >> >> "Event Manager Thread_QuartzSchedulerThread": >> at java.lang.ClassLoader.defineClass1(Native Method) >> at java.lang.ClassLoader.defineClass(ClassLoader.java:769) >> at >> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) >> at >> org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:2283) >> >> - locked <0x00001000bbc1dd20> (a java.lang.Object) >> at >> org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:811) >> >> at >> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1260) >> >> - locked <0x00001000bbc1dd20> (a java.lang.Object) >> at >> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1119) >> >> at >> org.quartz.simpl.LoadingLoaderClassLoadHelper.loadClass(LoadingLoaderClassLoadHelper.java:59) >> >> at >> org.quartz.simpl.CascadingClassLoadHelper.loadClass(CascadingClassLoadHelper.java:99) >> >> at >> org.quartz.simpl.CascadingClassLoadHelper.loadClass(CascadingClassLoadHelper.java:138) >> >> at >> org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectJobDetail(StdJDBCDelegate.java:852) >> >> at >> org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveJob(JobStoreSupport.java:1385) >> >> at >> org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTrigger(JobStoreSupport.java:2818) >> >> at >> org.quartz.impl.jdbcjobstore.JobStoreSupport$40.execute(JobStoreSupport.java:2759) >> >> at >> org.quartz.impl.jdbcjobstore.JobStoreSupport$40.execute(JobStoreSupport.java:2757) >> >> at >> org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3799) >> >> at >> org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTriggers(JobStoreSupport.java:2756) >> >> at >> org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:272) >> >> --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
