Tomcat 8.5.24 Java - (1.8.0-zing_16.12.3.0-b2-product-azlinuxM-X86_64 mixed mode [Mon Aug 13 05:18:16 PM UTC 2018]):
On Wed, Nov 21, 2018 at 6:49 AM Mark Thomas <ma...@apache.org> wrote: > And which version of Java? > > See also comments below. > > On 20/11/2018 22:16, Mark Thomas wrote: > > Which version of Tomcat are you using? > > > > Mark > > > > > > On 20/11/2018 21:02, Andrew Carr wrote: > >> Wait, I think I might see what I did, let me try again. > >> > >> Found one Java-level deadlock: > >> ============================= > >> "Thread-96": > >> waiting to lock monitor 0x3a0d6938 (object 0x80048462700, a > >> java.util.concurrent.ConcurrentHashMap), > >> which is held by "Thread-29" > > The above thread can be ignored as it is blocked by the deadlock but > isn't part of it. > > >> "Thread-29": > >> waiting to lock monitor 0x3a0d58b0 (object 0x1000bbc1dd20, a > >> java.lang.Object), > >> which is held by "Event Manager Thread_QuartzSchedulerThread" > > I can see all of this in the provided stack traces. > > > >> "Event Manager Thread_QuartzSchedulerThread": > >> waiting to lock monitor 0x3803a2c0 (object 0x800674b7830, a > >> org.apache.catalina.loader.ParallelWebappClassLoader), > >> which is held by "Thread-29" > > There is no evidence in this stack trace that Thread 29 is holding that > lock. > > There is no evidence in this stack trace that the Quartz thread is > waiting for that lock. > > Hence my comment that the stack traces are not consistent with the > reported deadlock. > > We really need to see where the lock on the ParallelWebappClassLoader > object is being held. Maybe try getting a thread dump with a different > tool? > > Until we can see where the locks are being held there isn't anything we > can do. > > Generally, I'd recommend an update to the latest Tomcat point release > and latest Java point release for whichever versions you are using just > in case there is a JRE bug or similar contributing to this. I don't see > anything related in the Tomcat changelog but I'd still always recommend > updating. > > Mark > > > > >> > >> "VM Thread" os_prio=0 tid=0x440000c00000 nid=0x1cbc > >> > >> "GC task thread#0 (GenPauselessGC)" os_prio=0 tid=0x440000200000 > >> nid=0x1cb6 > >> > >> "GC task thread#1 (GenPauselessGC)" os_prio=0 tid=0x440000400000 > >> nid=0x1cb7 > >> > >> "GC task thread#0 (GenPauselessGC)" os_prio=0 tid=0x440000600000 > >> nid=0x1cb8 > >> > >> "VM Periodic Task Thread" os_prio=0 tid=0x440002e00000 nid=0x1ccd > >> > >> JNI global references: 98048 > >> > >> > >> Java stack information for the threads listed above: > >> =================================================== > >> "Thread-96": > >> at > >> > org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:187) > >> > >> - waiting to lock <0x0000080048462700> (a > >> java.util.concurrent.ConcurrentHashMap) > >> at > >> > org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:486) > >> > >> at > >> > org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:432) > >> > >> at > >> > org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:403) > >> > >> at > >> > org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:389) > >> > >> at > >> > org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1002) > >> > >> at > >> > org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:345) > >> > >> at > >> > org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:340) > >> > >> at > >> > org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1093) > >> > >> at > >> > com.XXXXXXX.tenantprovisioning.component.tool.falcon.cluster.provisioning.ClusterProvisioningThread.launchProvisioning(ClusterProvisioningThread.java:75) > >> > >> at > >> > com.XXXXXXX.tenantprovisioning.component.tool.falcon.cluster.provisioning.ClusterProvisioningThread.run(ClusterProvisioningThread.java:63) > >> > >> at java.lang.Thread.run(Thread.java:745) > >> "Thread-29": > >> at > >> > org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1152) > >> > >> - waiting to lock <0x00001000bbc1dd20> (a java.lang.Object) > >> at > >> > org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1119) > >> > >> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:488) > >> at > >> > com.XXXXXXX.wfc.commonapp.eventmanager.business.tasksched.ScheduleEngine.generateJobDetail(ScheduleEngine.java:982) > >> > >> at > >> > com.XXXXXXX.wfc.commonapp.eventmanager.business.tasksched.ScheduleEngine.scheduleAtFixedRate(ScheduleEngine.java:107) > >> > >> at > >> > com.XXXXXXX.wfc.commonapp.eventmanager.business.tasksched.ScheduleEngine.scheduleMinutely(ScheduleEngine.java:1316) > >> > >> at > >> > com.XXXXXXX.wfc.commonapp.eventmanager.business.tasksched.ScheduleEngineService.scheduleMinutely(ScheduleEngineService.java:493) > >> > >> - locked <0x00001000aa2752d8> (a > >> > com.XXXXXXX.wfc.commonapp.eventmanager.business.tasksched.KQuartzEngineService) > >> > >> at > >> > com.XXXXXXX.wfc.commonapp.eventmanager.business.engine.EvtMgrSched.scheduleTask(EvtMgrSched.java:1057) > >> > >> at > >> > com.XXXXXXX.wfc.commonapp.eventmanager.business.engine.EvtMgrSched.scheduleTask(EvtMgrSched.java:876) > >> > >> at > >> > com.XXXXXXX.wfc.commonapp.eventmanager.business.startup.EvtMgrStartUp.tenantBasedSchedAndDeleteTask(EvtMgrStartUp.java:186) > >> > >> at > >> > com.XXXXXXX.wfc.commonapp.eventmanager.business.startup.EvtMgrStartUp.initialize(EvtMgrStartUp.java:138) > >> > >> - locked <0x0000080058b5c5e8> (a java.lang.Class) > >> at > >> > com.XXXXXXX.wfc.commonapp.eventmanager.business.startup.EvtMgrStartUp.startup(EvtMgrStartUp.java:73) > >> > >> at sun.reflect.GeneratedMethodAccessor311.invoke(Unknown Source) > >> at > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >> > >> at java.lang.reflect.Method.invoke(Method.java:498) > >> at > >> > org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:366) > >> > >> at > >> > org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:311) > >> > >> at > >> > org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:134) > >> > >> at > >> > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:409) > >> > >> at > >> > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1620) > >> > >> at > >> > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555) > >> > >> at > >> > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483) > >> > >> at > >> > org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) > >> > >> at > >> > org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) > >> > >> - locked <0x0000080048462700> (a java.util.concurrent.ConcurrentHashMap) > >> at > >> > org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) > >> > >> at > >> > org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:220) > >> > >> at > >> > org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1018) > >> > >> at > >> > org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:345) > >> > >> at > >> > org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:340) > >> > >> at > >> > org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1093) > >> > >> at > >> > com.XXXXXXX.container.api.access.SpringContext.getBean(SpringContext.java:62) > >> > >> at > >> > com.XXXXXXX.wfc.platform.startup.framework.DeferredSpringServicesStarter.startDeferredStartupServices(DeferredSpringServicesStarter.java:83) > >> > >> at > >> > com.XXXXXXX.wfc.platform.startup.framework.DeferredSpringServicesStarter.startup(DeferredSpringServicesStarter.java:64) > >> > >> at sun.reflect.GeneratedMethodAccessor311.invoke(Unknown Source) > >> at > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >> > >> at java.lang.reflect.Method.invoke(Method.java:498) > >> at > >> > org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:366) > >> > >> at > >> > org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:311) > >> > >> at > >> > org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:134) > >> > >> at > >> > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:409) > >> > >> at > >> > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1620) > >> > >> at > >> > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555) > >> > >> at > >> > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483) > >> > >> at > >> > org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) > >> > >> at > >> > org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) > >> > >> - locked <0x0000080048462700> (a java.util.concurrent.ConcurrentHashMap) > >> at > >> > org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) > >> > >> at > >> > org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:220) > >> > >> at > >> > org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1018) > >> > >> at > >> > org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:345) > >> > >> at > >> > org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:340) > >> > >> at > >> > org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1093) > >> > >> at > >> > com.XXXXXXX.container.api.access.SpringContext.getBean(SpringContext.java:62) > >> > >> at > >> > com.XXXXXXX.wfc.platform.startup.framework.WFCStartUp.startDeferredServices(WFCStartUp.java:1264) > >> > >> at > >> > com.XXXXXXX.wfc.platform.startup.framework.WFCStartUp.WFCInit(WFCStartUp.java:1159) > >> > >> at > >> > com.XXXXXXX.wfc.platform.startup.framework.WFCStartUp.WFCInit(WFCStartUp.java:1277) > >> > >> at > >> > com.XXXXXXX.wfc.platform.startup.framework.WFCStartUp$WFCInitThread.run(WFCStartUp.java:1429) > >> > >> "Event Manager Thread_QuartzSchedulerThread": > >> at java.lang.ClassLoader.defineClass1(Native Method) > >> at java.lang.ClassLoader.defineClass(ClassLoader.java:769) > >> at > >> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > >> at > >> > org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:2283) > >> > >> - locked <0x00001000bbc1dd20> (a java.lang.Object) > >> at > >> > org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:811) > >> > >> at > >> > org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1260) > >> > >> - locked <0x00001000bbc1dd20> (a java.lang.Object) > >> at > >> > org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1119) > >> > >> at > >> > org.quartz.simpl.LoadingLoaderClassLoadHelper.loadClass(LoadingLoaderClassLoadHelper.java:59) > >> > >> at > >> > org.quartz.simpl.CascadingClassLoadHelper.loadClass(CascadingClassLoadHelper.java:99) > >> > >> at > >> > org.quartz.simpl.CascadingClassLoadHelper.loadClass(CascadingClassLoadHelper.java:138) > >> > >> at > >> > org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectJobDetail(StdJDBCDelegate.java:852) > >> > >> at > >> > org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveJob(JobStoreSupport.java:1385) > >> > >> at > >> > org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTrigger(JobStoreSupport.java:2818) > >> > >> at > >> > org.quartz.impl.jdbcjobstore.JobStoreSupport$40.execute(JobStoreSupport.java:2759) > >> > >> at > >> > org.quartz.impl.jdbcjobstore.JobStoreSupport$40.execute(JobStoreSupport.java:2757) > >> > >> at > >> > org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3799) > >> > >> at > >> > org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTriggers(JobStoreSupport.java:2756) > >> > >> at > >> > org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:272) > >> > >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org > For additional commands, e-mail: dev-h...@tomcat.apache.org > > -- With Regards, Andrew Carr e. andrewlanec...@gmail.com w. andrew.c...@openlogic.com c. 4239489206 a. P.O. Box 1231, Greeneville, TN, 37744
