Wait, I think I might see what I did, let me try again. Found one Java-level deadlock: ============================= "Thread-96": waiting to lock monitor 0x3a0d6938 (object 0x80048462700, a java.util.concurrent.ConcurrentHashMap), which is held by "Thread-29" "Thread-29": waiting to lock monitor 0x3a0d58b0 (object 0x1000bbc1dd20, a java.lang.Object), which is held by "Event Manager Thread_QuartzSchedulerThread" "Event Manager Thread_QuartzSchedulerThread": waiting to lock monitor 0x3803a2c0 (object 0x800674b7830, a org.apache.catalina.loader.ParallelWebappClassLoader), which is held by "Thread-29"
"VM Thread" os_prio=0 tid=0x440000c00000 nid=0x1cbc "GC task thread#0 (GenPauselessGC)" os_prio=0 tid=0x440000200000 nid=0x1cb6 "GC task thread#1 (GenPauselessGC)" os_prio=0 tid=0x440000400000 nid=0x1cb7 "GC task thread#0 (GenPauselessGC)" os_prio=0 tid=0x440000600000 nid=0x1cb8 "VM Periodic Task Thread" os_prio=0 tid=0x440002e00000 nid=0x1ccd JNI global references: 98048 Java stack information for the threads listed above: =================================================== "Thread-96": at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:187) - waiting to lock <0x0000080048462700> (a java.util.concurrent.ConcurrentHashMap) at org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:486) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:432) at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:403) at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:389) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1002) at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:345) at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:340) at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1093) at com.XXXXXXX.tenantprovisioning.component.tool.falcon.cluster.provisioning.ClusterProvisioningThread.launchProvisioning(ClusterProvisioningThread.java:75) at com.XXXXXXX.tenantprovisioning.component.tool.falcon.cluster.provisioning.ClusterProvisioningThread.run(ClusterProvisioningThread.java:63) at java.lang.Thread.run(Thread.java:745) "Thread-29": at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1152) - waiting to lock <0x00001000bbc1dd20> (a java.lang.Object) at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1119) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:488) at com.XXXXXXX.wfc.commonapp.eventmanager.business.tasksched.ScheduleEngine.generateJobDetail(ScheduleEngine.java:982) at com.XXXXXXX.wfc.commonapp.eventmanager.business.tasksched.ScheduleEngine.scheduleAtFixedRate(ScheduleEngine.java:107) at com.XXXXXXX.wfc.commonapp.eventmanager.business.tasksched.ScheduleEngine.scheduleMinutely(ScheduleEngine.java:1316) at com.XXXXXXX.wfc.commonapp.eventmanager.business.tasksched.ScheduleEngineService.scheduleMinutely(ScheduleEngineService.java:493) - locked <0x00001000aa2752d8> (a com.XXXXXXX.wfc.commonapp.eventmanager.business.tasksched.KQuartzEngineService) at com.XXXXXXX.wfc.commonapp.eventmanager.business.engine.EvtMgrSched.scheduleTask(EvtMgrSched.java:1057) at com.XXXXXXX.wfc.commonapp.eventmanager.business.engine.EvtMgrSched.scheduleTask(EvtMgrSched.java:876) at com.XXXXXXX.wfc.commonapp.eventmanager.business.startup.EvtMgrStartUp.tenantBasedSchedAndDeleteTask(EvtMgrStartUp.java:186) at com.XXXXXXX.wfc.commonapp.eventmanager.business.startup.EvtMgrStartUp.initialize(EvtMgrStartUp.java:138) - locked <0x0000080058b5c5e8> (a java.lang.Class) at com.XXXXXXX.wfc.commonapp.eventmanager.business.startup.EvtMgrStartUp.startup(EvtMgrStartUp.java:73) at sun.reflect.GeneratedMethodAccessor311.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:366) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:311) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:134) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:409) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1620) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) - locked <0x0000080048462700> (a java.util.concurrent.ConcurrentHashMap) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:220) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1018) at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:345) at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:340) at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1093) at com.XXXXXXX.container.api.access.SpringContext.getBean(SpringContext.java:62) at com.XXXXXXX.wfc.platform.startup.framework.DeferredSpringServicesStarter.startDeferredStartupServices(DeferredSpringServicesStarter.java:83) at com.XXXXXXX.wfc.platform.startup.framework.DeferredSpringServicesStarter.startup(DeferredSpringServicesStarter.java:64) at sun.reflect.GeneratedMethodAccessor311.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:366) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:311) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:134) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:409) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1620) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) - locked <0x0000080048462700> (a java.util.concurrent.ConcurrentHashMap) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:220) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1018) at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:345) at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:340) at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1093) at com.XXXXXXX.container.api.access.SpringContext.getBean(SpringContext.java:62) at com.XXXXXXX.wfc.platform.startup.framework.WFCStartUp.startDeferredServices(WFCStartUp.java:1264) at com.XXXXXXX.wfc.platform.startup.framework.WFCStartUp.WFCInit(WFCStartUp.java:1159) at com.XXXXXXX.wfc.platform.startup.framework.WFCStartUp.WFCInit(WFCStartUp.java:1277) at com.XXXXXXX.wfc.platform.startup.framework.WFCStartUp$WFCInitThread.run(WFCStartUp.java:1429) "Event Manager Thread_QuartzSchedulerThread": at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:769) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:2283) - locked <0x00001000bbc1dd20> (a java.lang.Object) at org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:811) at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1260) - locked <0x00001000bbc1dd20> (a java.lang.Object) at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1119) at org.quartz.simpl.LoadingLoaderClassLoadHelper.loadClass(LoadingLoaderClassLoadHelper.java:59) at org.quartz.simpl.CascadingClassLoadHelper.loadClass(CascadingClassLoadHelper.java:99) at org.quartz.simpl.CascadingClassLoadHelper.loadClass(CascadingClassLoadHelper.java:138) at org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectJobDetail(StdJDBCDelegate.java:852) at org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveJob(JobStoreSupport.java:1385) at org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTrigger(JobStoreSupport.java:2818) at org.quartz.impl.jdbcjobstore.JobStoreSupport$40.execute(JobStoreSupport.java:2759) at org.quartz.impl.jdbcjobstore.JobStoreSupport$40.execute(JobStoreSupport.java:2757) at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3799) at org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTriggers(JobStoreSupport.java:2756) at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:272) On Tue, Nov 20, 2018 at 3:59 PM Andrew Carr <andrewlanec...@gmail.com> wrote: > Do you mean because the thread that is deadlocked is the > QuartzSchedulerThread? > > The classloader and the concurrent thread are both waiting on Quartz. > Quartz thread is defined below: (Other than that I do not know what you > mean when you say it is not consistent.) Should I provide the full dump in > a ticket? > > "Event Manager Thread_QuartzSchedulerThread": > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:769) > at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at > org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:2283) > - locked <0x00001000bbc1dd20> (a java.lang.Object) > at > org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:811) > at > org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1260) > - locked <0x00001000bbc1dd20> (a java.lang.Object) > at > org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1119) > at > org.quartz.simpl.LoadingLoaderClassLoadHelper.loadClass(LoadingLoaderClassLoadHelper.java:59) > at > org.quartz.simpl.CascadingClassLoadHelper.loadClass(CascadingClassLoadHelper.java:99) > at > org.quartz.simpl.CascadingClassLoadHelper.loadClass(CascadingClassLoadHelper.java:138) > at > org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectJobDetail(StdJDBCDelegate.java:852) > at > org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveJob(JobStoreSupport.java:1385) > at > org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTrigger(JobStoreSupport.java:2818) > at > org.quartz.impl.jdbcjobstore.JobStoreSupport$40.execute(JobStoreSupport.java:2759) > at > org.quartz.impl.jdbcjobstore.JobStoreSupport$40.execute(JobStoreSupport.java:2757) > at > org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3799) > at > org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTriggers(JobStoreSupport.java:2756) > at > org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:272) > > On Tue, Nov 20, 2018 at 3:20 PM Mark Thomas <ma...@apache.org> wrote: > >> The thread dump you posted doesn't seem to be consistent with the >> summary you provide below. Specifically for the "Event Manager >> Thread_QuartzSchedulerThread". Might there be multiple threads with that >> name and we are looking at the wrong one? >> >> Mark >> >> >> On 20/11/2018 19:06, Andrew Carr wrote: >> > My apologies, that is the offending thread, see below: (Should I open a >> > ticket to attach full dumps?) >> > >> > Found one Java-level deadlock: >> > ============================= >> > "Thread-96": >> > waiting to lock monitor 0x3a0d6938 (object 0x80048462700, a >> > java.util.concurrent.ConcurrentHashMap), >> > which is held by "Thread-29" >> > "Thread-29": >> > waiting to lock monitor 0x3a0d58b0 (object 0x1000bbc1dd20, a >> > java.lang.Object), >> > which is held by "Event Manager Thread_QuartzSchedulerThread" >> > "Event Manager Thread_QuartzSchedulerThread": >> > waiting to lock monitor 0x3803a2c0 (object 0x800674b7830, a >> > org.apache.catalina.loader.ParallelWebappClassLoader), >> > which is held by "Thread-29" >> > >> > >> > On Tue, Nov 20, 2018 at 12:50 PM Mark Thomas <ma...@apache.org> wrote: >> > >> >> On 20/11/2018 15:21, Andrew Carr wrote: >> >>> Hello, >> >>> >> >>> We have been seeing some intermittent issues with the parallel >> >>> classloader. I do not want to jump to any conclusions and say this >> is a >> >>> bug w/ the classloader, but we are getting deadlocked threads, pretty >> >>> randomly, when launching the server. I was wondering if anyone had >> >>> experienced this issue or if you could review the attached thread dump >> >> and >> >>> give me some feedback. >> >>> >> >>> There is no associated log file for the time the deadlock was >> occurring, >> >> I >> >>> am working on getting that. In unrelated log files I saw some >> >> concurrency >> >>> exceptions, non-thread safe access to lists basically. I believe that >> >>> issue has been corrected in the source code, and I am not sure the >> >>> concurrency exception has anything to do with the deadlock occurring >> >> during >> >>> classloading. >> >>> >> >>> Personally I am of the belief that unsafe maps / lists could possibly >> >> cause >> >>> the classloader to deadlock. >> >>> >> >>> I can put the full dumps in a ticket, I just hate to open another >> >>> classloader ticket. >> >> >> >> I don't see a deadlock in the provided stack trace. >> >> >> >> Mark >> >> >> >> --------------------------------------------------------------------- >> >> To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org >> >> For additional commands, e-mail: dev-h...@tomcat.apache.org >> >> >> >> >> > >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: dev-h...@tomcat.apache.org >> >> > > -- > With Regards, > Andrew Carr > > e. andrewlanec...@gmail.com > w. andrew.c...@openlogic.com > c. 4239489206 > a. P.O. Box 1231, Greeneville, TN, 37744 > -- With Regards, Andrew Carr e. andrewlanec...@gmail.com w. andrew.c...@openlogic.com c. 4239489206 a. P.O. Box 1231, Greeneville, TN, 37744
