Which version of Tomcat are you using?
Mark
On 20/11/2018 21:02, Andrew Carr wrote:
Wait, I think I might see what I did, let me try again.
Found one Java-level deadlock:
=============================
"Thread-96":
waiting to lock monitor 0x3a0d6938 (object 0x80048462700, a
java.util.concurrent.ConcurrentHashMap),
which is held by "Thread-29"
"Thread-29":
waiting to lock monitor 0x3a0d58b0 (object 0x1000bbc1dd20, a
java.lang.Object),
which is held by "Event Manager Thread_QuartzSchedulerThread"
"Event Manager Thread_QuartzSchedulerThread":
waiting to lock monitor 0x3803a2c0 (object 0x800674b7830, a
org.apache.catalina.loader.ParallelWebappClassLoader),
which is held by "Thread-29"
"VM Thread" os_prio=0 tid=0x440000c00000 nid=0x1cbc
"GC task thread#0 (GenPauselessGC)" os_prio=0 tid=0x440000200000 nid=0x1cb6
"GC task thread#1 (GenPauselessGC)" os_prio=0 tid=0x440000400000 nid=0x1cb7
"GC task thread#0 (GenPauselessGC)" os_prio=0 tid=0x440000600000 nid=0x1cb8
"VM Periodic Task Thread" os_prio=0 tid=0x440002e00000 nid=0x1ccd
JNI global references: 98048
Java stack information for the threads listed above:
===================================================
"Thread-96":
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:187)
- waiting to lock <0x0000080048462700> (a
java.util.concurrent.ConcurrentHashMap)
at
org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:486)
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:432)
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:403)
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:389)
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1002)
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:345)
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:340)
at
org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1093)
at
com.XXXXXXX.tenantprovisioning.component.tool.falcon.cluster.provisioning.ClusterProvisioningThread.launchProvisioning(ClusterProvisioningThread.java:75)
at
com.XXXXXXX.tenantprovisioning.component.tool.falcon.cluster.provisioning.ClusterProvisioningThread.run(ClusterProvisioningThread.java:63)
at java.lang.Thread.run(Thread.java:745)
"Thread-29":
at
org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1152)
- waiting to lock <0x00001000bbc1dd20> (a java.lang.Object)
at
org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1119)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:488)
at
com.XXXXXXX.wfc.commonapp.eventmanager.business.tasksched.ScheduleEngine.generateJobDetail(ScheduleEngine.java:982)
at
com.XXXXXXX.wfc.commonapp.eventmanager.business.tasksched.ScheduleEngine.scheduleAtFixedRate(ScheduleEngine.java:107)
at
com.XXXXXXX.wfc.commonapp.eventmanager.business.tasksched.ScheduleEngine.scheduleMinutely(ScheduleEngine.java:1316)
at
com.XXXXXXX.wfc.commonapp.eventmanager.business.tasksched.ScheduleEngineService.scheduleMinutely(ScheduleEngineService.java:493)
- locked <0x00001000aa2752d8> (a
com.XXXXXXX.wfc.commonapp.eventmanager.business.tasksched.KQuartzEngineService)
at
com.XXXXXXX.wfc.commonapp.eventmanager.business.engine.EvtMgrSched.scheduleTask(EvtMgrSched.java:1057)
at
com.XXXXXXX.wfc.commonapp.eventmanager.business.engine.EvtMgrSched.scheduleTask(EvtMgrSched.java:876)
at
com.XXXXXXX.wfc.commonapp.eventmanager.business.startup.EvtMgrStartUp.tenantBasedSchedAndDeleteTask(EvtMgrStartUp.java:186)
at
com.XXXXXXX.wfc.commonapp.eventmanager.business.startup.EvtMgrStartUp.initialize(EvtMgrStartUp.java:138)
- locked <0x0000080058b5c5e8> (a java.lang.Class)
at
com.XXXXXXX.wfc.commonapp.eventmanager.business.startup.EvtMgrStartUp.startup(EvtMgrStartUp.java:73)
at sun.reflect.GeneratedMethodAccessor311.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:366)
at
org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:311)
at
org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:134)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:409)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1620)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
at
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
- locked <0x0000080048462700> (a java.util.concurrent.ConcurrentHashMap)
at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:220)
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1018)
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:345)
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:340)
at
org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1093)
at
com.XXXXXXX.container.api.access.SpringContext.getBean(SpringContext.java:62)
at
com.XXXXXXX.wfc.platform.startup.framework.DeferredSpringServicesStarter.startDeferredStartupServices(DeferredSpringServicesStarter.java:83)
at
com.XXXXXXX.wfc.platform.startup.framework.DeferredSpringServicesStarter.startup(DeferredSpringServicesStarter.java:64)
at sun.reflect.GeneratedMethodAccessor311.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:366)
at
org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:311)
at
org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:134)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:409)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1620)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
at
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
- locked <0x0000080048462700> (a java.util.concurrent.ConcurrentHashMap)
at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:220)
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1018)
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:345)
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:340)
at
org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1093)
at
com.XXXXXXX.container.api.access.SpringContext.getBean(SpringContext.java:62)
at
com.XXXXXXX.wfc.platform.startup.framework.WFCStartUp.startDeferredServices(WFCStartUp.java:1264)
at
com.XXXXXXX.wfc.platform.startup.framework.WFCStartUp.WFCInit(WFCStartUp.java:1159)
at
com.XXXXXXX.wfc.platform.startup.framework.WFCStartUp.WFCInit(WFCStartUp.java:1277)
at
com.XXXXXXX.wfc.platform.startup.framework.WFCStartUp$WFCInitThread.run(WFCStartUp.java:1429)
"Event Manager Thread_QuartzSchedulerThread":
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:769)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at
org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:2283)
- locked <0x00001000bbc1dd20> (a java.lang.Object)
at
org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:811)
at
org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1260)
- locked <0x00001000bbc1dd20> (a java.lang.Object)
at
org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1119)
at
org.quartz.simpl.LoadingLoaderClassLoadHelper.loadClass(LoadingLoaderClassLoadHelper.java:59)
at
org.quartz.simpl.CascadingClassLoadHelper.loadClass(CascadingClassLoadHelper.java:99)
at
org.quartz.simpl.CascadingClassLoadHelper.loadClass(CascadingClassLoadHelper.java:138)
at
org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectJobDetail(StdJDBCDelegate.java:852)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveJob(JobStoreSupport.java:1385)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTrigger(JobStoreSupport.java:2818)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport$40.execute(JobStoreSupport.java:2759)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport$40.execute(JobStoreSupport.java:2757)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3799)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTriggers(JobStoreSupport.java:2756)
at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:272)
On Tue, Nov 20, 2018 at 3:59 PM Andrew Carr <andrewlanec...@gmail.com>
wrote:
Do you mean because the thread that is deadlocked is the
QuartzSchedulerThread?
The classloader and the concurrent thread are both waiting on Quartz.
Quartz thread is defined below: (Other than that I do not know what you
mean when you say it is not consistent.) Should I provide the full dump in
a ticket?
"Event Manager Thread_QuartzSchedulerThread":
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:769)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at
org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:2283)
- locked <0x00001000bbc1dd20> (a java.lang.Object)
at
org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:811)
at
org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1260)
- locked <0x00001000bbc1dd20> (a java.lang.Object)
at
org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1119)
at
org.quartz.simpl.LoadingLoaderClassLoadHelper.loadClass(LoadingLoaderClassLoadHelper.java:59)
at
org.quartz.simpl.CascadingClassLoadHelper.loadClass(CascadingClassLoadHelper.java:99)
at
org.quartz.simpl.CascadingClassLoadHelper.loadClass(CascadingClassLoadHelper.java:138)
at
org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectJobDetail(StdJDBCDelegate.java:852)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveJob(JobStoreSupport.java:1385)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTrigger(JobStoreSupport.java:2818)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport$40.execute(JobStoreSupport.java:2759)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport$40.execute(JobStoreSupport.java:2757)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3799)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTriggers(JobStoreSupport.java:2756)
at
org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:272)
On Tue, Nov 20, 2018 at 3:20 PM Mark Thomas <ma...@apache.org> wrote:
The thread dump you posted doesn't seem to be consistent with the
summary you provide below. Specifically for the "Event Manager
Thread_QuartzSchedulerThread". Might there be multiple threads with that
name and we are looking at the wrong one?
Mark
On 20/11/2018 19:06, Andrew Carr wrote:
My apologies, that is the offending thread, see below: (Should I open a
ticket to attach full dumps?)
Found one Java-level deadlock:
=============================
"Thread-96":
waiting to lock monitor 0x3a0d6938 (object 0x80048462700, a
java.util.concurrent.ConcurrentHashMap),
which is held by "Thread-29"
"Thread-29":
waiting to lock monitor 0x3a0d58b0 (object 0x1000bbc1dd20, a
java.lang.Object),
which is held by "Event Manager Thread_QuartzSchedulerThread"
"Event Manager Thread_QuartzSchedulerThread":
waiting to lock monitor 0x3803a2c0 (object 0x800674b7830, a
org.apache.catalina.loader.ParallelWebappClassLoader),
which is held by "Thread-29"
On Tue, Nov 20, 2018 at 12:50 PM Mark Thomas <ma...@apache.org> wrote:
On 20/11/2018 15:21, Andrew Carr wrote:
Hello,
We have been seeing some intermittent issues with the parallel
classloader. I do not want to jump to any conclusions and say this
is a
bug w/ the classloader, but we are getting deadlocked threads, pretty
randomly, when launching the server. I was wondering if anyone had
experienced this issue or if you could review the attached thread dump
and
give me some feedback.
There is no associated log file for the time the deadlock was
occurring,
I
am working on getting that. In unrelated log files I saw some
concurrency
exceptions, non-thread safe access to lists basically. I believe that
issue has been corrected in the source code, and I am not sure the
concurrency exception has anything to do with the deadlock occurring
during
classloading.
Personally I am of the belief that unsafe maps / lists could possibly
cause
the classloader to deadlock.
I can put the full dumps in a ticket, I just hate to open another
classloader ticket.
I don't see a deadlock in the provided stack trace.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
--
With Regards,
Andrew Carr
e. andrewlanec...@gmail.com
w. andrew.c...@openlogic.com
c. 4239489206
a. P.O. Box 1231, Greeneville, TN, 37744
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
