Hello, We have been seeing some intermittent issues with the parallel classloader. I do not want to jump to any conclusions and say this is a bug w/ the classloader, but we are getting deadlocked threads, pretty randomly, when launching the server. I was wondering if anyone had experienced this issue or if you could review the attached thread dump and give me some feedback.
There is no associated log file for the time the deadlock was occurring, I am working on getting that. In unrelated log files I saw some concurrency exceptions, non-thread safe access to lists basically. I believe that issue has been corrected in the source code, and I am not sure the concurrency exception has anything to do with the deadlock occurring during classloading. Personally I am of the belief that unsafe maps / lists could possibly cause the classloader to deadlock. I can put the full dumps in a ticket, I just hate to open another classloader ticket. Java stack information for the threads listed above: =================================================== "Thread-96": at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:187) - waiting to lock <0x0000080048462700> (a java.util.concurrent.ConcurrentHashMap) at org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:486) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:432) at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:403) at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:389) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1002) at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:345) at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:340) at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1093) at com.XXXXXXX.tenantprovisioning.component.tool.falcon.cluster.provisioning.ClusterProvisioningThread.launchProvisioning(ClusterProvisioningThread.java:75) at com.XXXXXXX.tenantprovisioning.component.tool.falcon.cluster.provisioning.ClusterProvisioningThread.run(ClusterProvisioningThread.java:63) at java.lang.Thread.run(Thread.java:745) "Thread-29": at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1152) - waiting to lock <0x00001000bbc1dd20> (a java.lang.Object) at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1119) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:488) at com.XXXXXXX.wfc.commonapp.eventmanager.business.tasksched.ScheduleEngine.generateJobDetail(ScheduleEngine.java:982) at com.XXXXXXX.wfc.commonapp.eventmanager.business.tasksched.ScheduleEngine.scheduleAtFixedRate(ScheduleEngine.java:107) at com.XXXXXXX.wfc.commonapp.eventmanager.business.tasksched.ScheduleEngine.scheduleMinutely(ScheduleEngine.java:1316) at com.XXXXXXX.wfc.commonapp.eventmanager.business.tasksched.ScheduleEngineService.scheduleMinutely(ScheduleEngineService.java:493) - locked <0x00001000aa2752d8> (a com.XXXXXXX.wfc.commonapp.eventmanager.business.tasksched.KQuartzEngineService) at com.XXXXXXX.wfc.commonapp.eventmanager.business.engine.EvtMgrSched.scheduleTask(EvtMgrSched.java:1057) at com.XXXXXXX.wfc.commonapp.eventmanager.business.engine.EvtMgrSched.scheduleTask(EvtMgrSched.java:876) at com.XXXXXXX.wfc.commonapp.eventmanager.business.startup.EvtMgrStartUp.tenantBasedSchedAndDeleteTask(EvtMgrStartUp.java:186) at com.XXXXXXX.wfc.commonapp.eventmanager.business.startup.EvtMgrStartUp.initialize(EvtMgrStartUp.java:138) - locked <0x0000080058b5c5e8> (a java.lang.Class) at com.XXXXXXX.wfc.commonapp.eventmanager.business.startup.EvtMgrStartUp.startup(EvtMgrStartUp.java:73) at sun.reflect.GeneratedMethodAccessor311.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:366) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:311) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:134) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:409) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1620) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) - locked <0x0000080048462700> (a java.util.concurrent.ConcurrentHashMap) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:220) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1018) at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:345) at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:340) at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1093) at com.XXXXXXX.container.api.access.SpringContext.getBean(SpringContext.java:62) at com.XXXXXXX.wfc.platform.startup.framework.DeferredSpringServicesStarter.startDeferredStartupServices(DeferredSpringServicesStarter.java:83) at com.XXXXXXX.wfc.platform.startup.framework.DeferredSpringServicesStarter.startup(DeferredSpringServicesStarter.java:64) at sun.reflect.GeneratedMethodAccessor311.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:366) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:311) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:134) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:409) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1620) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) - locked <0x0000080048462700> (a java.util.concurrent.ConcurrentHashMap) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:220) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1018) at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:345) at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:340) at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1093) at com.XXXXXXX.container.api.access.SpringContext.getBean(SpringContext.java:62) at com.XXXXXXX.wfc.platform.startup.framework.WFCStartUp.startDeferredServices(WFCStartUp.java:1264) at com.XXXXXXX.wfc.platform.startup.framework.WFCStartUp.WFCInit(WFCStartUp.java:1159) at com.XXXXXXX.wfc.platform.startup.framework.WFCStartUp.WFCInit(WFCStartUp.java:1277) at com.XXXXXXX.wfc.platform.startup.framework.WFCStartUp$WFCInitThread.run(WFCStartUp.java:1429) "Event Manager Thread_QuartzSchedulerThread": at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:769) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:2283) - locked <0x00001000bbc1dd20> (a java.lang.Object) at org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:811) at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1260) - locked <0x00001000bbc1dd20> (a java.lang.Object) at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1119) at org.quartz.simpl.LoadingLoaderClassLoadHelper.loadClass(LoadingLoaderClassLoadHelper.java:59) at org.quartz.simpl.CascadingClassLoadHelper.loadClass(CascadingClassLoadHelper.java:99) at org.quartz.simpl.CascadingClassLoadHelper.loadClass(CascadingClassLoadHelper.java:138) at org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectJobDetail(StdJDBCDelegate.java:852) at org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveJob(JobStoreSupport.java:1385) at org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTrigger(JobStoreSupport.java:2818) at org.quartz.impl.jdbcjobstore.JobStoreSupport$40.execute(JobStoreSupport.java:2759) at org.quartz.impl.jdbcjobstore.JobStoreSupport$40.execute(JobStoreSupport.java:2757) at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3799) at org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTriggers(JobStoreSupport.java:2756) at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:272) Found 1 deadlock. -- With Regards, Andrew Carr e. andrewlanec...@gmail.com w. andrew.c...@openlogic.com h. 4235255668 c. 4239489206 a. P.O. Box 1231, Greeneville, TN, 37744
