Hello,
in the meantime we have tried the latest nightly builds but the deployment deadlock still remains.
After analyzing java thread dumps from the deadlock situation we identified some potential problematic code and changed it.
But still one problem remains:
we have a Singleton EJB that reads some property files at init time. The Singleton is annotated with
@Startup
@Lock(LockType.READ)
Additionally we are using servlet filter that are used from different war and jar projects. These filter are trying to read properties delivered from our Singleton EJB.
The deadlock occurs during the deployment of the war and jars that are using the filter. Our Singleton EJB isn't even loaded in the moment of the deadlock, the PostConstruct method has no chance to read any properties.
In stacktrace you see the 2 blocking and waiting threads (XmlConfigLocal is the local interface of the Singleton EJB):
"MSC service thread 1-2" prio=10 tid=0x80d06000 nid=0x44ca in Object.wait() [0x8076c000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at org.jboss.as.ee.component.BasicComponent.waitForComponentStart(BasicComponent.java:109)
- locked <0x83432890> (a org.jboss.as.ejb3.component.singleton.SingletonComponent)
at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:127)
at org.jboss.as.ee.component.BasicComponent.createInstance(BasicComponent.java:85)
at org.jboss.as.ejb3.component.singleton.SingletonComponent.getComponentInstance(SingletonComponent.java:116)
- locked <0x83467f00> (a java.lang.Object)
at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:48)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:228)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:304)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:190)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:32)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165)
at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:173)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72)
at serverview.common.javaee.xml_config.XmlConfigLocal$$$view9.getJAXBElement(Unknown Source)
at serverview.common.javaee.lib.utilities.ConfigCore.update(ConfigCore.java:112)
at serverview.common.javaee.lib.utilities.ConfigCore.<init>(ConfigCore.java:75)
at serverview.common.javaee.security.common.lib.config.Config.<init>(Config.java:38)
at serverview.common.javaee.security.common.lib.config.SecUtils.getConfig(SecUtils.java:28)
at serverview.common.javaee.security.servlet_filter.TGTInsertionFilter.init(TGTInsertionFilter.java:218)
at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:447)
at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:3269)
at org.apache.catalina.core.StandardContext.start(StandardContext.java:3865)
- locked <0x8e202e18> (a org.apache.catalina.core.StandardContext)
at org.jboss.as.web.deployment.WebDeploymentService.start(WebDeploymentService.java:90)
- locked <0x8e1c7790> (a org.jboss.as.web.deployment.WebDeploymentService)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"MSC service thread 1-1" prio=10 tid=0x80568c00 nid=0x44c9 waiting for monitor entry [0x80980000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.jboss.as.ejb3.component.singleton.SingletonComponent.getComponentInstance(SingletonComponent.java:115)
- waiting to lock <0x83467f00> (a java.lang.Object)
at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:48)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:228)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:304)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:190)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:32)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165)
at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:173)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72)
at serverview.common.javaee.xml_config.XmlConfigLocal$$$view9.getJAXBElement(Unknown Source)
at serverview.common.javaee.lib.utilities.ConfigCore.update(ConfigCore.java:112)
at serverview.common.javaee.lib.utilities.ConfigCore.<init>(ConfigCore.java:75)
at serverview.common.javaee.security.common.lib.config.Config.<init>(Config.java:38)
at serverview.common.javaee.security.common.lib.config.SecUtils.getConfig(SecUtils.java:28)
at serverview.common.javaee.security.servlet_filter.TGTInsertionFilter.init(TGTInsertionFilter.java:218)
at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:447)
at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:3269)
at org.apache.catalina.core.StandardContext.start(StandardContext.java:3865)
- locked <0x8e58aa40> (a org.apache.catalina.core.StandardContext)
at org.jboss.as.web.deployment.WebDeploymentService.start(WebDeploymentService.java:90)
- locked <0x8e58cc78> (a org.jboss.as.web.deployment.WebDeploymentService)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
We are wondering why the singleton isn't started before.
With our workaround (using jboss-cli.sh for deploying in a defined serial order) we have no deployment problems. The problem occurs again, when stopping and starting JBoss. Therefore we are deactivating(activating our applications at shutdown/start of JBoss.
Baerbel