Cache locking issue after one node of the cluster is restart
michalkozakiewicz Nov 7, 2008 6:53 AMHi,
we are running 2 nodes on Jboss 5.0.0.Beta4. We had to restart one of the nodes and the deployment failed:
Caused by: org.hibernate.HibernateException: Could not instantiate cache implementation
Caused by: org.hibernate.cache.CacheException: org.jboss.cache.lock.TimeoutException read lock for /zcapitalspreads_ear,services_jar,capital_spreads/com/chaucerdigital/capitalspreads/account could not be acquired by Thread[main,5,jboss] after 15000 ms. Locks: Read lock owners: [] Write lock owner: GlobalTransaction:<10.0.0.23:3336>:40235 , lock info: write owner=GlobalTransaction:<10.0.0.23:3336>:40235 (activeReaders=0, activeWriter=Thread[Multiplexer,udp,10.0.0.24:1723,10,MultiplexerThreads], waitingReaders=0, waitingWriters=0, waitingUpgrader=0)
Any ideas please?
Michal
The full stack trace:
2:36:24,837 WARN [FC] Received two credit requests from 10.0.0.23:3336 without any intervening messages; sending 1999822 credits 12:36:34,869 WARN [FC] Received two credit requests from 10.0.0.23:3336 without any intervening messages; sending 1999822 credits 12:36:44,853 WARN [FC] Received two credit requests from 10.0.0.23:3336 without any intervening messages; sending 1999822 credits 12:36:54,885 WARN [FC] Received two credit requests from 10.0.0.23:3336 without any intervening messages; sending 1999822 credits 12:36:55,385 ERROR [RegionManager] failed to activate /zcapitalspreads_ear,services_jar,capital_spreads/com/chaucerdigital/capitalspreads/account/cash/AccountJournalEntry org.jboss.cache.lock.TimeoutException: read lock for /zcapitalspreads_ear,services_jar,capital_spreads/com/chaucerdigital/capitalspreads/account could not be acquired by Thread[main,5,jboss] after 15000 ms. Locks: Read lock owners: [] Write lock owner: GlobalTransaction:<10.0.0.23:3336>:40235 , lock info: write owner=GlobalTransaction:<10.0.0.23:3336>:40235 (activeReaders=0, activeWriter=Thread[Multiplexer,udp,10.0.0.24:1723,10,MultiplexerThreads], waitingReaders=0, waitingWriters=0, waitingUpgrader=0) at org.jboss.cache.lock.IdentityLock.acquireReadLock0(IdentityLock.java:323) at org.jboss.cache.lock.IdentityLock.acquireReadLock(IdentityLock.java:267) at org.jboss.cache.lock.IdentityLock.acquire(IdentityLock.java:493) at org.jboss.cache.interceptors.PessimisticLockInterceptor.acquireNodeLock(PessimisticLockInterceptor.java:526) at org.jboss.cache.interceptors.PessimisticLockInterceptor.lock(PessimisticLockInterceptor.java:420) at org.jboss.cache.interceptors.PessimisticLockInterceptor.acquireLocksWithTimeout(PessimisticLockInterceptor.java:348) at org.jboss.cache.interceptors.PessimisticLockInterceptor.handleGetNodeMethod(PessimisticLockInterceptor.java:301) at org.jboss.cache.interceptors.MethodDispacherInterceptor.invoke(MethodDispacherInterceptor.java:90) at org.jboss.cache.interceptors.PessimisticLockInterceptor.invoke(PessimisticLockInterceptor.java:83) at org.jboss.cache.interceptors.Interceptor.nextInterceptor(Interceptor.java:111) at org.jboss.cache.interceptors.MethodDispacherInterceptor.invoke(MethodDispacherInterceptor.java:56) at org.jboss.cache.interceptors.Interceptor.nextInterceptor(Interceptor.java:111) at org.jboss.cache.interceptors.MethodDispacherInterceptor.invoke(MethodDispacherInterceptor.java:62) at org.jboss.cache.interceptors.Interceptor.nextInterceptor(Interceptor.java:111) at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:452) at org.jboss.cache.interceptors.TxInterceptor.handleGetNodeMethod(TxInterceptor.java:245) at org.jboss.cache.interceptors.MethodDispacherInterceptor.invoke(MethodDispacherInterceptor.java:90) at org.jboss.cache.interceptors.Interceptor.nextInterceptor(Interceptor.java:111) at org.jboss.cache.interceptors.MethodDispacherInterceptor.invoke(MethodDispacherInterceptor.java:62) at org.jboss.cache.interceptors.Interceptor.nextInterceptor(Interceptor.java:111) at org.jboss.cache.interceptors.InvocationContextInterceptor.invoke(InvocationContextInterceptor.java:73) at org.jboss.cache.invocation.AbstractInvocationDelegate.invoke(AbstractInvocationDelegate.java:123) at org.jboss.cache.invocation.AbstractInvocationDelegate.invoke(AbstractInvocationDelegate.java:64) at org.jboss.cache.invocation.CacheInvocationDelegate.getNode(CacheInvocationDelegate.java:450) at org.jboss.cache.invocation.NodeInvocationDelegate.getChild(NodeInvocationDelegate.java:307) at org.jboss.cache.invocation.NodeInvocationDelegate.addChild(NodeInvocationDelegate.java:278) at org.jboss.cache.RegionManager.activateRegion(RegionManager.java:410) at org.jboss.cache.RegionManager.activate(RegionManager.java:325) at org.jboss.cache.RegionManager.activate(RegionManager.java:288) at org.jboss.cache.RegionImpl.activate(RegionImpl.java:82) at org.jboss.ejb3.entity.JBCCache.activateCacheRegion(JBCCache.java:321) at org.jboss.ejb3.entity.JBCCache.<init>(JBCCache.java:84) at org.jboss.ejb3.entity.JBCCacheFactory.buildCache(JBCCacheFactory.java:54) at org.jboss.ejb3.entity.TreeCacheProviderHook.buildCache(TreeCacheProviderHook.java:74) at org.hibernate.cache.CacheFactory.createCache(CacheFactory.java:61) at org.hibernate.impl.SessionFactoryImpl.<init>(SessionFactoryImpl.java:214) at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1294) at org.hibernate.cfg.AnnotationConfiguration.buildSessionFactory(AnnotationConfiguration.java:918) at org.hibernate.ejb.Ejb3Configuration.buildEntityManagerFactory(Ejb3Configuration.java:668) at org.hibernate.ejb.HibernatePersistence.createContainerEntityManagerFactory(HibernatePersistence.java:132) at org.jboss.ejb3.entity.PersistenceUnitDeployment.start(PersistenceUnitDeployment.java:259) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.reflect.plugins.introspection.ReflectionUtils.invoke(ReflectionUtils.java:56) at org.jboss.reflect.plugins.introspection.ReflectMethodInfoImpl.invoke(ReflectMethodInfoImpl.java:110) at org.jboss.joinpoint.plugins.BasicMethodJoinPoint.dispatch(BasicMethodJoinPoint.java:66) at org.jboss.kernel.plugins.dependency.KernelControllerContextAction$JoinpointDispatchWrapper.execute(KernelControllerContextAction.java:214) at org.jboss.kernel.plugins.dependency.ExecutionWrapper.execute(ExecutionWrapper.java:45) at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.dispatchExecutionWrapper(KernelControllerContextAction.java:108) at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.dispatchJoinPoint(KernelControllerContextAction.java:69) at org.jboss.kernel.plugins.dependency.LifecycleAction.installActionInternal(LifecycleAction.java:221) at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.installAction(KernelControllerContextAction.java:135) at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.installAction(KernelControllerContextAction.java:46) at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:62) at org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:71) at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51) at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:327) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1309) at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:734) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:862) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:784) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:574) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:398) at org.jboss.kernel.plugins.dependency.AbstractKernelController.install(AbstractKernelController.java:96) at org.jboss.kernel.plugins.dependency.AbstractKernelController.install(AbstractKernelController.java:90) at org.jboss.ejb3.MCKernelAbstraction.install(MCKernelAbstraction.java:153) at org.jboss.ejb3.Ejb3Deployment.startPersistenceUnits(Ejb3Deployment.java:606) at org.jboss.ejb3.Ejb3Deployment.start(Ejb3Deployment.java:402) at org.jboss.ejb3.deployers.EJBStage2Deployer.deploy(EJBStage2Deployer.java:53) at org.jboss.ejb3.deployers.EJBStage2Deployer.deploy(EJBStage2Deployer.java:37) at org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.internalDeploy(AbstractSimpleRealDeployer.java:65) at org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(AbstractRealDeployer.java:50) at org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:169) at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:853) at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:906) at org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:794) at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:327) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1309) at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:734) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:862) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:784) at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:622) at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:411) at org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:498) at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:506) at org.jboss.system.server.profileservice.ProfileServiceBootstrap.loadProfile(ProfileServiceBootstrap.java:246) at org.jboss.system.server.profileservice.ProfileServiceBootstrap.start(ProfileServiceBootstrap.java:131) at org.jboss.bootstrap.AbstractServerImpl.start(AbstractServerImpl.java:408) at org.jboss.Main.boot(Main.java:208) at org.jboss.Main$1.run(Main.java:534) at java.lang.Thread.run(Thread.java:595) 12:36:55,385 ERROR [RegionManager] failed inactivating /zcapitalspreads_ear,services_jar,capital_spreads/com/chaucerdigital/capitalspreads/account/cash/AccountJournalEntry org.jboss.cache.CacheException: Region /zcapitalspreads_ear,services_jar,capital_spreads/com/chaucerdigital/capitalspreads/account/cash/AccountJournalEntry is already being activated/deactivated at org.jboss.cache.RegionManager.inactivateRegion(RegionManager.java:523) at org.jboss.cache.RegionManager.activateRegion(RegionManager.java:458) at org.jboss.cache.RegionManager.activate(RegionManager.java:325) at org.jboss.cache.RegionManager.activate(RegionManager.java:288) at org.jboss.cache.RegionImpl.activate(RegionImpl.java:82) at org.jboss.ejb3.entity.JBCCache.activateCacheRegion(JBCCache.java:321) at org.jboss.ejb3.entity.JBCCache.<init>(JBCCache.java:84) at org.jboss.ejb3.entity.JBCCacheFactory.buildCache(JBCCacheFactory.java:54) at org.jboss.ejb3.entity.TreeCacheProviderHook.buildCache(TreeCacheProviderHook.java:74) at org.hibernate.cache.CacheFactory.createCache(CacheFactory.java:61) at org.hibernate.impl.SessionFactoryImpl.<init>(SessionFactoryImpl.java:214) at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1294) at org.hibernate.cfg.AnnotationConfiguration.buildSessionFactory(AnnotationConfiguration.java:918) at org.hibernate.ejb.Ejb3Configuration.buildEntityManagerFactory(Ejb3Configuration.java:668) at org.hibernate.ejb.HibernatePersistence.createContainerEntityManagerFactory(HibernatePersistence.java:132) at org.jboss.ejb3.entity.PersistenceUnitDeployment.start(PersistenceUnitDeployment.java:259) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.reflect.plugins.introspection.ReflectionUtils.invoke(ReflectionUtils.java:56) at org.jboss.reflect.plugins.introspection.ReflectMethodInfoImpl.invoke(ReflectMethodInfoImpl.java:110) at org.jboss.joinpoint.plugins.BasicMethodJoinPoint.dispatch(BasicMethodJoinPoint.java:66) at org.jboss.kernel.plugins.dependency.KernelControllerContextAction$JoinpointDispatchWrapper.execute(KernelControllerContextAction.java:214) at org.jboss.kernel.plugins.dependency.ExecutionWrapper.execute(ExecutionWrapper.java:45) at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.dispatchExecutionWrapper(KernelControllerContextAction.java:108) at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.dispatchJoinPoint(KernelControllerContextAction.java:69) at org.jboss.kernel.plugins.dependency.LifecycleAction.installActionInternal(LifecycleAction.java:221) at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.installAction(KernelControllerContextAction.java:135) at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.installAction(KernelControllerContextAction.java:46) at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:62) at org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:71) at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51) at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:327) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1309) at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:734) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:862) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:784) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:574) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:398) at org.jboss.kernel.plugins.dependency.AbstractKernelController.install(AbstractKernelController.java:96) at org.jboss.kernel.plugins.dependency.AbstractKernelController.install(AbstractKernelController.java:90) at org.jboss.ejb3.MCKernelAbstraction.install(MCKernelAbstraction.java:153) at org.jboss.ejb3.Ejb3Deployment.startPersistenceUnits(Ejb3Deployment.java:606) at org.jboss.ejb3.Ejb3Deployment.start(Ejb3Deployment.java:402) at org.jboss.ejb3.deployers.EJBStage2Deployer.deploy(EJBStage2Deployer.java:53) at org.jboss.ejb3.deployers.EJBStage2Deployer.deploy(EJBStage2Deployer.java:37) at org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.internalDeploy(AbstractSimpleRealDeployer.java:65) at org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(AbstractRealDeployer.java:50) at org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:169) at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:853) at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:906) at org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:794) at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:327) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1309) at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:734) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:862) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:784) at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:622) at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:411) at org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:498) at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:506) at org.jboss.system.server.profileservice.ProfileServiceBootstrap.loadProfile(ProfileServiceBootstrap.java:246) at org.jboss.system.server.profileservice.ProfileServiceBootstrap.start(ProfileServiceBootstrap.java:131) at org.jboss.bootstrap.AbstractServerImpl.start(AbstractServerImpl.java:408) at org.jboss.Main.boot(Main.java:208) at org.jboss.Main$1.run(Main.java:534) at java.lang.Thread.run(Thread.java:595) 12:36:55,385 ERROR [AbstractKernelController] Error installing to Start: name=persistence.units:ear=zcapitalspreads.ear,jar=services.jar,unitName=capital.spreads state=Create javax.persistence.PersistenceException: [PersistenceUnit: capital.spreads] Unable to build EntityManagerFactory at org.hibernate.ejb.Ejb3Configuration.buildEntityManagerFactory(Ejb3Configuration.java:676) at org.hibernate.ejb.HibernatePersistence.createContainerEntityManagerFactory(HibernatePersistence.java:132) at org.jboss.ejb3.entity.PersistenceUnitDeployment.start(PersistenceUnitDeployment.java:259) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.reflect.plugins.introspection.ReflectionUtils.invoke(ReflectionUtils.java:56) at org.jboss.reflect.plugins.introspection.ReflectMethodInfoImpl.invoke(ReflectMethodInfoImpl.java:110) at org.jboss.joinpoint.plugins.BasicMethodJoinPoint.dispatch(BasicMethodJoinPoint.java:66) at org.jboss.kernel.plugins.dependency.KernelControllerContextAction$JoinpointDispatchWrapper.execute(KernelControllerContextAction.java:214) at org.jboss.kernel.plugins.dependency.ExecutionWrapper.execute(ExecutionWrapper.java:45) at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.dispatchExecutionWrapper(KernelControllerContextAction.java:108) at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.dispatchJoinPoint(KernelControllerContextAction.java:69) at org.jboss.kernel.plugins.dependency.LifecycleAction.installActionInternal(LifecycleAction.java:221) at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.installAction(KernelControllerContextAction.java:135) at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.installAction(KernelControllerContextAction.java:46) at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:62) at org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:71) at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51) at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:327) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1309) at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:734) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:862) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:784) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:574) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:398) at org.jboss.kernel.plugins.dependency.AbstractKernelController.install(AbstractKernelController.java:96) at org.jboss.kernel.plugins.dependency.AbstractKernelController.install(AbstractKernelController.java:90) at org.jboss.ejb3.MCKernelAbstraction.install(MCKernelAbstraction.java:153) at org.jboss.ejb3.Ejb3Deployment.startPersistenceUnits(Ejb3Deployment.java:606) at org.jboss.ejb3.Ejb3Deployment.start(Ejb3Deployment.java:402) at org.jboss.ejb3.deployers.EJBStage2Deployer.deploy(EJBStage2Deployer.java:53) at org.jboss.ejb3.deployers.EJBStage2Deployer.deploy(EJBStage2Deployer.java:37) at org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.internalDeploy(AbstractSimpleRealDeployer.java:65) at org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(AbstractRealDeployer.java:50) at org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:169) at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:853) at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:906) at org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:794) at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:327) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1309) at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:734) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:862) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:784) at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:622) at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:411) at org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:498) at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:506) at org.jboss.system.server.profileservice.ProfileServiceBootstrap.loadProfile(ProfileServiceBootstrap.java:246) at org.jboss.system.server.profileservice.ProfileServiceBootstrap.start(ProfileServiceBootstrap.java:131) at org.jboss.bootstrap.AbstractServerImpl.start(AbstractServerImpl.java:408) at org.jboss.Main.boot(Main.java:208) at org.jboss.Main$1.run(Main.java:534) at java.lang.Thread.run(Thread.java:595) Caused by: org.hibernate.HibernateException: Could not instantiate cache implementation at org.hibernate.cache.CacheFactory.createCache(CacheFactory.java:64) at org.hibernate.impl.SessionFactoryImpl.<init>(SessionFactoryImpl.java:214) at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1294) at org.hibernate.cfg.AnnotationConfiguration.buildSessionFactory(AnnotationConfiguration.java:918) at org.hibernate.ejb.Ejb3Configuration.buildEntityManagerFactory(Ejb3Configuration.java:668) ... 54 more Caused by: org.hibernate.cache.CacheException: org.jboss.cache.lock.TimeoutException read lock for /zcapitalspreads_ear,services_jar,capital_spreads/com/chaucerdigital/capitalspreads/account could not be acquired by Thread[main,5,jboss] after 15000 ms. Locks: Read lock owners: [] Write lock owner: GlobalTransaction:<10.0.0.23:3336>:40235 , lock info: write owner=GlobalTransaction:<10.0.0.23:3336>:40235 (activeReaders=0, activeWriter=Thread[Multiplexer,udp,10.0.0.24:1723,10,MultiplexerThreads], waitingReaders=0, waitingWriters=0, waitingUpgrader=0) at org.jboss.cache.lock.IdentityLock.acquireReadLock0(IdentityLock.java:323) at org.jboss.cache.lock.IdentityLock.acquireReadLock(IdentityLock.java:267) at org.jboss.cache.lock.IdentityLock.acquire(IdentityLock.java:493) at org.jboss.cache.interceptors.PessimisticLockInterceptor.acquireNodeLock(PessimisticLockInterceptor.java:526) at org.jboss.cache.interceptors.PessimisticLockInterceptor.lock(PessimisticLockInterceptor.java:420) at org.jboss.cache.interceptors.PessimisticLockInterceptor.acquireLocksWithTimeout(PessimisticLockInterceptor.java:348) at org.jboss.cache.interceptors.PessimisticLockInterceptor.handleGetNodeMethod(PessimisticLockInterceptor.java:301) at org.jboss.cache.interceptors.MethodDispacherInterceptor.invoke(MethodDispacherInterceptor.java:90) at org.jboss.cache.interceptors.PessimisticLockInterceptor.invoke(PessimisticLockInterceptor.java:83) at org.jboss.cache.interceptors.Interceptor.nextInterceptor(Interceptor.java:111) at org.jboss.cache.interceptors.MethodDispacherInterceptor.invoke(MethodDispacherInterceptor.java:56) at org.jboss.cache.interceptors.Interceptor.nextInterceptor(Interceptor.java:111) at org.jboss.cache.interceptors.MethodDispacherInterceptor.invoke(MethodDispacherInterceptor.java:62) at org.jboss.cache.interceptors.Interceptor.nextInterceptor(Interceptor.java:111) at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:452) at org.jboss.cache.interceptors.TxInterceptor.handleGetNodeMethod(TxInterceptor.java:245) at org.jboss.cache.interceptors.MethodDispacherInterceptor.invoke(MethodDispacherInterceptor.java:90) at org.jboss.cache.interceptors.Interceptor.nextInterceptor(Interceptor.java:111) at org.jboss.cache.interceptors.MethodDispacherInterceptor.invoke(MethodDispacherInterceptor.java:62) at org.jboss.cache.interceptors.Interceptor.nextInterceptor(Interceptor.java:111) at org.jboss.cache.interceptors.InvocationContextInterceptor.invoke(InvocationContextInterceptor.java:73) at org.jboss.cache.invocation.AbstractInvocationDelegate.invoke(AbstractInvocationDelegate.java:123) at org.jboss.cache.invocation.AbstractInvocationDelegate.invoke(AbstractInvocationDelegate.java:64) at org.jboss.cache.invocation.CacheInvocationDelegate.getNode(CacheInvocationDelegate.java:450) at org.jboss.cache.invocation.NodeInvocationDelegate.getChild(NodeInvocationDelegate.java:307) at org.jboss.cache.invocation.NodeInvocationDelegate.addChild(NodeInvocationDelegate.java:278) at org.jboss.cache.RegionManager.activateRegion(RegionManager.java:410) at org.jboss.cache.RegionManager.activate(RegionManager.java:325) at org.jboss.cache.RegionManager.activate(RegionManager.java:288) at org.jboss.cache.RegionImpl.activate(RegionImpl.java:82) at org.jboss.ejb3.entity.JBCCache.activateCacheRegion(JBCCache.java:321) at org.jboss.ejb3.entity.JBCCache.<init>(JBCCache.java:84) at org.jboss.ejb3.entity.JBCCacheFactory.buildCache(JBCCacheFactory.java:54) at org.jboss.ejb3.entity.TreeCacheProviderHook.buildCache(TreeCacheProviderHook.java:74) at org.hibernate.cache.CacheFactory.createCache(CacheFactory.java:61) ... 58 more