1 Reply Latest reply on Jan 14, 2013 4:32 AM by tomjenkinson

    Deadlock during JBoss transaction processing

    girish_n

      Hi,

       

      I have deployed WebMethods Broker in jboss-eap-5.1. We are testing case where transactions times out and new transactions will be started, these also times out so on..This process will repeat as long as JBoss is running.

       

      During this we have noticed deadlock sometimes. Transactions never timeout.

       

      Captured thread dump is attached.

       

      Deadlock is as below:

       

      Found one Java-level deadlock:

      =============================

      "WorkManager(2)-1":

        waiting to lock monitor 0x00000000561c5490 (object 0x00000000cc460c08, a com.arjuna.ats.internal.jta.transaction.arjunacore.AtomicAction),

        which is held by "Thread-10"

      "Thread-10":

        waiting for ownable synchronizer 0x00000000cc461658, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),

        which is held by "WorkManager(2)-1"

       

       

      Java stack information for the threads listed above:

      ===================================================

      "WorkManager(2)-1":

                at com.arjuna.ats.arjuna.coordinator.BasicAction.removeChildThread(BasicAction.java:916)

                - waiting to lock <0x00000000cc460c08> (a com.arjuna.ats.internal.jta.transaction.arjunacore.AtomicAction)

                at com.arjuna.ats.internal.arjuna.thread.ThreadActionData.purgeActions(ThreadActionData.java:248)

                at com.arjuna.ats.internal.arjuna.thread.ThreadActionData.purgeActions(ThreadActionData.java:224)

                at com.arjuna.ats.arjuna.AtomicAction.suspend(AtomicAction.java:334)

                at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionManagerImple.suspend(TransactionManagerImple.java:76)

                at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.suspend(BaseTransactionManagerDelegate.java:143)

                at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:363)

                at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)

                at org.jboss.ejb.plugins.RunAsSecurityInterceptor.process(RunAsSecurityInterceptor.java:133)

                at org.jboss.ejb.plugins.RunAsSecurityInterceptor.invoke(RunAsSecurityInterceptor.java:103)

                at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)

                at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)

                at org.jboss.ejb.MessageDrivenContainer.internalInvoke(MessageDrivenContainer.java:420)

                at org.jboss.ejb.Container.invoke(Container.java:1092)

                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                at java.lang.reflect.Method.invoke(Method.java:597)

                at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)

                at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)

                at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)

                at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)

                at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)

                at org.jboss.system.server.jmx.LazyMBeanServer.invoke(LazyMBeanServer.java:283)

                at org.jboss.invocation.local.LocalInvoker$MBeanServerAction.invoke(LocalInvoker.java:169)

                at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:118)

                at org.jboss.invocation.InvokerInterceptor.invokeLocal(InvokerInterceptor.java:209)

                at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:195)

                at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:61)

                at org.jboss.ejb.plugins.inflow.MessageEndpointInterceptor.delivery(MessageEndpointInterceptor.java:249)

                at org.jboss.ejb.plugins.inflow.MessageEndpointInterceptor.invoke(MessageEndpointInterceptor.java:128)

                at org.jboss.proxy.ClientMethodInterceptor.invoke(ClientMethodInterceptor.java:74)

                at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:101)

                at $Proxy111.onMessage(Unknown Source)

                at com.sun.genericra.inbound.async.DeliveryHelper.deliverMessage(DeliveryHelper.java:301)

                at com.sun.genericra.inbound.async.DeliveryHelper.deliver(DeliveryHelper.java:196)

                at com.sun.genericra.inbound.async.DeliveryHelper.deliver(DeliveryHelper.java:186)

                at com.sun.genericra.inbound.async.EndpointConsumer.consumeMessage(EndpointConsumer.java:176)

                at com.sun.genericra.inbound.async.MessageListener.onMessage(MessageListener.java:56)

                at com.webmethods.jms.loadbalance.connection.WmClusterSessionImpl.run(WmClusterSessionImpl.java:2014)

                at com.sun.genericra.inbound.async.WorkImpl.run(WorkImpl.java:53)

                at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:205)

                at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:260)

                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                at java.lang.Thread.run(Thread.java:662)

      "Thread-10":

                at sun.misc.Unsafe.park(Native Method)

                - parking to wait for  <0x00000000cc461658> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)

                at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)

                at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)

                at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)

                at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)

                at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)

                at com.webmethods.locks.BasicJMSLockManager.takeObjectWriteLock(BasicJMSLockManager.java:53)

                at com.webmethods.jms.loadbalance.connection.WmClusterXASessionImpl.rollback(WmClusterXASessionImpl.java:1585)

                at com.sun.genericra.inbound.InboundXAResourceProxy.rollback(InboundXAResourceProxy.java:260)

                at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelAbort(XAResourceRecord.java:420)

                at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:3270)

                at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:3248)

                at com.arjuna.ats.arjuna.coordinator.BasicAction.Abort(BasicAction.java:1933)

                - locked <0x00000000cc460c08> (a com.arjuna.ats.internal.jta.transaction.arjunacore.AtomicAction)

                at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.cancel(TwoPhaseCoordinator.java:114)

                at com.arjuna.ats.arjuna.AtomicAction.cancel(AtomicAction.java:213)

                at com.arjuna.ats.arjuna.coordinator.TransactionReaper.doCancellations(TransactionReaper.java:444)

                at com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(ReaperWorkerThread.java:91)

       

       

      Found 1 deadlock.

       

       

      Any idea what caused this deadlock?

       

      Found a issue reported similar to this: https://community.jboss.org/message/509851

        • 1. Re: Deadlock during JBoss transaction processing
          tomjenkinson

          Hi Girish,

           

          Firstly, I don't think that issue is caused by the same reason as this deadlock.

           

          Looking at your stack traces I see:

           

          Thread-10: A transaction is timing out, which required JBTM to lock the transaction and call rollback on all the resources.

           

          WorkManager(2)-1: Simultaneously your application is (unsuprisingly) still operating on this transaction. It appears to be trying to invoke something remote, aspart of this, it is trying to suspend the transaction to flow it to (presumably) another server. Calling suspend requires the transaction to be locked too.

           

          The reason this is not proceeding is because Thread-10 is trying to takeObjectWriteLock in webmethods code.

           

          Basically this does not appear to be a JBTM issue.

           

          I suggest you look at Thread-10:

                    at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)

                    at com.webmethods.locks.BasicJMSLockManager.takeObjectWriteLock(BasicJMSLockManager.java:53)

                    at com.webmethods.jms.loadbalance.connection.WmClusterXASessionImpl.rollback(WmClusterXASessionImpl.java:1585)

                    at com.sun.genericra.inbound.InboundXAResourceProxy.rollback(InboundXAResourceProxy.java:260)

           

          And try to undestand why webmethods rollback code is unable to lock what it needs to. As a starter, the only part of the stack from WorkManager(2)-1 that seems to contain webmethods is:

          at com.webmethods.jms.loadbalance.connection.WmClusterSessionImpl.run(WmClusterSessionImpl.java:2014)

           

          Perhaps that has "locked" whatever locks the rollback code needs, maybe webmethods does not allow access from multiple threads to the same WmClusterSessionImpl?

           

          Tom