2 Replies Latest reply on Oct 7, 2010 10:38 PM by brian.stansberry

    Shutdown of JTS

    kabirkhan

      For AS 7 we are using jbossjts 4.11.0.Final, and I am seeing some problems on shutdown of a server instance. This does not happen every time but often enough on my machine. For about 65 seconds I see stack traces like:

       

      "Attach Listener" daemon prio=9 tid=12e309000 nid=0x130102000 waiting on condition [00000000]
         java.lang.Thread.State: RUNNABLE
      "Exit thread" prio=5 tid=101b85000 nid=0x13134e000 waiting for monitor entry [13134d000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      at java.lang.Shutdown.exit(Shutdown.java:168)
      - waiting to lock <1256c5a88> (a java.lang.Class for java.lang.Shutdown)
      at java.lang.Runtime.exit(Runtime.java:90)
      at java.lang.System.exit(System.java:921)
      at org.jboss.as.server.SystemExiter$DefaultExiter.exit(SystemExiter.java:53)
      at org.jboss.as.server.SystemExiter.exit(SystemExiter.java:40)
      at org.jboss.as.server.ProcessManagerServerCommunicationHandler$Controller$1.run(ProcessManagerServerCommunicationHandler.java:118)
      at java.lang.Thread.run(Thread.java:637)
      "Thread-2" daemon prio=5 tid=12e1c9000 nid=0x13185d000 waiting on condition [13185c000]
         java.lang.Thread.State: WAITING (parking)
      at sun.misc.Unsafe.park(Native Method)
      - parking to wait for  <1057a6b78> (a java.util.concurrent.CountDownLatch$Sync)
      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:905)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1217)
      at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
      at org.jboss.msc.service.ServiceContainerImpl$ShutdownHookHolder$1$1.run(ServiceContainerImpl.java:110)
      at java.lang.Thread.run(Thread.java:637)
      "SIGINT handler" daemon prio=9 tid=101aa1000 nid=0x13175a000 in Object.wait() [131759000]
         java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      - waiting on <107d1cc70> (a java.lang.Thread)
      at java.lang.Thread.join(Thread.java:1167)
      - locked <107d1cc70> (a java.lang.Thread)
      at java.lang.Thread.join(Thread.java:1220)
      at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:79)
      at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:24)
      at java.lang.Shutdown.runHooks(Shutdown.java:79)
      at java.lang.Shutdown.sequence(Shutdown.java:123)
      at java.lang.Shutdown.exit(Shutdown.java:168)
      - locked <1256c5a88> (a java.lang.Class for java.lang.Shutdown)
      at java.lang.Terminator$1.handle(Terminator.java:35)
      at sun.misc.Signal$1.run(Signal.java:195)
      at java.lang.Thread.run(Thread.java:637)
      "Thread-8" daemon prio=5 tid=12e311800 nid=0x130cde000 in Object.wait() [130cdd000]
         java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      - waiting on <1064bf630> (a java.util.LinkedList)
      at java.lang.Object.wait(Object.java:485)
      at com.arjuna.ats.arjuna.coordinator.TransactionReaper.waitForCancellations(TransactionReaper.java:397)
      - locked <1064bf630> (a java.util.LinkedList)
      at com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(ReaperWorkerThread.java:65)
      "Thread-7" daemon prio=5 tid=101a53800 nid=0x130bdb000 in Object.wait() [130bda000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      - waiting on <1064bf658> (a com.arjuna.ats.arjuna.coordinator.TransactionReaper)
      at com.arjuna.ats.internal.arjuna.coordinator.ReaperThread.run(ReaperThread.java:95)
      - locked <1064bf658> (a com.arjuna.ats.arjuna.coordinator.TransactionReaper)
      "Periodic Recovery" daemon prio=5 tid=12e308800 nid=0x130ad8000 in Object.wait() [130ad7000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      - waiting on <1063c0340> (a java.lang.Object)
      at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doPeriodicWait(PeriodicRecovery.java:696)
      at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:416)
      - locked <1063c0340> (a java.lang.Object)
      "Listener:4862" daemon prio=5 tid=12e307800 nid=0x1309d5000 runnable [1309d4000]
         java.lang.Thread.State: RUNNABLE
      at java.net.PlainSocketImpl.socketAccept(Native Method)
      at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
      - locked <1064bf858> (a java.net.SocksSocketImpl)
      at java.net.ServerSocket.implAccept(ServerSocket.java:453)
      at java.net.ServerSocket.accept(ServerSocket.java:421)
      at com.arjuna.ats.internal.arjuna.recovery.Listener.run(Listener.java:124)
      "Thread-6" daemon prio=5 tid=12e305000 nid=0x1308d2000 runnable [1308d1000]
         java.lang.Thread.State: RUNNABLE
      at java.net.PlainSocketImpl.socketConnect(Native Method)
      at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
      - locked <1064c0338> (a java.net.SocksSocketImpl)
      at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
      at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
      at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
      at java.net.Socket.connect(Socket.java:529)
      at java.net.Socket.connect(Socket.java:478)
      at java.net.Socket.<init>(Socket.java:375)
      at java.net.Socket.<init>(Socket.java:189)
      at com.arjuna.ats.internal.arjuna.recovery.TransactionStatusConnector.establishConnection(TransactionStatusConnector.java:209)
      at com.arjuna.ats.internal.arjuna.recovery.TransactionStatusConnector.test(TransactionStatusConnector.java:75)
      at com.arjuna.ats.internal.arjuna.recovery.ExpiredTransactionStatusManagerScanner.scan(ExpiredTransactionStatusManagerScanner.java:136)
      at com.arjuna.ats.internal.arjuna.recovery.ExpiredEntryMonitor.run(ExpiredEntryMonitor.java:191)
      "pool-2-thread-1" prio=5 tid=12e22c000 nid=0x1306cc000 waiting on condition [1306cb000]
         java.lang.Thread.State: WAITING (parking)
      at sun.misc.Unsafe.park(Native Method)
      - parking to wait for  <1063c06e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
      at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
      at java.lang.Thread.run(Thread.java:637)
      "Reference Reaper" daemon prio=5 tid=12e1fa800 nid=0x1305c0000 in Object.wait() [1305bf000]
         java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      - waiting on <1063c0e30> (a java.lang.ref.ReferenceQueue$Lock)
      at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
      - locked <1063c0e30> (a java.lang.ref.ReferenceQueue$Lock)
      at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
      at org.jboss.msc.ref.References$ReaperThread.run(References.java:64)
      "Thread-3" daemon prio=5 tid=101a54800 nid=0x1304bd000 in Object.wait() [1304bc000]
         java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      - waiting on <1064bfa30> (a com.arjuna.ats.internal.arjuna.recovery.ExpiredEntryMonitor)
      at java.lang.Thread.join(Thread.java:1167)
      - locked <1064bfa30> (a com.arjuna.ats.internal.arjuna.recovery.ExpiredEntryMonitor)
      at java.lang.Thread.join(Thread.java:1220)
      at com.arjuna.ats.internal.arjuna.recovery.ExpiredEntryMonitor.shutdown(ExpiredEntryMonitor.java:130)
      - locked <1260aba50> (a java.lang.Class for com.arjuna.ats.internal.arjuna.recovery.ExpiredEntryMonitor)
      at com.arjuna.ats.internal.arjuna.recovery.RecoveryManagerImple.stop(RecoveryManagerImple.java:209)
      at com.arjuna.ats.arjuna.recovery.RecoveryManager.terminate(RecoveryManager.java:213)
      - locked <1064b6c68> (a com.arjuna.ats.arjuna.recovery.RecoveryManager)
      at com.arjuna.ats.arjuna.recovery.RecoveryManager.terminate(RecoveryManager.java:197)
      at com.arjuna.ats.jbossatx.jta.RecoveryManagerService.stop(RecoveryManagerService.java:84)
      at org.jboss.as.txn.TransactionManagerService.stop(TransactionManagerService.java:224)
      - locked <1064b6bc0> (a org.jboss.as.txn.TransactionManagerService)
      at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1061)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:637)
      "Poller SunPKCS11-Darwin" daemon prio=1 tid=12e103000 nid=0x130361000 waiting on condition [130360000]
         java.lang.Thread.State: TIMED_WAITING (sleeping)
      at java.lang.Thread.sleep(Native Method)
      at sun.security.pkcs11.SunPKCS11$TokenPoller.run(SunPKCS11.java:692)
      at java.lang.Thread.run(Thread.java:637)
      "DestroyJavaVM" prio=5 tid=12e102000 nid=0x100501000 waiting on condition [00000000]
         java.lang.Thread.State: RUNNABLE
      "ClassLoader Thread" daemon prio=5 tid=1018e0000 nid=0x12fd7a000 in Object.wait() [12fd79000]
         java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      - waiting on <107b27cb8> (a java.util.ArrayDeque)
      at java.lang.Object.wait(Object.java:485)
      at org.jboss.modules.ConcurrentClassLoader$LoaderThread.run(ConcurrentClassLoader.java:329)
      - locked <107b27cb8> (a java.util.ArrayDeque)
      "Low Memory Detector" daemon prio=5 tid=101878800 nid=0x12fa90000 runnable [00000000]
         java.lang.Thread.State: RUNNABLE
      "CompilerThread1" daemon prio=9 tid=101878000 nid=0x12f98d000 waiting on condition [00000000]
         java.lang.Thread.State: RUNNABLE
      "CompilerThread0" daemon prio=9 tid=101877000 nid=0x12f88a000 waiting on condition [00000000]
         java.lang.Thread.State: RUNNABLE
      "Signal Dispatcher" daemon prio=9 tid=101876800 nid=0x12f787000 runnable [00000000]
         java.lang.Thread.State: RUNNABLE
      "Surrogate Locker Thread (CMS)" daemon prio=5 tid=12e03c800 nid=0x12f684000 waiting on condition [00000000]
         java.lang.Thread.State: RUNNABLE
      "Finalizer" daemon prio=8 tid=101875800 nid=0x12f307000 in Object.wait() [12f306000]
         java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      - waiting on <107ca9848> (a java.lang.ref.ReferenceQueue$Lock)
      at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
      - locked <107ca9848> (a java.lang.ref.ReferenceQueue$Lock)
      at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
      at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
      "Reference Handler" daemon prio=10 tid=101875000 nid=0x12f204000 in Object.wait() [12f203000]
         java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      - waiting on <107ca8f60> (a java.lang.ref.Reference$Lock)
      at java.lang.Object.wait(Object.java:485)
      at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
      - locked <107ca8f60> (a java.lang.ref.Reference$Lock)
      "VM Thread" prio=9 tid=12e022000 nid=0x12f101000 runnable
      "Gang worker#0 (Parallel GC Threads)" prio=9 tid=101802800 nid=0x10171b000 runnable
      "Gang worker#1 (Parallel GC Threads)" prio=9 tid=101803000 nid=0x105102000 runnable
      "Concurrent Mark-Sweep GC Thread" prio=9 tid=101842800 nid=0x12dbe6000 runnable
      "VM Periodic Task Thread" prio=10 tid=10187a000 nid=0x12fb93000 waiting on condition
      "Exception Catcher Thread" prio=10 tid=101801800 nid=0x100636000 runnable
      JNI global references: 889

      "Attach Listener" daemon prio=9 tid=12e309000 nid=0x130102000 waiting on condition [00000000]

         java.lang.Thread.State: RUNNABLE

       

      "Exit thread" prio=5 tid=101b85000 nid=0x13134e000 waiting for monitor entry [13134d000]

         java.lang.Thread.State: BLOCKED (on object monitor)

           at java.lang.Shutdown.exit(Shutdown.java:168)

           - waiting to lock <1256c5a88> (a java.lang.Class for java.lang.Shutdown)

           at java.lang.Runtime.exit(Runtime.java:90)

           at java.lang.System.exit(System.java:921)

           at org.jboss.as.server.SystemExiter$DefaultExiter.exit(SystemExiter.java:53)

           at org.jboss.as.server.SystemExiter.exit(SystemExiter.java:40)

           at org.jboss.as.server.ProcessManagerServerCommunicationHandler$Controller$1.run(ProcessManagerServerCommunicationHandler.java:118)

           at java.lang.Thread.run(Thread.java:637)

       

      "Thread-2" daemon prio=5 tid=12e1c9000 nid=0x13185d000 waiting on condition [13185c000]

         java.lang.Thread.State: WAITING (parking)

           at sun.misc.Unsafe.park(Native Method)

           - parking to wait for  <1057a6b78> (a java.util.concurrent.CountDownLatch$Sync)

           at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

           at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)

           at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:905)

           at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1217)

           at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)

           at org.jboss.msc.service.ServiceContainerImpl$ShutdownHookHolder$1$1.run(ServiceContainerImpl.java:110)

           at java.lang.Thread.run(Thread.java:637)

       

      "SIGINT handler" daemon prio=9 tid=101aa1000 nid=0x13175a000 in Object.wait() [131759000]

         java.lang.Thread.State: WAITING (on object monitor)

           at java.lang.Object.wait(Native Method)

           - waiting on <107d1cc70> (a java.lang.Thread)

           at java.lang.Thread.join(Thread.java:1167)

           - locked <107d1cc70> (a java.lang.Thread)

           at java.lang.Thread.join(Thread.java:1220)

           at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:79)

           at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:24)

           at java.lang.Shutdown.runHooks(Shutdown.java:79)

           at java.lang.Shutdown.sequence(Shutdown.java:123)

           at java.lang.Shutdown.exit(Shutdown.java:168)

           - locked <1256c5a88> (a java.lang.Class for java.lang.Shutdown)

           at java.lang.Terminator$1.handle(Terminator.java:35)

           at sun.misc.Signal$1.run(Signal.java:195)

           at java.lang.Thread.run(Thread.java:637)

       

      "Thread-8" daemon prio=5 tid=12e311800 nid=0x130cde000 in Object.wait() [130cdd000]

         java.lang.Thread.State: WAITING (on object monitor)

           at java.lang.Object.wait(Native Method)

           - waiting on <1064bf630> (a java.util.LinkedList)

           at java.lang.Object.wait(Object.java:485)

           at com.arjuna.ats.arjuna.coordinator.TransactionReaper.waitForCancellations(TransactionReaper.java:397)

           - locked <1064bf630> (a java.util.LinkedList)

           at com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(ReaperWorkerThread.java:65)

       

      "Thread-7" daemon prio=5 tid=101a53800 nid=0x130bdb000 in Object.wait() [130bda000]

         java.lang.Thread.State: TIMED_WAITING (on object monitor)

           at java.lang.Object.wait(Native Method)

           - waiting on <1064bf658> (a com.arjuna.ats.arjuna.coordinator.TransactionReaper)

           at com.arjuna.ats.internal.arjuna.coordinator.ReaperThread.run(ReaperThread.java:95)

           - locked <1064bf658> (a com.arjuna.ats.arjuna.coordinator.TransactionReaper)

       

      "Periodic Recovery" daemon prio=5 tid=12e308800 nid=0x130ad8000 in Object.wait() [130ad7000]

         java.lang.Thread.State: TIMED_WAITING (on object monitor)

           at java.lang.Object.wait(Native Method)

           - waiting on <1063c0340> (a java.lang.Object)

           at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doPeriodicWait(PeriodicRecovery.java:696)

           at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:416)

           - locked <1063c0340> (a java.lang.Object)

       

      "Listener:4862" daemon prio=5 tid=12e307800 nid=0x1309d5000 runnable [1309d4000]

         java.lang.Thread.State: RUNNABLE

           at java.net.PlainSocketImpl.socketAccept(Native Method)

           at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)

           - locked <1064bf858> (a java.net.SocksSocketImpl)

           at java.net.ServerSocket.implAccept(ServerSocket.java:453)

           at java.net.ServerSocket.accept(ServerSocket.java:421)

           at com.arjuna.ats.internal.arjuna.recovery.Listener.run(Listener.java:124)

       

      "Thread-6" daemon prio=5 tid=12e305000 nid=0x1308d2000 runnable [1308d1000]

         java.lang.Thread.State: RUNNABLE

           at java.net.PlainSocketImpl.socketConnect(Native Method)

           at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)

           - locked <1064c0338> (a java.net.SocksSocketImpl)

           at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)

           at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)

           at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)

           at java.net.Socket.connect(Socket.java:529)

           at java.net.Socket.connect(Socket.java:478)

           at java.net.Socket.<init>(Socket.java:375)

           at java.net.Socket.<init>(Socket.java:189)

           at com.arjuna.ats.internal.arjuna.recovery.TransactionStatusConnector.establishConnection(TransactionStatusConnector.java:209)

           at com.arjuna.ats.internal.arjuna.recovery.TransactionStatusConnector.test(TransactionStatusConnector.java:75)

           at com.arjuna.ats.internal.arjuna.recovery.ExpiredTransactionStatusManagerScanner.scan(ExpiredTransactionStatusManagerScanner.java:136)

           at com.arjuna.ats.internal.arjuna.recovery.ExpiredEntryMonitor.run(ExpiredEntryMonitor.java:191)

       

      "pool-2-thread-1" prio=5 tid=12e22c000 nid=0x1306cc000 waiting on condition [1306cb000]

         java.lang.Thread.State: WAITING (parking)

           at sun.misc.Unsafe.park(Native Method)

           - parking to wait for  <1063c06e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

           at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

           at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)

           at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

           at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

           at java.lang.Thread.run(Thread.java:637)

       

      "Reference Reaper" daemon prio=5 tid=12e1fa800 nid=0x1305c0000 in Object.wait() [1305bf000]

         java.lang.Thread.State: WAITING (on object monitor)

           at java.lang.Object.wait(Native Method)

           - waiting on <1063c0e30> (a java.lang.ref.ReferenceQueue$Lock)

           at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)

           - locked <1063c0e30> (a java.lang.ref.ReferenceQueue$Lock)

           at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)

           at org.jboss.msc.ref.References$ReaperThread.run(References.java:64)

       

      "Thread-3" daemon prio=5 tid=101a54800 nid=0x1304bd000 in Object.wait() [1304bc000]

         java.lang.Thread.State: WAITING (on object monitor)

           at java.lang.Object.wait(Native Method)

           - waiting on <1064bfa30> (a com.arjuna.ats.internal.arjuna.recovery.ExpiredEntryMonitor)

           at java.lang.Thread.join(Thread.java:1167)

           - locked <1064bfa30> (a com.arjuna.ats.internal.arjuna.recovery.ExpiredEntryMonitor)

           at java.lang.Thread.join(Thread.java:1220)

           at com.arjuna.ats.internal.arjuna.recovery.ExpiredEntryMonitor.shutdown(ExpiredEntryMonitor.java:130)

           - locked <1260aba50> (a java.lang.Class for com.arjuna.ats.internal.arjuna.recovery.ExpiredEntryMonitor)

           at com.arjuna.ats.internal.arjuna.recovery.RecoveryManagerImple.stop(RecoveryManagerImple.java:209)

           at com.arjuna.ats.arjuna.recovery.RecoveryManager.terminate(RecoveryManager.java:213)

           - locked <1064b6c68> (a com.arjuna.ats.arjuna.recovery.RecoveryManager)

           at com.arjuna.ats.arjuna.recovery.RecoveryManager.terminate(RecoveryManager.java:197)

           at com.arjuna.ats.jbossatx.jta.RecoveryManagerService.stop(RecoveryManagerService.java:84)

           at org.jboss.as.txn.TransactionManagerService.stop(TransactionManagerService.java:224)

           - locked <1064b6bc0> (a org.jboss.as.txn.TransactionManagerService)

           at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1061)

           at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

           at java.lang.Thread.run(Thread.java:637)

       

      "Poller SunPKCS11-Darwin" daemon prio=1 tid=12e103000 nid=0x130361000 waiting on condition [130360000]

         java.lang.Thread.State: TIMED_WAITING (sleeping)

           at java.lang.Thread.sleep(Native Method)

           at sun.security.pkcs11.SunPKCS11$TokenPoller.run(SunPKCS11.java:692)

           at java.lang.Thread.run(Thread.java:637)

       

      "DestroyJavaVM" prio=5 tid=12e102000 nid=0x100501000 waiting on condition [00000000]

         java.lang.Thread.State: RUNNABLE

       

      "ClassLoader Thread" daemon prio=5 tid=1018e0000 nid=0x12fd7a000 in Object.wait() [12fd79000]

         java.lang.Thread.State: WAITING (on object monitor)

           at java.lang.Object.wait(Native Method)

           - waiting on <107b27cb8> (a java.util.ArrayDeque)

           at java.lang.Object.wait(Object.java:485)

           at org.jboss.modules.ConcurrentClassLoader$LoaderThread.run(ConcurrentClassLoader.java:329)

           - locked <107b27cb8> (a java.util.ArrayDeque)

       

      "Low Memory Detector" daemon prio=5 tid=101878800 nid=0x12fa90000 runnable [00000000]

         java.lang.Thread.State: RUNNABLE

       

      "CompilerThread1" daemon prio=9 tid=101878000 nid=0x12f98d000 waiting on condition [00000000]

         java.lang.Thread.State: RUNNABLE

       

      "CompilerThread0" daemon prio=9 tid=101877000 nid=0x12f88a000 waiting on condition [00000000]

         java.lang.Thread.State: RUNNABLE

       

      "Signal Dispatcher" daemon prio=9 tid=101876800 nid=0x12f787000 runnable [00000000]

         java.lang.Thread.State: RUNNABLE

       

      "Surrogate Locker Thread (CMS)" daemon prio=5 tid=12e03c800 nid=0x12f684000 waiting on condition [00000000]

         java.lang.Thread.State: RUNNABLE

       

      "Finalizer" daemon prio=8 tid=101875800 nid=0x12f307000 in Object.wait() [12f306000]

         java.lang.Thread.State: WAITING (on object monitor)

           at java.lang.Object.wait(Native Method)

           - waiting on <107ca9848> (a java.lang.ref.ReferenceQueue$Lock)

           at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)

           - locked <107ca9848> (a java.lang.ref.ReferenceQueue$Lock)

           at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)

           at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

       

      "Reference Handler" daemon prio=10 tid=101875000 nid=0x12f204000 in Object.wait() [12f203000]

         java.lang.Thread.State: WAITING (on object monitor)

           at java.lang.Object.wait(Native Method)

           - waiting on <107ca8f60> (a java.lang.ref.Reference$Lock)

           at java.lang.Object.wait(Object.java:485)

           at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)

           - locked <107ca8f60> (a java.lang.ref.Reference$Lock)

       

      "VM Thread" prio=9 tid=12e022000 nid=0x12f101000 runnable

       

      "Gang worker#0 (Parallel GC Threads)" prio=9 tid=101802800 nid=0x10171b000 runnable

       

      "Gang worker#1 (Parallel GC Threads)" prio=9 tid=101803000 nid=0x105102000 runnable

       

      "Concurrent Mark-Sweep GC Thread" prio=9 tid=101842800 nid=0x12dbe6000 runnable

      "VM Periodic Task Thread" prio=10 tid=10187a000 nid=0x12fb93000 waiting on condition

       

      "Exception Catcher Thread" prio=10 tid=101801800 nid=0x100636000 runnable

      JNI global references: 889

       

      Thread-3 is run by an executor as part of the server shutdown process, and from looking at the code it is waiting for Thread-6 to finish. Thread-6 is stuck in a Socket.connect(), so it sounds like whatever it is trying to connect to is open but no longer accepting connections. Can someone point me towards what TransactionStatusConnector is trying to connect to?