-
1. Re: Domain controller loses connection to slaves
mackerman May 14, 2012 3:26 PM (in response to mackerman)I should add that there is no other indication that either the master or slaves have encountered any issues, communication or otherwise. Also, the same cluster configuration worked fine in JBoss 7.0.2, from which I have just upgraded.
-
2. Re: Domain controller loses connection to slaves
emuckenhuber May 15, 2012 4:00 AM (in response to mackerman)That should not happen. In case this is easy to reproduce for you, you might want to try with a nightly build: https://community.jboss.org/thread/167590
There should have been some fixes for reconnecting and also error reporting in the meantime. Perhaps you can also attach a thread-dump of the Host Controller process when you run into this issue again.
-
3. Re: Domain controller loses connection to slaves
mackerman May 15, 2012 4:37 PM (in response to emuckenhuber)Here is the host controller log file with thread-dump:
=========================================================================
JBoss Bootstrap Environment
JBOSS_HOME: /home/jboss/jboss
JAVA: java
JAVA_OPTS: -Xms64m -Xmx512m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.domain.default.config=domain.xml -Djboss.host.default.config=host.xml
=========================================================================
15:18:27,335 INFO [org.jboss.modules] (main) JBoss Modules version 1.1.1.GA
15:18:27,483 INFO [org.jboss.as.process.Host Controller.status] (main) JBAS012017: Starting process 'Host Controller'
[Host Controller] 15:18:28,280 INFO [org.jboss.modules] (main) JBoss Modules version 1.1.1.GA
[Host Controller] 15:18:28,550 INFO [org.jboss.msc] (main) JBoss MSC version 1.0.2.GA
[Host Controller] 15:18:28,702 INFO [org.jboss.as] (MSC service thread 1-3) JBAS015899: JBoss AS 7.1.1.Final "Brontes" starting
[Host Controller] 15:18:29,651 INFO [org.xnio] (MSC service thread 1-4) XNIO Version 3.0.3.GA
[Host Controller] 15:18:29,676 INFO [org.xnio.nio] (MSC service thread 1-4) XNIO NIO Implementation Version 3.0.3.GA
[Host Controller] 15:18:29,698 INFO [org.jboss.remoting] (MSC service thread 1-4) JBoss Remoting version 3.2.3.GA
[Host Controller] 15:18:31,089 INFO [org.jboss.as] (Controller Boot Thread) JBAS010902: Creating http management service using network interface (management) port (9990) securePort (-1)
[Host Controller] 15:18:31,109 INFO [org.jboss.as.remoting] (MSC service thread 1-3) JBAS017100: Listening on /0.0.0.0:9999
[Host Controller] 15:18:31,194 INFO [org.jboss.as] (Controller Boot Thread) JBAS015951: Admin console listening on http://0.0.0.0:9990
[Host Controller] 15:18:31,195 INFO [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss AS 7.1.1.Final "Brontes" (Host Controller) started in 3595ms - Started 11 of 11 services (0 services are passive or on-demand)
[Host Controller] 15:19:00,550 INFO [org.jboss.as.domain] (domain-mgmt-handler-thread - 1) JBAS010918: Registered remote slave host "BobServer10810227", JBoss AS 7.1.1.Final "Brontes"
[Host Controller] 17:46:09,228 INFO [org.jboss.as.domain] (domain-mgmt-handler-thread - 2) JBAS010918: Registered remote slave host "BobServer108165227", JBoss AS 7.1.1.Final "Brontes"
[Host Controller] 20:07:57,100 ERROR [org.jboss.remoting.remote.connection] (Remoting "bob-domain-mgr:MANAGEMENT" read-1) JBREM000200: Remote connection failed: java.io.IOException: Connection reset by peer
[Host Controller] 20:07:57,117 WARN [org.jboss.as.domain] (Remoting "bob-domain-mgr:MANAGEMENT" task-3) JBAS010929: Connection to remote host "BobServer10810227" closed unexpected
[Host Controller] 20:07:57,119 INFO [org.jboss.as.domain] (Remoting "bob-domain-mgr:MANAGEMENT" task-3) JBAS010925: Unregistered remote slave host "BobServer10810227"
[Host Controller] 20:17:03,082 ERROR [org.jboss.remoting.remote.connection] (Remoting "bob-domain-mgr:MANAGEMENT" read-1) JBREM000200: Remote connection failed: java.io.IOException: Connection reset by peer
[Host Controller] 20:17:03,087 WARN [org.jboss.as.domain] (Remoting "bob-domain-mgr:MANAGEMENT" task-1) JBAS010929: Connection to remote host "BobServer108165227" closed unexpected
[Host Controller] 20:17:03,089 INFO [org.jboss.as.domain] (Remoting "bob-domain-mgr:MANAGEMENT" task-1) JBAS010925: Unregistered remote slave host "BobServer108165227"
[Host Controller] 2012-05-15 20:31:47
[Host Controller] Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.5-b03 mixed mode):
[Host Controller]
[Host Controller] "Remoting "bob-domain-mgr:MANAGEMENT" task-4" prio=10 tid=0x00007fa4e0417800 nid=0x4ca5 waiting on condition [0x00007fa50433f000]
[Host Controller] java.lang.Thread.State: WAITING (parking)
[Host Controller] at sun.misc.Unsafe.park(Native Method)
[Host Controller] - parking to wait for <0x00000000e0058560> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[Host Controller] at java.util.concurrent.locks.LockSupport.park(Unknown Source)
[Host Controller] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
[Host Controller] at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
[Host Controller] at org.xnio.LimitedBlockingQueue.take(LimitedBlockingQueue.java:95)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[Host Controller] at java.lang.Thread.run(Unknown Source)
[Host Controller]
[Host Controller] "Remoting "bob-domain-mgr:MANAGEMENT" task-3" prio=10 tid=0x00007fa4e0416800 nid=0x4ca4 waiting on condition [0x00007fa504440000]
[Host Controller] java.lang.Thread.State: WAITING (parking)
[Host Controller] at sun.misc.Unsafe.park(Native Method)
[Host Controller] - parking to wait for <0x00000000e0058560> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[Host Controller] at java.util.concurrent.locks.LockSupport.park(Unknown Source)
[Host Controller] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
[Host Controller] at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
[Host Controller] at org.xnio.LimitedBlockingQueue.take(LimitedBlockingQueue.java:95)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[Host Controller] at java.lang.Thread.run(Unknown Source)
[Host Controller]
[Host Controller] "Remoting "bob-domain-mgr:MANAGEMENT" task-2" prio=10 tid=0x00007fa4e0538800 nid=0x4ca3 waiting on condition [0x00007fa504541000]
[Host Controller] java.lang.Thread.State: WAITING (parking)
[Host Controller] at sun.misc.Unsafe.park(Native Method)
[Host Controller] - parking to wait for <0x00000000e0058560> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[Host Controller] at java.util.concurrent.locks.LockSupport.park(Unknown Source)
[Host Controller] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
[Host Controller] at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
[Host Controller] at org.xnio.LimitedBlockingQueue.take(LimitedBlockingQueue.java:95)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[Host Controller] at java.lang.Thread.run(Unknown Source)
[Host Controller]
[Host Controller] "Remoting "bob-domain-mgr:MANAGEMENT" task-1" prio=10 tid=0x00007fa4e042f800 nid=0x4ca2 waiting on condition [0x00007fa506572000]
[Host Controller] java.lang.Thread.State: WAITING (parking)
[Host Controller] at sun.misc.Unsafe.park(Native Method)
[Host Controller] - parking to wait for <0x00000000e0058560> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[Host Controller] at java.util.concurrent.locks.LockSupport.park(Unknown Source)
[Host Controller] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
[Host Controller] at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
[Host Controller] at org.xnio.LimitedBlockingQueue.take(LimitedBlockingQueue.java:95)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[Host Controller] at java.lang.Thread.run(Unknown Source)
[Host Controller]
[Host Controller] "Thread-29" prio=10 tid=0x000000004117d000 nid=0x4c9e runnable [0x00007fa504648000]
[Host Controller] java.lang.Thread.State: RUNNABLE
[Host Controller] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
[Host Controller] at sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)
[Host Controller] at sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)
[Host Controller] at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
[Host Controller] - locked <0x00000000e041e9c8> (a sun.nio.ch.Util$2)
[Host Controller] - locked <0x00000000e041e9b8> (a java.util.Collections$UnmodifiableSet)
[Host Controller] - locked <0x00000000e041e760> (a sun.nio.ch.EPollSelectorImpl)
[Host Controller] at sun.nio.ch.SelectorImpl.select(Unknown Source)
[Host Controller] at org.jboss.sun.net.httpserver.ServerImpl$Dispatcher.run(ServerImpl.java:382)
[Host Controller] at java.lang.Thread.run(Unknown Source)
[Host Controller]
[Host Controller] "server-timer1" daemon prio=10 tid=0x0000000041139800 nid=0x4c9d in Object.wait() [0x00007fa5047f0000]
[Host Controller] java.lang.Thread.State: TIMED_WAITING (on object monitor)
[Host Controller] at java.lang.Object.wait(Native Method)
[Host Controller] - waiting on <0x00000000e0438630> (a java.util.TaskQueue)
[Host Controller] at java.util.TimerThread.mainLoop(Unknown Source)
[Host Controller] - locked <0x00000000e0438630> (a java.util.TaskQueue)
[Host Controller] at java.util.TimerThread.run(Unknown Source)
[Host Controller]
[Host Controller] "server-timer" daemon prio=10 tid=0x0000000041165800 nid=0x4c9c in Object.wait() [0x00007fa5048f1000]
[Host Controller] java.lang.Thread.State: TIMED_WAITING (on object monitor)
[Host Controller] at java.lang.Object.wait(Native Method)
[Host Controller] - waiting on <0x00000000e041f088> (a java.util.TaskQueue)
[Host Controller] at java.util.TimerThread.mainLoop(Unknown Source)
[Host Controller] - locked <0x00000000e041f088> (a java.util.TaskQueue)
[Host Controller] at java.util.TimerThread.run(Unknown Source)
[Host Controller]
[Host Controller] "Remoting "bob-domain-mgr:MANAGEMENT" write-1" prio=10 tid=0x000000004112c000 nid=0x4c99 runnable [0x00007fa504b3b000]
[Host Controller] java.lang.Thread.State: RUNNABLE
[Host Controller] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
[Host Controller] at sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)
[Host Controller] at sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)
[Host Controller] at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
[Host Controller] - locked <0x00000000e044f218> (a sun.nio.ch.Util$2)
[Host Controller] - locked <0x00000000e044f208> (a java.util.Collections$UnmodifiableSet)
[Host Controller] - locked <0x00000000e0439720> (a sun.nio.ch.EPollSelectorImpl)
[Host Controller] at sun.nio.ch.SelectorImpl.select(Unknown Source)
[Host Controller] at org.xnio.nio.WorkerThread.run(WorkerThread.java:157)
[Host Controller]
[Host Controller] "Remoting "bob-domain-mgr:MANAGEMENT" read-1" prio=10 tid=0x000000004112b000 nid=0x4c98 runnable [0x00007fa504c3c000]
[Host Controller] java.lang.Thread.State: RUNNABLE
[Host Controller] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
[Host Controller] at sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)
[Host Controller] at sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)
[Host Controller] at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
[Host Controller] - locked <0x00000000e00582b8> (a sun.nio.ch.Util$2)
[Host Controller] - locked <0x00000000e00582a8> (a java.util.Collections$UnmodifiableSet)
[Host Controller] - locked <0x00000000e0033600> (a sun.nio.ch.EPollSelectorImpl)
[Host Controller] at sun.nio.ch.SelectorImpl.select(Unknown Source)
[Host Controller] at sun.nio.ch.SelectorImpl.select(Unknown Source)
[Host Controller] at org.xnio.nio.WorkerThread.run(WorkerThread.java:153)
[Host Controller]
[Host Controller] "pool-3-thread-20" prio=10 tid=0x00007fa4c412f800 nid=0x4c97 waiting on condition [0x00007fa504d86000]
[Host Controller] java.lang.Thread.State: WAITING (parking)
[Host Controller] at sun.misc.Unsafe.park(Native Method)
[Host Controller] - parking to wait for <0x00000000f9907600> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[Host Controller] at java.util.concurrent.locks.LockSupport.park(Unknown Source)
[Host Controller] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
[Host Controller] at java.util.concurrent.DelayQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[Host Controller] at java.lang.Thread.run(Unknown Source)
[Host Controller] at org.jboss.threads.JBossThread.run(JBossThread.java:122)
[Host Controller]
[Host Controller] "pool-3-thread-19" prio=10 tid=0x00007fa4c412e000 nid=0x4c96 waiting on condition [0x00007fa504e87000]
[Host Controller] java.lang.Thread.State: WAITING (parking)
[Host Controller] at sun.misc.Unsafe.park(Native Method)
[Host Controller] - parking to wait for <0x00000000f9907600> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[Host Controller] at java.util.concurrent.locks.LockSupport.park(Unknown Source)
[Host Controller] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
[Host Controller] at java.util.concurrent.DelayQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[Host Controller] at java.lang.Thread.run(Unknown Source)
[Host Controller] at org.jboss.threads.JBossThread.run(JBossThread.java:122)
[Host Controller]
[Host Controller] "pool-3-thread-18" prio=10 tid=0x00007fa4c412c000 nid=0x4c95 waiting on condition [0x00007fa504f88000]
[Host Controller] java.lang.Thread.State: WAITING (parking)
[Host Controller] at sun.misc.Unsafe.park(Native Method)
[Host Controller] - parking to wait for <0x00000000f9907600> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[Host Controller] at java.util.concurrent.locks.LockSupport.park(Unknown Source)
[Host Controller] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
[Host Controller] at java.util.concurrent.DelayQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[Host Controller] at java.lang.Thread.run(Unknown Source)
[Host Controller] at org.jboss.threads.JBossThread.run(JBossThread.java:122)
[Host Controller]
[Host Controller] "pool-3-thread-17" prio=10 tid=0x00007fa4c4129800 nid=0x4c94 waiting on condition [0x00007fa505097000]
[Host Controller] java.lang.Thread.State: WAITING (parking)
[Host Controller] at sun.misc.Unsafe.park(Native Method)
[Host Controller] - parking to wait for <0x00000000f9907600> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[Host Controller] at java.util.concurrent.locks.LockSupport.park(Unknown Source)
[Host Controller] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
[Host Controller] at java.util.concurrent.DelayQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[Host Controller] at java.lang.Thread.run(Unknown Source)
[Host Controller] at org.jboss.threads.JBossThread.run(JBossThread.java:122)
[Host Controller]
[Host Controller] "pool-3-thread-16" prio=10 tid=0x00007fa4c4127800 nid=0x4c93 waiting on condition [0x00007fa5051ab000]
[Host Controller] java.lang.Thread.State: WAITING (parking)
[Host Controller] at sun.misc.Unsafe.park(Native Method)
[Host Controller] - parking to wait for <0x00000000f9907600> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[Host Controller] at java.util.concurrent.locks.LockSupport.park(Unknown Source)
[Host Controller] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
[Host Controller] at java.util.concurrent.DelayQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[Host Controller] at java.lang.Thread.run(Unknown Source)
[Host Controller] at org.jboss.threads.JBossThread.run(JBossThread.java:122)
[Host Controller]
[Host Controller] "pool-3-thread-15" prio=10 tid=0x00007fa4c4125800 nid=0x4c92 waiting on condition [0x00007fa5052e0000]
[Host Controller] java.lang.Thread.State: WAITING (parking)
[Host Controller] at sun.misc.Unsafe.park(Native Method)
[Host Controller] - parking to wait for <0x00000000f9907600> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[Host Controller] at java.util.concurrent.locks.LockSupport.park(Unknown Source)
[Host Controller] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
[Host Controller] at java.util.concurrent.DelayQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[Host Controller] at java.lang.Thread.run(Unknown Source)
[Host Controller] at org.jboss.threads.JBossThread.run(JBossThread.java:122)
[Host Controller]
[Host Controller] "pool-3-thread-14" prio=10 tid=0x00007fa4c4124000 nid=0x4c91 waiting on condition [0x00007fa5054b3000]
[Host Controller] java.lang.Thread.State: WAITING (parking)
[Host Controller] at sun.misc.Unsafe.park(Native Method)
[Host Controller] - parking to wait for <0x00000000f9907600> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[Host Controller] at java.util.concurrent.locks.LockSupport.park(Unknown Source)
[Host Controller] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
[Host Controller] at java.util.concurrent.DelayQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[Host Controller] at java.lang.Thread.run(Unknown Source)
[Host Controller] at org.jboss.threads.JBossThread.run(JBossThread.java:122)
[Host Controller]
[Host Controller] "pool-3-thread-13" prio=10 tid=0x00007fa4c4122000 nid=0x4c90 waiting on condition [0x00007fa505645000]
[Host Controller] java.lang.Thread.State: WAITING (parking)
[Host Controller] at sun.misc.Unsafe.park(Native Method)
[Host Controller] - parking to wait for <0x00000000f9907600> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[Host Controller] at java.util.concurrent.locks.LockSupport.park(Unknown Source)
[Host Controller] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
[Host Controller] at java.util.concurrent.DelayQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[Host Controller] at java.lang.Thread.run(Unknown Source)
[Host Controller] at org.jboss.threads.JBossThread.run(JBossThread.java:122)
[Host Controller]
[Host Controller] "pool-3-thread-12" prio=10 tid=0x00007fa4c4120000 nid=0x4c8f waiting on condition [0x00007fa505755000]
[Host Controller] java.lang.Thread.State: WAITING (parking)
[Host Controller] at sun.misc.Unsafe.park(Native Method)
[Host Controller] - parking to wait for <0x00000000f9907600> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[Host Controller] at java.util.concurrent.locks.LockSupport.park(Unknown Source)
[Host Controller] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
[Host Controller] at java.util.concurrent.DelayQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[Host Controller] at java.lang.Thread.run(Unknown Source)
[Host Controller] at org.jboss.threads.JBossThread.run(JBossThread.java:122)
[Host Controller]
[Host Controller] "pool-3-thread-11" prio=10 tid=0x00007fa4c411e000 nid=0x4c8e waiting on condition [0x00007fa505961000]
[Host Controller] java.lang.Thread.State: WAITING (parking)
[Host Controller] at sun.misc.Unsafe.park(Native Method)
[Host Controller] - parking to wait for <0x00000000f9907600> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[Host Controller] at java.util.concurrent.locks.LockSupport.park(Unknown Source)
[Host Controller] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
[Host Controller] at java.util.concurrent.DelayQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[Host Controller] at java.lang.Thread.run(Unknown Source)
[Host Controller] at org.jboss.threads.JBossThread.run(JBossThread.java:122)
[Host Controller]
[Host Controller] "pool-3-thread-10" prio=10 tid=0x00007fa4c411c000 nid=0x4c8d waiting on condition [0x00007fa505a62000]
[Host Controller] java.lang.Thread.State: WAITING (parking)
[Host Controller] at sun.misc.Unsafe.park(Native Method)
[Host Controller] - parking to wait for <0x00000000f9907600> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[Host Controller] at java.util.concurrent.locks.LockSupport.park(Unknown Source)
[Host Controller] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
[Host Controller] at java.util.concurrent.DelayQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[Host Controller] at java.lang.Thread.run(Unknown Source)
[Host Controller] at org.jboss.threads.JBossThread.run(JBossThread.java:122)
[Host Controller]
[Host Controller] "pool-3-thread-9" prio=10 tid=0x00007fa4c411a000 nid=0x4c8c waiting on condition [0x00007fa505b63000]
[Host Controller] java.lang.Thread.State: WAITING (parking)
[Host Controller] at sun.misc.Unsafe.park(Native Method)
[Host Controller] - parking to wait for <0x00000000f9907600> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[Host Controller] at java.util.concurrent.locks.LockSupport.park(Unknown Source)
[Host Controller] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
[Host Controller] at java.util.concurrent.DelayQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[Host Controller] at java.lang.Thread.run(Unknown Source)
[Host Controller] at org.jboss.threads.JBossThread.run(JBossThread.java:122)
[Host Controller]
[Host Controller] "pool-3-thread-8" prio=10 tid=0x00007fa4c4118000 nid=0x4c8b waiting on condition [0x00007fa505c67000]
[Host Controller] java.lang.Thread.State: WAITING (parking)
[Host Controller] at sun.misc.Unsafe.park(Native Method)
[Host Controller] - parking to wait for <0x00000000f9907600> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[Host Controller] at java.util.concurrent.locks.LockSupport.park(Unknown Source)
[Host Controller] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
[Host Controller] at java.util.concurrent.DelayQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[Host Controller] at java.lang.Thread.run(Unknown Source)
[Host Controller] at org.jboss.threads.JBossThread.run(JBossThread.java:122)
[Host Controller]
[Host Controller] "pool-3-thread-7" prio=10 tid=0x00007fa4c4116000 nid=0x4c8a waiting on condition [0x00007fa505d68000]
[Host Controller] java.lang.Thread.State: WAITING (parking)
[Host Controller] at sun.misc.Unsafe.park(Native Method)
[Host Controller] - parking to wait for <0x00000000f9907600> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[Host Controller] at java.util.concurrent.locks.LockSupport.park(Unknown Source)
[Host Controller] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
[Host Controller] at java.util.concurrent.DelayQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[Host Controller] at java.lang.Thread.run(Unknown Source)
[Host Controller] at org.jboss.threads.JBossThread.run(JBossThread.java:122)
[Host Controller]
[Host Controller] "pool-3-thread-6" prio=10 tid=0x00007fa4c4114800 nid=0x4c87 waiting on condition [0x00007fa505e6c000]
[Host Controller] java.lang.Thread.State: WAITING (parking)
[Host Controller] at sun.misc.Unsafe.park(Native Method)
[Host Controller] - parking to wait for <0x00000000f9907600> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[Host Controller] at java.util.concurrent.locks.LockSupport.park(Unknown Source)
[Host Controller] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
[Host Controller] at java.util.concurrent.DelayQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[Host Controller] at java.lang.Thread.run(Unknown Source)
[Host Controller] at org.jboss.threads.JBossThread.run(JBossThread.java:122)
[Host Controller]
[Host Controller] "pool-3-thread-5" prio=10 tid=0x00007fa4c40e1800 nid=0x4c86 waiting on condition [0x00007fa505f6d000]
[Host Controller] java.lang.Thread.State: WAITING (parking)
[Host Controller] at sun.misc.Unsafe.park(Native Method)
[Host Controller] - parking to wait for <0x00000000f9907600> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[Host Controller] at java.util.concurrent.locks.LockSupport.park(Unknown Source)
[Host Controller] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
[Host Controller] at java.util.concurrent.DelayQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[Host Controller] at java.lang.Thread.run(Unknown Source)
[Host Controller] at org.jboss.threads.JBossThread.run(JBossThread.java:122)
[Host Controller]
[Host Controller] "pool-3-thread-4" prio=10 tid=0x00007fa4c40e0800 nid=0x4c85 waiting on condition [0x00007fa50606e000]
[Host Controller] java.lang.Thread.State: WAITING (parking)
[Host Controller] at sun.misc.Unsafe.park(Native Method)
[Host Controller] - parking to wait for <0x00000000f9907600> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[Host Controller] at java.util.concurrent.locks.LockSupport.park(Unknown Source)
[Host Controller] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
[Host Controller] at java.util.concurrent.DelayQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[Host Controller] at java.lang.Thread.run(Unknown Source)
[Host Controller] at org.jboss.threads.JBossThread.run(JBossThread.java:122)
[Host Controller]
[Host Controller] "pool-3-thread-3" prio=10 tid=0x00007fa4c40a6000 nid=0x4c84 waiting on condition [0x00007fa50616f000]
[Host Controller] java.lang.Thread.State: WAITING (parking)
[Host Controller] at sun.misc.Unsafe.park(Native Method)
[Host Controller] - parking to wait for <0x00000000f9907600> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[Host Controller] at java.util.concurrent.locks.LockSupport.park(Unknown Source)
[Host Controller] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
[Host Controller] at java.util.concurrent.DelayQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[Host Controller] at java.lang.Thread.run(Unknown Source)
[Host Controller] at org.jboss.threads.JBossThread.run(JBossThread.java:122)
[Host Controller]
[Host Controller] "pool-3-thread-2" prio=10 tid=0x00007fa4c40a0800 nid=0x4c83 waiting on condition [0x00007fa506270000]
[Host Controller] java.lang.Thread.State: WAITING (parking)
[Host Controller] at sun.misc.Unsafe.park(Native Method)
[Host Controller] - parking to wait for <0x00000000f9907600> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[Host Controller] at java.util.concurrent.locks.LockSupport.park(Unknown Source)
[Host Controller] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
[Host Controller] at java.util.concurrent.DelayQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[Host Controller] at java.lang.Thread.run(Unknown Source)
[Host Controller] at org.jboss.threads.JBossThread.run(JBossThread.java:122)
[Host Controller]
[Host Controller] "pool-3-thread-1" prio=10 tid=0x00007fa4c40cd800 nid=0x4c82 waiting on condition [0x00007fa506371000]
[Host Controller] java.lang.Thread.State: WAITING (parking)
[Host Controller] at sun.misc.Unsafe.park(Native Method)
[Host Controller] - parking to wait for <0x00000000f9907600> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[Host Controller] at java.util.concurrent.locks.LockSupport.park(Unknown Source)
[Host Controller] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
[Host Controller] at java.util.concurrent.DelayQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[Host Controller] at java.lang.Thread.run(Unknown Source)
[Host Controller] at org.jboss.threads.JBossThread.run(JBossThread.java:122)
[Host Controller]
[Host Controller] "Read thread for localhost/127.0.0.1:51406" prio=10 tid=0x00007fa4d0007800 nid=0x4c7f runnable [0x00007fa506673000]
[Host Controller] java.lang.Thread.State: RUNNABLE
[Host Controller] at java.net.SocketInputStream.socketRead0(Native Method)
[Host Controller] at java.net.SocketInputStream.read(Unknown Source)
[Host Controller] at java.net.SocketInputStream.read(Unknown Source)
[Host Controller] at org.jboss.as.process.protocol.ConnectionImpl$1.run(ConnectionImpl.java:190)
[Host Controller] at java.lang.Thread.run(Unknown Source)
[Host Controller] at org.jboss.threads.JBossThread.run(JBossThread.java:122)
[Host Controller]
[Host Controller] "MSC service thread 1-4" prio=10 tid=0x00007fa4c8019800 nid=0x4c7d waiting on condition [0x00007fa506856000]
[Host Controller] java.lang.Thread.State: WAITING (parking)
[Host Controller] at sun.misc.Unsafe.park(Native Method)
[Host Controller] - parking to wait for <0x00000000e00338b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[Host Controller] at java.util.concurrent.locks.LockSupport.park(Unknown Source)
[Host Controller] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
[Host Controller] at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[Host Controller] at java.lang.Thread.run(Unknown Source)
[Host Controller]
[Host Controller] "MSC service thread 1-3" prio=10 tid=0x00007fa4c4001800 nid=0x4c7a waiting on condition [0x00007fa506957000]
[Host Controller] java.lang.Thread.State: WAITING (parking)
[Host Controller] at sun.misc.Unsafe.park(Native Method)
[Host Controller] - parking to wait for <0x00000000e00338b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[Host Controller] at java.util.concurrent.locks.LockSupport.park(Unknown Source)
[Host Controller] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
[Host Controller] at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[Host Controller] at java.lang.Thread.run(Unknown Source)
[Host Controller]
[Host Controller] "MSC service thread 1-2" prio=10 tid=0x00007fa4d0001000 nid=0x4c79 waiting on condition [0x00007fa506a58000]
[Host Controller] java.lang.Thread.State: WAITING (parking)
[Host Controller] at sun.misc.Unsafe.park(Native Method)
[Host Controller] - parking to wait for <0x00000000e00338b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[Host Controller] at java.util.concurrent.locks.LockSupport.park(Unknown Source)
[Host Controller] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
[Host Controller] at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[Host Controller] at java.lang.Thread.run(Unknown Source)
[Host Controller]
[Host Controller] "MSC service thread 1-1" prio=10 tid=0x00007fa508422800 nid=0x4c78 waiting on condition [0x00007fa506b59000]
[Host Controller] java.lang.Thread.State: WAITING (parking)
[Host Controller] at sun.misc.Unsafe.park(Native Method)
[Host Controller] - parking to wait for <0x00000000e00338b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[Host Controller] at java.util.concurrent.locks.LockSupport.park(Unknown Source)
[Host Controller] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
[Host Controller] at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
[Host Controller] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[Host Controller] at java.lang.Thread.run(Unknown Source)
[Host Controller]
[Host Controller] "Reference Reaper" daemon prio=10 tid=0x00007fa5080fe800 nid=0x4c75 in Object.wait() [0x00007fa506f49000]
[Host Controller] java.lang.Thread.State: WAITING (on object monitor)
[Host Controller] at java.lang.Object.wait(Native Method)
[Host Controller] - waiting on <0x00000000e04aca10> (a java.lang.ref.ReferenceQueue$Lock)
[Host Controller] at java.lang.ref.ReferenceQueue.remove(Unknown Source)
[Host Controller] - locked <0x00000000e04aca10> (a java.lang.ref.ReferenceQueue$Lock)
[Host Controller] at java.lang.ref.ReferenceQueue.remove(Unknown Source)
[Host Controller] at org.jboss.modules.ref.References$ReaperThread.run(References.java:68)
[Host Controller]
[Host Controller] "Low Memory Detector" daemon prio=10 tid=0x00007fa5080b2000 nid=0x4c73 runnable [0x0000000000000000]
[Host Controller] java.lang.Thread.State: RUNNABLE
[Host Controller]
[Host Controller] "C2 CompilerThread1" daemon prio=10 tid=0x00007fa5080b0000 nid=0x4c72 waiting on condition [0x0000000000000000]
[Host Controller] java.lang.Thread.State: RUNNABLE
[Host Controller]
[Host Controller] "C2 CompilerThread0" daemon prio=10 tid=0x00007fa5080ad000 nid=0x4c71 waiting on condition [0x0000000000000000]
[Host Controller] java.lang.Thread.State: RUNNABLE
[Host Controller]
[Host Controller] "Signal Dispatcher" daemon prio=10 tid=0x00007fa5080ab000 nid=0x4c70 waiting on condition [0x0000000000000000]
[Host Controller] java.lang.Thread.State: RUNNABLE
[Host Controller]
[Host Controller] "Finalizer" daemon prio=10 tid=0x00007fa508091000 nid=0x4c6f in Object.wait() [0x00007fa507559000]
[Host Controller] java.lang.Thread.State: WAITING (on object monitor)
[Host Controller] at java.lang.Object.wait(Native Method)
[Host Controller] - waiting on <0x00000000e0059098> (a java.lang.ref.ReferenceQueue$Lock)
[Host Controller] at java.lang.ref.ReferenceQueue.remove(Unknown Source)
[Host Controller] - locked <0x00000000e0059098> (a java.lang.ref.ReferenceQueue$Lock)
[Host Controller] at java.lang.ref.ReferenceQueue.remove(Unknown Source)
[Host Controller] at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)
[Host Controller]
[Host Controller] "Reference Handler" daemon prio=10 tid=0x00007fa50808f000 nid=0x4c6e in Object.wait() [0x00007fa50765a000]
[Host Controller] java.lang.Thread.State: WAITING (on object monitor)
[Host Controller] at java.lang.Object.wait(Native Method)
[Host Controller] - waiting on <0x00000000e0313c58> (a java.lang.ref.Reference$Lock)
[Host Controller] at java.lang.Object.wait(Object.java:485)
[Host Controller] at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)
[Host Controller] - locked <0x00000000e0313c58> (a java.lang.ref.Reference$Lock)
[Host Controller]
[Host Controller] "main" prio=10 tid=0x00007fa508007800 nid=0x4c67 runnable [0x00007fa50f848000]
[Host Controller] java.lang.Thread.State: RUNNABLE
[Host Controller] at java.io.FileInputStream.readBytes(Native Method)
[Host Controller] at java.io.FileInputStream.read(Unknown Source)
[Host Controller] at java.io.BufferedInputStream.fill(Unknown Source)
[Host Controller] at java.io.BufferedInputStream.read(Unknown Source)
[Host Controller] - locked <0x00000000e00591a8> (a java.io.BufferedInputStream)
[Host Controller] at org.jboss.as.host.controller.Main.main(Main.java:106)
[Host Controller] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[Host Controller] at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
[Host Controller] at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
[Host Controller] at java.lang.reflect.Method.invoke(Unknown Source)
[Host Controller] at org.jboss.modules.Module.run(Module.java:260)
[Host Controller] at org.jboss.modules.Main.main(Main.java:291)
[Host Controller]
[Host Controller] "VM Thread" prio=10 tid=0x00007fa508088000 nid=0x4c6d runnable
[Host Controller]
[Host Controller] "GC task thread#0 (ParallelGC)" prio=10 tid=0x00007fa50801a000 nid=0x4c6b runnable
[Host Controller]
[Host Controller] "GC task thread#1 (ParallelGC)" prio=10 tid=0x00007fa50801c000 nid=0x4c6c runnable
[Host Controller]
[Host Controller] "VM Periodic Task Thread" prio=10 tid=0x00007fa5080b5000 nid=0x4c74 waiting on condition
[Host Controller]
[Host Controller] JNI global references: 1687
[Host Controller]
[Host Controller] Heap
[Host Controller] PSYoungGen total 68480K, used 23023K [0x00000000f5560000, 0x00000000f9ae0000, 0x0000000100000000)
[Host Controller] eden space 65792K, 30% used [0x00000000f5560000,0x00000000f693d738,0x00000000f95a0000)
[Host Controller] from space 2688K, 99% used [0x00000000f9840000,0x00000000f9ade8a0,0x00000000f9ae0000)
[Host Controller] to space 2688K, 0% used [0x00000000f95a0000,0x00000000f95a0000,0x00000000f9840000)
[Host Controller] PSOldGen total 43712K, used 6885K [0x00000000e0000000, 0x00000000e2ab0000, 0x00000000f5560000)
[Host Controller] object space 43712K, 15% used [0x00000000e0000000,0x00000000e06b96d0,0x00000000e2ab0000)
[Host Controller] PSPermGen total 29120K, used 29077K [0x00000000d0000000, 0x00000000d1c70000, 0x00000000e0000000)
[Host Controller] object space 29120K, 99% used [0x00000000d0000000,0x00000000d1c65520,0x00000000d1c70000)
[Host Controller]
-
4. Re: Domain controller loses connection to slaves
emuckenhuber May 21, 2012 1:15 PM (in response to mackerman)I actually meant the connecting slave host-controller. I remember there was an issue with reconnect, which is fixed upstream.
We are also going to add some additional handling which hopefully is going to improve your situation further: https://issues.jboss.org/browse/AS7-4841
-
5. Re: Domain controller loses connection to slaves
mackerman May 24, 2012 11:20 AM (in response to emuckenhuber)Emanuel, tried the nightly build, unfortunately, the app would not deploy:
[Server:server-ip-10-81-0-227.pgilab.net] 15:10:07,606 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-4) MSC00001: Failed to start service jboss.deployment.unit."BobApp.war".STRUCTURE: org.jboss.msc.service.StartException in service jboss.deployment.unit."BobApp.war".STRUCTURE: JBAS018733: Failed to process phase STRUCTURE of deployment "BobApp.war"
[Server:server-ip-10-81-0-227.pgilab.net] at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:123) [jboss-as-server-7.2.0.Alpha1-SNAPSHOT.jar:7.2.0.Alpha1-SNAPSHOT]
[Server:server-ip-10-81-0-227.pgilab.net] at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]
[Server:server-ip-10-81-0-227.pgilab.net] at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]
[Server:server-ip-10-81-0-227.pgilab.net] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) [rt.jar:1.6.0_30]
[Server:server-ip-10-81-0-227.pgilab.net] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [rt.jar:1.6.0_30]
[Server:server-ip-10-81-0-227.pgilab.net] at java.lang.Thread.run(Unknown Source) [rt.jar:1.6.0_30]
[Server:server-ip-10-81-0-227.pgilab.net] Caused by: org.jboss.as.server.deployment.DeploymentUnitProcessingException: JBAS018740: Failed to mount deployment content
[Server:server-ip-10-81-0-227.pgilab.net] at org.jboss.as.server.deployment.module.DeploymentRootMountProcessor.deploy(DeploymentRootMountProcessor.java:92) [jboss-as-server-7.2.0.Alpha1-SNAPSHOT.jar:7.2.0.Alpha1-SNAPSHOT]
[Server:server-ip-10-81-0-227.pgilab.net] at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:116) [jboss-as-server-7.2.0.Alpha1-SNAPSHOT.jar:7.2.0.Alpha1-SNAPSHOT]
[Server:server-ip-10-81-0-227.pgilab.net] ... 5 more
[Server:server-ip-10-81-0-227.pgilab.net] Caused by: java.util.zip.ZipException: error in opening zip file
[Server:server-ip-10-81-0-227.pgilab.net] at java.util.zip.ZipFile.open(Native Method) [rt.jar:1.6.0_30]
[Server:server-ip-10-81-0-227.pgilab.net] at java.util.zip.ZipFile.<init>(Unknown Source) [rt.jar:1.6.0_30]
[Server:server-ip-10-81-0-227.pgilab.net] at java.util.zip.ZipFile.<init>(Unknown Source) [rt.jar:1.6.0_30]
[Server:server-ip-10-81-0-227.pgilab.net] at org.jboss.vfs.VFSUtils.unzip(VFSUtils.java:845) [jboss-vfs-3.1.0.Final.jar:3.1.0.Final]
[Server:server-ip-10-81-0-227.pgilab.net] at org.jboss.vfs.VFS.mountZipExpanded(VFS.java:536) [jboss-vfs-3.1.0.Final.jar:3.1.0.Final]
[Server:server-ip-10-81-0-227.pgilab.net] at org.jboss.vfs.VFS.mountZipExpanded(VFS.java:567) [jboss-vfs-3.1.0.Final.jar:3.1.0.Final]
[Server:server-ip-10-81-0-227.pgilab.net] at org.jboss.as.server.deployment.DeploymentMountProvider$Factory$ServerDeploymentRepositoryImpl.mountDeploymentContent(DeploymentMountProvider.java:97) [jboss-as-server-7.2.0.Alpha1-SNAPSHOT.jar:7.2.0.Alpha1-SNAPSHOT]
[Server:server-ip-10-81-0-227.pgilab.net] at org.jboss.as.server.deployment.module.DeploymentRootMountProcessor.deploy(DeploymentRootMountProcessor.java:88) [jboss-as-server-7.2.0.Alpha1-SNAPSHOT.jar:7.2.0.Alpha1-SNAPSHOT]
[Server:server-ip-10-81-0-227.pgilab.net] ... 6 more
[Server:server-ip-10-81-0-227.pgilab.net]
[Server:server-ip-10-81-0-227.pgilab.net] 15:10:07,629 INFO [org.jboss.as.server] (host-controller-connection-threads - 3) JBAS015870: Deploy of deployment "BobApp.war" was rolled back with failure message {"JBAS014671: Failed services" => {"jboss.deployment.unit.\"BobApp.war\".STRUCTURE" => "org.jboss.msc.service.StartException in service jboss.deployment.unit.\"BobApp.war\".STRUCTURE: JBAS018733: Failed to process phase STRUCTURE of deployment \"BobApp.war\"
[Server:server-ip-10-81-0-227.pgilab.net] Caused by: org.jboss.as.server.deployment.DeploymentUnitProcessingException: JBAS018740: Failed to mount deployment content
[Server:server-ip-10-81-0-227.pgilab.net] Caused by: java.util.zip.ZipException: error in opening zip file"}}
[Server:server-ip-10-81-0-227.pgilab.net] 15:10:07,632 INFO [org.jboss.as.server.deployment] (MSC service thread 1-3) JBAS015877: Stopped deployment BobApp.war in 3ms
-
6. Re: Domain controller loses connection to slaves
ctomc May 25, 2012 4:30 PM (in response to mackerman)This was on boot, or after depoying new application?
-
7. Re: Domain controller loses connection to slaves
mackerman May 25, 2012 5:07 PM (in response to ctomc)after deploying a new app
-
8. Re: Domain controller loses connection to slaves
emuckenhuber May 29, 2012 5:16 PM (in response to mackerman)Hmm, does this happen when using the CLI or the http API? I am wondering, because you mentioned using the http API on the other post.
-
9. Re: Domain controller loses connection to slaves
mackerman May 29, 2012 5:41 PM (in response to emuckenhuber)this was via the JBoss Management Console. I haven't used the CLI recently, the last I tried that the CLI agent hung.
-
10. Re: Domain controller loses connection to slaves
jaikiran May 30, 2012 2:33 AM (in response to mackerman)What kind of an application is this? Does it have any RichFaces jar(s) packaged in it? I see a couple of threads in this forum where there are reports of some issue with RichFaces jars failing with this error.
-
11. Re: Domain controller loses connection to slaves
emuckenhuber May 30, 2012 7:19 AM (in response to jaikiran)Yeah, the reference should be standalone. If it works there it should work in the domain as well. Otherewise it's a different issue.
One thing you can check when uploading the deployment - there should be a log message about content added to the deployment repository. The path contains the sha1 hash of the actual deployment content. So if that hash matches the sha1sum of the deployment you have locally the upload itself should be fine.
-
12. Re: Domain controller loses connection to slaves
mackerman May 30, 2012 10:45 AM (in response to jaikiran)It is not a rich faces application. here is the complete of 3rd party jars:
10/27/2011 01:27 PM 114,926 aspectjrt-1.6.1.jar
10/27/2011 01:27 PM 1,567,728 aspectjweaver-1.6.1.jar
10/14/2011 11:02 AM 188,671 commons-beanutils-1.7.0.jar
10/14/2011 11:02 AM 206,035 commons-beanutils-core-1.8.0.jar
10/14/2011 11:30 AM 30,085 commons-codec-1.2.jar
10/14/2011 11:02 AM 575,389 commons-collections-3.2.1.jar
10/14/2011 11:02 AM 298,829 commons-configuration-1.6.jar
10/14/2011 11:02 AM 107,631 commons-dbcp-1.2.1.jar
10/14/2011 11:02 AM 143,602 commons-digester-1.8.jar
10/14/2011 11:30 AM 305,001 commons-httpclient-3.1.jar
10/14/2011 11:03 AM 132,165 commons-jexl-1.1.jar
10/14/2011 11:02 AM 261,809 commons-lang-2.4.jar
10/14/2011 11:02 AM 60,686 commons-logging-1.1.1.jar
10/14/2011 11:02 AM 100,472 commons-pool-1.5.6.jar
10/14/2011 11:03 AM 176,153 commons-scxml-0.9.jar
10/14/2011 11:02 AM 171,851 concurrent-1.3.3.jar
05/03/2012 05:01 PM 1,491,672 hornetq-core-2.2.14.Final.jar
05/03/2012 05:01 PM 240,057 hornetq-jms-2.2.14.Final.jar
11/01/2011 10:30 AM 830,457 hornetq-ra.rar
05/03/2012 05:04 PM 107,176 hornetq-rest-2.2.14.Final.jar
10/14/2011 11:02 AM 706,710 hsqldb-1.8.0.10.jar
10/14/2011 03:33 PM 292,893 httpclient-4.0.3.jar
10/14/2011 03:33 PM 172,888 httpcore-4.0.1.jar
04/11/2012 02:01 PM 142,844 infinispan-client-hotrod-5.1.3.FINAL.jar
04/11/2012 02:01 PM 1,994,835 infinispan-core-5.1.3.FINAL.jar
05/08/2012 04:06 PM 765,648 jackson-mapper-asl-1.9.2.jar
02/06/2012 05:44 PM 53,814 jain-sip-api-1.2.jar
02/06/2012 05:44 PM 767,815 jain-sip-ri-1.2.1990.jar
10/14/2011 03:33 PM 542,241 javassist-3.6.0.GA.jar
10/14/2011 03:33 PM 103,515 jaxb-api-2.1.jar
10/14/2011 03:33 PM 867,801 jaxb-impl-2.1.12.jar
10/14/2011 03:33 PM 46,435 jaxrs-api-2.1.0.GA.jar
02/06/2012 05:44 PM 225,202 jboss-marshalling-1.3.6.GA.jar
02/06/2012 05:44 PM 81,389 jboss-marshalling-river-1.3.6.GA.jar
10/14/2011 11:02 AM 2,254 jcip-annotations-1.0.jar
10/17/2011 04:28 PM 72,478 jettison-1.2.jar
04/11/2012 02:01 PM 1,783,399 jgroups-3.0.8.Final.jar
10/14/2011 11:02 AM 1,904,171 jmf-2.1.1e.jar
10/14/2011 11:03 AM 6,567 jmxservice-1.1.1.jar
10/14/2011 11:02 AM 46,047 jsr173-1.0.jar
10/14/2011 03:33 PM 6,165 jsr250-api-1.0.jar
10/14/2011 11:02 AM 481,535 log4j-1.2.16.jar
02/16/2012 02:35 PM 2,897 LoggingAspect-1.0.jar
10/14/2011 11:02 AM 224,958 marshalling-api-1.3.0.GA.jar
10/14/2011 11:02 AM 1,768,445 msml-xbean-1.1.jar
11/16/2011 12:38 PM 792,314 netty-3.2.5.Final.jar
10/14/2011 11:02 AM 93,139 nist-sdp-1.0.jar
10/14/2011 03:33 PM 60,018 resteasy-jaxb-provider-2.1.0.GA.jar
10/14/2011 03:33 PM 545,700 resteasy-jaxrs-2.1.0.GA.jar
10/17/2011 04:28 PM 30,063 resteasy-jettison-provider-2.2.1.GA.jar
10/14/2011 11:02 AM 7,635 rhq-pluginAnnotations-3.0.1.jar
10/14/2011 11:02 AM 80,929 river-1.3.0.GA.jar
10/14/2011 03:33 PM 18,274 scannotation-1.0.2.jar
10/14/2011 11:03 AM 397,907 seam-solder-3.0.0.Final.jar
12/20/2011 06:30 PM 202,696 sipunit-1.0.0.jar
10/14/2011 03:33 PM 332,055 sjsxp-1.0.1.jar
10/27/2011 01:27 PM 2,949,500 spring-2.5.6.jar
10/27/2011 01:27 PM 20,422 spring-aspects-2.5.6.jar
10/27/2011 01:27 PM 488,282 spring-beans-2.5.6.jar
10/27/2011 01:27 PM 285,491 spring-core-2.5.6.jar
10/27/2011 01:33 PM 103,960 spring-mock-2.0.8.jar
10/14/2011 03:33 PM 26,310 stax-api-1.0.jar
12/19/2011 12:15 PM 182,112 stax2-api-3.1.1.jar
12/19/2011 12:15 PM 478,446 woodstox-core-asl-4.1.1.jar
10/14/2011 11:03 AM 3,149,655 xalan-2.6.0.jar
10/14/2011 11:02 AM 2,666,926 xbean-2.3.0.jar
11/01/2011 10:58 AM 1,412,376 xercesImpl-2.6.2-jaxb-1.0.6.jar
10/14/2011 11:02 AM 109,318 xml-apis-1.0.b2.jar
-
13. Re: Domain controller loses connection to slaves
ctomc May 30, 2012 2:43 PM (in response to mackerman)Hi,
i would start by trimming your deployment as you have lots of jars bundleted inside of your application that are already present in app server.
so real quick:
- aspectjrt-1.6.1.jar
- aspectjweaver-1.6.1.jar <-- you really still need this?
- commons-beanutils-1.7.0.jar <-- could be imported from system modules
- commons-beanutils-core-1.8.0.jar
- commons-codec-1.2.jar
- commons-collections-3.2.1.jar
- commons-configuration-1.6.jar
- commons-dbcp-1.2.1.jar
- commons-digester-1.8.jar
- commons-httpclient-3.1.jar <-- you need this? you have also httpcore that is newer version of this...
- commons-jexl-1.1.jar
- commons-lang-2.4.jar <-- could be imported from system module
- commons-logging-1.1.1.jar <-- should be removed
- commons-pool-1.5.6.jar
- commons-scxml-0.9.jar
- concurrent-1.3.3.jar <-- should be removed
- hornetq-core-2.2.14.Final.jar <-- should be removed
- hornetq-jms-2.2.14.Final.jar <-- should be removed
- hornetq-ra.rar <-- should be removed
- hornetq-rest-2.2.14.Final.jar <--should be removed
- hsqldb-1.8.0.10.jar <-- should be removed
- httpclient-4.0.3.jar <-- could be imported from system modules
- httpcore-4.0.1.jar <-- could be imported from system modules
- infinispan-client-hotrod-5.1.3.FINAL.jar <-- you really need this?
- infinispan-core-5.1.3.FINAL.jar <-- should be removed / replaced by module import
- jackson-mapper-asl-1.9.2.jar <-- should probably be removed
- jain-sip-api-1.2.jar
- jain-sip-ri-1.2.1990.jar
- javassist-3.6.0.GA.jar <-- should be removed unless expliticly needed
- jaxb-api-2.1.jar <-- part of jdk, should be removed
- jaxb-impl-2.1.12.jar <-- should be removed
- jaxrs-api-2.1.0.GA.jar <-- should be removed
- jboss-marshalling-1.3.6.GA.jar <-- should be removed, if really needed should be replaced by module import
- jboss-marshalling-river-1.3.6.GA.jar <-- should be removed, if really needed should be replaced by module import
- jcip-annotations-1.0.jar <-- you sure you need this?
- jettison-1.2.jar
- jgroups-3.0.8.Final.jar <-- should be removed, if really needed explicitly, should be replaced by module import
- jmf-2.1.1e.jar
- jmxservice-1.1.1.jar
- jsr173-1.0.jar <-- should be removed
- jsr250-api-1.0.jar <-- should be removed
- log4j-1.2.16.jar <-- should be removed
- LoggingAspect-1.0.jar <-- really need this?
- marshalling-api-1.3.0.GA.jar <-- should be removed, if really needed explicitly, should be replaced by module import
- msml-xbean-1.1.jar
- netty-3.2.5.Final.jar <-- could be removed, if really needed explicitly, should be replaced by module import
- nist-sdp-1.0.jar
- resteasy-jaxb-provider-2.1.0.GA.jar <-- should be removed, if really needed explicitly, should be replaced by module import
- resteasy-jaxrs-2.1.0.GA.jar <-- should be removed, if really needed explicitly, should be replaced by module import
- resteasy-jettison-provider-2.2.1.GA.jar <-- should be removed, if really needed explicitly, should be replaced by module import
- rhq-pluginAnnotations-3.0.1.jar <-- should be removed
- river-1.3.0.GA.jar <-- should be removed, if really needed explicitly, should be replaced by module import
- scannotation-1.0.2.jar
- seam-solder-3.0.0.Final.jar
- sipunit-1.0.0.jar <-- could be removed probably
- sjsxp-1.0.1.jar
- spring-2.5.6.jar
- spring-aspects-2.5.6.jar
- spring-beans-2.5.6.jar
- spring-core-2.5.6.jar
- spring-mock-2.0.8.jar
- stax-api-1.0.jar <-- should be removed, if really needed explicitly, should be replaced by module import
- stax2-api-3.1.1.jar <-- should be removed, if really needed explicitly, should be replaced by module import
- woodstox-core-asl-4.1.1.jar <-- should be removed, if really needed explicitly, should be replaced by module import
- xalan-2.6.0.jar <-- should be removed, if really needed explicitly, should be replaced by module import
- xbean-2.3.0.jar <-- should be removed, if really needed explicitly, should be replaced by module import
- xercesImpl-2.6.2-jaxb-1.0.6.jar <-- should be removed, if really needed explicitly, should be replaced by module import
- xml-apis-1.0.b2.jar <-- should be removed, if really needed explicitly, should be replaced by module import
if you use maven, set this dependancies to scope=provided
--
tomaz
-
14. Re: Domain controller loses connection to slaves
mackerman May 30, 2012 5:00 PM (in response to ctomc)thanks, i'll take a look at that. One complicating factor is that we can't use some of the modules configured in the app server, such as infinispan and jgroups, as the kind of configuration we need to do is not yet supported - we need to use a gossiprouter as multicast is not available in our environment. Similarly, we couldn't configure hornet Q via the app server as we couldn't get the combination of hornetQ, rest & jndi to work as needed, though this may be fixed now, i haven't checked. Some of the versions, such as infinispan-5.1.3 are also important (7.1.1 is at 5.1.2), as we need some improvements in the cluster management & synchronization.
would be nice if we could remove a bunch of these as it would speed up deployment considerably.
Mitchell