Hi,

We have jboss running in a cluster mode. Both the nodes are in different physical machine. We have configured the cluster using unicast method.

Problem is that the server goes down automatically, sometime one of the node and sometime both the nodes. We are running JBOSS AS 5.1.0.GA server on Redhat 6.3.

We are not able to troubleshoot the issue. Strangely whenever we restart the server, the problem occurs. Below is the only info that we see in the server.log file.

2014-07-07 07:32:26,807 INFO  [STDOUT] (Thread-68) 2014-07-07 07:32:26,807 [Thread-68] INFO  com.workpoint.monitor.GeneralMonitor  - Shutting down...

2014-07-07 07:32:26,807 INFO  [com.workpoint.monitor.GeneralMonitor] (Thread-68) Shutting down...

2014-07-07 07:32:26,808 INFO  [STDOUT] (Thread-68) 2014-07-07 07:32:26,808 [Thread-68] INFO  com.workpoint.monitor.AbstractMonitor  - AlertMonitor: shutdown requested.

2014-07-07 07:32:26,808 INFO  [com.workpoint.monitor.AbstractMonitor] (Thread-68) AlertMonitor: shutdown requested.

2014-07-07 07:32:26,808 INFO  [STDOUT] (Thread-68) 2014-07-07 07:32:26,808 [Thread-68] WARN  com.workpoint.pattern.RemoteClientTracker  - Client rmi://sso-idm1.state.ar.us:1577/Alert_WPDS_68862df4:1471096ca4f:-7ff1 is requesting its task list be restored to the master queue. This probably means that client is shutting down.

2014-07-07 07:32:26,808 WARN  [com.workpoint.pattern.RemoteClientTracker] (Thread-68) Client rmi://sso-idm1.state.ar.us:1577/Alert_WPDS_68862df4:1471096ca4f:-7ff1 is requesting its task list be restored to the master queue. This probably means that client is shutting down.

2014-07-07 07:32:27,145 WARN  [ims.tmt.events] (JBoss Shutdown Hook) Attempting to shutdown long running events if any.

2014-07-07 07:32:27,147 WARN  [ims.tmt.EnvironmentService] (JBoss Shutdown Hook) * Stopping environment: Arkansas SSO

2014-07-07 07:32:27,153 WARN  [ims.tmt.EnvironmentService] (JBoss Shutdown Hook) * Stopped environment: Arkansas SSO

2014-07-07 07:32:27,525 INFO  [STDOUT] (Thread-68) 2014-07-07 07:32:27,525 [Thread-68] INFO  com.workpoint.monitor.AbstractMonitor  - JobMonitor: shutdown requested.

2014-07-07 07:32:27,525 INFO  [com.workpoint.monitor.AbstractMonitor] (Thread-68) JobMonitor: shutdown requested.

2014-07-07 07:32:27,525 INFO  [STDOUT] (Thread-68) 2014-07-07 07:32:27,525 [Thread-68] WARN  com.workpoint.pattern.RemoteClientTracker  - Client rmi://sso-idm1.state.ar.us:1577/JobEval_WPDS_68862df4:1471096ca4f:-7fee is requesting its task list be restored to the master queue. This probably means that client is shutting down.

2014-07-07 07:32:27,525 WARN  [com.workpoint.pattern.RemoteClientTracker] (Thread-68) Client rmi://sso-idm1.state.ar.us:1577/JobEval_WPDS_68862df4:1471096ca4f:-7fee is requesting its task list be restored to the master queue. This probably means that client is shutting down.

2014-07-07 07:32:28,041 INFO  [STDOUT] (Thread-68) 2014-07-07 07:32:28,041 [Thread-68] INFO  com.workpoint.monitor.AbstractMonitor  - MailMonitor: shutdown requested.

2014-07-07 07:32:28,041 INFO  [com.workpoint.monitor.AbstractMonitor] (Thread-68) MailMonitor: shutdown requested.

2014-07-07 07:32:28,042 INFO  [STDOUT] (Thread-68) 2014-07-07 07:32:28,042 [Thread-68] WARN  com.workpoint.pattern.RemoteClientTracker  - Client rmi://sso-idm1.state.ar.us:1577/Mail_WPDS_68862df4:1471096ca4f:-7feb is requesting its task list be restored to the master queue. This probably means that client is shutting down.

2014-07-07 07:32:28,042 WARN  [com.workpoint.pattern.RemoteClientTracker] (Thread-68) Client rmi://sso-idm1.state.ar.us:1577/Mail_WPDS_68862df4:1471096ca4f:-7feb is requesting its task list be restored to the master queue. This probably means that client is shutting down.

2014-07-07 07:32:28,579 INFO  [STDOUT] (Thread-68) 2014-07-07 07:32:28,579 [Thread-68] INFO  com.workpoint.monitor.AbstractMonitor  - ScriptMonitor: shutdown requested.

2014-07-07 07:32:28,579 INFO  [com.workpoint.monitor.AbstractMonitor] (Thread-68) ScriptMonitor: shutdown requested.

2014-07-07 07:32:28,579 INFO  [STDOUT] (Thread-68) 2014-07-07 07:32:28,579 [Thread-68] WARN  com.workpoint.pattern.RemoteClientTracker  - Client rmi://sso-idm1.state.ar.us:1577/Script_WPDS_68862df4:1471096ca4f:-7fe8 is requesting its task list be restored to the master queue. This probably means that client is shutting down.

2014-07-07 07:32:28,579 WARN  [com.workpoint.pattern.RemoteClientTracker] (Thread-68) Client rmi://sso-idm1.state.ar.us:1577/Script_WPDS_68862df4:1471096ca4f:-7fe8 is requesting its task list be restored to the master queue. This probably means that client is shutting down.

2014-07-07 07:32:30,127 INFO  [STDOUT] (ScriptMonitor) 2014-07-07 07:32:30,126 [ScriptMonitor] ERROR com.workpoint.monitor.support.ScriptMonitorThread  - Exception caught while removing monitor rmi://sso-idm1.state.ar.us:1577/Script_WPDS_68862df4:1471096ca4f:-7fe8

java.rmi.RemoteException: NamingException caught attempting to locate the 'iam/im/ejb/MonitorPvt_EJB' object. Please verify that the server is running and the client configuration is correct.; nested exception is:

  javax.naming.CommunicationException [Root exception is java.rmi.UnmarshalException: Error unmarshaling return; nested exception is:

  java.lang.IllegalStateException: BaseClassLoader@415a6fc3{vfsfile:/opt/jboss/server/all/farm/iam_im.ear/user_console.war/} classLoader is not connected to a domain (probably undeployed?) for class javax.naming.NameNotFoundException]; nested exception is:

  java.rmi.RemoteException: NamingException caught attempting to locate the 'iam/im/ejb/MonitorPvt_EJB' object. Please verify that the server is running and the client configuration is correct.; nested exception is:

  javax.naming.CommunicationException [Root exception is java.rmi.UnmarshalException: Error unmarshaling return; nested exception is:

  java.lang.IllegalStateException: BaseClassLoader@415a6fc3{vfsfile:/opt/jboss/server/all/farm/iam_im.ear/user_console.war/} classLoader is not connected to a domain (probably undeployed?) for class javax.naming.NameNotFoundException]

  at com.workpoint.client.connect.ClientContextEJB.getMonitorPvt(Unknown Source)

  at com.workpoint.client.Monitor.deassignScriptMonitors(Unknown Source)

  at com.workpoint.monitor.support.ScriptMonitorThread.monitorDown(Unknown Source)

  at com.workpoint.monitor.support.AbstractMonitorThread.checkForWork(Unknown Source)

  at com.workpoint.monitor.support.AbstractMonitorThread.startRemoteTaskClient(Unknown Source)

  at com.workpoint.monitor.support.ScriptMonitorThread.becomeMaster(Unknown Source)

  at com.workpoint.monitor.support.AbstractMonitorThread.A(Unknown Source)

  at com.workpoint.monitor.support.AbstractMonitorThread.start(Unknown Source)

  at com.workpoint.monitor.support.ScriptMonitorThread.run(Unknown Source)

Caused by: java.rmi.RemoteException: NamingException caught attempting to locate the 'iam/im/ejb/MonitorPvt_EJB' object. Please verify that the server is running and the client configuration is correct.; nested exception is:

  javax.naming.CommunicationException [Root exception is java.rmi.UnmarshalException: Error unmarshaling return; nested exception is:

  java.lang.IllegalStateException: BaseClassLoader@415a6fc3{vfsfile:/opt/jboss/server/all/farm/iam_im.ear/user_console.war/} classLoader is not connected to a domain (probably undeployed?) for class javax.naming.NameNotFoundException]

  at com.workpoint.client.connect.ClientContextEJB.B(Unknown Source)

  ... 9 more

Caused by: javax.naming.CommunicationException [Root exception is java.rmi.UnmarshalException: Error unmarshaling return; nested exception is:

  java.lang.IllegalStateException: BaseClassLoader@415a6fc3{vfsfile:/opt/jboss/server/all/farm/iam_im.ear/user_console.war/} classLoader is not connected to a domain (probably undeployed?) for class javax.naming.NameNotFoundException]

  at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:839)

  at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:686)

  at javax.naming.InitialContext.lookup(InitialContext.java:392)

  at com.workpoint.client.connect.ClientContextEJB.A(Unknown Source)

  ... 10 more

Caused by: java.rmi.UnmarshalException: Error unmarshaling return; nested exception is:

  java.lang.IllegalStateException: BaseClassLoader@415a6fc3{vfsfile:/opt/jboss/server/all/farm/iam_im.ear/user_console.war/} classLoader is not connected to a domain (probably undeployed?) for class javax.naming.NameNotFoundException

  at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:227)

  at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:142)

  at org.jnp.server.NamingServer_Stub.lookup(Unknown Source)

  at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:726)

  ... 13 more

Caused by: java.lang.IllegalStateException: BaseClassLoader@415a6fc3{vfsfile:/opt/jboss/server/all/farm/iam_im.ear/user_console.war/} classLoader is not connected to a domain (probably undeployed?) for class javax.naming.NameNotFoundException

  at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:793)

  at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:441)

  at java.lang.ClassLoader.loadClass(ClassLoader.java:295)

  at java.lang.ClassLoader.loadClass(ClassLoader.java:247)

  at java.lang.Class.forName0(Native Method)

  at java.lang.Class.forName(Class.java:249)

  at sun.rmi.server.LoaderHandler.loadClass(LoaderHandler.java:152)

  at java.rmi.server.RMIClassLoader$2.loadClass(RMIClassLoader.java:620)

  at org.jboss.system.JBossRMIClassLoader.loadClass(JBossRMIClassLoader.java:91)

  at java.rmi.server.RMIClassLoader.loadClass(RMIClassLoader.java:247)

  at sun.rmi.server.MarshalInputStream.resolveClass(MarshalInputStream.java:201)

  at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1589)

  at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1494)

  at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1748)

  at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1327)

  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:349)

  at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:225)

  ... 16 more

2014-07-07 07:32:30,126 ERROR [com.workpoint.monitor.support.ScriptMonitorThread] (ScriptMonitor) Exception caught while removing monitor rmi://sso-idm1.state.ar.us:1577/Script_WPDS_68862df4:1471096ca4f:-7fe8

java.rmi.RemoteException: NamingException caught attempting to locate the 'iam/im/ejb/MonitorPvt_EJB' object. Please verify that the server is running and the client configuration is correct.; nested exception is:

Thanks,

Priyotosh