1 Reply Latest reply on Feb 9, 2015 2:46 AM by saladimurthys

    Need help in resolving clustering issue in jboss 5.1.0 GA

    saladimurthys

      Hello,

       

      We are using jboss.5.1.0.GA in two servers which are in clustering. Recently we got an issue like if 2nd node is up then the application is not working. When we stop the 2nd node application is working fine the we came to know that there is an issue in node2. The below is the error we found in the log in node2.

      ---------------------------------------------------------

      GMS: address is 10.31.12.103:7900 (cluster=HMM_PR_WEB_GROUP-DATA)

      ----------------------------------------------------------

      2015-02-06 07:06:59,751 ERROR [ExceptionUtil]|||org.jboss.messaging.core.jmx.MessagingPostOfficeService@5eb9ec47 startService

      java.lang.IllegalArgumentException: 10.31.12.3:7901 is not a member of the group [10.31.12.3:7900, 10.31.12.103:7900] (enabled_members=[])

              at org.jgroups.protocols.UNICAST.down(UNICAST.java:332)

              at org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:316)

              at org.jgroups.protocols.VIEW_SYNC.down(VIEW_SYNC.java:204)

              at org.jgroups.protocols.pbcast.GMS.down(GMS.java:905)

              at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:475)

              at org.jgroups.JChannel.down(JChannel.java:1517)

              at org.jgroups.JChannel.send(JChannel.java:626)

              at org.jboss.messaging.core.impl.postoffice.GroupMember.unicastData(GroupMember.java:320)

              at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.requestDeliveries(MessagingPostOffice.java:1716)

              at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.loadBindings(MessagingPostOffice.java:2583)

              at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.start(MessagingPostOffice.java:394)

              at org.jboss.messaging.core.jmx.MessagingPostOfficeService.startService(MessagingPostOfficeService.java:462)

              at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:376)

              at org.jboss.system.ServiceMBeanSupport.start(ServiceMBeanSupport.java:269)

              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

              at java.lang.reflect.Method.invoke(Method.java:597)

              at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)

              at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)

              at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:138)

              at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)

              at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:140)

              at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)

              at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)

              at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)

              at org.jboss.system.microcontainer.ServiceProxy.invoke(ServiceProxy.java:206)

              at com.sun.proxy.$Proxy38.start(Unknown Source)

              at org.jboss.system.microcontainer.StartStopLifecycleAction.installAction(StartStopLifecycleAction.java:42)

              at org.jboss.system.microcontainer.StartStopLifecycleAction.installAction(StartStopLifecycleAction.java:37)

              at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:62)

              at org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:71)

              at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51)

              at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)

              at org.jboss.system.microcontainer.ServiceControllerContext.install(ServiceControllerContext.java:286)

              at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1631)

              at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:934)

              at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)

              at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:984)

              at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:822)

              at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:553)

              at org.jboss.system.ServiceController.doChange(ServiceController.java:688)

              at org.jboss.system.ServiceController.start(ServiceController.java:460)

              at org.jboss.system.deployers.ServiceDeployer.start(ServiceDeployer.java:163)

              at org.jboss.system.deployers.ServiceDeployer.deploy(ServiceDeployer.java:99)

              at org.jboss.system.deployers.ServiceDeployer.deploy(ServiceDeployer.java:46)

       

      2015-02-06 07:06:59,753 ERROR [AbstractKernelController]|||Error installing to Start: name=jboss.messaging:service=PostOffice state=Create mode=Manual requiredState=Installed

      java.lang.IllegalArgumentException: 10.31.12.3:7901 is not a member of the group [10.31.12.3:7900, 10.31.12.103:7900] (enabled_members=[])

              at org.jgroups.protocols.UNICAST.down(UNICAST.java:332)

              at org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:316)

              at org.jgroups.protocols.VIEW_SYNC.down(VIEW_SYNC.java:204)

              at org.jgroups.protocols.pbcast.GMS.down(GMS.java:905)

              at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:475)

              at org.jgroups.JChannel.down(JChannel.java:1517)

              at org.jgroups.JChannel.send(JChannel.java:626)

              at org.jboss.messaging.core.impl.postoffice.GroupMember.unicastData(GroupMember.java:320)

              at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.requestDeliveries(MessagingPostOffice.java:1716)

              at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.loadBindings(MessagingPostOffice.java:2583)

              at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.start(MessagingPostOffice.java:394)

              at org.jboss.messaging.core.jmx.MessagingPostOfficeService.startService(MessagingPostOfficeService.java:462)

              at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:376)

              at org.jboss.system.ServiceMBeanSupport.start(ServiceMBeanSupport.java:269)

              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

              at java.lang.reflect.Method.invoke(Method.java:597)

              at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)

              at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)

              at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:138)

              at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)

              at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:140)

              at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)

              at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)

              at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)

              at org.jboss.system.microcontainer.ServiceProxy.invoke(ServiceProxy.java:206)

              at com.sun.proxy.$Proxy38.start(Unknown Source)

              at org.jboss.system.microcontainer.StartStopLifecycleAction.installAction(StartStopLifecycleAction.java:42)

              at org.jboss.system.microcontainer.StartStopLifecycleAction.installAction(StartStopLifecycleAction.java:37)

              at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:62)

              at org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:71)

              at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51)

              at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)

              at org.jboss.system.microcontainer.ServiceControllerContext.install(ServiceControllerContext.java:286)

              at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1631)

              at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:934)

              at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)

              at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:984)

              at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:822)

              at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:553)

              at org.jboss.system.ServiceController.doChange(ServiceController.java:688)

              at org.jboss.system.ServiceController.start(ServiceController.java:460)

              at org.jboss.system.deployers.ServiceDeployer.start(ServiceDeployer.java:163)

              at org.jboss.system.deployers.ServiceDeployer.deploy(ServiceDeployer.java:99)

              at org.jboss.system.deployers.ServiceDeployer.deploy(ServiceDeployer.java:46)

              at org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.internalDeploy(AbstractSimpleRealDeployer.java:62)

       

       

      DEPLOYMENTS IN ERROR:

        Deployment "jboss.messaging:service=PostOffice" is in error due to the following reason(s): **ERROR**, java.lang.IllegalArgumentException: 10.31.12.3:7901 is not a member of the group [10.31.12.3:7900, 10.31.12.103:7900] (enabled_members=[])

        Deployment "vfszip:/app/jboss/jboss-5.1.0.GA/server/airman/deploy/airman-web.ear/" is in error due to the following reason(s): org.jboss.deployers.spi.DeploymentException: URL file:/data/jboss/jboss-5.1.0.GA/airman/tmp/avc2v-c5rvp4-i5t8bwx3-1-i5t8ek26-ar/cache-web-3.0.PR3.war/ deployment failed

       

      **************************************

      And in node 1 if found only one error. that is

       

      2015-02-06 07:08:44,751 ERROR [ConnectionTable]|||failed sending data to 10.31.12.103:7900: java.net.SocketException: Socket closed

      2015-02-06 07:08:44,752 ERROR [ConnectionTable]|||failed sending data to 10.31.12.103:7900: java.net.SocketException: Socket closed

       

      At present we stopped node2 and the application is running on only node1. Please help in resolving this issue as it is major now.

       

      Thanks in advance