4 Replies Latest reply on Nov 14, 2011 1:00 AM by dasmurali

    Cluster UDP exception in JBoss EAP 5.1.1

    dasmurali

      Hi All Gurus,

       

      We are migrating from JBoss AS 5.1.0(Open Soruce) to JBoss EAP 5.1.1. Our application is in cluster and running good in production with Jboss AS 5.1.0(OS version).

       

      When we migrated to JBoss EAP 5.1.1, we are gettting exceptions in JBoss Messgaging cluster. Please find the below exception track trace for the same.

       

       

      2011-11-04 08:36:29,859 ERROR [org.jgroups.protocols.UDP] (Timer-7,10.19.47.140:55200) failed sending message to null (160 bytes)

      java.lang.Exception: dest=/228.11.21.21:45788 (163 bytes)

      at org.jgroups.protocols.UDP._send(UDP.java:374)

      at org.jgroups.protocols.UDP.sendToAllMembers(UDP.java:302)

      at org.jgroups.protocols.TP.doSend(TP.java:1478)

      at org.jgroups.protocols.TP.send(TP.java:1468)

      at org.jgroups.protocols.TP.down(TP.java:1186)

      at org.jgroups.protocols.TP$ProtocolAdapter.down(TP.java:2308)

      at org.jgroups.protocols.Discovery.down(Discovery.java:374)

      at org.jgroups.protocols.MERGE2.down(MERGE2.java:175)

      at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:381)

      at org.jgroups.protocols.FD.down(FD.java:315)

      at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:95)

      at org.jgroups.protocols.BARRIER.down(BARRIER.java:107)

      at org.jgroups.protocols.pbcast.NAKACK.send(NAKACK.java:803)

      at org.jgroups.protocols.pbcast.NAKACK.down(NAKACK.java:604)

      at org.jgroups.protocols.UNICAST.down(UNICAST.java:455)

      at org.jgroups.protocols.pbcast.STABLE$StabilitySendTask.run(STABLE.java:855)

      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)

      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

      at java.util.concurrent.FutureTask.run(FutureTask.java:138)

      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)

      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)

      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

      at java.lang.Thread.run(Thread.java:662)

      Caused by: java.io.IOException: Invalid argument

      at java.net.PlainDatagramSocketImpl.send(Native Method)

      at java.net.DatagramSocket.send(DatagramSocket.java:625)

      at org.jgroups.protocols.UDP._send(UDP.java:358)

      ... 23 more

       

       

      Our application is running in Suse linux 11.

       

      Thanks and Regards

      Murali Reddy

        • 1. Re: Cluster UDP exception in JBoss EAP 5.1.1
          dasmurali

          Further worked on this to resolve this cluster issue.

           

          Followed jgroups manual link --> http://www.jgroups.org/manual/html/ch02.html

           

          Interstingly when I tested whether the IP Multicast  is setup properly or not (point 2.8 in the article), it works fine. What ever I type in 'Sender' is coming to 'Receiver'(both are running in two different instances/jvm's).

           

          But when I tested the 'Draw' demo java program (point 2.6 in the article), it gace me the same exception. Please find the below stack trace for the same.

           

          > java -cp /home/jboss/jboss-5.1.1/lib/concurrent.jar:/home/jboss/jboss-5.1.1/server/DasMonitor/lib/jgroups.jar:/home/jboss/jboss-5.1.1/client/commons-logging.jar org.jgroups.demos.Draw
          Nov 7, 2011 3:32:25 AM org.jgroups.JChannel init
          INFO: JGroups version: 2.6.19.Final

          ---------------------------------------------------------
          GMS: address is 127.0.0.2:44446 (cluster=DrawGroupDemo)
          ---------------------------------------------------------
          Nov 7, 2011 3:32:25 AM org.jgroups.protocols.TP down
          SEVERE: failed sending message to null (48 bytes)
          java.lang.Exception: dest=/228.10.10.10:45588 (51 bytes)
                  at org.jgroups.protocols.UDP._send(UDP.java:374)
                  at org.jgroups.protocols.UDP.sendToAllMembers(UDP.java:302)
                  at org.jgroups.protocols.TP.doSend(TP.java:1478)
                  at org.jgroups.protocols.TP.send(TP.java:1468)
                  at org.jgroups.protocols.TP.down(TP.java:1186)
                  at org.jgroups.protocols.PING.sendMcastDiscoveryRequest(PING.java:278)
                  at org.jgroups.protocols.PING.sendGetMembersRequest(PING.java:259)
                  at org.jgroups.protocols.Discovery$PingSenderTask$1.run(Discovery.java:407)
                  at org.jgroups.util.TimeScheduler$RobustRunnable.run(TimeScheduler.java:196)
                  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
                  at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
                  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
                  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
                  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
                  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                  at java.lang.Thread.run(Thread.java:662)
          Caused by: java.io.IOException: Invalid argument
                  at java.net.PlainDatagramSocketImpl.send(Native Method)
                  at java.net.DatagramSocket.send(DatagramSocket.java:625)
                  at org.jgroups.protocols.UDP._send(UDP.java:

           

           

          Thanks and Regards

          Murali Reddy

          • 2. Re: Cluster UDP exception in JBoss EAP 5.1.1
            dasmurali

            One more test result.


            Followed Testing JBoss link -->: http://community.jboss.org/docs/DOC-12375


            Tested with 'jbm-control' jgroups stack. Got the similar exception.

             

            ~> java -cp /home/jboss/jboss-5.1.1/lib/concurrent.jar:/home/jboss/jboss-5.1.1/server/DasMonitor/lib/jgroups.jar:/home/jboss/jboss-5.1.1/client/commons-logging.jar  org.jgroups.demos.ViewDemo -props /home/jboss/jbm-control.xml


            Nov 7, 2011 3:42:01 AM org.jgroups.JChannel init
            INFO: JGroups version: 2.6.19.Final

            ---------------------------------------------------------
            GMS: address is 127.0.0.2:55200 (cluster=ViewDemo)
            ---------------------------------------------------------
            Nov 7, 2011 3:42:01 AM org.jgroups.protocols.TP down
            SEVERE: failed sending message to null (43 bytes)
            java.lang.Exception: dest=/228.11.11.11:45688 (46 bytes)
                    at org.jgroups.protocols.UDP._send(UDP.java:374)
                    at org.jgroups.protocols.UDP.sendToAllMembers(UDP.java:302)
                    at org.jgroups.protocols.TP.doSend(TP.java:1478)
                    at org.jgroups.protocols.TP.send(TP.java:1468)
                    at org.jgroups.protocols.TP.down(TP.java:1186)
                    at org.jgroups.protocols.TP$ProtocolAdapter.down(TP.java:2308)
                    at org.jgroups.protocols.PING.sendMcastDiscoveryRequest(PING.java:278)
                    at org.jgroups.protocols.PING.sendGetMembersRequest(PING.java:259)
                    at org.jgroups.protocols.Discovery$PingSenderTask$1.run(Discovery.java:407)
                    at org.jgroups.util.TimeScheduler$RobustRunnable.run(TimeScheduler.java:196)
                    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
                    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
                    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
                    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
                    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
                    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                    at java.lang.Thread.run(Thread.java:662)
            Caused by: java.io.IOException: Invalid argument
                    at java.net.PlainDatagramSocketImpl.send(Native Method)
                    at java.net.DatagramSocket.send(DatagramSocket.java:625)
                    at org.jgroups.protocols.UDP._send(UDP.java:358)
                    ... 18 more
                   

            Thanks and Regards
            Murali Reddy

             

            Added 'jbm-control' jgroups stack configuration file.

            • 3. Re: Cluster UDP exception in JBoss EAP 5.1.1
              dasmurali

              Tested same test case(with jbm-control.xml) in SUSE 9 with out any issues. So guessing some fight between JBoss EAP 5.1.1 'jbm-control' jgroups stack with SUSE 11.

               

              Thanks and Regards

              Murali Reddy

              • 4. Re: Cluster UDP exception in JBoss EAP 5.1.1
                dasmurali

                Finally able to resolve(I am guessing) this issue.

                 

                Added "-Djava.net.preferIPv4Stack=true" for above tests. Also added same in JBoss start up script. This resolved the issue. Now cluster is working good.

                 

                Not sure what"-Djava.net.preferIPv4Stack=true" will do.

                 

                Thanks and Regards

                Murali Reddy