5 Replies Latest reply on May 17, 2012 4:39 PM by jbertram Branched from an earlier discussion.

    HornetQ reactivation/reconnect attempts

    thebravedave

      Hello, I am having this problem where remote ejbs/JMS aren't working correctly when the client server is

      started after the host server.  I have 2 servers on seperate boxes. One is the JMS host server, the

      other is the JMS client server.

       

      Here is how we have our setup working.  We have our server A acting as the hornetQ JMS host. There are topics that are available on it.  We have our server B acting as the client.

      The client server gets a remote EJB RemoteConnectionFactory from the host server and publishes a topic message to it.  We then have a message driven bean on the client (server B) server that is listening to that topic,

      and should get triggered when a message is published to the topic.

       

      Im configuring everything in standalone.xml, and not using jboss-ejb-clients.properties.

       

      We are using jboss 7.1.1.Final, which uses jboss-ejb-client-1.0.5.Final, so the fix should work in this version.

      I want to test that the fix is working correctly.

      To do so I will start up the client server and then afterwards start up the host server and see if the client server reconnects to the host server after the host server has completed its startup.

       

      I start up the client server, and get this error in the logs. This error is to be expected as the JMS host server is not up at this point.

       

      2012-05-17 10:22:18,320 INFO  [org.hornetq.ra.inflow.HornetQActivation] Attempting to reconnect org.hornetq.ra.inflow.HornetQActivationSpec(ra=org.hornetq.ra.HornetQResourceAdapter@39477a84 destination=areavaultResourceupdate destinationType=javax.jms.Topic ack=Auto-acknowledge durable=false clientID=null user=Theuser password=**** maxSession=15)

      2012-05-17 10:22:18,322 ERROR [org.hornetq.ra.inflow.HornetQActivation] Unable to reconnect org.hornetq.ra.inflow.HornetQActivationSpec(ra=org.hornetq.ra.HornetQResourceAdapter@39477a84 destination=areavaultResourceupdate destinationType=javax.jms.Topic ack=Auto-acknowledge durable=false clientID=null user=sococouser password=**** maxSession=15): HornetQException[errorCode=2 message=Cannot connect to server(s). Tried with all available servers.]

              at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:769)

              at org.hornetq.ra.inflow.HornetQActivation.setupSession(HornetQActivation.java:362)

              at org.hornetq.ra.inflow.HornetQActivation.setup(HornetQActivation.java:294)

              at org.hornetq.ra.inflow.HornetQActivation.handleFailure(HornetQActivation.java:566)

              at org.hornetq.ra.inflow.HornetQActivation$SetupActivation.run(HornetQActivation.java:609)

              at org.jboss.jca.core.workmanager.WorkWrapper.run(WorkWrapper.java:212)

              at org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:33)

              at org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:801)

              at org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:45)

              at org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:821)

              at java.lang.Thread.run(Thread.java:662)

              at org.jboss.threads.JBossThread.run(JBossThread.java:122)

       

       

      I then start up the server that is the JMS Host. I see some communication between the client and the

      host on port 5445. I think this is when the client starts communicating with the host. All of this traffic (right below) happens pretty much all at once and right after the

      JMS host finishes starting up.

      Heres the tcpdump packet info (restricted to port 5445 only):

       

       

      10:40:32.347045 IP asdeveug02.eug.mycompany.net.50224 > userdeveug01.eug.mycompany.net.5445: Flags [S], seq 3659829354, win 14600, options [mss 1460,sackOK,TS val 262568567 ecr 0,nop,wscale 7], length 0

      10:40:32.347128 IP userdeveug01.eug.mycompany.net.5445 > asdeveug02.eug.mycompany.net.50224: Flags [S.], seq 267315287, ack 3659829355, win 14480, options [mss 1460,sackOK,TS val 1902849926 ecr 262568567,nop,wscale 6], length 0

      10:40:32.347255 IP asdeveug02.eug.mycompany.net.50224 > userdeveug01.eug.mycompany.net.5445: Flags [.], ack 1, win 115, options [nop,nop,TS val 262568567 ecr 1902849926], length 0

      10:40:32.378593 IP asdeveug02.eug.mycompany.net.50224 > userdeveug01.eug.mycompany.net.5445: Flags [P.], seq 1:22, ack 1, win 115, options [nop,nop,TS val 262568599 ecr 1902849926], length 21

      10:40:32.378660 IP userdeveug01.eug.mycompany.net.5445 > asdeveug02.eug.mycompany.net.50224: Flags [.], ack 22, win 227, options [nop,nop,TS val 1902849958 ecr 262568599], length 0

      10:40:32.378812 IP userdeveug01.eug.mycompany.net.5445 > asdeveug02.eug.mycompany.net.50224: Flags [P.], seq 1:22, ack 22, win 227, options [nop,nop,TS val 1902849958 ecr 262568599], length 21

      10:40:32.378947 IP asdeveug02.eug.mycompany.net.50224 > userdeveug01.eug.mycompany.net.5445: Flags [.], ack 22, win 115, options [nop,nop,TS val 262568599 ecr 1902849958], length 0

      10:40:32.379892 IP asdeveug02.eug.mycompany.net.50224 > userdeveug01.eug.mycompany.net.5445: Flags [P.], seq 22:43, ack 22, win 115, options [nop,nop,TS val 262568600 ecr 1902849958], length 21

      10:40:32.380010 IP userdeveug01.eug.mycompany.net.5445 > asdeveug02.eug.mycompany.net.50224: Flags [P.], seq 22:43, ack 43, win 227, options [nop,nop,TS val 1902849959 ecr 262568600], length 21

      10:40:32.382852 IP asdeveug02.eug.mycompany.net.50224 > userdeveug01.eug.mycompany.net.5445: Flags [P.], seq 43:61, ack 43, win 115, options [nop,nop,TS val 262568603 ecr 1902849959], length 18

      10:40:32.384551 IP asdeveug02.eug.mycompany.net.50224 > userdeveug01.eug.mycompany.net.5445: Flags [P.], seq 61:143, ack 43, win 115, options [nop,nop,TS val 262568605 ecr 1902849959], length 82

      10:40:32.384625 IP userdeveug01.eug.mycompany.net.5445 > asdeveug02.eug.mycompany.net.50224: Flags [.], ack 143, win 227, options [nop,nop,TS val 1902849964 ecr 262568603], length 0

      10:40:32.390737 IP userdeveug01.eug.mycompany.net.5445 > asdeveug02.eug.mycompany.net.50224: Flags [P.], seq 43:60, ack 143, win 227, options [nop,nop,TS val 1902849970 ecr 262568603], length 17

      10:40:32.398461 IP asdeveug02.eug.mycompany.net.50224 > userdeveug01.eug.mycompany.net.5445: Flags [P.], seq 143:156, ack 60, win 115, options [nop,nop,TS val 262568618 ecr 1902849970], length 13

      10:40:32.398858 IP userdeveug01.eug.mycompany.net.5445 > asdeveug02.eug.mycompany.net.50224: Flags [P.], seq 60:77, ack 156, win 227, options [nop,nop,TS val 1902849978 ecr 262568618], length 17

      10:40:32.438497 IP asdeveug02.eug.mycompany.net.50224 > userdeveug01.eug.mycompany.net.5445: Flags [.], ack 77, win 115, options [nop,nop,TS val 262568659 ecr 1902849978], length 0

       

       

      The only problem is that after this when I perform a task that should trigger a JMS message on

      the client, the message driven bean is not triggered. In fact I am using tcpdump to listen to all traffic to the JMS client

      and when I perform a task that should trigger the message driven bean (which is configured to listen to a JMS topic from the JMS host server)

      I don't get an traffic from the client server to the host server (or host to client).

       

      If I  start the host up first, then afterwards start up the client server, everytime I perform

      a task that should trigger the message driven bean, I see the traffic occur on port 5445 (from client->host and host->client) and the

      message driven bean is triggered correctly.

       

      I do see some intermitten packets being sent on port 5445 as such

      ...

      10:54:32.389758 IP asdeveug02.eug.mycompany.net.50224 > userdeveug01.eug.mycompany.net.5445: Flags [.], ack 767, win 115, options [nop,nop,TS val 263408610 ecr 1903689969], length 0

       

      10:55:02.389588 IP asdeveug02.eug.mycompany.net.50224 > userdeveug01.eug.mycompany.net.5445: Flags [P.], seq 822:843, ack 767, win 115, options [nop,nop,TS val 263438610 ecr 1903689969], length 21

       

      10:55:02.389782 IP userdeveug01.eug.mycompany.net.5445 > asdeveug02.eug.mycompany.net.50224: Flags [P.], seq 767:788, ack 843, win 227, options [nop,nop,TS val 1903719969 ecr 263438610], length 21

       

      10:55:02.389900 IP asdeveug02.eug.mycompany.net.50224 > userdeveug01.eug.mycompany.net.5445: Flags [.], ack 788, win 115, options [nop,nop,TS val 263438610 ecr 1903719969], length 0

      ....

       

      But none of these packets are being sent when I trigger the event that should publish a message to the topic and hence the message driven bean.  When I start up the JMS host first

      and the client server afterwards, and trigger the even that should trigger the topic, it is very obvious that the packets being sent on port 5445 are coming from the triggering event.

      I find this confusing, because I think that our message utility class is getting the RemoteConnectionFactory class correctly, due to the large chunk of packets sent on port 5445 when the host server starts up as well

      as the fact that when I kill java running on the host server I get a

       

      HornetQException[errorCode=2 message=Channel disconnected]


      error, which makes me think that the client server was actually connected to the host server.

       

       

      Please help me. As it is we have to always do startup in a certain order, and since we have automatic weekly restarts of the client server, if the host server were down at that time for some reason (or being restarted) then the

      JMS would just not work, which would cause bugs in our software.

       

      Let me know if you need any more information.

       

      Thank you!

        • 1. Re: Standalone to remote EJB reconnect issues..
          jbertram

          What activation configuration properties are you using on your MDB?

          • 2. Re: Standalone to remote EJB reconnect issues..
            thebravedave

            Thanks for answering so quickly.

             

            @TransactionAttribute(TransactionAttributeType.REQUIRED)

            @MessageDriven(name = "ResourceUpdateMessageListenerBean", activationConfig = {

                    @ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Topic"),

                    @ActivationConfigProperty(propertyName = "destination", propertyValue = "areavaultResourceupdate"),

                    @ActivationConfigProperty(propertyName = "acknowledgeMode", propertyValue = "Auto-acknowledge"),

                    @ActivationConfigProperty(propertyName="user", propertyValue="Theuser"),

                    @ActivationConfigProperty(propertyName="password", propertyValue="User45")

             

            })

            public class ResourceUpdateMessageListener implements MessageListener  {

            ....

            }

             

            But I don't see any traffic going from the client server to the host server when I use my messageutil utility class to try and publish a message to the host server, so I'm thinking that the issue lies upstream from the mdb. ( I could be wrong though).

             

            Thanks again,

             

            David

            • 3. Re: Standalone to remote EJB reconnect issues..
              jbertram

              Try using these:

               

                @ActivationConfigProperty(propertyName = "reconnectAttempts", propertyValue = "-1"),   

                    @ActivationConfigProperty(propertyName = "setupAttempts", propertyValue = "-1")

               

              Let me know if that helps.

              • 4. Re: Standalone to remote EJB reconnect issues..
                thebravedave

                This totally fixed my issue, thanks so much Justin Bertram!

                Once I added these 2 new ActivationConfigPropertys, my client server was constantly sending HornetQ reconnect attempts as such

                 

                2012-05-17 13:15:23,015 INFO  [org.hornetq.ra.inflow.HornetQActivation] Attempting to reconnect org.hornetq.ra.inflow.HornetQActivationSpec(ra=org.hornetq.ra.HornetQResourceAdapter@68c9e408

                destination=areavaultResourceupdate destinationType=javax.jms.Topic ack=Auto-acknowledge durable=false clientID=null user=Theuser password=**** maxSession=15)

                2012-05-17 13:15:23,017 ERROR [org.hornetq.ra.inflow.HornetQActivation] Unable to reconnect org.hornetq.ra.inflow.HornetQActivationSpec(ra=org.hornetq.ra.HornetQResourceAdapter@68c9e408 des

                tination=areavaultResourceupdate destinationType=javax.jms.Topic ack=Auto-acknowledge durable=false clientID=null user=Theuser password=**** maxSession=15): HornetQException[errorCode=2

                message=Cannot connect to server(s). Tried with all available servers.]

                        at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:769)

                        at org.hornetq.ra.inflow.HornetQActivation.setupSession(HornetQActivation.java:362)

                        at org.hornetq.ra.inflow.HornetQActivation.setup(HornetQActivation.java:294)

                        at org.hornetq.ra.inflow.HornetQActivation.handleFailure(HornetQActivation.java:566)

                        at org.hornetq.ra.inflow.HornetQActivation$SetupActivation.run(HornetQActivation.java:609)

                        at org.jboss.jca.core.workmanager.WorkWrapper.run(WorkWrapper.java:212)

                        at org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:33)

                        at org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:801)

                        at org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:45)

                        at org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:821)

                        at java.lang.Thread.run(Thread.java:662)

                        at org.jboss.threads.JBossThread.run(JBossThread.java:122)

                 

                Before it would only send these for a while and then quit sending them.

                I had read about these properties, but didn't think they were needed since the fix for jboss-ejb-client was supposed to entail the client reconnecting with the host when the host was not up first. I didn't know

                that the implementation (in this case an mdb over jms) had to be configured in order to make this work as well.

                Could you explain the reasoning behind this, as I find it a bit strange.

                 

                Thanks again, you rock!

                 

                David

                • 5. Re: Standalone to remote EJB reconnect issues..
                  jbertram

                  I believe the main problem with your understanding is conflating EJB connections with JMS connections.  The two are altogether different.

                   

                  When the server starts it attempts to "activate" all appropriate MDBs.  For MDBs using the HornetQ JCA RA (which they do by default) all the activation configuration properties are inspected and obeyed accordingly.  If activation fails (for whatever reason) then the activation is retried again based on the "setupAttempts" activation configuration property (10 by default).  If the connection fails after activation has already succeeded then the connection will be retried based on the "reconnectionAttempts" activation configuration property (-1 by default).