1 2 Previous Next 22 Replies Latest reply on Jun 22, 2011 7:55 PM by david_b

    JMS bridge doesn't complain, but no msg in destination

    atijms

      I've set up the HornetQ JMS bridge between two EAR applications running on JBoss AS 6 instances on different servers (using the default supplied HornetQ).

       

      The bridge reports that everything is okay. There is the message about succeeding to connect and when I send a message to the local source queue, it is succesfully removed (I used twiddle.sh to verify this).

       

      There are no exceptions or errors reported by the bridge, yet there does not seem to be any communication going on to the remote destination.

       

      I patched the bridge to log its messages at INFO, and everything seems to be okay:

       

      2011-03-15 22:38:44,246 INFO  (pool-11-thread-2) org.hornetq.jms.bridge.impl.patch.trace.JMSBridgeImpl$BatchTimeChecker@13123be5 woke up
      2011-03-15 22:38:44,246 INFO  (pool-11-thread-2) org.hornetq.jms.bridge.impl.patch.trace.JMSBridgeImpl$BatchTimeChecker@13123be5 waiting for 2008
      2011-03-15 22:38:46,782 INFO  (pool-11-thread-2) org.hornetq.jms.bridge.impl.patch.trace.JMSBridgeImpl$BatchTimeChecker@13123be5 woke up
      2011-03-15 22:38:46,782 INFO  (pool-11-thread-2) org.hornetq.jms.bridge.impl.patch.trace.JMSBridgeImpl$BatchTimeChecker@13123be5 waited enough
      2011-03-15 22:38:46,782 INFO  (pool-11-thread-2) org.hornetq.jms.bridge.impl.patch.trace.JMSBridgeImpl$BatchTimeChecker@13123be5 got some messages so sending batch
      2011-03-15 22:38:46,782 INFO  (pool-11-thread-2) Sending batch of 1 messages
      2011-03-15 22:38:46,782 INFO  (pool-11-thread-2) Adding old message id in Message header
      2011-03-15 22:38:46,786 INFO  (pool-11-thread-2) Sending message HornetQMessage[ID:981c5fc0-4f4c-11e0-9367-00163e6c3f7b]:PERSISTENT
      2011-03-15 22:38:46,786 INFO  (pool-11-thread-2) Sent message HornetQMessage[ID:9b9b2641-4f4c-11e0-9367-00163e6c3f7b]:PERSISTENT
      2011-03-15 22:38:46,786 INFO  (pool-11-thread-2) Delisting resources from tx
      2011-03-15 22:38:46,806 INFO  (pool-11-thread-2) Delisted resources from tx
      2011-03-15 22:38:46,806 INFO  (pool-11-thread-2) Committing JTA transaction
      2011-03-15 22:38:46,918 INFO  (pool-11-thread-2) Committed JTA transaction
      2011-03-15 22:38:46,918 INFO  (pool-11-thread-2) Starting JTA transaction
      2011-03-15 22:38:46,918 INFO  (pool-11-thread-2) Started JTA transaction
      2011-03-15 22:38:46,918 INFO  (pool-11-thread-2) Enlisting resources in tx
      2011-03-15 22:38:46,918 INFO  (pool-11-thread-2) Enlisted resources in tx
      2011-03-15 22:38:46,918 INFO  (pool-11-thread-2) org.hornetq.jms.bridge.impl.patch.trace.JMSBridgeImpl$BatchTimeChecker@13123be5 sent batch
      2011-03-15 22:38:46,918 INFO  (pool-11-thread-2) org.hornetq.jms.bridge.impl.patch.trace.JMSBridgeImpl$BatchTimeChecker@13123be5 waiting for 5000
      

       

      Only, as mentioned the message simply doesn't arrive in the remote destination.

       

      Now comes the awkward thing, when I kill the remote server, the exact same sequence of log lines appears and there is no exception whatsoever. The messages seem to be lost in thin air. I also checked the DLQ queue, but there's nothing there.

       

      Only when I restart the server on which the bridge lives, while the target server is still down, the bridge starts complaining that it can't connect.

       

      So there are two problems:

       

      1. When everything should be connected (the target destination can be retrieved from remote JNDI), the message just doesn't arrive.
      2. When the target server is down, the bridge doesn't complain about this at all.

       

      When I run both EAR applications on my localhost (either deployed to the same JBoss AS instance or two different ones) the messages do arrive. I'm very puzzled about this and any help would be highly appreciated.

        • 1. Re: JMS bridge doesn't complain, but no msg in destination
          atijms

          p.s.

           

          If I kill the remote server on my localhost, I do almost immediately get the following exception:

           

           

          18:32:43,597 WARN  [org.hornetq.jms.bridge.impl.patch.JMSBridgeImpl] Detected failure on bridge connection
          
          18:32:59,467 DEBUG [org.jnp.interfaces.NamingContext] Failed to connect to localhost:1199: javax.naming.CommunicationException: Failed to connect to server localhost/127.0.0.1:1199 [Root exception is javax.naming.ServiceUnavailableException: Failed to connect to server localhost/127.0.0.1:1199 [Root exception is java.net.ConnectException: Connection refused]]
              at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:337) [:5.0.5.Final]
              at org.jnp.interfaces.NamingContext.checkRef(NamingContext.java:1746) [:5.0.5.Final]
              at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:695) [:5.0.5.Final]
              at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:688) [:5.0.5.Final]
              at javax.naming.InitialContext.lookup(InitialContext.java:392) [:1.6.0_23]
              at org.hornetq.jms.bridge.impl.JNDIFactorySupport.createObject(JNDIFactorySupport.java:58) [:6.0.0.Final]
              at org.hornetq.jms.bridge.impl.JNDIDestinationFactory.createDestination(JNDIDestinationFactory.java:40) [:6.0.0.Final]
              at org.hornetq.jms.bridge.impl.patch.JMSBridgeImpl.setupJMSObjects(JMSBridgeImpl.java:1089)
              at org.hornetq.jms.bridge.impl.patch.JMSBridgeImpl.setupJMSObjectsWithRetry(JMSBridgeImpl.java:1347)
              at org.hornetq.jms.bridge.impl.patch.JMSBridgeImpl.access$19(JMSBridgeImpl.java:1336)
              at org.hornetq.jms.bridge.impl.patch.JMSBridgeImpl$FailureHandler.run(JMSBridgeImpl.java:1863)
              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_23]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_23]
              at java.lang.Thread.run(Thread.java:662) [:1.6.0_23]
          

           

          In this case this exception is thus what is required. On this other machine, the fact that I don't even get this exception probably means starting the connection has failed or hangs, but why is there no exception from the bridge or HornetQ about this when setting up the JMS objecs?

          • 2. JMS bridge doesn't complain, but no msg in destination
            atijms

            After a lot of debugging and fiddling with stuff, I finally found that it's related to the binding ports of JBoss AS.

             

            If I start up JBoss AS with -b 0.0.0.0 the strange behavior as outlined above happens. No exceptions, but also no messages that arrive. If I start up with -b [some IP address], everything works as expected.

             

            It's a little problematic to have JBoss AS started with -b [IP address], since in our case a server may have multiple IP addresses and we have multiple servers, so this would not exactly be trivial to configure.

            • 3. JMS bridge doesn't complain, but no msg in destination
              ataylor

              you could use a core bridge instead.

              • 4. JMS bridge doesn't complain, but no msg in destination
                atijms

                Andy Taylor wrote:

                 

                you could use a core bridge instead.

                I did indeed briefly looked into this, but the following in the manual weirds me out:

                If you're using JMS then normally the JMS configuration hornetq-jms.xml is loaded after the core configuration file                            hornetq-configuration.xml is loaded. If your bridge                        is consuming from a JMS queue then you'll need to make sure the JMS queue is                        also deployed as a core queue in the core configuration. Take a look at the                        bridge example for an example of how this is done.

                If possible, I would like to avoid this. Our queues are defined inside the EAR. Would it be possible to define such a core queue in a file inside the EAR? I'm also not overly happy about having to define the same queue twice (as core and as JMS), but this would be a small price to pay.

                 

                But at any length, shouldn't the JMS bridge also just work? Since the JMS bridge is internally not doing anything special, I guess that this effects any 'regular' JMS usage (without a bridge thus) as well. Afterall, the bridge simply gets a destination from remote JNDI, creates a session and a standard JMS MessageProducer. We've always used the JMS bridge in the previous version (JBoss AS 5.1/JBM) and there was no issue with binding the server to 0.0.0.0.

                • 5. JMS bridge doesn't complain, but no msg in destination
                  ataylor

                  the core bridge is designed to be performant and i would always recomend using it ahead of the jms bridge, like you say the jms bridge is just using basic jms stuff and jndi so should just work, im guessing that this is an issue with jndi rather than hornetq.

                  • 6. JMS bridge doesn't complain, but no msg in destination
                    atijms

                    Andy Taylor wrote:

                     

                    the core bridge is designed to be performant and i would always recomend using it ahead of the jms bridge, like you say the jms bridge is just using basic jms stuff and jndi so should just work, im guessing that this is an issue with jndi rather than hornetq.

                    You would say, but it isn't. The destination (a queue in this case) is succesfullly retrieved from the JNDI of the remote server. Between the same two servers JNDI lookups in general and EJB communication works perfectly.

                     

                    The problem is when the bridge (or as mentioned, any JMS client) creates a MessageProducer for this remote destination and uses it to send a message.

                     

                    There is no exception or error logged, but it clearly doesn't work.

                    • 7. JMS bridge doesn't complain, but no msg in destination
                      ataylor

                      hmmm, weird, at the point that the jms resources have been looked up there is no difference. I can't see how it wouldnt work. All that i can think is that jndi is some how returning the wrong source or target connection factories, maybe the bridge is being created to itself or something, that would definately cause messages to go missing. if you could post your config with easy to run set up instructions i'll see if i get time to take a look. or you could debug the creation of the jms resources t each end of the bridge and also check using the jmx-console to ee whats connected to where.

                      • 8. Re: JMS bridge doesn't complain, but no msg in destination
                        atijms

                        he Andy Taylor wrote:

                         

                        hmmm, weird, at the point that the jms resources have been looked up there is no difference. I can't see how it wouldnt work. All that i can think is that jndi is some how returning the wrong source or target connection factories, maybe the bridge is being created to itself or something, that would definately cause messages to go missing

                         

                        Indeed, if the bridge would be configured wronly, you would expect that.

                         

                        But it works as soon as I restart the target server with -b [one of its IP addresses], without changing the bridge. Therefor, the bridge should be configured correctly, shouldn't it?

                         

                        It also works between two JBoss AS instances on my machine, or between two such instances on 2 machines on my local network. For the two machines where it silently fails, a difference might be that the target server has multiple IP addresses. Since those machines are remote servers, debugging is not exactly trivial there. I'm now doing this via patched HornetQ classes that do extra logging and by deploying test code.

                         

                         

                        or you could debug the creation of the jms resources t each end of the bridge and also check using the jmx-console to ee whats connected to where.

                         

                        All JMS resources are seemingly created normally. This is for instance what the local queue on the client server shows:

                         

                        ./twiddle.sh get 'org.hornetq:module=JMS,type=Queue,name="testLocalQueue"'
                        Selector=null
                        Paused=false
                        JNDIBindings=[Ljava.lang.String;@5d61dfb5
                        DeadLetterAddress=jms.queue.DLQ
                        ExpiryAddress=jms.queue.ExpiryQueue
                        ConsumerCount=1
                        DeliveringCount=0
                        ScheduledCount=0
                        MessagesAdded=3
                        Address=jms.queue.testLocalQueue
                        Name=testLocalQueue
                        Temporary=false
                        MessageCount=0
                        

                         

                        If I send a message to it, it's this:

                         

                         

                        ./twiddle.sh get 'org.hornetq:module=JMS,type=Queue,name="testLocalQueue"'
                        Selector=null
                        Paused=false
                        JNDIBindings=[Ljava.lang.String;@21aed5f9
                        DeadLetterAddress=jms.queue.DLQ
                        ExpiryAddress=jms.queue.ExpiryQueue
                        ConsumerCount=1
                        DeliveringCount=1
                        ScheduledCount=0
                        MessagesAdded=4
                        Address=jms.queue.testLocalQueue
                        Name=testLocalQueue
                        Temporary=false
                        MessageCount=1
                        

                         

                        And a small while after that (the configured batch time of the bridge) it's this:

                         

                         

                        ./twiddle.sh get 'org.hornetq:module=JMS,type=Queue,name="testLocalQueue"'
                        Selector=null
                        Paused=false
                        JNDIBindings=[Ljava.lang.String;@44a613f8
                        DeadLetterAddress=jms.queue.DLQ
                        ExpiryAddress=jms.queue.ExpiryQueue
                        ConsumerCount=1
                        DeliveringCount=0
                        ScheduledCount=0
                        MessagesAdded=4
                        Address=jms.queue.testLocalQueue
                        Name=testLocalQueue
                        Temporary=false
                        MessageCount=0
                        

                         

                        Since the bridge is the only listener for the queue, it's thus the one that removed the message.

                         

                        The bridge shows this:

                         

                        ./twiddle.sh get 'org.hornetq:service=JMSBridge'
                        Selector=null
                        Failed=false
                        Paused=false
                        ClientID=null
                        TransactionManagerLocatorClass=org.hornetq.integration.jboss.tm.JBoss5TransactionManagerLocator
                        TransactionManagerLocatorMethod=getTm
                        SourceUsername=null
                        SourcePassword=null
                        TargetUsername=null
                        TargetPassword=null
                        FailureRetryInterval=5000
                        MaxRetries=-1
                        QualityOfServiceMode=ONCE_AND_ONLY_ONCE
                        MaxBatchSize=25
                        MaxBatchTime=5000
                        SubscriptionName=null
                        AddMessageIDInHeader=true
                        Started=true
                        

                         

                        There is not too much useful information for this specific case (like messages processed or amount of failures etc), but at least it's clear it's started and at that point not failed.

                         

                        At the other (target) server, the following is shown:

                         

                        ./twiddle.sh get 'org.hornetq:module=JMS,type=Queue,name="testQueue"'
                        Selector=null
                        Paused=false
                        JNDIBindings=[Ljava.lang.String;@21aed5f9
                        DeadLetterAddress=jms.queue.DLQ
                        ExpiryAddress=jms.queue.ExpiryQueue
                        ConsumerCount=15
                        DeliveringCount=0
                        ScheduledCount=0
                        MessagesAdded=0
                        Address=jms.queue.testQueue
                        Name=testQueue
                        Temporary=false
                        MessageCount=0
                        

                         

                        The actual configuration for the bridge is fairly trivial, I could post it, but as mentioned the problem doesn't occur on my local machine or between machines on my local network. On those remote machines, I deployed the exact same code with the exact same configuration and only made one single change: java.naming.provider.url is changed into the name of the other remote server.

                         

                        It's thus something specific to those remote servers, which would normally of course be for me to find out (e.g. firewall issues, etc), but in this case the major problem is that HornetQ, Netty, JBoss AS etc don't give any exception whatsoever. For the code everything seems to work fine, but the messages just disappear.

                         

                        What I could think of is that when java.naming.provider.url points to jnp://test.myserver.com, and the JBoss AS instance there is bound to 0.0.0.0 AND has multiple NICs (and thus multiple IPs), that at some place HornetQ and/or Netty get IP address 'A' back that they use for communication. Maybe internally in that target JBoss AS instance, the destination queue only happens to listen to IP address 'B' and thus never sees the messages that come in via 'A'.

                         

                        Would that be a possibility?

                        • 9. Re: JMS bridge doesn't complain, but no msg in destination
                          ataylor
                          But it works as soon as I restart the target server with -b [one of its IP addresses], without changing the bridge. Therefor, the bridge should be configured correctly, shouldn't it?

                          I guess that depends on whether your restarting the server with the bridge on or not. This info would be useful as the bridge can live on either source or target server, in fact it could sit in a server all by itself.

                           

                          ./twiddle.sh get 'org.hornetq:module=JMS,type=Queue,name="testLocalQueue"'
                          Selector=null
                          Paused=false
                          JNDIBindings=[Ljava.lang.String;@21aed5f9
                          DeadLetterAddress=jms.queue.DLQ
                          ExpiryAddress=jms.queue.ExpiryQueue
                          ConsumerCount=1
                          DeliveringCount=1
                          ScheduledCount=0
                          MessagesAdded=4
                          Address=jms.queue.testLocalQueue
                          Name=testLocalQueue
                          Temporary=false
                          MessageCount=1

                          hmmm, this show 4 messages added altho you have only sent one

                           

                          What I could think of is that when java.naming.provider.url points to jnp://test.myserver.com, and the JBoss AS instance there is bound to 0.0.0.0 AND has multiple NICs (and thus multiple IPs), that at some place HornetQ and/or Netty get IP address 'A' back that they use for communication. Maybe internally in that target JBoss AS instance, the destination queue only happens to listen to IP address 'B' and thus never sees the messages that come in via 'A'.

                           

                          well jndi will expose itself over all NiCS, HornetQ exposes itself over what ever is configured in Hornetq-configuration.xml, when the bridge connection is made, it connects to jndi and jndi gives it a connection factory which has a host n port, once connected it doesnt matter about destinations etc, it will always send/receive to this server.

                          • 10. Re: JMS bridge doesn't complain, but no msg in destination
                            atijms

                            Andy Taylor wrote:

                             

                            I guess that depends on whether your restarting the server with the bridge on or not. This info would be useful as the bridge can live on either source or target server, in fact it could sit in a server all by itself.


                             

                            You're right, sorry for not making this clear.

                             

                            The bridge lives on the source server, which is the same server where testLocalQueue sits. The target server has no bridge and is the server where testQueue sits.

                             

                            I did some more testing and found the problem is indeed caused by the binding (-b startup parameter). When the target server is started with -b 0.0.0.0 and the source server with -b [some IP] there's an exception. But when both servers are started with -b 0.0.0.0 there is the silent failure.

                             

                            To summarize these findings:

                             

                                          
                            Source server Target server Works Exception/error
                            -b 0.0.0.0 -b 0.0.0.0 No No(!)
                            -b 10.0.0.1 -b 0.0.0.0 No Yes
                            -b 10.0.0.1 -b 10.0.0.2 Yes -
                            -b 0.0.0.0 -b 10.0.0.2 Yes -

                             

                             

                            If the target server binds to a fixed IP, it always works. If the target server does not bind to a fixed IP, it depends on the source server whether there will an exception or a silent failure. For each test situation in the table above, I restarted both servers but did not change any configuration or code other than the -b startup parameter for JBoss AS.

                             

                             

                            Andy Taylor wrote:


                            ./twiddle.sh get 'org.hornetq:module=JMS,type=Queue,name="testLocalQueue"'
                            Selector=null
                            Paused=false
                            JNDIBindings=[Ljava.lang.String;@21aed5f9
                            DeadLetterAddress=jms.queue.DLQ
                            ExpiryAddress=jms.queue.ExpiryQueue
                            ConsumerCount=1
                            DeliveringCount=1
                            ScheduledCount=0
                            MessagesAdded=4
                            Address=jms.queue.testLocalQueue
                            Name=testLocalQueue
                            Temporary=false
                            MessageCount=1

                            hmmm, this show 4 messages added altho you have only sent one


                             

                            Oh, yes, that's a bit confusting but correct. Before I started the test there had already been 3 messages send to the queue in the past.

                            • 11. Re: JMS bridge doesn't complain, but no msg in destination
                              atijms

                              I did some debugging in the HornetQ source code to see where things go wrong. I debugged how the connection is established from this line onwards in JMSBridgeImpl#setupJMSObjects():

                               

                               targetConn = createConnection(targetUsername, targetPassword, targetCff);
                              

                               

                              For the situation "source server binds to IP, target server binds to 0.0.0.0", the interesting bits happened in org.hornetq.core.remoting.impl.netty.NettyConnector.NettyConnector#createConnection():

                               

                              Right in the beginning of the method, the code tries to create a socket connection to 0.0.0.0. This is obviously not correct! It happens in this line (comment added by me).

                               

                              address = new InetSocketAddress(host, port); // port = 0.0.0.0, port 5445
                              

                               

                              The actual connection is done asynchronously, so the following returns false:

                               

                              ChannelFuture future = bootstrap.connect(address);
                              future.awaitUninterruptibly();
                              
                              if (future.isSuccess()) // false
                              

                               

                              The actual exception is now swallowed:

                               

                               

                               else
                                    {
                                       Throwable t = future.getCause();
                              
                                       if (t != null && !(t instanceof ConnectException))
                                       {
                                          NettyConnector.log.error("Failed to create netty connection", future.getCause());
                                       }
                              
                                       return null;
                                    }
                              

                               

                              But in the debugger one can see it's an ConnectException saying the connection to 0.0.0.0 is denied (which is expected really). Later on however, the error condition is caught and a new exception is being thrown. This is eventually logged as:

                               

                               

                              14:02:13,852 TRACE [org.hornetq.jms.bridge.impl.patch.JMSBridgeImpl] Failed to connect bridge: javax.jms.JMSException: Unable to connect to server using configuration org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=0-0-0-0
                                  at org.hornetq.core.client.impl.FailoverManagerImpl.createSession(FailoverManagerImpl.java:375) [:6.0.0.Final]
                                  at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSessionInternal(ClientSessionFactoryImpl.java:1123) [:6.0.0.Final]
                                  at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSession(ClientSessionFactoryImpl.java:849) [:6.0.0.Final]
                                  at org.hornetq.jms.client.HornetQConnection.authorize(HornetQConnection.java:565) [:6.0.0.Final]
                                  at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:624) [:6.0.0.Final]
                                  at org.hornetq.jms.client.HornetQConnectionFactory.createXAConnection(HornetQConnectionFactory.java:152) [:6.0.0.Final]
                                  at org.hornetq.jms.client.HornetQConnectionFactory.createXAConnection(HornetQConnectionFactory.java:147) [:6.0.0.Final]
                                  at org.hornetq.jms.bridge.impl.patch.JMSBridgeImpl.createConnection(JMSBridgeImpl.java:992) [:]
                                  at org.hornetq.jms.bridge.impl.patch.JMSBridgeImpl.setupJMSObjects(JMSBridgeImpl.java:1095) [:]
                                  at org.hornetq.jms.bridge.impl.patch.JMSBridgeImpl.setupJMSObjectsWithRetry(JMSBridgeImpl.java:1347) [:]
                                  at org.hornetq.jms.bridge.impl.patch.JMSBridgeImpl.access$19(JMSBridgeImpl.java:1336) [:]
                                  at org.hornetq.jms.bridge.impl.patch.JMSBridgeImpl$FailureHandler.run(JMSBridgeImpl.java:1863) [:]
                                  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_23]
                                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_23]
                                  at java.lang.Thread.run(Thread.java:662) [:1.6.0_23]
                              Caused by: HornetQException[errorCode=2 message=Unable to connect to server using configuration org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=0-0-0-0]
                              

                               

                               

                               

                              Then the situation "source server binds to 0.0.0.0, target server binds to 0.0.0.0", the interesting bits are again in

                              NettyConnector#createConnection().

                               

                              First the host is again 0.0.0.0 in the following line:

                               

                              address = new InetSocketAddress(host, port); // port = 0.0.0.0, port 5445
                              

                               

                              Nothing new here, but now the following happens: the previously shown "if (future.isSuccess())" now returns true: (comment added by me):

                               

                               

                              if (future.isSuccess()) // true
                                    {
                                       final Channel ch = future.getChannel();
                              
                                       // ...
                              
                                       NettyConnection conn = new NettyConnection(ch, new Listener(), !httpEnabled && batchDelay > 0, false);
                              
                                      // org.hornetq.core.remoting.impl.netty.NettyConnection@158dbf4d[local= /10.0.0.1:38423, remote=my-workstation/10.0.0.1:5445]
                              

                               

                              In the comment I've put the result of conn.toString(). As can be seen, it clearly seems to connect back to itself. The target server most likely erroneously returns 0.0.0.0 as the address to connect to, and in the case the source server is also bound to 0.0.0.0, it is accepted as a valid local IP.

                               

                              Then another strange thing happens. Whenever this connection back to the source server happens, the target queue, which is normally only available at the target server now appears on the source server. Via mbean explorer I found this at: org.hornetq -> Core -> Queue -> jms.queue.testQueue.

                               

                              I double and tripple validated that "testQueue" is absolutely not defined at the source server. It dynamically appears whenever both source and target servers bind to 0.0.0.0. There is nowhere any specific configuration that the source server connects to itself (as mentioned, when I restart the servers without changing any configuration or code and ONLY provide another value for -b when starting the two JBoss AS instances, everything works).

                              • 12. Re: JMS bridge doesn't complain, but no msg in destination
                                atijms

                                After some more testing, it appears that the 0.0.0.0 IP address used to connect to the target server, originates from the connection factory obtained from JNDI.

                                 

                                A very basic JNDI lookup of this factory already reveals the problem:

                                 

                                Properties env = new Properties();
                                env.put("java.naming.factory.initial", "org.jnp.interfaces.NamingContextFactory");
                                env.put("java.naming.factory.url.pkgs", "org.jboss.naming:org.jnp.interfaces");
                                env.put("java.naming.provider.url", "jnp://myserver.com:1099"); // a remote server
                                InitialContext context = new InitialContext(env);
                                
                                Object test = context.lookup("/ConnectionFactory");
                                

                                 

                                In this case, "test' is a org.hornetq.jms.client.HornetQConnectionFactory, which internally contains the reference back to the remove server where it came from. This appears to be this wrong 0.0.0.0 again, when the target server is binding to 0.0.0.0.

                                 

                                The exact object reference chain is:

                                 

                                test (HornetQConnectionFactory) -> sessionFactory (ClientSessionFactoryImpl) -> staticConnectors (ArrayList<E>) -> [0] (Pair) -> A (TransportConfiguration) -> params (HashMap) -> {host=0.0.0.0, port=5445}

                                 

                                So when a server is started with -b 0.0.0.0, this 0.0.0.0 isn't an actual IP address, but merely a convenience to say "bind to all IP addresses". I don't think this particular value should be communicated to external servers as a callback IP, should it?

                                 

                                (for comparison; JBoss EJB3 remote proxies contain the host name in an internal object called SessionRemoteProxyInvocationHandler.url, e.g. "socket://mymachine0.dmz.mynetwork.com:3873/?timeout=300000", this does work.)

                                • 13. Re: JMS bridge doesn't complain, but no msg in destination
                                  ataylor

                                  ahhh, ok, if you look at your connector definition in hornetq-configuration.xml its probably set as "${jboss.bind.address:localhost}" which is set by the container to 0.0.0.0, obviously this only works for explicit ip addresses. you can just change this to the ip address and it will work.

                                  • 14. Re: JMS bridge doesn't complain, but no msg in destination
                                    atijms

                                    Andy Taylor wrote:

                                     

                                    ahhh, ok, if you look at your connector definition in hornetq-configuration.xml its probably set as "${jboss.bind.address:localhost}" which is set by the container to 0.0.0.0, obviously this only works for explicit ip addresses. you can just change this to the ip address and it will work.

                                     

                                    I noticed this indeed. With JBoss AS 5/JBoss Messaging it wasn't needed to configure the server's own IP address anywhere. For the sake of deployment and maintainablity, configuring the server with its own IP address is less than desirable. It will work on my local workstation / network, but this will be a very hard (if not impossible) sell to get into production.

                                    1 2 Previous Next