1 2 Previous Next 23 Replies Latest reply on Oct 22, 2012 5:38 AM by smswamy Go to original post
      • 15. Re: HornetQ overlapping address(Multicast cluster) problem in JBOSS AS 7.1.1
        jbertram

        To my knowledge, nothing in HornetQ needs jboss.node.name to be set, but I cannot speak for other components within AS7. 

         

        At the end of the day your two clusters should be using unique multicast addresses and ports.  If you are already doing this (which it seems like you may be) please work up a simple example that reproduces the problem you're seeing and attach it to the thread.  If possible, make the reproducer use a single machine because that's all I have to work with where I am currently.  I'll try it out and see if I can reproduce it as well. 

        • 16. Re: HornetQ overlapping address(Multicast cluster) problem in JBOSS AS 7.1.1
          smswamy

          Hi Justin, I am not abe to reproduce it in local. I created 2 clusters. Each cluster had 2 instances. All the 4 instances are in my local system. The bind address is set to localhost and I changed all the port numbers of all the instances. I started all the instances. Its not showing up any errors.

           

           

          Still I am getting the same binding issue in the log when I deploy it in linux server,

          08-08-2012 15:14:41,035 ERROR [org.hornetq.core.server.cluster.impl.ClusterConnectionImpl] (Thread-3 (HornetQ-client-global-threads-1288110412)) Failed to handle message: java.lang.IllegalStateException: Cannot find binding for 92e18543-5dd4-424f-8059-6adbdf48b6bdc4b0d943-e16b-11e1-9972-634e4086895c on ClusterConnectionImpl@1694429802[nodeUUID=c4b0d943-e16b-11e1-9972-634

           

          This error/warning which I had earlier posted happens only when the server is started and not after that ??? Also I see packets movement from one node to other node in the cluster. So cluster is working but still it throws the warning/error messages during server startup. Not sure why.

           

          Although both the servers are alive and running I see the following message. I couldn't understand why I am getting this messsage.


          14-08-2012 09:43:30,739 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] (hornetq-failure-check-thread) Connection failure has been detected: Did not receive data from /17.158.36.19:61029. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=3]

          • 17. Re: HornetQ overlapping address(Multicast cluster) problem in JBOSS AS 7.1.1
            smswamy

            Hi Justin, Although I am not able to reproduce it in a single system, I am still getting it when we have more than one systems. I am still analyzing the issue. Will you please confirm me this,

             

            In the standalone.xml my connetor and acceptor is missing the netty-throughput.

                                <netty-connector name="netty-throughput" socket-binding="messaging-throughput">

                                    <param key="batch-delay" value="50"/>

                                </netty-connector>

             

            Is it mandatory to have it as part of messaging ?

             

             

                            <connectors>

                                <netty-connector name="netty" socket-binding="messaging"/>

                                <in-vm-connector name="in-vm" server-id="0"/>

                             </connectors>

             

            Please confirm.

            • 18. Re: HornetQ overlapping address(Multicast cluster) problem in JBOSS AS 7.1.1
              smswamy

              Hi Justin,

                    Will you please confirm me if the <local-bind-port> which we use as part of the multicast cluster broadcast group needs to be same with all the nodes in the cluster ?

              • 19. Re: HornetQ overlapping address(Multicast cluster) problem in JBOSS AS 7.1.1
                jbertram

                You don't need the netty-throughput connector by default.  You only need it if you change the default configuration to actually reference it.  It's there just for convenience if you need it for higher thoughput than the default connector.

                • 20. Re: HornetQ overlapping address(Multicast cluster) problem in JBOSS AS 7.1.1
                  jbertram

                  The <local-bind-port> does not need to be the same on members of a cluster.  In fact, you don't have to specify <local-bind-port> at all if you don't want to as it isn't required.  The default configuration in standalone-full-ha.xml doesn't have it.

                  • 21. Re: HornetQ overlapping address(Multicast cluster) problem in JBOSS AS 7.1.1
                    smswamy

                    Thanks for all those Justin. Let me consider my problem as local environmental issue. Will you please suggest things I may need to test/explore. I have put up the configuration below.

                     

                    I face these problems,

                     

                    (1) It throws up the cluster connected message multiple times although I got only 3 physical servers connected (this clearly shows there is a loop in the cluster)

                    (2) Then it gets me a warning saying

                              "Remote queue binding jms.queue.expiryQueue458fb681-ed23-11e1-8bbf-c42c031d96aa has already been bound in the post office. Most likely cause for this is you have a loop in your cluster due to cluster max-hops being too large or you have multiple cluster connections to the same nodes using overlapping addresses"

                        

                         For each queue/topic I get this message

                     

                    (3) Then it throws an error "Failed to handle message: java.lang.IllegalStateException: Cannot find binding for jms.queue.",

                    (4) Sometimes it says, nodeID is getting duplicated. I had removed the data and temp directories before starting the server. The systems are also separate physical systems. Stil wonder why am I getting it.

                     

                    I couldnt explore more...Is there a way I can debug ?

                     

                    This scenario I am able to get it even with desktop machines. I have  3 systems. Mostly the last system that connects to the cluster gets this message, other two do not get these errors/warnings. Whichever system I start as the last server gets these messages.

                     

                    All the three servers I had made sure I delete all the content in the data and temp directory before starting the server. Not Sure whats more I need to do.

                     

                    (1) Please clarify me if my nodeID/serverUUId is getting duplicated, how to avoid it other than deleting the data/temp directories.

                    (2) How to check max-hops is too large as it says there may be a chance of max hops too large.

                    (3) In what scenarios "cannot binding for queue may occur" and avoid it.

                     

                     

                    <hornetq-server>
                                <clustered>true</clustered>
                                <persistence-enabled>true</persistence-enabled>
                                <cluster-user>myconfig</cluster-user>
                                <cluster-password>mycluster</cluster-password>
                                <journal-file-size>102400</journal-file-size>
                                <journal-min-files>2</journal-min-files>

                                <connectors>
                                         <netty-connector name="netty" socket-binding="messaging"/>
                                         <in-vm-connector name="in-vm" server-id="5"/>
                                </connectors>

                                <acceptors>
                                         <netty-acceptor name="netty" socket-binding="messaging"/>
                                         <in-vm-acceptor name="in-vm" server-id="5"/>
                                </acceptors>

                                <broadcast-groups>
                                    <broadcast-group name="bg-config">
                                        <local-bind-port>9877</local-bind-port>
                                        <group-address>224.1.22.1</group-address>
                                        <group-port>9876</group-port>
                                        <broadcast-period>5000</broadcast-period>
                                        <connector-ref>
                                            netty
                                        </connector-ref>
                                    </broadcast-group>
                                </broadcast-groups>

                                <discovery-groups>
                                    <discovery-group name="dg-config">
                                        <group-address>224.1.22.1</group-address>
                                        <group-port>9876</group-port>
                                        <refresh-timeout>10000</refresh-timeout>
                                    </discovery-group>
                                </discovery-groups>

                                <cluster-connections>
                                    <cluster-connection name="config-cluster">
                                        <address>jms</address>
                                        <connector-ref>netty</connector-ref>
                                        <forward-when-no-consumers>true</forward-when-no-consumers>
                                        <max-hops>1</max-hops>
                                        <discovery-group-ref discovery-group-name="dg-config"/>
                                    </cluster-connection>
                                </cluster-connections>

                                <security-settings>
                                    <security-setting match="#">
                                        <permission type="send" roles="guest"/>
                                        <permission type="consume" roles="guest"/>
                                        <permission type="createNonDurableQueue" roles="guest"/>
                                        <permission type="deleteNonDurableQueue" roles="guest"/>
                                    </security-setting>
                                </security-settings>

                                <address-settings>
                                    <address-setting match="#">
                                        <dead-letter-address>jms.queue.DLQ</dead-letter-address>
                                        <expiry-address>jms.queue.ExpiryQueue</expiry-address>
                                        <redelivery-delay>500</redelivery-delay>
                                        <max-delivery-attempts>5000</max-delivery-attempts>
                                        <max-size-bytes>10485760</max-size-bytes>
                                        <address-full-policy>BLOCK</address-full-policy>
                                        <message-counter-history-day-limit>10</message-counter-history-day-limit>
                                    </address-setting>
                                </address-settings>

                                <jms-connection-factories>
                                    <connection-factory name="InVmConnectionFactory">
                                        <connectors>
                                            <connector-ref connector-name="in-vm"/>
                                        </connectors>
                                        <entries>
                                            <entry name="java:/ConnectionFactory"/>
                                        </entries>
                                        <connection-ttl>-1</connection-ttl>
                                    </connection-factory>
                                    <connection-factory name="RemoteConnectionFactory">
                                        <connectors>
                                            <connector-ref connector-name="netty"/>
                                        </connectors>
                                        <entries>
                                            <entry name="RemoteConnectionFactory"/>
                                            <entry name="java:jboss/exported/jms/RemoteConnectionFactory"/>
                                        </entries>
                                        <connection-ttl>-1</connection-ttl>
                                    </connection-factory>
                                    <pooled-connection-factory name="hornetq-ra">
                                        <transaction mode="xa"/>
                                        <connectors>
                                            <connector-ref connector-name="in-vm"/>
                                        </connectors>
                                        <entries>
                                            <entry name="java:/JmsXA"/>
                                        </entries>
                                    </pooled-connection-factory>
                                </jms-connection-factories>

                                <jms-destinations>
                                    <jms-queue name="testQueue">
                                        <entry name="queue/test"/>
                                        <entry name="java:jboss/exported/jms/queue/test"/>
                                    </jms-queue>
                                    <jms-queue name="expiryQueue">
                                        <entry name="queue/ExpiryQueue"/>
                                        <entry name="java:jboss/exported/jms/queue/ExpiryQueue"/>
                                    </jms-queue>
                                    <jms-topic name="testTopic">
                                        <entry name="topic/test"/>
                                        <entry name="java:jboss/exported/jms/topic/test"/>
                                    </jms-topic>
                                </jms-destinations>
                            </hornetq-server>

                    • 22. Re: HornetQ overlapping address(Multicast cluster) problem in JBOSS AS 7.1.1
                      smswamy

                      Hi Justin, will you please let me know, which file we need to delete in the data directory to make sure the same node id is not being used.

                       

                      Will you please let me know, if there is a way we can set the node id(manually specify) through code or during startup the server as command line parameter.

                       

                      Thanks.

                      • 23. Re: HornetQ overlapping address(Multicast cluster) problem in JBOSS AS 7.1.1
                        smswamy

                        Hi Justin, I identified the issue. Its a local environment issue. Although the servers are stopped and started properly, there are times when the server is still running in the background, although it is showing us stopped in the background.

                         

                        There was an issue with our local scripts that starts and stops the server. We fixed it and everything worked fine.

                        Thanks for you support !!!.

                         

                        For all those who face similar issue, please make sure .

                        (1) The server.lock file(inside data directory) is not shared between 2 instances/nodes in a cluster. I got it from Justin, in this file the JBOSS maintains the NodeId.

                        (2) I faced a different scenario, wherein I didnt the shared the server.lock file. When we restart the server, the server stop cleanup was not proper and in the background the server was still running, although in our automated deployment scripts, it was showing as server stopped.

                        (3) When our autometed script restarted the server we started getting the issues mentioned in this discussion.

                         

                        Thanks.

                        1 2 Previous Next