-
1. Re: HornetQ overlapping address(Multicast cluster) problem in JBOSS AS 7.1.1
rhusar Jul 30, 2012 6:07 AM (in response to smswamy)Hello
IMHO
problem #1: wrong binding. You are binding to 127.0.0.x addresses. These are loopback addresses -- read on http://www.rfc-editor.org/rfc/rfc3330.txt -- that means the nodes can't communicate directly which is needed in many places. Bind to addresses accessible from other nodes.
problem #2: unsupported hornetQ topology. You should all cluster on 1 UDP address.
-
2. Re: HornetQ overlapping address(Multicast cluster) problem in JBOSS AS 7.1.1
smswamy Jul 30, 2012 7:44 AM (in response to rhusar)Hi, Although I have said the bind adresses are 127.0.0...I mentioned just for an example. Its not actual. In actual we have bound the Host Names to their IP adresses.
Group A: Multicast address - 224.2.2.1, 9876
Host A - 127.0.0.1(Bind address)
Host B - 127.0.0.2(Bind address)
Group B: Multicast address - 224.2.2.2, 9876
Host A - 127.0.0.1(Bind address)
Host C - 127.0.0.3(Bind address)
One more thing is Group A and Group B are 2 different clusters altogether(2 different environments we use). So, I have given 2 different Multicast addresses. Even when the Multicast address is same it is getting me the same problem.
Host A is common for both the environments consider Dev and Test env. Our application is deployed in Host A in 2 separate instances. The Netty connector port numbers are different, but both the instance in Host A share the same bind address that is the Host A IP address.
So For Dev,
Netty Connector will be Host A IP : 2345(port number for example)
For Test
Netty Connector will be Host A IP : 2348(port number for example)
Will you please able to confirm me, why am I getting this error ? Is it any bug with JBOSS HornetQ ?
"most likely cause for this is you have a loop in your cluster due to cluster max-hops being too large or you have multiple cluster connections to the same nodes using overlapping addresses"
-
3. Re: HornetQ overlapping address(Multicast cluster) problem in JBOSS AS 7.1.1
mnovak Jul 30, 2012 8:50 AM (in response to smswamy)Hi,
could you attach stack trace and configuration of the servers, please?
could you try to use unique name for each cluster connection in <cluster-connection name="my-cluster">?
Thanks
-
4. Re: HornetQ overlapping address(Multicast cluster) problem in JBOSS AS 7.1.1
smswamy Jul 30, 2012 9:16 AM (in response to mnovak)For both the clusters we have the same name, <cluster-connection name="my-cluster">
I will change the cluster names in both the multicast groups and try.
-
5. Re: HornetQ overlapping address(Multicast cluster) problem in JBOSS AS 7.1.1
smswamy Jul 30, 2012 9:38 AM (in response to mnovak)Meanwhile, this is the stack trace I am getting,
08:22:45,568 ERROR [org.hornetq.core.server.cluster.impl.ClusterConnectionImpl] (Thread-15 (HornetQ-client-global-threads-1527318062))
Failed to handle message: java.lang.IllegalStateException: Cannot find binding for jms.queue.queue1Queuebaa6a3f8-da1f-11e1-a7a5-00163eaa0f5a on ClusterConnectionImpl@1993101708[nodeUUID=baa6a3f8-da1f-11e1-a7a5-00163eaa0f5a, connector=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=55070&host=abc.server.com, address=jms, server=HornetQServerImpl::serverUUID=baa6a3f8-da1f-11e1-a7a5-00163eaa0f5a]
at org.hornetq.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.doConsumerCreated(ClusterConnectionImpl.java:1458) [hornetq-core-2.2.13.Final.jar:]
at org.hornetq.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.onMessage(ClusterConnectionImpl.java:1225) [hornetq-core-2.2.13.Final.jar:]
at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:983) [hornetq-core-2.2.13.Final.jar:]
at org.hornetq.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:48) [hornetq-core-2.2.13.Final.jar:]
at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1113) [hornetq-core-2.2.13.Final.jar:]
at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [hornetq-core-2.2.13.Final.jar:]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_19]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_19]
at java.lang.Thread.run(Thread.java:619) [rt.jar:1.6.0_19]
08:22:45,603 WARN [org.hornetq.core.server.cluster.impl.ClusterConnectionImpl] (Thread-9 (HornetQ-client-global-threads-1527318062)) MessageFlowRecordImpl [nodeID=f3dc24d4-d728-11e1-9b50-00163eaa0f5a, connector=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=55070&host=abc.server.com, queueName=sf.my-cluster.f3dc24d4-d728-11e1-9b50-00163eaa0f5a, queue=QueueImpl[name=sf.my-cluster.f3dc24d4-d728-11e1-9b50-00163eaa0f5a, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=baa6a3f8-da1f-11e1-a7a5-00163eaa0f5a]]@6099210f, isClosed=false, firstReset=true]::Remote queue binding 54a7c877-7adf-4bdb-b67c-dd52b9108e0fbaa6a3f8-da1f-11e1-a7a5-00163eaa0f5a has already been bound in the post office. Most likely cause for this is you have a loop in your cluster due to cluster max-hops being too large or you have multiple cluster connections to the same nodes using overlapping addresses
08:22:45,603 WARN [org.hornetq.core.server.cluster.impl.ClusterConnectionImpl] (Thread-19 (HornetQ-client-global-threads-1527318062)) MessageFlowRecordImpl [nodeID=f63a634e-d7f0-11e1-aa43-00163eaa0f5a, connector=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=55070&host=abc.server.com, queueName=sf.my-cluster.f63a634e-d7f0-11e1-aa43-00163eaa0f5a, queue=QueueImpl[name=sf.my-cluster.f63a634e-d7f0-11e1-aa43-00163eaa0f5a, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=baa6a3f8-da1f-11e1-a7a5-00163eaa0f5a]]@4aa168c, isClosed=false, firstReset=true]::Remote queue binding 54a7c877-7adf-4bdb-b67c-dd52b9108e0fbaa6a3f8-da1f-11e1-a7a5-00163eaa0f5a has already been bound in the post office. Most likely cause for this is you have a loop in your cluster due to cluster max-hops being too large or you have multiple cluster connections to the same nodes using overlapping addresses
08:22:45,606 ERROR [org.hornetq.core.server.cluster.impl.ClusterConnectionImpl] (Thread-9 (HornetQ-client-global-threads-1527318062)) Failed to handle message: java.lang.IllegalStateException: Cannot find binding for 54a7c877-7adf-4bdb-b67c-dd52b9108e0fbaa6a3f8-da1f-11e1-a7a5-00163eaa0f5a on ClusterConnectionImpl@1993101708[nodeUUID=baa6a3f8-da1f-11e1-a7a5-00163eaa0f5a, connector=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=55070&host=abc.server.com, address=jms, server=HornetQServerImpl::serverUUID=baa6a3f8-da1f-11e1-a7a5-00163eaa0f5a]
at org.hornetq.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.doConsumerCreated(ClusterConnectionImpl.java:1458) [hornetq-core-2.2.13.Final.jar:]
at org.hornetq.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.onMessage(ClusterConnectionImpl.java:1225) [hornetq-core-2.2.13.Final.jar:]
at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:983) [hornetq-core-2.2.13.Final.jar:]
at org.hornetq.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:48) [hornetq-core-2.2.13.Final.jar:]
at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1113) [hornetq-core-2.2.13.Final.jar:]
at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [hornetq-core-2.2.13.Final.jar:]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_19]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_19]
at java.lang.Thread.run(Thread.java:619) [rt.jar:1.6.0_19]
08:22:45,608 ERROR [org.hornetq.core.server.cluster.impl.ClusterConnectionImpl] (Thread-19 (HornetQ-client-global-threads-1527318062)) Failed to handle message: java.lang.IllegalStateException: Cannot find binding for 54a7c877-7adf-4bdb-b67c-dd52b9108e0fbaa6a3f8-da1f-11e1-a7a5-00163eaa0f5a on ClusterConnectionImpl@1993101708[nodeUUID=baa6a3f8-da1f-11e1-a7a5-00163eaa0f5a, connector=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=55070&host=abc.server.com, address=jms, server=HornetQServerImpl::serverUUID=baa6a3f8-da1f-11e1-a7a5-00163eaa0f5a]
at org.hornetq.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.doConsumerCreated(ClusterConnectionImpl.java:1458) [hornetq-core-2.2.13.Final.jar:]
at org.hornetq.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.onMessage(ClusterConnectionImpl.java:1225) [hornetq-core-2.2.13.Final.jar:]
at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:983) [hornetq-core-2.2.13.Final.jar:]
at org.hornetq.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:48) [hornetq-core-2.2.13.Final.jar:]
at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1113) [hornetq-core-2.2.13.Final.jar:]
at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [hornetq-core-2.2.13.Final.jar:]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_19]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_19]
at java.lang.Thread.run(Thread.java:619) [rt.jar:1.6.0_19]
-
6. Re: HornetQ overlapping address(Multicast cluster) problem in JBOSS AS 7.1.1
jbertram Jul 30, 2012 10:22 AM (in response to smswamy)1 of 1 people found this helpfulYou may be hitting https://issues.jboss.org/browse/HORNETQ-874. I recommend you try the latest nightly to see if your issue is resolved there.
-
7. Re: HornetQ overlapping address(Multicast cluster) problem in JBOSS AS 7.1.1
smswamy Jul 30, 2012 11:41 AM (in response to jbertram)I am using HornetQ as part of JBOSS AS 7.1.1 Final version. I see a HornetQ fixed version in the link you have provided me. Will you please let me know in which version of will this fix be available ?
-
8. Re: HornetQ overlapping address(Multicast cluster) problem in JBOSS AS 7.1.1
rhusar Jul 30, 2012 11:54 AM (in response to smswamy)I am using HornetQ as part of JBOSS AS 7.1.1 Final version. I see a HornetQ fixed version in the link you have provided me. Will you please let me know in which version of will this fix be available ?
Well, the next community release will be 7.2, until then you need to build yourself or download it from the link above.
-
9. Re: HornetQ overlapping address(Multicast cluster) problem in JBOSS AS 7.1.1
smswamy Aug 2, 2012 11:02 AM (in response to mnovak)Hi,
I tried out different cluster names, multicast address, port numbers and even I changed my queue names. Its throwing the same error.
The issue seems to be closer to what Justin had mentioned, although I couldnt confirm 100%. Host A is common to Dev and Test environment. So bind address of Host A is same and the Netty Connector port number is different for both the instances in the Host A.
From the Test environment, when HornetQ tries to do a auto discovery in multicast cluster, test environment(Host A) instance gets port number of the Dev environment instance(Host A) first and does the connection. There is a Test Environment Host A instance which also tries to do a cluster with a different port number and it throws the overlapping address error.
This I could confirm from the logs. The Test Environment cluster is getting connected to all the Dev Environment Hosts because of the breach that has happened.
I am looking at taking the latest build to test this scenario, in between If I set the <local-bind-port> in broadcast group will it give me a solution ??
-
10. Re: HornetQ overlapping address(Multicast cluster) problem in JBOSS AS 7.1.1
smswamy Aug 3, 2012 3:00 AM (in response to rhusar)Hi, If I just take the HornetQ module from the latest JBOSS 7.2.0 Alpla Snapshot Nightly build and use it in my JBOSS 7.1.1 Final version, will it work? Is there any issues in doing as such ?
Thanks
-
11. Re: HornetQ overlapping address(Multicast cluster) problem in JBOSS AS 7.1.1
jbertram Aug 3, 2012 9:14 AM (in response to smswamy)I believe it will work.
-
12. Re: HornetQ overlapping address(Multicast cluster) problem in JBOSS AS 7.1.1
smswamy Aug 7, 2012 2:13 PM (in response to jbertram)Hi Justin,
If I replace just the hornetQ module(whole hornetq module folder), I am getting these exceptions.. Will you please able to suggest me, is there a possibility if I can replace any specific jars within the hornetQ module or I need to replace more modules.
23:34:43,392 WARN [org.jboss.modules] (MSC service thread 1-1) Failed to define class org.jboss.as.messaging.jms.AS7RecoveryRegistry in Module "org.jboss.as.messaging:main" from local module loader @1d95492 (roots: /Users/mysystem/JbossTest/jboss-as-7.1.1.Final/modules): java.lang.LinkageError: Failed to link org/jboss/as/messaging/jms/AS7RecoveryRegistry (Module "org.jboss.as.messaging:main" from local module loader @1d95492 (roots: /Users/mysystem/JbossTest/jboss-as-7.1.1.Final/modules))
at org.jboss.modules.ModuleClassLoader.defineClass(ModuleClassLoader.java:396)
at org.jboss.modules.ModuleClassLoader.loadClassLocal(ModuleClassLoader.java:243)
at org.jboss.modules.ModuleClassLoader$1.loadClassLocal(ModuleClassLoader.java:73)
at org.jboss.modules.Module.loadModuleClass(Module.java:517)
at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:182)
at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:468)
at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:456)
at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:423)
at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:398)
at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:120)
at org.jboss.as.messaging.jms.PooledConnectionFactoryService.createService(PooledConnectionFactoryService.java:245) [jboss-as-messaging-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.as.messaging.jms.PooledConnectionFactoryService.start(PooledConnectionFactoryService.java:180) [jboss-as-messaging-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]
at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [classes.jar:1.6.0_33]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [classes.jar:1.6.0_33]
at java.lang.Thread.run(Thread.java:680) [classes.jar:1.6.0_33]
Caused by: java.lang.NoClassDefFoundError: org/hornetq/jms/server/recovery/RecoveryRegistry
at java.lang.ClassLoader.defineClass1(Native Method) [classes.jar:1.6.0_33]
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631) [classes.jar:1.6.0_33]
at java.lang.ClassLoader.defineClass(ClassLoader.java:615) [classes.jar:1.6.0_33]
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) [classes.jar:1.6.0_33]
at org.jboss.modules.ModuleClassLoader.doDefineOrLoadClass(ModuleClassLoader.java:327)
at org.jboss.modules.ModuleClassLoader.defineClass(ModuleClassLoader.java:391)
... 16 more
Caused by: java.lang.ClassNotFoundException: org.hornetq.jms.server.recovery.RecoveryRegistry from [Module "org.jboss.as.messaging:main" from local module loader @1d95492 (roots: /Users/mysystem/JbossTest/jboss-as-7.1.1.Final/modules)]
at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:190)
at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:468)
at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:456)
at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:423)
at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:423)
at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:398)
at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:120)
... 22 more
-
13. Re: HornetQ overlapping address(Multicast cluster) problem in JBOSS AS 7.1.1
jbertram Aug 7, 2012 2:48 PM (in response to smswamy)Looks like there were some changes in the AS 7 messaging subsystem since 7.1.1 that went along with the HornetQ upgrade so using that version of HornetQ doesn't appear to be as straight-forward as it first appeared.
Have you actually tried 7.2.0.Alpha to see if it resolves the problem?
You can always pull HornetQ 2.2.13 (https://github.com/hornetq/hornetq/tree/HornetQ_2_2_13_AS7_Final) and apply the fix from HORNETQ-874 (https://github.com/hornetq/hornetq/commit/d872b84b0098eacfb0029a19c06ec4fa4b07b9e7).
-
14. Re: HornetQ overlapping address(Multicast cluster) problem in JBOSS AS 7.1.1
smswamy Aug 8, 2012 11:34 AM (in response to jbertram)Hi Justin,
From the link you had given me, I found that the change is only with hornetq-core.jar. So I replaced the hornetq-core-2.2.13 jar present in jboss 7.1.1 Final with hornetq-core-2.2.18 jar. The change is reflected in the following message. Everything is working fine except that, the overlapping issue didnt got solved.
Will you please clarify me, do I need to give a different node name(-Djboss.node.name=node1) through the runtime parameter, for all the nodes present in the multicast cluster, as specified in the below link. Is it mandatory ?
http://middlewaremagic.com/jboss/?p=1952
08-08-2012 15:14:41,030 WARN [org.hornetq.core.server.cluster.impl.ClusterConnectionImpl] (Thread-3 (HornetQ-client-global-threads-1288110412)) MessageFlowRecordImpl [nodeID=2076c198-e0bd-11e1-9639-00163eaa0f5a, connector=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=40297&host=our-server-com, queueName=sf.myconfig-cluster.2076c198-e0bd-11e1-9639-00163eaa0f5a, queue=QueueImpl[name=sf.myconfig-cluster.2076c198-e0bd-11e1-9639-00163eaa0f5a, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=c4b0d943-e16b-11e1-9972-634e4086895c]]@10e80317, isClosed=false, firstReset=true]::Remote queue binding 92e18543-5dd4-424f-8059-6adbdf48b6bdc4b0d943-e16b-11e1-9972-634e4086895c has already been bound in the post office. Most likely cause for this is you have a loop in your cluster due to cluster max-hops being too large or you have multiple cluster connections to the same nodes using overlapping addresses
08-08-2012 15:14:41,035 ERROR [org.hornetq.core.server.cluster.impl.ClusterConnectionImpl] (Thread-3 (HornetQ-client-global-threads-1288110412)) Failed to handle message: java.lang.IllegalStateException: Cannot find binding for 92e18543-5dd4-424f-8059-6adbdf48b6bdc4b0d943-e16b-11e1-9972-634e4086895c on ClusterConnectionImpl@1694429802[nodeUUID=c4b0d943-e16b-11e1-9972-634e4086895c, connector=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=40297&host=our-server-com, address=jms, server=HornetQServerImpl::serverUUID=c4b0d943-e16b-11e1-9972-634e4086895c]
at org.hornetq.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.doConsumerCreated(ClusterConnectionImpl.java:1458) [hornetq-core-2.2.18.Final.jar:2.2.18.Final (HQ_2_2_18_FINAL, 122)]
at org.hornetq.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.onMessage(ClusterConnectionImpl.java:1225) [hornetq-core-2.2.18.Final.jar:2.2.18.Final (HQ_2_2_18_FINAL, 122)]
at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1028) [hornetq-core-2.2.18.Final.jar:2.2.18.Final (HQ_2_2_18_FINAL, 122)]
at org.hornetq.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:50) [hornetq-core-2.2.18.Final.jar:2.2.18.Final (HQ_2_2_18_FINAL, 122)]
at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1173) [hornetq-core-2.2.18.Final.jar:2.2.18.Final (HQ_2_2_18_FINAL, 122)]
at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [hornetq-core-2.2.18.Final.jar:2.2.18.Final (HQ_2_2_18_FINAL, 122)]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_19]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_19]
at java.lang.Thread.run(Thread.java:619) [rt.jar:1.6.0_19]