5 Replies Latest reply on Feb 7, 2010 8:25 PM by yingyuan.cheng

strange cluster problem:broadcasting the same node id

yingyuan.cheng Feb 4, 2010 12:55 AM

I have two nodes to form a cluster under fedora linux 12. If I close then start the first node three times, the second node will complain same broadcasting node id received.

The first node command line and log:

^C[yingyuan@localhost bin]$ ./run.sh ../config/stand-alone/hqcluster/
***********************************************************************************
java -Xms32M -Xmx64M -Dhornetq.config.dir=../config/stand-alone/hqcluster/ -Djava.util.logging.config.file=../config/stand-alone/hqcluster//logging.properties -Djava.library.path=. -classpath ../lib/netty.jar:../lib/jnpserver.jar:../lib/jboss-mc.jar:../lib/jboss-jms-api.jar:../lib/hornetq-transports.jar:../lib/hornetq-logging.jar:../lib/hornetq-jms.jar:../lib/hornetq-jboss-as-integration.jar:../lib/hornetq-core.jar:../lib/hornetq-bootstrap.jar:../config/stand-alone/hqcluster/:../schemas/ org.hornetq.integration.bootstrap.HornetQBootstrapServer hornetq-beans.xml
***********************************************************************************
[main] 13:42:18,319 INFO [org.hornetq.integration.bootstrap.HornetQBootstrapServer] Starting HornetQ Server
[main] 13:42:20,112 INFO [org.hornetq.core.server.impl.HornetQServerImpl] live server is starting..
[main] 13:42:20,234 INFO [org.hornetq.core.persistence.impl.journal.JournalStorageManager] Using AIO Journal
[main] 13:42:20,295 WARNING [org.hornetq.core.server.impl.HornetQServerImpl] Security risk! It has been detected that the cluster admin user and password have not been changed from the installation default. Please see the HornetQ user guide, cluster chapter, for instructions on how to do this.
[main] 13:42:21,797 INFO [org.hornetq.integration.transports.netty.NettyAcceptor] Started Netty Acceptor version 3.1.5.GA-r1772
[main] 13:42:21,800 INFO [org.hornetq.core.server.impl.HornetQServerImpl] HornetQ Server version 2.0.0.GA (Hornet Queen, 113) started
[Thread-2 (group:HornetQ-server-threads31548547-9296787)] 13:42:26,353 INFO [org.hornetq.core.server.cluster.impl.BridgeImpl] Connecting bridge sf.my-cluster.80cfa78c-1137-11df-b5d3-080027b6f106 to its destination
[Thread-2 (group:HornetQ-server-threads31548547-9296787)] 13:42:26,463 INFO [org.hornetq.core.server.cluster.impl.BridgeImpl] Bridge sf.my-cluster.80cfa78c-1137-11df-b5d3-080027b6f106 is connected to its destination

^C[yingyuan@localhost bin]$ ./run.sh ../config/stand-alone/hqcluster/
***********************************************************************************
java -Xms32M -Xmx64M -Dhornetq.config.dir=../config/stand-alone/hqcluster/ -Djava.util.logging.config.file=../config/stand-alone/hqcluster//logging.properties -Djava.library.path=. -classpath ../lib/netty.jar:../lib/jnpserver.jar:../lib/jboss-mc.jar:../lib/jboss-jms-api.jar:../lib/hornetq-transports.jar:../lib/hornetq-logging.jar:../lib/hornetq-jms.jar:../lib/hornetq-jboss-as-integration.jar:../lib/hornetq-core.jar:../lib/hornetq-bootstrap.jar:../config/stand-alone/hqcluster/:../schemas/ org.hornetq.integration.bootstrap.HornetQBootstrapServer hornetq-beans.xml
***********************************************************************************
[main] 13:42:39,782 INFO [org.hornetq.integration.bootstrap.HornetQBootstrapServer] Starting HornetQ Server
[main] 13:42:41,592 INFO [org.hornetq.core.server.impl.HornetQServerImpl] live server is starting..
[main] 13:42:41,712 INFO [org.hornetq.core.persistence.impl.journal.JournalStorageManager] Using AIO Journal
[main] 13:42:41,772 WARNING [org.hornetq.core.server.impl.HornetQServerImpl] Security risk! It has been detected that the cluster admin user and password have not been changed from the installation default. Please see the HornetQ user guide, cluster chapter, for instructions on how to do this.
[main] 13:42:43,213 INFO [org.hornetq.integration.transports.netty.NettyAcceptor] Started Netty Acceptor version 3.1.5.GA-r1772
[main] 13:42:43,216 INFO [org.hornetq.core.server.impl.HornetQServerImpl] HornetQ Server version 2.0.0.GA (Hornet Queen, 113) started
[Thread-2 (group:HornetQ-server-threads31548547-9296787)] 13:42:46,363 INFO [org.hornetq.core.server.cluster.impl.BridgeImpl] Connecting bridge sf.my-cluster.80cfa78c-1137-11df-b5d3-080027b6f106 to its destination
[Thread-2 (group:HornetQ-server-threads31548547-9296787)] 13:42:46,498 INFO [org.hornetq.core.server.cluster.impl.BridgeImpl] Bridge sf.my-cluster.80cfa78c-1137-11df-b5d3-080027b6f106 is connected to its destination

//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

The second node command line and log:

[yingyuan@localhost bin]$ ./run.sh ../config/stand-alone/hqcluster/
***********************************************************************************
java -Xms32M -Xmx64M -Dhornetq.config.dir=../config/stand-alone/hqcluster/ -Djava.util.logging.config.file=../config/stand-alone/hqcluster//logging.properties -Djava.library.path=. -classpath ../lib/netty.jar:../lib/jnpserver.jar:../lib/jboss-mc.jar:../lib/jboss-jms-api.jar:../lib/hornetq-transports.jar:../lib/hornetq-logging.jar:../lib/hornetq-jms.jar:../lib/hornetq-jboss-as-integration.jar:../lib/hornetq-core.jar:../lib/hornetq-bootstrap.jar:../config/stand-alone/hqcluster/:../schemas/ org.hornetq.integration.bootstrap.HornetQBootstrapServer hornetq-beans.xml
***********************************************************************************
[main] 13:41:54,834 INFO [org.hornetq.integration.bootstrap.HornetQBootstrapServer] Starting HornetQ Server
[main] 13:41:56,754 INFO [org.hornetq.core.server.impl.HornetQServerImpl] live server is starting..
[main] 13:41:56,883 INFO [org.hornetq.core.persistence.impl.journal.JournalStorageManager] Using AIO Journal
[main] 13:41:56,951 WARNING [org.hornetq.core.server.impl.HornetQServerImpl] Security risk! It has been detected that the cluster admin user and password have not been changed from the installation default. Please see the HornetQ user guide, cluster chapter, for instructions on how to do this.
[main] 13:41:58,456 INFO [org.hornetq.integration.transports.netty.NettyAcceptor] Started Netty Acceptor version 3.1.5.GA-r1772
[main] 13:41:58,459 INFO [org.hornetq.core.server.impl.HornetQServerImpl] HornetQ Server version 2.0.0.GA (Hornet Queen, 113) started
[Thread-1 (group:HornetQ-server-threads31548547-9296787)] 13:41:59,874 INFO [org.hornetq.core.server.cluster.impl.BridgeImpl] Connecting bridge sf.my-cluster.d77230e6-1138-11df-b1e7-080027eaa684 to its destination
[Thread-1 (group:HornetQ-server-threads31548547-9296787)] 13:42:00,215 INFO [org.hornetq.core.server.cluster.impl.BridgeImpl] Bridge sf.my-cluster.d77230e6-1138-11df-b1e7-080027eaa684 is connected to its destination
[Thread-1 (group:HornetQ-client-global-threads-29290924)] 13:42:12,085 WARNING [org.hornetq.core.remoting.impl.RemotingConnectionImpl] Connection failure has been detected: The connection was disconnected because of server shutdown [code=4]
[Old I/O client worker (channelId: 27668940, /172.16.31.6:39869 => /172.16.31.5:5445)] 13:42:25,771 WARNING [org.hornetq.core.remoting.impl.ChannelImpl] Can't find packet to clear: last received command id 9 first stored command id 0
[Thread-1 (group:HornetQ-client-global-threads-29290924)] 13:42:36,321 WARNING [org.hornetq.core.remoting.impl.RemotingConnectionImpl] Connection failure has been detected: The connection was disconnected because of server shutdown [code=4]
[Old I/O client worker (channelId: 7200207, /172.16.31.6:39888 => /172.16.31.5:5445)] 13:42:47,185 WARNING [org.hornetq.core.remoting.impl.ChannelImpl] Can't find packet to clear: last received command id 9 first stored command id 0
[hornetq-discovery-group-thread-dg-group1] 13:42:50,026 WARNING [org.hornetq.core.cluster.impl.DiscoveryGroupImpl] There seem to be more than one broadcasters on the network broadcasting the same node id
[hornetq-discovery-group-thread-dg-group1] 13:42:55,026 WARNING [org.hornetq.core.cluster.impl.DiscoveryGroupImpl] There seem to be more than one broadcasters on the network broadcasting the same node id
[Thread-1 (group:HornetQ-client-global-threads-29290924)] 13:42:59,504 WARNING [org.hornetq.core.remoting.impl.RemotingConnectionImpl] Connection failure has been detected: The connection was disconnected because of server shutdown [code=4]

...

Did I mis-configure my nodes? I need HornetQ to reform current message system, so it is very important to me. Can anyone help me?

Thanks.

leo

hqcluster-node-b.tar.gz 2.6 KB
hqcluster-node-a.tar.gz 2.6 KB

1. Re: strange cluster problem:broadcasting the same node id

yingyuan.cheng Feb 6, 2010 11:18 AM (in response to yingyuan.cheng)

I patched one file, now it stops complaining after repeating 2 times. But the following WARN message still exists, does it really make significant?

[Old I/O client worker (channelId: <...Ignored here...>)] 23:55:30,338 WARNING [org.hornetq.core.remoting.impl.ChannelImpl] Can't find packet to clear: last received command id 9 first stored command id 0

--- hornetq-2.0.0.GA-src.org/src/main/org/hornetq/core/cluster/impl/DiscoveryGroupImpl.java 2010-01-13 10:59:14.000000000 +0800
+++ hornetq-2.0.0.GA-src/src/main/org/hornetq/core/cluster/impl/DiscoveryGroupImpl.java 2010-02-06 23:54:02.000000000 +0800
@@ -226,6 +226,7 @@
       String uniqueID;

       boolean changed;
+ int sameCounter;

       UniqueIDEntry(final String uniqueID)
       {
@@ -237,9 +238,9 @@
          return changed;
       }

-      void setChanged()
+      void setChanged(boolean changed)
       {
-         changed = true;
+         this.changed = changed;
       }

       String getUniqueID()
@@ -251,6 +252,16 @@
       {
          this.uniqueID = uniqueID;
       }
+
+ void setSameCounter(int sameCounter)
+ {
+ this.sameCounter = sameCounter;
+ }
+
+ int getSameCounter()
+ {
+ return this.sameCounter;
+ }
    }

    /*
@@ -273,6 +284,8 @@
       {
          if (entry.getUniqueID().equals(uniqueID))
          {
+ // reset counter to avoid being replaced
+ entry.setSameCounter(0);
             return true;
          }
          else
@@ -281,7 +294,8 @@
             // has same node id but different unique id
             if (!entry.isChanged())
             {
-               entry.setChanged();
+               entry.setChanged(true);
+    entry.setSameCounter(0);

                entry.setUniqueID(uniqueID);

@@ -289,6 +303,15 @@
             }
             else
             {
+ entry.setSameCounter(entry.getSameCounter()+1);
+
+ // replace stale uniqueID if happens more than 3 times
+ if (entry.getSameCounter() == 3) {
+ entry.setChanged(false);
+ entry.setUniqueID(uniqueID);
+ return true;
+ }
+
                return false;
             }
          }
@@ -336,7 +359,6 @@
             if (!uniqueIDOK(originatingNodeID, uniqueID))
             {
                DiscoveryGroupImpl.log.warn("There seem to be more than one broadcasters on the network broadcasting the same node id");
-
                continue;
             }
Actions
2. Re: strange cluster problem:broadcasting the same node id

timfox Feb 6, 2010 2:06 PM (in response to yingyuan.cheng)

Please post your config and someone will take a look
Actions
3. Re: strange cluster problem:broadcasting the same node id

yingyuan.cheng Feb 6, 2010 11:21 PM (in response to timfox)
Hi Tim.

I have posted my configuration files
hqcluster-node-b.tar.gz
hqcluster-node-a.tar.gz
Actions
4. Re: strange cluster problem:broadcasting the same node id

timfox Feb 7, 2010 2:56 AM (in response to yingyuan.cheng)

Can you post a JIRA please?
Actions
5. Re: strange cluster problem:broadcasting the same node id

yingyuan.cheng Feb 7, 2010 8:25 PM (in response to timfox)

I created one, https://jira.jboss.org/jira/browse/HORNETQ-291
Actions

Go to original post