Doesn't http://community.jboss.org/wiki/settingupaninfinispancluster work?  You just need to start up instances on > 1 server and they should discover each other.

Dear Manik\All,

Clustering different instances with same Server in Single Machine:

I was using hot rod server. Within one server, If I start different instances using the below command with different port numbers, Its working fine and detecting each other.

startServer.bat -l 10.38.54.210 -p 6904 -m 20 -t 20 -c ../etc/config-samples/minimal.xml -r hotrod -i 600 -n true -s 1024000 -e 1024000 -o 10.38.54.210 -x 6904
startServer.bat -l 10.38.54.210 -p 6905 -m 20 -t 20 -c ../etc/config-samples/minimal.xml -r hotrod -i 600 -n true -s 1024000 -e 1024000 -o 10.38.54.210 -x 6905

Log Info:

Start1

Start2

Start3

2010-08-16 13:45:36,037 INFO [HotRodOperation] (main) New topology: {/10.38.54.210:6905=5357, /10.38.54.210:6904=8872}

2010-08-16 13:45:36,037 INFO [TcpTransportFactory] (main) New server added(/10.38.54.210:6905), adding to the pool.

2010-08-16 13:45:36,037 INFO [TcpTransportFactory] (main) New server added(/10.38.54.210:6904), adding to the pool.

Start4

Able to form clustering with different instances of same server.

Hotrod properties:
infinispan.client.hotrod.server_list = 10.38.54.210:6905

Clustering different Server in different Machine: Log Info:

Whereas If I start one more server in different machine[ip -> 10.38.54.192 and port 6900] on the same network with below command, Its failing to detect the already running server instances in the other machine.[above mentioned servers running on port 6904 & 6905 running in ip 10.38.54.210]

startServer.bat -l 10.38.54.192 -p 6900 -m 20 -t 20 -c ../etc/config-samples/minimal.xml -r hotrod -i 600 -n true -s 1024000 -e 1024000 -o 10.38.54.192 -x 6900

Log Info:

2010-08-16 13:53:24,802 INFO [TestCache] (main) Inside TestCache

Start1

Start2

Start3

2010-08-16 13:53:24,880 INFO [HotRodOperation] (main) New topology: {/10.38.54.192:6900=7045}

2010-08-16 13:53:24,880 INFO [TcpTransportFactory] (main) New server added(/10.38.54.192:6900), adding to the pool.

2010-08-16 13:53:24,880 INFO [TcpTransportFactory] (main) Server not in cluster anymore(/10.38.54.210:6905), removing from the pool.

Start4

Its giving the info like server (/10.38.54.210:6905) not in cluster and its removig it from the pool but the server was started and running in the another machine but its failing to detect it.

Hotrod properties:

infinispan.client.hotrod.server_list = 10.38.54.210:6905;10.38.54.192:6900

Your issue does not appear to be related to Hot Rod. You seem to have communications issues between these machines. UDP communication should work between them. You should start testing this http://www.jgroups.org/manual/html/ch02.html#ItDoesntWork, check firewalls...etc.

Dear Galder,

As you said, there seems to be a communication issues between those machines. Thanks for the intimation. Some ports in those machines are closed. When we resolve that issue, it’s started working fine.

As part of understanding, how it’s forming a cluster. I have two questions. If time permits, kindly clarify me.

Machine 1: Server is started using below command

startServer.bat -l 192.168.1.7 -p 6904 -m 20 -t 20 -c ../etc/config-samples/minimal.xml -r hotrod -i 600 -n true -s 1024000 -e 1024000 -o 192.168.1.7 -x 6904

Machine 2: Server is started using below command

startServer.bat -l 192.168.1.14 -p 6905 -m 20 -t 20 -c ../etc/config-samples/minimal.xml -r hotrod -i 600 -n true -s 1024000 -e 1024000 -o 192.168.1.14 -x 6905

Question1: [How hotrod Servers detecting each other and forming a Cluster?]

When you infer both these above commands, we are not mentioning the other machine identity to which hotrod server has to form a cluster. As per my understanding, since both the servers using the same minimal.xml, which in turn uses same jgroups-udp.xml. They form a cluster based on the same mcast_addr and same mcast_port mentioned in the jgroups-udp.xml. Please let me know, whether my understanding is correct. If I miss understand something, how the hotrod servers detecting each other. Kindly clarify it.

Question2: [Each time you start the server, Hotrod servers are connecting to dynamic port of the other machine and forming a cluster. Can we able to fix the ports that the Hotrod server has to connect, instead of connecting to a dynamic port?]

In the first machine, I have used 'netstat -ano' command. I got the below result

Proto  Local Address              Foreign Address        State                  PID

TCP    192.168.1.7:3155       192.168.1.14:4679      ESTABLISHED     3852

In the second machine, I have used 'netstat -ano' command. I got the below result

Proto  Local Address              Foreign Address        State                  PID

TCP    192.168.1.14:4679      192.168.1.7:3155       ESTABLISHED     11080

From the above two results, we can able to see that they connect to dynamic ports [3155 & 4679], which are not specified anywhere by us. Why this is happening. Is there a way I can able to fix those connecting ports or Will the ports be always decided by Jgroups Transport ? Kindly clarify.

My advance thanks for the same.

Thanks & Regards,

Savin

Re Q1: Correct. They see each other thanks to messages they sent to a particular mcast addr+port.

Re Q2: Most likely, that TCP connection is the one belonging to JGroups' FD_SOCK protocol. Set start_port as per: http://community.jboss.org/wiki/JGroupsFDSOCK

Hi,

The example of GUIDemo of Infinispan shows data accessibility among  inter clusters, created on starting/running multiple instances. So which mode of Clustering this example uses? Is it Distributed mode/Local/mode.. ? Also what technique is used in maintaining the cache here?

Thanks n Regards,

Vinay

The mode is defined by the config file it uses, see etc/gui-demo-cache-config.xml : https://github.com/infinispan/infinispan/blob/master/demos/gui/src/main/resources/config-samples/gui-demo-cache-config.xml

For the example of GUIDemo of Infinispan, what technique is used in maintaining the cache across all clusters..how is it working?

How is the caching happening here?

Hi Savin,

May I know why HotRod ports needs to be different even they run on complete different machine and JVM?

6904 and 6905?

I tried using same port it didn't worked. I need to change the port to make it run  but I am not clear why the ports should be unique?

Can anyone help me why the ports should be unique?

Ports on different machines or bound to different interfaces/addresses of the same machine can definitely be the same. When you say "it didn't work" can you provide a stack trace ?

inifinispan 9.1.1

jgroups-4.0.6

Infinispan :

<?xml version="1.0" encoding="UTF-8"?>

<infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

xsi:schemaLocation="urn:infinispan:config:9.1 http://www.infinispan.org/schemas/infinispan-config-9.1.xsd"

xmlns="urn:infinispan:config:9.1">

<jgroups>

<stack-file name="external-file" path="jgroups-tcp-${spring.profiles.active}.xml" />

</jgroups>

<cache-container default-cache="fuseCache">

<transport stack="external-file" />

<distributed-cache name="fuseCache" mode="SYNC" start="EAGER">

<locking isolation="READ_COMMITTED" acquire-timeout="30000" concurrency-level="1000" striping="false"/>

</distributed-cache>

</cache-container>

</infinispan>

jgroups-tcp-Test

<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

xmlns="urn:org:jgroups" xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups.xsd">

<TCP bind_port="7801" bind_addr="SITE_LOCAL" />

<TCPPING async_discovery="true" initial_hosts="${jgroups.tcpping.initial_hosts:hostA[7801],hostB[7801]}}" port_range="2" />

<MERGE3 min_interval="10000" max_interval="30000" />

<FD_SOCK />

<FD timeout="3000" max_tries="3" />

<VERIFY_SUSPECT timeout="1500" />

<BARRIER />

<pbcast.NAKACK2 use_mcast_xmit="false" discard_delivered_msgs="true" />

<UNICAST3 />

<pbcast.STABLE desired_avg_gossip="50000" max_bytes="4M" />

<pbcast.GMS print_local_addr="true" join_timeout="2000" />

<MFC max_credits="2M" min_threshold="0.4" />

<FRAG2 frag_size="60K" />

<!--RSVP resend_interval="2000" timeout="10000"/ -->

<pbcast.STATE_TRANSFER />

</config>

StackTrace:

{

  "timestamp" : "2017-11-14T10:43:32.912+10:00",

  "message" : "ISPN000136: Error executing command PutKeyValueCommand, writing keys [HostA-19961]",

  "thread" : "jgroups-23,ISPN,HostA-19961",

  "level" : "ERROR",

  "stack_trace" : "org.infinispan.remoting.RemoteException: ISPN000217: Received exception from HostB-21393, see cause for remote stack trace\n\tat org.infinispan.remoting.transport.ResponseCollectors.wrapRemoteException(ResponseCollectors.java:27)\n\tat org.infinispan.remoting.transport.jgroups.SyncMapResponseCollector.addException(SyncMapResponseCollector.java:48)\n\tat org.infinispan.remoting.transport.jgroups.SyncMapResponseCollector.addException(SyncMapResponseCollector.java:26)\n\tat org.infinispan.remoting.transport.ValidResponseCollector.addResponse(ValidResponseCollector.java:29)\n\tat org.infinispan.remoting.transport.impl.MultiTargetRequest.onResponse(MultiTargetRequest.java:91)\n\tat org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:53)\n\tat org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1328)\n\tat org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1238)\n\tat org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$200(JGroupsTransport.java:121)\n\tat org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.receive(JGroupsTransport.java:1366)\n\tat org.jgroups.JChannel.up(JChannel.java:819)\n\tat org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:893)\n\tat org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:128)\n\tat org.jgroups.protocols.FRAG2.up(FRAG2.java:177)\n\tat org.jgroups.protocols.FlowControl.up(FlowControl.java:343)\n\tat org.jgroups.protocols.pbcast.GMS.up(GMS.java:864)\n\tat org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:240)\n\tat org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1002)\n\tat org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:728)\n\tat org.jgroups.protocols.UNICAST3.up(UNICAST3.java:383)\n\tat org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)\n\tat org.jgroups.protocols.BARRIER.up(BARRIER.java:170)\n\tat org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:119)\n\tat org.jgroups.protocols.FD.up(FD.java:212)\n\tat org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:252)\n\tat org.jgroups.protocols.MERGE3.up(MERGE3.java:276)\n\tat org.jgroups.protocols.Discovery.up(Discovery.java:262)\n\tat org.jgroups.protocols.TP.passMessageUp(TP.java:1229)\n\tat org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat java.lang.Thread.run(Thread.java:745)\n\tSuppressed: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from HostA-14530, see cause for remote stack trace\n\t\t... 32 common frames omitted\n\tCaused by: java.lang.NullPointerException: null\n\t\tat org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:694)\n\t\tat org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:679)\n\t\tat org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:361)\n\t\tat org.infinispan.marshall.core.BytesObjectInput.readObject(BytesObjectInput.java:40)\n\t\tat org.infinispan.commands.write.PutKeyValueCommand.readFrom(PutKeyValueCommand.java:121)\n\t\tat org.infinispan.marshall.exts.ReplicableCommandExternalizer.readCommandParameters(ReplicableCommandExternalizer.java:103)\n\t\tat org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:92)\n\t\tat org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:54)\n\t\tat org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:694)\n\t\tat org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:677)\n\t\tat org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:361)\n\t\tat org.infinispan.marshall.core.BytesObjectInput.readObject(BytesObjectInput.java:40)\n\t\tat org.infinispan.commands.remote.SingleRpcCommand.readFrom(SingleRpcCommand.java:47)\n\t\tat org.infinispan.marshall.exts.ReplicableCommandExternalizer.readCommandParameters(ReplicableCommandExternalizer.java:103)\n\t\tat org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:116)\n\t\tat org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:57)\n\t\tat org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:694)\n\t\tat org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:677)\n\t\tat org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:361)\n\t\tat org.infinispan.marshall.core.GlobalMarshaller.objectFromObjectInput(GlobalMarshaller.java:199)\n\t\tat org.infinispan.marshall.core.GlobalMarshaller.objectFromByteBuffer(GlobalMarshaller.java:228)\n\t\tat org.infinispan.remoting.transport.jgroups.JGroupsTransport.processRequest(JGroupsTransport.java:1288)\n\t\tat org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1235)\n\t\tat org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$200(JGroupsTransport.java:121)\n\t\tat org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.receive(JGroupsTransport.java:1366)\n\t\tat org.jgroups.JChannel.up(JChannel.java:819)\n\t\tat org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:893)\n\t\tat org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:128)\n\t\tat org.jgroups.protocols.FRAG2.up(FRAG2.java:177)\n\t\tat org.jgroups.protocols.FlowControl.up(FlowControl.java:359)\n\t\tat org.jgroups.protocols.pbcast.GMS.up(GMS.java:867)\n\t\tat org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:240)\n\t\tat org.jgroups.protocols.UNICAST3.up(UNICAST3.java:369)\n\t\tat org.jgroups.protocols.pbcast.NAKACK2.deliver(NAKACK2.java:941)\n\t\tat org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:830)\n\t\tat org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:610)\n\t\tat org.jgroups.protocols.BARRIER.up(BARRIER.java:170)\n\t\tat org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:129)\n\t\t... 9 common frames omitted\nCaused by: java.lang.NullPointerException: null\n\tat org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:694)\n\tat org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:679)\n\tat org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:361)\n\tat org.infinispan.marshall.core.BytesObjectInput.readObject(BytesObjectInput.java:40)\n\tat org.infinispan.commands.write.PutKeyValueCommand.readFrom(PutKeyValueCommand.java:121)\n\tat org.infinispan.marshall.exts.ReplicableCommandExternalizer.readCommandParameters(ReplicableCommandExternalizer.java:103)\n\tat org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:92)\n\tat org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:54)\n\tat org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:694)\n\tat org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:677)\n\tat org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:361)\n\tat org.infinispan.marshall.core.BytesObjectInput.readObject(BytesObjectInput.java:40)\n\tat org.infinispan.commands.remote.SingleRpcCommand.readFrom(SingleRpcCommand.java:47)\n\tat org.infinispan.marshall.exts.ReplicableCommandExternalizer.readCommandParameters(ReplicableCommandExternalizer.java:103)\n\tat org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:116)\n\tat org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:57)\n\tat org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:694)\n\tat org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:677)\n\tat org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:361)\n\tat org.infinispan.marshall.core.GlobalMarshaller.objectFromObjectInput(GlobalMarshaller.java:199)\n\tat org.infinispan.marshall.core.GlobalMarshaller.objectFromByteBuffer(GlobalMarshaller.java:228)\n\tat org.infinispan.remoting.transport.jgroups.JGroupsTransport.processRequest(JGroupsTransport.java:1288)\n\tat org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1235)\n\tat org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$200(JGroupsTransport.java:121)\n\tat org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.receive(JGroupsTransport.java:1366)\n\tat org.jgroups.JChannel.up(JChannel.java:819)\n\tat org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:893)\n\tat org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:128)\n\tat org.jgroups.protocols.FRAG2.up(FRAG2.java:177)\n\tat org.jgroups.protocols.FlowControl.up(FlowControl.java:359)\n\tat org.jgroups.protocols.pbcast.GMS.up(GMS.java:867)\n\tat org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:240)\n\tat org.jgroups.protocols.UNICAST3.up(UNICAST3.java:369)\n\tat org.jgroups.protocols.pbcast.NAKACK2.deliver(NAKACK2.java:941)\n\tat org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:830)\n\tat org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:610)\n\tat org.jgroups.protocols.BARRIER.up(BARRIER.java:170)\n\tat org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:129)\n\t... 9 common frames omitted\n",

  "hostname" : "HostA.tattsnonprod.com",

  "appName" : "fuse-shared-cache"

}