-
1. Re: How to deploy infinispan cluster across several servers????
manik Aug 11, 2010 7:38 AM (in response to llq_9)Doesn't http://community.jboss.org/wiki/settingupaninfinispancluster work? You just need to start up instances on > 1 server and they should discover each other.
-
2. Re: How to deploy infinispan cluster across several servers????
savin7 Aug 16, 2010 6:31 AM (in response to manik)Dear Manik\All,
Clustering different instances with same Server in Single Machine:
I was using hot rod server. Within one server, If I start different instances using the below command with different port numbers, Its working fine and detecting each other.
startServer.bat -l 10.38.54.210 -p 6904 -m 20 -t 20 -c ../etc/config-samples/minimal.xml -r hotrod -i 600 -n true -s 1024000 -e 1024000 -o 10.38.54.210 -x 6904
startServer.bat -l 10.38.54.210 -p 6905 -m 20 -t 20 -c ../etc/config-samples/minimal.xml -r hotrod -i 600 -n true -s 1024000 -e 1024000 -o 10.38.54.210 -x 6905Log Info:
Start1
Start2
Start3
2010-08-16 13:45:36,037 INFO [HotRodOperation] (main) New topology: {/10.38.54.210:6905=5357, /10.38.54.210:6904=8872}
2010-08-16 13:45:36,037 INFO [TcpTransportFactory] (main) New server added(/10.38.54.210:6905), adding to the pool.
2010-08-16 13:45:36,037 INFO [TcpTransportFactory] (main) New server added(/10.38.54.210:6904), adding to the pool.
Start4
Able to form clustering with different instances of same server.
Hotrod properties:
infinispan.client.hotrod.server_list = 10.38.54.210:6905Clustering different Server in different Machine: Log Info:
Whereas If I start one more server in different machine[ip -> 10.38.54.192 and port 6900] on the same network with below command, Its failing to detect the already running server instances in the other machine.[above mentioned servers running on port 6904 & 6905 running in ip 10.38.54.210]
startServer.bat -l 10.38.54.192 -p 6900 -m 20 -t 20 -c ../etc/config-samples/minimal.xml -r hotrod -i 600 -n true -s 1024000 -e 1024000 -o 10.38.54.192 -x 6900
Log Info:
2010-08-16 13:53:24,802 INFO [TestCache] (main) Inside TestCache
Start1
Start2
Start3
2010-08-16 13:53:24,880 INFO [HotRodOperation] (main) New topology: {/10.38.54.192:6900=7045}
2010-08-16 13:53:24,880 INFO [TcpTransportFactory] (main) New server added(/10.38.54.192:6900), adding to the pool.
2010-08-16 13:53:24,880 INFO [TcpTransportFactory] (main) Server not in cluster anymore(/10.38.54.210:6905), removing from the pool.
Start4
Its giving the info like server (/10.38.54.210:6905) not in cluster and its removig it from the pool but the server was started and running in the another machine but its failing to detect it.
Hotrod properties:
infinispan.client.hotrod.server_list = 10.38.54.210:6905;10.38.54.192:6900
Attached are the files that I have used for clustering. Please let me know, Is there I have to change any configurations in the attached files to achieve clustering hotrod server in different machines. So that it can detect the server that is running in another machine.
My advance thanks for the same.
Thanks & Regards,
Savin
-
TestCache.java.zip 602 bytes
-
hotrod-client.properties.zip 690 bytes
-
jgroups-udp.xml.zip 1.1 KB
-
minimal.xml 1.1 KB
-
-
3. Re: How to deploy infinispan cluster across several servers????
galder.zamarreno Aug 19, 2010 6:27 AM (in response to savin7)Your issue does not appear to be related to Hot Rod. You seem to have communications issues between these machines. UDP communication should work between them. You should start testing this http://www.jgroups.org/manual/html/ch02.html#ItDoesntWork, check firewalls...etc.
-
4. Re: How to deploy infinispan cluster across several servers????
savin7 Aug 25, 2010 5:18 AM (in response to galder.zamarreno)Dear Galder,
As you said, there seems to be a communication issues between those machines. Thanks for the intimation. Some ports in those machines are closed. When we resolve that issue, it’s started working fine.
As part of understanding, how it’s forming a cluster. I have two questions. If time permits, kindly clarify me.
Machine 1: Server is started using below command
startServer.bat -l 192.168.1.7 -p 6904 -m 20 -t 20 -c ../etc/config-samples/minimal.xml -r hotrod -i 600 -n true -s 1024000 -e 1024000 -o 192.168.1.7 -x 6904
Machine 2: Server is started using below command
startServer.bat -l 192.168.1.14 -p 6905 -m 20 -t 20 -c ../etc/config-samples/minimal.xml -r hotrod -i 600 -n true -s 1024000 -e 1024000 -o 192.168.1.14 -x 6905
Question1: [How hotrod Servers detecting each other and forming a Cluster?]
When you infer both these above commands, we are not mentioning the other machine identity to which hotrod server has to form a cluster. As per my understanding, since both the servers using the same minimal.xml, which in turn uses same jgroups-udp.xml. They form a cluster based on the same mcast_addr and same mcast_port mentioned in the jgroups-udp.xml. Please let me know, whether my understanding is correct. If I miss understand something, how the hotrod servers detecting each other. Kindly clarify it.
Question2: [Each time you start the server, Hotrod servers are connecting to dynamic port of the other machine and forming a cluster. Can we able to fix the ports that the Hotrod server has to connect, instead of connecting to a dynamic port?]
In the first machine, I have used 'netstat -ano' command. I got the below result
Proto Local Address Foreign Address State PID
TCP 192.168.1.7:3155 192.168.1.14:4679 ESTABLISHED 3852
In the second machine, I have used 'netstat -ano' command. I got the below result
Proto Local Address Foreign Address State PID
TCP 192.168.1.14:4679 192.168.1.7:3155 ESTABLISHED 11080
From the above two results, we can able to see that they connect to dynamic ports [3155 & 4679], which are not specified anywhere by us. Why this is happening. Is there a way I can able to fix those connecting ports or Will the ports be always decided by Jgroups Transport ? Kindly clarify.
My advance thanks for the same.
Thanks & Regards,
Savin
-
5. Re: How to deploy infinispan cluster across several servers????
galder.zamarreno Aug 30, 2010 11:32 AM (in response to savin7)Re Q1: Correct. They see each other thanks to messages they sent to a particular mcast addr+port.
Re Q2: Most likely, that TCP connection is the one belonging to JGroups' FD_SOCK protocol. Set start_port as per: http://community.jboss.org/wiki/JGroupsFDSOCK
-
6. How to deploy infinispan cluster across several servers????
vinayananth07 Feb 1, 2011 4:53 AM (in response to manik)Hi,
The example of GUIDemo of Infinispan shows data accessibility among inter clusters, created on starting/running multiple instances. So which mode of Clustering this example uses? Is it Distributed mode/Local/mode.. ? Also what technique is used in maintaining the cache here?
Thanks n Regards,
Vinay
-
7. How to deploy infinispan cluster across several servers????
galder.zamarreno Feb 4, 2011 10:24 AM (in response to vinayananth07)The mode is defined by the config file it uses, see etc/gui-demo-cache-config.xml : https://github.com/infinispan/infinispan/blob/master/demos/gui/src/main/resources/config-samples/gui-demo-cache-config.xml
-
8. How to deploy infinispan cluster across several servers????
vinayananth07 Feb 7, 2011 4:53 AM (in response to galder.zamarreno)For the example of GUIDemo of Infinispan, what technique is used in maintaining the cache across all clusters..how is it working?
How is the caching happening here?
-
9. Re: How to deploy infinispan cluster across several servers????
sarada.ch Nov 13, 2017 4:47 AM (in response to savin7)Hi Savin,
May I know why HotRod ports needs to be different even they run on complete different machine and JVM?
6904 and 6905?
I tried using same port it didn't worked. I need to change the port to make it run but I am not clear why the ports should be unique?
Can anyone help me why the ports should be unique?
-
10. Re: How to deploy infinispan cluster across several servers????
nadirx Nov 13, 2017 6:35 AM (in response to sarada.ch)Ports on different machines or bound to different interfaces/addresses of the same machine can definitely be the same. When you say "it didn't work" can you provide a stack trace ?
-
11. Re: How to deploy infinispan cluster across several servers????
sarada.ch Nov 14, 2017 4:55 AM (in response to nadirx)inifinispan 9.1.1
jgroups-4.0.6
Infinispan :
<?xml version="1.0" encoding="UTF-8"?>
<infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:9.1 http://www.infinispan.org/schemas/infinispan-config-9.1.xsd"
xmlns="urn:infinispan:config:9.1">
<jgroups>
<stack-file name="external-file" path="jgroups-tcp-${spring.profiles.active}.xml" />
</jgroups>
<cache-container default-cache="fuseCache">
<transport stack="external-file" />
<distributed-cache name="fuseCache" mode="SYNC" start="EAGER">
<locking isolation="READ_COMMITTED" acquire-timeout="30000" concurrency-level="1000" striping="false"/>
</distributed-cache>
</cache-container>
</infinispan>
jgroups-tcp-Test
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="urn:org:jgroups" xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups.xsd">
<TCP bind_port="7801" bind_addr="SITE_LOCAL" />
<TCPPING async_discovery="true" initial_hosts="${jgroups.tcpping.initial_hosts:hostA[7801],hostB[7801]}}" port_range="2" />
<MERGE3 min_interval="10000" max_interval="30000" />
<FD_SOCK />
<FD timeout="3000" max_tries="3" />
<VERIFY_SUSPECT timeout="1500" />
<BARRIER />
<pbcast.NAKACK2 use_mcast_xmit="false" discard_delivered_msgs="true" />
<UNICAST3 />
<pbcast.STABLE desired_avg_gossip="50000" max_bytes="4M" />
<pbcast.GMS print_local_addr="true" join_timeout="2000" />
<MFC max_credits="2M" min_threshold="0.4" />
<FRAG2 frag_size="60K" />
<!--RSVP resend_interval="2000" timeout="10000"/ -->
<pbcast.STATE_TRANSFER />
</config>
StackTrace:
{
"timestamp" : "2017-11-14T10:43:32.912+10:00",
"message" : "ISPN000136: Error executing command PutKeyValueCommand, writing keys [HostA-19961]",
"thread" : "jgroups-23,ISPN,HostA-19961",
"level" : "ERROR",
"stack_trace" : "org.infinispan.remoting.RemoteException: ISPN000217: Received exception from HostB-21393, see cause for remote stack trace\n\tat org.infinispan.remoting.transport.ResponseCollectors.wrapRemoteException(ResponseCollectors.java:27)\n\tat org.infinispan.remoting.transport.jgroups.SyncMapResponseCollector.addException(SyncMapResponseCollector.java:48)\n\tat org.infinispan.remoting.transport.jgroups.SyncMapResponseCollector.addException(SyncMapResponseCollector.java:26)\n\tat org.infinispan.remoting.transport.ValidResponseCollector.addResponse(ValidResponseCollector.java:29)\n\tat org.infinispan.remoting.transport.impl.MultiTargetRequest.onResponse(MultiTargetRequest.java:91)\n\tat org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:53)\n\tat org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1328)\n\tat org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1238)\n\tat org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$200(JGroupsTransport.java:121)\n\tat org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.receive(JGroupsTransport.java:1366)\n\tat org.jgroups.JChannel.up(JChannel.java:819)\n\tat org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:893)\n\tat org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:128)\n\tat org.jgroups.protocols.FRAG2.up(FRAG2.java:177)\n\tat org.jgroups.protocols.FlowControl.up(FlowControl.java:343)\n\tat org.jgroups.protocols.pbcast.GMS.up(GMS.java:864)\n\tat org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:240)\n\tat org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1002)\n\tat org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:728)\n\tat org.jgroups.protocols.UNICAST3.up(UNICAST3.java:383)\n\tat org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)\n\tat org.jgroups.protocols.BARRIER.up(BARRIER.java:170)\n\tat org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:119)\n\tat org.jgroups.protocols.FD.up(FD.java:212)\n\tat org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:252)\n\tat org.jgroups.protocols.MERGE3.up(MERGE3.java:276)\n\tat org.jgroups.protocols.Discovery.up(Discovery.java:262)\n\tat org.jgroups.protocols.TP.passMessageUp(TP.java:1229)\n\tat org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat java.lang.Thread.run(Thread.java:745)\n\tSuppressed: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from HostA-14530, see cause for remote stack trace\n\t\t... 32 common frames omitted\n\tCaused by: java.lang.NullPointerException: null\n\t\tat org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:694)\n\t\tat org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:679)\n\t\tat org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:361)\n\t\tat org.infinispan.marshall.core.BytesObjectInput.readObject(BytesObjectInput.java:40)\n\t\tat org.infinispan.commands.write.PutKeyValueCommand.readFrom(PutKeyValueCommand.java:121)\n\t\tat org.infinispan.marshall.exts.ReplicableCommandExternalizer.readCommandParameters(ReplicableCommandExternalizer.java:103)\n\t\tat org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:92)\n\t\tat org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:54)\n\t\tat org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:694)\n\t\tat org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:677)\n\t\tat org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:361)\n\t\tat org.infinispan.marshall.core.BytesObjectInput.readObject(BytesObjectInput.java:40)\n\t\tat org.infinispan.commands.remote.SingleRpcCommand.readFrom(SingleRpcCommand.java:47)\n\t\tat org.infinispan.marshall.exts.ReplicableCommandExternalizer.readCommandParameters(ReplicableCommandExternalizer.java:103)\n\t\tat org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:116)\n\t\tat org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:57)\n\t\tat org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:694)\n\t\tat org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:677)\n\t\tat org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:361)\n\t\tat org.infinispan.marshall.core.GlobalMarshaller.objectFromObjectInput(GlobalMarshaller.java:199)\n\t\tat org.infinispan.marshall.core.GlobalMarshaller.objectFromByteBuffer(GlobalMarshaller.java:228)\n\t\tat org.infinispan.remoting.transport.jgroups.JGroupsTransport.processRequest(JGroupsTransport.java:1288)\n\t\tat org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1235)\n\t\tat org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$200(JGroupsTransport.java:121)\n\t\tat org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.receive(JGroupsTransport.java:1366)\n\t\tat org.jgroups.JChannel.up(JChannel.java:819)\n\t\tat org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:893)\n\t\tat org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:128)\n\t\tat org.jgroups.protocols.FRAG2.up(FRAG2.java:177)\n\t\tat org.jgroups.protocols.FlowControl.up(FlowControl.java:359)\n\t\tat org.jgroups.protocols.pbcast.GMS.up(GMS.java:867)\n\t\tat org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:240)\n\t\tat org.jgroups.protocols.UNICAST3.up(UNICAST3.java:369)\n\t\tat org.jgroups.protocols.pbcast.NAKACK2.deliver(NAKACK2.java:941)\n\t\tat org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:830)\n\t\tat org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:610)\n\t\tat org.jgroups.protocols.BARRIER.up(BARRIER.java:170)\n\t\tat org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:129)\n\t\t... 9 common frames omitted\nCaused by: java.lang.NullPointerException: null\n\tat org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:694)\n\tat org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:679)\n\tat org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:361)\n\tat org.infinispan.marshall.core.BytesObjectInput.readObject(BytesObjectInput.java:40)\n\tat org.infinispan.commands.write.PutKeyValueCommand.readFrom(PutKeyValueCommand.java:121)\n\tat org.infinispan.marshall.exts.ReplicableCommandExternalizer.readCommandParameters(ReplicableCommandExternalizer.java:103)\n\tat org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:92)\n\tat org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:54)\n\tat org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:694)\n\tat org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:677)\n\tat org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:361)\n\tat org.infinispan.marshall.core.BytesObjectInput.readObject(BytesObjectInput.java:40)\n\tat org.infinispan.commands.remote.SingleRpcCommand.readFrom(SingleRpcCommand.java:47)\n\tat org.infinispan.marshall.exts.ReplicableCommandExternalizer.readCommandParameters(ReplicableCommandExternalizer.java:103)\n\tat org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:116)\n\tat org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:57)\n\tat org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:694)\n\tat org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:677)\n\tat org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:361)\n\tat org.infinispan.marshall.core.GlobalMarshaller.objectFromObjectInput(GlobalMarshaller.java:199)\n\tat org.infinispan.marshall.core.GlobalMarshaller.objectFromByteBuffer(GlobalMarshaller.java:228)\n\tat org.infinispan.remoting.transport.jgroups.JGroupsTransport.processRequest(JGroupsTransport.java:1288)\n\tat org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1235)\n\tat org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$200(JGroupsTransport.java:121)\n\tat org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.receive(JGroupsTransport.java:1366)\n\tat org.jgroups.JChannel.up(JChannel.java:819)\n\tat org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:893)\n\tat org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:128)\n\tat org.jgroups.protocols.FRAG2.up(FRAG2.java:177)\n\tat org.jgroups.protocols.FlowControl.up(FlowControl.java:359)\n\tat org.jgroups.protocols.pbcast.GMS.up(GMS.java:867)\n\tat org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:240)\n\tat org.jgroups.protocols.UNICAST3.up(UNICAST3.java:369)\n\tat org.jgroups.protocols.pbcast.NAKACK2.deliver(NAKACK2.java:941)\n\tat org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:830)\n\tat org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:610)\n\tat org.jgroups.protocols.BARRIER.up(BARRIER.java:170)\n\tat org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:129)\n\t... 9 common frames omitted\n",
"hostname" : "HostA.tattsnonprod.com",
"appName" : "fuse-shared-cache"
}