7 Replies Latest reply: Sep 6, 2012 12:12 AM by Subash Chaturanga RSS

Range of ports infinispan accepts by default: replication timeout happens due to port blocking

Subash Chaturanga Newbie

Hi all,

 

I came across with this issue, where infinispan cache replication time outs.

Here is the error log I got. Seems this ports got blocked, and hence time outs. "Replication timeout for blx25ao03-38795"

So what is the default range of ports that infinispan supports or is it possible to change the range of ports ?

In fact I looked at the jgroups-tcp and udp xmls. There in  jgroups-tcp.xml I saw an attribute called, port_range="30".(I didn't see such for jgroups-udp.xml).

If this is this range what is the initial port value. Then  range should be( initvalue - initivalue+30). Isn't it ?

Here I am assuming 38795 is the port which hangs, according to logs. Correct me if I have read the logs incorrectly.

I need to identify the udp/tcp port range infinispan accepts to get rid from this issue. Appreciate any help on this?



org.infinispan.util.concurrent.TimeoutException: Replication timeout for blx25ao03-38795

 

    at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:421)

    at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:100)

    at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:124)

    at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229)

    at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:217)

    at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:199)

    at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:193)

    at org.infinispan.interceptors.ReplicationInterceptor.handleCrudMethod(ReplicationInterceptor.java:114)

    at org.infinispan.interceptors.ReplicationInterceptor.visitPutKeyValueCommand(ReplicationInterceptor.java:78)

    at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)

    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

    at org.infinispan.interceptors.LockingInterceptor.visitPutKeyValueCommand(LockingInterceptor.java:198)

    at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)

    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

    at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)

    at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)

    at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)

    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

    at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:183)

    at org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:132)

    at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)

    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

  • 1. Re: Range of ports infinispan accepts by default: replication timeout happens due to port blocking
    Mircea Markus Master

    Can you post the logs from all the servers?

    What version of ISPN are you using?

  • 2. Re: Range of ports infinispan accepts by default: replication timeout happens due to port blocking
    Subash Chaturanga Newbie

    Hi Markus,

    Thanks for the quick response. The version of infinispan I use is 4.1.0.CR2.

    Actually the logs are same. This is a scenario which has two identical servers deployed. From each server I am getting this.

     

    After digging deep, the issue seems blocking the mentioned port from a firewall. If I can get to know the port range what infinispan uses, I can try my end opening those ports

    and check whether this issue still reproduces.

  • 3. Re: Range of ports infinispan accepts by default: replication timeout happens due to port blocking
    Mircea Markus Master

    The last Final ISPN release is 5.1.6.FInal, you might want to conisder upgrading to that.

    I doubt it that it is a port issue, as if at that stage ISPN would only do an RPC if it could contact the other members of the cluster.

    The port should be present in the jgroups configuration file.

  • 4. Re: Range of ports infinispan accepts by default: replication timeout happens due to port blocking
    Subash Chaturanga Newbie

    Hi Markus,

    For the moment I can't move to the new version.

    Following is the jgroups-tcp.xml I used. Can  you plase tell mew how this port range suppose to work ? i..e

    Here port_range =30  bind port 7800.(hence range should be 7800-7830 ?? or Am I doing something wrong ?)

    If so, my problem is how can then it timeouts on port 38795 (I have bold this ort number in the initial error log)

     

     

     

     

    <config xmlns="urn:org:jgroups"

            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

            xsi:schemaLocation="urn:org:jgroups file:schema/JGroups-2.8.xsd">

       <TCP bind_port="7800"

            loopback="true"

            port_range="30"

            recv_buf_size="20000000"

            send_buf_size="640000"

            discard_incompatible_packets="true"

            max_bundle_size="64000"

            max_bundle_timeout="30"

            enable_bundling="true"

            use_send_queues="true"

            sock_conn_timeout="300"

            enable_diagnostics="false"

     

     

            thread_pool.enabled="true"

            thread_pool.min_threads="2"

            thread_pool.max_threads="30"

            thread_pool.keep_alive_time="5000"

            thread_pool.queue_enabled="false"

            thread_pool.queue_max_size="100"

            thread_pool.rejection_policy="Discard"

     

     

            oob_thread_pool.enabled="true"

            oob_thread_pool.min_threads="2"

            oob_thread_pool.max_threads="30"

            oob_thread_pool.keep_alive_time="5000"

            oob_thread_pool.queue_enabled="false"

            oob_thread_pool.queue_max_size="100"

            oob_thread_pool.rejection_policy="Discard"       

             />

     

     

       <!--<TCPPING timeout="3000"-->

                <!--initial_hosts="localhost[7800],localhost[7801]}"-->

                <!--port_range="5"-->

                <!--num_initial_members="3"/>-->

     

     

       <MPING bind_addr="${jgroups.bind_addr:127.0.0.1}" break_on_coord_rsp="true"

          mcast_addr="${jgroups.udp.mcast_addr:228.6.7.8}" mcast_port="${jgroups.udp.mcast_port:46655}" ip_ttl="${jgroups.udp.ip_ttl:2}"

          num_initial_members="3"/>

     

     

       <MERGE2 max_interval="30000"

               min_interval="10000"/>

       <FD_SOCK/>

       <FD timeout="3000" max_tries="3"/>

       <VERIFY_SUSPECT timeout="1500"/>

       <pbcast.NAKACK

             use_mcast_xmit="false" gc_lag="0"

             retransmit_timeout="300,600,1200,2400,4800"

             discard_delivered_msgs="false"/>

       <UNICAST timeout="300,600,1200"/>

       <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"

                      max_bytes="400000"/>

       <pbcast.GMS print_local_addr="false" join_timeout="7000" view_bundling="true"/>

       <FC max_credits="2000000"

           min_threshold="0.10"/>

       <FRAG2 frag_size="60000"/>

       <pbcast.STREAMING_STATE_TRANSFER/>

       <!-- <pbcast.STATE_TRANSFER/> -->

       <pbcast.FLUSH timeout="0"/>

    </config>

  • 5. Re: Range of ports infinispan accepts by default: replication timeout happens due to port blocking
    Subash Chaturanga Newbie

    Hi Markus,

    So as I am reproducing this I wonder if I increase timeouts would help this. If so, what are the possible timeouts that should be increased to overcome this timeout issue (there are multiple timeouts available) ?

    In other words If I change that  through system properties, what are the properties need to be added . Here [1] are the system props I found. But no timeout related stuff there.

     

    [1] - https://community.jboss.org/wiki/SystemProps

  • 6. Re: Range of ports infinispan accepts by default: replication timeout happens due to port blocking
    Mircea Markus Master

    I think it's replTmeout you're after.

     

    I've just seen that you posted this question on a different thread as well, please keep related qustions on the same forum thread from now on - this would help other users to get acces to this info.

  • 7. Re: Range of ports infinispan accepts by default: replication timeout happens due to port blocking
    Subash Chaturanga Newbie

    Hi Markus,

    Thanks for the response. Will try this and update this thread.