5 Replies Latest reply: Oct 10, 2011 8:43 AM by Galder Zamarreño RSS

    Join problems in DIST-SYNC cluster

    gernot.bauer Newbie

      Hi!

       

      I am experiencing problems when trying to increase the size of a dist sync cluster. My configuration is as follows:

      • Infinispan 4.2.1-FINAL
      • DIST-SYNC
      • Hotrod
      • Running on EC2 using gossip
      • One BerkeleyDB per node as cache store on elastic block storage

       

      When I restarted a cluster with data in the cache store, I gradually increased the cluster size to 4 nodes. Upon starting the 4th node, I received the following exception (IP addresses are masked):

       

       

      2011-09-07 09:33:41,051 ERROR [org.infinispan.remoting.rpc.RpcManagerImpl] (Rehasher-ip-0.0.0.4-30751-Worker-1) unexpected error while replicating
      org.infinispan.loaders.CacheLoaderException: error loading all entries
              at org.infinispan.loaders.bdbje.BdbjeCacheStore.convertToCacheLoaderException(BdbjeCacheStore.java:553)
              at org.infinispan.loaders.bdbje.BdbjeCacheStore.loadAllKeys(BdbjeCacheStore.java:461)
              at org.infinispan.loaders.decorators.AbstractDelegatingStore.loadAllKeys(AbstractDelegatingStore.java:100)
              at org.infinispan.commands.control.RehashControlCommand.pullStateForJoin(RehashControlCommand.java:200)
              at org.infinispan.commands.control.RehashControlCommand.perform(RehashControlCommand.java:146)
              at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:142)
              at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:156)
              at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithRetry(InboundInvocationHandlerImpl.java:246)
              at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:129)
              at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:159)
              at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:144)
              at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:577)
              at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:488)
              at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:364)
              at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:770)
              at org.jgroups.JChannel.up(JChannel.java:1484)
              at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1074)
              at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:429)
              at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:263)
              at org.jgroups.protocols.FRAG2.up(FRAG2.java:189)
              at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
              at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
              at org.jgroups.protocols.pbcast.GMS.up(GMS.java:891)
              at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:246)
              at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:575)
              at org.jgroups.protocols.UNICAST.up(UNICAST.java:294)
              at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:703)
              at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:133)
              at org.jgroups.protocols.FD.up(FD.java:275)
              at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:275)
              at org.jgroups.protocols.MERGE2.up(MERGE2.java:209)
              at org.jgroups.protocols.Discovery.up(Discovery.java:291)
              at org.jgroups.protocols.TP.passMessageUp(TP.java:1102)
              at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1658)
              at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1640)
              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
              at java.lang.Thread.run(Thread.java:662)
      Caused by: java.lang.IllegalStateException: This method can only be called after associating the current thread with a context
              at org.infinispan.context.InvocationContextContainerImpl.getInvocationContext(InvocationContextContainerImpl.java:148)
              at org.infinispan.eviction.EvictionManagerImpl.getInvocationContext(EvictionManagerImpl.java:164)
              at org.infinispan.eviction.EvictionManagerImpl.onEntryEviction(EvictionManagerImpl.java:146)
              at org.infinispan.eviction.EvictionManagerImpl.onEntryEviction(EvictionManagerImpl.java:140)
              at org.infinispan.container.DefaultDataContainer$DefaultEvictionListener.onEntryEviction(DefaultDataContainer.java:179)
              at org.infinispan.util.concurrent.BoundedConcurrentHashMap$Segment.notifyEvictionListener(BoundedConcurrentHashMap.java:1176)
              at org.infinispan.util.concurrent.BoundedConcurrentHashMap$Segment.get(BoundedConcurrentHashMap.java:871)
              at org.infinispan.util.concurrent.BoundedConcurrentHashMap.get(BoundedConcurrentHashMap.java:1429)
              at org.infinispan.container.DefaultDataContainer.peek(DefaultDataContainer.java:95)
              at org.infinispan.container.DefaultDataContainer.containsKey(DefaultDataContainer.java:130)
              at org.infinispan.util.ReadOnlyDataContainerBackedKeySet.contains(ReadOnlyDataContainerBackedKeySet.java:37)
              at org.infinispan.loaders.bdbje.BdbjeCacheStore.loadAllKeys(BdbjeCacheStore.java:458)
              ... 36 more
      2011-09-07 09:33:41,064 ERROR [org.infinispan.distribution.JoinTask] (Rehasher-ip-0.0.0.4-30751) Caught exception!  Aborting join.
      java.util.concurrent.ExecutionException: org.infinispan.CacheException: org.infinispan.loaders.CacheLoaderException: error loading all entries
              at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
              at java.util.concurrent.FutureTask.get(FutureTask.java:83)
              at org.infinispan.distribution.JoinTask.performRehash(JoinTask.java:111)
              at org.infinispan.distribution.RehashTask.call(RehashTask.java:75)
              at org.infinispan.distribution.RehashTask.call(RehashTask.java:41)
              at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
              at java.util.concurrent.FutureTask.run(FutureTask.java:138)
              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
              at java.lang.Thread.run(Thread.java:662)
      Caused by: org.infinispan.CacheException: org.infinispan.loaders.CacheLoaderException: error loading all entries
              at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:115)
              at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:126)
              at org.infinispan.distribution.RehashTask$StateGrabber.call(RehashTask.java:139)
              at org.infinispan.distribution.RehashTask$StateGrabber.call(RehashTask.java:125)
              ... 5 more
      Caused by: org.infinispan.loaders.CacheLoaderException: error loading all entries
              at org.infinispan.loaders.bdbje.BdbjeCacheStore.convertToCacheLoaderException(BdbjeCacheStore.java:553)
              at org.infinispan.loaders.bdbje.BdbjeCacheStore.loadAllKeys(BdbjeCacheStore.java:461)
              at org.infinispan.loaders.decorators.AbstractDelegatingStore.loadAllKeys(AbstractDelegatingStore.java:100)
              at org.infinispan.commands.control.RehashControlCommand.pullStateForJoin(RehashControlCommand.java:200)
              at org.infinispan.commands.control.RehashControlCommand.perform(RehashControlCommand.java:146)
              at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:142)
              at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:156)
              at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithRetry(InboundInvocationHandlerImpl.java:246)
              at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:129)
              at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:159)
              at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:144)
              at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:577)
              at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:488)
              at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:364)
              at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:770)
              at org.jgroups.JChannel.up(JChannel.java:1484)
              at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1074)
              at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:429)
              at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:263)
              at org.jgroups.protocols.FRAG2.up(FRAG2.java:189)
              at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
              at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
              at org.jgroups.protocols.pbcast.GMS.up(GMS.java:891)
              at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:246)
              at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:575)
              at org.jgroups.protocols.UNICAST.up(UNICAST.java:294)
              at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:703)
              at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:133)
              at org.jgroups.protocols.FD.up(FD.java:275)
              at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:275)
              at org.jgroups.protocols.MERGE2.up(MERGE2.java:209)
              at org.jgroups.protocols.Discovery.up(Discovery.java:291)
              at org.jgroups.protocols.TP.passMessageUp(TP.java:1102)
              at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1658)
              at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1640)
              ... 3 more
      Caused by: java.lang.IllegalStateException: This method can only be called after associating the current thread with a context
              at org.infinispan.context.InvocationContextContainerImpl.getInvocationContext(InvocationContextContainerImpl.java:148)
              at org.infinispan.eviction.EvictionManagerImpl.getInvocationContext(EvictionManagerImpl.java:164)
              at org.infinispan.eviction.EvictionManagerImpl.onEntryEviction(EvictionManagerImpl.java:146)
              at org.infinispan.eviction.EvictionManagerImpl.onEntryEviction(EvictionManagerImpl.java:140)
              at org.infinispan.container.DefaultDataContainer$DefaultEvictionListener.onEntryEviction(DefaultDataContainer.java:179)
              at org.infinispan.util.concurrent.BoundedConcurrentHashMap$Segment.notifyEvictionListener(BoundedConcurrentHashMap.java:1176)
              at org.infinispan.util.concurrent.BoundedConcurrentHashMap$Segment.get(BoundedConcurrentHashMap.java:871)
              at org.infinispan.util.concurrent.BoundedConcurrentHashMap.get(BoundedConcurrentHashMap.java:1429)
              at org.infinispan.container.DefaultDataContainer.peek(DefaultDataContainer.java:95)
              at org.infinispan.container.DefaultDataContainer.containsKey(DefaultDataContainer.java:130)
              at org.infinispan.util.ReadOnlyDataContainerBackedKeySet.contains(ReadOnlyDataContainerBackedKeySet.java:37)
              at org.infinispan.loaders.bdbje.BdbjeCacheStore.loadAllKeys(BdbjeCacheStore.java:458)
              ... 36 more
      2011-09-07 09:33:41,067 INFO  [org.infinispan.distribution.JoinTask] (Rehasher-ip-0.0.0.4-30751) ip-0.0.0.4-30751 aborted join rehash after 130 milliseconds!
      

       

       

      When I stop this node and try to restart it again, I see the following log output, the last lines continuing forever (well, at least for a couple of minutes - after that, I stopped the node):

       

      2011-09-07 09:37:07,970 DEBUG [org.infinispan.distribution.JoinTask] (Rehasher-ip-0-0-0-4-2899) Retrieved old consistent hash address list null
      2011-09-07 09:37:08,369 DEBUG [org.infinispan.loaders.CacheLoaderManagerImpl] (InfinispanServer-Main) Preloaded 140 keys in 461 milliseconds
      2011-09-07 09:37:09,053 DEBUG [org.infinispan.distribution.JoinTask] (Rehasher-ip-0-0-0-4-2899) Retrieved old consistent hash address list null
      2011-09-07 09:37:10,115 DEBUG [org.infinispan.distribution.JoinTask] (Rehasher-ip-0-0-0-4-2899) Retrieved old consistent hash address list null
      2011-09-07 09:37:11,217 DEBUG [org.infinispan.distribution.JoinTask] (Rehasher-ip-0-0-0-4-2899) Retrieved old consistent hash address list null
      2011-09-07 09:37:12,330 DEBUG [org.infinispan.distribution.JoinTask] (Rehasher-ip-0-0-0-4-2899) Retrieved old consistent hash address list null
      2011-09-07 09:37:13,422 DEBUG [org.infinispan.distribution.JoinTask] (Rehasher-ip-0-0-0-4-2899) Retrieved old consistent hash address list null
      2011-09-07 09:37:15,144 DEBUG [org.infinispan.distribution.JoinTask] (Rehasher-ip-0-0-0-4-2899) Retrieved old consistent hash address list null
      2011-09-07 09:37:16,536 DEBUG [org.infinispan.distribution.JoinTask] (Rehasher-ip-0-0-0-4-2899) Retrieved old consistent hash address list null
      2011-09-07 09:37:17,359 DEBUG [org.infinispan.distribution.JoinTask] (Rehasher-ip-0-0-0-4-2899) Retrieved old consistent hash address list null
      2011-09-07 09:37:18,321 DEBUG [org.infinispan.distribution.JoinTask] (Rehasher-ip-0-0-0-4-2899) Retrieved old consistent hash address list null
      2011-09-07 09:37:19,853 DEBUG [org.infinispan.distribution.JoinTask] (Rehasher-ip-0-0-0-4-2899) Retrieved old consistent hash address list null
      2011-09-07 09:37:21,295 DEBUG [org.infinispan.distribution.JoinTask] (Rehasher-ip-0-0-0-4-2899) Retrieved old consistent hash address list null
      2011-09-07 09:37:22,437 DEBUG [org.infinispan.distribution.JoinTask] (Rehasher-ip-0-0-0-4-2899) Retrieved old consistent hash address list null
      2011-09-07 09:37:24,220 DEBUG [org.infinispan.distribution.JoinTask] (Rehasher-ip-0-0-0-4-2899) Retrieved old consistent hash address list null
      

       

      If I restart the node, I get the same messages over and over (there's no difference whether or not I delete the local cache store in between).

       

      I've attached the Infinispan and JGroups config I use on my nodes.

       

      Does anyone have an idea of what is happening here? I'm willing to debug the cluster if I have to, but I'm not even sure where to start looking. Any help is appreciated.

        • 1. Re: Join problems in DIST-SYNC cluster
          Galder Zamarreño Master

          The invocation context handling is changing considerably for Infinispan 5.1.0.BETA1 as a result of https://issues.jboss.org/browse/ISPN-1397, so I'd suggest you give your use case a try when 5.1.0.BETA1 is released which should be this coming Monday/Tuesday.

          • 2. Re: Join problems in DIST-SYNC cluster
            gernot.bauer Newbie

            Hi Galder,

             

            Thanks for your reply. Since we encountered library version conflicts concerning the 5.0 hotrod-client and our JBoss 5 AS which we coulnd't solve by now (there was a discussion somewhere describing this problem), we are currently bound to hotrod client v4.2.1. Is it safe to use the Hotrod 4 client with Infinispan 5.1 (I made a short test with client 4.2.1 and server 5.0 and didn't encounter problems)?

            • 3. Re: Join problems in DIST-SYNC cluster
              gernot.bauer Newbie

              Today I made some tests with Infinispan 5.1-BETA1. I successfully managed to setup a cluster with a similar configuration from above (slightly adopting the jgroups config to get the nodes up and running) and added random data with the 4.2.1 hotrod client. During these write operations, I shut down the second node, waited some time and restarted the second node. In the log file, I saw many exceptions similar to this one:

               

              2011-10-06 14:02:27,296 DEBUG (InfinispanServer-Main) [org.infinispan.loaders.CacheLoaderManagerImpl] Preloading transient state from cache loader org.infinispan.loaders.decorators.AsyncStore@7a258c74
              2011-10-06 14:02:29,998 DEBUG (OOB-2,infinispan-cluster-set,ip-0-0-0-2-12649) [org.infinispan.statetransfer.BaseStateTransferManagerImpl] Applying new state from ip-0-0-0-1-7198: received 84042 keys
              2011-10-06 14:02:30,005 ERROR (OOB-2,infinispan-cluster-set,ip-0-0-0-1-12649) [org.infinispan.interceptors.InvocationContextInterceptor] ISPN000136: Execution error
              java.lang.NullPointerException
                      at org.infinispan.distribution.DistributionManagerImpl.locate(DistributionManagerImpl.java:137)
                      at org.infinispan.interceptors.DistCacheStoreInterceptor.skip(DistCacheStoreInterceptor.java:137)
                      at org.infinispan.interceptors.DistCacheStoreInterceptor.visitPutKeyValueCommand(DistCacheStoreInterceptor.java:80)
                      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
                      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)
                      at org.infinispan.interceptors.CacheLoaderInterceptor.visitPutKeyValueCommand(CacheLoaderInterceptor.java:81)
                      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
                      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)
                      at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:181)
                      at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:136)
                      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
                      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)
                      at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitPutKeyValueCommand(NonTransactionalLockingInterceptor.java:59)
                      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
                      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)
                      at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:133)
                      at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:60)
                      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
                      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)
                      at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:133)
                      at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:60)
                      at org.infinispan.interceptors.StateTransferLockInterceptor.visitPutKeyValueCommand(StateTransferLockInterceptor.java:110)
                      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
                      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)
                      at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:114)
                      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
                      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)
                      at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:133)
                      at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:60)
                      at org.infinispan.interceptors.IsMarshallableInterceptor.visitPutKeyValueCommand(IsMarshallableInterceptor.java:98)
                      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
                      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)
                      at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:104)
                      at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:64)
                      at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:60)
                      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
                      at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:318)
                      at org.infinispan.statetransfer.BaseStateTransferManagerImpl.applyState(BaseStateTransferManagerImpl.java:249)
                      at org.infinispan.commands.control.StateTransferControlCommand.perform(StateTransferControlCommand.java:116)
                      at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:173)
                      at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:181)
                      at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithRetry(InboundInvocationHandlerImpl.java:249)
                      at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:159)
                      at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:160)
                      at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:139)
                      at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:446)
                      at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:353)
                      at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:229)
                      at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:556)
                      at org.jgroups.JChannel.up(JChannel.java:720)
                      at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)
                      at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:430)
                      at org.jgroups.protocols.FRAG2.unfragment(FRAG2.java:302)
                      at org.jgroups.protocols.FRAG2.up(FRAG2.java:162)
                      at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
                      at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
                      at org.jgroups.protocols.pbcast.GMS.up(GMS.java:870)
                      at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)
                      at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:572)
                      at org.jgroups.protocols.UNICAST.up(UNICAST.java:282)
                      at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:583)
                      at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:134)
                      at org.jgroups.protocols.FD.up(FD.java:270)
                      at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:273)
                      at org.jgroups.protocols.MERGE2.up(MERGE2.java:208)
                      at org.jgroups.protocols.Discovery.up(Discovery.java:334)
                      at org.jgroups.protocols.TP.passMessageUp(TP.java:1091)
                      at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1646)
                      at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1628)
                      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                      at java.lang.Thread.run(Thread.java:662)
              
              • 4. Re: Join problems in DIST-SYNC cluster
                Galder Zamarreño Master

                Yes, it should be safe to use Hot Rod client 4.2 with Hot Rod server 5 because there's a wire contract provided by the Hot Rod protocol.

                 

                That NPE looks like a bug. Would you mind filling a JIRA in https://issues.jboss.org/browse/ISPN and provide information on how to replicate the issue exacltly? Also, if you repeat the test, please enable TRACE logging on org.infinispan package and attach the log to the JIRA isssue.

                • 5. Re: Join problems in DIST-SYNC cluster
                  Galder Zamarreño Master

                  Please also attach the configuration file used