4 Replies Latest reply on Jan 27, 2014 8:21 AM by dan.berindei

    ISPN000071: Caught exception when handling command SingleRpcCommand

    ckriley0814

      We are trying to perform an insert of a cache value into a distributed cache that is replicated via Cross Site Replication in JBoss 6.0.0.Alpha4. The cross site replication has two nodes in the global cluster. 

      When we do the put on the local site we get the following error on the remote site. We just upgraded to 6.0.0.Alpha4 because of JIRA ISSUE ISPN-3346, which we reproduced in 5.3.0.Final. I have attached our jboss configuration file for review.

       

      17:22:41,798 WARN  [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (Incoming-2,shared=tcp) ISPN000071: Caught exception when handling command SingleRpcCommand{cacheName='importantCache', command=PutKeyValueCommand{key=1, value=[B@363f983f, flags=null, putIfAbsent=false, metadata=MimeMetadata(contentType=text/plain), successful=true, ignorePreviousValue=false}}: java.lang.NullPointerException

              at org.infinispan.xsite.BackupReceiverRepositoryImpl.isBackupForRemoteCache(BackupReceiverRepositoryImpl.java:112) [infinispan-core-6.0.0.Alpha4.jar:6.0.0.Alpha4]

              at org.infinispan.xsite.BackupReceiverRepositoryImpl.getBackupCacheManager(BackupReceiverRepositoryImpl.java:95) [infinispan-core-6.0.0.Alpha4.jar:6.0.0.Alpha4]

              at org.infinispan.xsite.BackupReceiverRepositoryImpl.handleRemoteCommand(BackupReceiverRepositoryImpl.java:67) [infinispan-core-6.0.0.Alpha4.jar:6.0.0.Alpha4]

              at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromRemoteSite(CommandAwareRpcDispatcher.java:234) [infinispan-core-6.0.0.Alpha4.jar:6.0.0.Alpha4]

              at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:209) [infinispan-core-6.0.0.Alpha4.jar:6.0.0.Alpha4]

              at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460) [jgroups-3.4.0.Alpha1.jar:3.4.0.Alpha1]

              at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377) [jgroups-3.4.0.Alpha1.jar:3.4.0.Alpha1]

              at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:247) [jgroups-3.4.0.Alpha1.jar:3.4.0.Alpha1]

              at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:665) [jgroups-3.4.0.Alpha1.jar:3.4.0.Alpha1]

              at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) [jgroups-3.4.0.Alpha1.jar:3.4.0.Alpha1]

              at org.jgroups.JChannel.up(JChannel.java:719) [jgroups-3.4.0.Alpha1.jar:3.4.0.Alpha1]

              at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1002) [jgroups-3.4.0.Alpha1.jar:3.4.0.Alpha1]

              at org.jgroups.protocols.relay.RELAY2.deliver(RELAY2.java:612) [jgroups-3.4.0.Alpha1.jar:3.4.0.Alpha1]

              at org.jgroups.protocols.relay.RELAY2.route(RELAY2.java:508) [jgroups-3.4.0.Alpha1.jar:3.4.0.Alpha1]

              at org.jgroups.protocols.relay.RELAY2.handleMessage(RELAY2.java:483) [jgroups-3.4.0.Alpha1.jar:3.4.0.Alpha1]

              at org.jgroups.protocols.relay.RELAY2.handleRelayMessage(RELAY2.java:461) [jgroups-3.4.0.Alpha1.jar:3.4.0.Alpha1]

              at org.jgroups.protocols.relay.Relayer$Bridge.receive(Relayer.java:263) [jgroups-3.4.0.Alpha1.jar:3.4.0.Alpha1]

              at org.jgroups.JChannel.up(JChannel.java:749) [jgroups-3.4.0.Alpha1.jar:3.4.0.Alpha1]

              at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1006) [jgroups-3.4.0.Alpha1.jar:3.4.0.Alpha1]

              at org.jgroups.stack.Protocol.up(Protocol.java:406) [jgroups-3.4.0.Alpha1.jar:3.4.0.Alpha1]

              at org.jgroups.protocols.FRAG2.up(FRAG2.java:195) [jgroups-3.4.0.Alpha1.jar:3.4.0.Alpha1]

              at org.jgroups.protocols.FlowControl.up(FlowControl.java:439) [jgroups-3.4.0.Alpha1.jar:3.4.0.Alpha1]

              at org.jgroups.protocols.FlowControl.up(FlowControl.java:439) [jgroups-3.4.0.Alpha1.jar:3.4.0.Alpha1]

              at org.jgroups.stack.Protocol.up(Protocol.java:406) [jgroups-3.4.0.Alpha1.jar:3.4.0.Alpha1]

              at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:304) [jgroups-3.4.0.Alpha1.jar:3.4.0.Alpha1]

              at org.jgroups.protocols.UNICAST.removeAndDeliver(UNICAST.java:748) [jgroups-3.4.0.Alpha1.jar:3.4.0.Alpha1]

              at org.jgroups.protocols.UNICAST.handleBatchReceived(UNICAST.java:704) [jgroups-3.4.0.Alpha1.jar:3.4.0.Alpha1]

              at org.jgroups.protocols.UNICAST.up(UNICAST.java:454) [jgroups-3.4.0.Alpha1.jar:3.4.0.Alpha1]

              at org.jgroups.stack.Protocol.up(Protocol.java:406) [jgroups-3.4.0.Alpha1.jar:3.4.0.Alpha1]

              at org.jgroups.stack.Protocol.up(Protocol.java:406) [jgroups-3.4.0.Alpha1.jar:3.4.0.Alpha1]

              at org.jgroups.protocols.FD.up(FD.java:274) [jgroups-3.4.0.Alpha1.jar:3.4.0.Alpha1]

              at org.jgroups.stack.Protocol.up(Protocol.java:406) [jgroups-3.4.0.Alpha1.jar:3.4.0.Alpha1]

              at org.jgroups.protocols.MERGE2.up(MERGE2.java:223) [jgroups-3.4.0.Alpha1.jar:3.4.0.Alpha1]

              at org.jgroups.stack.Protocol.up(Protocol.java:406) [jgroups-3.4.0.Alpha1.jar:3.4.0.Alpha1]

              at org.jgroups.stack.Protocol.up(Protocol.java:406) [jgroups-3.4.0.Alpha1.jar:3.4.0.Alpha1]

              at org.jgroups.protocols.TP.passBatchUp(TP.java:1409) [jgroups-3.4.0.Alpha1.jar:3.4.0.Alpha1]

              at org.jgroups.protocols.TP$BatchHandler.run(TP.java:1564) [jgroups-3.4.0.Alpha1.jar:3.4.0.Alpha1]

              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.6.0_24]

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.6.0_24]

              at java.lang.Thread.run(Thread.java:679) [rt.jar:1.6.0_24]

        • 1. Re: ISPN000071: Caught exception when handling command SingleRpcCommand
          jugglingcats

          I am seeing a similar error (same error code) on 6.0.0.Final.

           

          WARN  cheTopologyControlCommand | ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=sessions, type=CH_UPDATE, sender=alfie-lt-46127, joinInfo=null, topologyId=3, currentCH=DefaultConsistentHash{numSegments=60, numOwners=1, members=[alfie-lt-46127]}, pendingCH=null, throwable=null, viewId=1}

          java.util.ConcurrentModificationException

            at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926)

            at java.util.HashMap$KeyIterator.next(HashMap.java:960)

            at org.infinispan.persistence.async.AdvancedAsyncCacheLoader.process(AdvancedAsyncCacheLoader.java:80)

            at org.infinispan.persistence.manager.PersistenceManagerImpl.processOnAllStores(PersistenceManagerImpl.java:414)

            at org.infinispan.statetransfer.StateConsumerImpl.invalidateSegments(StateConsumerImpl.java:910)

            at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:393)

            at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:178)

            at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:38)

            at org.infinispan.statetransfer.StateTransferManagerImpl$1.updateConsistentHash(StateTransferManagerImpl.java:100)

            at org.infinispan.topology.LocalTopologyManagerImpl.handleConsistentHashUpdate(LocalTopologyManagerImpl.java:191)

            at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:152)

            at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:124)

            at org.infinispan.topology.ClusterTopologyManagerImpl$3.run(ClusterTopologyManagerImpl.java:606)

            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

            at java.util.concurrent.FutureTask.run(FutureTask.java:262)

            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

            at java.lang.Thread.run(Thread.java:744)

           

          I prefer your cache name btw... ;-)

           

          Should we be worried by this error? It is only at WARN level.

           

          Alfie.

          • 2. Re: ISPN000071: Caught exception when handling command SingleRpcCommand
            pruivo

            Hi Chris,

             

            Is the cache 'importantCache' defined in remote site?

             

            EDIT: btw, are you using the same configuration file in both sites?

             

            Cheers,

            Pedro

            • 3. Re: ISPN000071: Caught exception when handling command SingleRpcCommand
              pruivo

              Hi Alfie,

               

              Next time open a new topic

               

              Anyway, that is definitely a bug. Do you mind to open a JIRA? (Infinispan - JBoss Issue Tracker)

               

              Although it does not make you lose data, it may cause a memory leak...

               

              Cheers,

              Pedro

              • 4. Re: ISPN000071: Caught exception when handling command SingleRpcCommand
                dan.berindei

                Alfie, I've created an issue in JIRA for the second problem: ISPN-3938 AdvancedAsyncCacheLoader.process() concurrency issues.

                 

                Chris, do you still see your problem with the Infinispan 6.0.1.Final? I did notice a problem at line 94 in your configuration, a `<!--` inside a comment, but I don't think that's the real problem. I've created another issue to improve the test coverage for x-site in the server: ISPN-3939 Improve test coverage for for x-site configuration parsing.