7 Replies Latest reply: Sep 26, 2012 2:58 PM by jchildress2 RSS

Errors using Infinispan unicast cluster, HotRod server and a HotRod client - any idea?

Andrian Belinschi Newbie

Infinispan version 5.1.4.Final

 

Using a cluster of 6 nodes in unicast tcp mode configured through Jgroups 2.0.9 configuration and a HotRod server that joins the cluster successfully on start up using also unicast tcp.

After the cluster is formed correctly, using a client Java application I am trying to 'put' data in default cache in the cluster but getting the following errors in one of cluster nodes immediately after issuing the 'put("key", "value")':


WARN: Problems unmarshalling remote command from byte buffer

org.infinispan.CacheException: Type of data read is unknown. Id=-2147482548 is not amongst known reader indexes.

        at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:216)

        at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)

        at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)

        at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)

        at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:119)

        at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:107)

        at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:58)

        at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:391)

        at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:222)

        at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)

        at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)

        at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)

        at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:119)

        at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:163)

        at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:67)

        at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:391)

        at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:222)

        at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)

        at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)

        at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)

        at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:154)

        at org.infinispan.marshall.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:114)

        at org.infinispan.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:85)

        at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:50)

        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:200)

        at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456)

        at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363)

        at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238)

        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543)

        at org.jgroups.JChannel.up(JChannel.java:716)

        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)

        at org.jgroups.protocols.RSVP.up(RSVP.java:192)

        at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)

        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)

        at org.jgroups.protocols.FlowControl.up(FlowControl.java:400)

        at org.jgroups.protocols.pbcast.GMS.up(GMS.java:889)

        at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)

        at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:759)

        at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:365)

        at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:602)

        at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143)

        at org.jgroups.protocols.FD.up(FD.java:273)

        at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288)

        at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)

        at org.jgroups.protocols.Discovery.up(Discovery.java:359)

        at org.jgroups.protocols.TP.passMessageUp(TP.java:1180)

        at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1728)

        at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1710)

        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

        at java.lang.Thread.run(Thread.java:619)

 

Any idea why I am getting the error?

 

My HotRod client reports the following error:

 

May 10, 2012 4:19:28 PM org.infinispan.client.hotrod.impl.protocol.Codec10 checkForErrorsInResponseStatus

WARN: ISPN004005: Error received from the server: org.infinispan.CacheException: Problems invoking command.

Exception in thread "main" org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for message id[4] returned server error (status=0x85): org.infinispan.CacheException: Problems invoking command.

    at org.infinispan.client.hotrod.impl.protocol.Codec10.checkForErrorsInResponseStatus(Codec10.java:156)

    at org.infinispan.client.hotrod.impl.protocol.Codec10.readHeader(Codec10.java:109)

    at org.infinispan.client.hotrod.impl.operations.HotRodOperation.readHeaderAndValidate(HotRodOperation.java:78)

    at org.infinispan.client.hotrod.impl.operations.AbstractKeyValueOperation.sendPutOperation(AbstractKeyValueOperation.java:72)

    at org.infinispan.client.hotrod.impl.operations.PutOperation.executeOperation(PutOperation.java:52)

    at org.infinispan.client.hotrod.impl.operations.PutOperation.executeOperation(PutOperation.java:41)

    at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:68)

    at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:220)

    at org.infinispan.CacheSupport.put(CacheSupport.java:53)

    at org.infinispan.quickstart.clusteredcache.distribution.HotRodClient.main(HotRodClient.java:24)

 

 

Here is the HotRod client code:

 

BasicCacheContainer cm = new RemoteCacheManager("myhostnam:11222", true);

BasicCache<Object, Object> cache = cm.getCache();

String value = cache.get("key"); // no errors

cache.put("key", "value");  // errors out

 

...

 

 

Any suggestions appreciated.

 

thanks

  • 1. Re: Errors using Infinispan unicast cluster, HotRod server and a HotRod client - any idea?
    Galder Zamarreño Master

    No idea. Are you sure it's JGroups 2.0.9? Shouldn't be 3.x?

     

    Maybe each cluster server might be using different jars.

     

    I'd suggest you build a test case and attach it here.

  • 2. Re: Errors using Infinispan unicast cluster, HotRod server and a HotRod client - any idea?
    Andrian Belinschi Newbie

    I made a typo, I was going to write that it is using JGroups 3.0.9

  • 3. Re: Errors using Infinispan unicast cluster, HotRod server and a HotRod client - any idea?
    Galder Zamarreño Master

    Thought that might be the case . So, can you build a test case for us? This is a weird scenario.

  • 4. Re: Errors using Infinispan unicast cluster, HotRod server and a HotRod client - any idea?
    Andrian Belinschi Newbie

    I have just downloaded Infinispan 5.1.5.CR1 and using new jars for testing the cluster with a simple HotRod client (the code shown in previous posts). It looks like the logic is working now. Here is what I am running:

     

    1. An Infinispan cluster of two nodes (different hosts running Solaris) clustered using TCP unicast (with TCPPING enabled) with a pre-defined list of hosts + a HotRod server (running an OpenSuse box) that joins the cluster also in TCP unicast mode (total three members in the cluster running on different hosts). The cluster forms correctly without any errors.
    2. A simple HotRod simple testing code running on Win7 64 bits. However when running the HotRod client, the HotRod server is throwing the following errors (the client doesn't report any errors and still reads and updates the cache successfully):

     

    2012-05-15 15:05:27,583 ERROR [HotRodDecoder] (HotRodServerWorker-1-1) ISPN005009: Unexpected error before any request parameters read

    1. java.io.IOException: Connection reset by peer

            at sun.nio.ch.FileDispatcher.read0(Native Method)

            at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)

            at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:251)

            at sun.nio.ch.IOUtil.read(IOUtil.java:218)

            at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:254)

            at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:323)

            at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)

            at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)

            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

            at java.lang.Thread.run(Thread.java:679)

    2012-05-15 15:05:27,585 ERROR [HotRodDecoder] (HotRodServerWorker-1-1) ISPN005009: Unexpected error before any request parameters read

    1. java.io.IOException: Broken pipe

            at sun.nio.ch.FileDispatcher.write0(Native Method)

            at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)

            at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:122)

            at sun.nio.ch.IOUtil.write(IOUtil.java:78)

            at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:352)

            at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$PooledSendBuffer.transferTo(SocketSendBufferPool.java:240)

            at org.jboss.netty.channel.socket.nio.NioWorker.write0(NioWorker.java:472)

            at org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:390)

            at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:137)

            at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76)

            at org.jboss.netty.channel.Channels.write(Channels.java:632)

            at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70)

            at org.jboss.netty.channel.Channels.write(Channels.java:611)

            at org.jboss.netty.channel.Channels.write(Channels.java:578)

            at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:251)

            at org.infinispan.server.core.AbstractProtocolDecoder.exceptionCaught(AbstractProtocolDecoder.scala:279)

            at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:432)

            at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:333)

            at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)

            at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)

            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

            at java.lang.Thread.run(Thread.java:679)

     

     

     

     

     

     

     

    Let me know if you need more details. The error is generated each time I run the HotRod client Java app.

     

    -Andrian

  • 5. Re: Errors using Infinispan unicast cluster, HotRod server and a HotRod client - any idea?
    Andrian Belinschi Newbie

    It looks like if I use 'cm.close()' before exisiting the client app then all is good, no more errors on HotRod server.

  • 6. Re: Errors using Infinispan unicast cluster, HotRod server and a HotRod client - any idea?
    jchildress2 Newbie

    Andrian,

     

    I'm getting the same error and seem to have a similar setup to yours.

     

    I don't see where cm.close() is available in the API....  I'm using 5.1.6

     

    thanks,

     

    John

  • 7. Re: Errors using Infinispan unicast cluster, HotRod server and a HotRod client - any idea?
    jchildress2 Newbie

    It looks like stop() takes care of it...