3 Replies Latest reply on May 11, 2012 10:09 AM by galder.zamarreno

    OutOfMemory Exception in HotRodServer

    alex.shvidky

      Hi All,

       

      I performed next test scenario with Infinispan 5.1.3:

      1. Run hotrod infinispan server (Only one node).

      2. Store big amount of objects in it (Arround 1 million objects, ~500B every object).

      3. Retrieve all objects via HotRodClient - I have to do it to initialize my local caches.

       

      I store successfully all of my objects and it looks like singe gets work fine.

      But when I try to retrieve all objects from cache using RemoteCache.getBulk() I get OutOfMemory exception in the server:

      ERROR [HotRodDecoder] (HotRodServerWorker-1-62) ISPN005009: Unexpected error before any request parameters read

      java.lang.OutOfMemoryError: Java heap space

              at org.jboss.netty.buffer.HeapChannelBuffer.<init>(HeapChannelBuffer.java:47)

              at org.jboss.netty.buffer.BigEndianHeapChannelBuffer.<init>(BigEndianHeapChannelBuffer.java:39)

              at org.jboss.netty.buffer.ChannelBuffers.buffer(ChannelBuffers.java:139)

              at org.jboss.netty.buffer.HeapChannelBufferFactory.getBuffer(HeapChannelBufferFactory.java:73)

              at org.jboss.netty.buffer.DynamicChannelBuffer.ensureWritableBytes(DynamicChannelBuffer.java:84)

              at org.jboss.netty.buffer.DynamicChannelBuffer.writeBytes(DynamicChannelBuffer.java:233)

              at org.jboss.netty.buffer.AbstractChannelBuffer.writeBytes(AbstractChannelBuffer.java:446)

              at org.infinispan.server.core.transport.ExtendedChannelBuffer$.writeRangedBytes(ExtendedChannelBuffer.scala:64)

              at org.infinispan.server.hotrod.Encoder10$$anonfun$writeResponse$5.apply(Encoder10.scala:104)

              at org.infinispan.server.hotrod.Encoder10$$anonfun$writeResponse$5.apply(Encoder10.scala:101)

              at scala.collection.Iterator$class.foreach(Iterator.scala:660)

              at scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:573)

              at org.infinispan.server.hotrod.Encoder10$.writeResponse(Encoder10.scala:101)

              at org.infinispan.server.hotrod.Encoder11$.writeResponse(Encoder11.scala:67)

              at org.infinispan.server.hotrod.HotRodEncoder.encode(HotRodEncoder.scala:67)

              at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:66)

              at org.jboss.netty.channel.Channels.write(Channels.java:611)

              at org.jboss.netty.channel.Channels.write(Channels.java:578)

              at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:251)

              at org.infinispan.server.core.AbstractProtocolDecoder.writeResponse(AbstractProtocolDecoder.scala:174)

              at org.infinispan.server.hotrod.HotRodDecoder.customDecodeKey(HotRodDecoder.scala:152)

              at org.infinispan.server.core.AbstractProtocolDecoder.decodeKey(AbstractProtocolDecoder.scala:115)

              at org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:70)

              at org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:45)

              at org.infinispan.server.core.transport.CustomReplayingDecoder.callDecode(CustomReplayingDecoder.java:250)

              at org.infinispan.server.core.transport.CustomReplayingDecoder.messageReceived(CustomReplayingDecoder.java:223)

              at org.infinispan.server.core.AbstractProtocolDecoder.messageReceived(AbstractProtocolDecoder.scala:360)

       

      Any ideas?

      Please suggest.

       

      I need the way to read all objects from Infinispan cache. Does another way exist except RemoteCache.getBulk()?

       

      Thanks in advance.

      Alex

        • 1. Re: OutOfMemory Exception in HotRodServer
          galder.zamarreno

          Streaming has not yet been implemented in Infinispan servers, so if you're calling getBulk, the server will make copies of the key/value requested and store them in a buffer to send them back to the client.

           

          So, what's likely happening is that your cache has not been given enough memory to support both storing the data and retrieving all the data in bulk.

           

          There's several ways you can work around this:

           

          1. Increase -Xmx of the JVM that runs the Hot Rod server.

           

          or

           

          2. Call RemoteCache.getBulk(size), limiting the number of entries to return.

          • 2. Re: OutOfMemory Exception in HotRodServer
            alex.shvidky

            Hi Galder,

             

            Thank you very much for your answer.

             

            1. How can I provide -Xmx parameter when I run Hot Rod server? How much memory it uses by default?

            2. You are right that RemoteCache.getBulk(size) will limit number of entries returned from server. But how in this way can I fetch all entries from server? I had not seen any paging API in RemoteCache? Had I missed something?

             

            Thanks,

            Alex

            • 3. Re: OutOfMemory Exception in HotRodServer
              galder.zamarreno

              Re 1.) Depends how you start it, but if you use the script, add -Xmx there, there're tons of examples out there... we use no default, depends on your JVM.

               

              Re 2.) Don't think you can use it that way yet.