5 Replies Latest reply on May 2, 2012 9:04 AM by galder.zamarreno

    Errors with rehash and cache loader

    dvlinh

      Hello,

       

      I'm using Infinispan in DIST mode with FileCacheStore. Here is my config:

       

          <default>
              <clustering mode="distribution">
                  <hash numOwners="2" rehashRpcTimeout="120000"/>
                  <async asyncMarshalling="false" />
                  <l1 enabled="true" lifespan="60000" />
              </clustering>
              <invocationBatching enabled="true"/>
              <locking useLockStriping="false" />
              <loaders shared="true" preload="true" passivation="false">
                  <loader class="org.infinispan.loaders.file.FileCacheStore" fetchPersistentState="true" purgeOnStartup="false">
                      <properties>
                          <property name="location" value="/pathToStore"/>
                      </properties>
                      <async enabled="true" flushLockTimeout="15000" threadPoolSize="5"/>
                  </loader>
              </loaders>
          </default>
      

       

      But I've got an issue by doing the following:

      • Start node 1,
      • Insert data in to the cache
      • Start node 2
      • I see changes of the cache store in file system, so I guess infinispan is rehashing the data
      • Start node 3
      • Here, I've got several errors in the logs of both nodes:

       

      ERROR (FileCacheStore.java:310) - ISPN000062: Error while reading from file: /pathToStore/myNamedCache/0
      java.io.EOFException: The stream ended unexpectedly.  Please check whether the source of the stream encountered any issues generating the stream.
          at org.infinispan.marshall.VersionAwareMarshaller.objectFromObjectStream(VersionAwareMarshaller.java:190)
          at org.infinispan.marshall.AbstractDelegatingMarshaller.objectFromObjectStream(AbstractDelegatingMarshaller.java:59)
          at org.infinispan.loaders.file.FileCacheStore.objectFromInputStreamInReentrantMode(FileCacheStore.java:433)
          at org.infinispan.loaders.file.FileCacheStore.loadBucket(FileCacheStore.java:306)
          at org.infinispan.loaders.file.FileCacheStore.loadBucket(FileCacheStore.java:280)
          at org.infinispan.loaders.bucket.BucketBasedCacheStore.storeLockSafe(BucketBasedCacheStore.java:86)
          at org.infinispan.loaders.bucket.BucketBasedCacheStore.storeLockSafe(BucketBasedCacheStore.java:49)
          at org.infinispan.loaders.LockSupportCacheStore.store(LockSupportCacheStore.java:207)
          at org.infinispan.loaders.decorators.AbstractDelegatingStore.store(AbstractDelegatingStore.java:68)
          at org.infinispan.loaders.decorators.AsyncStore.applyModificationsSync(AsyncStore.java:245)
          at org.infinispan.loaders.decorators.AsyncStore$AsyncProcessor.put(AsyncStore.java:405)
          at org.infinispan.loaders.decorators.AsyncStore$AsyncProcessor.innerRun(AsyncStore.java:389)
          at org.infinispan.loaders.decorators.AsyncStore$AsyncProcessor.run(AsyncStore.java:312)
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
          at java.lang.Thread.run(Thread.java:619)
      Caused by: java.io.EOFException: Read past end of file
          at org.jboss.marshalling.SimpleDataInput.eofOnRead(SimpleDataInput.java:126)
          at org.jboss.marshalling.SimpleDataInput.readUnsignedByteDirect(SimpleDataInput.java:263)
          at org.jboss.marshalling.SimpleDataInput.readIntDirect(SimpleDataInput.java:306)
          at org.jboss.marshalling.SimpleDataInput.readLongDirect(SimpleDataInput.java:294)
          at org.jboss.marshalling.SimpleDataInput.readLong(SimpleDataInput.java:289)
          at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:369)
          at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
          at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1673)
          at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1589)
          at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1553)
          at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1231)
          at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272)
          at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
          at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)
          at org.infinispan.container.entries.ImmortalCacheEntry$Externalizer.readObject(ImmortalCacheEntry.java:158)
          at org.infinispan.container.entries.ImmortalCacheEntry$Externalizer.readObject(ImmortalCacheEntry.java:148)
          at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:389)
          at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:220)
          at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)
          at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
          at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)
          at org.infinispan.loaders.bucket.Bucket$Externalizer.readObject(Bucket.java:157)
          at org.infinispan.loaders.bucket.Bucket$Externalizer.readObject(Bucket.java:139)
          at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:389)
          at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:220)
          at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)
          at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
          at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)
          at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:148)
          at org.infinispan.marshall.VersionAwareMarshaller.objectFromObjectStream(VersionAwareMarshaller.java:188)
          ... 15 more
      

       

      I don't know if my config is right or how I could deal with that. Do I need to wait the rehash finished to run the node 3 ?

        • 1. Re: Errors with rehash and cache loader
          galder.zamarreno

          What Infinispan version? I'd suggest trying latest 5.1.4.CR1 since we resolved some EOFException cases in the past.

          • 2. Re: Errors with rehash and cache loader
            dvlinh

            I used 5.1.3.FINAL. I will try 5.1.4.FINAL to see if it happens again.

            Thank you

            • 3. Re: Errors with rehash and cache loader
              dvlinh

              I always have theses errors with 5.1.4.FINAL.

              And I found that the rehash process is somewhat long. It took me nearly 1 hour for about 200,000 simple objects (50 MB on FileCacheStore)

              • 4. Re: Errors with rehash and cache loader
                dvlinh

                I also have warnings like this:

                 

                WARN (FileCacheStore.java:270) - ISPN000060: Problems purging file /pathToStore/myNamedCache/1024
                org.infinispan.loaders.CacheLoaderException: Error while reading from file
                    at org.infinispan.loaders.file.FileCacheStore.loadBucket(FileCacheStore.java:311)
                    at org.infinispan.loaders.file.FileCacheStore.doPurge(FileCacheStore.java:251)
                    at org.infinispan.loaders.file.FileCacheStore.purgeInternal(FileCacheStore.java:230)
                    at org.infinispan.loaders.AbstractCacheStore$2.run(AbstractCacheStore.java:111)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                    at java.lang.Thread.run(Thread.java:619)
                Caused by: java.io.EOFException: The stream ended unexpectedly.  Please check whether the source of the stream encountered any issues generating the stream.
                    at org.infinispan.marshall.VersionAwareMarshaller.objectFromObjectStream(VersionAwareMarshaller.java:191)
                    at org.infinispan.marshall.AbstractDelegatingMarshaller.objectFromObjectStream(AbstractDelegatingMarshaller.java:60)
                    at org.infinispan.loaders.file.FileCacheStore.objectFromInputStreamInReentrantMode(FileCacheStore.java:433)
                    at org.infinispan.loaders.file.FileCacheStore.loadBucket(FileCacheStore.java:306)
                    ... 6 more
                Caused by: java.io.EOFException: Read past end of file
                    at org.jboss.marshalling.SimpleDataInput.eofOnRead(SimpleDataInput.java:126)
                    at org.jboss.marshalling.SimpleDataInput.readUnsignedByteDirect(SimpleDataInput.java:263)
                    at org.jboss.marshalling.SimpleDataInput.readIntDirect(SimpleDataInput.java:306)
                    at org.jboss.marshalling.SimpleDataInput.readLongDirect(SimpleDataInput.java:294)
                    at org.jboss.marshalling.SimpleDataInput.readLong(SimpleDataInput.java:289)
                    at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:369)
                    at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
                    at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1677)
                    at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1593)
                    at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1557)
                    at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1235)
                    at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272)
                    at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
                    at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)
                    at org.infinispan.container.entries.ImmortalCacheEntry$Externalizer.readObject(ImmortalCacheEntry.java:160)
                    at org.infinispan.container.entries.ImmortalCacheEntry$Externalizer.readObject(ImmortalCacheEntry.java:150)
                    at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:391)
                    at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:222)
                    at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)
                    at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
                    at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)
                    at org.infinispan.loaders.bucket.Bucket$Externalizer.readObject(Bucket.java:157)
                    at org.infinispan.loaders.bucket.Bucket$Externalizer.readObject(Bucket.java:139)
                    at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:391)
                    at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:222)
                    at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)
                    at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
                    at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)
                    at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:154)
                    at org.infinispan.marshall.VersionAwareMarshaller.objectFromObjectStream(VersionAwareMarshaller.java:189)
                    ... 9 more
                
                • 5. Re: Errors with rehash and cache loader
                  galder.zamarreno

                  Do you have a unit test that we can run to reproduce this issue? Were the cache store contents generated with the same Infinispan version or with a different one?

                   

                  For the rehash issue, what about you replicate the issue again and generate some stracktraces from the nodes involve in the rehashing? Zip them up and attach them here.