5 Replies Latest reply: May 2, 2012 9:04 AM by Galder Zamarreño RSS

Errors with rehash and cache loader

Viet Linh Dang Newbie

Hello,

 

I'm using Infinispan in DIST mode with FileCacheStore. Here is my config:

 

    <default>
        <clustering mode="distribution">
            <hash numOwners="2" rehashRpcTimeout="120000"/>
            <async asyncMarshalling="false" />
            <l1 enabled="true" lifespan="60000" />
        </clustering>
        <invocationBatching enabled="true"/>
        <locking useLockStriping="false" />
        <loaders shared="true" preload="true" passivation="false">
            <loader class="org.infinispan.loaders.file.FileCacheStore" fetchPersistentState="true" purgeOnStartup="false">
                <properties>
                    <property name="location" value="/pathToStore"/>
                </properties>
                <async enabled="true" flushLockTimeout="15000" threadPoolSize="5"/>
            </loader>
        </loaders>
    </default>

 

But I've got an issue by doing the following:

  • Start node 1,
  • Insert data in to the cache
  • Start node 2
  • I see changes of the cache store in file system, so I guess infinispan is rehashing the data
  • Start node 3
  • Here, I've got several errors in the logs of both nodes:

 

ERROR (FileCacheStore.java:310) - ISPN000062: Error while reading from file: /pathToStore/myNamedCache/0
java.io.EOFException: The stream ended unexpectedly.  Please check whether the source of the stream encountered any issues generating the stream.
    at org.infinispan.marshall.VersionAwareMarshaller.objectFromObjectStream(VersionAwareMarshaller.java:190)
    at org.infinispan.marshall.AbstractDelegatingMarshaller.objectFromObjectStream(AbstractDelegatingMarshaller.java:59)
    at org.infinispan.loaders.file.FileCacheStore.objectFromInputStreamInReentrantMode(FileCacheStore.java:433)
    at org.infinispan.loaders.file.FileCacheStore.loadBucket(FileCacheStore.java:306)
    at org.infinispan.loaders.file.FileCacheStore.loadBucket(FileCacheStore.java:280)
    at org.infinispan.loaders.bucket.BucketBasedCacheStore.storeLockSafe(BucketBasedCacheStore.java:86)
    at org.infinispan.loaders.bucket.BucketBasedCacheStore.storeLockSafe(BucketBasedCacheStore.java:49)
    at org.infinispan.loaders.LockSupportCacheStore.store(LockSupportCacheStore.java:207)
    at org.infinispan.loaders.decorators.AbstractDelegatingStore.store(AbstractDelegatingStore.java:68)
    at org.infinispan.loaders.decorators.AsyncStore.applyModificationsSync(AsyncStore.java:245)
    at org.infinispan.loaders.decorators.AsyncStore$AsyncProcessor.put(AsyncStore.java:405)
    at org.infinispan.loaders.decorators.AsyncStore$AsyncProcessor.innerRun(AsyncStore.java:389)
    at org.infinispan.loaders.decorators.AsyncStore$AsyncProcessor.run(AsyncStore.java:312)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.EOFException: Read past end of file
    at org.jboss.marshalling.SimpleDataInput.eofOnRead(SimpleDataInput.java:126)
    at org.jboss.marshalling.SimpleDataInput.readUnsignedByteDirect(SimpleDataInput.java:263)
    at org.jboss.marshalling.SimpleDataInput.readIntDirect(SimpleDataInput.java:306)
    at org.jboss.marshalling.SimpleDataInput.readLongDirect(SimpleDataInput.java:294)
    at org.jboss.marshalling.SimpleDataInput.readLong(SimpleDataInput.java:289)
    at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:369)
    at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
    at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1673)
    at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1589)
    at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1553)
    at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1231)
    at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272)
    at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
    at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)
    at org.infinispan.container.entries.ImmortalCacheEntry$Externalizer.readObject(ImmortalCacheEntry.java:158)
    at org.infinispan.container.entries.ImmortalCacheEntry$Externalizer.readObject(ImmortalCacheEntry.java:148)
    at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:389)
    at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:220)
    at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)
    at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
    at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)
    at org.infinispan.loaders.bucket.Bucket$Externalizer.readObject(Bucket.java:157)
    at org.infinispan.loaders.bucket.Bucket$Externalizer.readObject(Bucket.java:139)
    at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:389)
    at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:220)
    at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)
    at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
    at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)
    at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:148)
    at org.infinispan.marshall.VersionAwareMarshaller.objectFromObjectStream(VersionAwareMarshaller.java:188)
    ... 15 more

 

I don't know if my config is right or how I could deal with that. Do I need to wait the rehash finished to run the node 3 ?

  • 1. Re: Errors with rehash and cache loader
    Galder Zamarreño Master

    What Infinispan version? I'd suggest trying latest 5.1.4.CR1 since we resolved some EOFException cases in the past.

  • 2. Re: Errors with rehash and cache loader
    Viet Linh Dang Newbie

    I used 5.1.3.FINAL. I will try 5.1.4.FINAL to see if it happens again.

    Thank you

  • 3. Re: Errors with rehash and cache loader
    Viet Linh Dang Newbie

    I always have theses errors with 5.1.4.FINAL.

    And I found that the rehash process is somewhat long. It took me nearly 1 hour for about 200,000 simple objects (50 MB on FileCacheStore)

  • 4. Re: Errors with rehash and cache loader
    Viet Linh Dang Newbie

    I also have warnings like this:

     

    WARN (FileCacheStore.java:270) - ISPN000060: Problems purging file /pathToStore/myNamedCache/1024
    org.infinispan.loaders.CacheLoaderException: Error while reading from file
        at org.infinispan.loaders.file.FileCacheStore.loadBucket(FileCacheStore.java:311)
        at org.infinispan.loaders.file.FileCacheStore.doPurge(FileCacheStore.java:251)
        at org.infinispan.loaders.file.FileCacheStore.purgeInternal(FileCacheStore.java:230)
        at org.infinispan.loaders.AbstractCacheStore$2.run(AbstractCacheStore.java:111)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
    Caused by: java.io.EOFException: The stream ended unexpectedly.  Please check whether the source of the stream encountered any issues generating the stream.
        at org.infinispan.marshall.VersionAwareMarshaller.objectFromObjectStream(VersionAwareMarshaller.java:191)
        at org.infinispan.marshall.AbstractDelegatingMarshaller.objectFromObjectStream(AbstractDelegatingMarshaller.java:60)
        at org.infinispan.loaders.file.FileCacheStore.objectFromInputStreamInReentrantMode(FileCacheStore.java:433)
        at org.infinispan.loaders.file.FileCacheStore.loadBucket(FileCacheStore.java:306)
        ... 6 more
    Caused by: java.io.EOFException: Read past end of file
        at org.jboss.marshalling.SimpleDataInput.eofOnRead(SimpleDataInput.java:126)
        at org.jboss.marshalling.SimpleDataInput.readUnsignedByteDirect(SimpleDataInput.java:263)
        at org.jboss.marshalling.SimpleDataInput.readIntDirect(SimpleDataInput.java:306)
        at org.jboss.marshalling.SimpleDataInput.readLongDirect(SimpleDataInput.java:294)
        at org.jboss.marshalling.SimpleDataInput.readLong(SimpleDataInput.java:289)
        at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:369)
        at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
        at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1677)
        at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1593)
        at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1557)
        at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1235)
        at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272)
        at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
        at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)
        at org.infinispan.container.entries.ImmortalCacheEntry$Externalizer.readObject(ImmortalCacheEntry.java:160)
        at org.infinispan.container.entries.ImmortalCacheEntry$Externalizer.readObject(ImmortalCacheEntry.java:150)
        at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:391)
        at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:222)
        at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)
        at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
        at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)
        at org.infinispan.loaders.bucket.Bucket$Externalizer.readObject(Bucket.java:157)
        at org.infinispan.loaders.bucket.Bucket$Externalizer.readObject(Bucket.java:139)
        at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:391)
        at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:222)
        at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)
        at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
        at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)
        at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:154)
        at org.infinispan.marshall.VersionAwareMarshaller.objectFromObjectStream(VersionAwareMarshaller.java:189)
        ... 9 more
    
  • 5. Re: Errors with rehash and cache loader
    Galder Zamarreño Master

    Do you have a unit test that we can run to reproduce this issue? Were the cache store contents generated with the same Infinispan version or with a different one?

     

    For the rehash issue, what about you replicate the issue again and generate some stracktraces from the nodes involve in the rehashing? Zip them up and attach them here.