1 2 Previous Next 15 Replies Latest reply: Apr 15, 2013 2:27 PM by kjordan2001 RSS

Read Past EOF errors with Infinispan - Hibernate Search

smnarayam Newbie

We're currently attempting to setup a 10 node Infinispan distributed cluster, used as a HibernateSearch backend. Our stack is Hibernate 3.6.4, HibernateSearch 3.4.1, Infinispan 4.2.1 Final, JGroups 2.11.1 Final and Lucene 3.1.0. Our setup is a JMS Master / Slave configuration. We are using JNDI to lookup the Infinispan CacheManager. Following the Hibernate Search recommendation, we have the Metadata & Lock Caches replicated, and we are distributing the Data cache with hash = 3.

 

Under low usage, it holds up fine. However, whenever we start to stress the system, we start seeing a lot of 'read past EOF' errors on the master during index writes.

 

We have tried various options, like switching cache store to Jdbm (based on the last comment in this post - https://forum.hibernate.org/viewtopic.php?p=2449501), async write-behind to the cache store, exclusive_index_use set to false, and with all 3 caches distributed. Switching the Metadata / Lock caches to replicated and using exclusive_index_use gave us positive improvements (got us over "java.io.IOException: No sub-file with id .fnm found (files: [.fdt, .fdx])" exceptions), but we continue to see these ''read past eof' errors.

 

We believe we must be misconfigured somewhere. Can anyone recommend a fix?

 

The errors we see are -
04-02-2012 20:45:07 ERROR Hibernate Search: Directory writer-1 impl.LogErrorHandler: Exception occurred java.io.IOException: read past EOF

java.io.IOException: read past EOF

    at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:207)

    at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:39)

    at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:92)

    at org.apache.lucene.store.BufferedIndexInput.readVInt(BufferedIndexInput.java:181)

    at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:339)

    at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)

    at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:118)

    at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:578)

    at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:684)

    at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:659)

    at org.apache.lucene.index.BufferedDeletes.applyDeletes(BufferedDeletes.java:283)

    at org.apache.lucene.index.BufferedDeletes.applyDeletes(BufferedDeletes.java:191)

    at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3358)

    at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3296)

    at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:3159)

    at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3232)

    at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3214)

    at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3198)

    at org.hibernate.search.backend.Workspace.commitIndexWriter(Workspace.java:220)

    at org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor.run(PerDPQueueProcessor.java:109)

    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)

    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

    at java.util.concurrent.FutureTask.run(FutureTask.java:138)

    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

    at java.lang.Thread.run(Thread.java:662)


Another variation we see is -

04-02-2012 14:37:43 ERROR Lucene Merge Thread #8 impl.LogErrorHandler: Exception occurred org.apache.lucene.index.MergePolicy$MergeException: java.io.IOExcept

org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException: read past EOF

    at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:517)

    at org.hibernate.search.backend.impl.lucene.overrides.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:49)

    at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)

Caused by: java.io.IOException: read past EOF

    at org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:138)

    at org.apache.lucene.index.SegmentReader$Norm.bytes(SegmentReader.java:409)

    at org.apache.lucene.index.SegmentReader$Norm.bytes(SegmentReader.java:404)

    at org.apache.lucene.index.SegmentReader.norms(SegmentReader.java:1084)

    at org.apache.lucene.index.SegmentMerger.mergeNorms(SegmentMerger.java:636)

    at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:112)

    at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3938)

    at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3614)

    at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:388)

    at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:456)

 

Our Infinispan configuration is -

 

<?xml version="1.0" encoding="UTF-8"?>

<infinispan

    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

    xsi:schemaLocation="urn:infinispan:config:4.2 http://www.infinispan.org/schemas/infinispan-config-4.2.xsd"

    xmlns="urn:infinispan:config:4.2">

   <!-- *************************** -->

    <!-- System-wide global settings -->

    <!-- *************************** -->

      <global>

          <!-- Duplicate domains are allowed so that multiple deployments with default configuration

            of Hibernate Search applications work - if possible it would be better to use JNDI to share

            the CacheManager across applications -->

        <globalJmxStatistics

            enabled="true"

            cacheManagerName="HibernateSearch"

            allowDuplicateDomains="true" />

 

        <!-- If the transport is omitted, there is no way to create distributed or clustered

            caches. There is no added cost to defining a transport but not creating a cache that uses one,

            since the transport is created and initialized lazily. -->

        <transport

            clusterName="HibernateSearch-Infinispan-cluster-MT"

            distributedSyncTimeout="90000">

            <properties>

                <property name="configurationFile" value="infinispan-tcp.cfg.xml" />

            </properties>

            <!-- Note that the JGroups transport uses sensible defaults if no configuration

                property is defined. See the JGroupsTransport javadocs for more flags -->

        </transport>

 

        <!-- Used to register JVM shutdown hooks. hookBehavior: DEFAULT, REGISTER, DONT_REGISTER.

            Hibernate Search takes care to stop the CacheManager so registering is not needed -->

        <shutdown

            hookBehavior="DONT_REGISTER" />

      </global>

 

    <!-- *************************** -->

    <!-- Default "template" settings -->

    <!-- *************************** -->

      <default>

          <locking

            lockAcquisitionTimeout="20000"

            writeSkewCheck="false"

            concurrencyLevel="5000"

            useLockStriping="false" />

 

        <lazyDeserialization

            enabled="false" />

 

        <!-- Invocation batching is required for use with the Lucene Directory -->

        <invocationBatching

            enabled="true" />

 

        <!-- This element specifies that the cache is clustered. modes supported: distribution

            (d), replication (r) or invalidation (i). Don't use invalidation to store Lucene indexes (as

            with Hibernate Search DirectoryProvider). Replication is recommended for best performance of

            Lucene indexes, but make sure you have enough memory to store the index in your heap.

            Also distribution scales much better than replication on high number of nodes in the cluster. -->

        <clustering

            mode="d">

            <sync replTimeout="90000" />

            <l1 enabled="false" />

        </clustering>

 

        <jmxStatistics

            enabled="true" />

 

        <eviction

            maxEntries="-1"

            strategy="NONE" />

 

        <expiration

            maxIdle="-1" />

    </default>

 

    <!-- ******************************************************************************* -->

    <!-- Individually configured "named" caches.                                         -->

    <!--                                                                                 -->

    <!-- While default configuration happens to be fine with similar settings across the -->

    <!-- three caches, they should generally be different in a production environment.   -->

    <!--                                                                                 -->

    <!-- Current settings could easily lead to OutOfMemory exception as a CacheStore     -->

    <!-- should be enabled, and maybe distribution is desired.                           -->

    <!-- ******************************************************************************* -->

 

    <!-- *************************************** -->

    <!--  Cache to store Lucene's file metadata  -->

    <!-- *************************************** -->

    <namedCache

        name="LuceneIndexesMetadata">

        <clustering

            mode="replication">

            <stateRetrieval fetchInMemoryState="true" logFlushTimeout="30000" />

            <sync replTimeout="90000" />

            <l1 enabled="false" />

        </clustering>

        <loaders shared="true" preload="true">

            <loader class="org.infinispan.loaders.file.FileCacheStore" fetchPersistentState="false" ignoreModifications="false" purgeOnStartup="false">

                <properties>

                    <property name="location" value="/usr/local/tc/.index/metadata" />

                </properties>

            </loader>

        </loaders>

    </namedCache>

 

    <!-- **************************** -->

    <!--  Cache to store Lucene data  -->

    <!-- **************************** -->

    <namedCache

        name="LuceneIndexesData">

        <clustering

            mode="d">

            <hash numOwners="3" />

            <sync replTimeout="90000" />

            <l1 enabled="false" />

        </clustering>

        <loaders shared="true" preload="true">

            <loader class="org.infinispan.loaders.file.FileCacheStore" fetchPersistentState="false" ignoreModifications="false" purgeOnStartup="false">

                <properties>

                    <property name="location" value="/usr/local/tc/.index/data" />

                </properties>

            </loader>

        </loaders>

    </namedCache>

 

    <!-- ***************************** -->

    <!--  Cache to store Lucene locks  -->

    <!-- ***************************** -->

    <namedCache

        name="LuceneIndexesLocking">

        <clustering

            mode="replication">

            <stateRetrieval fetchInMemoryState="true" logFlushTimeout="30000" />

            <sync replTimeout="90000" />

            <l1 enabled="false" />

        </clustering>

    </namedCache>

   

</infinispan>

 

Our hibernate configuration on the master -

 

        <property name="hibernate.search.default.directory_provider">infinispan</property>

        <property name="hibernate.search.infinispan.cachemanager_jndiname">InfinispanCacheManager</property>

        <property name="hibernate.search.infinispan.chunk_size">40960</property>

        <property name="hibernate.search.default.exclusive_index_use">true</property>

 

Hibernate configuration on the slaves -

 

        <property name="hibernate.search.default.directory_provider">infinispan</property>

        <property name="hibernate.search.infinispan.cachemanager_jndiname">InfinispanCacheManager</property>

        <property name="hibernate.search.infinispan.chunk_size">40960</property>

        <property name="hibernate.search.worker.backend">jms</property>

        <property name="hibernate.search.worker.jndi.class">org.apache.naming.java.javaURLContextFactory</property>               

        <property name="hibernate.search.worker.jms.connection_factory">ConnectionFactory</property>

        <property name="hibernate.search.worker.jms.queue">/queue/hibernateSearchQueue</property>


  • 1. Re: Read Past EOF errors with Infinispan - Hibernate Search
    Sanne Grinovero Master

    Hi,

    this was being caused by subtle race conditions and small bugs in both Lucene and Infinispan, but it's a long time ago, these versions are quite old; it would require several backports in both projects to fix, which is not practical: I'd advise to upgrade.

     

    I've got very good results from testing the latest stable versions:

    • Apache Lucene 3.5
    • Infinispan 5.1.3.FINAL
    • Hibernate Search 4.1.0.Final

     

    You would get Hibernate ORM 4.1.2.Final as well, which includes a lot of improvements; I realize it's quite some work to update it all, but there are many other reasons for this to be a good idea.

     

    If you really can't upgrade, or can't now, you can workaround the problem by using a chunk_size
    which is large enough to contain the full segment. It will require some experimentation to find a good value for you, as if you set it way too high you'll be consuming more memory than necessary.

  • 2. Re: Read Past EOF errors with Infinispan - Hibernate Search
    smnarayam Newbie

    Hi Sanne,

     

    Thanks for your fast response.

     

    Unfortunately, we are not in a position to upgrade right now. We'll experiment with the chunk_size, and report back our findings.

     

    Thanks.

  • 3. Re: Read Past EOF errors with Infinispan - Hibernate Search
    Sanne Grinovero Master

    thanks, please keep me updated.

  • 4. Re: Read Past EOF errors with Infinispan - Hibernate Search
    Francisco Lozano Newbie

    I am having a similar problem but running with the latest versions (Lucene 3.5.0, Infinispan 5.1.2/5.1.3 - tried both). No hibernate search, just pure lucene usage here.

     

    My configuration for tests is:

     

    <?xml version="1.0" encoding="UTF-8"?>

    <infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

              xsi:schemaLocation="urn:infinispan:config:5.1 http://www.infinispan.org/schemas/infinispan-config-5.1.xsd"

              xmlns="urn:infinispan:config:5.1">

              <global>

              </global>

              <default>

                        <locking lockAcquisitionTimeout="20000" />

                        <invocationBatching enabled="true" />

                        <eviction maxEntries="-1" strategy="NONE" />

                        <expiration maxIdle="-1" />

              </default>

     

     

              <namedCache name="indexMetadata">

                        <loaders passivation="false" shared="false" preload="false">

                                  <loader class="org.infinispan.loaders.jdbm.JdbmCacheStore"

                                            fetchPersistentState="true" purgeSynchronously="false"

                                            purgeOnStartup="false">

                                            <async enabled="true" threadPoolSize="10"/>

                                            <properties>

                                                      <property name="location" value="/tmp/infinispan/index_metadata" />

                                            </properties>

                                  </loader>

                        </loaders>

              </namedCache>

     

     

              <namedCache name="indexChunks">

                        <eviction wakeUpInterval="5000" maxEntries="100000" strategy="LIRS" />

                        <loaders shared="false" preload="false">

                                  <loader class="org.infinispan.loaders.jdbm.JdbmCacheStore"

                                            fetchPersistentState="true" purgeSynchronously="false"

                                            purgeOnStartup="false">

                                            <async enabled="true" threadPoolSize="10"/>

                                            <properties>

                                                      <property name="location" value="/tmp/infinispan/index_chunks" />

                                            </properties>

                                  </loader>

                        </loaders>

              </namedCache>

     

     

              <namedCache name="indexDistLocks">

              </namedCache>

     

     

    </infinispan>

     

    I had to add the async option because without it the write performance was so slow that it was totally unusable.

     

    EDIT: Now I'm able to reproduce everywhere. If I remove the async clause it doesn't reproduce.

     

    The 1st time I open an index in a set of tests, it works well. Then I shutdown everything (whole Spring context), and start everything for the next test, and when I try to get the index I get:

     

    java.io.IOException: Read past EOF

    at org.infinispan.lucene.SingleChunkIndexInput.readByte(SingleChunkIndexInput.java:76) ~[infinispan-lucene-directory-5.1.3.FINAL.jar:5.1.3.FINAL]

    at org.apache.lucene.store.ChecksumIndexInput.readByte(ChecksumIndexInput.java:41) ~[lucene-core-3.5.0.jar:3.5.0 1204988 - simon - 2011-11-22 14:46:51]

    at org.apache.lucene.store.DataInput.readInt(DataInput.java:84) ~[lucene-core-3.5.0.jar:3.5.0 1204988 - simon - 2011-11-22 14:46:51]

    at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:272) ~[lucene-core-3.5.0.jar:3.5.0 1204988 - simon - 2011-11-22 14:46:51]

    at org.apache.lucene.index.IndexFileDeleter.<init>(IndexFileDeleter.java:182) ~[lucene-core-3.5.0.jar:3.5.0 1204988 - simon - 2011-11-22 14:46:51]

    at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1178) ~[lucene-core-3.5.0.jar:3.5.0 1204988 - simon - 2011-11-22 14:46:51]

    ...

     

    Or similar stuff, like:

    java.io.FileNotFoundException: Error loading medatada for index file: _6.fnm|M|idx12345678

              at org.infinispan.lucene.InfinispanDirectory.openInput(InfinispanDirectory.java:282) ~[infinispan-lucene-directory-5.1.3.FINAL.jar:5.1.3.FINAL]

              at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:74) ~[lucene-core-3.5.0.jar:3.5.0 1204988 - simon - 2011-11-22 14:46:51]

              at org.apache.lucene.index.IndexWriter.getFieldInfos(IndexWriter.java:1222) ~[lucene-core-3.5.0.jar:3.5.0 1204988 - simon - 2011-11-22 14:46:51]

              at org.apache.lucene.index.IndexWriter.getCurrentFieldInfos(IndexWriter.java:1238) ~[lucene-core-3.5.0.jar:3.5.0 1204988 - simon - 2011-11-22 14:46:51]

              at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1171) ~[lucene-core-3.5.0.jar:3.5.0 1204988 - simon - 2011-11-22 14:46:51]

     

    I have tried with BerkeleyDB backend and I have the same result.

     

    I setup the caches in Spring with:

     

    <bean factory-bean="infinispanDefaultCacheManager" factory-method="getCache"

    id="infinispanIndexMetadataCache">

    <constructor-arg value="indexMetadata" />

    </bean>

     

     

    <bean factory-bean="infinispanDefaultCacheManager" factory-method="getCache"

    id="infinispanIndexChunksCache">

    <constructor-arg value="indexChunks" />

    </bean>

     

     

    <bean factory-bean="infinispanDefaultCacheManager" factory-method="getCache"

    id="infinispanIndexDistLocksCache">

    <constructor-arg value="indexDistLocks" />

    </bean>

     

     

    <bean factory-bean="infinispanDefaultCacheManager" factory-method="getCache"

    id="infinispanIndexDatabaseCache">

    <constructor-arg value="indexDatabase" />

    </bean>

     

     

    <bean class="org.infinispan.manager.DefaultCacheManager" id="infinispanDefaultCacheManager"

    init-method="start" destroy-method="stop">

    <constructor-arg value="infinispan-test-bdb.xml" />

    </bean>

     

    Shall I do some kind of shutdown hook or anything like that?

  • 5. Re: Read Past EOF errors with Infinispan - Hibernate Search
    Francisco Lozano Newbie

    I have checked and added <eviction to every <namedCache>, and it happens something interesting:

    - If I put a very small maxEntries value (eg: 5), it reproduces immediately. If I put a very large maxEntries value (eg: 100000) it does not reproduce - which makes me think that the problem is that the async cache store is not properly "flushed" before shutdown. I have added the init-method="start" destroy-method="stop" to the DefaultCacheManager definition, but maybe I'm missing something :?

     

    EDIT:

    I have fixed the problem above. After each test, the CacheManager is properly closed and I get in the logs:

    INFO o.i.eviction.PassivationManagerImpl.passivateAll - ISPN000029: Passivating all entries to disk

    INFO o.i.eviction.PassivationManagerImpl.passivateAll - ISPN000030: Passivated 0 entries in 2 milliseconds

    INFO o.i.eviction.PassivationManagerImpl.passivateAll - ISPN000029: Passivating all entries to disk

    INFO o.i.eviction.PassivationManagerImpl.passivateAll - ISPN000030: Passivated 4 entries in 1 milliseconds

    INFO o.i.eviction.PassivationManagerImpl.passivateAll - ISPN000029: Passivating all entries to disk

    INFO o.i.eviction.PassivationManagerImpl.passivateAll - ISPN000030: Passivated 4 entries in 0 milliseconds


    But this was not the cause of the problem, I still have messages like:

    java.io.FileNotFoundException: Error loading medatada for index file: segments_1|M|idx12345678

              at org.infinispan.lucene.InfinispanDirectory.openInput(InfinispanDirectory.java:282) ~[infinispan-lucene-directory-5.1.3.FINAL.jar:5.1.3.FINAL]

              at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:265) ~[lucene-core-3.5.0.jar:3.5.0 1204988 - simon - 2011-11-22 14:46:51]

              at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:363) ~[lucene-core-3.5.0.jar:3.5.0 1204988 - simon - 2011-11-22 14:46:51]

              at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:754) ~[lucene-core-3.5.0.jar:3.5.0 1204988 - simon - 2011-11-22 14:46:51]

              at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:593) ~[lucene-core-3.5.0.jar:3.5.0 1204988 - simon - 2011-11-22 14:46:51]

              at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:359) ~[lucene-core-3.5.0.jar:3.5.0 1204988 - simon - 2011-11-22 14:46:51]

              at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1148) ~[lucene-core-3.5.0.jar:3.5.0 1204988 - simon - 2011-11-22 14:46:51]

     

     

    And:

     

     

    java.io.FileNotFoundException: Error loading medatada for index file: _0.fnm|M|idx12345678

              at org.infinispan.lucene.InfinispanDirectory.openInput(InfinispanDirectory.java:282) ~[infinispan-lucene-directory-5.1.3.FINAL.jar:5.1.3.FINAL]

              at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:74) ~[lucene-core-3.5.0.jar:3.5.0 1204988 - simon - 2011-11-22 14:46:51]

              at org.apache.lucene.index.IndexWriter.getFieldInfos(IndexWriter.java:1222) ~[lucene-core-3.5.0.jar:3.5.0 1204988 - simon - 2011-11-22 14:46:51]

              at org.apache.lucene.index.IndexWriter.getCurrentFieldInfos(IndexWriter.java:1238) ~[lucene-core-3.5.0.jar:3.5.0 1204988 - simon - 2011-11-22 14:46:51]

              at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1171) ~[lucene-core-3.5.0.jar:3.5.0 1204988 - simon - 2011-11-22 14:46:51].

     

     

    And a new one:

     

    INFO o.i.eviction.PassivationManagerImpl.passivateAll - ISPN000029: Passivating all entries to disk

    INFO o.i.eviction.PassivationManagerImpl.passivateAll - ISPN000030: Passivated 0 entries in 0 milliseconds

    INFO o.i.eviction.PassivationManagerImpl.passivateAll - ISPN000029: Passivating all entries to disk

    INFO o.i.eviction.PassivationManagerImpl.passivateAll - ISPN000030: Passivated 361 entries in 0 milliseconds

    ERROR o.i.loaders.decorators.AsyncStore.run - ISPN000051: Unexpected error

    org.infinispan.CacheException: Unable to acquire lock on update map

              at org.infinispan.loaders.decorators.AsyncStore.acquireLock(AsyncStore.java:293) ~[infinispan-core-5.1.3.FINAL.jar:5.1.3.FINAL]

              at org.infinispan.loaders.decorators.AsyncStore.access$900(AsyncStore.java:86) ~[infinispan-core-5.1.3.FINAL.jar:5.1.3.FINAL]

              at org.infinispan.loaders.decorators.AsyncStore$AsyncProcessor.innerRun(AsyncStore.java:336) ~[infinispan-core-5.1.3.FINAL.jar:5.1.3.FINAL]

              at org.infinispan.loaders.decorators.AsyncStore$AsyncProcessor.run(AsyncStore.java:312) ~[infinispan-core-5.1.3.FINAL.jar:5.1.3.FINAL]

              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) ~[na:1.6.0_31]

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) ~[na:1.6.0_31]

              at java.lang.Thread.run(Thread.java:662) ~[na:1.6.0_31]

     

     

    Right now my infinispan configuration file looks like (single-node setup for tests):

     

    <?xml version="1.0" encoding="UTF-8"?>

    <infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

              xsi:schemaLocation="urn:infinispan:config:5.1 http://www.infinispan.org/schemas/infinispan-config-5.1.xsd"

              xmlns="urn:infinispan:config:5.1">

              <global>

              </global>

              <default>

                        <locking lockAcquisitionTimeout="20000" />

                        <invocationBatching enabled="true" />

                        <eviction maxEntries="-1" strategy="NONE" />

                        <expiration maxIdle="-1" />

              </default>

     

     

              <namedCache name="indexMetadata">

                        <eviction wakeUpInterval="5000" maxEntries="1000" strategy="LRU" />

                        <loaders shared="false" preload="true" passivation="true">

                                  <loader class="org.infinispan.loaders.bdbje.BdbjeCacheStore"

                                            fetchPersistentState="true" purgeSynchronously="false"

                                            purgeOnStartup="false">

                                            <async enabled="true" threadPoolSize="10" />

                                            <properties>

                                                      <property name="location" value="/tmp/infinispan/index_metadata" />

                                            </properties>

                                  </loader>

                        </loaders>

              </namedCache>

     

     

              <namedCache name="indexChunks" >

                        <eviction wakeUpInterval="5000" maxEntries="1000" strategy="LRU" />

                        <loaders shared="false" preload="false" passivation="true">

                                  <loader class="org.infinispan.loaders.bdbje.BdbjeCacheStore"

                                            fetchPersistentState="true" purgeSynchronously="false"

                                            purgeOnStartup="false">

                                            <async enabled="true" threadPoolSize="10" />

                                            <properties>

                                                      <property name="location" value="/tmp/infinispan/index_chunks" />

                                            </properties>

                                  </loader>

                        </loaders>

              </namedCache>

     

     

              <namedCache name="indexDistLocks">

              </namedCache>

     

     

    </infinispan>

  • 6. Re: Read Past EOF errors with Infinispan - Hibernate Search
    Francisco Lozano Newbie

    Sorry to update this thread so frequently, but I hope it's useful for someone else...

     

    I just changed my "purgeSynchronously" to true for every cache, and added flushLockTimeout="60000" to every <async /> clause and the problem seems to be solved it still reproduces - or, at least, does not reproduce in my tests.

    (Actually the problem seems only solved with BerkeleyDB, but with Jdbm it still fails until I disable also the "preload" of the index metadata - I'll open a new thread for that to avoid confusion).

  • 7. Re: Read Past EOF errors with Infinispan - Hibernate Search
    smnarayam Newbie

    Hi Sanne,

     

    We were unable to get past the 'read past eof' errors with a distributed cluster. Finally, we reverted back to a replicated setup. With replication, we are able to run the same stress tests without any errors.

     

    For now, we are planning to move ahead with replication. In the coming months, we plan to upgrade Infinispan, Lucene etc, and we will try distribution again after the upgrade.

     

    Thanks.

  • 8. Re: Read Past EOF errors with Infinispan - Hibernate Search
    Sanne Grinovero Master

    right that seems a good choice. Please let me know if you have any problem after upgrading!

    Sanne

  • 9. Re: Read Past EOF errors with Infinispan - Hibernate Search
    Francisco Lozano Newbie

    Hi Sanne,

     

    Actually I am reproducing the same issue with 5.1.3 / lucene 3.5.0. I created a test in git for it: https://github.com/flozano/infinispan-lucene-eof-problem. May I ask you to take a look and tell me if I'm doing something wrong or if I'm hitting a bug actually?

  • 10. Re: Read Past EOF errors with Infinispan - Hibernate Search
    Sanne Grinovero Master

    Hi Francisco,

    thanks for the tests, that's very interesting and I'm glad to look into it in the weekend, unfortunately I can't before that as I'm away.

  • 11. Re: Read Past EOF errors with Infinispan - Hibernate Search
    Francisco Lozano Newbie

    That'd be great Sanne, thanks a lot!

  • 12. Re: Read Past EOF errors with Infinispan - Hibernate Search
    Sanne Grinovero Master

    Hi Francisco,

    sorry for the delay, I've been very busy and then.. I forgot.

     

    I've opened https://issues.jboss.org/browse/ISPN-2046 to track this directly.

     

    Your test is amazing: it has been very helpfull to reproduce it. The good news is it helped be to immediately spot a just-introduced problem ISPN-2049, but then I realized the branch you where testing wasn't affected by this.

     

    The main problem, as you had guessed already, lies with async stores:  ISPN-1174.

    Until that is solved, it's not safe to use an async CacheLoader with data strictures which can't deal with it; Lucene definitely not.

  • 13. Re: Read Past EOF errors with Infinispan - Hibernate Search
    Francisco Lozano Newbie

    Hi Sanne,

     

    Thanks a lot for taking care of this. I also lost track of this problem it was an experiment, not a project-with-deadline.

     

    I see the JIRA is scheduled for fix in 5.2.0.Final - that would be great, looks like a very complex issue...

  • 14. Re: Read Past EOF errors with Infinispan - Hibernate Search
    kjordan2001 Newbie

    I get a similar error with and without async turned on in 5.2.4 when I try to start up a 3rd node:

    java.io.IOException: Read past EOF

            at org.infinispan.lucene.SingleChunkIndexInput.readByte(SingleChunkIndexInput.java:77)

            at org.apache.lucene.store.DataInput.readVInt(DataInput.java:107)

            at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:314)

            at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:74)

            at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:80)

            at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116)

            at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:94)

            at org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:105)

            at org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(ReadOnlyDirectoryReader.java:27)

            at org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:78)

            at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:709)

            at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:72)

            at org.apache.lucene.index.IndexReader.open(IndexReader.java:256)

            at org.hibernate.search.indexes.impl.SharingBufferReaderProvider.readerFactory(SharingBufferR

    eaderProvider.java:143)

            at org.hibernate.search.indexes.impl.SharingBufferReaderProvider$PerDirectoryLatestReader.<in

    it>(SharingBufferReaderProvider.java:217)

            at org.hibernate.search.indexes.impl.SharingBufferReaderProvider.createReader(SharingBufferRe

    aderProvider.java:120)

     

    Sometimes I also get:

    Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: org.infinispan.lucene.locking.BaseLuceneLock@1caf7e42

            at org.apache.lucene.store.Lock.obtain(Lock.java:84)

            at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1098)

            at org.hibernate.search.store.impl.DirectoryProviderHelper.initializeIndexIfNeeded(DirectoryProviderHelper.java:157)

     

    All the nodes seem to be configured correctly as I can start 2 up but no matter which one the 3rd one is, it gets one of those errors.

     

    I have it configured with JGroups JDBC and a JDBC CacheStore.  So far Infinispan has been pretty aggravating for me as I also had a very hard time getting it set up due to errors that shouldn't have happened since I used an equivalent configuration and then it worked fine (and never got it working with S3 as a store): https://community.jboss.org/thread/222803

     

    For the second error:

    2013-04-11 16:52:12,501 [org.infinispan.lucene.locking.BaseLuceneLock] TRACE: Lock: write.lock not aquired for index: <myindex>, was taken already.

     

    Which doesn't make any sense since no writing was being done to the index at that point in time.  I even cleared out the backing store for it to make sure no corruption occured before starting the nodes.  Node 1 recreated the index.  Then I started Node 2 and that went fine.  But then Node 3 gives an error always.

1 2 Previous Next