5 Replies Latest reply on Aug 19, 2015 11:48 PM by prashant.thakur

    near-real-time settings for Indexing with Infinispan Cache as backend

    prashant.thakur

      Can we use near-real-time as IndexManager settings while providing inifinispan as a directory_provider.

      This setting works when we use single node but with multiple node cluster we are facing multiple locking issues.

      Can someone please share if we are missing something or some extra parameters need to be set for using this combination.

        • 1. Re: near-real-time settings for Indexing with Infinispan Cache as backend
          prashant.thakur

          Following is exception seen

           

           

          INFO [org.infinispan.query.impl.massindex.IndexUpdater] [pool-16-thread-1] [2015-08-18 10:36:13,860] (IndexUpdater.java:48) - ISPN014016: Purging instances of 'org.infinispan.query.remote.indexing.ProtobufValueWrapper' from the index

           

           

          ERROR [org.hibernate.search.exception.impl.LogErrorHandler] [pool-16-thread-1] [2015-08-18 10:36:18,997] (LogErrorHandler.java:67) - HSEARCH000058: HSEARCH000117: IOException on the IndexWriter

          org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: org.infinispan.lucene.locking.BaseLuceneLock@a357128

                  at org.apache.lucene.store.Lock.obtain(Lock.java:89)

                  at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:755)

                  at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.createNewIndexWriter(IndexWriterHolder.java:138)

                  at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.getIndexWriter(IndexWriterHolder.java:98)

                  at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.getIndexWriter(IndexWriterHolder.java:117)

                  at org.hibernate.search.backend.impl.lucene.AbstractWorkspaceImpl.getIndexWriter(AbstractWorkspaceImpl.java:108)

                  at org.hibernate.search.backend.impl.lucene.LuceneBackendTaskStreamer.doWork(LuceneBackendTaskStreamer.java:45)

                  at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueProcessor.applyStreamWork(LuceneBackendQueueProcessor.java:76)

                  at org.hibernate.search.indexes.spi.DirectoryBasedIndexManager.performStreamOperation(DirectoryBasedIndexManager.java:107)

                  at org.infinispan.query.impl.massindex.ExtendedBatchBackend.purge(ExtendedBatchBackend.java:38)

                  at org.infinispan.query.impl.massindex.IndexUpdater.purge(IndexUpdater.java:49)

                  at org.infinispan.query.impl.massindex.IndexWorker.preIndex(IndexWorker.java:54)

                  at org.infinispan.query.impl.massindex.IndexWorker.call(IndexWorker.java:75)

                  at org.infinispan.query.impl.massindex.IndexWorker.call(IndexWorker.java:31)

                  at org.infinispan.commands.read.DistributedExecuteCommand.perform(DistributedExecuteCommand.java:98)

                  at org.infinispan.distexec.DefaultExecutorService$LocalDistributedTaskPart$1.doLocalInvoke(DefaultExecutorService.java:1104)

                  at org.infinispan.distexec.DefaultExecutorService$LocalDistributedTaskPart$1.call(DefaultExecutorService.java:1094)

                  at java.util.concurrent.FutureTask.run(FutureTask.java:266)

                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

                  at java.lang.Thread.run(Thread.java:745)

          ERROR [org.hibernate.search.backend.impl.lucene.LuceneBackendTaskStreamer] [pool-16-thread-1] [2015-08-18 10:36:18,999] (LuceneBackendTaskStreamer.java:47) - HSEARCH000072: Couldn't open the IndexWriter because of previous error: operation skipped, index ouf of sync!

           

          ERROR [org.hibernate.search.exception.impl.LogErrorHandler] [pool-16-thread-1] [2015-08-18 10:36:24,076] (LogErrorHandler.java:67) - HSEARCH000058: HSEARCH000117: IOException on the IndexWriter

          org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: org.infinispan.lucene.locking.BaseLuceneLock@a357128

                  at org.apache.lucene.store.Lock.obtain(Lock.java:89)

                  at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:755)

                  at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.createNewIndexWriter(IndexWriterHolder.java:138)

                  at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.getIndexWriter(IndexWriterHolder.java:98)

                  at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.getIndexWriter(IndexWriterHolder.java:117)

                  at org.hibernate.search.backend.impl.lucene.AbstractWorkspaceImpl.getIndexWriter(AbstractWorkspaceImpl.java:108)

                  at org.hibernate.search.backend.impl.lucene.LuceneBackendTaskStreamer.doWork(LuceneBackendTaskStreamer.java:45)

                  at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueProcessor.applyStreamWork(LuceneBackendQueueProcessor.java:76)

                  at org.hibernate.search.indexes.spi.DirectoryBasedIndexManager.performStreamOperation(DirectoryBasedIndexManager.java:107)

                  at org.hibernate.search.backend.impl.batch.DefaultBatchBackend.flush(DefaultBatchBackend.java:76)

                  at org.infinispan.query.impl.massindex.ExtendedBatchBackend.flush(ExtendedBatchBackend.java:57)

                  at org.infinispan.query.impl.massindex.ExtendedBatchBackend.purge(ExtendedBatchBackend.java:42)

                  at org.infinispan.query.impl.massindex.IndexUpdater.purge(IndexUpdater.java:49)

                  at org.infinispan.query.impl.massindex.IndexWorker.preIndex(IndexWorker.java:54)

                  at org.infinispan.query.impl.massindex.IndexWorker.call(IndexWorker.java:75)

                  at org.infinispan.query.impl.massindex.IndexWorker.call(IndexWorker.java:31)

                  at org.infinispan.commands.read.DistributedExecuteCommand.perform(DistributedExecuteCommand.java:98)

                  at org.infinispan.distexec.DefaultExecutorService$LocalDistributedTaskPart$1.doLocalInvoke(DefaultExecutorService.java:1104)

                  at org.infinispan.distexec.DefaultExecutorService$LocalDistributedTaskPart$1.call(DefaultExecutorService.java:1094)

                  at java.util.concurrent.FutureTask.run(FutureTask.java:266)

                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

                  at java.lang.Thread.run(Thread.java:745)

          • 2. Re: near-real-time settings for Indexing with Infinispan Cache as backend
            gustavonalle

            It's technically possible to use the near-real-time index manager with Infinispan directory, but the caches that will hold the index (LuceneIndexesLocking,LuceneIndexesData and LuceneIndexesMetadata) need to be LOCAL, otherwise they will be competing to get the Lock when writing to the index.

            • 3. Re: near-real-time settings for Indexing with Infinispan Cache as backend
              prashant.thakur

              the indexing starts but all entries are not getting indexed . We see locking issue after half of data is being indexed

              Below is the error seen

              ERROR [org.hibernate.search.exception.impl.LogErrorHandler] [remote-thread--p2-t4] [2015-08-19 05:46:50,333] (LogErrorHandler.java:67) - HSEARCH000058: HSEARCH000117: IOException on the IndexWriter

              org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: org.infinispan.lucene.locking.BaseLuceneLock@1bc8d0bf

                      at org.apache.lucene.store.Lock.obtain(Lock.java:89)

                      at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:755)

                      at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.createNewIndexWriter(IndexWriterHolder.java:138)

                      at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.getIndexWriter(IndexWriterHolder.java:98)

                      at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.getIndexWriter(IndexWriterHolder.java:117)

                      at org.hibernate.search.backend.impl.lucene.AbstractWorkspaceImpl.getIndexWriter(AbstractWorkspaceImpl.java:108)

                      at org.hibernate.search.backend.impl.lucene.LuceneBackendTaskStreamer.doWork(LuceneBackendTaskStreamer.java:45)

                      at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueProcessor.applyStreamWork(LuceneBackendQueueProcessor.java:76)

                      at org.hibernate.search.indexes.spi.DirectoryBasedIndexManager.performStreamOperation(DirectoryBasedIndexManager.java:107)

                      at org.infinispan.query.impl.massindex.ExtendedBatchBackend.purge(ExtendedBatchBackend.java:38)

                      at org.infinispan.query.impl.massindex.IndexUpdater.purge(IndexUpdater.java:49)

                      at org.infinispan.query.impl.massindex.IndexWorker.preIndex(IndexWorker.java:54)

                      at org.infinispan.query.impl.massindex.IndexWorker.call(IndexWorker.java:75)

                      at org.infinispan.query.impl.massindex.IndexWorker.call(IndexWorker.java:31)

                      at org.infinispan.commands.read.DistributedExecuteCommand.perform(DistributedExecuteCommand.java:98)

                      at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokePerform(BasePerCacheInboundInvocationHandler.java:92)

                      at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:34)

                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

                      at java.lang.Thread.run(Thread.java:745)

              ERROR [org.hibernate.search.backend.impl.lucene.LuceneBackendTaskStreamer] [remote-thread--p2-t4] [2015-08-19 05:46:50,334] (LuceneBackendTaskStreamer.java:47) - HSEARCH000072: Couldn't open the IndexWriter because of previous error: operation skipped, index ouf of sync!

              WARN [org.jgroups.protocols.MPING] [MPING] [2015-08-19 05:46:54,624] (Discovery.java:294) - a8140cda-41ac-7943-b66a-157b88da98d7: discarding discovery request for cluster 'MyGalaxy1' from a8140cda-41ac-7943-b66a-157b88da98d7; our cluster name is 'PST_CLUSTER_TEST'. Please separate your clusters properly

              ERROR [org.hibernate.search.exception.impl.LogErrorHandler] [pool-16-thread-1] [2015-08-19 05:46:55,417] (LogErrorHandler.java:67) - HSEARCH000058: HSEARCH000117: IOException on the IndexWriter

              org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: org.infinispan.lucene.locking.BaseLuceneLock@1bc8d0bf

                      at org.apache.lucene.store.Lock.obtain(Lock.java:89)

                      at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:755)

                      at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.createNewIndexWriter(IndexWriterHolder.java:138)

                      at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.getIndexWriter(IndexWriterHolder.java:98)

                      at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.getIndexWriter(IndexWriterHolder.java:117)

                      at org.hibernate.search.backend.impl.lucene.AbstractWorkspaceImpl.getIndexWriter(AbstractWorkspaceImpl.java:108)

                      at org.hibernate.search.backend.impl.lucene.LuceneBackendTaskStreamer.doWork(LuceneBackendTaskStreamer.java:45)

                      at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueProcessor.applyStreamWork(LuceneBackendQueueProcessor.java:76)

                      at org.hibernate.search.indexes.spi.DirectoryBasedIndexManager.performStreamOperation(DirectoryBasedIndexManager.java:107)

                      at org.infinispan.query.impl.massindex.ExtendedBatchBackend.purge(ExtendedBatchBackend.java:38)

                      at org.infinispan.query.impl.massindex.IndexUpdater.purge(IndexUpdater.java:49)

                      at org.infinispan.query.impl.massindex.IndexWorker.preIndex(IndexWorker.java:54)

                      at org.infinispan.query.impl.massindex.IndexWorker.call(IndexWorker.java:75)

                      at org.infinispan.query.impl.massindex.IndexWorker.call(IndexWorker.java:31)

                      at org.infinispan.commands.read.DistributedExecuteCommand.perform(DistributedExecuteCommand.java:98)

                      at org.infinispan.distexec.DefaultExecutorService$LocalDistributedTaskPart$1.doLocalInvoke(DefaultExecutorService.java:1104)

                      at org.infinispan.distexec.DefaultExecutorService$LocalDistributedTaskPart$1.call(DefaultExecutorService.java:1094)

                      at java.util.concurrent.FutureTask.run(FutureTask.java:266)

                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

                      at java.lang.Thread.run(Thread.java:745)

              ERROR [org.hibernate.search.backend.impl.lucene.LuceneBackendTaskStreamer] [pool-16-thread-1] [2015-08-19 05:46:55,418] (LuceneBackendTaskStreamer.java:47) - HSEARCH000072: Couldn't open the IndexWriter because of previous error: operation skipped, index ouf of sync!

              ERROR [org.hibernate.search.exception.impl.LogErrorHandler] [remote-thread--p2-t1] [2015-08-19 05:47:00,511] (LogErrorHandler.java:67) - HSEARCH000058: HSEARCH000117: IOException on the IndexWriter

              org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: org.infinispan.lucene.locking.BaseLuceneLock@1bc8d0bf

                      at org.apache.lucene.store.Lock.obtain(Lock.java:89)

                      at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:755)

                      at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.createNewIndexWriter(IndexWriterHolder.java:138)

                      at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.getIndexWriter(IndexWriterHolder.java:98)

                      at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.getIndexWriter(IndexWriterHolder.java:117)

                      at org.hibernate.search.backend.impl.lucene.AbstractWorkspaceImpl.getIndexWriter(AbstractWorkspaceImpl.java:108)

                      at org.hibernate.search.backend.impl.lucene.LuceneBackendTaskStreamer.doWork(LuceneBackendTaskStreamer.java:45)

                      at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueProcessor.applyStreamWork(LuceneBackendQueueProcessor.java:76)

                      at org.hibernate.search.indexes.spi.DirectoryBasedIndexManager.performStreamOperation(DirectoryBasedIndexManager.java:107)

                      at org.hibernate.search.backend.impl.batch.DefaultBatchBackend.flush(DefaultBatchBackend.java:76)

                      at org.infinispan.query.impl.massindex.ExtendedBatchBackend.flush(ExtendedBatchBackend.java:57)

                      at org.infinispan.query.impl.massindex.ExtendedBatchBackend.purge(ExtendedBatchBackend.java:42)

                      at org.infinispan.query.impl.massindex.IndexUpdater.purge(IndexUpdater.java:49)

                      at org.infinispan.query.impl.massindex.IndexWorker.preIndex(IndexWorker.java:54)

                      at org.infinispan.query.impl.massindex.IndexWorker.call(IndexWorker.java:75)

                      at org.infinispan.query.impl.massindex.IndexWorker.call(IndexWorker.java:31)

                      at org.infinispan.commands.read.DistributedExecuteCommand.perform(DistributedExecuteCommand.java:98)

                      at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokePerform(BasePerCacheInboundInvocationHandler.java:92)

                      at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:34)

                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

                      at java.lang.Thread.run(Thread.java:745)

              ERROR [org.hibernate.search.backend.impl.lucene.LuceneBackendTaskStreamer] [remote-thread--p2-t1] [2015-08-19 05:47:00,512] (LuceneBackendTaskStreamer.java:47) - HSEARCH000072: Couldn't open the IndexWriter because of previous error: operation skipped, index ouf of sync!

              INFO [org.infinispan.query.impl.massindex.IndexUpdater] [remote-thread--p2-t1] [2015-08-19 05:47:00,588] (IndexUpdater.java:43) - ISPN014017: Flushing index 'org.infinispan.query.remote.indexing.ProtobufValueWrapper'

              • 4. Re: near-real-time settings for Indexing with Infinispan Cache as backend
                gustavonalle

                Hi, can you post your configuration?

                • 5. Re: near-real-time settings for Indexing with Infinispan Cache as backend
                  prashant.thakur

                  the configurations from xml

                  <distributed-cache name="SUBSCRIBER" mode="SYNC" owners="2" segments="100" capacity="1" l1-lifespan="0">
                    <custom-interceptors>
                      <interceptor after="org.infinispan.interceptors.compat.TypeConverterInterceptor" class="com.subex.spark.common.distributedcaching.server.interceptors.LastModifiedDateInterceptor"/>
                    </custom-interceptors>
                    <transaction transaction-manager-lookup="org.infinispan.transaction.lookup.GenericTransactionManagerLookup" mode="NON_XA" locking="OPTIMISTIC"/>
                  </distributed-cache>
                        <!-- *************************************** -->
                    <!--  Cache to store Lucene's file metadata  -->
                    <!-- *************************************** -->
                    <local-cache name="LuceneIndexesMetadata">
                          </local-cache>
                                <!-- **************************** -->
                    <!--  Cache to store Lucene data  -->
                    <!-- **************************** -->
                    <local-cache name="LuceneIndexesData">
                          </local-cache>
                    <local-cache name="LuceneIndexesLocking">
                          </local-cache>

                   

                  From Java code related to index configuration..

                   

                     SearchMapping mapping = new SearchMapping();

                                                                  mapping.entity(classType).indexed().providedId() ;

                                                                  Properties props = new Properties();

                                                                  props.put(org.hibernate.search.cfg.Environment.MODEL_MAPPING, mapping);

                   

                                                                  props.put("hibernate.search.default.directory_provider", "infinispan");

                                                                  props.put("hibernate.search.default.indexmanager", "near-real-time");

                  configurationBuilder.indexing().index(Index.LOCAL).autoConfig(true).withProperties(props);