1 2 3 4 Previous Next 51 Replies Latest reply on Jul 18, 2013 12:23 PM by kasik Go to original post
      • 30. Re: Infinispan on EC2
        raulraja

        The fixes were introduced here https://github.com/hibernate/hibernate-core/commit/f01505591d137689c0dbd58df56946e5c55d91eb#hibernate-infinispan/src/main/java/org/hibernate/cache/infinispan/util/CacheHelper.java as fix for this http://opensource.atlassian.com/projects/hibernate/browse/HHH-6186

         

        They affected me because I'm running Infinispan 5 due to some other issues I had with 4.x trying to get it running on ec2.

        I found that the infinispan 4.x series had some bugs that were fix on 5.x and those affected me when I started the integration and trying to get it running on ec2. 5.x solved the issues but introduce dependencies not compatible with the hibernate-infinispan code. I should have mentioned that in my earlier post, my apologies.

         

        After all there may be no need to submit any patches since really I'm targetting a infinispan version not supported on the hibernate dependencies.

        Still couldn't get inifnispan 4.x to work with the cache and search but at this point I can't remember the specific issue. As for the misleading log trace statemenets I will try to submit a patch or point out the places where they appear.

         

        Is there a sample config I can follow to get the lucene index on the db running with infinispan or is this something I'm suppose to implement as a loader?. I have read in multiple places that the Lucene JDBC based directory is pretty slow. Also is the JDBC directory the right approach to use when running infinispan as clustered cache?

        • 31. Re: Infinispan on EC2
          sannegrinovero

          The fixes were introduced here https://github.com/hibernate/hibernate-core/commit/f01505591d137689c0dbd58df56946e5c55d91eb#hibernate-infinispan/src/main/java/org/hibernate/cache/infinispan/util/CacheHelper.java as fix for this http://opensource.atlassian.com/projects/hibernate/browse/HHH-6186

           

          They affected me because I'm running Infinispan 5 due to some other issues I had with 4.x trying to get it running on ec2.

          I found that the infinispan 4.x series had some bugs that were fix on 5.x and those affected me when I started the integration and trying to get it running on ec2. 5.x solved the issues but introduce dependencies not compatible with the hibernate-infinispan code. I should have mentioned that in my earlier post, my apologies.

           

          After all there may be no need to submit any patches since really I'm targetting a infinispan version not supported on the hibernate dependencies.

          Still couldn't get inifnispan 4.x to work with the cache and search but at this point I can't remember the specific issue. As for the misleading log trace statemenets I will try to submit a patch or point out the places where they appear.

          Thank you, yes sorry Infinispan 5 was introduced in Hibernate after starting work on Hibernate core v.4.

           

          Is there a sample config I can follow to get the lucene index on the db running with infinispan or is this something I'm suppose to implement as a loader?. I have read in multiple places that the Lucene JDBC based directory is pretty slow. Also is the JDBC directory the right approach to use when running infinispan as clustered cache?

          I'm not suggesting to use the old JDBC based directory, which is indeed pretty slow (and thus was not interesting to integrate with Hibernate Search), but to use the Infinispan directory like you did, just changing the cacheloader to store in a database instead of storing it on local filesystem: it won't be as slow as the database only solution as you will still benefit from the distributed cache, but it's nice to have the index stored so you can shutdown the application nodes, and also to make backups of the index together to the data.

           

          There are some example configurations in other posts, here: http://community.jboss.org/message/561017#561017 you can find a MySQL backend configured with reasonable values for the Lucene's purpose. Make sure to enable such a cacheloader for both the metadataCache and chunksCache, but not for the locking metadata.

          It would be great if you could post your final all-around configuration and EC2 howto to our WIKI: http://community.jboss.org/en/infinispan or to your own blog if you like that as I think you're almost there and this would be very helpfull for others, I can keep giving you hints but I'm not having a full application to double-check that my settings would be fine (I had one a year ago, with older jgroups and Infinispan - I'll have to make a new one).

          • 32. Re: Infinispan on EC2
            raulraja

            Thanks Sanne, I'm working on configuring the cache store with org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore. Once it is all configured I will post it to our blog and the WIKI.

            • 33. Re: Infinispan on EC2
              raulraja

              Trying to get the JDBC cache sore to work but I'm not seing the class is ever called when placing TRACE statements or debug points on it.

              Here is the configuration I'm using. Am I missing anything obvious?

               

              {code:xml}

              <namedCache name="LuceneIndexesData">

                                  <loaders passivation="false" shared="true" preload="true">

                                            <loader class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore" fetchPersistentState="true">

                                                      <properties>

                                                                <property name="stringsTableNamePrefix" value="ISPN_LuceneIndexesData"/>

                                                                <property name="idColumnName" value="id"/>

                                                                <property name="dataColumnName" value="data"/>

                                                                <property name="timestampColumnName" value="timestamp"/>

                                                                <property name="timestampColumnType" value="BIGINT"/>

                                                                <property name="connectionFactoryClass" value="org.infinispan.loaders.jdbc.connectionfactory.PooledConnectionFactory"/>

                                                                <property name="connectionUrl" value="jdbc:mysql://..."/>

                                                                <property name="userName" value="..."/>

                                                                <property name="password" value="..." />

                                                                <property name="driverClass" value="com.mysql.jdbc.Driver"/>

                                                                <property name="idColumnType" value="VARCHAR(255)"/>

                                                                <property name="dataColumnType" value="BINARY"/>

                                                                <property name="dropTableOnExit" value="false"/>

                                                                <property name="createTableOnStart" value="true"/>

                                                      </properties>

                                            </loader>

                                  </loaders>

                                  <clustering mode="replication">

                                            <stateRetrieval fetchInMemoryState="true" logFlushTimeout="60000"/>

                                            <sync replTimeout="60000"/>

                                  </clustering>

                        </namedCache>

              {code}

               

              • 34. Re: Infinispan on EC2
                sannegrinovero

                Hi,

                you need to set this property too:

                <property name="key2StringMapperClass" value="org.infinispan.lucene.LuceneKey2StringMapper" />

                 

                But this should be self-explanatory as I just double checked that using an org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore

                in this case will throw a org.infinispan.loaders.keymappers.UnsupportedKeyTypeException... don't you have this error in the logs?

                • 35. Re: Infinispan on EC2
                  raulraja

                  I did not get that exception but digged depper and got this configuration running...

                   

                   

                  {code:xml}

                  <?xml version="1.0" encoding="UTF-8"?>

                  <infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

                                                xsi:schemaLocation="urn:infinispan:config:5.0 http://www.infinispan.org/schemas/infinispan-config-5.0.xsd"

                                                xmlns="urn:infinispan:config:5.0">

                   

                   

                            <!-- *************************** -->

                            <!-- System-wide global settings -->

                            <!-- *************************** -->

                   

                   

                            <global>

                   

                   

                                      <!-- Duplicate domains are allowed so that multiple deployments with default configuration

                                                                    of Hibernate Search applications work - if possible it would be better to use JNDI to share

                                                                    the CacheManager across applications -->

                                      <globalJmxStatistics enabled="false" cacheManagerName="HibernateSearch" allowDuplicateDomains="true"/>

                   

                   

                   

                   

                                      <transport clusterName="infinispan-hibernate-search-cluster" distributedSyncTimeout="60000" transportClass="org.infinispan.remoting.transport.jgroups.JGroupsTransport">

                                                <properties>

                                                          <property name="configurationFile" value="jgroups-ec2-search.xml"/>

                                                </properties>

                                      </transport>

                   

                   

                                      <!-- Used to register JVM shutdown hooks. hookBehavior: DEFAULT, REGISTER, DONT_REGISTER.

                                                                    Hibernate Search takes care to stop the CacheManager so registering is not needed -->

                                      <shutdown hookBehavior="DONT_REGISTER"/>

                   

                   

                            </global>

                   

                   

                            <!-- *************************** -->

                            <!-- Default "template" settings -->

                            <!-- *************************** -->

                   

                   

                            <default>

                   

                   

                                      <locking lockAcquisitionTimeout="60000" writeSkewCheck="false" concurrencyLevel="500" useLockStriping="false"/>

                   

                   

                                      <!--<lazyDeserialization enabled="false"/>-->

                                      <storeAsBinary enabled="false"/>

                   

                   

                                      <!-- Invocation batching is required for use with the Lucene Directory -->

                                      <invocationBatching enabled="true"/>

                   

                   

                                      <!-- This element specifies that the cache is clustered. modes supported: distribution

                                                                    (d), replication (r) or invalidation (i). Don't use invalidation to store Lucene indexes (as

                                                                    with Hibernate Search DirectoryProvider). Replication is recommended for best performance of

                                                                    Lucene indexes, but make sure you have enough memory to store the index in your heap.

                                                                    Also distribution scales much better than replication on high number of nodes in the cluster. -->

                                      <clustering mode="distribution">

                   

                   

                                                <!-- Prefer loading all data at startup than later -->

                                                <stateRetrieval timeout="60000" logFlushTimeout="60000" fetchInMemoryState="true" alwaysProvideInMemoryState="true"/>

                   

                   

                                                <!-- Network calls are synchronous by default -->

                                                <sync replTimeout="60000"/>

                                      </clustering>

                                      <jmxStatistics enabled="false"/>

                                      <eviction maxEntries="-1" strategy="NONE"/>

                                      <expiration maxIdle="-1"/>

                   

                   

                            </default>

                   

                   

                            <!-- ******************************************************************************* -->

                            <!-- Individually configured "named" caches.                                         -->

                            <!--                                                                                 -->

                            <!-- While default configuration happens to be fine with similar settings across the -->

                            <!-- three caches, they should generally be different in a production environment.   -->

                            <!--                                                                                 -->

                            <!-- Current settings could easily lead to OutOfMemory exception as a CacheStore     -->

                            <!-- should be enabled, and maybe distribution is desired.                           -->

                            <!-- ******************************************************************************* -->

                   

                   

                            <!-- *************************************** -->

                            <!--  Cache to store Lucene's file metadata  -->

                            <!-- *************************************** -->

                            <namedCache name="LuceneIndexesMetadata">

                                      <loaders passivation="false" shared="true" preload="true">

                              <loader class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"

                                  fetchPersistentState="false" ignoreModifications="false"

                                  purgeOnStartup="false">

                                  <properties>

                                      <property name="stringsTableNamePrefix" value="ISPN_LuceneIndexesMetadata" />

                                      <property name="key2StringMapperClass" value="org.infinispan.lucene.LuceneKey2StringMapper" />

                                      <property name="idColumnName" value="ID_COLUMN" />

                                      <property name="dataColumnName" value="DATA_COLUMN" />

                                      <property name="timestampColumnName" value="TIMESTAMP_COLUMN" />

                                      <property name="timestampColumnType" value="BIGINT" />

                                      <property name="connectionFactoryClass" value="org.infinispan.loaders.jdbc.connectionfactory.ManagedConnectionFactory"/>

                                                               <property name="datasourceJndiLocation" value="java:AppDatasource"/>

                                      <property name="idColumnType" value="VARCHAR(256)" />

                                      <property name="dataColumnType" value="BLOB" />

                                      <property name="dropTableOnExit" value="false" />

                                      <property name="createTableOnStart" value="true" />

                                  </properties>

                                  <async enabled="true" flushLockTimeout="15000" threadPoolSize="10" />

                              </loader>

                          </loaders>

                                      <clustering mode="replication">

                                                <stateRetrieval fetchInMemoryState="true" logFlushTimeout="60000" />

                                                <sync replTimeout="60000"/>

                                      </clustering>

                            </namedCache>

                   

                   

                            <!-- **************************** -->

                            <!--  Cache to store Lucene data  -->

                            <!-- **************************** -->

                            <namedCache name="LuceneIndexesData">

                                      <loaders passivation="false" shared="true" preload="true">

                              <loader class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"

                                  fetchPersistentState="false" ignoreModifications="false"

                                  purgeOnStartup="false">

                                  <properties>

                                      <property name="stringsTableNamePrefix" value="ISPN_LuceneIndexesData" />

                                      <property name="key2StringMapperClass" value="org.infinispan.lucene.LuceneKey2StringMapper" />

                                      <property name="idColumnName" value="ID_COLUMN" />

                                      <property name="dataColumnName" value="DATA_COLUMN" />

                                      <property name="timestampColumnName" value="TIMESTAMP_COLUMN" />

                                      <property name="timestampColumnType" value="BIGINT" />

                                      <property name="connectionFactoryClass" value="org.infinispan.loaders.jdbc.connectionfactory.ManagedConnectionFactory"/>

                                                               <property name="datasourceJndiLocation" value="java:AppDatasource"/>

                                      <property name="idColumnType" value="VARCHAR(256)" />

                                      <property name="dataColumnType" value="BLOB" />

                                      <property name="dropTableOnExit" value="false" />

                                      <property name="createTableOnStart" value="true" />

                                  </properties>

                                  <async enabled="true" flushLockTimeout="15000" threadPoolSize="10" />

                              </loader>

                          </loaders>

                                      <clustering mode="replication">

                                                <stateRetrieval fetchInMemoryState="true" logFlushTimeout="60000"/>

                                                <sync replTimeout="60000"/>

                                      </clustering>

                            </namedCache>

                   

                   

                            <!-- ***************************** -->

                            <!--  Cache to store Lucene locks  -->

                            <!-- ***************************** -->

                            <namedCache name="LuceneIndexesLocking">

                                      <clustering mode="replication">

                                                <stateRetrieval fetchInMemoryState="true" logFlushTimeout="60000"/>

                                                <sync replTimeout="60000"/>

                                      </clustering>

                            </namedCache>

                   

                   

                  </infinispan>

                  {code}

                   

                   

                  Unfortunatly I'm getting the following exception when the cache tries to flush to the db...

                   

                  Hibernate Search: Directory writer-1 07/11 17:18:18 ERROR org.hibernate.search.exception.impl.LogErrorHandler - Exception occurred org.infinispan.CacheException: Unable to end batch

                   

                   

                  org.infinispan.CacheException: Unable to end batch

                            at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:111)

                            at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:93)

                            at org.infinispan.CacheImpl.endBatch(CacheImpl.java:440)

                            at org.infinispan.lucene.InfinispanIndexOutput.close(InfinispanIndexOutput.java:208)

                            at org.apache.lucene.index.FormatPostingsDocsWriter.close(FormatPostingsDocsWriter.java:124)

                            at org.apache.lucene.index.FormatPostingsTermsWriter.close(FormatPostingsTermsWriter.java:71)

                            at org.apache.lucene.index.FormatPostingsFieldsWriter.finish(FormatPostingsFieldsWriter.java:70)

                            at org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:135)

                            at org.apache.lucene.index.TermsHash.flush(TermsHash.java:109)

                            at org.apache.lucene.index.DocInverter.flush(DocInverter.java:72)

                            at org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:59)

                            at org.apache.lucene.index.DocumentsWriter.flush(DocumentsWriter.java:552)

                            at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3331)

                            at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3296)

                            at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:3159)

                            at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3232)

                            at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3214)

                            at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3198)

                            at org.hibernate.search.backend.Workspace.commitIndexWriter(Workspace.java:220)

                            at org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor.run(PerDPQueueProcessor.java:109)

                            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)

                            at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

                            at java.util.concurrent.FutureTask.run(FutureTask.java:138)

                            at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                            at java.lang.Thread.run(Thread.java:680)

                  Caused by: javax.transaction.HeuristicMixedException

                            at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:315)

                            at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:99)

                            at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:97)

                            at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:131)

                            at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:108)

                            ... 25 more

                  Hibernate Search: Directory writer-1 07/11 17:18:18 WARN org.hibernate.search.backend.Workspace - going to force release of the IndexWriter lock

                   

                   

                  Not sure how to get around this. My transactions are managed by spring's @Transactional and should be thread bound.

                  • 36. Re: Infinispan on EC2
                    sannegrinovero

                    Hi,

                    I'm not understandig why you should get an HeuristicMixedException since you're not using transactions.

                    Looking into DummyTransaction.java:315 it seems it should have logged some more details about what exactly whent wrong.. could you please post that part too?

                     

                    The configuration looks good, it's very similar to one I know was working fine.

                    Since it's failing to commit the resource.. maybe there's something wrong while storing into the database? Please check more logs.

                    • 37. Re: Infinispan on EC2
                      mircea.markus

                      Some more context information around that would be gret. There should be an error message containing "exception while committing"

                      • 38. Re: Infinispan on EC2
                        sannegrinovero

                        could you also try using the alternative strategy to get database connections? (the one taking a JDBC driver, connection URL, password & login).

                         

                        In the way you're creating them I'm afraid that the application server might be messing with the connection, maybe enlistening it in a transaction. What application server is it? what kind of connection pool is it?

                        • 39. Re: Infinispan on EC2
                          raulraja

                          The datasource is spring configured bean which is what's used by the hibernate session factory. I fired it up this morning with infinispan "trace" on and I didn't see the exception but also have noticed that when looking at the database none of the tables for the JdbcStringBasedCacheStore are created.

                          I get the same result when hardcoding the connection information and not using the jndi datasource.

                           

                          This is the datasource:

                           

                          {code:xml}

                          <bean id="dataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource" destroy-method="close">

                                              <property name="driverClass" value="${datasource.driverClass}"/>

                                              <property name="jdbcUrl" value="${datasource.jdbcUrl}"/>

                                              <property name="user" value="${datasource.user}"/>

                                              <property name="password" value="${datasource.password}"/>

                                              <property name="acquireIncrement" value="5"/>

                                              <property name="acquireRetryAttempts" value="30"/>

                                              <property name="idleConnectionTestPeriod" value="14400"/>

                                              <property name="maxPoolSize" value="200"/>

                                              <property name="maxStatements" value="50"/>

                                              <property name="minPoolSize" value="10"/>

                                              <property name="checkoutTimeout" value="25200"/>

                                              <property name="maxIdleTime" value="60"/>

                                              <property name="preferredTestQuery" value="select 1"/>

                                    </bean>

                          {code}

                           

                           

                          Now I get this exception from JDBC_PING, I'm guessing due to the app not shutting down gracefully.

                           

                          Timer-3,infinispan-hibernate-search-cluster,Raul-Raja-Martinezs-MacBook-Pro-46980 07/12 10:47:52 ERROR org.jgroups.protocols.JDBC_PING - Error updating JDBC_PING table

                          com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry 'b3be0c91-bcc2-a590-e674-b02f728400e6-infinispan-hibernate-search' for key 'PRIMARY'

                                    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

                                    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)

                                    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)

                                    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)

                                    at com.mysql.jdbc.Util.handleNewInstance(Util.java:409)

                                    at com.mysql.jdbc.Util.getInstance(Util.java:384)

                                    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1041)

                                    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3562)

                                    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3494)

                                    at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1960)

                                    at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2114)

                                    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2696)

                                    at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2105)

                                    at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2398)

                                    at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2316)

                                    at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2301)

                                    at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:105)

                                    at org.jgroups.protocols.JDBC_PING.insert(JDBC_PING.java:265)

                                    at org.jgroups.protocols.JDBC_PING.writeToFile(JDBC_PING.java:246)

                                    at org.jgroups.protocols.FILE_PING$WriterTask.run(FILE_PING.java:246)

                                    at org.jgroups.util.TimeScheduler2$RecurringTask.run(TimeScheduler2.java:592)

                                    at org.jgroups.util.TimeScheduler2$MyTask.run(TimeScheduler2.java:524)

                                    at org.jgroups.util.TimeScheduler2$Entry.execute(TimeScheduler2.java:429)

                                    at org.jgroups.util.TimeScheduler2$1.run(TimeScheduler2.java:295)

                                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                                    at java.lang.Thread.run(Thread.java:680)

                           

                           

                          I'm the only member in the cluster and my config has not changed at all. I placed debug points all over the place on JdbcStringBasedCacheStore and it never gets called. I'm seeing log entries now where the infinispan lucene directory seems to be using the filesystem, which makes no sense to me as the configuration should be pointing to the JdbcStringBasedCacheStore.

                           

                          Hibernate Search: Directory writer-1 07/12 10:46:35 DEBUG org.infinispan.lucene.InfinispanDirectory - Removed file: segments_1 from index: com.fortysevendeg.corp.trips.frontend.model.Itinerary

                           

                          The app server is Tomcat6 and the app is spring + jpa hibernate app

                          • 40. Re: Infinispan on EC2
                            raulraja

                            I was able to repro the transaction exception again.

                            I placed debug points all over the place on JdbcStringBasedCacheStore and still no tables created or any hits. It looks like that configuration is never used.

                             

                            Here is the full stack trace that I get and copy of the config below:

                             

                            INFO: Server startup in 34131 ms

                            Connected to server

                            Hibernate Search: Directory writer-1 07/13 13:11:39 DEBUG org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor - Opening an IndexWriter for update

                            http-8080-2 07/13 13:11:41 WARN nu.localhost.tapestry5.springsecurity.services.internal.StaticSecurityChecker - Could not validate configuration attributes as the SecurityMetadataSource did not return any attributes from getAllConfigAttributes()

                            Hibernate Search: Directory writer-1 07/13 13:11:41 DEBUG org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor - Opening an IndexWriter for update

                            Hibernate Search: Directory writer-1 07/13 13:11:41 DEBUG org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor - Opening an IndexWriter for update

                            Hibernate Search: Directory writer-1 07/13 13:11:41 DEBUG org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor - Opening an IndexWriter for update

                            http-8080-2 07/13 13:11:42 DEBUG org.hibernate.search.reader.SharingBufferReaderProvider - Opening IndexReader for directoryProviders: 1

                            http-8080-2 07/13 13:11:42 DEBUG org.hibernate.search.reader.SharingBufferReaderProvider - Closing MultiReader: CacheableMultiReader(ReadOnlyDirectoryReader(segments_1))

                            http-8080-2 07/13 13:11:42 DEBUG org.hibernate.search.reader.SharingBufferReaderProvider - Opening IndexReader for directoryProviders: 1

                            http-8080-2 07/13 13:11:42 DEBUG org.hibernate.search.reader.SharingBufferReaderProvider - Closing MultiReader: CacheableMultiReader(ReadOnlyDirectoryReader(segments_1))

                            Hibernate Search: Directory writer-1 07/13 13:11:50 DEBUG org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor - Opening an IndexWriter for update

                            Hibernate Search: Directory writer-1 07/13 13:11:52 DEBUG org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor - Opening an IndexWriter for update

                            Hibernate Search: Directory writer-1 07/13 13:11:52 DEBUG org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor - Opening an IndexWriter for update

                            Hibernate Search: Directory writer-1 07/13 13:11:52 DEBUG org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor - Opening an IndexWriter for update

                            Hibernate Search: Directory writer-1 07/13 13:11:52 DEBUG org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor - Opening an IndexWriter for update

                            http-8080-2 07/13 13:11:54 DEBUG org.hibernate.search.reader.SharingBufferReaderProvider - Opening IndexReader for directoryProviders: 1

                            http-8080-2 07/13 13:11:54 DEBUG org.hibernate.search.reader.SharingBufferReaderProvider - Closing MultiReader: CacheableMultiReader(ReadOnlyDirectoryReader(segments_1))

                            http-8080-2 07/13 13:11:54 DEBUG org.hibernate.search.reader.SharingBufferReaderProvider - Opening IndexReader for directoryProviders: 1

                            http-8080-2 07/13 13:11:54 DEBUG org.hibernate.search.reader.SharingBufferReaderProvider - Closing MultiReader: CacheableMultiReader(ReadOnlyDirectoryReader(segments_1))

                            http-8080-4 07/13 13:12:01 DEBUG org.hibernate.search.reader.SharingBufferReaderProvider - Opening IndexReader for directoryProviders: 1

                            http-8080-4 07/13 13:12:01 DEBUG org.hibernate.search.reader.SharingBufferReaderProvider - Closing MultiReader: CacheableMultiReader(ReadOnlyDirectoryReader(segments_1))

                            http-8080-4 07/13 13:12:01 DEBUG org.hibernate.search.reader.SharingBufferReaderProvider - Opening IndexReader for directoryProviders: 1

                            http-8080-4 07/13 13:12:01 DEBUG org.hibernate.search.reader.SharingBufferReaderProvider - Closing MultiReader: CacheableMultiReader(ReadOnlyDirectoryReader(segments_1))

                            http-8080-4 07/13 13:12:02 DEBUG org.hibernate.search.reader.SharingBufferReaderProvider - Opening IndexReader for directoryProviders: 1

                            http-8080-4 07/13 13:12:02 DEBUG org.hibernate.search.reader.SharingBufferReaderProvider - Closing MultiReader: CacheableMultiReader(ReadOnlyDirectoryReader(segments_1))

                            http-8080-4 07/13 13:12:02 DEBUG org.hibernate.search.reader.SharingBufferReaderProvider - Opening IndexReader for directoryProviders: 1

                            http-8080-4 07/13 13:12:02 DEBUG org.hibernate.search.reader.SharingBufferReaderProvider - Closing MultiReader: CacheableMultiReader(ReadOnlyDirectoryReader(segments_1))

                            http-8080-1 07/13 13:12:19 DEBUG com.fortysevendeg.commons.services.events.impl.EventPublisherImpl - Intercepted [ItineraryService.ON_ITINERARY_CHANGED]

                            http-8080-1 07/13 13:12:19 DEBUG com.fortysevendeg.commons.services.events.impl.EventPublisherImpl - Publishing ItineraryService.ON_ITINERARY_CHANGED

                            Hibernate Search: Directory writer-1 07/13 13:12:19 DEBUG org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor - Opening an IndexWriter for update

                            Hibernate Search: Directory writer-1 07/13 13:12:19 DEBUG org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor - Opening an IndexWriter for update

                            Hibernate Search: Directory writer-1 07/13 13:12:19 DEBUG org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor - Opening an IndexWriter for update

                            Hibernate Search: Directory writer-1 07/13 13:12:19 DEBUG org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor - Opening an IndexWriter for update

                            Hibernate Search: Directory writer-1 07/13 13:12:19 WARN org.infinispan.transaction.tm.DummyTransaction - ISPN00112: exception while committing

                            javax.transaction.xa.XAException

                                      at org.infinispan.transaction.xa.TransactionXaAdapter.getLocalTransactionAndValidateImpl(TransactionXaAdapter.java:233)

                                      at org.infinispan.transaction.xa.TransactionXaAdapter.getLocalTransactionAndValidate(TransactionXaAdapter.java:226)

                                      at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:103)

                                      at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:312)

                                      at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:99)

                                      at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:97)

                                      at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:131)

                                      at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:108)

                                      at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:93)

                                      at org.infinispan.CacheImpl.endBatch(CacheImpl.java:440)

                                      at org.infinispan.lucene.InfinispanIndexOutput.close(InfinispanIndexOutput.java:208)

                                      at org.apache.lucene.util.IOUtils.closeSafely(IOUtils.java:80)

                                      at org.apache.lucene.index.FieldsWriter.close(FieldsWriter.java:111)

                                      at org.apache.lucene.index.StoredFieldsWriter.flush(StoredFieldsWriter.java:52)

                                      at org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:58)

                                      at org.apache.lucene.index.DocumentsWriter.flush(DocumentsWriter.java:552)

                                      at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3331)

                                      at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3296)

                                      at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:3159)

                                      at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3232)

                                      at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3214)

                                      at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3198)

                                      at org.hibernate.search.backend.Workspace.commitIndexWriter(Workspace.java:220)

                                      at org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor.run(PerDPQueueProcessor.java:109)

                                      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)

                                      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

                                      at java.util.concurrent.FutureTask.run(FutureTask.java:138)

                                      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                                      at java.lang.Thread.run(Thread.java:680)

                            Hibernate Search: Directory writer-1 07/13 13:12:19 ERROR org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor - Unexpected error in Lucene Backend:

                            org.infinispan.CacheException: Unable to end batch

                                      at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:111)

                                      at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:93)

                                      at org.infinispan.CacheImpl.endBatch(CacheImpl.java:440)

                                      at org.infinispan.lucene.InfinispanIndexOutput.close(InfinispanIndexOutput.java:208)

                                      at org.apache.lucene.util.IOUtils.closeSafely(IOUtils.java:80)

                                      at org.apache.lucene.index.FieldsWriter.close(FieldsWriter.java:111)

                                      at org.apache.lucene.index.StoredFieldsWriter.flush(StoredFieldsWriter.java:52)

                                      at org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:58)

                                      at org.apache.lucene.index.DocumentsWriter.flush(DocumentsWriter.java:552)

                                      at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3331)

                                      at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3296)

                                      at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:3159)

                                      at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3232)

                                      at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3214)

                                      at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3198)

                                      at org.hibernate.search.backend.Workspace.commitIndexWriter(Workspace.java:220)

                                      at org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor.run(PerDPQueueProcessor.java:109)

                                      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)

                                      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

                                      at java.util.concurrent.FutureTask.run(FutureTask.java:138)

                                      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                                      at java.lang.Thread.run(Thread.java:680)

                            Caused by: javax.transaction.HeuristicMixedException

                                      at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:315)

                                      at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:99)

                                      at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:97)

                                      at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:131)

                                      at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:108)

                                      ... 22 more

                            Hibernate Search: Directory writer-1 07/13 13:12:19 ERROR org.hibernate.search.exception.impl.LogErrorHandler - Exception occurred org.infinispan.CacheException: Unable to end batch

                             

                             

                            org.infinispan.CacheException: Unable to end batch

                                      at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:111)

                                      at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:93)

                                      at org.infinispan.CacheImpl.endBatch(CacheImpl.java:440)

                                      at org.infinispan.lucene.InfinispanIndexOutput.close(InfinispanIndexOutput.java:208)

                                      at org.apache.lucene.util.IOUtils.closeSafely(IOUtils.java:80)

                                      at org.apache.lucene.index.FieldsWriter.close(FieldsWriter.java:111)

                                      at org.apache.lucene.index.StoredFieldsWriter.flush(StoredFieldsWriter.java:52)

                                      at org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:58)

                                      at org.apache.lucene.index.DocumentsWriter.flush(DocumentsWriter.java:552)

                                      at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3331)

                                      at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3296)

                                      at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:3159)

                                      at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3232)

                                      at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3214)

                                      at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3198)

                                      at org.hibernate.search.backend.Workspace.commitIndexWriter(Workspace.java:220)

                                      at org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor.run(PerDPQueueProcessor.java:109)

                                      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)

                                      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

                                      at java.util.concurrent.FutureTask.run(FutureTask.java:138)

                                      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                                      at java.lang.Thread.run(Thread.java:680)

                            Caused by: javax.transaction.HeuristicMixedException

                                      at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:315)

                                      at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:99)

                                      at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:97)

                                      at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:131)

                                      at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:108)

                                      ... 22 more

                            Hibernate Search: Directory writer-1 07/13 13:12:19 WARN org.hibernate.search.backend.Workspace - going to force release of the IndexWriter lock

                             

                             

                             

                            {code:xml}

                            <?xml version="1.0" encoding="UTF-8"?>

                            <infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

                                                          xsi:schemaLocation="urn:infinispan:config:5.0 http://www.infinispan.org/schemas/infinispan-config-5.0.xsd"

                                                          xmlns="urn:infinispan:config:5.0">

                             

                             

                                      <!-- *************************** -->

                                      <!-- System-wide global settings -->

                                      <!-- *************************** -->

                             

                             

                                      <global>

                             

                             

                                                <!-- Duplicate domains are allowed so that multiple deployments with default configuration

                                                                              of Hibernate Search applications work - if possible it would be better to use JNDI to share

                                                                              the CacheManager across applications -->

                                                <globalJmxStatistics enabled="false" cacheManagerName="HibernateSearch" allowDuplicateDomains="true"/>

                             

                             

                             

                             

                                                <transport clusterName="infinispan-hibernate-search-cluster" distributedSyncTimeout="60000" transportClass="org.infinispan.remoting.transport.jgroups.JGroupsTransport">

                                                          <properties>

                                                                    <property name="configurationFile" value="jgroups-ec2-search.xml"/>

                                                          </properties>

                                                </transport>

                             

                             

                                                <!-- Used to register JVM shutdown hooks. hookBehavior: DEFAULT, REGISTER, DONT_REGISTER.

                                                                              Hibernate Search takes care to stop the CacheManager so registering is not needed -->

                                                <shutdown hookBehavior="DONT_REGISTER"/>

                             

                             

                                      </global>

                             

                             

                                      <!-- *************************** -->

                                      <!-- Default "template" settings -->

                                      <!-- *************************** -->

                             

                             

                                      <default>

                             

                             

                                                <locking lockAcquisitionTimeout="60000" writeSkewCheck="false" concurrencyLevel="500" useLockStriping="false"/>

                             

                             

                                                <!--<lazyDeserialization enabled="false"/>-->

                                                <storeAsBinary enabled="false"/>

                             

                             

                                                <!-- Invocation batching is required for use with the Lucene Directory -->

                                                <invocationBatching enabled="true"/>

                             

                             

                                                <!-- This element specifies that the cache is clustered. modes supported: distribution

                                                                              (d), replication (r) or invalidation (i). Don't use invalidation to store Lucene indexes (as

                                                                              with Hibernate Search DirectoryProvider). Replication is recommended for best performance of

                                                                              Lucene indexes, but make sure you have enough memory to store the index in your heap.

                                                                              Also distribution scales much better than replication on high number of nodes in the cluster. -->

                                                <clustering mode="distribution">

                             

                             

                                                          <!-- Prefer loading all data at startup than later -->

                                                          <stateRetrieval timeout="60000" logFlushTimeout="60000" fetchInMemoryState="true" alwaysProvideInMemoryState="true"/>

                             

                             

                                                          <!-- Network calls are synchronous by default -->

                                                          <sync replTimeout="60000"/>

                                                </clustering>

                                                <jmxStatistics enabled="false"/>

                                                <eviction maxEntries="-1" strategy="NONE"/>

                                                <expiration maxIdle="-1"/>

                             

                             

                                      </default>

                             

                             

                                      <!-- ******************************************************************************* -->

                                      <!-- Individually configured "named" caches.                                         -->

                                      <!--                                                                                 -->

                                      <!-- While default configuration happens to be fine with similar settings across the -->

                                      <!-- three caches, they should generally be different in a production environment.   -->

                                      <!--                                                                                 -->

                                      <!-- Current settings could easily lead to OutOfMemory exception as a CacheStore     -->

                                      <!-- should be enabled, and maybe distribution is desired.                           -->

                                      <!-- ******************************************************************************* -->

                             

                             

                                      <!-- *************************************** -->

                                      <!--  Cache to store Lucene's file metadata  -->

                                      <!-- *************************************** -->

                                      <namedCache name="LuceneIndexesMetadata">

                                                <loaders passivation="false" shared="true" preload="true">

                                        <loader class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"

                                            fetchPersistentState="true" ignoreModifications="false"

                                            purgeOnStartup="false">

                                            <properties>

                                                <property name="stringsTableNamePrefix" value="ISPN_LuceneIndexesMetadata" />

                                                <property name="key2StringMapperClass" value="org.infinispan.lucene.LuceneKey2StringMapper" />

                                                <property name="idColumnName" value="ID_COLUMN" />

                                                <property name="dataColumnName" value="DATA_COLUMN" />

                                                <property name="timestampColumnName" value="TIMESTAMP_COLUMN" />

                                                <property name="timestampColumnType" value="BIGINT" />

                                                <property name="connectionFactoryClass" value="org.infinispan.loaders.jdbc.connectionfactory.ManagedConnectionFactory"/>

                                                                         <property name="datasourceJndiLocation" value="java:AppDatasource"/>

                                                <property name="idColumnType" value="VARCHAR(256)" />

                                                <property name="dataColumnType" value="BLOB" />

                                                <property name="dropTableOnExit" value="false" />

                                                <property name="createTableOnStart" value="true" />

                                            </properties>

                                            <async enabled="true" flushLockTimeout="15000" threadPoolSize="10" />

                                        </loader>

                                    </loaders>

                                                <clustering mode="replication">

                                                          <stateRetrieval fetchInMemoryState="true" logFlushTimeout="60000" />

                                                          <sync replTimeout="60000"/>

                                                </clustering>

                                      </namedCache>

                             

                             

                                      <!-- **************************** -->

                                      <!--  Cache to store Lucene data  -->

                                      <!-- **************************** -->

                                      <namedCache name="LuceneIndexesData">

                                                <loaders passivation="false" shared="true" preload="true">

                                        <loader class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"

                                            fetchPersistentState="true" ignoreModifications="false"

                                            purgeOnStartup="false">

                                            <properties>

                                                <property name="stringsTableNamePrefix" value="ISPN_LuceneIndexesData" />

                                                <property name="key2StringMapperClass" value="org.infinispan.lucene.LuceneKey2StringMapper" />

                                                <property name="idColumnName" value="ID_COLUMN" />

                                                <property name="dataColumnName" value="DATA_COLUMN" />

                                                <property name="timestampColumnName" value="TIMESTAMP_COLUMN" />

                                                <property name="timestampColumnType" value="BIGINT" />

                                                <property name="connectionFactoryClass" value="org.infinispan.loaders.jdbc.connectionfactory.ManagedConnectionFactory"/>

                                                                         <property name="datasourceJndiLocation" value="java:AppDatasource"/>

                                                <property name="idColumnType" value="VARCHAR(256)" />

                                                <property name="dataColumnType" value="BLOB" />

                                                <property name="dropTableOnExit" value="false" />

                                                <property name="createTableOnStart" value="true" />

                                            </properties>

                                            <async enabled="true" flushLockTimeout="15000" threadPoolSize="10" />

                                        </loader>

                                    </loaders>

                                                <clustering mode="replication">

                                                          <stateRetrieval fetchInMemoryState="true" logFlushTimeout="60000"/>

                                                          <sync replTimeout="60000"/>

                                                </clustering>

                                      </namedCache>

                             

                             

                                      <!-- ***************************** -->

                                      <!--  Cache to store Lucene locks  -->

                                      <!-- ***************************** -->

                                      <namedCache name="LuceneIndexesLocking">

                                                <clustering mode="replication">

                                                          <stateRetrieval fetchInMemoryState="true" logFlushTimeout="60000"/>

                                                          <sync replTimeout="60000"/>

                                                </clustering>

                                      </namedCache>

                             

                             

                            </infinispan>

                            {code}

                             

                            • 41. Re: Infinispan on EC2
                              raulraja

                              I step through the configuration and I see that the cache store loaders are never enabled or registered.

                              See attached screenshot and note that the CacheLoaderManagerConfig after finalizing intialization shows an empty collection for the "cacheLoadersConfigs"

                               

                              Any idea of what I may be doing wrong? Also if it's not registering the cache store why would it ever throw the transaction exception.

                               

                              As I was stepping through the code I noticed how the config for the JdbcStringBasedCacheStore was created but never stored since both "async" and "singleton" where flagged as false when going through the "Visiting" pattern to register the config.

                               

                              I'm on Infinispan 5.0.0.CR7.

                              • 42. Re: Infinispan on EC2
                                raulraja

                                Seems to work on I'm on Infinispan 5.0.0.CR6.

                                • 43. Re: Infinispan on EC2
                                  sannegrinovero

                                  Hi,

                                  maybe you found a bug in the latest CRs.

                                  So it works fine with 5.0.0.CR6 and it does NOT work with 5.0.0.CR7 ?

                                  Could you please try the latest version as committed on master?

                                  It looks like you might be hitting https://issues.jboss.org/browse/ISPN-1226

                                  • 44. Re: Infinispan on EC2
                                    raulraja

                                    Seems to work with 5.0.0.CR8, thanks!

                                    Making progress with an optimal setup and everything seems to be running if the nodes don't start at the same time.

                                    It always fails to start when the nodes start at the same time since it seems like all hit each other because the SessionFactory starts at the same time and the nodes are discoverable but not ready for state transfers.