7 Replies Latest reply on Feb 23, 2011 9:00 AM by galder.zamarreno

    JBoss AS 6 Hibernate L2C, Infinispan RemoteCacheStore

    matt_filion

      I've been toying with JBoss AS 6 a bit and can get all of my variations of our cache strategies to work except when we need to store the entities in a RemoteCacheStore. For some reason hibernate simply does not call any of the methods on the cache store to load and persist the data.

       

      What I'm basically trying to accomplish is a "Distant" cache for our entities that will be frequently used across all nodes in the cluster. This will reduce the memory footprint of our JBoss instances and reduce the amount of effort spent marshalling/demarshalling inforamtion to and from the database since everything stored in cache is a bit more "native" to the applications environment. This is the cache configuration that I have inside my "infinispan-configs.xml" file in JBoss AS 6 (CR1) but i never see my RemoteCacheStore being utilized.

       

      Has anyone have gone down this route yet or have any knowledge if CacheStores work within the infinispan-configs.xml configurations?

       

       

      <!-- *************************************** -->
      <!-- *          100% Remote Cache          * -->
      <!-- // infinispan-configs.xml // -->
      
      <!-- *************************************** -->
      <!-- *          100% Remote Cache          * -->
      <!-- *************************************** -->
      <!-- 
                This configuration passes all cache gets and puts to a remote server and never
                stored locally within the VM.
      
      
                passivation="false" Must be set to false otherwise the cache loader 
                will not be used for each transaction, it ill only update the cache store 
                when it triggers a "passivation" event at the end of a transaction or during 
                eviction. 
      
                shared=true Setting this to true avoids multiple cache instances 
                writing the same modification multiple times. If enabled, only the node where 
                the modification originated will write to the cache store. 
      
                preload=false 
                If true, when the cache starts, data stored in the cache store will be pre-loaded 
                into memory. You can set this attribute to true if we want to establish some 
                sort of "warm" cache that loads from a remote source.  
      -->
      <!-- *************************************** -->
      <namedCache name="entity-remote">
           <loaders preload="false" shared="true" passivation="false">
                <loader 
                     class="org.infinispan.loaders.remote.RemoteCacheStore"
                     fetchPersistentState="false" 
                     ignoreModifications="false"
                     purgeOnStartup="false" 
                     purgeSynchronously="false">
                     <properties>
                          <property name="hotRodClientPropertiesFile"     value="${jboss.common.base.url}cluster/infinispan-cache-registry.sar/hotrod-client.properties" />
                          <property name="useDefaultRemoteCache" value="false" />
                          <property name="remoteCacheName" value="dist" />
                     </properties>
                </loader>
           </loaders>
      </namedCache>
      
      
      
      <!-- // persistence.xml // -->
      <property name="hibernate.transaction.manager_lookup_class" value="org.hibernate.transaction.JBossTransactionManagerLookup"/>
      <property name="hibernate.cache.region.factory_class"       value="org.hibernate.cache.infinispan.JndiInfinispanRegionFactory"/>
      <property name="hibernate.cache.infinispan.cachemanager"    value="java:CacheManager/entity"/>
      
      <!-- 
           To be able to define custom cache for a specific entity or entities collection use the following.
           When adding new class names to the custom definitions the "name" attribute must start with 
           'hibernate.cache.infinispan' and end with 'cfg'. Its easy to miss the last part. The middle is
           the full package and class name of the entity that is currently managed in cache. 
      -->
      <property name="hibernate.cache.infinispan.com.model.Entity.cfg"            value="entity-remote"/>
      <property name="hibernate.cache.infinispan.com.model.Entity.collection.cfg" value="entity-remote"/>
      
      <!--  Default Configurations for Entities that dont have explicit definitions. -->
      <property name="hibernate.cache.infinispan.entity.cfg"      value="entity-remote"/>
      <property name="hibernate.cache.infinispan.collection.cfg"  value="entity-remote"/>
      <property name="hibernate.cache.infinispan.query.cfg"       value="entity-remote"/>
      <property name="hibernate.cache.infinispan.timestamp.cfg"   value="timestamps"/>
      
      <!-- *************************************** -->
      <!--
      This configuration passes all cache gets and puts to a remote server and never
      stored locally within the VM.
      passivation="false" Must be set to false otherwise the cache loader
      will not be used for each transaction, it ill only update the cache store
      when it triggers a "passivation" event at the end of a transaction or during
      eviction.
      shared=true Setting this to true avoids multiple cache instances
      writing the same modification multiple times. If enabled, only the node where
      the modification originated will write to the cache store.
      preload=false
      If true, when the cache starts, data stored in the cache store will be pre-loaded
      into memory. You can set this attribute to true if we want to establish some
      sort of "warm" cache that loads from a remote source.
      -->
      <!-- *************************************** -->
      <namedCache name="entity-remote">
      <loaders preload="false" shared="true" passivation="false">
      <loader
      class="org.infinispan.loaders.remote.RemoteCacheStore"
      fetchPersistentState="false"
      ignoreModifications="false"
      purgeOnStartup="false"
      purgeSynchronously="false">
      <properties>
      <property name="hotRodClientPropertiesFile"     value="${jboss.common.base.url}cluster/infinispan-cache-registry.sar/hotrod-client.properties" />
      <property name="useDefaultRemoteCache" value="false" />
      <property name="remoteCacheName" value="dist" />
      </properties>
      </loader>
      </loaders>

      </namedCache>

       

        • 1. Re: JBoss AS 6 Hibernate L2C, Infinispan RemoteCacheStore
          matt_filion

          I was able to get past this by not using jboss's supplied cache manager and instead defining the configurations in my application. However, i would prefer to offload these kinds of configurations the container so I consider this a work around and not necessarily the right answer to the problem.

           

          <!-- // persistence.xml // -->
          <!-- *************************************************** -->
          <!--  Use the application provided configurations.       -->
          <!-- *************************************************** -->
          <property name="hibernate.cache.region.factory_class"  value="org.hibernate.cache.infinispan.InfinispanRegionFactory"/>
          <property name="hibernate.cache.infinispan.cfg"        value="ati-infinispan-configs.xml"/>
          

          <property name="hibernate.cache.region.factory_class"  value="org.hibernate.cache.infinispan.InfinispanRegionFactory"/>

          <property name="hibernate.cache.infinispan.cfg"        value="ati-infinispan-<!-- // persistence.xml // -->

          <property name="hibernate.cache.region.factory_class"  value="org.hibernate.cache.infinispan.InfinispanRegionFactory"/>

          <property name="hibernate.cache.infinispan.cfg"        value="app-infinispan-configs.xml"/>configs.xml"/>

           

          • 2. Re: JBoss AS 6 Hibernate L2C, Infinispan RemoteCacheStore
            matt_filion

            Had to turn on debug logging on the bootup but figured out that the problem was my hotrod-client.properties file was not getting picked up by jboss and therefore never being configured. I ended up hard coding an absolute path to the \server\all\cluster\infinispan-cache-registry.sar\infinispan-configs.xml and that fixed my issue.

             

            <namedCache name="entity-remote">
                 <loaders preload="false" shared="true" passivation="false">
                      <loader 
                           class="org.infinispan.loaders.remote.RemoteCacheStore"
                           fetchPersistentState="false" 
                           ignoreModifications="false"
                           purgeOnStartup="false" 
                           purgeSynchronously="false">
                           <properties>
                                <property name="hotRodClientPropertiesFile"     value="/absolute/file/path/hotrod-client.properties" />
                                <property name="useDefaultRemoteCache" value="false" />
                                <property name="remoteCacheName" value="dist" />
                           </properties>
                      </loader>
                 </loaders>
            </namedCache>
            • 3. Re: JBoss AS 6 Hibernate L2C, Infinispan RemoteCacheStore
              galder.zamarreno

              Matt, two things:

               

              1. Did you have to enable debugging to see that the file was not being picked up? According to the code this should be logged at error level, see https://github.com/infinispan/infinispan/blob/4.2.x/cachestore/remote/src/main/java/org/infinispan/loaders/remote/RemoteCacheStoreConfig.java

               

              2. I'm not sure about your use case. The whole point of a 2LC is that is fast to local and it's fast to access, much faster than having to go via TCP/IP to a database. Now, by adding a remote cache store, you're potentially adding further delays cos get() calls that are empty in the local cache now will go remotely to the remote Hot Rod server to double check. So, you're adding unnecessary delays there.

               

              Sure, there're situations when using a cache loader makes sense, for example a cluster cache loader instead of enabling state transfer, but I have doubts about your remote cache store.

               

                 public void setHotRodClientPropertiesFile(String hotRodClientProperties) {
                    FileLookup fileLookup = new FileLookup();
                    InputStream inputStream = fileLookup.lookupFile(hotRodClientProperties);
                    try {
                       this.hotRodClientProperties.load(inputStream);
                    } catch (IOException e) {
                       log.error("Issues while loading properties from file", e);
                       throw new CacheException(e);
                    }
              • 4. Re: JBoss AS 6 Hibernate L2C, Infinispan RemoteCacheStore
                matt_filion

                Galder,

                 

                1. I didn't see any error messages, this isn't to say that it doesn't exist but I didn't notice it. I turned on debugging and saw it as an info statement saying that it couldn't find it on the classpath and trying to look it up as a file.
                2. In some cases we have entities that need to have their state persisted across restarts of the servers, but don't need super fast access. Which is ultimately the perfect fit for a RemoteCacheStore setup like this where it simply gets the data from the RemoteCacheStore in all instances.

                  We also have a very large collection of entities that we need to be fast and persist across restarts as well. So in these 2 scenarios we have different strategies in place which uses the RemoteCacheStore for a second "tier" of caching when its expired from the in-vm cache. The in-vm cache of course has a much shorter eviction strategy than the remote cache store because the volume of entities that could potentially get loaded over time is enormous. The data grid gives the data the protection we need it to have because with the volume of reads that are done the db gets fairly heavily hit.

                 

                The main benefit of having the RemoteCacheStore is that we don't have to have to programatically declare different loading for each entity, its defined by the underlying Level 2 Cache strategy. Gives our engineers a bit less to worry about. Also, since the infinispan server cluster is as close as you declare the server it can be maintained on the same machine and lead to much faster access of entities than a DB since that will always have to live on a remote server.

                • 5. Re: JBoss AS 6 Hibernate L2C, Infinispan RemoteCacheStore
                  galder.zamarreno

                  Re 1. An info statement? Hmmm, sounds like it should be pointed rather as an ERROR or WARN message at least. Would you mind opening a jira in https://issues.jboss.org/browse/ISPN with the debugging that you saw and the configuration you used? Thanks

                   

                  Re 2. The only advantage I could see RemoteCacheStore giving you here is precisely quicker access to those entities as opposed to having Hibernate retrieve it from the database, but then again, I think you'd be losing out in cache queries that return nothing, cos they'd incur a cost of having to go to the RemoteCacheStore and check.

                   

                  You also have to remember that entity cache, which is communicates synchronously, is configured with state transfer. So, this means that when a server is restarted, it will try to get the entity cache contents from another node in the cluster. So, unless you do full cluster restarts, you shouldn't find yourself in situation where the cache needs to be completely repopulated out of the db. Instead, if you use rolling restart strategies, you might be able to always keep several nodes with entity caches in the cluster, while other nodes get restarted.

                   

                  I think what you're trying to do is interesting, but I really think you should test both scenarios, first with default configurations provided by the Infinispan 2LC jar, and then with a RCS, and see which one performs better. I don't think the decision is that clear cut.

                  • 6. JBoss AS 6 Hibernate L2C, Infinispan RemoteCacheStore
                    matt_filion

                    Sorry for the long delay, the holidays and an up comming release at work has left me with little spare time. I'll get to the docs within the next week or so.

                    • 7. JBoss AS 6 Hibernate L2C, Infinispan RemoteCacheStore
                      galder.zamarreno

                      Added a section using Infinispan as remote second level cache to http://community.jboss.org/docs/DOC-14105