Inconsistent cache between 2 cluster nodes
fharms Oct 15, 2013 4:03 AMHi
We are from time to time experience inconsistent in JBoss Cache between two nodes. We use JBoss Cache as second level cache for Hibernate
We have 2 nodes both setup with a jboss cache in a cluster with the “READ_COMMIT” as isolation level and cache mode “INVALIDATION_SYNC”
A snapshot of the jboss cache configuration:
<entry><key>mvcc-entity</key> <value> <bean name="MVCCEntityCache" class="org.jboss.cache.config.Configuration"> <!-- Mode of communication with peer caches. INVALIDATION_SYNC is highly recommended as the mode for use with entity and collection caches. --> <property name="cacheMode">INVALIDATION_SYNC</property> <!-- Name of cluster. Needs to be the same for all members --> <property name="clusterName">${jboss.partition.name:DefaultPartition}-mvcc-entity</property> <!-- Specifies the number of shared locks to use for write locks acquired. --> <property name="concurrencyLevel">100000</property> <!-- Whether or not to fetch state on joining a cluster. --> <property name="fetchInMemoryState">true</property> <!-- Must match the value of "useRegionBasedMarshalling" --> <property name="inactiveOnStartup">true</property> <!-- The isolation level used for transactions. --> <property name="isolationLevel">READ_COMMITTED</property> <!-- We have no asynchronous notification listeners --> <property name="listenerAsyncPoolSize">0</property> <!-- Max number of milliseconds to wait for a lock acquisition match the transaction timeout on 5 min--> <property name="lockAcquisitionTimeout">300000</property> <!-- Specifies whether parent nodes are locked when inserting or removing children. --> <property name="lockParentForChildInsertRemove">false</property>
Problem :
We seen this problem when 2 clients is routed to each server client #1 add a new item to a collection on server #1, The collection on server #2 is invalidated as expected.
Client #2 is now reading this collection from server #2 which force it read the data from the database into the cache(putForExternalRead). Concurrently client #1 is adding a new item to the same collection on server #1 and invalidate the collection on server #2, this update is finish before Client #2 has inserted all data into the cache. Client #2 is finished populating all data to the cache, but do not see the changes from Client #1, in fact it just insert it's data.
Since the putForExternalRead only will update the cache if it’s not present, it will never read the latest update into cache and by this you end up with a inconsistent cache between the nodes, but also between the cache and the database.
I have tried to illustrate in this simple diagram
The JBoss Application server is version 5.1, JBoss Cache 3.2.5.GA, Hibernate JBossCache2.x Integration 3.3.2.GA_CP03
Is this a bug or because of a misconfiguration?
Thanks!
/Flemming