locked keys after replication timeout problem
jsl Aug 26, 2011 11:08 AMHi - please could you help me with the following problem:
I have distributed caches configured using JBossTM. During stress we found following blocker: some nodes got repl. timeout on node CACHE23
2011-08-26 16:28:56,901 ERROR local -> dist worker| tionContextInterceptor: 111| ISPN000136: Execution error
org.infinispan.util.concurrent.TimeoutException: Replication timeout for CACHE13-46306
therefore PreparedCommand failed as well:
2011-08-26 16:28:56,979 ERROR local -> dist worker| TransactionCoordinator:1640| Error while processing PrepareCommand
org.infinispan.util.concurrent.TimeoutException: Replication timeout for CACHE13-46306
but after that following error occured:
2011-08-26 16:28:57,057 WARN | local -> dist worker| TransactionXaAdapter: 118| ISPN000141: Could not rollback prepared 1PC transaction. This transaction will be rolled back by the recovery process, if enabled. Transaction: LocalXaTransaction{xid=< formatId=131076, gtrid_length=29, bqual_length=28, tx_uid=0:ffff0a000ae5:e6f8:4e57ad79:7, node_name=1, branch_uid=0:ffff0a000ae5:e6f8:4e57ad79:8, eis_name=unknown eis name >} LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, transaction=TransactionImple < ac, BasicAction: 0:ffff0a000ae5:e6f8:4e57ad79:7 status: ActionStatus.COMMITTING >} org.infinispan.transaction.xa.LocalXaTransaction@44574163 |
2011-08-26 16:28:57,166 ERROR local -> dist worker| IfspnCacheSynchronizer: 139| Error during commit: Could not commit transaction.
javax.transaction.RollbackException: Could not commit transaction.
the transaction dissapeared:
2011-08-26 16:28:57,182 WARN | local -> dist worker| IfspnCacheSynchronizer: 129| Error during rollback: Error during commit: Could not commit transaction.java.lang.IllegalStateException: BaseTransaction.rollback - no transaction! |
The problem is that after that there are locked nodes by itself (node CACHE23) and distributed cache is completely locked for it:
org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [1 seconds] on key [ID8541566201966883263] for requestor [GlobalTransaction:<CACHE23-28362>:3:remote]! Lock held by [GlobalTransaction:<CACHE23-28362>:2:remote]
Do you guys have any idea if this is bug or improperly configured TM or something? Same error I've got using BatchingTM, DummyTM applying various configurations
Is there some way how to 'unlock' acquired lock explicitly or how to bring things back to live?
Thanks alot!
Jan
Configuration (full log attached):
<global> | |
<transport clusterName="cachecluster"> | |
<properties> | |
<property name="configurationFile" value="cw-jgroups-udp.xml"/> | |
</properties> | |
</transport> | |
<globalJmxStatistics enabled="true" jmxDomain="distCache"/> | |
</global> | |
| <default> |
| <transaction |
transactionManagerLookupClass="org.infinispan.transaction.lookup.JBossStandaloneJTAManagerLookup" | |
syncRollbackPhase="false" | |
syncCommitPhase="false" | |
useEagerLocking="false"/> | |
| <locking isolationLevel="REPEATABLE_READ" |
lockAcquisitionTimeout="1000" | |
writeSkewCheck="false" | |
concurrencyLevel="512" | |
useLockStriping="false"/> | |
| <clustering mode="distribution"> |
<sync replTimeout="30000"/> | |
<l1 enabled="false"/> | |
<hash numOwners="100"/> | |
</clustering> | |
| <jmxStatistics enabled="true"/> |
<!--<invocationBatching enabled="true"/>--> | |
</default> |
-
cachecluster.log.zip 18.4 KB