-
15. Re: Lock is not released on remote node after transaction commit
manik Aug 5, 2010 6:09 AM (in response to pengyan)It looks like your StringKey type cannot be marshalled by our marshalling layer.
Caused by: org.infinispan.CacheException: Unable to marshall value TestKey{key=key2}at org.infinispan.marshall.MarshalledValue.serialize(MarshalledValue.java:101)at org.infinispan.marshall.MarshalledValue.getRaw(MarshalledValue.java:157)at org.infinispan.marshall.MarshalledValue$Externalizer.writeObject(MarshalledValue.java:244)at org.infinispan.marshall.jboss.ConstantObjectTable$ExternalizerAdapter.writeObject(ConstantObjectTable.java:271)at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:147)at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:423)at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeObject(ReplicableCommandExternalizer.java:61)at org.infinispan.marshall.jboss.ConstantObjectTable$ExternalizerAdapter.writeObject(ConstantObjectTable.java:271)at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:147)at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:423)at org.infinispan.marshall.jboss.GenericJBossMarshaller.objectToObjectStream(GenericJBossMarshaller.java:100)at org.infinispan.marshall.VersionAwareMarshaller.objectToBuffer(VersionAwareMarshaller.java:92)at org.infinispan.marshall.AbstractMarshaller.objectToBuffer(AbstractMarshaller.java:31)at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectToBuffer(MarshallerAdapter.java:22)at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$ReplicationTask.marshallCall(CommandAwareRpcDispatcher.java:241)... 39 moreCaused by: java.io.InvalidObjectException: Unexpected illegal access exceptionat org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1094)at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:964)at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:423)at org.infinispan.marshall.jboss.GenericJBossMarshaller.objectToObjectStream(GenericJBossMarshaller.java:100)at org.infinispan.marshall.VersionAwareMarshaller.objectToObjectStream(VersionAwareMarshaller.java:146)at org.infinispan.marshall.MarshalledValue.serialize(MarshalledValue.java:92)... 53 moreCaused by: java.lang.IllegalAccessException: Class org.jboss.marshalling.river.RiverMarshaller can not access a member of class test.StringKey with modifiers "private"at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:65)Is this the same StringKey implementation you sent me earlier?
-
16. Re: Lock is not released on remote node after transaction commit
galder.zamarreno Aug 5, 2010 6:19 AM (in response to pengyan)Re: marshall exception
Manik, this has happened before, in fact a couple of days back. I saw it in hudson just a couple of days ago (hence why I was after sun.reflect.* classes on a Mac ):
What JDK version, provider and plattform are you using?
-
17. Re: Lock is not released on remote node after transaction commit
galder.zamarreno Aug 5, 2010 6:26 AM (in response to pengyan)Re: NPE
java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768) at org.infinispan.util.concurrent.locks.containers.OwnableReentrantPerEntryLockContainer.getLockFromMap(OwnableReentrantPerEntryLockContainer.java:38) at org.infinispan.util.concurrent.locks.containers.OwnableReentrantPerEntryLockContainer.ownsLock(OwnableReentrantPerEntryLockContainer.java:28) at org.infinispan.util.concurrent.locks.LockManagerImpl.ownsLock(LockManagerImpl.java:123) at org.infinispan.util.concurrent.locks.DeadlockDetectingLockManager.localVsLocalDld(DeadlockDetectingLockManager.java:94) at org.infinispan.util.concurrent.locks.DeadlockDetectingLockManager.lockAndRecord(DeadlockDetectingLockManager.java:78) at org.infinispan.container.EntryFactoryImpl.acquireLock(EntryFactoryImpl.java:205)
Hmmmm, seems like your keys are null? What version are you using? If the key was null and you're using a version CR1 or higher, the exception should be different (https://jira.jboss.org/browse/ISPN-520)
-
18. Re: Lock is not released on remote node after transaction commit
manik Aug 5, 2010 6:42 AM (in response to galder.zamarreno)The NPE is probably the result of the marshalling issue. Inability to marshall key, hence a nul key on the remote end. That's what I suspect anyway, I don't have full logs.
-
19. Re: Lock is not released on remote node after transaction commit
pengyan Aug 5, 2010 10:16 AM (in response to galder.zamarreno)the snapshot version, as the output I listed for my maven repository.
Also those two exception happens just occasionally.
-
20. Re: Lock is not released on remote node after transaction commit
manik Aug 5, 2010 12:22 PM (in response to pengyan)So the marshalling issue was a problem in JBoss Marshalling. Reported as JBMAR-111 and already fixed in trunk. Expect a new snapshot to be out with this fix soon.
-
21. Re: Lock is not released on remote node after transaction commit
pengyan Aug 5, 2010 8:00 PM (in response to manik)A bit confused... Is it a new snapshot for jboss marshalling or infinispan?
How could I configure my pom file to make it work?
-
22. Re: Lock is not released on remote node after transaction commit
galder.zamarreno Aug 6, 2010 4:11 AM (in response to pengyan)Expect a new snapshot of Infinispan. We'll let you know when ready.
-
23. Re: Lock is not released on remote node after transaction commit
galder.zamarreno Aug 6, 2010 4:56 AM (in response to galder.zamarreno)A new Infinispan snapshot has been uploaded which contains the fix for marshalling issue. For your own tracking, the snapshot is: 4.1.0-20100806.082729
-
24. Re: Lock is not released on remote node after transaction commit
manik Aug 6, 2010 6:40 AM (in response to galder.zamarreno)Thanks for releasing this, Galder.
@Chun Li, could you please give this a try and let us know if this works for you.
-
25. Re: Lock is not released on remote node after transaction commit
pengyan Aug 9, 2010 7:02 AM (in response to manik)I could still see occasionally there's NullPointerException. Following is the stack trace.
I can be sure that the key is not null.
and line 103 for GenericInfinispanCache.java is
cache.getAdvancedCache().lock(key);
and line 210 ~ 211 for ChunkAllocatorCache.java is
ChunkKey<E> chunkKey = new ChunkKey<E>(key, chunkIdProvider.generateChunkId());
Integer availableResource = cache.findForUpdate(chunkKey);So I can be sure the key passed to lock method would never be null. Also I'm using infinispan-core-4.1.0-20100806.082729-7.jar.
2010-08-09 05:36:58,360 ERROR [org.infinispan.interceptors.InvocationContextInterceptor](pool-184-thread-8) Execution error:
java.lang.NullPointerException
at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)
at org.infinispan.util.concurrent.locks.containers.OwnableReentrantPerEntryLockContainer.getLockFromMap(OwnableReentrantPerEntryLockContainer.java:38)
at org.infinispan.util.concurrent.locks.containers.OwnableReentrantPerEntryLockContainer.ownsLock(OwnableReentrantPerEntryLockContainer.java:28)
at org.infinispan.util.concurrent.locks.LockManagerImpl.ownsLock(LockManagerImpl.java:123)
at org.infinispan.util.concurrent.locks.DeadlockDetectingLockManager.localVsLocalDld(DeadlockDetectingLockManager.java:94)
at org.infinispan.util.concurrent.locks.DeadlockDetectingLockManager.lockAndRecord(DeadlockDetectingLockManager.java:78)
at org.infinispan.container.EntryFactoryImpl.acquireLock(EntryFactoryImpl.java:205)
at org.infinispan.container.EntryFactoryImpl.wrapEntryForWriting(EntryFactoryImpl.java:148)
at org.infinispan.container.EntryFactoryImpl.wrapEntryForWriting(EntryFactoryImpl.java:106)
at org.infinispan.interceptors.LockingInterceptor.visitLockControlCommand(LockingInterceptor.java:146)
at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:136)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:147)
at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:136)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:147)
at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:136)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at org.infinispan.interceptors.MarshalledValueInterceptor.visitLockControlCommand(MarshalledValueInterceptor.java:92)
at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:136)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:171)
at org.infinispan.interceptors.TxInterceptor.visitLockControlCommand(TxInterceptor.java:115)
at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:136)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:58)
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:39)
at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:147)
at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:136)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:271)
at org.infinispan.CacheDelegate.lock(CacheDelegate.java:296)
at org.infinispan.CacheDelegate.lock(CacheDelegate.java:289)
at test.p.common.persistence.cache.GenericInfinispanCache.findForUpdate(GenericInfinispanCache.java:103)
at test.p.common.persistence.cache.advanced.ChunkAllocatorCache.release(ChunkAllocatorCache.java:211)
at test.p.odrm.resmanagement.cache.SOPServiceImpl.releaseBandwidth(SOPServiceImpl.java:41)
at sun.reflect.GeneratedMethodAccessor611.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:198)
at $Proxy289.releaseBandwidth(Unknown Source)
at test.p.odrm.resmanagement.ResourcePersistorImpl.releaseSessionResources(ResourcePersistorImpl.java:63) -
26. Re: Lock is not released on remote node after transaction commit
manik Aug 9, 2010 7:42 AM (in response to pengyan)This happens occasionally? Is it under specific load? Do you have a unit/stress test for this?
-
27. Re: Lock is not released on remote node after transaction commit
pengyan Aug 9, 2010 10:50 PM (in response to manik)occasionally.
I could reproduce with 5 threads in attached testcase.
-
infinispan-lock.tgz 9.6 KB
-
-
28. Re: Lock is not released on remote node after transaction commit
manik Aug 11, 2010 6:54 AM (in response to pengyan)To help isolate the problem, do you only see the occasional NPE when you use deadlock detection? Or even without deadlock detection?
-
29. Re: Lock is not released on remote node after transaction commit
manik Aug 11, 2010 7:12 AM (in response to manik)Ok, this is definitely a problem with deadlock detection, as I haven't been able to recreate it with deadlock detection disabled. Only with it enabled.
I've created a JIRA for this. ISPN-594.