-
15. Re: Problem with a PUT on a large row in 4.2.0.FINAL - works with stateTransfer
I have a major update on this. I can reproduce the problem and have found the issue. There are actually 2 guilty parties here.
The first guilty party is our application. The entry to the cache in question has a value object which is an ArrayList. We initially load it with just 25,000 of its eventual 2 million entries and then do a put() into the cache. Moments later we add to the ArrayList until it reaches it full 2 milllion entry size. What I have not noticed before is that the application neglects to do another put() after updating the value object.
After figuring out the application problem we started to become suspicious about the idea that a ConcurrentModificationException must be occurring and is silent (i.e. swallowed by someone in the stack). We know we are modifying the value object and that is taking a while since there are so many items in the ArrayLIst which is happening while the background thread is marshalling and flushing the first put(). I wrote a small BTrace script to trap such a scenario and print a stacktrace in the case when it is occurring. Sure enough it is. Here is the stacktrace:
java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
java.util.AbstractList$Itr.next(AbstractList.java:343)
org.infinispan.marshall.MarshallUtil.marshallCollection(MarshallUtil.java:47)
org.infinispan.marshall.exts.ArrayListExternalizer.writeObject(ArrayListExternalizer.java:49)
org.infinispan.marshall.jboss.ConstantObjectTable$ExternalizerAdapter.writeObject(ConstantObjectTable.java:322)
org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:147)
org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1140)
org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1096)
org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:966)
org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1140)
org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1096)
org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:966)
org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:423)
org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeObject(ReplicableCommandExternalizer.java:61)
org.infinispan.marshall.jboss.ConstantObjectTable$ExternalizerAdapter.writeObject(ConstantObjectTable.java:322)
org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:147)
org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:423)
org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeObject(ReplicableCommandExternalizer.java:61)
org.infinispan.marshall.jboss.ConstantObjectTable$ExternalizerAdapter.writeObject(ConstantObjectTable.java:322)
org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:147)
org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:423)
org.infinispan.marshall.jboss.GenericJBossMarshaller.objectToObjectStream(GenericJBossMarshaller.java:98)
org.infinispan.marshall.VersionAwareMarshaller.objectToBuffer(VersionAwareMarshaller.java:93)
org.infinispan.marshall.AbstractMarshaller.objectToBuffer(AbstractMarshaller.java:31)
org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectToBuffer(MarshallerAdapter.java:22)
org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$ReplicationTask.marshallCall(CommandAwareRpcDispatcher.java:246)
org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$ReplicationTask.call(CommandAwareRpcDispatcher.java:273)
org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$ReplicationTask.call(CommandAwareRpcDispatcher.java:205)
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
java.util.concurrent.FutureTask.run(FutureTask.java:138)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:619)
I have not yet found where in the stack it is getting swallowed.
-
16. Re: Problem with a PUT on a large row in 4.2.0.FINAL - works with stateTransfer
I think the problem is somehow related to https://issues.jboss.org/browse/ISPN-960
When there's a serialization problem in either of the cases mentioned in that jira, the marshalling error only appears as TRACE and it does not get propagated to the user because it's fire and forget mode.
ISPN-960 does not solve your problem but could be further enhanced by making sure that in the cases that marshalling errors are not sent back to user, they're logged as ERROR rather than TRACE.