6 Replies Latest reply on Jul 24, 2006 6:20 AM by mwelss

    ConcurrentModificationException

    mwelss

      Using jboss-messaging 1.0.1CR3 on jboss 4.0.4GA with ejb3 mdb I got the following Exception:

      13:38:20,920 ERROR [ServerConnectionEndpoint] Exception occured
      java.util.ConcurrentModificationException
      at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:449)
      at java.util.AbstractList$Itr.next(AbstractList.java:420)
      at org.jboss.jms.server.endpoint.ServerConsumerEndpoint$DeliveryCallback.afterCommit(ServerConsumerEndpoint.java:730)
      at org.jboss.messaging.core.tx.Transaction.commit(Transaction.java:186)
      at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendTransaction(ServerConnectionEndpoint.java:390)
      at org.jboss.jms.server.endpoint.advised.ConnectionAdvised.org$jboss$jms$server$endpoint$advised$ConnectionAdvised$sendTransaction$aop(ConnectionAdvised.java:99)
      at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
      at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
      at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
      at org.jboss.aop.Advisor.dynamicInvoke(Advisor.java:723)
      at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:101)
      at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:126)
      at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:842)
      at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:82)
      at org.jboss.remoting.Client.invoke(Client.java:610)
      at org.jboss.remoting.Client.invoke(Client.java:602)
      at org.jboss.jms.client.delegate.DelegateSupport.invoke(DelegateSupport.java:112)
      at org.jboss.jms.client.delegate.ClientConnectionDelegate$sendTransaction_N4986868250254447300.invokeNext(ClientConnectionDelegate$sendTransaction_N4986868250254447300.java)
      at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:134)
      at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117)
      at org.jboss.jms.client.delegate.ClientConnectionDelegate$sendTransaction_N4986868250254447300.invokeNext(ClientConnectionDelegate$sendTransaction_N4986868250254447300.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69)
      at org.jboss.jms.client.delegate.ClientConnectionDelegate$sendTransaction_N4986868250254447300.invokeNext(ClientConnectionDelegate$sendTransaction_N4986868250254447300.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientConnectionDelegate$sendTransaction_N4986868250254447300.invokeNext(ClientConnectionDelegate$sendTransaction_N4986868250254447300.java)
      at org.jboss.jms.client.delegate.ClientConnectionDelegate.sendTransaction(ClientConnectionDelegate.java)
      at org.jboss.jms.tx.ResourceManager.sendTransactionXA(ResourceManager.java:406)
      at org.jboss.jms.tx.ResourceManager.commit(ResourceManager.java:194)
      at org.jboss.jms.tx.MessagingXAResource.commit(MessagingXAResource.java:116)
      at org.jboss.tm.TransactionImpl$Resource.commit(TransactionImpl.java:2253)
      at org.jboss.tm.TransactionImpl.commitResources(TransactionImpl.java:1784)
      at org.jboss.tm.TransactionImpl.commit(TransactionImpl.java:340)
      at org.jboss.tm.TxManager.commit(TxManager.java:240)
      at org.jboss.jms.asf.StdServerSession.onMessage(StdServerSession.java:351)
      at org.jboss.jms.client.remoting.MessageCallbackHandler.callOnMessage(MessageCallbackHandler.java:82)
      at org.jboss.jms.client.container.AsfAspect.handleRun(AsfAspect.java:165)
      at sun.reflect.GeneratedMethodAccessor82.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:130)
      at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
      at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:134)
      at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117)
      at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69)
      at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
      at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
      at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
      at org.jboss.jms.asf.StdServerSession.run(StdServerSession.java:194)
      at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:748)
      at java.lang.Thread.run(Thread.java:595)
      13:38:20,921 INFO [STDOUT] [Client] Message 71/100 was successfully sent to the testQueue queue
      13:38:20,925 ERROR [ServerConnectionEndpoint] Failed to rollback tx
      org.jboss.messaging.core.tx.TransactionException: Failed to remove delivery 16730
      at org.jboss.jms.server.endpoint.ServerConsumerEndpoint$DeliveryCallback.afterRollback(ServerConsumerEndpoint.java:752)
      at org.jboss.messaging.core.tx.Transaction.rollback(Transaction.java:256)
      at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendTransaction(ServerConnectionEndpoint.java:399)
      at org.jboss.jms.server.endpoint.advised.ConnectionAdvised.org$jboss$jms$server$endpoint$advised$ConnectionAdvised$sendTransaction$aop(ConnectionAdvised.java:99)
      at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
      at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
      at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
      at org.jboss.aop.Advisor.dynamicInvoke(Advisor.java:723)
      at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:101)
      at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:126)
      at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:842)
      at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:82)
      at org.jboss.remoting.Client.invoke(Client.java:610)
      at org.jboss.remoting.Client.invoke(Client.java:602)
      at org.jboss.jms.client.delegate.DelegateSupport.invoke(DelegateSupport.java:112)
      at org.jboss.jms.client.delegate.ClientConnectionDelegate$sendTransaction_N4986868250254447300.invokeNext(ClientConnectionDelegate$sendTransaction_N4986868250254447300.java)
      at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:134)
      at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117)
      at org.jboss.jms.client.delegate.ClientConnectionDelegate$sendTransaction_N4986868250254447300.invokeNext(ClientConnectionDelegate$sendTransaction_N4986868250254447300.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69)
      at org.jboss.jms.client.delegate.ClientConnectionDelegate$sendTransaction_N4986868250254447300.invokeNext(ClientConnectionDelegate$sendTransaction_N4986868250254447300.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientConnectionDelegate$sendTransaction_N4986868250254447300.invokeNext(ClientConnectionDelegate$sendTransaction_N4986868250254447300.java)
      at org.jboss.jms.client.delegate.ClientConnectionDelegate.sendTransaction(ClientConnectionDelegate.java)
      at org.jboss.jms.tx.ResourceManager.sendTransactionXA(ResourceManager.java:406)
      at org.jboss.jms.tx.ResourceManager.commit(ResourceManager.java:194)
      at org.jboss.jms.tx.MessagingXAResource.commit(MessagingXAResource.java:116)
      at org.jboss.tm.TransactionImpl$Resource.commit(TransactionImpl.java:2253)
      at org.jboss.tm.TransactionImpl.commitResources(TransactionImpl.java:1784)
      at org.jboss.tm.TransactionImpl.commit(TransactionImpl.java:340)
      at org.jboss.tm.TxManager.commit(TxManager.java:240)
      at org.jboss.jms.asf.StdServerSession.onMessage(StdServerSession.java:351)
      at org.jboss.jms.client.remoting.MessageCallbackHandler.callOnMessage(MessageCallbackHandler.java:82)
      at org.jboss.jms.client.container.AsfAspect.handleRun(AsfAspect.java:165)
      at sun.reflect.GeneratedMethodAccessor82.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:130)
      at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
      at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:134)
      at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117)
      at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
      at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69)
      at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
      at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
      at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
      at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
      at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
      at org.jboss.jms.asf.StdServerSession.run(StdServerSession.java:194)
      at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:748)
      at java.lang.Thread.run(Thread.java:595)
      13:38:20,927 INFO [STDOUT] [PlexMDB#0] message received: 300

        • 1. Re: ConcurrentModificationException
          timfox

          Martin-

          Please can you tell me under what circumstances this happens.

          Can you please replicate in a simple case I can run so I can investigate?

          Thanks.

          • 2. Re: ConcurrentModificationException
            mwelss

            Tim,

            basically it's my jms test case (version 2+ ;-)
            on a server with 4 CPUs (which I believe is important!)

            Unfortunately it happened only once and I was not able to reproduce it so far.

            Regards,

            Martin

            • 3. Re: ConcurrentModificationException
              timfox

              Ok thx.

              Someone else saw this one too. But like you it only happened rarely and they couldn't reproduce it either.

              I'll look into it.

              • 4. Re: ConcurrentModificationException
                timfox

                Martin-

                I've started looking at this issue this morning.

                I've run your latest (v3) test application against 4.0.4.GA and ejb3 and I'm not seeing any memory leaks or exceptions, when I test against the latest code base.

                However in the last few weeks I have made quite a few changes in the way delivery is done so I may have inadvertently fixed your problem while I did that.

                I shall also try this on a dual core machine.

                • 5. Re: ConcurrentModificationException
                  timfox

                  Martin - we reproduced this problem on a dual core (it seems to happen quickly with multiple cores), and eventually managed to reproduce it on a single core (it took me 8 hours of running and 8 million messages).

                  There was a problem related to transaction isolation with XA transactions, which I have now fixed.

                  We have then run the test again on both dual and single cores with millions of messages and the problem seems to have been solved.

                  The fix will be in the next release. We should hopefully be out in the next week.

                  Thanks.

                  • 6. Re: ConcurrentModificationException
                    mwelss

                    sounds good!
                    Thanks.