6 Replies Latest reply on Jul 15, 2010 2:13 AM by gaohoward

    Remote MDBs seem to stop after marshalling exception

    mikemil
    mikemil Jun 17, 2010 1:45 PM

      Environment

      -------------------------

      JBoss 4.2.3

      JBoss Messaging 1.4.4

      JBoss Remoting  2.2.3

       

      We have a couple servers and each have local queues and remote mdbs.   After running for a couple of hours, we see the following exception - which mentions 'marshalling exception'.   The problem is - after this exception, the remote mdbs for that server never seem to get triggered.   We have to restart the JBoss server in order to the MDBs triggered.

       

      {noformat}

      2010-06-11 11:05:18,777 ERROR [org.jboss.remoting.transport.socket.SocketClientInvoker] Got marshalling exception, exiting
      java.net.SocketException: Connection reset
          at java.net.SocketInputStream.read(SocketInputStream.java:197)
          at java.io.BufferedInputStream.fill(BufferedInputStream.java:229)
          at java.io.BufferedInputStream.read(BufferedInputStream.java:246)
          at java.io.FilterInputStream.read(FilterInputStream.java:89)
          at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.readVersion(MicroSocketClientInvoker.java:1077)
          at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:682)
          at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:458)
          at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:141)
          at org.jboss.remoting.MicroRemoteClientInvoker.establishLease(MicroRemoteClientInvoker.java:474)
          at org.jboss.remoting.Client.setupClientLease(Client.java:1912)
          at org.jboss.remoting.Client.connect(Client.java:1800)
          at org.jboss.remoting.Client.connect(Client.java:652)
          at org.jboss.jms.client.remoting.JMSRemotingConnection$1.run(JMSRemotingConnection.java:362)
          at java.security.AccessController.doPrivileged(AccessController.java:246)
          at org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:356)
          at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:165)
          at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
          at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:80)
          at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java)
          at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
          at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
          at org.jboss.jms.client.JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:205)
          at org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:101)
          at org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:95)
          at org.jboss.ejb.plugins.jms.DLQHandler.createService(DLQHandler.java:187)
          at org.jboss.system.ServiceMBeanSupport.jbossInternalCreate(ServiceMBeanSupport.java:260)
          at org.jboss.system.ServiceMBeanSupport.create(ServiceMBeanSupport.java:188)
          at org.jboss.ejb.plugins.jms.JMSContainerInvoker.innerStartDelivery(JMSContainerInvoker.java:615)
          at org.jboss.ejb.plugins.jms.JMSContainerInvoker$ExceptionListenerImpl$ExceptionListenerRunnable.run(JMSContainerInvoker.java:1533)
          at java.lang.Thread.run(Thread.java:810)
      2010-06-11 11:05:18,778 DEBUG [org.jboss.remoting.InvokerRegistry] removed SocketClientInvoker[1d2c1d2c, bisocket://cholpos01p.na.odcorp.net:14457] from registry
      2010-06-11 11:05:18,778 DEBUG [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[1d2c1d2c, bisocket://cholpos01p.na.odcorp.net:14457] disconnecting ...
      2010-06-11 11:05:18,778 DEBUG [org.jboss.remoting.Client] Client[302125570:a1jn4v-s8ubem-gaai3x6d-1-gab58h4q-2yw] is disconnected
      2010-06-11 11:05:18,778 DEBUG [org.jboss.remoting.Client] Client[376575602:a1jn4v-s8ubem-gaai3x6d-1-gab58h4q-2yx] is disconnected
      2010-06-11 11:05:18,778 DEBUG [org.jboss.ejb.plugins.jms.DLQHandler] Initialization failed DLQHandler
      org.jboss.jms.exception.MessagingNetworkFailureException
          at org.jboss.jms.client.delegate.DelegateSupport.handleThrowable(DelegateSupport.java:240)
          at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:198)
          at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
          at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:80)
          at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java)
          at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
          at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
          at org.jboss.jms.client.JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:205)
          at org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:101)
          at org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:95)
          at org.jboss.ejb.plugins.jms.DLQHandler.createService(DLQHandler.java:187)
          at org.jboss.system.ServiceMBeanSupport.jbossInternalCreate(ServiceMBeanSupport.java:260)
          at org.jboss.system.ServiceMBeanSupport.create(ServiceMBeanSupport.java:188)
          at org.jboss.ejb.plugins.jms.JMSContainerInvoker.innerStartDelivery(JMSContainerInvoker.java:615)
          at org.jboss.ejb.plugins.jms.JMSContainerInvoker$ExceptionListenerImpl$ExceptionListenerRunnable.run(JMSContainerInvoker.java:1533)
          at java.lang.Thread.run(Thread.java:810)
      Caused by:
      org.jboss.remoting.CannotConnectException: Error setting up client lease upon performing connect.
          at org.jboss.remoting.Client.connect(Client.java:1804)
          at org.jboss.remoting.Client.connect(Client.java:652)
          at org.jboss.jms.client.remoting.JMSRemotingConnection$1.run(JMSRemotingConnection.java:362)
          at java.security.AccessController.doPrivileged(AccessController.java:246)
          at org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:356)
          at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:165)
          ... 14 more
      Caused by:
      java.lang.Exception: Error setting up client lease
          at org.jboss.remoting.MicroRemoteClientInvoker.establishLease(MicroRemoteClientInvoker.java:508)
          at org.jboss.remoting.Client.setupClientLease(Client.java:1912)
          at org.jboss.remoting.Client.connect(Client.java:1800)
          ... 19 more
      Caused by:
      java.rmi.MarshalException: Failed to communicate.  Problem during marshalling/unmarshalling; nested exception is:
          java.net.SocketException: Connection reset
          at org.jboss.remoting.transport.socket.SocketClientInvoker.handleException(SocketClientInvoker.java:127)
          at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:740)
          at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:458)
          at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:141)
          at org.jboss.remoting.MicroRemoteClientInvoker.establishLease(MicroRemoteClientInvoker.java:474)
          ... 21 more
      Caused by:
      java.net.SocketException: Connection reset
          at java.net.SocketInputStream.read(SocketInputStream.java:197)
          at java.io.BufferedInputStream.fill(BufferedInputStream.java:229)
          at java.io.BufferedInputStream.read(BufferedInputStream.java:246)
          at java.io.FilterInputStream.read(FilterInputStream.java:89)
          at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.readVersion(MicroSocketClientInvoker.java:1077)
          at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:682)
          ... 24 more
      2010-06-11 11:05:18,778 ERROR [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Reconnect failed: JMS provider failure detected for ImportMerchandiseMDB
      org.jboss.jms.exception.MessagingNetworkFailureException
          at org.jboss.jms.client.delegate.DelegateSupport.handleThrowable(DelegateSupport.java:240)
          at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:198)
          at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
          at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:80)
          at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java)
          at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
          at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
          at org.jboss.jms.client.JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:205)
          at org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:101)
          at org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:95)
          at org.jboss.ejb.plugins.jms.DLQHandler.createService(DLQHandler.java:187)
          at org.jboss.system.ServiceMBeanSupport.jbossInternalCreate(ServiceMBeanSupport.java:260)
          at org.jboss.system.ServiceMBeanSupport.create(ServiceMBeanSupport.java:188)
          at org.jboss.ejb.plugins.jms.JMSContainerInvoker.innerStartDelivery(JMSContainerInvoker.java:615)
          at org.jboss.ejb.plugins.jms.JMSContainerInvoker$ExceptionListenerImpl$ExceptionListenerRunnable.run(JMSContainerInvoker.java:1533)
          at java.lang.Thread.run(Thread.java:810)
      Caused by:
      org.jboss.remoting.CannotConnectException: Error setting up client lease upon performing connect.
          at org.jboss.remoting.Client.connect(Client.java:1804)
          at org.jboss.remoting.Client.connect(Client.java:652)
          at org.jboss.jms.client.remoting.JMSRemotingConnection$1.run(JMSRemotingConnection.java:362)
          at java.security.AccessController.doPrivileged(AccessController.java:246)
          at org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:356)
          at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:165)
          ... 14 more
      Caused by:
      java.lang.Exception: Error setting up client lease
          at org.jboss.remoting.MicroRemoteClientInvoker.establishLease(MicroRemoteClientInvoker.java:508)
          at org.jboss.remoting.Client.setupClientLease(Client.java:1912)
          at org.jboss.remoting.Client.connect(Client.java:1800)
          ... 19 more
      Caused by:
      java.rmi.MarshalException: Failed to communicate.  Problem during marshalling/unmarshalling; nested exception is:
          java.net.SocketException: Connection reset
          at org.jboss.remoting.transport.socket.SocketClientInvoker.handleException(SocketClientInvoker.java:127)
          at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:740)
          at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:458)
          at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:141)
          at org.jboss.remoting.MicroRemoteClientInvoker.establishLease(MicroRemoteClientInvoker.java:474)
          ... 21 more
      Caused by:
      java.net.SocketException: Connection reset
          at java.net.SocketInputStream.read(SocketInputStream.java:197)
          at java.io.BufferedInputStream.fill(BufferedInputStream.java:229)
          at java.io.BufferedInputStream.read(BufferedInputStream.java:246)
          at java.io.FilterInputStream.read(FilterInputStream.java:89)
          at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.readVersion(MicroSocketClientInvoker.java:1077)
          at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:682)
          ... 24 more
      2010-06-11 11:05:18,778 DEBUG [org.jboss.ejb.plugins.jms.JMSContainerInvoker] innerStop
      2010-06-11 11:05:18,778 DEBUG [org.jboss.jms.asf.StdServerSessionPool] Clearing 0 from ServerSessionPool
      2010-06-11 11:05:18,778 DEBUG [org.jboss.ejb.plugins.jms.DLQHandler] Destroying DLQHandler
      2010-06-11 11:05:18,778 DEBUG [org.jboss.ejb.plugins.jms.DLQHandler] Destroyed DLQHandler
      2010-06-11 11:05:18,778 INFO  [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Waiting for reconnect internal 10000ms for ImportMerchandiseMDB
      2010-06-11 11:05:28,778 INFO  [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Trying to reconnect to JMS provider for ImportMerchandiseMDB
      2010-06-11 11:05:28,778 DEBUG [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Initializing
      2010-06-11 11:05:28,779 DEBUG [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Looking up provider adapter: java:/RemoteJMSProvider
      2010-06-11 11:05:28,779 DEBUG [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Provider adapter: org.jboss.jms.jndi.JNDIProviderAdapter@13ae13ae
      2010-06-11 11:05:28,779 DEBUG [org.jboss.ejb.plugins.jms.DLQHandler] Creating DLQHandler
      2010-06-11 11:05:28,956 DEBUG [org.jboss.ejb.plugins.jms.DLQHandler] Using factory: JBossConnectionFactory->ConnectionFactoryDelegate[jboss.messaging.connectionfactory:service=ConnectionFactory, SID=0]

      {noformat}

       

      Looking for suggestions on this.   Customer is considering migrating back to MQ from Messaging because we didn't see these types of connection issues.

        • 1. Re: Remote MDBs seem to stop after marshalling exception
          mikemil
          mikemil Jun 24, 2010 11:40 AM (in response to mikemil)

          With the help of someone from the network group, we have been able to find a slight difference between our MDBS that get 'stuck' and stop delivering messages and those that continue to operate smoothly.   According to the network folks the difference is the number of connections to port 4457 (serverBindPort).   For a server/mdb still operating, we have 4 connections active, whereas mdbs that are 'stuck' have 2 less connections to port 4457.

           

          Does this ring a bell with anyone?   We never really had these problems when running MQ.   It started with our migration to Messaging 1.4.4.

            • Actions
          • 2. Re: Remote MDBs seem to stop after marshalling exception
            ron_sigal
            ron_sigal Jul 5, 2010 8:36 PM (in response to mikemil)

            Hi Mike,

             

            I can't tell you what is happening, but the fact that the innermost Exception is a SocketException("Connection reset") indicates that the connection was closed at the other end.  Do you have a firewall between the client and server?  Do you see any indication in server.log that connections were closing on the server?

             

            I don't know what the two versus four connections means.  You might pose the question on the JBossMessaging users forum.

             

            -Ron

              • Actions
            • 3. Re: Remote MDBs seem to stop after marshalling exception
              mikemil
              mikemil Jul 14, 2010 4:26 PM (in response to ron_sigal)

              Thanks for the update Ron - looks like 4th of July weekend was time for you to respond to forum posts because I saw a lot of responses.

               

              I think we have worked thru our problem, which at this point we will categorize as 'bad architecture'.   We had a hub & spoke architecture with over 800 spokes.  The hub had a remote MDB pointing to each spoke and each spoke had an remote MDB pointing to the hub.   The main problem mostly at the hub - using Messaging w/Remoting bisocket - this meant that, if all spokes were connected to the hub, we ended up with over 2400 threads (800 hubs each getting a Connection Consumer thread, along with 2 separate control socket thread (firewall between hub and spokes)) along with everything else our hub server was trying to do.

               

              Data flowed correctly for some time, then things appeared to break and apparently we were getting networking problems on port 4457.    We have changed our architecture to no longer use the remote MDBs and seem to be much more stable.

               

              Not sure exactly what in the flow was broken but things are much more stable w/o the remote mdbs - may just too many remote mdbs....

               

              Thanks again!

                • Actions
              • 4. Re: Remote MDBs seem to stop after marshalling exception
                gaohoward
                gaohoward Jul 14, 2010 10:52 PM (in response to mikemil)

                Hi,

                 

                you probably have hit this one: https://jira.jboss.org/browse/JBMESSAGING-1776

                 

                This issue may cause JCA not to receive next connection failure notification, therefore stops reconnection.

                 

                The scenario could be:

                 

                1. connection failure happened, JCA tries to reconnect.

                2. During reconnect, the JCA will create a new connection internally and register a exception listener.

                3. Just after the new connection has been created but before the listener is registered, the new connection is broken.

                4. So the JCA eventually registers a listener on a already 'bad' connection.

                5. due to this issue, JCA won't get notified for this bad connection and thinks it a good one.

                6. So the MDB won't get any more messages since.

                  • Actions
                • 5. Re: Remote MDBs seem to stop after marshalling exception
                  mikemil
                  mikemil Jul 14, 2010 11:05 PM (in response to gaohoward)

                  Does that defect apply to Messaging 1.4.4, because it doesn't appear to be listed in that defect?

                   

                  Thanks for the response.

                    • Actions
                  • 6. Re: Remote MDBs seem to stop after marshalling exception
                    gaohoward
                    gaohoward Jul 15, 2010 2:13 AM (in response to mikemil)

                    I think so.

                      • Actions