1 Reply Latest reply on Aug 20, 2011 10:23 PM by ron_sigal

    detected failure on control connection

    tim.eapen

      Dear All,

       

      We are using:

       

      JBoss EAP 4.3.0GA_CP08

      JBoss Remoting 2.2.3.SP2-brew

      JBoss Messaging 1.4.0.SP3_CP10-brew

       

      We have an application that receives notification using JMS Topics.  One of our QA testers has noticed that if the client is left up over night the client seems to stop receiving notifications.

       

      We see the following 'control connection' warning on the client:

       

      2011-06-01 14:56:39,835 WARN 2011-06-01 09:34:16,199 WARN org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlMonitorTimerTask@12a9dc0: detected failure on control connection Thread[control: fdfc58[SSL_RSA_WITH_RC4_128_MD5: Socket[addr=/10.90.7.1,port=4458,localport=3823]],5,] (5c4o02a-q8t994-god83ybh-1-god83yyh-b: requesting new control connection

       

       

      We see the marshalling error on the server side due to a java.io.IOException: Connection timed out error:

       

      09:34:03,699 ERROR [SocketClientInvoker] Got marshalling exception, exiting

      java.io.IOException: Connection timed out

      at java.net.SocketOutputStream.socketWrite0(Native Method)

      at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)

      at java.net.SocketOutputStream.write(SocketOutputStream.java:136)

      at com.sun.net.ssl.internal.ssl.OutputRecord.writeBuffer(OutputRecord.java:295)

      at com.sun.net.ssl.internal.ssl.OutputRecord.write(OutputRecord.java:284)

      at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:734)

      at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:722)

      at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59)

      at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)

      at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)

      at java.io.DataOutputStream.flush(DataOutputStream.java:106)

      at org.jboss.jms.wireformat.ClientDelivery.write(ClientDelivery.java:93)

      at org.jboss.jms.wireformat.JMSWireFormat.write(JMSWireFormat.java:237)

      at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.versionedWrite(MicroSocketClientInvoker.java:1060)

      at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:667)

      at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:458)

      at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:141)

      at org.jboss.remoting.Client.invoke(Client.java:1935)

      at org.jboss.remoting.Client.invoke(Client.java:788)

      at org.jboss.remoting.Client.invokeOneway(Client.java:839)

      at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallback(ServerInvokerCallbackHandler.java:829)

      at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallbackOneway(ServerInvokerCallbackHandler.java:700)

      at org.jboss.jms.server.endpoint.ServerSessionEndpoint.performDelivery(ServerSessionEndpoint.java:1467)

      at org.jboss.jms.server.endpoint.ServerSessionEndpoint.handleDelivery(ServerSessionEndpoint.java:1379)

      at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.handle(ServerConsumerEndpoint.java:327)

      at org.jboss.messaging.core.impl.RoundRobinDistributor.handle(RoundRobinDistributor.java:119)

      at org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.handle(MessagingQueue.java:590)

      at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.handle(ClusterRoundRobinDistributor.java:79)

      at org.jboss.messaging.core.impl.ChannelSupport.deliverInternal(ChannelSupport.java:665)

      at org.jboss.messaging.core.impl.MessagingQueue.deliverInternal(MessagingQueue.java:513)

      at org.jboss.messaging.core.impl.ChannelSupport.handle(ChannelSupport.java:246)

      at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2442)

      at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:518)

      at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:779)

      at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:399)

      at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)

      at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)

      at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)

      at sun.reflect.GeneratedMethodAccessor2519.invoke(Unknown Source)

      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

      at java.lang.reflect.Method.invoke(Method.java:597)

      at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)

      at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)

      at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)

      at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)

      at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:157)

      at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:862)

      at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:101)

      at org.jboss.remoting.Client.invoke(Client.java:1935)

      at org.jboss.remoting.Client.invoke(Client.java:788)

      at org.jboss.remoting.Client.invoke(Client.java:776)

      at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:189)

      at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160)

      at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$send$aop(ClientSessionDelegate.java:499)

      at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)

      at org.jboss.jms.client.container.SessionAspect.handleSend(SessionAspect.java:661)

      at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect28.invoke(SessionAspect28.java)

      at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)

      at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)

      at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)

      at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)

      at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:172)

      at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)

      at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)

      at org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java)

      at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:276)

      at org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect39.invoke(ProducerAspect39.java)

      at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)

      at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:172)

      at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)

      at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)

      at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java)

      at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:165)

      at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:208)

      at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:146)

      at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:137)

      at org.jboss.jms.client.JBossMessageProducer.publish(JBossMessageProducer.java:178)

      at com.sdm.hw.notification.jms.JmsNotifcationPublisher.publish(JmsNotifcationPublisher.java:59)

      at com.sdm.hw.batchworkflow.domain.BatchEventPublicationServiceImpl.send(BatchEventPublicationServiceImpl.java:126)

      at com.sdm.hw.batchworkflow.domain.BatchEventPublicationServiceImpl.sendChanged(BatchEventPublicationServiceImpl.java:67)

      at com.sdm.hw.batchworkflow.domain.BatchEventPublicationServiceImpl.publishBatchForQuickFeedback(BatchEventPublicationServiceImpl.java:32)

      at com.sdm.hw.batchworkflow.domain.BatchCreator$5.run(BatchCreator.java:157)

      at com.sdm.hw.command.handler.CodeBlockCommand.invoke(CodeBlockCommand.java:15)

      at com.sdm.hw.general.service.ejb.CommandInvoker.handleRequest(CommandInvoker.java:83)

      at com.sdm.hw.general.service.ejb.CommandInvoker.invoke(CommandInvoker.java:36)

      at com.sdm.hw.general.service.ejb.CommandTransactionSupportServiceBean.invoke(CommandTransactionSupportServiceBean.java:46)

      at sun.reflect.GeneratedMethodAccessor238.invoke(Unknown Source)

      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

      at java.lang.reflect.Method.invoke(Method.java:597)

      at org.jboss.invocation.Invocation.performCall(Invocation.java:359)

      at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:237)

      at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:169)

      at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)

      at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)

      at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:404)

      at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)

      at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:168)

      at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)

      at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)

      at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:648)

      at org.jboss.ejb.Container.invoke(Container.java:960)

      at org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java:430)

      at org.jboss.ejb.plugins.local.StatelessSessionProxy.invoke(StatelessSessionProxy.java:103)

      at $Proxy1029.invoke(Unknown Source)

      at com.sdm.hw.general.service.ejb.CommandServiceBean.executeInTransaction(CommandServiceBean.java:132)

      at com.sdm.hw.general.service.ejb.CommandServiceBean.invoke(CommandServiceBean.java:99)

      at sun.reflect.GeneratedMethodAccessor237.invoke(Unknown Source)

      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

      at java.lang.reflect.Method.invoke(Method.java:597)

      at org.jboss.invocation.Invocation.performCall(Invocation.java:359)

      at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:237)

      at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:169)

      at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)

      at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)

      at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:378)

      at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)

      at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:168)

      at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)

      at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)

      at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:648)

      at org.jboss.ejb.Container.invoke(Container.java:960)

      at org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java:430)

      at org.jboss.ejb.plugins.local.StatelessSessionProxy.invoke(StatelessSessionProxy.java:103)

      at $Proxy1054.invoke(Unknown Source)

      at com.sdm.hw.command.ServerRequestProcessor.processRequest(ServerRequestProcessor.java:50)

      at com.sdm.hw.command.ServerRequestProcessor.processRequest(ServerRequestProcessor.java:36)

      at com.sdm.hw.command.Requestor.invoke(Requestor.java:32)

      at com.sdm.hw.command.TransactionService.invoke(TransactionService.java:32)

      at com.sdm.hw.command.TransactionService.invoke(TransactionService.java:24)

      at com.sdm.hw.batchworkflow.domain.BatchCreator.publishQuickFeedbackBatchCreate(BatchCreator.java:156)

      at com.sdm.hw.batchworkflow.domain.BatchCreator.createBatch(BatchCreator.java:96)

      at com.sdm.hw.batchworkflow.domain.BatchCreator.createBatchTimed(BatchCreator.java:81)

      at com.sdm.hw.batchworkflow.domain.BatchCreator.create(BatchCreator.java:72)

      at com.sdm.hw.batchworkflow.application.LaunchBatchAsyncCommand.invoke(LaunchBatchAsyncCommand.java:36)

      at com.sdm.hw.general.service.ejb.CommandInvoker.handleRequest(CommandInvoker.java:83)

      at com.sdm.hw.general.service.ejb.CommandInvoker.invoke(CommandInvoker.java:36)

      at com.sdm.hw.general.service.ejb.CommandServiceBean.executeInTransaction(CommandServiceBean.java:128)

      at com.sdm.hw.general.service.ejb.CommandServiceBean.invoke(CommandServiceBean.java:99)

      at sun.reflect.GeneratedMethodAccessor237.invoke(Unknown Source)

      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

      at java.lang.reflect.Method.invoke(Method.java:597)

      at org.jboss.invocation.Invocation.performCall(Invocation.java:359)

      at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:237)

      at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:169)

      at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)

      at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)

      at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:378)

      at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)

      at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:168)

      at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)

      at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)

      at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:648)

      at org.jboss.ejb.Container.invoke(Container.java:960)

      at org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java:430)

      at org.jboss.ejb.plugins.local.StatelessSessionProxy.invoke(StatelessSessionProxy.java:103)

      at $Proxy1054.invoke(Unknown Source)

      at com.sdm.hw.command.ServerRequestProcessor.processRequest(ServerRequestProcessor.java:50)

      at com.sdm.hw.general.service.ejb.AbstractAsynchronousCommandMDBean.onMessage(AbstractAsynchronousCommandMDBean.java:40)

      at sun.reflect.GeneratedMethodAccessor4208.invoke(Unknown Source)

      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

      at java.lang.reflect.Method.invoke(Method.java:597)

      at org.jboss.invocation.Invocation.performCall(Invocation.java:359)

      at org.jboss.ejb.MessageDrivenContainer$ContainerInterceptor.invoke(MessageDrivenContainer.java:492)

      at org.jboss.ejb.plugins.MessageDrivenInstanceInterceptor.invoke(MessageDrivenInstanceInterceptor.java:116)

      at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)

      at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)

      at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:315)

      at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)

      at org.jboss.ejb.plugins.RunAsSecurityInterceptor.invoke(RunAsSecurityInterceptor.java:109)

      at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)

      at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)

      at org.jboss.ejb.MessageDrivenContainer.internalInvoke(MessageDrivenContainer.java:399)

      at org.jboss.ejb.Container.invoke(Container.java:960)

      at org.jboss.ejb.plugins.jms.JMSContainerInvoker.invoke(JMSContainerInvoker.java:1139)

      at org.jboss.ejb.plugins.jms.JMSContainerInvoker$MessageListenerImpl.onMessage(JMSContainerInvoker.java:1452)

      at org.jboss.jms.asf.StdServerSession.onMessage(StdServerSession.java:266)

      at org.jboss.jms.client.container.ClientConsumer.callOnMessageStatic(ClientConsumer.java:160)

      at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:831)

      at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect14.invoke(SessionAspect14.java)

      at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)

      at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:172)

      at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)

      at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)

      at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)

      at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)

      at org.jboss.jms.asf.StdServerSession.run(StdServerSession.java:194)

      at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:761)

      at java.lang.Thread.run(Thread.java:619)

       

       

      Is this a configuration problem?

        • 1. Re: detected failure on control connection
          ron_sigal

          Hi Tim,

           

          Are you still seeing this problem?

           

          I'll repeat the remark I just made on another thread: 

           

          When you get a "detected failure on control connection" message, it means a PING from the server to the client isn't arriving in time.  These failures might be a sign of a congested network, in which case you might want to lengthen the PING cycle.  You can play with the parameters

           

          * pingFrequency: how often the ping is sent

           

          * pingWindowFactor: multiplied by pingFrequency to give the window within which the PING is expected.

           

          -Ron