1 Reply Latest reply: Aug 20, 2011 10:23 PM by Ron Sigal RSS

detected failure on control connection

Tim Eapen Newbie

Dear All,

 

We are using:

 

JBoss EAP 4.3.0GA_CP08

JBoss Remoting 2.2.3.SP2-brew

JBoss Messaging 1.4.0.SP3_CP10-brew

 

We have an application that receives notification using JMS Topics.  One of our QA testers has noticed that if the client is left up over night the client seems to stop receiving notifications.

 

We see the following 'control connection' warning on the client:

 

2011-06-01 14:56:39,835 WARN 2011-06-01 09:34:16,199 WARN org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlMonitorTimerTask@12a9dc0: detected failure on control connection Thread[control: fdfc58[SSL_RSA_WITH_RC4_128_MD5: Socket[addr=/10.90.7.1,port=4458,localport=3823]],5,] (5c4o02a-q8t994-god83ybh-1-god83yyh-b: requesting new control connection

 

 

We see the marshalling error on the server side due to a java.io.IOException: Connection timed out error:

 

09:34:03,699 ERROR [SocketClientInvoker] Got marshalling exception, exiting

java.io.IOException: Connection timed out

at java.net.SocketOutputStream.socketWrite0(Native Method)

at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)

at java.net.SocketOutputStream.write(SocketOutputStream.java:136)

at com.sun.net.ssl.internal.ssl.OutputRecord.writeBuffer(OutputRecord.java:295)

at com.sun.net.ssl.internal.ssl.OutputRecord.write(OutputRecord.java:284)

at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:734)

at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:722)

at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59)

at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)

at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)

at java.io.DataOutputStream.flush(DataOutputStream.java:106)

at org.jboss.jms.wireformat.ClientDelivery.write(ClientDelivery.java:93)

at org.jboss.jms.wireformat.JMSWireFormat.write(JMSWireFormat.java:237)

at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.versionedWrite(MicroSocketClientInvoker.java:1060)

at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:667)

at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:458)

at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:141)

at org.jboss.remoting.Client.invoke(Client.java:1935)

at org.jboss.remoting.Client.invoke(Client.java:788)

at org.jboss.remoting.Client.invokeOneway(Client.java:839)

at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallback(ServerInvokerCallbackHandler.java:829)

at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallbackOneway(ServerInvokerCallbackHandler.java:700)

at org.jboss.jms.server.endpoint.ServerSessionEndpoint.performDelivery(ServerSessionEndpoint.java:1467)

at org.jboss.jms.server.endpoint.ServerSessionEndpoint.handleDelivery(ServerSessionEndpoint.java:1379)

at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.handle(ServerConsumerEndpoint.java:327)

at org.jboss.messaging.core.impl.RoundRobinDistributor.handle(RoundRobinDistributor.java:119)

at org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.handle(MessagingQueue.java:590)

at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.handle(ClusterRoundRobinDistributor.java:79)

at org.jboss.messaging.core.impl.ChannelSupport.deliverInternal(ChannelSupport.java:665)

at org.jboss.messaging.core.impl.MessagingQueue.deliverInternal(MessagingQueue.java:513)

at org.jboss.messaging.core.impl.ChannelSupport.handle(ChannelSupport.java:246)

at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2442)

at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:518)

at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:779)

at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:399)

at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)

at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)

at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)

at sun.reflect.GeneratedMethodAccessor2519.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)

at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)

at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)

at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)

at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:157)

at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:862)

at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:101)

at org.jboss.remoting.Client.invoke(Client.java:1935)

at org.jboss.remoting.Client.invoke(Client.java:788)

at org.jboss.remoting.Client.invoke(Client.java:776)

at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:189)

at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160)

at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$send$aop(ClientSessionDelegate.java:499)

at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)

at org.jboss.jms.client.container.SessionAspect.handleSend(SessionAspect.java:661)

at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect28.invoke(SessionAspect28.java)

at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)

at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)

at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)

at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)

at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:172)

at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)

at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)

at org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java)

at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:276)

at org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect39.invoke(ProducerAspect39.java)

at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)

at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:172)

at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)

at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)

at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java)

at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:165)

at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:208)

at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:146)

at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:137)

at org.jboss.jms.client.JBossMessageProducer.publish(JBossMessageProducer.java:178)

at com.sdm.hw.notification.jms.JmsNotifcationPublisher.publish(JmsNotifcationPublisher.java:59)

at com.sdm.hw.batchworkflow.domain.BatchEventPublicationServiceImpl.send(BatchEventPublicationServiceImpl.java:126)

at com.sdm.hw.batchworkflow.domain.BatchEventPublicationServiceImpl.sendChanged(BatchEventPublicationServiceImpl.java:67)

at com.sdm.hw.batchworkflow.domain.BatchEventPublicationServiceImpl.publishBatchForQuickFeedback(BatchEventPublicationServiceImpl.java:32)

at com.sdm.hw.batchworkflow.domain.BatchCreator$5.run(BatchCreator.java:157)

at com.sdm.hw.command.handler.CodeBlockCommand.invoke(CodeBlockCommand.java:15)

at com.sdm.hw.general.service.ejb.CommandInvoker.handleRequest(CommandInvoker.java:83)

at com.sdm.hw.general.service.ejb.CommandInvoker.invoke(CommandInvoker.java:36)

at com.sdm.hw.general.service.ejb.CommandTransactionSupportServiceBean.invoke(CommandTransactionSupportServiceBean.java:46)

at sun.reflect.GeneratedMethodAccessor238.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.jboss.invocation.Invocation.performCall(Invocation.java:359)

at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:237)

at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:169)

at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)

at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)

at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:404)

at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)

at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:168)

at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)

at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)

at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:648)

at org.jboss.ejb.Container.invoke(Container.java:960)

at org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java:430)

at org.jboss.ejb.plugins.local.StatelessSessionProxy.invoke(StatelessSessionProxy.java:103)

at $Proxy1029.invoke(Unknown Source)

at com.sdm.hw.general.service.ejb.CommandServiceBean.executeInTransaction(CommandServiceBean.java:132)

at com.sdm.hw.general.service.ejb.CommandServiceBean.invoke(CommandServiceBean.java:99)

at sun.reflect.GeneratedMethodAccessor237.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.jboss.invocation.Invocation.performCall(Invocation.java:359)

at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:237)

at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:169)

at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)

at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)

at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:378)

at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)

at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:168)

at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)

at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)

at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:648)

at org.jboss.ejb.Container.invoke(Container.java:960)

at org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java:430)

at org.jboss.ejb.plugins.local.StatelessSessionProxy.invoke(StatelessSessionProxy.java:103)

at $Proxy1054.invoke(Unknown Source)

at com.sdm.hw.command.ServerRequestProcessor.processRequest(ServerRequestProcessor.java:50)

at com.sdm.hw.command.ServerRequestProcessor.processRequest(ServerRequestProcessor.java:36)

at com.sdm.hw.command.Requestor.invoke(Requestor.java:32)

at com.sdm.hw.command.TransactionService.invoke(TransactionService.java:32)

at com.sdm.hw.command.TransactionService.invoke(TransactionService.java:24)

at com.sdm.hw.batchworkflow.domain.BatchCreator.publishQuickFeedbackBatchCreate(BatchCreator.java:156)

at com.sdm.hw.batchworkflow.domain.BatchCreator.createBatch(BatchCreator.java:96)

at com.sdm.hw.batchworkflow.domain.BatchCreator.createBatchTimed(BatchCreator.java:81)

at com.sdm.hw.batchworkflow.domain.BatchCreator.create(BatchCreator.java:72)

at com.sdm.hw.batchworkflow.application.LaunchBatchAsyncCommand.invoke(LaunchBatchAsyncCommand.java:36)

at com.sdm.hw.general.service.ejb.CommandInvoker.handleRequest(CommandInvoker.java:83)

at com.sdm.hw.general.service.ejb.CommandInvoker.invoke(CommandInvoker.java:36)

at com.sdm.hw.general.service.ejb.CommandServiceBean.executeInTransaction(CommandServiceBean.java:128)

at com.sdm.hw.general.service.ejb.CommandServiceBean.invoke(CommandServiceBean.java:99)

at sun.reflect.GeneratedMethodAccessor237.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.jboss.invocation.Invocation.performCall(Invocation.java:359)

at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:237)

at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:169)

at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)

at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)

at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:378)

at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)

at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:168)

at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)

at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)

at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:648)

at org.jboss.ejb.Container.invoke(Container.java:960)

at org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java:430)

at org.jboss.ejb.plugins.local.StatelessSessionProxy.invoke(StatelessSessionProxy.java:103)

at $Proxy1054.invoke(Unknown Source)

at com.sdm.hw.command.ServerRequestProcessor.processRequest(ServerRequestProcessor.java:50)

at com.sdm.hw.general.service.ejb.AbstractAsynchronousCommandMDBean.onMessage(AbstractAsynchronousCommandMDBean.java:40)

at sun.reflect.GeneratedMethodAccessor4208.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.jboss.invocation.Invocation.performCall(Invocation.java:359)

at org.jboss.ejb.MessageDrivenContainer$ContainerInterceptor.invoke(MessageDrivenContainer.java:492)

at org.jboss.ejb.plugins.MessageDrivenInstanceInterceptor.invoke(MessageDrivenInstanceInterceptor.java:116)

at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)

at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)

at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:315)

at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)

at org.jboss.ejb.plugins.RunAsSecurityInterceptor.invoke(RunAsSecurityInterceptor.java:109)

at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)

at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)

at org.jboss.ejb.MessageDrivenContainer.internalInvoke(MessageDrivenContainer.java:399)

at org.jboss.ejb.Container.invoke(Container.java:960)

at org.jboss.ejb.plugins.jms.JMSContainerInvoker.invoke(JMSContainerInvoker.java:1139)

at org.jboss.ejb.plugins.jms.JMSContainerInvoker$MessageListenerImpl.onMessage(JMSContainerInvoker.java:1452)

at org.jboss.jms.asf.StdServerSession.onMessage(StdServerSession.java:266)

at org.jboss.jms.client.container.ClientConsumer.callOnMessageStatic(ClientConsumer.java:160)

at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:831)

at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect14.invoke(SessionAspect14.java)

at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)

at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:172)

at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)

at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)

at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)

at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)

at org.jboss.jms.asf.StdServerSession.run(StdServerSession.java:194)

at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:761)

at java.lang.Thread.run(Thread.java:619)

 

 

Is this a configuration problem?

  • 1. Re: detected failure on control connection
    Ron Sigal Master

    Hi Tim,

     

    Are you still seeing this problem?

     

    I'll repeat the remark I just made on another thread: 

     

    When you get a "detected failure on control connection" message, it means a PING from the server to the client isn't arriving in time.  These failures might be a sign of a congested network, in which case you might want to lengthen the PING cycle.  You can play with the parameters

     

    * pingFrequency: how often the ping is sent

     

    * pingWindowFactor: multiplied by pingFrequency to give the window within which the PING is expected.

     

    -Ron