-
1. Re: HornetQ client keeps reconnecting when JBoss server behind reverse proxy is down
ataylor Apr 20, 2012 3:25 AM (in response to dafr)are you saying the retry interval is being ignored, could you debug to see what is happening in getConnectionWithRetry(), we dont have this sort of setup with a proxy to hand so i cant really test it.
-
2. Re: HornetQ client keeps reconnecting when JBoss server behind reverse proxy is down
dafr Apr 20, 2012 11:16 AM (in response to ataylor)The retry interval (and other parameters regarding reconnection) only applies to the making of the connection if I'm not mistaken.
If the server is down but apache is still running, the client is able to connect (to the apache) and gets a 503 before the connection is closed. getConnectionWithRetry() is called after this. getConnectionWithRetry is called a few hundred times per second.
If the apache is down, getConnectionWithRetry() is called once every 11 seconds or so which is expected behaviour as a connection cannot be made.
(I'm looking towards connectionDestroyed() at the moment.)
-
3. Re: HornetQ client keeps reconnecting when JBoss server behind reverse proxy is down
dafr May 14, 2012 11:27 AM (in response to dafr)I have attached two stack traces (export from TPTP) wherein getConnectionWithRetry() is called. In 10 seconds time, I get about 500 of these. There seem to be 2 slightly different stack traces so I included an example of each of these two types. As they are rather big, I included them as XML so you can easily expand/collapse each method in any editor that supports XML formatting.
I hope this makes some sense to you.
-
handleConnectionFailure_2.xml 81.5 KB
-
handleConnectionFailure_1.xml 77.7 KB
-
-
4. Re: HornetQ client keeps reconnecting when JBoss server behind reverse proxy is down
clebert.suconic May 14, 2012 12:20 PM (in response to dafr)The retry interval is used at reconnection as well.
I never say never.. but I don't think there's a bug on this value. Feel free to prove it otherwise though.
Message was edited by: Clebert Suconic
-
5. Re: HornetQ client keeps reconnecting when JBoss server behind reverse proxy is down
dafr May 15, 2012 4:25 AM (in response to clebert.suconic)getConnectionWithRetry in ClientSessionFactoryImpl will do a retry after retryInterval with multiplier for reconnectAttempts, but only if connection == null. However, HornetQ is able to connect (to the apache reverse proxy) so connection != null and the call returns. You can also see this in the attached stack traces.
{code:java}
private void getConnectionWithRetry(final int reconnectAttempts)
{
long interval = retryInterval;
int count = 0;
synchronized (waitLock)
{
while (true)
{
if (exitLoop)
{
return;
}
if (log.isDebugEnabled())
{
log.debug("Trying reconnection attempt " + count);
}
getConnection();
if (connection == null)
{
// Failed to get connection
if (reconnectAttempts != 0)
{
count++;
if (reconnectAttempts != -1 && count == reconnectAttempts)
{
log.warn("Tried " + reconnectAttempts + " times to connect. Now giving up on reconnecting it.");
return;
}
try
{
waitLock.wait(interval);
}
catch (InterruptedException ignore)
{
}
// Exponential back-off
long newInterval = (long)(interval * retryIntervalMultiplier);
if (newInterval > maxRetryInterval)
{
newInterval = maxRetryInterval;
}
interval = newInterval;
}
else
{
return;
}
}
else
{
return;
}
}
}
}
{code}
-
6. Re: HornetQ client keeps reconnecting when JBoss server behind reverse proxy is down
ataylor May 15, 2012 5:54 AM (in response to dafr)sounds like an issue we can fix, is there anyway you can provide what the proxy returns (Im assuming its an HTTP response) on the wire so we can debug, we dont have a proxy we can easily set up.
-
7. Re: HornetQ client keeps reconnecting when JBoss server behind reverse proxy is down
dafr May 16, 2012 8:17 AM (in response to ataylor)Attached file is a single HTTPS conversation between our client (that also happens to be a JBoss server) and the Apache reverse proxy, taken with Wireshark. I have decripted the SSL data but removed things like server hostnames and most of the content of the static 503 html page that Apache returns.
-
8. Re: HornetQ client keeps reconnecting when JBoss server behind reverse proxy is down
dafr May 24, 2012 11:39 AM (in response to ataylor)FYI: logging does not provide much info. Every 15ms we get the following lines (log level TRACE on org.jboss.netty):
{noformat}
...
2012-05-24 16:05:15,604 DEBUG [Thread-0 (group:HornetQ-client-global-threads-10943145)] [impl.ClientSessionFactoryImpl] Trying to connect at the main server using connector rg-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=443&host=local-mydomain-com&use-servlet=true&proxyHost=&proxyAuthenticateUser=&ssl-enabled=true&proxyPort=8080&servlet-path=/hornetq/HornetQServlet&proxyAuthenticatePassword=
2012-05-24 16:05:15,609 DEBUG [Thread-0 (group:HornetQ-client-global-threads-10943145)] [impl.ClientSessionFactoryImpl] Trying reconnection attempt 0
2012-05-24 16:05:15,609 DEBUG [Thread-0 (group:HornetQ-client-global-threads-10943145)] [netty.NettyConnector] Started Netty Connector version 3.2.4.Final-0b47c34
...
{noformat}
Any suggestions for a fix or a workaround?