8 Replies Latest reply on May 24, 2012 11:39 AM by dafr

    HornetQ client keeps reconnecting when JBoss server behind reverse proxy is down

    dafr

      We have a HornetQ client that is getting messages over HTTPS from a server that is behind a reverse proxy (mod_proxy_ajp). All goes well under normal conditions. However, if the server is down, the client starts to reconnect to the reverse proxy as fast as possible (several hundred connections per second per client).

      When profiling, it looks like ClientSessionFactoryImpl.failoverOrReconnect() is doing this. Things like retryInterval, maxRetryInterval, reconnectAttempts are all set to their default value but probably don't have anything to do with this as the connection can be made to the proxy. The client does not get an answer from the server as expected but gets a 503 from the apache.

      Can someone give at least a hint please? Thanks.

        • 1. Re: HornetQ client keeps reconnecting when JBoss server behind reverse proxy is down
          ataylor

          are you saying the retry interval is being ignored, could you debug to see what is happening in getConnectionWithRetry(), we dont have this sort of setup with a proxy to hand so i cant really test it.

          • 2. Re: HornetQ client keeps reconnecting when JBoss server behind reverse proxy is down
            dafr

            The retry interval (and other parameters regarding reconnection) only applies to the making of the connection if I'm not mistaken.

            If the server is down but apache is still running, the client is able to connect (to the apache) and gets a 503 before the connection is closed. getConnectionWithRetry() is called after this. getConnectionWithRetry is called a few hundred times per second.

            If the apache is down, getConnectionWithRetry() is called once every 11 seconds or so which is expected behaviour as a connection cannot be made.

            (I'm looking towards connectionDestroyed() at the moment.)

            • 3. Re: HornetQ client keeps reconnecting when JBoss server behind reverse proxy is down
              dafr

              I have attached two stack traces (export from TPTP) wherein getConnectionWithRetry() is called. In 10 seconds time, I get about 500 of these. There seem to be 2 slightly different stack traces so I included an example of each of these two types. As they are rather big, I included them as XML so you can easily expand/collapse each method in any editor that supports XML formatting.

              I hope this makes some sense to you.

              • 4. Re: HornetQ client keeps reconnecting when JBoss server behind reverse proxy is down
                clebert.suconic

                The retry interval is used at reconnection as well.

                 

                I never say never.. but I don't think there's a bug on this value. Feel free to prove it otherwise though.

                 

                Message was edited by: Clebert Suconic

                • 5. Re: HornetQ client keeps reconnecting when JBoss server behind reverse proxy is down
                  dafr

                  getConnectionWithRetry in ClientSessionFactoryImpl will do a retry after retryInterval with multiplier for reconnectAttempts, but only if connection == null. However, HornetQ is able to connect (to the apache reverse proxy) so connection != null and the call returns. You can also see this in the attached stack traces.

                   

                  {code:java}

                     private void getConnectionWithRetry(final int reconnectAttempts)

                     {

                        long interval = retryInterval;

                         int count = 0;

                         synchronized (waitLock)

                        {

                           while (true)

                           {

                              if (exitLoop)

                              {

                                 return;

                              }

                               if (log.isDebugEnabled())

                              {

                                 log.debug("Trying reconnection attempt " + count);

                              }

                               getConnection();

                               if (connection == null)

                              {

                                 // Failed to get connection

                                  if (reconnectAttempts != 0)

                                 {

                                    count++;

                                     if (reconnectAttempts != -1 && count == reconnectAttempts)

                                    {

                                       log.warn("Tried " + reconnectAttempts + " times to connect. Now giving up on reconnecting it.");

                                        return;

                                    }

                                     try

                                    {

                                       waitLock.wait(interval);

                                    }

                                    catch (InterruptedException ignore)

                                    {

                                    }

                                     // Exponential back-off

                                    long newInterval = (long)(interval * retryIntervalMultiplier);

                                     if (newInterval > maxRetryInterval)

                                    {

                                       newInterval = maxRetryInterval;

                                    }

                                     interval = newInterval;

                                 }

                                 else

                                 {

                                    return;

                                 }

                              }

                              else

                              {

                                 return;

                              }

                           }

                        }

                     }

                  {code}

                  • 6. Re: HornetQ client keeps reconnecting when JBoss server behind reverse proxy is down
                    ataylor

                    sounds like an issue we can fix, is there anyway you can provide what the proxy returns (Im assuming its an HTTP response) on the wire so we can debug, we dont have a proxy we can easily set up.

                    • 7. Re: HornetQ client keeps reconnecting when JBoss server behind reverse proxy is down
                      dafr

                      Attached file is a single HTTPS conversation between our client (that also happens to be a JBoss server) and the Apache reverse proxy, taken with Wireshark. I have decripted the SSL data but removed things like server hostnames and most of the content of the static 503 html page that Apache returns.

                      • 8. Re: HornetQ client keeps reconnecting when JBoss server behind reverse proxy is down
                        dafr

                        FYI: logging does not provide much info. Every 15ms we get the following lines (log level TRACE on org.jboss.netty):

                         

                         

                        {noformat}

                         

                        ...

                        2012-05-24 16:05:15,604 DEBUG [Thread-0 (group:HornetQ-client-global-threads-10943145)] [impl.ClientSessionFactoryImpl] Trying to connect at the main server using connector rg-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=443&host=local-mydomain-com&use-servlet=true&proxyHost=&proxyAuthenticateUser=&ssl-enabled=true&proxyPort=8080&servlet-path=/hornetq/HornetQServlet&proxyAuthenticatePassword=

                        2012-05-24 16:05:15,609 DEBUG [Thread-0 (group:HornetQ-client-global-threads-10943145)] [impl.ClientSessionFactoryImpl] Trying reconnection attempt 0

                        2012-05-24 16:05:15,609 DEBUG [Thread-0 (group:HornetQ-client-global-threads-10943145)] [netty.NettyConnector] Started Netty Connector version 3.2.4.Final-0b47c34

                        ...

                         

                        {noformat}

                         

                         

                        Any suggestions for a fix or a workaround?