Thanks for the "nodeTimeout" pointer! There may be a documentation issue here: the default is documented at -1 (none), but in practice it seems to default to 300 seconds.

I don't think there is ever a ping error, at least I see nothing indicating that in error_log, even at "debug" level. The log snippet below suggests that there are successfull pings to the same instance between the timeout and the next STATUS (PINGs are performed over the remaining AJP connections that are still open):

[Fri Mar 18 11:14:44 2011] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header

[Fri Mar 18 11:14:44 2011] [error] ajp_read_header: ajp_ilink_receive failed

[Fri Mar 18 11:14:44 2011] [error] (120006)APR does not understand this error code: proxy: read response failed from 169.70.169.12:33832 (169.70.169.12)

[Fri Mar 18 11:14:44 2011] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state

...

[Fri Mar 18 11:14:44 2011] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state

[Fri Mar 18 11:14:47 2011] [debug] proxy_util.c(2011): proxy: ajp: has acquired connection for (169.70.169.12)

[Fri Mar 18 11:14:47 2011] [debug] proxy_util.c(2067): proxy: connecting ajp://169.70.169.12:43530/ to 169.70.169.12:43530

[Fri Mar 18 11:14:47 2011] [debug] proxy_util.c(2193): proxy: connected ajp://169.70.169.12:43530/ to 169.70.169.12:43530

[Fri Mar 18 11:14:47 2011] [debug] mod_proxy_cluster.c(703): ajp_cping_cpong: Done

[Fri Mar 18 11:14:44 2011] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state

...

I think what happens is that the worker is marked bad on the basis of a single connection timeout, even though other connections are still good and pings are still succeeding. Is there a configuration option to treat ping/pong error as the only reason to mark the worker bad, essentially ignoring individual connection timeouts?

Thank you,

--Alexey

After some more digging I understand how the worker is getting marked bad: the logic is in proxy_handler (mod_proxy.c):

      else if (access_status == HTTP_INTERNAL_SERVER_ERROR) {

            /* Unrecoverable server error.

             * We can not failover to another worker.

             * Mark the worker as unusable if member of load balancer

             */

            if (balancer) {

                worker->s->status |= PROXY_WORKER_IN_ERROR;

                worker->s->error_time = apr_time_now();

            }

            break;

        }

As long as AJP timeout is bubbled up as HTTP_INTERNAL_SERVER_ERROR, the worker is marked bad, and the request is reported as a 500 failure to the client. A more graceful way of handling the AJP header read timeout would be to report it as something less severe, for example HTTP_GATEWAY_TIME_OUT, which is still reported as a failure to the client, but has not effect on worker status. To try it out, I made a change in ap_proxy_ajp_request (mod_proxy_ajp.c):

    /* read the response */

    conn->data = NULL;

    status = ajp_read_header(conn->sock, r, maxsize,

                             (ajp_msg_t **)&(conn->data));

    if (status != APR_SUCCESS) {

        /* We had a failure: Close connection to backend */

        conn->close++;

        apr_brigade_destroy(input_brigade);

        ap_log_error(APLOG_MARK, APLOG_ERR, status, r->server,

                     "proxy: read response failed from %pI (%s)",

                     conn->worker->cp->addr,

                     conn->worker->hostname);

        /*

         * This is only non fatal when we have not sent (parts) of a possible

         * request body so far (we do not store it and thus cannot sent it

         * again) and the method is idempotent. In this case we can dare to

         * retry it with a different worker if we are a balancer member.

         */

        if (!send_body && (is_idempotent(r) == METHOD_IDEMPOTENT)) {

            return HTTP_SERVICE_UNAVAILABLE;

        }

        return HTTP_GATEWAY_TIME_OUT;

    }

This seems to do the trick - a request longer than "nodeTimeout" is reported as a 504, but the worker can continue serve other requests normally in parallel to the long request and after it times out, with no interruptions. My understanding is that a true worker hang would be detected regardless of this change, by the ping/pong probe.

Are there any hidden implications of such change I am missing?

Thank you,
--Alexey

Agreed. The full patch that I made in the end touches a couple more files:

ajp.h  at line 115:

#define AJP_ELOGFAIL            (APR_OS_START_USERERR + 9)

/** Bad requestmethod */

#defineAJP_EBAD_METHOD        (APR_OS_START_USERERR + 10)

+/** Send/receive timeout */

+#define AJP_ETIMEOUT            (APR_OS_START_USERERR + 10)

ajp_link.c  at line 95:

     if (status !=APR_SUCCESS) {

        ap_log_error(APLOG_MARK, APLOG_ERR, status, NULL,

                     "ajp_ilink_receive() can't receive header");

-        returnAJP_ENO_HEADER;

+        return(status == APR_TIMEUP ? AJP_ETIMEOUT : AJP_ENO_HEADER);

     }

mod_proxy_ajp.c at line 343 :

         if(!send_body && (is_idempotent(r) == METHOD_IDEMPOTENT)) {

             returnHTTP_SERVICE_UNAVAILABLE;

         }

+        if (status ==AJP_ETIMEOUT)

+        {

+               /*Connection may time out due to a long-running request, which

+                *means the worker deserves a second chance. Returning a 500 here

+                * wouldbring the worker to error status, a mild 504 will keep it

+                *usable for subsequent requests.

+                */

+            return HTTP_GATEWAY_TIME_OUT;

+        }

         return HTTP_INTERNAL_SERVER_ERROR;

     }

Do you think this is sufficiently selective? It definitely made a marked improvement in my case, so hopefully you could make a change like this in one of next releases of mod_cluster.

Thank you,
--Alexey