[Please see http://kbase.redhat.com/faq/docs/DOC-15866 for the latest up-to-date version of this article]
Optimal mod_jk configuration
There are many potential problems associated with the default configuration of mod_jk. Let's say it is perfectly adequate for a very low traffic website, but when pushing any moderate to high load to mod_jk, there will be connection problems. This is not due to any bug in mod_jk whatsoever, however, it is because the default configuration makes no assumption about your existing hardware or potential load, so, therefore, it is not tuned accordingly.
Note that the configuration recommendations here are optimal as a base configuration to avoid many of the common problems users experience with mod_jk. There exist many other useful optimizations, but these depend on the environment and web application in use. See http://tomcat.apache.org/connectors-doc/reference/workers.html for details on all available mod_jk properties.
Let's take a look at a typical default configuration for Apache/Tomcat/mod_jk:
workers.properties
worker.list=loadbalancer,status worker.node1.port=8009 worker.node1.host=node1.mydomain.com worker.node1.type=ajp13 worker.node1.lbfactor=1 worker.node2.port=8009 worker.node2.host= node2.mydomain.com worker.node2.type=ajp13 worker.node2.lbfactor=1 worker.loadbalancer.type=lb worker.loadbalancer.balance_workers=node1,node2 worker.status.type=status
JBoss Web's (Tomcat) server.xml AJP snippet:
<Connector port="8009" address="${jboss.bind.address}" protocol="AJP/1.3"
emptySessionPath="true" enableLookups="false" redirectPort="8443" ></Connector>
Apache's httpd.conf:
<IfModule prefork.c> StartServers 8 MinSpareServers 5 MaxSpareServers 20 ServerLimit 256 MaxClients 256 MaxRequestsPerChild 4000 </IfModule>
The above configuration, under load, may cause mod_jk to be very slow and unresponsive, cause http errors, and cause half-closed connections. These problems can arise because there are no connection timeouts specified to take care of orphaned connections, no error handling properties defined in workers.properties, and no connection limits set in Apache and Tomcat.
First off, lets take care of Tomcat:
Configuring server.xml:
The main concern with server.xml is setting the connectionTimeout which
sets the SO_TIMEOUT of the underlying socket. So when a connection in
Tomcat hasn't had a request in the amount of time specified by
connectionTimeout, then the connection dies off. This is necessary because if the connection hasn't been used for a certain period of
time then there is the chance that it is half-close on the mod_jk end.
If the connection isn't closed there will be an inflation of threads
which can over time hit the maxThreads count in Tomcat then Tomcat will
not be able to accept any new connections. A connectionTimeout of 600000 (10 minutes) is a good number to start out with. There may be a situation where the connections are not being recycled fast enough, in this instance the connectionTimeout could be lowered to 60000 or 1 minute.
When setting connectionTimeout in Tomcat, mod_jk should also have
connect_timeout/prepost_timeout set, which allows detection that the
Tomcat connection has been closed and preventing a retry request.
The recommended value of maxThreads is 200 per CPU, so here we assume the server is a single core machine. If it has been quad core, we could push that value to 800, and more depending on RAM and other machine specs.
<Connector port="8009"
address="${jboss.bind.address}"
emptySessionPath="true"
enableLookups="false"
redirectPort="8443"
protocol="AJP/1.3"
maxThreads="200"
connectionTimeout="600000"></Connector>
Configuring workers.properties:
See comments inline.
worker.list=loadbalancer,status worker.template.port=8009 worker.template.type=ajp13 worker.template.lbfactor=1 #ping_timeout was introduced in 1.2.27 worker.template.ping_timeout=1000 #ping_mode was introduced in 1.2.27, if not using 1.2.27 please specify connect_timeout=10000 and prepost_timeout=10000 as an alternative worker.template.ping_mode=A worker.template.socket_timeout=10 #It is not necessary to specify connection_pool_timeout if you are running the worker mpm worker.template.connection_pool_timeout=600 #Referencing the template worker properties makes the workers.properties shorter and more concise worker.node1.reference=worker.template worker.node1.host=192.168.1.2 worker.node2.reference=worker.template worker.node2.host=192.168.1.3 worker.loadbalancer.type=lb worker.loadbalancer.balance_workers=node1,node2 worker.loadbalancer.sticky_session=True worker.status.type=status
The key points in the above workers.properties is we've added limits for the connections mod_jk makes. With the base configuration, socket timeouts default to infinite. The other important properties are ping_mode and ping_timeout which handle probing a connection for errors and connection_pool_timeout which must be set to equal server.xml's connectionTimeout when using the prefork mpm. When these two values are the same, after a connection has been inactive for x amount of time, the connection in mod_jk and Tomcat will be closed at the same time, preventing a half-closed connection.
Configuring Apache
Make note that maxThreads for the AJP connection should coincide with
the MaxClients set in Apache's httpd.conf. MaxClients needs to be set
in the correct module in Apache.
This can be determined by running httpd -V:
# httpd -V Server version: Apache/2.2.3 Server built: Sep 11 2006 09:43:05 Server's Module Magic Number: 20051115:3 Server loaded: APR 1.2.7, APR-Util 1.2.8 Compiled using: APR 1.2.7, APR-Util 1.2.7 Architecture: 32-bit Server MPM: Prefork threaded: no forked: yes (variable process count) Server compiled with.... -D APACHE_MPM_DIR="server/mpm/prefork" -D APR_HAS_SENDFILE -D APR_HAS_MMAP -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) -D APR_USE_SYSVSEM_SERIALIZE -D APR_USE_PTHREAD_SERIALIZE -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT -D APR_HAS_OTHER_CHILD -D AP_HAVE_RELIABLE_PIPED_LOGS -D DYNAMIC_MODULE_LIMIT=128 -D HTTPD_ROOT="/etc/httpd" -D SUEXEC_BIN="/usr/sbin/suexec" -D DEFAULT_PIDLOG="logs/httpd.pid" -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" -D DEFAULT_LOCKFILE="logs/accept.lock" -D DEFAULT_ERRORLOG="logs/error_log" -D AP_TYPES_CONFIG_FILE="conf/mime.types" -D SERVER_CONFIG_FILE="conf/httpd.conf"
Which tells me the Server MPM is Prefork. This is not always 100% accurate so you should also view the output of /etc/sysconfig/httpd to see if the following line is there: HTTPD=/usr/sbin/httpd.worker. If it is commented out you are running prefork, otherwise if uncommented worker.
httpd.conf:
<IfModule prefork.c> StartServers 8 MinSpareServers 5 MaxSpareServers 20 MaxClients 200 MaxRequestsPerChild 0 </IfModule>
Or if Apache is using worker, it is <IfModule worker.c> StartServers 2 MaxClients 200 MinSpareThreads 25 MaxSpareThreads 75 ThreadsPerChild 25 MaxRequestsPerChild 0 </IfModule>
MaxRequestsPerChild is 0, this is the recommended value when using
mod_jk as mod_jk keeps open persistent connections. The key values in
the above configuration are MaxClients and MaxRequestsPerChild, the rest
of the values are left as default. Note that MaxRequestsPerChild is
recommended to be 0 however the value may need to be greater than 0
depending on if Apache is used for other modules also, especially in the
case of resource leakage.
Advanced worker-mpm Configuration
To get the most out of your mod_jk setup you should be using Apache's worker mpm which provides a definite performance improvement over the prefork mpm. The following section will detail how to configure Apache/mod_jk/Tomcat with the worker mpm and the math behind the configuration.
Let's start out with the worker mpm configuration
<IfModule mpm_worker_module> ThreadLimit 100 StartServers 5 MaxClients 1000 MinSpareThreads 100 MaxSpareThreads 1000 ThreadsPerChild 100 MaxRequestsPerChild 0 </IfModule>
The optimal configuration completely depends on the hardware being used and the load requirements. But a general rule of thumb, keep processes low and thread count high. To determine the number of processes Apache will use simply divide MaxClients by ThreadPerChild. So in this case MaxClients (1000) / ThreadsPerChild (100) = Processes (10), so Apache will allocate a maximum of 100 threads per each 10 child processes resulting in a total of 1000 possible clients.
Now to translate this to mod_jk, mod_jk maintains a connection pool for each worker defined in workers.properties. By default with Apache mod_jk sets connection_pool_size to ThreadsPerChild, so in the above case that would translate to 100, giving 1000 possible connections to JBoss. This may or may not be desired.
Let's take a common example, there will be 3 JBoss servers that combined needed to be able to handle 900 concurrent connections
worker.list=loadbalancer,status worker.template.type=ajp13 worker.template.port=8009 worker.template.ping_mode=A worker.template.connection_pool_size=30 worker.template.socket_timeout=10 worker.template.retries=20 worker.node1.reference=worker.template worker.node1.host=192.168.0.101 worker.node2.reference=worker.template worker.node2.host=192.168.0.102 worker.node3.reference=worker.template worker.node3.host=192.168.0.103 worker.loadbalancer.type=lb worker.loadbalancer.balance_workers=node1,node2, node3 worker.loadbalancer.sticky_session=True worker.status.type=status
The above configuration tells mod_jk to multiplex 30 connections to the available Apache processes, which is 10 processes. So that means 30 connections multiplexed over 10 processes gives 300 possible connections to each backend worker. Furthermore, the total connections able to be used in this configuration from Apache is 900 which means 100 connections will be left over for static content or whatnot.
Next configure maxThreads in each ajp connector to match the above.
Node 1 ajp connector:
<Connector port="8009"
address="${jboss.bind.address}"
emptySessionPath="true"
enableLookups="false"
redirectPort="8443"
protocol="AJP/1.3"
maxThreads="300"
connectionTimeout="600000"></Connector>
Node 2 ajp connector:
<Connector port="8009"
address="${jboss.bind.address}"
emptySessionPath="true"
enableLookups="false"
redirectPort="8443"
protocol="AJP/1.3"
maxThreads="300"
connectionTimeout="600000"></Connector>
Node 3 ajp connector:
<Connector port="8009"
address="${jboss.bind.address}"
emptySessionPath="true"
enableLookups="false"
redirectPort="8443"
protocol="AJP/1.3"
maxThreads="300"
connectionTimeout="600000"></Connector>
Remember when using connectionTimeout which is always recommended, prepost_timeout and connect_timeout also need to be set, which is done. I'm not showing sticky session configuration, but that covered in the main mod_jk article in using mod_jk with JBoss.
Referenced by:
Comments