Mod-Cluster_Management_Protocol

The document describes the protocol elements of the mod_cluster protocol between a container (AS) and a load balancer (Apache httpd).

 

There 8 messages:

 

  • CONFIG: Send configuration information for a node or set of nodes.

  • ENABLE-APP: Send requests and assign new sessions to the specified app. Use of  to identify the app means enable all apps on the given node.

  • DISABLE-APP: apache should not create new session for this webapp, but still continue serving existing session on this node. Use of  to identify the app means disable all apps on the given node.

  • STOP-APP: New requests for this webapp should not be sent to this node. Use of  to identify the app means stop all apps on the given node.

  • STOP-APP-RSP: Response to STOP-APP (since version 0.2.0)
  • REMOVE-APP: Remove the information about this webapp from mod_cluster tables.

  • STATUS: Send the current load balance factor for this node (or a set of nodes). Periodically sent.

  • STATUS-RSP mod_cluster_manager response to a STATUS.

  • INFO: Request configuration info from mod_cluster_manager.

  • INFO-RSP: Response to INFO virtual host and listen address/port.

  • DUMP: Request a text dump of the current configuration seen by mod_cluster_manager.

  • DUMP-RSP: Response to DUMP.

  • PING: Request a ping to httpd or node. The node could defined by JVMRoute or Scheme,Host and Port (since version 0.1.0).
  • PING-RSP: Response to PING.

 

Message pay-load:

 

The information is in ASCII and URL encoded if needed.

All numbers are integer and a string representation is used.

The command is string it is the first part of a message it is followed by a / HTTP/1.0

<CR><LF>
Example:

DISABLE-APP / HTTP/1.0<CR><LF>

All parameters are send as <PARAMETER_NAME>=<VALUE> and the separator between their pairs is <&>

<SPACE> in value is escaped as <+>

<&> in value is escaped as &26

<=> in value is escaped %3D

<CR><LF> means end of <COMMAND>
example:

DISABLE / HTTP/1.0<CR><LF>

Content-length: 44<CR><LF>

<CR><LF>

Jvmroute=node1&Context=myapp&Context=ourapp<CR><LF>

 

To make the document more readable the " / HTTP/1.0" is not added a the end of the commands.

 

CONFIG:

CONFIGBalancer: <Balancer><balancer conf>JvmRoute: <JvmRoute>Domain: <Domain><Host: <Node IP>Port: <Connector Port>Type: <Type of the connector>(<node conf>)
(Alias: <vhost list>Context: <context list>)m

Where:

<Balancer> is the name of the balancer in httpd (for example mycluster in ProxyPass /myapp balancer://mycluster/). If not present the default value will be used. (max size: 40)

 

<balancer conf>

 

is the configuration of the balancer (See http://community.jboss.org/wiki/Mod-Clusternodebalancer).

 

<JvmRoute> is what is after the . in the JSESSIONID cookie or the parameter jsessionid. (max size: 80)

 

<Domain> is the domain corresponding to the node. (max size: 20)

 

<Node IP> is the IP address (or hostname) where the node is going to receive requests from httpd. (max size: 64)

 

<Connector Port> is the port on which the node except to receive requests. (max size: 7)

 

<Type of the connector> http/https/ajp The protocol to use between httpd and AS to process requests. (max size: 6)

 

<node conf>

 

is the configuration of the node (See http://community.jboss.org/wiki/Mod-Clusternodebalancer).

 

<vhost list>

 

List of virtual hosts like host1,host2 (See Alias in http://tomcat.apache.org/tomcat-6.0-doc/config/host.html#Host Name Aliases

 

<context list>

 

List the context the virtual host list supports like /myapp,/ourapp.

ENABLE-APP:

ENABLE-APP

JvmRoute: <JvmRoute>

Alias: <vhost list>

Context: </context>

 

<JvmRoute> See CONFIG.

<vhost list> See CONFIG.

</context> That is the context of the application. </>

is the ROOT context.

 

DISABLE-APP:

Like ENABLE-APP.

 

STOP-APP:

Like ENABLE-APP.

 

REMOVE-APP:

Like ENABLE-APP.

 

STOP-APP-RSP:

That is a reponse to STOP-APP, like a http reponse it starts with HTTP/1.1 200 OK

<CR><LF>

and it has the format:

HTTP/1.1 200 OK

Type: STOP-APP-RSP

JvmRoute: <JvmRoute>

Alias: <vhost list>

Context: </context>

Requests: <num_requests>

<num_requests> That the number of requests active on the context at the time of STOP-APP command

Note that a Wildcard STOP-APP won't cause  a STOP-APP-RSP but just a 200 or 500

STATUS:

STATUS

JvmRoute: <JvmRoute>

Load: <Load Factor>

 

<JvmRoute> See CONFIG.

<Load Factor>

That is a number (string) It is a number between 1 and 100 and defines the normalized weighted load applied to the worker. Only possible values are real load fator. The value "0" means the node is on standby. "-1" means the node is broken (The cluster can't connect to it).

 

STATUS-RSP:

That is a reponse, like a http reponse it starts with HTTP/1.1 200 OK

<CR><LF>

and it has the format:

 

HTTP/1.1 200 OK

Type: STATUS-RSP

JvmRoute: <JvmRoute>

State: <State>

Id: <Id>

 

<JvmRoute> See CONFIG.
<State> That is the state httpd see for the node at the time of the processing (value from shared memory).
<Id> That is the generation id of process in httpd if it changes (increases) when httpd has been restarted and its view of the cluster

configuration could be incorrect. In this case ModClusterService should send a new CONFIG ASAP so the information could be updated.

 

INFO:

INFO

 

INFO-RSP:

INFO-RSP

vhost: <vhost>

vport: <vport>

rhost: <rhost>

rport: <rport>

 

<vhost> and <vport> are the host and port defined in the ServerName directive.
<rhost> and <rport>

are the host and port that has received the INFO request.

 

DUMP:

DUMP

 

DUMP-RSP:

HTTP/1.1 200 OK

Type: DUMP-RSP

Resp: <string>

 

<string>

is a "readable" description of the mod_cluster configuration corresponding to the messages the cluster has sent to mod_cluster.

 

PING:

There are 3 different pings: (for version 0.1.0 on)

PING

It is used to check that the proxy is alive.

PING

JvmRoute: <JvmRoute>

It is used to check if a node is alive.

PING

Type: <Type>

Host: <Host>

Port: <Port>

It is used to check that httpd can connect to a possible node defined by Type://Host:Port/

 

<JvmRoute> See CONFIG.

<Type> See CONFIG (node configuration).

<Host> See CONFIG (node configuration).

<Port> See CONFIG (node configuration)

 

PING-RSP:

That is a reponse, like a http reponse it starts with HTTP/1.1 200 OK

<CR><LF>

and it has the format:

 

HTTP/1.1 200 OK

Type: PING-RSP

State: <State>

Id: <Id>

 

<State> See STATUS_RSP.

<Id> See STATUS_RSP.

 

Using -APP command with wildcard:

In case a -APP command with wildcard is sent by ModClusterManager to mod_cluster only the JVMRoute is going to be relevant in the payload message.

 

 

For example:

 

+++

 

DISABLE  HTTP/1.0

<CR><LF>

Content-length: 44<CR><LF>

<CR><LF>

Jvmroute=node1&Context=myapp&Context=ourapp<CR><LF>

+++


Will be handled like:

+++

DISABLE  HTTP/1.0<CR><LF>

Content-length: 15<CR><LF>

<CR><LF>

Jvmroute=node1<CR><LF>

 

+++

 

 

Other values between the command and HTTP/1.0 (or 1.1) are ignored in the actual version of the protocol.

 

A shutdown of a node will cause the following events:

 

DISABLE-APP / for each application.

 

STOP-APP / for each application.

 

DISABLE-APP

 

STOP-APP

 

(http://community.jboss.org/wiki/ModClusterDesign suggests that ModClusterManager should wait until all sessions have been finished but that requires a to be written tool. The idea is that an administrator initiated step; similar to what people do now by changing workers.properties to quiesce a node in mod_jk, but it could be initiated from the JBoss side via a management tool).

If a request arrives for a context corresponding to this node 500 will be returned to the client.

 

An additional utility could be written to send a REMOVE-APP  once the JBoss node is stopped REMOTE-APP will remove all the node information from mod_cluster table and any socket between httpd and the node will be closed. (For a more complete description see http://wiki.jboss.org/wiki/Mod-Cluster_Internals). If a request arrives for a context corresponding to this node 404 will be returned to the client: in fact the mod_proxy will not be called for the request and an httpd page could be displayed. A REMOVE-APP / for example will just clean the mod_cluster table corresponding to the application defined in the payload.

 

Error handling:

Once a error occurs in mod_cluster 500 is returned and the status line tells the version of protocol mod_cluster is using and a short description of the error.

The error response status line will be something like:

HTTP/1.1 500 ERROR

 

Version: version_supported

 

Type: type

 

Mess: "error_string"

 

Where version_supported is the version of the protocol mod_cluster is able to support.

 

Where type is the type of error, for example: MEM to tell the message contains syntax errors (SYNTAX) or the data can't be updated to update the shared memory (MEM).

 

The "error_string" should help to understand what was wrong and the n.p.n VERSION information tells which highest version of the protocol mod_cluster understands.

 

The first part of the error_string should help to make a decision how to go on:

 

SYNTAX: mod_cluster can't understand the message or part of the message. Another version of the protocol should used or ModClusterService should be fixed.

 

MEM: mod_cluster can't update the shared memory. If that is the answer to a -APP messages or a STATUS message new CONFIG message should be send. If it is the answer to a CONFIG message the configuration of the cluster in ModCusterService should be checked or/and the CONFIG message should be resend to mod_cluster.

Example:

HTTP/1.1 500 ERROR
Date: Fri, 16 May 2008 09:55:21 GMT
Server: Apache/2.2.9-dev (Unix) mod_ssl/2.2.9-dev OpenSSL/0.9.8b DAV/2
Content-Length: 557
Connection: close
Version: 0.0.0
Type: SYNTAX
Mess: "Command is not supported"
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

mod_cluster-manager handler:

The mod_cluster-manager handler allows to do operation like ENABLE_APP/DISABLE_APP through a web interface. The format of the request string is the following:

Nonce: <nonce>Cmd: <cmd>Range: <range>MCMP String

where:

<nonce> Is a string like e17066b4-0cb1-4e58-93e3-cdc9efb6be9 corresponding to a unique id of httpd.

<cmd> Is the command: one of ENABLE_APP, DISABLE_APP etc.

<range> Is a "NODE" or "CONTEXT". "NODE" means that the _APP command is a wildcard command.

The MCMP String is a string containing a command described above.

Example:

http://localhost:8000/mod_cluster-manager?nonce=e17066b4-0cb1-4e58-93e3-cdc9efb6be9c&Cmd=DISABLE-APP&Range=CONTEXT&JVMRoute=jvm1&Alias=