Using Apache HTTPD as Reverse Proxy and Load Balancer for custom Netty HTTP engine

JBoss Netty has been my first choice for building a scalable HTTP server engine sans Servlet API and Containers, ever since I first discovered it around 1.5 years back. Its USP is its simplicity combined with extreme performance.

 

Here, I shall show a simple example of creating HTTP server with JBoss Netty which creates and provides a session identity to the user. When the user connects by providing the session identity as a URL parameter and without the name, it welcomes the user with its name. We will use this simple server as an example to do reverse proxy and then load balancing with sticky-sessions with Apache Web Server’s mod_proxy and mod_proxy_balancer modules. All source code as well as the httpd.conf file is in the attachment.

 

First, we need to create a handler, much like a Servlet that would receive HTTP requests. HTTP requests can be chunked and would have to be combined at the server side before processing. This chunking is not typical of high-bandwidth Internet requests, but is very much possible with for example, long range wireless mediums.

So first, we have an abstract template handler for receiving HTTP requests called AbstractHttpHandler. This extends Netty's SimpleChannelUpstreamHandler and assembles chunked HTTP requests in its messageReceived method. When a complete request is assembled, the abstract template method handleRequest is called with the complete HttpRequest and the channel.

 

Now, we create an implementation of AbstractHttpHandler which assigns unique sessions to users and later identifies the users from their sessions. The SimpleHttpHandler is created with a server instance id and a session instance separator. Each server is provided a unique identity in a cluster of servers. Sessions created for users would be given a session id with the following format - <Unique Id created><session instance separator><server instance id>. Thus, if the server instance name is ‘netty1’ and session instance separator is ‘.’ and the unique id created is ‘2C855D9FB0710111’, then the session id given to the user would be the concatenated value ‘2C855D9FB0710111.netty1’. This would help Apache Web Server to maintain sticky sessions, as it would be able to parse the session id and find that it has been created by netty1 instance and all the next requests with this session id must be passed to this instance.

 

If the request provides the username, we simply create a new session id and store the user session details in our local memory map in SessionIdHolder. We send the response to the user – ‘Session ID is <…..>’. In the next request, the user can copy this session id from the response and send it as a request parameter. Then the server extracts the user session details mapped to the session id, after stripping off the server instance id, and finds out the corresponding user name. It sends the response ‘Welcome user <….> to server <…>’. If it does not find the session in its map, it returns the response ‘No valid session, please login with user’.

 

The sessions can be shared across servers using replicated caching or a shared database. However, to keep our example simple, and to illustrate load balancing with session stickiness, we have not replicated the sessions.

 

Now that we have the handlers ready, we write the main class to start the HTTP server which is SessionTestServer. The program accepts the server instance id, the session separator and the server port number to bind for accepting client requests in the command-line arguments. These arguments are optional, the default value for session separator is ‘.’ and the server port number is 9080.

 

Having created the simple HTTP server, we start it using the command line

          java apachetest.SessionTestServer netty1 . 9080

 

This starts a server listening for HTTP requests at port 9080 and with server instance id ‘netty1’ and session id separator as ‘.’ We get the command-line print-

          Started server at port 9080 Press any key to stop server

 

We leave the server running and open a new browser window. We give the URL address as http://localhost:9080/sessionTest?user=Archanaa and press enter.

We get back a response on the browser window (for example) - 'Session Id is 0140C3892B4F5348.netty1'

 

Now, we copy the session id and change the URL address at the browser to http://localhost:9080/sessionTest?jsessionid=0140C3892B4F5348.netty1

where the ‘jsessionid’ request parameter value would be the session id which was assigned to us, in the last response.

 

We now get the response – ‘Welcome user Archanaa to server netty1’.

 

At the console where netty1 process has been started, press any key (like Enter). The server process will stop with the message – ‘Shutting down server’

 

Now, let us use Apache Web Server for load-balancing two such HTTP servers with session stickiness.

After installing the Apache Web Server, which would listen at port 80 on your machine, go to conf/ directory and make the following changes in httpd.conf configuration file.

  • Uncomment the modules mod_proxy, mod_proxy_balancer, mod_proxy_http and mod_rewrite.
  • Add the following VirtualHost configuration to the end of the httpd.conf file

 

<VirtualHost *:*>

    ServerAdmin archanaa.panda@gmail.com

    DocumentRoot "C:/apps/Apache Software Foundation/Apache2.2/htdocs"

    ProxyRequests Off

    ErrorLog logs/test.in-error_log

    CustomLog logs/test.in-access_log common

    <Proxy *>

          Order deny,allow

          Allow from all

    </Proxy>

 

    ProxyPreserveHost On

    ProxyPass /sessionTest/ balancer://mycluster/ stickysession=jsessionid

    ProxyPass /sessionTest balancer://mycluster stickysession=jsessionid

    <Proxy balancer://mycluster>

                Order deny,allow

                Allow from all

                BalancerMember http://localhost:9080/sessionTest route=netty1

                BalancerMember http://localhost:9081/sessionTest route=netty2

    </Proxy>

    <Location /balancer-manager>

          SetHandler balancer-manager

          Order deny,allow

          Allow from all

        </Location>

</VirtualHost>

 

 

We intend to start two servers with names netty1 and netty2 at ports 9080 and 9081, respectively. We have specified these URLs and their server instances in the BalancerMember configuration under Proxy tag. In the ProxyPass configuration, we have specified the stickysession as ‘jsessionid’, so that Apache would look for it in the HTTP request parameters or the cookies. We will pass the sessionid in the HTTP request parameter, as we had done earlier. We will choose the session id separator as ‘.’ Because that is the default separator recognized by Apache Web Server.

 

Now let us start the two server instances like so-

          java apachetest.SessionTestServer netty1 . 9080

 

          java apachetest.SessionTestServer netty2 . 9081

 

After that, let us start the Apache httpd.

 

With this, the Apache Web Server will now act as a reverse-proxy as well as load balancer with sticky-sessions for the two server instances.

To test that, let us connect with the browser URL http://localhost/sessionTest?user=Archanaa

We are provided with a user session id, say, FAABE9E8D955E7B4.netty1

 

Similarly, in a different browser window or tab, let us connect with a different user http://localhost/sessionTest?user=Ankur and we would get a new session id, say, 02C07FDA0A6DACA0.netty2

 

Now let us change the request to give the jsessionid parameter in the request instead of the user and see the results. We find that both users are correctly identified at their respective server instances. No matter how many times we connect the users with their session ids, and in whatever order, we always get the response from the correct server showing that session stickiness is working properly.

 

Now let us stop one particular server instance, say netty1 and try to connect its user. Now if we fire a request from our browser with netty1’s session, it would be failed over to the next instance, that is netty2. Since we have not replicated our session, we get the response –

          No valid session, please login with user

 

Meanwhile, the other user is able to connect as before successfully with netty2’s session.

 

Thus we see that our simple experiment has worked well.