7 Replies Latest reply: Jun 14, 2013 11:08 AM by Wolf-Dieter Fink Branched to a new discussion. RSS

Tutorial domain clustering in JBoss 7 / EAP 6

Willem Noorduin Newbie

A while ago I wrote https://community.jboss.org/thread/162859, which described how to setup clusters, mod_cluster, Apache 2.2+ and JBoss 5.1. The same works more-or-less when you want to do such a thing with JBoss 7 standalone nodes, so I think you can figure this one out for yourself. Clustering goes better (in terms that it is much easier to maintain) when you install a domain controller and a couple of slaves which listen to that domain controller.


In this document (typos excluded), I give a howto of such a setup. Maybe it is useful, bits and pieces of this stuff are "in the cloud called the internet" of course.



CHAPTER 1: General architecture



1.1.    Setup of JBoss and purpose of the document


In this chapter we work with three different kind of servers, of which the IP-addresses and the

canonical names are the following:     webserver               ws (Apache 2.2.x)    domain controller       dc (JBoss 7 / EAP6)    applicationserver       slave1 of dc (JBoss 7 / EAP6)    applicationserver       slave2 of dc (JBoss 7 / EAP6)



1.2.    JBoss Clusters, Instances and applications:


In this document we configure two apache virtual hosts (www.application1.nl and www.application2.nl).

Each virtual host shall be connected via mod_cluster with it's own jboss cluster. Clusters are:


cluster1 consists of {node11@as1 and node12@as2)

cluster2 consists of {node21@as1 and node22@as2}


application1.war is deployed to cluster1 and reacheable via http://www.application1.nl/application1

application2.war is deployed to cluster2 and reacheable via http://www.application2.nl/application2


In this situation we have two Apache Virtual hosts which sent requests to two different JBoss-AS clusters, each JBoss-AS cluster consists two instances, each one of them runs on a different (physical) Linux Server (or whatever).


In the DNS, both URLs www.application1.nl and www.application2.nl resolve to the webserver, and we are using mod_cluster in combination with ajp (Apache JServ Protocol) to pass application calls to the JBoss-AS cluster.


This document describes the configuration of the domain controller and application servers, and does not concentrate on the mod_cluster aspect so much. When you have a clustername to which you can send your application request, things are very much transparent (but see the CHAPTER 8 in https://community.jboss.org/thread/162859). The goal is to describe the cluster seperation completely from the standpoint of the domain controller / slave relationship. This means that, with exeption of creating a new profile, which has to do by "hand", much of the stuff we do with the JBoss client (jboss-cli) on the domain crontroller.



1.3.    Brief contents of this document


CHAPTER 2: Installation and configurating the domain crontroller


CHAPTER 3: Installation of the slaves


CHAPTER 4: Creating a new profile


CHAPTER 5: Configuring the architecture for running a cluster


CHAPTER 6: Deploying the applications


CHAPTER 7: What we have not discussed



CHAPTER 2: The domain crontroller



2.1.    A brief history of the installation of JBoss 7


After downloading JBoss 7 (mine is called jboss-as-7.1.1.Final.zip) and an Oracle JDK (jdk1.6.0_30.tar.gz) do the following:


root@dc # cd /opt

root@dc # cp /tmp/jdk1.6.0_30.tar.gz /tmp/jboss-as-7.1.1.Final.zip .

root@dc # tar xfz jdk1.6.0_30.tar.gz ; unzip jboss-as-7.1.1.Final.zip

root@dc # ln -s jdk1.6.0_30.tar.gz jdk ; ln -s jboss-as-7.1.1.Final jboss

root@dc # chown -R jboss:jboss jboss-as-7.1.1.Final


2.2.    Installing the domain-controller


The easiest way to start the domain controller is just fire up /opt/jboss/bin/domain.sh -b <bind-address> after setting JAVA_HOME to the correct value. Note that this is also the clumsiest way, since it starts in the foreground. Now there are plenty of ways to configure it to start in the background and barf into a console log instead of a console-window, but we do not want to go into that here. Now start:


root@dc # /opt/jboss/bin/domain.sh  --domain-config domain.xml --host-config=host-master.xml -P file:///apps/jboss/nlptc27b09/configs/domain.properties




root@ cat /apps/jboss/












Which is just a way of how to administer things more neatly. Anyway, the above processes should run.


You should end with something like this:


root@ ps -ef | grep jboss | grep -v grep

jboss     8339     1  0 08:40 ?        00:00:00 /bin/sh /opt/jboss/bin/domain.sh --domain-config domain.xml --host-config=host.xml -P file:///apps/jboss/

jboss     8382  8339  0 08:40 ?        00:00:03 /opt/jdk/bin/java -D[Process Controller] -server -Dorg.jboss.boot.log.file=/var/log/jboss/ -Dlogging.configuration=file:/apps/jboss/profiles/ -jar /opt/jboss/jboss-modules.jar -mp /opt/jboss/modules org.jboss.as.process-controller -jboss-home /opt/jboss -jvm /opt/jdk/bin/java -mp /opt/jboss/modules -- -Dorg.jboss.boot.log.file=/var/log/jboss/ -Dlogging.configuration=file:/apps/jboss/profiles/ -server -- -default-jvm /opt/jdk/bin/java --domain-config domain.xml --host-config=host.xml -P file:///apps/jboss/

jboss     8396  8382  0 08:40 ?        00:00:11 /opt/jdk/bin/java -D[Host Controller] -Dorg.jboss.boot.log.file=/var/log/jboss/ -Dlogging.configuration=file:/apps/jboss/profiles/ -server -jar /opt/jboss/jboss-modules.jar -mp /opt/jboss/modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.host-controller -mp /opt/jboss/modules --pc-address --pc-port 46303 -default-jvm /opt/jdk/bin/java --domain-config domain.xml --host-config=host.xml -P file:///apps/jboss/ -Djboss.home.dir=/opt/jboss



2.3.    Adding JBoss users


2.3.1    Adding the admin user


The user admin is used to login to the administration website:


root@dc : /opt/jboss/bin # ./add-user.sh


What type of user do you wish to add?

a) Management User (mgmt-users.properties)

b) Application User (application-users.properties)



Enter the details of the new user to add.

Realm (ManagementRealm) :

Username : admin

Password : <welkom>

Re-enter Password : <welkom>

Updated user 'admin' to file '/opt/jboss-eap-6.0.0/standalone/configuration/mgmt-users.properties'

Updated user 'admin' to file '/opt/jboss-eap-6.0.0/domain/configuration/mgmt-users.properties'

Is this new user going to be used for one AS process to connect to another AS process e.g. slave domain controller?

yes/no? no


You need this user when you want to administer the domain controller with jboss-cli.sh from somewhere else.



2.3.2    Adding the slave user


The user slaveuser is used by external node to login to the Domain Controller:


root@dc : /opt/jboss/bin # ./add-user.sh


What type of user do you wish to add?

a) Management User (mgmt-users.properties)

b) Application User (application-users.properties)



Enter the details of the new user to add.

Realm (ManagementRealm) :

Username : slaveuser

Password : <welkom>

Re-enter Password : <welkom>

About to add user 'slaveuser' for realm 'ManagementRealm'

Is this correct yes/no? yes

Added user 'slaveuser' to file '/opt/jboss/standalone/configuration/mgmt-users.properties'

Added user 'slaveuser' to file '/opt/jboss/domain/configuration/mgmt-users.properties'

Is this new user going to be used for one AS process to connect to another AS process e.g. slave domain controller?

yes/no? yes


To represent the user add the following to the server-identities definition <secret value="d2Vsa29t" />




CHAPTER 3: Installation of the slaves



Come up with a unique name (here slave1) for a slave and repeat this chapter for as much slaves as you want to have in your architecture. Note that slaves can be anywhere (that is, on production servers, somewhere else on the internet, and even on your laptop). The slavenames have to be unique (you can use for instance a DNS-like name for bigger architectures).


3.1.    A brief history of the installation of JBoss 7 for slaves.


This is easy, just follow 2.1


3.2.    Installing a slave server of the dc


Since this is the same thing over and over again, we tell you how to do it on the applicationserver slave1 (


3.2.1.    Adjusting host-slave.xml


<?xml version='1.0' encoding='UTF-8'?>


<host name="slave1" xmlns="urn:jboss:domain:1.3">




            <security-realm name="ManagementRealm">


                    <secret value="${jboss.domain.login.password}"/>



                    <local default-user="$local"/>

                    <properties path="mgmt-users.properties" relative-to="jboss.domain.config.dir"/>






There is some story here to tell, when you define it like this, you must offer a jboss.domain.login.password property during startup. Upshot is, this must be a cleartext property. If you are paranoid, you can work with:



            <security-realm name="ManagementRealm">


                    <secret value="d2Vsa29t"/>



                    <local default-user="$local"/>

                    <properties path="mgmt-users.properties" relative-to="jboss.domain.config.dir"/>





The secret value is the one which was given to you in 2.3.2. Now to make this really a slave of the dc, adjust as follows:




        <remote host="${jboss.domain.master.address}" port="${jboss.domain.master.port:9999}" security-realm="ManagementRealm" username="${jboss.domain.login.user}"/>




We use variables here, because we use a property file to feed our jboss process. If you don't want that, you can use the literal versions of that:



3.2.2.    Starting the slave.


The easiest way to start the domain controller is just fire up /opt/jboss/bin/domain.sh -b <bind-address> after setting JAVA_HOME to the correct value. Note that this is also the clumsiest way, since it starts in the foreground. Now there are plenty of ways to configure it to start in the background and barf into a console log instead of a console-window. Now start:


root@slave1 # /opt/jboss/bin/domain.sh  --domain-config domain.xml --host-config=host-slave.xml -P file:///apps/jboss/slave1/configs/domain.properties




root@slave1:/opt# cat /apps/jboss/alave1/configs/domain.properties











We end up with


root@slave1:/opt# ps -ef | grep jboss | grep -v grep

jboss     6670     1  0 11:44 ?        00:00:00 /bin/sh /opt/jboss/bin/domain.sh --domain-config domain.xml --host-config=host-slave.xml -P file:///apps/jboss/fbewslave/configs/domain.properties

jboss     6713  6670  0 11:44 ?        00:00:00 /opt/jdk/bin/java -D[Process Controller] -server -Dorg.jboss.boot.log.file=/var/log/jboss/fbewslave/process-controller.log -Dlogging.configuration=file:/apps/jboss/profiles/fbewslave/configuration/logging.properties -jar /opt/jboss/jboss-modules.jar -mp /opt/jboss/modules org.jboss.as.process-controller -jboss-home /opt/jboss -jvm /opt/jdk/bin/java -mp /opt/jboss/modules -- -Dorg.jboss.boot.log.file=/var/log/jboss/fbewslave/host-controller.log -Dlogging.configuration=file:/apps/jboss/profiles/fbewslave/configuration/logging.properties -server -- -default-jvm /opt/jdk/bin/java --domain-config domain.xml --host-config=host-slave.xml -P file:///apps/jboss/fbewslave/configs/domain.properties

jboss     6728  6713  3 11:44 ?        00:00:04 /opt/jdk/bin/java -D[Host Controller] -Dorg.jboss.boot.log.file=/var/log/jboss/fbewslave/host-controller.log -Dlogging.configuration=file:/apps/jboss/profiles/fbewslave/configuration/logging.properties -server -jar /opt/jboss/jboss-modules.jar -mp /opt/jboss/modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.host-controller -mp /opt/jboss/modules --pc-address --pc-port 37932 -default-jvm /opt/jdk/bin/java --domain-config domain.xml --host-config=host-slave.xml -P file:///apps/jboss/fbewslave/configs/domain.properties -Djboss.home.dir=/opt/jboss



3.2.3.    Controlling if the slave is connected to the domain controller


The slave is connected to the domain controller when you can see slave1 as a host in the jboss-cli as follows:


root@dc:/opt/jboss/bin# ./jboss-cli.sh --controller= --connect

[domain@ /] ls -l /host






CHAPTER 4: Creating a new profile



4.1.    Add a profile definition to the domain.xml


Since it is not (yet) possible to copy a profile with the JBoss client, we have to copy a profile by hand:


1.    Open /opt/jboss/domain/configuration/domain.xml on the domain crontroller.

2.    When you want a ha-typed profile, copy all the entries between:


<profile name="ha">




    to a new entry between the <profiles> ... </profiles> tags, for example:









The following properties of our architecture must be set on the profile level:


* The balancer-name, this is essentially what you need to put in the Apache Virtual Host in the <Proxy> ... </Proxy> tags.

* The jdbc-driver entries (that is, when you need one of-course, but an application without its data is like ...).

* The datasource entries.

* Access logging of the web system.



4.2.    Control if the profile is there


After hacking (eh adjusting) the domain.xml of the domain controller, restart the domain crontroller and check whether the profile is there:


[domain@ /] ls -l /profile








We see that we have repeat 4.1 with a cluster-name cluster2, since we are going to serve two (seperated clusters).



CHAPTER 5: Configuring the architecture for running a cluster



5.1.    A script to configure to configure the profile and the server-group


Remember: allthough it is not a 1-1 map, you can see the combination profile + server-group as a simple definition of what we had in JBoss 5 for a cluster.


Safe the following to a text file (say /tmp/cluster1.cli).




# Setting the balancer name in the profile



/profile=cluster1/subsystem=modcluster/mod-cluster-config=configuration:write-attribute(name=balancer, value="cluster1")



# Creating the server group



/server-group=cluster1-server-group:add(profile=cluster1, socket-binding-group=ha-sockets, socket-binding-port-offset=0)



# Creating JDBC Driver entries and datasource entry for cluster1

# Datasource definition shown for Oracle






data-source --name=cluster1_DS \

--connection-url="jdbc:oracle:thin:@databaseserver:port:database" \

--jndi-name="java:jboss/cluster1_DS" \

--driver-name="oracle" \

--user-name="cluster1_appl" --password="password" \

--use-java-context="true" \

--min-pool-size="0" \

--max-pool-size="64" \

--idle-timeout-minutes="30" \

--background-validation="false" \

--background-validation-millis="1" \

--validate-on-match="true" \

--allocation-retry="0" \

--share-prepared-statements="false" \

--set-tx-query-timeout="false" \

--query-timeout="0" \

--use-try-lock="0" \

--url-delimiter="|" \

--profile=cluster1 add



# Creating application properties entry (if needed of course)






# Creating access log entry



/profile=cluster1/subsystem=web/virtual-server=default-host/access-log=configuration:add(extended=false, pattern="%a %t %H %p %U %s ", prefix=false, resolve-hosts=false, rotate=true



Of course there could be more or less entries in this file (just put there what you want), after this:


root@dc # cd /opt/jboss/bin

root@dc # /jboss-cli.sh --controller= --connect --file=/tmp/cluster1.cli



5.2.    Creating and configuration server-configs


A server-group contains serveral server-configs, and the last ones are synonym with what we had as clusternodes in the good ol' dagys. These server-configs can be running everywhere where there is a JBoss slave is installed.  If you have not done so, follow 3.2.3 and check if the slave is there. Then (again in the jboss-cli):


[domain@ /] /host=slave1/server-config=node11:add(auto-start=true, group=cluster1-server-group, socket-binding-group=ha-sockets, socket-binding-port-offset=0)


    "outcome" => "success",

    "result" => undefined



5.3.    Starting the nodes and where are my files


You can start, stop or restart the server-configs as follows:


[domain@ /] /host=slave1/server-config=node11:start


    "outcome" => "success",

    "result" => "STARTING"



On slave1 (we can now see something like this:


root@slave1:/home/wnoordui# tree /opt/jboss/domain/servers/node11/


|-- data

|-- log

|   |-- accesslog

|   |   `-- access_log.2013-02-13

|   |-- boot.log

|   |-- server.log

|   `-- server.log.2013-02-11

`-- tmp



5.4.    On enabling datasources and testing them


Enable a datasource and test if it's there:


[domain@ /] /profile=cluster1/subsystem=datasources/data-source=cluster1_DS:enable


    "outcome" => "success",

    "result" => undefined,

    "server-groups" => {"cluster1-server-group" => {"host" => {"slave1" => {"node11" => {"response" => {

        "outcome" => "success",

        "result" => undefined




[domain@ /] /host=slave1/server=node11/subsystem=datasources/data-source=cluster1_DS:test-connection-in-pool


    "outcome" => "success",

    "result" => [true]




CHAPTER 6: Deploying and testing the applications



6.1.    Deploy an application


This is simple, suppose we have a /tmp/testapp.war which contains the application code. Then:


[domain@ /] deploy  /tmp/testapp.war --server-groups=cluster1-server-group


and follow the logfiles on one of the nodes of cluster1-server-group (node11 would be a great choice)/



6.2.    Test the application


For me: lynx http://slave1:8080/testapp works fine, but if you are gui-impaired you can you iceweazel, firefox or (even) ie.



CHAPTER 7: What we have not discussed



In this document only the simplest way of getting clusters or clusterlike architectures running on a domain cluster. I haven't delved into things like hibernation, messaging, etc. Also you can read somewhere else (for example https://community.jboss.org/thread/162859 in CHAPTER 8 there), how to setup an Apache / mod cluster for this thing

  • 1. Re: Tutorial domain clustering in JBoss 7 / EAP 6
    Radoslav Husar Master

    Neat. Lets make this into an article rather than forum post -- okay? I can do the conversion.

  • 2. Re: Tutorial domain clustering in JBoss 7 / EAP 6
    Wolf-Dieter Fink Master

    Remember that a domain is a total different concept. So please stay clear with that to not confuse somebody.


    • A domain is to manage/controll/configure several server instances across several physical boxes.
    • A cluster is a bunch of server-instances which share application requests (load balancing, failover).


    So a domain setup may contain non-cluster, one-cluster or many clusters. That depends on the used profile and the settings for JGroups communication.

    If you use the same JGroups multicast-address (default) the different servers will find each other and build a cluster.

  • 3. Re: Tutorial domain clustering in JBoss 7 / EAP 6
    kzivkovic Newbie

    In jboss-as-7.1.1.Final.zip my



    Starts with:

    <host xmlns="urn:jboss:domain:1.2">

    and not with

    <host xmlns="urn:jboss:domain:1.3"> that is case in example you gave.


    Therefore I get:


    [Host Controller] 11:14:35,202 ERROR [org.jboss.as.host.controller] (Controller Boot Thread) JBAS010932: Caught exception during boot: org.jboss.as.controller.persistence.Configura

    tionPersistenceException: JBAS014676: Failed to parse configuration

    [Host Controller]       at org.jboss.as.controller.persistence.XmlConfigurationPersister.load(XmlConfigurationPersister.java:141) [jboss-as-controller-7.1.1.Final.jar:7.1.1.Final]

    [Host Controller]       at org.jboss.as.host.controller.HostControllerConfigurationPersister.load(HostControllerConfigurationPersister.java:158) [jboss-as-host-controller-7.1.1.Fin


    [Host Controller]       at org.jboss.as.host.controller.DomainModelControllerService.boot(DomainModelControllerService.java:315) [jboss-as-host-controller-7.1.1.Final.jar:7.1.1.Fin


    [Host Controller]       at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:155) [jboss-as-controller-7.1.1.Final.jar:7.1.1.Final]

    [Host Controller]       at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_23]

    [Host Controller] Caused by: javax.xml.stream.XMLStreamException: ParseError at [row,col]:[31,8]

    [Host Controller] Message: JBAS014788: Unexpected attribute 'username' encountered

    [Host Controller]       at org.jboss.as.controller.parsing.ParseUtils.unexpectedAttribute(ParseUtils.java:104) [jboss-as-controller-7.1.1.Final.jar:7.1.1.Final]

    [Host Controller]       at org.jboss.as.host.controller.parsing.HostXml.parseRemoteDomainControllerAttributes(HostXml.java:832) [jboss-as-host-controller-7.1.1.Final.jar:7.1.1.Fina


    [Host Controller]       at org.jboss.as.host.controller.parsing.HostXml.parseRemoteDomainController1_1(HostXml.java:778) [jboss-as-host-controller-7.1.1.Final.jar:7.1.1.Final]

    [Host Controller]       at org.jboss.as.host.controller.parsing.HostXml.parseDomainController(HostXml.java:745) [jboss-as-host-controller-7.1.1.Final.jar:7.1.1.Final]

    [Host Controller]       at org.jboss.as.host.controller.parsing.HostXml.readHostElement_1_1(HostXml.java:392) [jboss-as-host-controller-7.1.1.Final.jar:7.1.1.Final]

    [Host Controller]       at org.jboss.as.host.controller.parsing.HostXml.readElement(HostXml.java:126) [jboss-as-host-controller-7.1.1.Final.jar:7.1.1.Final]

    [Host Controller]       at org.jboss.as.host.controller.parsing.HostXml.readElement(HostXml.java:103) [jboss-as-host-controller-7.1.1.Final.jar:7.1.1.Final]

    [Host Controller]       at org.jboss.staxmapper.XMLMapperImpl.processNested(XMLMapperImpl.java:110) [staxmapper-1.1.0.Final.jar:1.1.0.Final]

    [Host Controller]       at org.jboss.staxmapper.XMLMapperImpl.parseDocument(XMLMapperImpl.java:69) [staxmapper-1.1.0.Final.jar:1.1.0.Final]

    [Host Controller]       at org.jboss.as.controller.persistence.XmlConfigurationPersister.load(XmlConfigurationPersister.java:133) [jboss-as-controller-7.1.1.Final.jar:7.1.1.Final]

    [Host Controller]       ... 4 more

    [Host Controller]

    [Host Controller] 11:14:35,294 FATAL [org.jboss.as.host.controller] (Controller Boot Thread) JBAS010933: Host Controller boot has failed in an unrecoverable manner; exiting. See pr

    evious messages for details.

    [Host Controller] 11:14:35,331 INFO  [org.jboss.as] (MSC service thread 1-2) JBAS015950: JBoss AS 7.1.1.Final "Brontes" stopped in 6ms

    11:14:35,673 INFO  [org.jboss.as.process.Host Controller.status] (reaper for Host Controller) JBAS012010: Process 'Host Controller' finished with an exit status of 99

    11:14:35,683 INFO  [org.jboss.as.process] (Thread-8) JBAS012016: Shutting down process controller

    11:14:35,688 INFO  [org.jboss.as.process] (Thread-8) JBAS012015: All processes finished; exiting


    Help is appreciated...

  • 4. Re: Tutorial domain clustering in JBoss 7 / EAP 6
    Wolf-Dieter Fink Master


    welcome to the forum.


    You can not use a configuration from a newer version (this is the URN:domain x.x). The second issue is because you change the version but in this case the XML is not according to the schema.

    If you need another example try this ejb-multi-server quickstart.

  • 5. Re: Tutorial domain clustering in JBoss 7 / EAP 6
    kzivkovic Newbie

    Hello Dieter,


    I am completely aware that "you cannot use configuration from a newer version...".

    I just claim that with downloading jboss-as-7.1.1.Final.zip I didn't get host-slave.xml that contained:

    <host xmlns="urn:jboss:domain:1.3" line.


    It contained

    <host xmlns="urn:jboss:domain:1.2"


    I just followed the instructions in article you gave and wanted to warn you about this discrepancy.


    I also tried to run ejb-multi-server quickstart, but I got:

    #95 /server-group=quickstart-ejb-multi-appWeb-server:add(profile=default-web,socket-binding-group=standard-sockets-web)

    #96 /server-group=quickstart-ejb-multi-appWeb-server/jvm=default:add()

    #97 /host=master/server-config=app-web:add(auto-start=true, group=quickstart-ejb-multi-appWeb-server, socket-binding-port-offset=300)

    Failed to execute batch: JBAS010850: No handler for operation add at address [

        ("profile" => "default-web"),

        ("subsystem" => "naming"),

        ("service" => "remote-naming")



    But maybe this is off topic... thanks for reply.

  • 6. Re: Tutorial domain clustering in JBoss 7 / EAP 6
    Vivek Mittal Newbie


    I am also facing the same issue. How did you resolve it ?

  • 7. Re: Tutorial domain clustering in JBoss 7 / EAP 6
    Wolf-Dieter Fink Master

    You should use the latest version from github and read the README "Modify the CLI Scripts (if you are running JBoss Enterprise Application Platform 6.0 or JBoss AS7.1 servers"

    This is because the scripts are provided for use with the latest EAP and AS7.2 version and have issues if you use former versions.