A few days ago I faced the requirement to create an infinispan cluster composed by a JBoss node and a number of spring standalone applications.

Even though, I knew that this set up is feasible I was not able to find a working example.

I found this related article https://community.jboss.org/wiki/InfinispanClusterOnJBossASAndJavaSE very useful and I used it as a roadmap to my attempt.

I had to use the following Java frameworks:

- JBoss 7.1.2 with Spring 3.1

- Spring stanlone application with Spring 3.1 and Infinispan 5.1.4 (This version is included in JBoss 7.1.2).

This cluster set up requires different infinispan configurations in JBoss and in standalone nodes.

JBoss AS configuration - 1rst approach

My first approach was to set up the cluster cache in the JBoss standalone.xml configuration in the section with the built in infinispan configurations.

Therefore, I created a new cache-container element with the desirable infinispan configuration.
File jboss_standalone_inifinispan_config.xml shows the configuration added in the standalone.xml.

The jgroups configuration in the standalone.xml left untouched. By default, JBoss bundled jgroups uses UDP stack.

Standalone application configuration

The set up of the infinispan standalone included the extra effort to provide a correct jgroups configuration matching the one bundled with JBoss.

File infinispan-javase-config.xml shows the infinispan spring configuration in the standalone application.

File jgroups-udp.xml shows the jgroups configuration.

It is important to mention here that the default jgroups-upd.xml which is bundled in the infinispan 5.1.4 jar has a few (but important?) differences with the jgroups configuration which is used by JBoss.

In order to use exactly the same configuration elements and values, I inspected the JBoss jgroups module from the JMX console and I found all the differences between the two jgroups configurations.

Then I adjust the jgroups-udp.xml to match exactly the values exposed by the JBoss jmx console.

The attached jgroups-udp.xml contains the adjusted configuration.

It is mandatory that the placeholders in the xml be replaced by exactly the same values used in the JBoss standalone.xml. In my example, I used the following:

jgroups.udp.bind_port = 55300

jgroups.udp.diagnostics_addr = 224.1.75.75

jgroups.udp.diagnostics_port = 7500
jgroups.udp.mcast_addr = 230.1.0.4
jgroups.udp.mcast_port = 45688

Unfortunately, this approach didn't work. The infinispan cluster was formed (I was able to verify that from the JMX console) but the communication between the peers was problematic.

Unexpected exceptions were thrown when a value was inserted in the cache and the values in the cache put by one peer couldn't be read by the other.

An additional problem was that, all the cluster configurations defined in the JBoss standalone used the same jgroups ports and addresses. As a consequence, all cluster caches received the same jgroups multicast messages even if they didn't belong to the same cluster. This caused unecessry network traffic and annoying warnings in the server logs.

To be honest I didn't investigate the possibility to define different jgroups configuration in the JBoss standalone per cache container configuration. (I am not sure that this is feasible).

The files infinispan-clustering-logexceptions-jboss.xml and infinispan-clustering-logexceptions-standalone.xml show the exceptions thrown in JBoss and in standalone application respectively.

I found references for these erros in JBoss jira: https://issues.jboss.org/browse/AS7-5393, https://issues.jboss.org/browse/ISPN-2330, https://issues.jboss.org/browse/AS7-3180

but none of these issues mentioned a workaround to solve the communication problem, at least without having to migrate to a newer JBoss version (7.1.3 or 7.1.4)

JBoss AS configuration - 2rst approach

Our second approach was to define the cluster cache in the Spring configuration and deploy it as as standalone cache inside the war.

The standalone application was left untouched as in the first approach.

This attempt was successful!

The infinispan cluster was formed and it worked as expected. Additionally, by defining the cluster cache in the spring context, I was able to give a separate jgroups addresses and thus override the default used by JBoss standalone.xml and avoid the unnecessary multicast messages in the network.

In a nutshell, we took the spring configuration from the standalone spring applciation and we create exactly the same files in the JBoss application spring context.

File infinispan-jboss-spring-config.xml shows the infinispan configuration in the JBoss spring context. As you can see, it is exactly the same with the one in the standalone spring application.

The only difference is the path of the jgroups-udp.xml which should match the location of the file in the produced war tree.

Finally, two important points:

• The standalone application and the JBoss should use the same infinispan (and therefore jgroups) versions. Actually the set up might work with different infinispan versions but some of them are not compatible. In my set up, I was able to verify that version 5.1.4 (JBoss' version) was incompatible with 5.1.2 (standalone application) but tests with 5.1.4 and 5.1.8 were successful.
• All the cluster nodes should start their VM with the same IP stack. JBoss starts by default with the option java.net.preferIPv4Stack=true.
  Therefore, standalone application should start with the same parameter.

Hope to help someone with this article.