Monitoring Infinispan with RHQ

 

WARNING! Content out of date!

Please note that this wiki is out of date and only reflects how Infinispan used to work in versions 4.0 and 4.1. For any higher versions, please refer to the Infinispan documentation here. The most up to date instructions on monitoring Infinispan with RHQ can be found here.

Introduction

The preferred way to manage multiple Infinispan instances spread accross different servers is to use RHQ which is JBoss' enterprise management solution. Thanks to RHQ's agent and auto discovery capabilities, monitoring both Cache Manager and Cache instances is a very simple task. With RHQ, administrators have access to graphical views of key runtime parameters or statistics and can also be notified be these exceed or go below certain limits.

Getting Started

  1. Following instructions in the RHQ installation wiki, install a central RHQ server and agent in each node where Infinispan instances will be running.
  2. Download the latest Infinispan binary distribution (*-bin.zip or *-all.zip should do) from the downloads section and locate the  RHQ/JOPR plugin jar file. This is located under the modules/jopr-plugin directory.
  3. Go to the RHQ dashboard (i.e. http://localhost:7080/Dashboard.do) and select Administration->System Configuration->Plugins from the top menu.
  4. Go to the bottom of the page and click on Upload Plugin->Add and then select the jopr plugin pointed out in the previous step. Now click on Scan For Updates to verify that the Infinispan RHQ plugin was loaded.
    • If you need to update the Infinispan RHQ plugin, follow the exact same procedure.
  5. Issue a plugin update so that the agents can pick up the Infinispan RHQ plugin. You can do so via either:
    • Go to Resources->Servers, and for each "<host-name> RHQ Agent" instance, execute its Update All Plugins operation.
    • Or, execute "plugins update" at each agent's command prompt.
    • Or, restart each agent.

 

At this point, RHQ is now ready to start monitoring Infinispan instances. See the next section to find out more.

Monitoring Infinispan Instances

Currently, the RHQ agent that runs the Infinispan RHQ/JOPR plugin can only automatically discover a single Infinispan instance running in the same machine as the agent and this Infinispan instance needs to have port 6996 open for remote JMX monitoring and management. So, Infinispan instances need to start with the following system properties:

 

-Dcom.sun.management.jmxremote.port=6996 -Dcom.sun.management.jmxremote.ssl=false 
-Dcom.sun.management.jmxremote.authenticate=false

 

You can optionally have multiple Infinispan instances running on the same machine but these will need to use a different port and they'll have to be imported into RHQ server manually. To add Infinispan instances manually, go to Resources > Platforms > localhost and then click on Inventory. At the bottom of the page, there's a section that says "Manually Add" and next to it there's a drop down list. Open this list, select "Infinispan Cache Manager" and click Ok. In the next page, select "default" template and this will lead to a page where you need to enter both the JMX connector address of the new Infinispan instance you're trying to monitor and the Cache Manager MBean object name. For example:

 

Connector Address:

service:jmx:rmi://127.0.0.1/jndi/rmi://127.0.0.1:7997/jmxrmi

 

Object Name (Infinispan 4.1 or earlier):

 infinispan:cache-name=[global],jmx-resource=CacheManager

 

Object Name (Infinispan 4.2 or later) where cache manager name is substituted within quotes:

org.infinispan:type=CacheManager,name="<name_of_cache_manager>"

 

The object name is always the same but the connector address varies depending on the host and particularly the JMX port where the new instance is listening. In this case, the instance would have been started with:

 

-Dcom.sun.management.jmxremote.port=7997 -Dcom.sun.management.jmxremote.ssl=false 
-Dcom.sun.management.jmxremote.authenticate=false

 

A detailed demo showing how to monitor Infinispan instances will be coming very soon.

Tips And Tricks

Finally, this section focuses on the lessons learned while developing the Infinispan RHQ plugin that are likely to be useful to anyone using RHQ.

 

By default, at least in version 2.3.1 of RHQ, the RHQ agent sends an availability report of any managed resources every 5 minutes. The problem with this is that if you're testing whether your Infinispan instance is automatically discovered by the RHQ server, it can take up to 5 minutes to do so! Also, it can take 5 minutes for the RHQ server to figure out that you've shutdown your Infinispan instance.

 

You can change this setting by the following property (default value is 300 seconds) in rhq-agent/conf/agent-configuration.xml. For example, if you wanted the availability to be sent every 1 minute, simply change the value to 60:

 

<entry key="rhq.agent.plugins.availability-scan.period-secs" value="60"/>

 

Please bear in mind the instructions given in the Configuring JOPR agent documentration and more specifically the paragraph below with regards to changes made to properties in agent-configuration.xml:

Once the agent is configured, it persists its configuration in the
Java Preferences backing store. Once this happens,
agent-configuration.xml is no longer needed or used. Editing
agent-configuration.xml will no longer have any effect on the agent,
even if you restart the agent. If you want the agent to pick up changes
you make to agent-configuration.xml, you must either restart the agent
with the "--cleanconfig" command line option or use the "config
--import" agent prompt command.

Screencasts

You find an screencast about monitoring Infinispan with RHQ console in Infinispan's youtube channel. This video gives a good overall view of what you can do with RHQ to monitor Infinispan.

Flash Movies

A series of flash movies has also been recorded that contain more detailed information on the topics briefly shown by the screencast. Here are the links to these flash movies and a short explanation of what they contain: