Several times over the last year I have experimented with various tools for profiling AS.  One of the frustrating things with that, is that the overhead of all the traditional Java profiling solutions is so high, that its hard to get meaningful results.  With that in mind, I started to turn my attention to system profiling solutions, of which there are two for Linux.  One is called perf, and the other oprofile.

Perf, is easier to use, but it currently does not have the ability to give Java method level detailed information.  It also has some very interesting capabilities around locks that I would like to exploit, but without the Java method level information its a non-starter.  I will definitely keep an eye on perf, because if it can add the method level information, then it may end up being the more useful too.  OProfile, on the other hand, does have the capability to show method level information.  The other item of note, is that both of these tools measure CPU utililzation, and can only give the method level information for methods that have been compiled to native code via the JIT (Just-in-Time compiler).  So, methods that are run interpreted will not show up in the details, and just be lumped together with all other intrepreted methods.  So, let's take a look at how to setup OProfile, and use it to profile a running JBoss Application Server.

First, we need to install OProfile.  To do this, you can use yum, and the command is as follows:

yum install oprofile oprofile-jit

The second package (oprofile-jit) is necessary to get the Java method level symbol information.  Next, depending on the JVM that you are using you may have to install the debug information package for the JVM.  I always use OpenJDK, so I have it installed along with its debug information package.  The Sun JDK has symbol information in it, so it does not need a separate installation.

To install the debug information package for OpenJDK, you first have to enable the debug information repository.

List the available repositories:

$ ls /etc/yum.repos.d
dropbox.repo                 rpmfusion-free.repo
fedora-rawhide.repo          rpmfusion-free-updates.repo
fedora.repo                  rpmfusion-free-updates-testing.repo
fedora-updates.repo          rpmfusion-nonfree-rawhide.repo
fedora-updates-testing.repo  rpmfusion-nonfree.repo
google-chrome.repo           rpmfusion-nonfree-updates.repo
rpmfusion-free-rawhide.repo  rpmfusion-nonfree-updates-testing.repo

If you look into the content of a .repo file, it will look like the following:

$ cat /etc/yum.repos.d/fedora.repo
...

[fedora-debuginfo]
name=Fedora $releasever - $basearch - Debug
failovermethod=priority
#baseurl=http://download.fedoraproject.org/pub/fedora/linux/releases/$releasever/Everything/$basearch/debug/
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=fedora-debug-$releasever&arch=$basearch
enabled=0
metadata_expire=7d
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$basearch

...

To enable the debuginfo repository, you must replace 'enabled=0' with 'enabled=1' for every repository you enabled.  For example, I am using dropbox, fedora, fedora-updates, google-chrome, rpmfusion-free, rpmfusion-free-updates, rpmfusion-nonfree, rpmfusion-nonfree-updates, so I updated them all.  If the .repo file does not have a section for a debuginfo repository, you might have to add it by yourself to enable it.

Now, as it turns out, just adding the repository is not enough.  If you want to keep the debug information packages in sync with their respective non-debug packages you also need to install a yum plugin that will search the debug information reposiitory for corresponding updates.  Otherwise, you will get them out of sync.  You can install that plugin via the following command:

yum install yum-plugin-auto-update-debug-info

Now that yum is setup properly, you can install the OpenJDK debug information package via the following command:

yum install java-1.6.0-openjdk-debuginfo

Now you have both OProfile and the necessary symbol information for the JDK installed, and we can move on to profiling a running AS instance.

So, to setup an AS 7 server to be profiled we first have to start the JVM up with the oprofile agent.  This is done very simply by adding the following to the JVM command line:

# 64-bit JVM
-agentpath:/usr/lib64/oprofile/libjvmti_oprofile.so



# 32-bit JVM
-agentpath:/usr/lib/oprofile/libjvmti_oprofile.so

This can be found in the standalone.conf or domain.conf in the bin directory for AS 7.  The following is an example of what it looks like:

#
# Specify options to pass to the Java VM.
#
if [ "x$JAVA_OPTS" = "x" ]; then
   JAVA_OPTS="-Xms10240m -Xmx10240m -XX:+UseLargePages -XX:+UseParallelOldGC"
   JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true"
   JAVA_OPTS="$JAVA_OPTS -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000"
   JAVA_OPTS="$JAVA_OPTS -agentpath:/usr/lib64/oprofile/libjvmti_oprofile.so"
fi

That's all there is to the setup for the AS, but this by itself does not enable profiling.  There is a separate daemon process that controls the profiling that has to be started up, and controlled to turn profiling on and off.  To start the daemon, and control when to start and stop profiling, there is a control program called opcontrol.  The following commands will get things rolling for you.  By the way, you can start up the application server at any time, after getting the agent configured on the JVM command line.  Just don't start your workload that you want to profile until you are ready with the below opcontrol commands.

opcontrol --start-daemon   <-- Start the deamon, but does not start profiling.
opcontrol --start  <-- Starts profiling - you want to issue this command after you have started your workload running, or potentially just before.
opcontrol --dump  <-- dumps the profiling data out to the default file (this can be done at any time during the workload, or right after the workload completes).
opcontrol --stop  <-- Stops profiling, but leaves the daemon running.
opcontrol --shutdown  <-- Shuts the daemon down, and it will no longer be running.

Once you have captured profiling data, and have dumped it, you can generate a report.  The simpliest report, is done with the following:

opreport -l --output-file=<filename>

This will give you a list, starting with the highest percentage use of CPU cycles to the lowest.  Keep in mind that this is a system wide profiler, so it will show everything that was running on the server, not just the java process.  There are many other reporting options that can be played with, and for reference, see the OProfile documentation:

So, if you are interested in helping out with performance tuning AS 7, this is a good place to start.