We are running on JBoss 5.1 / JDK 1.6 on Linux servers. During very high load we are seeing Out of Memory Error and it seems like related to the JBoss remoting.
We took a heap dump and the Memory Analyzer shows the below instance is taking 25% of the memory:
One instance of "org.jboss.mx.server.MBeanServerImpl" loaded by "org.jboss.classloader.spi.base.BaseClassLoader @ 0x415b2fc8" occupies 65,055,032 (24.64%) bytes. The memory is accumulated in one instance of "EDU.oswego.cs.dl.util.concurrent.ConcurrentReaderHashMap$Entry" loaded by "org.jboss.bootstrap.NoAnnotationURLClassLoader @ 0x42f8ba78".
The details which is pointing ot the JBoss remoting classes:
Class Name Shallow Heap Retained Heap
- EDU.oswego.cs.dl.util.concurrent.ConcurrentReaderHashMap$Entry Shallow Heap => 8,208 Retained Heap => 45,693,888
EDU.oswego.cs.dl.util.concurrent.ConcurrentReaderHashMap Shallow Heap => 56 Retained Heap => 45,761,640
EDU.oswego.cs.dl.util.concurrent.ConcurrentReaderHashMap$Entry Shallow Heap => 24 Retained Heap => 45,761,688
EDU.oswego.cs.dl.util.concurrent.ConcurrentReaderHashMap$Entry Shallow Heap => 272 Retained Heap => 64,584,776
EDU.oswego.cs.dl.util.concurrent.ConcurrentReaderHashMap Shallow Heap => 56 Retained Heap => 64,621,408
org.jboss.mx.server.registry.BasicMBeanRegistry Shallow Heap => 48 Retained Heap => 64,621,680
org.jboss.mx.server.MBeanServerImpl Shallow Heap => 24 Retained Heap => 65,055,032
java.lang.Thread @ 0x42135bb8 Signal Dispatcher Native Stack, Thread
Any thoughts on this please?
Sorry for the delay. Are you still having this problem?
I'm not really sure about what I'm looking at here; in particular, what is pointing at what? A Remoting server will try to find an MBeanServer. Is this dump saying that the MBeanServerImpl points to the BasicMBeanRegistry which points to the ConcurrentReaderHashMap? In any case, it doesn't seem as if Remoting has a direct pointer to the ConcurrentReaderHashMap, right?
Also, I'm not sure what to make about the presence of the PooledInvoker in the dump. That's not part of Remoting, and I don't know how it would be related to a Remoting server.
Can you add any information that would help me make sense of the dump?