way too many SLSB instances eventually cause heap to run out
ips Feb 8, 2011 11:00 AMIn the RHQ 3.0 server, we use JBossAS 4.2.3's bundled EJB3. A user has reported seeing way too many SLSB instances slowly building up in heap and eventually causing the server to run out of heap (see his post from the rhq-devel list below). Can anyone shed some light on what could be causing it? Is it a bug in the EJB container, a bug in our application code, or just some configuration setting we need to adjust?
Thanks,
Ian
-----
Following up a post from about a month ago. We were seeing a persistent slow memory leak in the rhq server in tenured gen space that eventually led to an out of memory exception after running the server for about a week. I captured a heap dump and found hundreds of thousands of stateless session beans in memory. Here's a snapshot from my profiler of a table of classes with greatest number of instances.
Name Objects Shallow Size Retained Size java.util.HashMap$Entry 1939755 93108240 189082696 java.util.HashMap$Entry[] 1090957 167796768 340273520 java.util.HashMap 1084265 69392960 408521632 java.util.LinkedList$Entry 860965 34438600 727956072 org.jboss.ejb3.BaseSessionContext 856281 34251240 34251240 org.rhq.enterprise.server.authz.RequiredPermissionsInterceptor 856281 13700496 13700496 org.rhq.enterprise.server.common.TransactionInterruptInterceptor 856281 13700496 13700496 org.jboss.ejb3.stateless.StatelessBeanContext 856265 68501200 490959040 java.lang.String 429025 17161000 48902064 char[] 379454 37897872 37897872 java.lang.Integer 171633 4119192 4119192 java.util.Hashtable$Entry 157623 7565904 34980432 java.util.TreeMap$Entry 105496 6751744 14950816 java.lang.String[] 98401 4340480 6555536 org.rhq.enterprise.server.auth.SubjectManagerBean 91116 6560352 49567104 org.rhq.enterprise.server.auth.TemporarySessionPasswordGenerator 91116 3644640 43006752 org.rhq.enterprise.server.authz.AuthorizationManagerBean 91115 2186760 2186760 org.rhq.enterprise.server.alert.AlertConditionManagerBean 91084 2914688 2914688 org.rhq.enterprise.server.alert.AlertManagerBean 90914 9455056 9455056 org.rhq.enterprise.server.alert.AlertDefinitionManagerBean 90911 4363728 4363728 org.rhq.enterprise.server.alert.AlertConditionLogManagerBean 90903 5090568 5090568 org.rhq.enterprise.server.alert.CachedConditionManagerBean 90903 4363344 4363344 org.rhq.enterprise.server.alert.AlertDampeningManagerBean 90903 3636120 3636120 org.jboss.security.SecurityAssociation$SubjectContext 49229 2362992 2362992 org.rhq.enterprise.server.cloud.instance.ServerManagerBean 39354 3463152 3463152 org.rhq.enterprise.server.cloud.CloudManagerBean 39354 2833488 2833488
Here are the merged paths from the SubjectManagerBean to GCRoot:
<All the objects> org.jboss.ejb3.stateless.StatelessBeanContext java.util.LinkedList$Entry java.util.LinkedList$Entry java.util.LinkedList org.jboss.ejb3.InfinitePool org.jboss.ejb3.ThreadlocalPool org.jboss.ejb3.stateless.StatelessContainer
All the other manager beans have similar merged paths. So I started to wonder why there were so many slsb's in the ThreadlocalPools and after some digging found this (http://community.jboss.org/message/363520) thread that sort of describes what I'm seeing. I still don't know why it's happening but it gave me something to try. I changed the Stateless Bean pool class in ejb3-interceptors-aop.xml from ThreadlocalPool to StrictMaxPool. Now when I run the server and watch it with my profiler I see at max 3 SubjectManagerBeans in memory. Same appears to be true for other slsb's. This isn't a solution to the problem but I'm hoping someone can shed light on what's really going on. I would be happy to upload the heap dump to somewhere public but it's almost a GB in size.
Bala Nair
SeaChange International