currently we are facing a blocking memory leak in AS 7.1.1.
In a test application I read 12000 EJB 2.1 EntityBeans. Hereafter, there are about 5 million (!!) Interceptor arrays firmely bound in the heap.
I made a heap dump and hat a look at where they are bound. A typical reference chain is this one:
GC Root: ServiceContainerImpl$ServiceThread
ServiceContainerImpl$ServiceThread.container = ServiceContainerImpl
ServiceContainerImpl.registry = UnlockedReadHashMap
UnlockedReadHashMap.table = AtomicReferenceArray
AtomicReferenceArray.array = Object
Object = UnlockedReadHashMap$Item
UnlockedReadHashMap$Item = UnlockedReadHashMap$Item
UnlockedReadHashMap$Item.value = ServiceRegistrationImpl
ServiceRegistrationImpl.instance = ServiceControllerImpl
ServiceControllerImpl.serviceValue = ImmediateValue
ImmediateValue.value = TimedObjectInvokerImpl
TimedObjectInvokerImpl.timeoutInterceptors = HashMap
HashMap.table = HashMap$Entry
HashMap$Entry = HashMap$Entry
HashMap$Entry.value = ChainedInterceptor
ChainedInterceptor.interceptors = Arrays$ArrayList
Arrays$ArrayList.a = Interceptor
It looks as nearly all of them are tied to the ServiceThread of the ServiceContainerImpl.
Is there any known issue covering this? We urgently need information on this since this makes EJB2.1 usage on AS7.1.1 impossible.
Looking at the code I can see a possible improvement. But is this really a leak? Do those instances stay behind after the application is undeployed? I understand 5 million interceptors shouldn't be around (and there's a way to improve that), but I want to understand the extent of this issue.
P.S: Can you also check this against latest nightly build https://community.jboss.org/thread/167590
Yes, it is a leak. If I read another 12000 beans I have 10 millions of them and so on. Eventually the AS goes OOM if I jut keep it going long enough. And they stay in memeory even after undeploying the applcation. So, IMO it looks serious.
I will try to check this against the latesr build asap and post the results here.
Now I managed to run the test on the latest 7.1.2 build. Unfortunately the situation has not changed. I used jmap with a preceeding full GC for counting the Interceptor arrays:
190762 (after startup and deployment)
5507902 (after reading 12000 beans)
10790063 (after reading the same 12000 beans again)
At exactly the same magnitude there are java.util.Arrays$ArrayList instances growing. Some interceptors follow behind. Here are the top scorers:
num #instances #bytes class name
1: 10790299 258967176 java.util.Arrays$ArrayList
2: 10790063 244139632 [Lorg.jboss.invocation.Interceptor;
3: 7059524 112952384 org.jboss.invocation.WeavedInterceptor
4: 3517332 84415968 org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor
5: 3517332 84415968 org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1
6: 227890 64023408 [Ljava.lang.Object;
7: 3730539 59688624 org.jboss.invocation.ChainedInterceptor
This looks strikingly similar to this (not mine, just helped me out): http://lists.jboss.org/pipermail/jboss-jira/2012-February/208418.html
In my application jsp tags were holding references to my objects long after they were no longer being used.
The fix is to add this to your system properties.
<property name="org.apache.jasper.Constants.USE_INSTANCE_MANAGER_FOR_TAGS" value="false"/>
This is just a guess since you don't mention anything jsp tag related. The jmap results just look very similar to my own symptoms.
This is marked as resolved, and the fix is in my version 7.2.0.Final, but I am hitting this big time.
I am calling a session bean (tx required) that fetches a lot of data, 45000 rows, using read-ahead. In less than a minute my old generation is exhausted and begins a cycle of Full GCs. I tried running on EAP 6.1.0, the result was the same.
I did a heapdump, debugged Jboss7 code and jprofiled There is an enormous amount of interceptors being created, for every getter I call on my bean 500 are created, that hold the memory for the duration of the transaction so no space can be freed. Why are so many interceptors being created? Can anyone give me any pointers?
Am am running Jboss-7.2.0.Final after migrating from Jboss-4.0.4.GA and the performance regression is significant but still acceptable. This problem, however, kills my server. My plan is to move away from ejb2 but that cannot be done at the moment.
In Jboss 4, from what I understand, transation synchronizations and certain interceptors where bypassed if we called a get* method. Is this possible in Jboss7?
OS: Fedora 18 (x86_64)