9 Replies Latest reply: Aug 21, 2013 6:11 AM by fredfrancisco RSS

Serious memory leak in AS7.1.1

Klaus Benary Newbie

Hi,

 

currently we are facing a blocking memory leak in AS 7.1.1.

 

In a test application I read 12000 EJB 2.1 EntityBeans. Hereafter,  there are about 5 million (!!) Interceptor[] arrays firmely bound in the heap.

I made a heap dump and hat a look at where they are bound. A typical reference chain is this one:

 

GC Root: ServiceContainerImpl$ServiceThread

ServiceContainerImpl$ServiceThread.container = ServiceContainerImpl

ServiceContainerImpl.registry = UnlockedReadHashMap

  UnlockedReadHashMap.table = AtomicReferenceArray

   AtomicReferenceArray.array = Object[16384]

    Object[6209] = UnlockedReadHashMap$Item[3]

     UnlockedReadHashMap$Item[1] = UnlockedReadHashMap$Item

       UnlockedReadHashMap$Item.value = ServiceRegistrationImpl

        ServiceRegistrationImpl.instance = ServiceControllerImpl

         ServiceControllerImpl.serviceValue = ImmediateValue

          ImmediateValue.value = TimedObjectInvokerImpl

            TimedObjectInvokerImpl.timeoutInterceptors = HashMap

              HashMap.table = HashMap$Entry[128]

               HashMap$Entry[123] = HashMap$Entry

                HashMap$Entry.value = ChainedInterceptor

                  ChainedInterceptor.interceptors = Arrays$ArrayList

                   Arrays$ArrayList.a = Interceptor[6]

 

It looks as nearly all of them are tied to the ServiceThread of the ServiceContainerImpl.

 

Is there any known issue covering this? We urgently need information on this since this makes EJB2.1 usage on AS7.1.1 impossible.

 

Thanks

Klaus

  • 1. Re: Serious memory leak in AS7.1.1
    jaikiran pai Master

    Looking at the code I can see a possible improvement. But is this really a leak? Do those instances stay behind after the application is undeployed? I understand 5 million interceptors shouldn't be around (and there's a way to improve that), but I want to understand the extent of this issue.

     

    P.S: Can you also check this against latest nightly build https://community.jboss.org/thread/167590

  • 2. Re: Serious memory leak in AS7.1.1
    Klaus Benary Newbie

    Yes, it is a leak. If I read another 12000 beans I have 10 millions of them and so on. Eventually the AS goes OOM if I jut keep it going long enough. And they stay in memeory even after undeploying the applcation. So, IMO it looks serious.

     

    I will try to check this against the latesr build asap and post the results here.

  • 3. Re: Serious memory leak in AS7.1.1
    Klaus Benary Newbie

    Now I managed to run the test on the latest 7.1.2 build. Unfortunately the situation has not changed. I used jmap with a preceeding full GC for counting the Interceptor arrays:

     

        190762   (after startup and deployment)

      5507902   (after reading 12000 beans)

    10790063   (after reading the same 12000 beans again)

     

    At exactly the same magnitude there are java.util.Arrays$ArrayList instances growing. Some interceptors follow behind. Here are the top scorers:

     

    num     #instances         #bytes  class name
    ----------------------------------------------
       1:      10790299      258967176  java.util.Arrays$ArrayList
       2:      10790063      244139632  [Lorg.jboss.invocation.Interceptor;
       3:       7059524      112952384  org.jboss.invocation.WeavedInterceptor
       4:       3517332       84415968  org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor
       5:       3517332       84415968  org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1
       6:        227890       64023408  [Ljava.lang.Object;
       7:       3730539       59688624  org.jboss.invocation.ChainedInterceptor

     

     

     

     

     

     

     

  • 4. Re: Serious memory leak in AS7.1.1
    Michael Nielson Newbie

    This looks strikingly similar to this (not mine, just helped me out): http://lists.jboss.org/pipermail/jboss-jira/2012-February/208418.html

     

    In my application jsp tags were holding references to my objects long after they were no longer being used.

     

    The fix is to add this to your system properties.

     

        <system-properties>

            <property name="org.apache.jasper.Constants.USE_INSTANCE_MANAGER_FOR_TAGS" value="false"/>

        </system-properties>

     

    This is just a guess since you don't mention anything jsp tag related. The jmap results just look very similar to my own symptoms.

     

    Michael

  • 5. Re: Serious memory leak in AS7.1.1
    Klaus Benary Newbie

    I am afraid this is not the point. 

    Firstly because this is no Servlet/JSP container issue, but en EJB container one. Second, the issue you mentioned was solved with AS7.1.0 Final.

     

    Thanks anyway

    Klaus

  • 6. Re: Serious memory leak in AS7.1.1
    jaikiran pai Master

    Klaus, I haven't been able to get to this yet. But please file a JIRA https://issues.jboss.org/browse/AS7 so that we don't forget about this one.

  • 7. Re: Serious memory leak in AS7.1.1
    Klaus Benary Newbie

    Done. https://issues.jboss.org/browse/AS7-4675

     

    Besides: There is another issue coming from EJB2.1 CMP/CMR which is completely blocking us: https://community.jboss.org/message/731706#731706

    There is no qualified answer to this so far. Should I file an issue for that too?

  • 8. Re: Serious memory leak in AS7.1.1
    fredfrancisco Newbie

    https://issues.jboss.org/browse/AS7-4675

     

    This is marked as resolved, and the fix is in my version 7.2.0.Final, but I am hitting this big time.

     

    I am calling a session bean (tx required) that fetches a lot of data, 45000 rows, using read-ahead. In less than a minute my old generation is exhausted and begins a cycle of Full GCs. I tried running on EAP 6.1.0, the result was the same.

     

    I did a heapdump, debugged Jboss7 code and jprofiled There is an enormous amount of interceptors being created, for every getter I call on my bean 500 are created, that hold the memory for the duration of the transaction so no space can be freed. Why are so many interceptors being created? Can anyone give me any pointers?

     

    Am am running Jboss-7.2.0.Final after migrating from Jboss-4.0.4.GA and the performance regression is significant but still acceptable. This problem, however, kills my server. My plan is to move away from ejb2 but that cannot be done at the moment.

     

    In Jboss 4, from what I understand, transation synchronizations and certain interceptors where bypassed if we called a get* method. Is this possible in Jboss7?

     

    OS: Fedora 18 (x86_64)
    Jboss: 7.2.0.Final

  • 9. Re: Serious memory leak in AS7.1.1
    fredfrancisco Newbie

    Hello jaikiran pai.

     

    Could you elaborate on the improvements you mentioned?

     

    Thanks.