6 Replies Latest reply on Oct 10, 2011 9:11 PM by vcorrea

    Memory leak on org.jboss.classloader.spi.base.BaseClassLoader

    johnnystrozzino

      Hello everybody,

       

      I've recently encountered some problems with memory leaks on JBoss, and even after having googled a lot, I still cannot manage to solve them.
      My application has several EJB3 web services and handles about 100k soap requests per day. Each request involves reading/writing Oracle db, using JPA and EJB3 over Hibernate. But after 10 days of production it runs out of memory and I must restart the application server. The application is normally deployed on a 4 nodes cluster.

       

       

       

      So I followed this strategy:
      1) Run only 1 cluster node, for 4 days
      2) GC and get an head dump every morning
      3) Compare the head dumps with Eclipse Memory Analyzer in order to understand what's growing
      I found out that there is an instance of class org.jboss.classloader.spi.base.BaseClassLoader which after 1 day is retaining 81 MB of memory, after 2 days 168 MB, after 3 days 245 MB. This object has a java.util.concurrent.ConcurrentHashMap called "blacklist" which actually is responsible for almost all the used memory (respectively 70 MB, 157 MB, 234 MB) of the container object.

       

      This blacklist is a ConcurrentHashMap<String,String> with each key equals to its value. The latest snapshot contains over 1 million of Strings.
      All Strings objects inside the map seem to appear in the following format: $obj$XXXXXXXX/E1/E2/.../En.class
      where [XXXXXXXX] is an hexadecimal number, and [E1]...[En] are names of application JPA entities, sorted in a way thet reflects the real relationship order. For instance, entity E1 has a reference to entity E2, which has a reference to entity E3, and so on.

       

      Some info about my setup:
      Max heap size: 1.2 GB
      JBoss 5.1.0-GA updated with JBoss EJB3 Plugin 1.0.19
      Quartz 1.8.3 in cluster configuration
      Hot deployment disabled
      Oracle Driver: 11g
      JDK and JRE 1.5 (latest) 32 bit
      CentOS Linux 5.3

       

      Any hint to help me figure out what's wrong in my code, or in my configuration is very welcome.

       

      In attachment there are three screenshots taken in different days.

       

      Thank you
        • 1. Re: Memory leak on org.jboss.classloader.spi.base.BaseClassLoader
          ryanhos

          This is the class blacklist.  I researched this back when we were on JBoss 4.2.3 and now I'm revisiting it for JBoss 5.1.0, since things seem to have changed.

           

          Hibernate calls class.forName() on every token in each unique query.  This is one of many reasons why it's crucially important to use bind parameters in your queries.  If you are using bind params, your query is parsed by Hibernate once, then placed in some parsed query cache.  This parsing is what calls Class.forName() on each token in the query.  If you're not using bind params, each query looks unique, and it gets reparsed each time the aruments in the WHERE clause change.  The endless stream of unique arguments in the where clause (from your 100k SOAP requests) end up as parameters to Class.forName().  JBoss's classloader caches failed class lookups, as a performance enhancement.  In JBoss 4.2.3, this was done in RepositoryClassLoader.  It was possible to set two properties which would not stop the accumulation in the class blacklist, but would stop the OutOfMemoryErrors from occurring by allowing the JVM to cull the blacklist when memory gets tight.

           

          -Dorg.jboss.mx.loading.blacklistMode=SoftSet

          -XX:SoftRefLRUPolicyMSPerMB=500

           

          However, in JBoss 5.1.0, (probably 5.x on), RepositoryClassLoader appears to be unused (zero instances in my heap dump today).  The class blacklist is now contained in BaseClassLoader, which does not have the same runaway allocation guard, as far as I can tell.

           

          You might try filing a JIRA since this seems like a regression, but Adrian Brock has already weighed-in on the topic once before, and he's of the opinion that it's the framework's problem, not JBoss'.  "There are many "stupid" webapp frameworks (and other code) around that try to continually load non-existant classes."  https://jira.jboss.org/browse/JBAS-3041

           

          In short, make sure that each and every one of your queries uses bind params, for this reason and numerous others all related to increased performance and security.

          • 2. Re: Memory leak on org.jboss.classloader.spi.base.BaseClassLoader
            johnnystrozzino

            Thank you for your kind answer.

             

            The queries are generated by jpacriteria library, another layer over hibernate.

            I found out that each object is given an alias, for example:

             

            SELECT $obj$5cdba0 FROM com.example.MyEntity  $obj$5cdba0

             

            Actually the query does work, but makes the class loader blacklist the String "$obj$5cdba0.class" because someone (hibernate?) is trying to forName it.

             

            I see these solutions, in order of dirtyness(?):

             

            1) patch jpacriteria library in order to avoid generating insane queries

            2) disable blacklisting by using jboss-classloading.xml (quick, any drawbacks?)

            3) clear blacklist every day with a quartz job, by using jmx

             

            Regarding 2) there are 104 classloader instances, but I should override the blacklist policy only for the classloader with id="vfsfile:$JBOSS_HOME/server/node1/conf/jboss-service.xml". Where should I put the jboss-classloading.xml?

             

            Any other ideas?

             

             

             

            (Someone tell Adrian Brock that jpacriteria should be added to the *blacklist* of stupid frameworks

            And btw why is the class loader using an unlimited cache for a blacklist?? Why not LRU?)

            • 3. Re: Memory leak on org.jboss.classloader.spi.base.BaseClassLoader
              felix-friedrich

              Is there any way to invoke the Java garbage collection via command line (and not from within the application itself)?

               

              Thanks...

               

              Felix

              • 4. Re: Memory leak on org.jboss.classloader.spi.base.BaseClassLoader
                wdfink

                Yes, you might use jconsole (bundled since Java6).

                Here you have a view to the running JVM and invocation of GC.

                Local you need only the PID, remote you must enable the access via JVM option.

                 

                The JConsole watch the given JVM, but if you observe too long you affect the application because some (statistic) data will be stored in the observed JVM.

                Also an explicit call of GC should only be done for test, production enviroment should run without.

                • 5. Re: Memory leak on org.jboss.classloader.spi.base.BaseClassLoader
                  vittal.rao

                  Hi ,

                   

                  When i analyzed memory leak problem through Eclipse Memory Analyzer tool. I see that most of the memory leak is caused by

                  org.jboss.classloader.spi.base.BaseClassLoader as mentioned below.

                   

                  org.jboss.classloader.spi.base.BaseClassLoader @ 0x7021143c837,6622,886,1841,020,624,66478.02%
                  <system class loader>74,9195,242,24081,034,7366.19%
                  org.jboss.bootstrap.NoAnnotationURLClassLoader @ 0x70001a22818,8731,854,12879,102,1526.05%
                  org.jboss.classloader.spi.base.BaseClassLoader @ 0x70210ed383,83877,08050,101,2803.83%
                  org.jboss.classloader.spi.base.BaseClassLoader @ 0x70001a1a047142,04047,995,6883.67%

                   

                  I wanted to know how baseClassLoader is causing a memory problem.

                   

                  Thanks in Advance.

                  Vittal

                  • 6. Re: Memory leak on org.jboss.classloader.spi.base.BaseClassLoader
                    vcorrea

                    Your answer is on JIRA:

                    https://issues.jboss.org/browse/JBAS-3041

                     

                    The baseclassloader stores the classNames of ClassNotFoundException on a blacklist hashmap and it can grow.