1 2 Previous Next 22 Replies Latest reply on Sep 20, 2012 3:21 AM by tomjenkinson

    JBoss AS 7.1.1 Final XA datasources memory leak

    karl_zhang

      Dear All,

      recently I also hit the memory leak problem described in AS7-4975 and JBTM-1183. A workaround is mentioned in these two tickets. But the workaround is seem to be related to the configuration of JBoss 7, the version of JBosss module ironjacamar and the version postgres jdbc driver. So I could not find an exact solution for this issue. Can anyone tell me, what exactly I have to do to work around this problem?

      1) how dose the configuration of JBoss 7 look like? Do I just need to add the recover-plugin class definition?

      2) do I need a nightly build version of JBoss7 for the recover-plugin xml element? Or it is possible, just exchange the module ironjacamar in JBoss 7.1.1 with a newer version? Because if I switched to ironjacamar-1.1.0.Beta1, JBoss gave an error during booting: "java.lang.NoClassDefFoundError: org/jboss/jca/common/api/metadata/ds/DataSource$Tag".

      3) which version of postgres jdbc driver do I need? Do I have to wait for the release of 9.2devel?

       

      Best Regards,

       

      Karl Gross

        • 1. Re: JBoss AS 7.1.1 Final XA datasources memory leak
          jesper.pedersen

          1) Yes, add the recover-plugin element, and make sure that you have a connection checker defined

          2) Use a nightly snapshot of AS, or build one of the branches for yourself. IronJacamar 1.1 is for EE7, so that won't work in AS7

          3) Use 902 - or build latest Git master / REL_9_2_STABLE yourself - the key is to have a connection checker defined

          • 2. Re: JBoss AS 7.1.1 Final XA datasources memory leak
            karl_zhang

            Dear Jesper,

            Thank you  for your detail explanation.

             

            Best Regards,

             

            Karl Gross

            • 3. Re: JBoss AS 7.1.1 Final XA datasources memory leak
              karl_zhang

              Dear Jesper,

               

              I did the following according to your instructions:

              1. download a AS7 nightly build: https://ci.jboss.org/jenkins/job/JBoss-AS-7.x-latest/lastSuccessfulBuild/artifact/build/target/jboss-as-7.x.zip

              2. update ironjacamar module to 1.1.0.Beta1, use postgres 9.1-902 JDBC 4

              3. add the following elements in xa-datasource defnition:

              <recovery>

                 <recover-plugin class-name="org.jboss.jca.core.recovery.ValidatingManagedConnectionFactoryRecoveryPlugin"/>

              </recovery>

              <validation>

                 <valid-connection-checker class-name="org.jboss.jca.adapters.jdbc.extensions.postgres.PostgreSQLValidConnectionChecker"></valid-connection-checker>

                 <exception-sorter class-name="org.jboss.jca.adapters.jdbc.extensions.postgres.PostgreSQLExceptionSorter"></exception-sorter>

              </validation>

              * In JBTM-1183 the attribute no-recovery="false" is recommended also. But I could not use it, because JBoss gave an error during booting: Caused by: org.jboss.msc.service.StartException in anonymous service: JBAS010432: Unable to start the ds because it generated more than one cf.

               

              Unfortunately the class com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule still used 74% of the total heap memory.

               

              Could you please tell me, if I have done correctly? Or do you know any other issues related to this class? Thank you!

               

              Best Regards,

               

              Karl Gross

              • 4. Re: JBoss AS 7.1.1 Final XA datasources memory leak
                jesper.pedersen

                2) You can't use IronJacamar 1.1 in AS7 - just use nightly build directly

                3) We will look at the no-recovery issue

                 

                Please, provide an Eclipse MAT report that shows what is holding on to the instances.

                • 5. Re: JBoss AS 7.1.1 Final XA datasources memory leak
                  karl_zhang

                  Dear Jesper,

                   

                  thank you for your quick response. I will check it again using the nightly build directly.

                   

                  A report from Eclipse MAT:

                   

                  Class Name                                                                                | Shallow Heap | Retained Heap | Percentage

                  ------------------------------------------------------------------------------------------------------------------------------------------

                  .*com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.*                      |          |           |      
                  com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule @ 0x6c89b628             |       48 |   673.829.016 | 73,88%
                  |- java.util.Hashtable @ 0x6c9d7d38                                                       |       40 |   673.828.208 | 73,88%
                  |  |- java.util.Hashtable$Entry[24575] @ 0x9014c2f8                                       |   98.312 |   673.828.136 | 73,88%
                  |  |  |- java.util.Hashtable$Entry @ 0xa1e562c0                                           |       24 |   230.872 |  0,03%
                  |  |  |  |- java.util.Hashtable$Entry @ 0xa1604c78                                        |       24 |   192.696 |  0,02%
                  |  |  |  |- org.jboss.jca.core.tx.jbossts.XAResourceWrapperImpl @ 0xa1e59460              |       32 |    37.848 |  0,00%
                  |  |  |  |- com.arjuna.ats.internal.jta.recovery.arjunacore.RecoveryXids @ 0xa1e59480     |       32 |       304 |  0,00%
                  |  |  |  '- Total: 3 entries                                                              |          |           |      
                  |  |  |- java.util.Hashtable$Entry @ 0x90140630                                           |       24 |   230.864 |  0,03%
                  |  |  |- java.util.Hashtable$Entry @ 0x9cd1e4b0                                           |       24 |   192.688 |  0,02%
                  |  |  |- java.util.Hashtable$Entry @ 0x9db84dd8                                           |       24 |   192.688 |  0,02%
                  |  |  |- java.util.Hashtable$Entry @ 0x9ec43e28                                           |       24 |   192.688 |  0,02%
                  |  |  |- java.util.Hashtable$Entry @ 0x65fc69d8                                           |       24 |   192.680 |  0,02%
                  |  |  |- java.util.Hashtable$Entry @ 0x9e38b9a0                                           |       24 |   192.680 |  0,02%
                  |  |  |- java.util.Hashtable$Entry @ 0xa1e560e0                                           |       24 |   192.664 |  0,02%
                  |  |  |- java.util.Hashtable$Entry @ 0x65fc8508                                           |       24 |   192.328 |  0,02%
                  |  |  |- java.util.Hashtable$Entry @ 0x9db83f20                                           |       24 |   192.328 |  0,02%
                  |  |  |- java.util.Hashtable$Entry @ 0x9c4a6ad0                                           |       24 |   192.320 |  0,02%
                  |  |  |- java.util.Hashtable$Entry @ 0x9cd1e6d8                                           |       24 |   192.320 |  0,02%
                  |  |  |- java.util.Hashtable$Entry @ 0xa05151f0                                           |       24 |   192.320 |  0,02%
                  |  |  |- java.util.Hashtable$Entry @ 0x9a24d188                                           |       24 |   191.968 |  0,02%
                  |  |  |- java.util.Hashtable$Entry @ 0xa0d8c3d8                                           |       24 |   191.968 |  0,02%
                  |  |  |- java.util.Hashtable$Entry @ 0x999de8e0                                           |       24 |   191.960 |  0,02%
                  |  |  |- java.util.Hashtable$Entry @ 0x9f483d40                                           |       24 |   154.168 |  0,02%
                  |  |  |- java.util.Hashtable$Entry @ 0x65fc66c0                                           |       24 |   154.160 |  0,02%
                  |  |  |- java.util.Hashtable$Entry @ 0x81ef5778                                           |       24 |   154.160 |  0,02%
                  |  |  |- java.util.Hashtable$Entry @ 0x65fc9510                                           |       24 |   154.160 |  0,02%
                  |  |  |- java.util.Hashtable$Entry @ 0x65fc9f18                                           |       24 |   154.160 |  0,02%
                  |  |  |- java.util.Hashtable$Entry @ 0x65fc6678                                           |       24 |   154.160 |  0,02%
                  |  |  |- java.util.Hashtable$Entry @ 0x90182ce0                                           |       24 |   154.160 |  0,02%
                  |  |  |- java.util.Hashtable$Entry @ 0x96bd9488                                           |       24 |   154.160 |  0,02%
                  |  |  |- java.util.Hashtable$Entry @ 0x97567ec0                                           |       24 |   154.160 |  0,02%
                  |  |  '- Total: 25 of 17.746 entries; 17.721 more                                         |          |           |      
                  |  |- java.util.Collections$SynchronizedSet @ 0x6ca89178                                  |       16 |        32 |  0,00%
                  |  '- Total: 2 entries                                                                    |          |           |      
                  |- java.util.LinkedList @ 0x6c9d7d08                                                      |       24 |       504 |  0,00%
                  |- java.lang.String @ 0x6bdff5f8  Local XARecoveryModule                                  |       24 |        80 |  0,00%
                  |- java.util.ArrayList @ 0x6c9d7d68                                                       |       24 |        80 |  0,00%
                  |- java.util.ArrayList @ 0x6c9d7d20                                                       |       24 |        48 |  0,00%
                  |- java.util.ArrayList @ 0x6c9d7cf0                                                       |       24 |        40 |  0,00%
                  |- com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryResourceManagerImple @ 0x6c9d7d60|        8 |         8 |  0,00%
                  '- Total: 7 entries                                                                       |          |           |      
                  class com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule @ 0x6be237f8       |        0 |         0 |  0,00%
                  Total: 2 entries (219.077 filtered)                                                       |          |           |      

                  ------------------------------------------------------------------------------------------------------------------------------------------

                  • 6. Re: JBoss AS 7.1.1 Final XA datasources memory leak
                    karl_zhang

                    Dear Jesper,

                     

                    I started my JBoss instance yestoday, simply using a nightly build. Meanwhile the retained heap for XARecoveryModule was already about 50% of the total heap size. I have no ideas how to proceed with this. The configuration of my datasource looks like:

                     

                    <xa-datasource jndi-name="java:/server.ds.core" pool-name="server.ds.core-pool" enabled="true" use-java-context="true">

                        <xa-datasource-property name="DatabaseName">

                            server_db

                        </xa-datasource-property>

                        <xa-datasource-property name="User">

                            user

                        </xa-datasource-property>

                        <xa-datasource-property name="ServerName">

                            server

                        </xa-datasource-property>

                        <xa-datasource-property name="PortNumber">

                            5432

                        </xa-datasource-property>

                        <xa-datasource-property name="Password">

                            xxx

                        </xa-datasource-property>

                        <xa-datasource-class>org.postgresql.xa.PGXADataSource</xa-datasource-class>

                        <driver>postgresql-jdbc4</driver>

                        <transaction-isolation>TRANSACTION_SERIALIZABLE</transaction-isolation>

                        <xa-pool>

                            <min-pool-size>1</min-pool-size>

                            <max-pool-size>75</max-pool-size>

                            <use-strict-min>true</use-strict-min>

                        </xa-pool>

                        <security>

                            <user-name>user</user-name>

                            <password>xxx</password>

                        </security>

                        <recovery>

                            <recover-plugin class-name="org.jboss.jca.core.recovery.ValidatingManagedConnectionFactoryRecoveryPlugin"/>

                        </recovery>

                        <validation>

                            <valid-connection-checker class-name="org.jboss.jca.adapters.jdbc.extensions.postgres.PostgreSQLValidConnectionChecker"></valid-connection-checker>

                            <exception-sorter class-name="org.jboss.jca.adapters.jdbc.extensions.postgres.PostgreSQLExceptionSorter"></exception-sorter>

                            <check-valid-connection-sql>select 1</check-valid-connection-sql>

                        </validation>

                        <timeout>

                            <idle-timeout-minutes>5</idle-timeout-minutes>

                        </timeout>

                        <statement>

                            <track-statements>TRUE</track-statements>

                        </statement>

                    </xa-datasource>

                    <drivers>

                        <driver name="postgresql-jdbc4" module="org.postgresql"/>

                    </drivers>

                     

                    The version of ironjacamar in the night build JBoss is 1.0.12. I am using postgresql-9.1-902.jdbc4.jar. Could you please tell me, if I have done something wrong in the configuration? Thank you!

                     

                    Best Regards,

                     

                    Karl Gross

                    • 7. Re: JBoss AS 7.1.1 Final XA datasources memory leak
                      jesper.pedersen

                      You don't need the <check-valid-connection-sql> element, otherwise it looks good. See if you can upload a small heap dump (compressed) to this thread that shows the problem.

                      • 8. Re: JBoss AS 7.1.1 Final XA datasources memory leak
                        jesper.pedersen

                        Could you try and see if this completely untested jar makes a difference ?

                        • 9. Re: JBoss AS 7.1.1 Final XA datasources memory leak
                          karl_zhang

                          Dear Jesper,

                           

                          thank you for creating the jar file. Unfortunately exchanging the jar file has made no difference. After JBoss was running not more than two days, the heap size for XARecoveryModule reached already 400MB.

                          Is this problem also related to the not implemented method Connection.isValid in postgresql-9.1-902?

                           

                          Because I am testing a  migrated project from JBoss 4, the memory dump is at least 600MB. It is difficult for me to provide a small dump file.

                           

                          Best Regards,

                           

                          Karl Gross

                          • 10. Re: JBoss AS 7.1.1 Final XA datasources memory leak
                            karl_zhang

                            Dear Jesper,

                             

                            I have uploaded a compressed heap dump file with the size of 21MB. You can get it from http://www.2shared.com/file/jznfgHor/datasource-memory-leak-idp-dum.html. The dump is from our JBoss instance for single sign on.

                             

                            Could you please help us to analyze this dump file and find a workaround also applicable for JBoss 7.1.1?

                             

                            Because even if the workaround works for JBoss nightly build, it is not enough for users who want to migrate to JBoss 7 at the moment. Moreover no timeline for the next release can be found in the JBoss AS roadmap. This uncertainty will also make lot of users hesitate to change to JBoss7.

                             

                            Thank you for your help!

                             

                            Best Regards,

                             

                            Karl Gross

                            • 11. Re: JBoss AS 7.1.1 Final XA datasources memory leak
                              jhalliday

                              ummm, oops.   per my comment on JBTM-1183, the work in JBTM-924 should actually prevent this even where isValid is not working. Unfortunately that fix got scrambled by some later work, effectively creating a regression. We're on the case.

                              • 12. Re: JBoss AS 7.1.1 Final XA datasources memory leak
                                karl_zhang

                                Dear Jonathan,

                                 

                                thank you for your reply! Because we are now almost at the end of migrating our application to JEE6, hopefully we could get a solution as soon as possible.

                                 

                                Best Regards,

                                 

                                Karl Gross

                                • 13. Re: JBoss AS 7.1.1 Final XA datasources memory leak
                                  jhalliday

                                  In that case reproduce it on your EAP server, log in to the customer portal and create a support ticket for it, referencing this thread. That way it's covered by the subscription SLA rather than the community 'when we get to it' model. We don't do community patch releases, so without a subscription you're waiting for a while.

                                  • 14. Re: JBoss AS 7.1.1 Final XA datasources memory leak
                                    didier2011

                                    We're also experiencing this memory leak and need a fix for this, urgently, but we want a patch for 7.1.1 community version!

                                    But as I understand this thread, there is no patch available at the moment? Isn't this a very heavy bug? We cannot run our application longer than three days without memory exception.

                                    1 2 Previous Next