1 2 3 Previous Next 41 Replies Latest reply on Apr 11, 2012 12:08 PM by smarlow Go to original post
      • 15. Re: Startup Goes Very Slow with Scanning Entity Classes from Different Jars
        smarlow

        If you can get the "Caused by: java.lang.OutOfMemoryError: Java heap space" to happen again, it would be very helpful to start the AS instance with an additional option  -XX:+HeapDumpOnOutOfMemoryError.

         

        This will generate a heapdump (probably in the as7/bin folder) that shows what is in memory at the time of running out of memory.  We can load the heap dump with the MAT tool and you can show me some screen shots of the top memory consumers (you will see a nice pie chart right away that shows the memory breakdown, take a picture of that also).

         

        Scott

        • 16. Re: Startup Goes Very Slow with Scanning Entity Classes from Different Jars
          ablet

          Stephan, yes, each ejb module is loading it's own persistence.xml, however they have same Unit Name.

           

          The point you have highlighted here is very valuable to us, and I will share and disscuss it with the project team. I would like you know that almalgamating the modules with th size of our project need meticulous consideration from technical and non-technical part of views.

          How do you think about sharing persistence.xml among some ejb-modules (persistence outside of ejb-jar). In that case will entity classes be auto scanned by container as it does when it is located inside ejb-jar?

           

          Thank you.

          • 17. Re: Startup Goes Very Slow with Scanning Entity Classes from Different Jars
            ablet

            I got the memory HeapDump by -XX:+HeapDumpOnOutOfMemoryError. and this is the report:

             

            The first Chart:

            memoryBreakDown.JPG

            Top Consumers:

            TopConsumer_1.JPG

            TopConsumer_2.JPG

             

              

            Dominator Tree: List the Biggest Objects ...

            treeDominator_extended.JPG

             

            Leaks:

             

            heapdump.JPG

             

             

            Problem Suspect 1:

            40 instances of "org.jboss.as.jpa.validator.SerializableValidatorFactory", loaded by "org.jboss.modules.ModuleClassLoader @ 0x14021a90" occupy 361,273,488 (37.10%) bytes.

             

            Problem Suspect 2:

            38 instances of "org.hibernate.internal.SessionFactoryImpl", loaded by "org.jboss.modules.ModuleClassLoader @ 0x13e1bae0" occupy 286,931,768 (29.47%) bytes.

            Problem Suspect 3:

            38 instances of "org.hibernate.cfg.Configuration", loaded by "org.jboss.modules.ModuleClassLoader @ 0x13e1bae0" occupy 151,909,248 (15.60%) bytes.

            • 18. Re: Startup Goes Very Slow with Scanning Entity Classes from Different Jars
              ablet

              I also have the report of the Duplicate classes (Detect classes loaded by multiple class loaders):

               

              DuplicateClasses1.JPG

               

              DuplicateClasses2.JPG

               

              Why there are many dublications and there is a huge number (193410) of instances?

               

              thanks again.

              • 19. Re: Startup Goes Very Slow with Scanning Entity Classes from Different Jars
                smarlow

                Lets start with a look at the biggest suspect "

                40 instances of "org.jboss.as.jpa.validator.SerializableValidatorFactory", loaded by "org.jboss.modules.ModuleClassLoader @ 0x14021a90" occupy 361,273,488 (37.10%) bytes.".

                 

                About how many persistence units are you deploying (more than 40)?  Do you have a large number or small number of bean validation rules in the application? 

                 

                Could you paste more details from the BeanMetaDataCache ConcurrentHashMap?  I think if you right mouse button click on the ConcurrentHashMap and choose to look at outgoing references (meaning to show what the map references as apposed to incoming which is what references the map).  Or perhaps you can keep drilling into the map from the dominator tree directly.  The important part is that when you left mouse button click on a field in the map, the left side of the screen should show all of the contents.  I'd like to get a better idea of what exactly is in map (perhaps its a lot of duplicate information or all of the validation info from your app). 

                 

                If that doesn't seem to work, try opening the histogram view and type in "org.hibernate.validator.metadata.BeanMetaDataCache" in the class name filter (or scroll to that class).  You should be able to drill into the map contents from there (if you can't from the dominator tree).

                 

                Regarding the duplicate class view, I'm not sure about the importance of that yet.  Maybe come back to that after drilling further into the memory issue associated with BeanMetaDataCache.

                 

                Scott

                • 20. Re: Startup Goes Very Slow with Scanning Entity Classes from Different Jars
                  hardy.ferentschik

                  The problem might be related to https://hibernate.onjira.com/browse/HV-479. Hibernate Validator 4.2 cached the bean metadata per class without implementing a proper eviction strategy. You could try the latest 4.3 SNAPSHOT, buy you would have to make adjustments in your validator module (modules/org/hibernate/validator/main). I could help you with this if you want to give it a go.

                   

                  Other than that it would really help to get more information about your actual deployment scenario. The very best would be if you could create a test case which you could share so that we can have a look at the actual code. As Scott is saying some more information about what's in the BeanMetaDataCache would be helpful as well.

                   

                  Another thing I am wondering about - are you using hot deploys?

                   

                  --Hardy

                  • 21. Re: Startup Goes Very Slow with Scanning Entity Classes from Different Jars
                    smarlow

                    Mostafa,

                     

                    Its been a few days since we have heard back from you.  Will you be able to continue to work with us, to help you get past this issue?  If not, can you answer some of the questions we asked in our previous responses (that might help us understand more about the cause in case it happens to someone else). 

                     

                    Scott

                    • 22. Re: Startup Goes Very Slow with Scanning Entity Classes from Different Jars
                      ablet

                      My apology not to be able to respond sooner. I wish i could provide any feedback on what you disscussed above, but at the current time i am not in the development site location of the project and it's not applicable to access them remotely. I had already shared this issue and disscussion with our project manager and he was looking for the right person to continue to work on this. I do remind him about this, hopefully one of my colleagues could continue to work on this. Meanwhile, I try to answer ur questions (or any testing that might not need development site.) as much as i can.

                       

                       

                      About how many persistence units are you deploying (more than 40)?  Do you have a large number or small number of bean validation rules in the application?

                      Yes we have more than 40 persistence Unit in our project. and as far as i know, we don't have beaen validation rules,  (correction: we do use many bean validation rules).

                      • 23. Re: Startup Goes Very Slow with Scanning Entity Classes from Different Jars
                        ablet

                        Thanks Hardy Ferentschik, For you information, we used Hibernate 3.2 on deploying application whith JBoss AS 4.3 which we didn't have this GC issue. while we were migrating to JBoss AS7 we faced an issue similar to https://issues.jboss.org/browse/AS7-3080 therefore we decided to migrate our project to use hibernate 4.0 (default Hibernate included in jboss-as-7.1.0.CR1b ). Based on what you  mentioned here, I think we have to update the hibernate validation module to use 4.3 SNAPSHOT, rather than current hibernate-validator-4.2.0.Final.jar and then test the deployment.

                         

                         

                        Other than that it would really help to get more information about your actual deployment scenario. The very best would be if you could create a test case which you could share so that we can have a look at the actual code. As Scott is saying some more information about what's in the BeanMetaDataCache would be helpful as well.

                         

                        Another thing I am wondering about - are you using hot deploys?

                        I am not sure what do u want me to explain about deployment scenario, at first record of these two disscussions, https://community.jboss.org/message/723709#723709 , and https://community.jboss.org/thread/197521?start=0&tstart=0, I provided some information about the application, and deployment settings, Please let me know if you need more information about that.

                         

                        I might not be able to provide a testCase as i explained before, currennt problem just can be seen when i deploy whole application (more than 80 modules that many of them have their own persistence unit with cross dependency with entity class scanning through <jar-file> on each other ).

                         

                        I try to provide more information regarding to what's in BeanMetaDataCache.

                         

                        Current problem is just based on fresh startup and not hot deploys.

                         

                        Thanks a lot.

                        • 24. Re: Startup Goes Very Slow with Scanning Entity Classes from Different Jars
                          smarlow

                          Mostafa,

                           

                          Thanks for the information about the number of persistence units and how you have no validation rules (that you know of) in the application. 

                           

                          If there aren't any validation rules in the app, try setting the javax.persistence.validation.mode property to NONE in each persistence unit and see if that helps improve deployment performance.  It would be helpful if you still tried the Hibernate Validator snapshot build first (probably easier).

                           

                          Scott

                          • 25. Re: Startup Goes Very Slow with Scanning Entity Classes from Different Jars
                            ablet

                            Scott, sorry for misunderstanding of mine, many bean validation rules have been used in the application.

                             

                            Regarding to trying the the Hibernate Validator snapshot build , as i already mentioned in previous post it's not applicable for me to test the deployment and provide feedback to you (I am not in the development site, and i am traveling to another location) . I am sure the project manager will assign someone to check that.

                            • 26. Re: Startup Goes Very Slow with Scanning Entity Classes from Different Jars
                              fabdouglas

                              Hi everybody,

                              I get this OutOfMemory on my Maven build but not on my lightweight Jetty server.

                              I've assigned 1.6G to the VM and this line causes the outofmemory :

                              Validation.buildDefaultValidatorFactory().getValidator().validate(bean)

                               

                              I don't know yet where is the memory issue : during the validate, during the validator creation,...

                              Anyway, when this line is executed, I get OutOfMemory : from 300M used, i reach the 1.4G used.

                              I think this is a classloader issue due to the important amount of available JARs in my continuous integration classpath. I guess in AS7 there are a plenty of classes too....

                               

                              I've been able to reproduce this issue on :

                              4.0.2.GA

                              4.2.0.Final

                              4.3.0.Beta1

                               

                              I will investigate to find the leak.

                              • 27. Re: Startup Goes Very Slow with Scanning Entity Classes from Different Jars
                                fabdouglas

                                Step by step... the issue is in Validation.buildDefaultValidatorFactory()

                                • 28. Re: Startup Goes Very Slow with Scanning Entity Classes from Different Jars
                                  smarlow

                                  Fabrice,

                                   

                                  What are the version numbers for?  Hibernate Validator?

                                   

                                  Could you also try the latest Hibernate Validator 4.3 snapshot (that Hardy mentioned earlier)?  Copy the new jar into as7/modules/org/hibernate/validator/main folder and update the module.xml file to reference new HV jar instead of hibernate-validator-4.2.0.Final.jar.

                                   

                                  Scott

                                  • 29. Re: Startup Goes Very Slow with Scanning Entity Classes from Different Jars
                                    hardy.ferentschik

                                    Hi Fabrice and Mostafa,

                                     

                                    regarding Hibernate Validator. You don't need to use a SNAPSHOT anymore. If you want to try a new version give Hibernate Validator 4.3.0.Beta1 a try - https://repository.jboss.org/nexus/content/repositories/public/org/hibernate/hibernate-validator/4.3.0.Beta1/

                                    This release addresses HV-479 which might cause you problem. As Scott is saying, you should be able to just replace the jar in the modules/org/hibernate/validator/main folder.

                                     

                                    Fabrice, regarding your maven build - are you using Arquillian and some sort of embedded AS in your build? Or do you have some other unit tests which trigger validation. I am just trying to understand if and where AS and Validator come into play. Any more information about what actually fails would be great.

                                     

                                    Mostafa, any chance you could give an estimate on the number of entities and number of constraints within the estimates. Indpendent of the amount of ValidatorFactorys in the system, it seems there is a lot of meta data per factory. If I got the numbers right we are talking around 344 MB of data taken up by 40 instances of SerializableValidatorFactory. That would make for around 9MB of metadata per factory. That would require either A LOT of entities with constraints or hot redeploys (basically you redeploying an ear/war etc by replacing it with a newer version or maybe you just touch the file to trigger the redeploy. In this case new meta data classes will be created, because even though the class itself might not have changed, the classloader has).