14 Replies Latest reply on Feb 24, 2011 8:59 AM by atijms

    Class loader issues with JB 6.0 and HornetQ

    cdollar393

      Hi All,

       

      I am migrating my application from JBoss 4.2 to JBoss 6.0 Final and am running into some classloading issues with HornetQ.

       

      I deploy my application as a single .ear file. It is a typical .ear with multiple .sar, .war, and .jar files. I keep all my common files, interfaces, etc. in their own .jar file, and that .jar goes into the lib/ folder in the .ear.

       

      One of my .war files has an application scoped backing bean that also implements MessageListener. It receives ObjectMessage (I know this is not the best for serialization) JMS messages.

       

      My issue comes in when this class receives a message. As soon as I call the getObject() method on the ObjectMessage I get an error:

       

      {code}16:36:03,890 ERROR [com.rydindecal.PermitExpress.web.beans.UserBean] Error while receiving JMS message: javax.jms.JMSException: com.rydindecal.PermitExpress.common.ObjectChangeNotification

              at java.net.URLClassLoader$1.run(URLClassLoader.java:217) [:1.6.0_18]

              at java.security.AccessController.doPrivileged(Native Method) [:1.6.0_18]

              at java.net.URLClassLoader.findClass(URLClassLoader.java:205) [:1.6.0_18]

              at java.lang.ClassLoader.loadClass(ClassLoader.java:321) [:1.6.0_18]

              at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294) [:1.6.0_18]

              at java.lang.ClassLoader.loadClass(ClassLoader.java:266) [:1.6.0_18]

              at java.lang.Class.forName0(Native Method) [:1.6.0_18]

              at java.lang.Class.forName(Class.java:264) [:1.6.0_18]

              at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:284) [jboss-classloader.jar:2.2.0.GA]

              at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1152) [jboss-classloader.jar:2.2.0.GA]

              at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:886) [jboss-classloader.jar:2.2.0.GA]

              at org.jboss.classloader.spi.base.BaseClassLoader.doLoadClass(BaseClassLoader.java:505) [jboss-classloader.jar:2.2.0.GA]

              at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:450) [jboss-classloader.jar:2.2.0.GA]

              at java.lang.ClassLoader.loadClass(ClassLoader.java:266) [:1.6.0_18]

              at java.lang.Class.forName0(Native Method) [:1.6.0_18]

              at java.lang.Class.forName(Class.java:264) [:1.6.0_18]

              at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:621) [:1.6.0_18]

              at org.hornetq.utils.ObjectInputStreamWithClassLoader.resolveClass(ObjectInputStreamWithClassLoader.java:69) [:6.0.0.Final]

              at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1592) [:1.6.0_18]

              at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1513) [:1.6.0_18]

              at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1749) [:1.6.0_18]

              at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1346) [:1.6.0_18]

              at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368) [:1.6.0_18]

              at org.hornetq.jms.client.HornetQObjectMessage.getObject(HornetQObjectMessage.java:158) [:6.0.0.Final]

              at com.rydindecal.PermitExpress.web.beans.UserBean.onMessage(UserBean.java:163)

              at org.hornetq.jms.client.JMSMessageListenerWrapper.onMessage(JMSMessageListenerWrapper.java:91) [:6.0.0.Final]

              at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:822) [:6.0.0.Final]

              at org.hornetq.core.client.impl.ClientConsumerImpl.access$100(ClientConsumerImpl.java:46) [:6.0.0.Final]

              at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:940) [:6.0.0.Final]

              at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [:6.0.0.Final]

              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [:1.6.0_18]

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [:1.6.0_18]

              at java.lang.Thread.run(Thread.java:636) [:1.6.0_18]{code}

       

      In this case "ObjectChangeNotification" is the actual object in the ObjectMessage.

       

      At this point I tried to change my code to use a MapMessage rather than ObjectMessage. This is when I ran into my second problem... In the onMessage() method in my class I was using JNDI and an interface to lookup an EJB and call one of it's methods. When the onMessage() was called it would fail with this error at the step that made the EJB call:

       

      {code}

      16:16:27,065 WARN  [org.hornetq.jms.client.JMSMessageListenerWrapper] Unhandled exception thrown from onMessage: java.lang.RuntimeException: Specified calling class, com.rydindecal.PermitExpress.interfaces.ClientManager could not be found for BaseClassLoader@4b4b50{vfs:///home/chris/java/projects/jboss-6.0.0.Final.git/server/permitexpress/conf/jboss-service.xml}

      {code}

       

      Now I'm really believing it is a classloading issue, so to test this I changed the "isolated" property in JBOSS_HOME/server/< servername>/deployer/ear-deployer-jboss-beans.xml to false as per this thread. This got it working.

       

      Then I went back and changed my MapMessage stuff back to ObjectMessage and tried again -- this also worked.

       

      As soon as I change "isolated" back to true it doesn't work.

       

      Now for my question... What is the best approach to fix this? The thread I linked to above states that setting that value to false is not recommended for production, so I'm hesitant to think this is a long-term solution. Based on the exception it appears that the onMessage() call was using the sytem-scoped classloader that JBoss uses (and is defined in JBOSS_HOME/server/< servername>/conf/jboss-service.xml) rather than the deployed .ear's "isolated" classloader, but I could be wrong there. I'm guessing that if I took my .jar of common objects and interfaces and put it in JBOSS_HOME/common/lib that things would work with "isolated" set to true, but this would be really inconvenient.

       

      Any thoughts or ideas?

       

      Thanks!

      Chris

        • 1. Class loader issues with JB 6.0 and HornetQ
          cdollar393

          Sorry for the bump, but does anyone have any thoughts on my issue?

           

          Thanks again,

          Chris

          • 2. Class loader issues with JB 6.0 and HornetQ
            henk53

            Hi,

             

            One of my .war files has an application scoped backing bean that also implements MessageListener. It receives ObjectMessage (I know this is not the best for serialization) JMS messages.

             

            What does the code for this look like? Is this a backing bean also an MDB or did you programmatically let it listen for messages?

            • 3. Class loader issues with JB 6.0 and HornetQ
              cdollar393

              henk de boer wrote:

               

              Hi,

               

              One of my .war files has an application scoped backing bean that also implements MessageListener. It receives ObjectMessage (I know this is not the best for serialization) JMS messages.

               

              What does the code for this look like? Is this a backing bean also an MDB or did you programmatically let it listen for messages?

               

              It is a plain backing bean that is listening for messages programatically. Basically I connect and create the message listener in the constructor for the class. When a message is received I update a number of local objects based on the contents of the message.

               

              I've actually recently done the same thing with a session scoped backing bean. My .ear application can have multiple people logged in at the same time, working on the same thing. If a user makes a change to something (backing bean calls an EJB) that EJB will send a JMS message about what was changed. The session-scoped bb gets that message, and if the original change was made by a different user (i.e. user A made the change but user B is getting the JMS message) then I use an a4j:push to let user B know that user A changed something and we need to grab the latest updates asap. There are a few other things going on in the background to handle race and concurrency issues, but thats the general idea.

               

              I'm not at all sure if this is a good design pattern, but so far it's worked pretty well for my needs.

              • 4. Class loader issues with JB 6.0 and HornetQ
                clebert.suconic

                This will be plain serialization.

                 

                 

                For *any* messaging system, an Object Message is just a thin layer to generate a big payload of bytes. Nothing special in terms of messaging.

                 

                You just need to make it work through simple Serialization. Nothing else.

                 

                 

                If you have a serialization exception, that would be the same as if you were doing a writeObject and readObject through Object*Stream.

                 

                 

                The only "special" arrangement messaging systems will make is to encapsulate the classLoader through a dumb extension of ObjectInputStream. (look at any JMS Provider's source and you will find something equivalent).

                 

                 

                So, in summary.. just make sure you serialization bits are matching with versioning and classloader.

                • 5. Class loader issues with JB 6.0 and HornetQ
                  cdollar393

                  Hi Clebert,

                   

                  I still don't quite understand...

                   

                  I deploy my application as an .ear file. All my common classes, including the Object that I use in my ObjectMessage, are packaged into a .jar file that then goes into the lib/ folder at the root of my .ear archive.

                   

                  As I understand it, the classes in the lib/ folder of my .ear should be available in the classloader used by all the additional resouces (such as .wars, EJBs packaged into .jars, etc.). In my case I only package my ObjectChangeNotification class into my common lib jar -- it is not included separately in the other .wars, etc.

                   

                  I have made sure my ObjectChangeNotification class implements Serializable, and it only contains primative objects in that class so all other items should be serializable.

                   

                  Since I only include 1 copy of the ObjectChangeNotification class in my .ear I'm not sure how this could be a versioning mismatch issue or the like.

                   

                  Am I missing some simple concept here?

                   

                  I really do appreaciate the help!

                  Chris

                  • 6. Class loader issues with JB 6.0 and HornetQ
                    john.waterwoodx

                    @mr Dollar,

                     

                    The design pattern sounds solid, although in JBoss AS 6 you could also consider the CDI event bus. This is a leightweight alternative for one of the usecases where JMS is traditionally used.

                    • 7. Class loader issues with JB 6.0 and HornetQ
                      clebert.suconic

                      I'm not sure how do you see this as a HornetQ issue. If there's any issue here it will be a classloader or packaging isolation.

                       

                      We can only help with the HornetQ part here. (Although I would help you if I knew the answer right away).

                       

                       

                      Anyway: All I know is that classLoading isolation is not a fun thing. You probably have a small mistake in your packaging. A common mistake I have seen is people adding hornetq clients.jar on the EAR (or the jms-api to the EAR) what will cause issues.  People tend to do that these days without realizing it because of maven automatic dependencies.

                      • 8. Class loader issues with JB 6.0 and HornetQ
                        cdollar393

                        Hi Clebert,

                        Clebert Suconic wrote:

                         

                        I'm not sure how do you see this as a HornetQ issue. If there's any issue here it will be a classloader or packaging isolation.

                         

                        We can only help with the HornetQ part here. (Although I would help you if I knew the answer right away).

                         

                         

                        Anyway: All I know is that classLoading isolation is not a fun thing. You probably have a small mistake in your packaging. A common mistake I have seen is people adding hornetq clients.jar on the EAR (or the jms-api to the EAR) what will cause issues.  People tend to do that these days without realizing it because of maven automatic dependencies.

                        First off, thank you for replying to my question - I do appreciate it.

                         

                        Second, I'm not at all sure that this is a HornetQ issue. It very well could be an AS issue or an issue with my code and its packaging. I first posed the question here because the only area that I am having issues with is consuming JMS messages. Using the classes contained in my ObjectMessage anywhere outside of the onMessage() method (even uses elsewhere in the same class that has the onMessage() method) works fine -- I only have issues with the classes while consuming messages.

                         

                        I have created a simple test case that illustrates the behavior I'm describing, although I'm not sure how to attach it to a forum post. If you'd like I can send the testcase to you, or if you'd prefer I can move this discussion to the AS forums.

                         

                        Thanks again,

                        Chris

                        • 9. Class loader issues with JB 6.0 and HornetQ
                          jaikiran

                          Looking at that stacktrace the code hasn't yet reached to the AS layer and is failing in the HornetQ layer. As an FYI, there was (is?) a similar bug in JBoss Messaging http://community.jboss.org/message/517805#517805

                          • 10. Class loader issues with JB 6.0 and HornetQ
                            clebert.suconic

                            That's just a ClassNotFound Exception.

                             

                            The ObjectMessage is using the application's classloader. (At least on the latest version)

                             

                             

                            Unless the user is using an old version or something.

                             

                             

                             

                            If there's a bug here, I need a replication test.

                            • 11. Class loader issues with JB 6.0 and HornetQ
                              cdollar393

                              jaikiran pai wrote:

                               

                              Looking at that stacktrace the code hasn't yet reached to the AS layer and is failing in the HornetQ layer. As an FYI, there was (is?) a similar bug in JBoss Messaging http://community.jboss.org/message/517805#517805

                              Yes, you are correct. I tried the workaround described (temporarily changing the classloader in onMessage()) and that worked for me. Thanks!

                              • 12. Class loader issues with JB 6.0 and HornetQ
                                cdollar393

                                I tried the workaround described by jaikiran and it worked. As far as I can tell I don't have any issues with my packaging.

                                 

                                I can provide you my test case if you'd like, just let me know who to send it to or where to upload it.

                                 

                                Thanks again!

                                Chris

                                • 13. Class loader issues with JB 6.0 and HornetQ
                                  jaikiran

                                  Chris Dollar wrote:

                                   

                                   

                                   

                                  I can provide you my test case if you'd like, just let me know who to send it to or where to upload it.

                                   

                                  You can attach the testcase (along with instructions on how to run it) to this thread.

                                  • 14. Re: Class loader issues with JB 6.0 and HornetQ
                                    atijms

                                    Chris Dollar wrote:

                                     

                                    It is a plain backing bean that is listening for messages programatically.

                                     

                                    I had a problem not unlike this in JBoss AS 5.1.

                                     

                                    I too am creating a JMS consumer in the web module. In the onMessage method I can use classes from the web module using the new operator, but resources the code is loading via the class loader fail to be loaded. Upon inspection, it appeared that when onMessage is invoked, Thread.getCurrentThread().getContextClassloader() returns the BaseClassLoader (BaseClassLoader@313037f1{vfsfile:/home/myuser/jboss-5.1.0.GA/server/default/deploy/my_main.ear/}) instead of the web class loader (org.jboss.web.tomcat.service.WebCtxLoader$ENCLoader).

                                     

                                    I solved the problem in the following way. It executes in an (application scoped) JSF managed bean:

                                     

                                     

                                    final ClassLoader webClassLoader = Thread.currentThread().getContextClassLoader();
                                    
                                    con = myFactory.createConnection();
                                    
                                    con.createSession(false, Session.AUTO_ACKNOWLEDGE).createConsumer(myQueue).setMessageListener(
                                        new MessageListener() {
                                    
                                            @Override
                                            public void onMessage(Message message) {
                                    
                                                // New Foo() with Foo definition in Web module works
                                                // without class loader tricks
                                                ClassLoader oldLoader = null;
                                                try {
                                                    oldLoader = Thread.currentThread().getContextClassLoader();
                                                    // Set classloader or else code using
                                                    // Thread.currentThread().getContextClassLoader().getResourceAsStream
                                                    // etc will fail.
                                                    Thread.currentThread().setContextClassLoader(webClassLoader);
                                    
                                                    // Msg handling code here
                                    
                                                }                        
                                                finally {
                                                    if (oldLoader != null) {
                                                        Thread.currentThread().setContextClassLoader(oldLoader);
                                                    }
                                                }
                                            }                        
                                    
                                        }        
                                    );
                                    
                                    con.start();