9 Replies Latest reply on May 8, 2012 2:09 AM by mglowacki

    Switch to Infinispan made my JBoss 5 unstable

    mglowacki

      hello,

       

        I have recently get rid of jboss cache 1.0/2.0 in my web apps hosted on JBoss 5.1 and switched to cache 3.2 and infinispan 5.1.

       

        I was hoping for a big performance boost and less troubles. Unfortunately, it is disaster now. After few hours of operations, my JBoss process is taking 30% of CPU (40 cores machine...) when after restart it takes 4-5%. When I take a look at web-console, full connectors status, I see a lot (more of them, bigger CPU usage) threads/requests which run infinitely - status shows "Servicing" and all the time same 3 or 4 pages - all where I use Infinispan. I made several threads dumps - but it looks clean - no tracks of my apps code, no blocks...

       

         What could it be? On Javaranch I got a good tip - perhaps my app is spawning new threads (Java SE way) - but those requests I see in web-console does not contain and thread spawning methods... Maybe I misconfigured Infinispan? Or maybe it does not work well with JBoss 5.1? My config (I do it programatically) is lack of config:

       

      DefaultCacheManager cm = new DefaultCacheManager();

       

      and then I create new caches: cm.getCache("cache_name")

       

      I will be really grateful for any tips - I have no clue what to do, what more I can to in order to find the reason for this rising CPU usage...

       

      Regards,

      Michal

        • 1. Re: Switch to Infinispan made my JBoss 5 unstable
          rhusar

          What do you mean you got rid of the JBoss Cache -- AS 5.1 uses JBoss Cache internally for all clustering.

          How are you integrating Infinispan in your app?

          Where are you creating the cache managers?

          Looks like you are creating extra cache managers, can you check the jstack of your AS?

          • 2. Re: Switch to Infinispan made my JBoss 5 unstable
            mglowacki

            Radoslav,

             

            thank you very much for your answer.

             

            By "getting rid of jboss cache 1.0" I meant upgrading jboss cache to 3.0/Infinispan in my web apps and also removal of Seam's s:cache tag (which is compatible only with cache 1.0/2.0).

             

            I create cache managers in my web app - in singleton class. Example below:

             

            public abstract class CacheManager implements Serializable {

                      private static Logger log = Logger.getLogger(CacheManager.class);

                      private static final long serialVersionUID = -6768491704769446425L;

             

                      @SuppressWarnings("rawtypes")

                      protected Cache cache;

             

                      protected static DefaultCacheManager cm;

             

                      static {

                                try {

                                          cm = new DefaultCacheManager(/*"infinispan.xml"*/);

                                } catch (Exception e) {

                                          log.fatal("cannot locate infinispan.xml!");

                                          e.printStackTrace();

                                }

                      }

             

             

                      protected CacheManager(String cacheName)

                          throws OperationsException, NullPointerException {    

                    this.cache = cm.getCache(cacheName);       

                }

            ..

            }

             

            public class MyCache extends CacheManager implements Serializable {

                      private static final long serialVersionUID = 4727646036264788873L;

             

                      private static MyCache _instance;

               

                public MyCache () throws OperationsException, NullPointerException {

                                super("MyCache ");

                      }

             

                      public static MyCache getInstance() throws OperationsException, NullPointerException  {

                    if(_instance == null) {

                        _instance = new MyCache ();

                    }

                    return _instance;

                }

            }

             

            and later when I want to use it:

             

            MyCache cache = MyCache.getInstance();

             

             

             

            jstack of my web app would be difficult to get good results - hard to simulate this in test environment and I cannot do it in production.

            • 3. Re: Switch to Infinispan made my JBoss 5 unstable
              mglowacki

              I have made a jstack of the running jboss (at 50% CPU): http://speedy.sh/QQwt5/jstack.txt

              • 4. Re: Switch to Infinispan made my JBoss 5 unstable
                galder.zamarreno

                Michal, what are we looking for in that thread dump?

                • 5. Re: Switch to Infinispan made my JBoss 5 unstable
                  mglowacki

                  Radoslav told me to check it - I found nothing wrong, so I put it here for reference.

                   

                  Week by week I am thinking more about leaving JBoss server after 6 years - I cannot find the root of the problem (I just restarted my production server and after 10 minutes CPU is constant at 20% where it should be 2-3% max...), no one can help me and I send 3 or 4 forms for commercial support and never get any reply.

                   

                  well, thanks for the attention at least.

                  • 6. Re: Switch to Infinispan made my JBoss 5 unstable
                    galder.zamarreno

                    There's no trace of Jgroups nor Infinispan in that stacktrace, so no idea what's causing the high CPU usage. Are you sure it's the Java process doing that? If it is, then i'd suggest you mimic production load on a separate server and attach a profiler to see where CPU time is being used.

                     

                    The reason commercial (I assume you mean Red Hat) support didn't reply is because Infinispan is not supported in JBoss 5, so they cannot help you. If you want to use Infinispan within an app server, I'd suggest AS 7.1.

                    • 7. Re: Switch to Infinispan made my JBoss 5 unstable
                      jaikiran

                      Just adding to what Galder and Rado mentioned - it's difficult to say what's wrong due to the lack of specific information in the posts.

                       

                       

                      Michal Glowacki wrote:

                       

                      When I take a look at web-console, full connectors status, I see a lot (more of them, bigger CPU usage) threads/requests which run infinitely - status shows "Servicing" and all the time same 3 or 4 pages - all where I use Infinispan. I made several threads dumps - but it looks clean - no tracks of my apps code, no blocks...

                       

                      I'm not sure what threads those are. Can you attach the screenshot or even those thread dumps that you generated to help us understand what those threads are about?

                       

                      When you see this high CPU behaviour, what kind of operations are happening in your application? For example, are some users accessing your application which leads to Infinispan or any other kind of activity being triggered within your application?

                      • 8. Re: Switch to Infinispan made my JBoss 5 unstable
                        jaikiran

                        Michal Glowacki wrote:

                         

                        On Javaranch I got a good tip - perhaps my app is spawning new threads (Java SE way) -

                        That thread has a bit more information about your application http://www.coderanch.com/t/573915/Tomcat/AJP-requested-serviced-all-time#2611697 and from what I see in there, it isn't clear if this has anything to do with Infinispan. As a first step, I would enable the access log valve (which logs the access to your web application) and see where those additional requests are coming from and if they are expected.

                        1 of 1 people found this helpful
                        • 9. Re: Switch to Infinispan made my JBoss 5 unstable
                          mglowacki

                          Thank you all for help - it looks like the problem was 3rd party application. My customer has replaced it and now I got 2 weeks without restart.

                           

                          The reason I don't switch to JBoss AS 7 is complexity of moving the Seam 2 projects. Now with with Seam 2.3.0Beta1 with JSF 2.0 support I hope it will be easier.

                           

                          Thanks,

                          Michal