3 Replies Latest reply on Aug 19, 2016 7:31 AM by dusty86

    Infinispan 5.3 memory problem

    dusty86
    dusty86 Sep 30, 2015 8:55 AM

      Hi all,

       

      we are using Infinispan 5.3 as transient data cache provider inside our web-application. Currently we are running two separate application instances:

      • physical 6 server cluster
      • physical 2 server cluster

       

      On the 6 server cluster everything is running fine. On the 2 server cluster we have the problem, that on one server inside the cluster the memory is increasing day by day until we have to restart the application server in order to prevent Out-Of-Memory Errors. The interesting point is, that after the restart it may be that server a has the memory problem and server b is doing fine or vice versa (never both: kind of a random master-slave effect).

       

      After doing some profilings we see that org.jboss.marshalling.util:identityIntMap is causing the 'memory leak':

       

      scrn1.png

      scrn2.png

       

      It seems that the dynamic data inside the Map gets collected properly. However, the Map structure itself seems to be kept during the lifecycle of a whole thread.

       

      My question: Has anybody had alredy an issue like this or does anybody know a solution regarding this issue?

       

       

      Thanks and BR

      Roman

        • 1. Re: Infinispan 5.3 memory problem
          rvansa
          rvansa Sep 30, 2015 7:14 AM (in response to dusty86)

          Hi Roman, Infinispan 5.3 is more than 2 years old release, so we don't fix (neither investigate thoroughly) bugs there. However, we would be very keen to fix any memory leaks if you are able to reproduce them on Infinispan 8.

            • Actions
          • 2. Re: Infinispan 5.3 memory problem
            dan.berindei
            dan.berindei Oct 15, 2015 4:19 PM (in response to dusty86)

            My guess is that the maps you're seeing is the marshallers' instanceCache, because the marshallers are cached in a thread-local variable. Maybe you could cut down on the size of your thread pools, or allow idle worker threads to shut down after 60 seconds?

             

            Like Radim said, we're not going to fix the problem in 5.3, but we'd like to hear back from you in case you can reproduce it with Infinispan 8. If you do, it would be good to attach a screenshot with the incoming references to the IdentityIntMap, to clarify if the per-thread marshallers cache is at fault.

              • Actions
            • 3. Re: Infinispan 5.3 memory problem
              dusty86
              dusty86 Aug 19, 2016 7:31 AM (in response to dusty86)

              Hi all,

               

              finally we did an upgrade to Infinispan 8 on our productive system and sadly we have exact the same effect as we had on Infinispan 5.3. Any suggestions?

               

               

              Best regards

              Jakob

                • Actions