6 Replies Latest reply on Jun 13, 2012 2:59 PM by dex80526

    locking timeout: Unable to aquire a lock on key

    dex80526

      With Infinispan 5.1.4.Final, I have a cache with contains 500,000 entries.

      The lokcing config like this:

        <locking

               isolationLevel="READ_COMMITTED"

               lockAcquisitionTimeout="15000"

               writeSkewCheck="false"

               concurrencyLevel="500"

               useLockStriping="false"

            />

       

       

       

      ERROR org.infinispan.interceptors.InvocationContextInterceptor[146] - ISPN000136: Execution error

      org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [1.5 seconds] on key [5e99e7148b7e37de0e73faab26f354e73836414a] for requestor [DldGlobalTransaction{coinToss=2655070111224623118, isMarkedForRollback=false, lockIntention=5e99e7148b7e37de0e73faab26f354e73836414a, affectedKeys=[], locksAtOrigin=[]} GlobalTransaction:<null>:692754:local]! Lock held by [null]

       

      What is the problem?

      How do I fix this?

      Thanks.

        • 1. Re: locking timeout: Unable to aquire a lock on key
          darrellburgan

          Also an issue for us using 5.0.1.FINAL. Our lock timeout is more like 25 seconds though.

           

          Our response is to retry the cache action after a short sleep period. We're finding that the system can handle a low volume of TimeoutExceptions but the cache performance degrades quickly as more and more of them occur. Eventually a critical mass is reached and the entire cache becomes dysfunctional.

           

          Interestingly, sometimes when this snowball effect occurs on a particular cache, it picks on only one node in the cluster. Other times the whole cluster starts having trouble for that cache. It's very mysterious to me why it would not always be a cluster-wide problem.

          • 2. Re: locking timeout: Unable to aquire a lock on key
            dex80526

            try to increase the concurrencyLevel in the locking element.

             

            It seems to make difference in my case.

            • 3. Re: locking timeout: Unable to aquire a lock on key
              darrellburgan

              dex chen wrote:

               

              try to increase the concurrencyLevel in the locking element.

               

              It seems to make difference in my case.

               

              We're currently at concurrencyLevel 256. Is this 'low'? Seems like it's already pretty high. I can certainly push it higher ....

              • 4. Re: locking timeout: Unable to aquire a lock on key
                dex80526

                It is low for us. I have half million entries, and the concurrencyLevel is 10 times bigger than what you have here. I also bump up lockAcquisitionTimeout.

                 

                I assume you set useLockStriping="false" already,

                • 5. Re: locking timeout: Unable to aquire a lock on key
                  darrellburgan

                  dex chen wrote:

                   

                  It is low for us. I have half million entries, and the concurrencyLevel is 10 times bigger than what you have here. I also bump up lockAcquisitionTimeout.

                   

                  I assume you set useLockStriping="false" already,

                   

                  Yeah I have lock striping turned off already. I will kick our concurrencyLevel up to 2048 and see if that makes an appreciable difference. Our lockAcquisitionTimeout is already 25000 millis, which seems pretty long. What is yours?

                  • 6. Re: locking timeout: Unable to aquire a lock on key
                    dex80526

                    yea, your number is similar to mine.  Let me know if the concurrncylevel makes difference for you or not.