I'm trying to stress test a cluster of HotRod servers, and I find that under a load the servers are failing requests from the clients with a TimeoutException:

2011-02-16 13:54:21,927 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (OOB-1,foo,node526-61879) Execution error:

org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [10 seconds] on key [ByteArrayKey{data=ByteArray{size=7, hashCode=6bf7dab1, array=[2, 62, 4, 52, 55, 51, 54, ..]}}] for requestor [Thread[OOB-1,foo,node526-61879,5,Thread Pools]]! Lock held by [(another thread)]

(see the full exception stack trace in the attached infinispan.log)

My test clients contain a loop of put/get operations that looks like this:

      for (int c = 0 ; c < numOps ; ++c)

      {

        int key = random.nextInt(numKeys) ;

        String keyStr = Integer.toString(key) ;

        String value = cache.get(keyStr) ;

        if (value == null)

        {

          cache.put(keyStr, data) ;

        }

      }

To paraphrase this: "For numOps, select a random key out of numKeys and attempt to get it's value from the cache. If it's not there, put a value for this key."

I have two servers  configured with the default cache in distribution mode (see the attached infinispan.xml and my-jgroups.xml) In my test, I run just two clients, each with numOps = 1000, numKeys = 10,000. The size of the cached data string is always 1 byte.

Any suggestions to resolve this issue would be appreciated.

Message was edited by: Steven Levitt Followup: I recreated the problem while running all servers and clients locally on my development machine. So, it would appear that the problem lies either in my configuration or in the Infinispan code.