2 Replies Latest reply on Jun 13, 2013 11:59 AM by cdman

    How do I (partially) synchronize a client with a set of HotRod servers?

    cdman

      Hello.

       

      If I have a set of HotRod servers in DIST mode, what is the best way to continiously synchronize a client with a subset of data? To make the question more concrete, consider the following example:

       

      • A set of HotRod servers is keeping documents for N clients
      • Each client connects to the servers and - while it is connected - it would like to get notifications as fast as possible about new/changed documents

      The only option which I see currently is using a continious query (2). Are there other (better) options? Is it possible to use a local cache + cache loader to speed up the synchronization between the servers and the client after it has been offline for a while? (something like: store the currrent state to disk and when it becomes online again, it should synchronize only the changed documents)

       

      PS. This is part of a bigger question (1) which I've broken up so that it is easier to "digest". Thank you in advance for your help.

       

      (1) http://community.jboss.org/thread/167245

      (2) http://community.jboss.org/wiki/ContinuousQueryWithInfinispan

        • 1. Re: How do I (partially) synchronize a client with a set of HotRod servers?
          mircea.markus

          Will solve that for you ISPN-374, but before that you can also:

          - have a HR server on top of a local cache on the client. The local cache can also be used as an L1 cache

          - have a listener on the server side that can connect to the HR server on the client

          - on changes the listener notifies the client

           

          Another approach is through JMS.

           

          Or the nicest is ISPN-374, if you feel like contributing would be great!

          • 2. Re: How do I (partially) synchronize a client with a set of HotRod servers?
            cdman

            Hello,

             

            Thank you for the feedback. It has been a long time coming, but finally I put together a concrete code example using some of techniques from above. You can read the article about it here: http://www.todaysoftmag.com/article/en/12/High_availability_performance_systems_using_data_grids_in_Java_437 and get the code here: https://github.com/cdman/infinispan-exchange

             

            I would also like to thank Dan Berindei for his help.

             

            Some updates:

            • hotrod is not used because watching for elements would have needed implementing an interceptor (I understand from Dan that listeners are called before the element is actually committed and using them would have created the risk of double-processing). Instead I just created a REST endpoint
            • also, hotrod doesn't support continious query, so there is no good solution for the output
            • an other issue with HotRod (in my opinion) is that implementation of non-java clients seems to have stalled a little bit
            • the transaction + failover support seems to have some bugs which Dan is (has?) ironed out

             

            Unfortunately the performance I got out of the system (~1000 TPS) is too low for the kind of solution I was looking for (feel free to look at the code and suggest improvements, but the matching engine itself supports ~400k operations on the same system and a test with Radargun confirmed that this value is in the range of values to be expected).

             

            Thank you,

            Attila