14 Replies Latest reply on Apr 17, 2012 4:30 PM by vblagojevic

    How to get execution result from a remote node when using DistributedExecutorService ?

    dudalov

      I use DistributedExecutorService.submitEverywhere on a replicated cache with several nodes.

      The task implements both DistributedCallable and Serializable. BTW, the latter is not documented by Infinispan! The task is simple; return type is a string. It's similar to https://docs.jboss.org/author/display/ISPN/Infinispan+Distributed+Execution+Framework

       

       

      From my tests I see that the task is executed once on each node, which is what I expect. However I can get the results from the local node only. For all remote nodes I'm getting an exception:

       

      java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Invalid response {Sound-24395=null}

              at org.infinispan.distexec.DefaultExecutorService$DistributedRunnableFuture.retrieveResult(DefaultExecutorService.java:611)

              at org.infinispan.distexec.DefaultExecutorService$DistributedRunnableFuture.get(DefaultExecutorService.java:558)

             

      I see that execution framework first creates DistributedExecuteCommand which is “transferred” to remote nodes as SingleRpcCommand (see RpcManagerImpl.invokeRemotely). However SingleRpcCommand doesn’t expect any return values, which causes DefaultResponseGenerator.getResponse returning null.

       

      I’m not sure if I as API user have any control over it. Do I? Can I specify that I want result be available when I create my DistributedCallable?

        • 1. Re: How to get execution result from a remote node when using DistributedExecutorService ?
          vblagojevic

          Hi Dmitry,

           

          Which method of DistributedExecutorService are you calling? You can view examples of DistributedExecutorService usage in our testsuite or in PI appx. example in demos. If you post some of your code I'll gladly help.

           

          Regards,

          Vladimir

          • 2. Re: How to get execution result from a remote node when using DistributedExecutorService ?
            vblagojevic

            Dmitry, serialization requirements are documented here http://docs.jboss.org/infinispan/5.0/apidocs/org/infinispan/distexec/DefaultExecutorService.html

            However, you are right, it should be documented in other places as well.

             

            Regards,

            Vladimir 

            • 3. Re: How to get execution result from a remote node when using DistributedExecutorService ?
              dudalov

              Hi Vladimir,

               

              Sorry, I didn't provide the complete description of the test. For the task I used Callable<String> that simply reported passed set of keys and a local address

                   cache.getCacheManager().getAddress()

              As for DistributedExecutorService then I used submitEverywhere

               

                 DistributedExecutorService service = new DefaultExecutorService(_infinispanCache);

                 if ( null == keys || keys.isEmpty() ) {

                     return service.submitEverywhere(task);

                 }  else {

                     return service.submitEverywhere(task, keys.toArray());

                 }

               

              With this test I wanted to see how I can collect some data from every node where a cache has some objects. Trying different sets of keys I saw that the task was executed on the expected cluster nodes. However I was never able to get the results of the execution on the remote nodes. The results from the submitted tasks were printed as

               

              for (Future<String> task : tasks ) {

              try {

                 String result = task.get();

                 log("  Result: " + result);

                 ...

               

              with remote nodes always failing into the catch block (pls. see the above stack trace)

               

              According to the description of ReplicableCommand.isReturnValueExpected(), it seems to be by design and I wonder if there is any setting that alter the default bahaviour or another way to get the results from all executed tasks including remote nodes as well. Please advise.

               

              Thanks,

              Dmitry

              • 4. Re: How to get execution result from a remote node when using DistributedExecutorService ?
                dudalov

                … and again the task was executed without any problems on the expected set of cluster nodes. The only problem that I had was inability to get the results from the “remote” nodes. I'm not sure if I missed anything.

                • 5. Re: How to get execution result from a remote node when using DistributedExecutorService ?
                  vblagojevic

                  Dmitry, start simple. Make your callable return a String.

                  {code}

                  class SimpleCallable implements Callable<String>, Serializable {

                   

                        @Override

                        public String call() throws Exception {

                           return "a";

                        }

                     }

                  {code}

                   

                  And see what happens. If that works, then we'll delve into your Callable. Otherwise, itmight be something with your setup.

                   

                  Regards,

                  Vladimir

                  • 6. Re: How to get execution result from a remote node when using DistributedExecutorService ?
                    dudalov

                    It's all the same. I modified the Callable to match your example exactly.

                     

                    I think that the problem is in the way the response value is wrapped into an object. The method (the one that reports the exception)   DefaultExecutorService$DistributedRunnableFuture.retrieveResult

                    returns not-null values only for SuccessfulResponse objects. But as I mentioned before DefaultResponseGenerator does it only for commands with isReturnValueExpected==true

                     

                    public class DefaultResponseGenerator implements ResponseGenerator {

                       public Response getResponse(CacheRpcCommand command, Object returnValue) {

                          if (returnValue == null) return null;

                          if (returnValue instanceof EntryVersionsMap || command.isReturnValueExpected()) {

                             return SuccessfulResponse.create(returnValue);

                          } else {

                             return null; // saves on serializing a response!

                          }

                       }

                    }

                     

                    But that doesn't seem to be the case for SingleRpcCommand. Do I get it wrong?

                     

                    Thanks,

                    Dmitry

                    • 7. Re: How to get execution result from a remote node when using DistributedExecutorService ?
                      vblagojevic

                      Dmitry,

                       

                      Please archive up your entire example, along with configuration files used, and post it here on the board and I'll have a look. Also, let me know which version of Infinispan are you using!

                       

                      Regards,

                      Vladimir

                      • 8. Re: How to get execution result from a remote node when using DistributedExecutorService ?
                        dudalov

                        Sure, I have an interactive test application that creates a new new cluster node every time it's started. The application allows to run some commands including EXECUTE. For conveniency all related classes are implemented as inner classes, so it's just one file. JGroups configuration is read from jgroups-udp.xml taken from the Infinispan tutorial.

                         

                        I would be glad to post it, but what exactly menu item I should use? Is it some from under "Thread Actions"?

                         

                        Thanks,

                        Dmitry

                        • 9. Re: How to get execution result from a remote node when using DistributedExecutorService ?
                          vblagojevic

                          Hit reply and then "Use advanced editor" which will allow you to attach file.

                           

                          Cheers,

                          Vladimir

                          • 10. Re: How to get execution result from a remote node when using DistributedExecutorService ?
                            dudalov

                            Attached please find the interactive test that creates a new cluster node every time it's started. The test uses JGroups configuration jgroups-udp.xml taken from Infinispan tutorial to build a cluster.

                            Once started the test also creates a replicated cache with the name "FooLookupCache" and prompts for the command (case insensitive). The see the list of supported commands with the short descriptions please type "help", which is one of the commands.

                             

                            To reproduce the exceptions that I saw:

                             

                              1. Start at least two instances of the test application  (java -cp ".;build/classes;lib/*"  dev.cache.infinispan.SimpleInteractiveTest)

                              2. After the prompt enter on one node:

                                      add  foo1

                                      add  foo2

                                  and on another node:

                                       add  bar1

                               3. After that the cache should contain 3 objects  (verify it with "list" command)

                               4. On either of the nodes run "execute" command. No arguments is required.

                             

                            For "execute" command the test application will submit inner class TestTask with additional parameter containing keys of all existing in the cache objects. For all returned Future<String> it will try to get the result or report the exception. Task during its "execution" logs the message that helps to see how the task was called on a node.

                             

                            Dmitry

                            • 11. Re: How to get execution result from a remote node when using DistributedExecutorService ?
                              dudalov

                              ... forgot to mention that I ran it against infinispan-5.1.2.FINAL and the test expects to find Infinispan modules under "lib" subfolder. To point to another folder please provide different settings in the "-cp" option in the command line when you start the test

                               

                              Dmitry

                              • 12. Re: How to get execution result from a remote node when using DistributedExecutorService ?
                                vblagojevic

                                Hi Dmitry,

                                 

                                The problem is that cache mode is not DIST_SYNC; distributed executors are intended to work in DIST_SYNC mode. However, we have also strived to have them working in REPL_SYNC mode although we have not had tests in place. I suspect this is a regression. I'll fix this one, add the tests for REPL_SYNC execution and in the meantime use DIST_SYNC if possible. Related JIRA is https://issues.jboss.org/browse/ISPN-1984

                                 

                                Thanks a lot for your effort. Keep your feedback coming!

                                 

                                Regards,

                                Vladimir

                                • 13. Re: How to get execution result from a remote node when using DistributedExecutorService ?
                                  dudalov

                                  You are welcome. I'm glad if it helped.

                                   

                                  Vladimir: please feel free to use that interactive test and modify it as you like for your needs. I'm doing evaluation of using Infinispan solution for our system. So I'm also interested in high quality of your product, which btw looks very good.

                                   

                                  Cheers,

                                  Dmitry

                                  • 14. Re: How to get execution result from a remote node when using DistributedExecutorService ?
                                    vblagojevic

                                    Dmitry,

                                     

                                    Thanks a lot. We already have an excellent framework for testing and I have now added tests for REPL_SYNC mode. These changes should be integrated into releases 5.1.4 and 5.2. Glad to hear that your evaluation is going well, keep your feedback coming. It is because of users like you that we are able to achieve such a high quality.

                                     

                                    Regards,

                                    Vladimir

                                    1 of 1 people found this helpful