7 Replies Latest reply on Dec 11, 2008 12:18 PM by brian.stansberry

    Load-balance the invocation of a MBean in the cluster from a

      Hello all

      My use case is the following:
      - in a JBoss cluster, I have a set of service MBeans, but not on all cluster nodes; for example, node A has an instance of the service, node B also, but node C has no such instance.
      - Each node may own at most one service instance.
      - The object name of the service is the same on each node.
      - The service extends the HAServiceMBeanSupport class.
      - A stateless session EJB3, running in the same cluster, needs to invoke a public method on one of these services (randomly chosen or not, this isn't so important); this EJB may of course run on node C.
      - I really would like that this EJB may access the public methods of the remote service through its public interface, and not with the MBeanServer "invoke" pattern: "Object invoke(ObjectName, String, Object[], String[])".

      I try to figure out the best solution to achieve this use case:
      - So far, I investigated the ProxyFactoryHA way (http://docs.jboss.org/jbossas/guides/j2eeguide/r2/en/html/ch2.chapter.html#d0e4902). The bad side is that I encountered a bug (https://jira.jboss.org/jira/browse/JBAS-6284). Otherwise it could fit my needs.
      - Nevertheless, I'm not sure this is the simplest method, or the most efficient; the clustered service that I mentioned earlier will not be the only one in the cluster: I will have numerous groups of such services in the cluster. With the HA proxy factory, I have one additional MBean for every service MBean.

      So, if you think there might be a better or simplier solution, or if you have any other clues, they are welcome.
      Thanks a lot.

        • 1. Re: Load-balance the invocation of a MBean in the cluster fr
          brian.stansberry

          Thanks for the bug report. I think this is an issue that's been around for a long time; it has to do with the way the HA proxy factories work with the invokers. I'm looking now to see if there is a straightforward fix.

          In the meantime, I'm going to post below the code from a test service that we use in the testsuite to test load balanced invocations to an mbean service. This "HAService" class overrides HAServiceMBeanSupport's startService() method and also adds an invoke(Invocation invocation) method. Those are the key bits; as a workaround perhaps you can try incorporating the same logic in your service.

          • 2. Re: Load-balance the invocation of a MBean in the cluster fr
            brian.stansberry

            First, the service implementation:

            package org.jboss.test.cluster.invokerha;
            
            import java.lang.reflect.Method;
            import java.lang.reflect.InvocationTargetException;
            import java.lang.reflect.UndeclaredThrowableException;
            import java.security.Principal;
            import java.util.Collections;
            import java.util.HashMap;
            import java.util.Map;
            
            import org.jboss.ha.jmx.HAServiceMBeanSupport;
            import org.jboss.invocation.Invocation;
            import org.jboss.invocation.MarshalledInvocation;
            import org.jboss.logging.Logger;
            import org.jboss.security.SecurityAssociation;
            import org.jboss.system.Registry;
            
            public class HAService
             extends HAServiceMBeanSupport
             implements HAServiceRemote, HAServiceMBean
            {
             private static final Logger log = Logger.getLogger(HAService.class);
            
             private int count = 0;
            
             private Map marshalledInvocationMapping;
            
             public void startService()
             throws Exception
             {
             super.startService();
            
             // Calulate method hashes for remote invocation
             Method[] methods = HAServiceRemote.class.getMethods();
             HashMap tmpMap = new HashMap(methods.length);
             for(int m = 0; m < methods.length; m ++)
             {
             Method method = methods[m];
             Long hash = new Long(MarshalledInvocation.calculateHash(method));
             tmpMap.put(hash, method);
             }
             marshalledInvocationMapping = Collections.unmodifiableMap(tmpMap);
            
             // Place our ObjectName hash into the Registry so invokers can resolve it
             Registry.bind(new Integer(serviceName.hashCode()), serviceName);
             }
            
             public void stopService()
             throws Exception
             {
             super.stopService();
            
             // No longer available to the invokers
             Registry.unbind(new Integer(serviceName.hashCode()));
             }
            
             /**
             * Expose the client mapping
             */
             public Map getMethodMap()
             {
             return marshalledInvocationMapping;
             }
            
             /**
             * This is the "remote" entry point
             */
             public Object invoke(Invocation invocation)
             throws Exception
             {
             log.info("Invoked");
            
             // Invoked remotely, inject method resolution
             if (invocation instanceof MarshalledInvocation)
             {
             MarshalledInvocation mi = (MarshalledInvocation) invocation;
             mi.setMethodMap(marshalledInvocationMapping);
             }
             Method method = invocation.getMethod();
             Object[] args = invocation.getArguments();
            
             log.info("Invocation of " + method);
            
             // Setup any security context (only useful if something checks it, this impl doesn't)
             Principal principal = invocation.getPrincipal();
             Object credential = invocation.getCredential();
             SecurityAssociation.setPrincipal(principal);
             SecurityAssociation.setCredential(credential);
            
             // Dispatch the invocation
             try
             {
             return method.invoke(this, args);
             }
             catch(InvocationTargetException e)
             {
             Throwable t = e.getTargetException();
             if( t instanceof Exception )
             throw (Exception) t;
             else
             throw new UndeclaredThrowableException(t, method.toString());
             }
             finally
             {
             // Clear the security context
             SecurityAssociation.clear();
             }
             }
            
             // Implementation of remote methods
            
             public String hello()
             {
             log.info("hello() invoked");
             return "Hello";
             }
            
             public String getClusterNode()
             {
             log.info("getClusterNode() invoked");
             return getHAPartition().getNodeName();
             }
            
            }


            • 3. Re: Load-balance the invocation of a MBean in the cluster fr
              brian.stansberry

              Now the service's remote interface (not really relevant, just FYI) and its mbean interface:

              package org.jboss.test.cluster.invokerha;
              
              import java.io.IOException;
              
              public interface HAServiceRemote
              {
               String hello() throws IOException;
              
               String getClusterNode();
              }


              package org.jboss.test.jmx.ha;
              
              import java.util.Map;
              
              import org.jboss.invocation.Invocation;
              
              public interface HAServiceMBean
               extends org.jboss.ha.jmx.HAServiceMBean
              {
               // For remoting
               Map getMethodMap();
               Object invoke(Invocation invocation) throws Exception;
              
               boolean getSendRemoteLifecycleNotifications();
               void setSendRemoteLifecycleNotifications(boolean send);
              }


              • 4. Re: Load-balance the invocation of a MBean in the cluster fr
                brian.stansberry

                Finally, the deployment descriptor:

                <?xml version="1.0" encoding="UTF-8"?>
                
                <server>
                
                 <!-- Create JRMPHA proxy for the service that sends notifications -->
                 <mbean code="org.jboss.proxy.generic.ProxyFactoryHA"
                 name="jboss.test:service=ProxyFactory,name=HAService,protocol=jrmpha">
                
                 <!-- Use the default partition -->
                 <depends optional-attribute-name="PartitionObjectName">jboss:service=DefaultPartition</depends>
                
                 <!-- Use the standard JRMPInvoker from conf/jboss-service.xml -->
                 <depends optional-attribute-name="InvokerName">jboss:service=invoker,type=jrmpha</depends>
                
                 <!-- The load balancing policy -->
                 <attribute name="LoadBalancePolicy">org.jboss.ha.framework.interfaces.RoundRobin</attribute>
                
                 <!-- The target MBean -->
                 <depends optional-attribute-name="TargetName">jboss.test:service=HAService</depends>
                
                 <!-- Where to bind the proxy -->
                 <attribute name="JndiName">jmx/HAService</attribute>
                
                 <!-- The interface exposed to the client -->
                 <attribute name="ExportedInterface">org.jboss.test.jmx.ha.HAServiceRemote</attribute>
                
                 <!-- Client side behaviour -->
                 <attribute name="ClientInterceptors">
                 <interceptors>
                 <interceptor>org.jboss.proxy.ClientMethodInterceptor</interceptor>
                 <interceptor>org.jboss.invocation.InvokerInterceptor</interceptor>
                 </interceptors>
                 </attribute>
                 </mbean>
                
                 <!-- Our service -->
                 <mbean code="org.jboss.test.jmx.ha.HAService"
                 name="jboss.test:service=HAService">
                 <attribute name="SendRemoteLifecycleNotifications">false</attribute>
                 </mbean>
                
                </server>


                • 5. Re: Load-balance the invocation of a MBean in the cluster fr
                  brian.stansberry

                  Now I bit more about the problem.

                  The detached invokers , e.g. JRMPInvoker or JRMPInvokerHA work by unmarshalling an Invocation object off the wire, getting the ObjectName of the target service from the invocation, and then making a call on the MBeanServer similar to this:

                  Object returnValue = mbeanServer.invoke(targetObjectName,
                   "invoke",
                   new Object[]{invocation},
                   new String[] {Invocation.class.getName()});


                  This approach requires that the service indicated by "targetObjectName" exposes a public void invoke(Invocation invocation) method in its mbean interface. You can see an impl of this method above.

                  That's a pretty heavy requirement for a service, so for a non-HA service, the "targetObjectName" is actually the ObjectName for the ProxyFactory mbean that created the proxy. The proxy factory actually handles the invoke(Invocation) call and delegates to the real target service.

                  The problem you are seeing is because ProxyFactoryHA doesn't work that way. It actually requires the end-user service to implement invoke(Invocation) itself. In addition, it forces the end-user service to do the

                   // Place our ObjectName hash into the Registry so invokers can resolve it
                   Registry.bind(new Integer(serviceName.hashCode()), serviceName);
                  


                  call that you see in the example HAService.startService() method above. If that call isn't made you end up with the "java.lang.IllegalArgumentException: null object name" exception you reported in your bug report.

                  So, the most obvious fix for JBAS-6284 is to get ProxyFactory HA to work the same as the non-HA factory. Not sure there aren't any lurking problems preventing that; we'll see.

                  • 6. Re: Load-balance the invocation of a MBean in the cluster fr

                    Brian,

                    I really thank you for your valuable response.
                    In the mean time, I figured out another workaround: use the "regular" JRMP proxy factory, which registers a local JNDI name on each target server, and go through HA-JNDI to perform the cluster lookup of one of my registered services. The bad side here is that I cannot really load-balance the service invocations: the HA-JNDI lookup always chooses the first valid response in the response list.
                    So, sooner or later, I'll migrate to your workaround or wait the next release.

                    I still have the following question, which in fact is not really related to any clustering issue: can I spare this extra MBean declaration (the proxy factory), but still access my remote MBean service through its remote interface (HAServiceRemote in your example)? Or maybe I want to have one's cake and eat it?

                    Another time, thank you very much, Brian.

                    • 7. Re: Load-balance the invocation of a MBean in the cluster fr
                      brian.stansberry

                       

                      "cdelory" wrote:

                      In the mean time, I figured out another workaround: use the "regular" JRMP proxy factory, which registers a local JNDI name on each target server, and go through HA-JNDI to perform the cluster lookup of one of my registered services. The bad side here is that I cannot really load-balance the service invocations: the HA-JNDI lookup always chooses the first valid response in the response list.


                      Do you mean this is because the mbean is only deployed on a few servers in your cluster, so most lookups involve a cluster-wide search on the server side, and those cluster wide searches are not returning evenly balanced results? That wouldn't surprise me at all.

                      If that's not what you mean, hmm, the lookup calls should be load balanced.

                      So, sooner or later, I'll migrate to your workaround or wait the next release.


                      OK. Note that I marked the issue as resolved for 4.2.4 and 5.0.1, but it could be a real long time before any 4.2.4 comes out; perhaps only the discovery of some critical security issue in 4.2.3 would drive the release of a 4.2.4.

                      I still have the following question, which in fact is not really related to any clustering issue: can I spare this extra MBean declaration (the proxy factory), but still access my remote MBean service through its remote interface (HAServiceRemote in your example)? Or maybe I want to have one's cake and eat it?.


                      No, to make invocations against the HAServiceRemote type in a remote VM you need an object in that VM that implements the interface and then passes the call back to the server via some transport. That is, you need a proxy. :-)

                      If you are willing to give up calling against the interface and just make detyped JMX invocations, you can use the AS's JMX Remoting service aka the RMIAdaptor.