-
1. Re: Load-balance the invocation of a MBean in the cluster fr
brian.stansberry Dec 9, 2008 4:33 PM (in response to cdelory)Thanks for the bug report. I think this is an issue that's been around for a long time; it has to do with the way the HA proxy factories work with the invokers. I'm looking now to see if there is a straightforward fix.
In the meantime, I'm going to post below the code from a test service that we use in the testsuite to test load balanced invocations to an mbean service. This "HAService" class overrides HAServiceMBeanSupport's startService() method and also adds an invoke(Invocation invocation) method. Those are the key bits; as a workaround perhaps you can try incorporating the same logic in your service. -
2. Re: Load-balance the invocation of a MBean in the cluster fr
brian.stansberry Dec 9, 2008 4:36 PM (in response to cdelory)First, the service implementation:
package org.jboss.test.cluster.invokerha; import java.lang.reflect.Method; import java.lang.reflect.InvocationTargetException; import java.lang.reflect.UndeclaredThrowableException; import java.security.Principal; import java.util.Collections; import java.util.HashMap; import java.util.Map; import org.jboss.ha.jmx.HAServiceMBeanSupport; import org.jboss.invocation.Invocation; import org.jboss.invocation.MarshalledInvocation; import org.jboss.logging.Logger; import org.jboss.security.SecurityAssociation; import org.jboss.system.Registry; public class HAService extends HAServiceMBeanSupport implements HAServiceRemote, HAServiceMBean { private static final Logger log = Logger.getLogger(HAService.class); private int count = 0; private Map marshalledInvocationMapping; public void startService() throws Exception { super.startService(); // Calulate method hashes for remote invocation Method[] methods = HAServiceRemote.class.getMethods(); HashMap tmpMap = new HashMap(methods.length); for(int m = 0; m < methods.length; m ++) { Method method = methods[m]; Long hash = new Long(MarshalledInvocation.calculateHash(method)); tmpMap.put(hash, method); } marshalledInvocationMapping = Collections.unmodifiableMap(tmpMap); // Place our ObjectName hash into the Registry so invokers can resolve it Registry.bind(new Integer(serviceName.hashCode()), serviceName); } public void stopService() throws Exception { super.stopService(); // No longer available to the invokers Registry.unbind(new Integer(serviceName.hashCode())); } /** * Expose the client mapping */ public Map getMethodMap() { return marshalledInvocationMapping; } /** * This is the "remote" entry point */ public Object invoke(Invocation invocation) throws Exception { log.info("Invoked"); // Invoked remotely, inject method resolution if (invocation instanceof MarshalledInvocation) { MarshalledInvocation mi = (MarshalledInvocation) invocation; mi.setMethodMap(marshalledInvocationMapping); } Method method = invocation.getMethod(); Object[] args = invocation.getArguments(); log.info("Invocation of " + method); // Setup any security context (only useful if something checks it, this impl doesn't) Principal principal = invocation.getPrincipal(); Object credential = invocation.getCredential(); SecurityAssociation.setPrincipal(principal); SecurityAssociation.setCredential(credential); // Dispatch the invocation try { return method.invoke(this, args); } catch(InvocationTargetException e) { Throwable t = e.getTargetException(); if( t instanceof Exception ) throw (Exception) t; else throw new UndeclaredThrowableException(t, method.toString()); } finally { // Clear the security context SecurityAssociation.clear(); } } // Implementation of remote methods public String hello() { log.info("hello() invoked"); return "Hello"; } public String getClusterNode() { log.info("getClusterNode() invoked"); return getHAPartition().getNodeName(); } }
-
3. Re: Load-balance the invocation of a MBean in the cluster fr
brian.stansberry Dec 9, 2008 4:41 PM (in response to cdelory)Now the service's remote interface (not really relevant, just FYI) and its mbean interface:
package org.jboss.test.cluster.invokerha; import java.io.IOException; public interface HAServiceRemote { String hello() throws IOException; String getClusterNode(); }
package org.jboss.test.jmx.ha; import java.util.Map; import org.jboss.invocation.Invocation; public interface HAServiceMBean extends org.jboss.ha.jmx.HAServiceMBean { // For remoting Map getMethodMap(); Object invoke(Invocation invocation) throws Exception; boolean getSendRemoteLifecycleNotifications(); void setSendRemoteLifecycleNotifications(boolean send); }
-
4. Re: Load-balance the invocation of a MBean in the cluster fr
brian.stansberry Dec 9, 2008 4:42 PM (in response to cdelory)Finally, the deployment descriptor:
<?xml version="1.0" encoding="UTF-8"?> <server> <!-- Create JRMPHA proxy for the service that sends notifications --> <mbean code="org.jboss.proxy.generic.ProxyFactoryHA" name="jboss.test:service=ProxyFactory,name=HAService,protocol=jrmpha"> <!-- Use the default partition --> <depends optional-attribute-name="PartitionObjectName">jboss:service=DefaultPartition</depends> <!-- Use the standard JRMPInvoker from conf/jboss-service.xml --> <depends optional-attribute-name="InvokerName">jboss:service=invoker,type=jrmpha</depends> <!-- The load balancing policy --> <attribute name="LoadBalancePolicy">org.jboss.ha.framework.interfaces.RoundRobin</attribute> <!-- The target MBean --> <depends optional-attribute-name="TargetName">jboss.test:service=HAService</depends> <!-- Where to bind the proxy --> <attribute name="JndiName">jmx/HAService</attribute> <!-- The interface exposed to the client --> <attribute name="ExportedInterface">org.jboss.test.jmx.ha.HAServiceRemote</attribute> <!-- Client side behaviour --> <attribute name="ClientInterceptors"> <interceptors> <interceptor>org.jboss.proxy.ClientMethodInterceptor</interceptor> <interceptor>org.jboss.invocation.InvokerInterceptor</interceptor> </interceptors> </attribute> </mbean> <!-- Our service --> <mbean code="org.jboss.test.jmx.ha.HAService" name="jboss.test:service=HAService"> <attribute name="SendRemoteLifecycleNotifications">false</attribute> </mbean> </server>
-
5. Re: Load-balance the invocation of a MBean in the cluster fr
brian.stansberry Dec 9, 2008 6:09 PM (in response to cdelory)Now I bit more about the problem.
The detached invokers , e.g. JRMPInvoker or JRMPInvokerHA work by unmarshalling an Invocation object off the wire, getting the ObjectName of the target service from the invocation, and then making a call on the MBeanServer similar to this:Object returnValue = mbeanServer.invoke(targetObjectName, "invoke", new Object[]{invocation}, new String[] {Invocation.class.getName()});
This approach requires that the service indicated by "targetObjectName" exposes a public void invoke(Invocation invocation) method in its mbean interface. You can see an impl of this method above.
That's a pretty heavy requirement for a service, so for a non-HA service, the "targetObjectName" is actually the ObjectName for the ProxyFactory mbean that created the proxy. The proxy factory actually handles the invoke(Invocation) call and delegates to the real target service.
The problem you are seeing is because ProxyFactoryHA doesn't work that way. It actually requires the end-user service to implement invoke(Invocation) itself. In addition, it forces the end-user service to do the// Place our ObjectName hash into the Registry so invokers can resolve it Registry.bind(new Integer(serviceName.hashCode()), serviceName);
call that you see in the example HAService.startService() method above. If that call isn't made you end up with the "java.lang.IllegalArgumentException: null object name" exception you reported in your bug report.
So, the most obvious fix for JBAS-6284 is to get ProxyFactory HA to work the same as the non-HA factory. Not sure there aren't any lurking problems preventing that; we'll see. -
6. Re: Load-balance the invocation of a MBean in the cluster fr
cdelory Dec 11, 2008 3:55 AM (in response to cdelory)Brian,
I really thank you for your valuable response.
In the mean time, I figured out another workaround: use the "regular" JRMP proxy factory, which registers a local JNDI name on each target server, and go through HA-JNDI to perform the cluster lookup of one of my registered services. The bad side here is that I cannot really load-balance the service invocations: the HA-JNDI lookup always chooses the first valid response in the response list.
So, sooner or later, I'll migrate to your workaround or wait the next release.
I still have the following question, which in fact is not really related to any clustering issue: can I spare this extra MBean declaration (the proxy factory), but still access my remote MBean service through its remote interface (HAServiceRemote in your example)? Or maybe I want to have one's cake and eat it?
Another time, thank you very much, Brian. -
7. Re: Load-balance the invocation of a MBean in the cluster fr
brian.stansberry Dec 11, 2008 12:18 PM (in response to cdelory)"cdelory" wrote:
In the mean time, I figured out another workaround: use the "regular" JRMP proxy factory, which registers a local JNDI name on each target server, and go through HA-JNDI to perform the cluster lookup of one of my registered services. The bad side here is that I cannot really load-balance the service invocations: the HA-JNDI lookup always chooses the first valid response in the response list.
Do you mean this is because the mbean is only deployed on a few servers in your cluster, so most lookups involve a cluster-wide search on the server side, and those cluster wide searches are not returning evenly balanced results? That wouldn't surprise me at all.
If that's not what you mean, hmm, the lookup calls should be load balanced.So, sooner or later, I'll migrate to your workaround or wait the next release.
OK. Note that I marked the issue as resolved for 4.2.4 and 5.0.1, but it could be a real long time before any 4.2.4 comes out; perhaps only the discovery of some critical security issue in 4.2.3 would drive the release of a 4.2.4.I still have the following question, which in fact is not really related to any clustering issue: can I spare this extra MBean declaration (the proxy factory), but still access my remote MBean service through its remote interface (HAServiceRemote in your example)? Or maybe I want to have one's cake and eat it?.
No, to make invocations against the HAServiceRemote type in a remote VM you need an object in that VM that implements the interface and then passes the call back to the server via some transport. That is, you need a proxy. :-)
If you are willing to give up calling against the interface and just make detyped JMX invocations, you can use the AS's JMX Remoting service aka the RMIAdaptor.