4 Replies Latest reply on Jan 14, 2008 10:40 AM by brian.stansberry

    HASingletonElectionPolicy

    smeng1

      I have setup 2 nodes (say Node A and B) in a cluster using Jboss 4.2.2 GA.

      My requirement is:
      1. Node A is set as the preferred master. If both nodes are running, Node A will always become the master i.e. in the event Node B was started before Node A, it will stop the service and Node A will take over running as the master.
      2. If the master node goes down, then the next available node in the cluster becomes the master.

      My problem is:
      When Nodes A & B are running, Node A as master, and Node A is properly shutdown, Node B still detects Node A in the cluster view and continues to elect Node A as the master rather than taking over.

      Here is my HASingletonElectionPolicy implementation:

      package com.unique.ha.singleton;
      
      import org.apache.commons.logging.Log;
      import org.apache.commons.logging.LogFactory;
      import org.jboss.ha.framework.interfaces.ClusterNode;
      import org.jboss.ha.framework.interfaces.HAPartition;
      import org.jboss.ha.singleton.HASingletonElectionPolicySimple;
      
      public class HASingletonElectionPolicyUnique extends HASingletonElectionPolicySimple
      implements HASingletonElectionPolicyUniqueMBean {
      
       private String preferredMasterAddress;
      
       private static final Log log = LogFactory.getLog(HASingletonElectionPolicyUnique.class);
      
       public void setPreferredMasterAddress(String preferredMasterAddress) {
      
       this.preferredMasterAddress = preferredMasterAddress;
       }
      
       public String getPreferredMasterAddress() {
      
       return this.preferredMasterAddress;
       }
      
       public ClusterNode pickSingleton() {
      
       return pickSingleton(getHAPartition());
       }
      
       public ClusterNode pickSingleton(HAPartition partition) {
      
       log.info( "Running HASingletonElectionPolicyUnique..." );
      
       ClusterNode[] nodes = partition.getClusterNodes();
       log.info( "CurrentView for " + partition.getPartitionName() + " is " + partition.getCurrentView().toString() );
      
       if( this.preferredMasterAddress != null ) {
      
       for ( int x=0; x < nodes.length; x++ ) {
      
       ClusterNode node = nodes[x];
      
       if ( node.getIpAddress().getHostAddress().equals( this.preferredMasterAddress ) ) {
      
       log.info("Set Master Status to preferred master address " + this.preferredMasterAddress);
       return node;
       }
       }
       }
      
       log.warn("Unable to select preferred Master at address " +
       this.preferredMasterAddress + ". Will select next available node.");
      
       return super.pickSingleton(partition);
       }
      
      
      }
      


      jboss-service.xml
      <server>
      
       <mbean code="com.unique.ha.singleton.HASingletonElectionPolicyUnique"
       name="com.unique.ha:name=HASingletonElectionPolicyUnique">
       <attribute name="PreferredMasterAddress">${unique.ha.preferred_master_address}</attribute>
       </mbean>
      
       <mbean code="com.unique.system.mbean.MyService"
       name="com.unique.system.mbean:name=MyService">
       </mbean>
      
       <mbean code="org.jboss.ha.singleton.HASingletonController"
       name="com.unique.system.mbean:service=MyService-HASingletonController">
       <depends>jboss:service=${jboss.partition.name}</depends>
       <depends optional-attribute-name="ElectionPolicy" proxy-type="attribute">com.unique.ha:name=HASingletonElectionPolicyUnique</depends>
       <depends optional-attribute-name="TargetName">com.unique.system.mbean:name=MyService</depends>
       <attribute name="PartitionName">${jboss.partition.name}</attribute>
       <attribute name="TargetStartMethod">startSingleton</attribute>
       <attribute name="TargetStopMethod">stopSingleton</attribute>
       </mbean>
      
      </server>


      Thanks
      Meng

        • 1. Re: HASingletonElectionPolicy
          smeng1

          Just to add that I have also tried using HASingletonElectionPolicySimple instead of my own customised policy i.e. HASingletonElectionPolicyUnique

          First MBean in jboss-service.xml replaced with the following:

          <mbean code="org.jboss.ha.singleton.HASingletonElectionPolicySimple"
           name="com.unique.ha:name=HASingletonElectionPolicyUnique">
           <attribute name="Position">0</attribute>
           </mbean>


          Node A was set with Position 0 and Node B was set with Position 1.

          The results were almost the same with the exception that in between bringing up and down the nodes, the two nodes seemed to have got themselves in a confused state on who is the master and at one point, the application was started on both nodes despite the HASingleton implementation.

          The following steps are what I have observed when Node A and B are both running with Node A running as the master. And then I properly shutdown Node A:
          1. Node A begins shutdown.
          2. Node B detects/informed by Node A?? that Node A is undergoing shutdown.
          3. Node B runs the HASingletonElectionPolicy but due to the fact that Node A is still in view, picks Node A as the master.
          4. Node B's view changes and notes that Node A is not within the cluster view anymore.

          After that, nothing happens and the application doesn't get started at all. Note that if an improper termination of Node A occurs, Node B does start the application, which is correct failover behaviour. The problem only occurs for proper shutdown of nodes.

          Is this behaviour correct? Is there a way to manually activate the HASingletonElectionPolicy or some way to get the current state of the other cluster node?

          • 2. Re: HASingletonElectionPolicy
            brian.stansberry

            I suspect this is a side-effect of http://jira.jboss.com/jira/browse/JBAS-4919. When you do a clean shutdown, B is getting a message telling it the *com.unique.system.mbean:service=MyService-HASingletonController service* is undeployed on A, but A is still in the view, since A overall hasn't shut down yet.

            Can you try your HASingletonElectionPolicy, but kill -9 A instead of a clean shutdown and let me know what happens? Obviously that's not a solution, but it can help confirm my thinking.

            • 3. Re: HASingletonElectionPolicy
              smeng1

              After killing A, B correctly removes A from view and then proceeds to start up the singleton and becomes the master. So the failover is successful in this case.

              • 4. Re: HASingletonElectionPolicy
                brian.stansberry

                Thanks for the report. Based on this I've bumped the priority of http://jira.jboss.com/jira/browse/JBAS-4919 to Critical.