6 Replies Latest reply on Jul 24, 2007 6:52 PM by konkimalla

Problem with removing a Node from Cluster - HAJNDI dies

redchili Mar 22, 2007 4:53 AM

Hello,

I have a problem when I remove a node from a cluster. The cluster consists of 2 or more identical configured nodes on which the application is deployed via farming.
Adding nodes to the cluster works fine. As soon as the deployment is completed the JNDIView shows all registered beans and proxies:

HA-JNDI Namespace
 +- QueueConnectionFactory
 +- XAConnectionFactory
 +- HTTPXAConnectionFactory
 +- queue
 | +- D
 | +- DLQ
 | +- C
 | +- ex
 | +- B
 | +- A
 | +- testQueue
 +- HTTPConnectionFactory
 +- UIL2XAConnectionFactory[link -> XAConnectionFactory]
 +- kusssdemo
 | +- TermPeriodEM
 | | +- local (proxy: $Proxy1060 implements interface at.jku......)
 | +- StudyCodeBusinessLogicBean
 | | +- remote (proxy: $Proxy934 implements interface at.jku.......)
 | +- StudyMajorFieldBusinessLogicBean

Now when I shutdown a node in the cluster the proxies are removed from all nodes and not restarted again (thus leaving any application inoperable):

HA-JNDI Namespace
 +- HTTPXAConnectionFactory
 +- XAConnectionFactory
 +- QueueConnectionFactory
 +- queue
 | +- D
 | +- C
 | +- DLQ
 | +- B
 | +- ex
 | +- A
 | +- testQueue
 +- HTTPConnectionFactory
 +- UIL2XAConnectionFactory[link -> XAConnectionFactory]
 +- kusssdemo
 | +- TermPeriodEM
 | +- StudyMajorFieldBusinessLogicBean
 | +- StudyCodeBusinessLogicBean

The only related suspect thing I can see in the log of another (still alive) node is the "was NOT removed !!!" message:

INFO [org.jboss.ha.framework.interfaces.HAPartition.DefaultPartition] New cluster view for partition DefaultPartition: 13 ([192.168.1.104:1099, 192.168.1.106:1099] delta: -1)
 DEBUG [org.jboss.ha.framework.interfaces.HAPartition.DefaultPartition] dead members: [192.168.1.105:1099]
 DEBUG [org.jboss.ha.framework.interfaces.HAPartition.DefaultPartition] membership changed from 2 to 2
 DEBUG [org.jgroups.protocols.pbcast.NAKACK] removing 192.168.1.105:7800 from received_msgs (not member anymore)
 DEBUG [org.jboss.ha.framework.interfaces.HAPartition.DefaultPartition] Begin notifyListeners, viewID: 13
 DEBUG [org.jgroups.protocols.FD_SOCK] VIEW_CHANGE received: [192.168.1.104:7800, 192.168.1.106:7800]
 INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.DefaultPartition] I am (192.168.1.106:1099) received membershipChanged event:
 INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.DefaultPartition] Dead members: 1 ([192.168.1.105:1099])
 INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.DefaultPartition] New Members : 0 ([])
 INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.DefaultPartition] All Members : 2 ([192.168.1.104:1099, 192.168.1.106:1099])
 DEBUG [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.DefaultPartition] purgeDeadMembers, [192.168.1.105:1099]
 DEBUG [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.DefaultPartition] trying to remove deadMember 192.168.1.105:1099 for key DCacheBridge-DefaultJGBridge
 DEBUG [org.jgroups.protocols.FD] suspected_mbrs: [], after adjustment: []
 DEBUG [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.DefaultPartition] 192.168.1.105:1099 was NOT removed!!!
 DEBUG [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.DefaultPartition] trying to remove deadMember 192.168.1.105:1099 for key jboss.ha:service=HASingletonDeployer
 DEBUG [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.DefaultPartition] 192.168.1.105:1099 was NOT removed!!!
 DEBUG [org.jgroups.protocols.FD_SOCK] determinePingDest()=192.168.1.104:7800, pingable_mbrs=[192.168.1.104:7800, 192.168.1.106:7800]
 DEBUG [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.DefaultPartition] trying to remove deadMember 192.168.1.105:1099 for key HAJNDI
 DEBUG [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.DefaultPartition] 192.168.1.105:1099 was NOT removed!!!
 DEBUG [org.jboss.ha.framework.interfaces.HAPartition.DefaultPartition] End notifyListeners, viewID: 13

As soon as I join any other node to the cluster again, everything continous to work fine.
Does anybody have an idea what might be wrong here?

1. Re: Problem with removing a Node from Cluster - HAJNDI dies

brian.stansberry Mar 22, 2007 11:07 AM (in response to redchili)

These things shouldn't be bound in the HA-JNDI Namespace; they should be in the Global JNDI Namespace.

Did you alter server/all/conf/jndi.properties? If you did, don't.
Actions
2. Re: Problem with removing a Node from Cluster - HAJNDI dies

redchili Mar 22, 2007 11:24 AM (in response to redchili)

I didnt change the jndi.properties but I guess that bites me then: http://wiki.jboss.org/wiki/Wiki.jsp?page=JBossHAJNDIUseCluster

There is a jndi.properties somewhere in the deployed application (which I don't know yet)
Actions
3. Re: Problem with removing a Node from Cluster - HAJNDI dies

brian.stansberry Mar 22, 2007 11:45 AM (in response to redchili)

Kudos to you for posting the link and thus helping others; I forgot about that page that explains the problem better. :-)
Actions
4. Re: Problem with removing a Node from Cluster - HAJNDI dies

konkimalla Jul 24, 2007 11:00 AM (in response to redchili)

After reading the clustering document, I am assuming that ejbs need to be registered in HA-JNDI tree instead of JNDI. I thought @Clustered will take care of this, but I could not find any ejbs in JNDIVIEW of jmxconsole.

Can someone please let me know the procedure to bind ejbs in HA-JNDI tree?
Actions
5. Re: Problem with removing a Node from Cluster - HAJNDI dies

brian.stansberry Jul 24, 2007 2:02 PM (in response to redchili)

EJBs are bound in the local JNDI tree, not the HA-JNDI tree.

If your client connects to HA-JNDI, the HA-JNDI server can find things bound in local JNDI on any node in the cluster.
Actions
6. Re: Problem with removing a Node from Cluster - HAJNDI dies

konkimalla Jul 24, 2007 6:52 PM (in response to redchili)

Now I understand the logic. However, I am facing a problem in clustered environment having two nodes when using the default cluster-service.xml file. I am able to successfully make two nodes in cluster.

I am running a client application that invokes a stateful bean and everything is working fine when I use the following code:

props.put("java.naming.factory.initial", "org.jnp.interfaces.NamingContextFactory");
props.put("java.naming.provider.url", "localhost:1100");
props.put("java.naming.factory.url.pkgs", "org.jboss.naming:org.jnp.interfaces");

My test is to fail over the node so that the other node will serve the client. so, as per the document I changed url to

props.put("java.naming.provider.url", "<currentnodeip1>:1100,<another nodeip2>:1100"); But, after shutting down the server the client is also dying.

The client could not even connect to the A/S when I changed the url to
props.put("java.naming.provider.url", "<currentnodeip1>:1100");

I am very confused as I am using the default configruation xml file. Please help me out.
Actions

Go to original post