HornetQ 2.2.13 - HA not working on MDB client topic - IllegalStateException: Cannot create session factory, server locator is closed (maybe it has been garbage collected)
ndipiazza Sep 18, 2012 2:02 PMI have a situation where HA failover fails to take place. Related to this issue: https://community.jboss.org/thread/204023
We have a JBoss 7.1.1.Final JMS Server cluster with two nodes (port 5445, and port 5545).
We have a third JBoss 7.1.1.Final JMS Client that connects to the HornetQ server cluster.
We are connecting with an MDB that has the two node's hard coded in the MDB annotations. It is a durable topic. Discovery is not enabled.
Here is the MDB:
package test.mdb; import javax.ejb.EJBException; import javax.ejb.MessageDrivenBean; import javax.ejb.MessageDrivenContext; import javax.jms.Message; import javax.jms.MessageListener; import javax.naming.Context; import javax.naming.InitialContext; import javax.naming.NamingException; import javax.ejb.ActivationConfigProperty; import javax.ejb.MessageDriven; @MessageDriven(name = "MDB_CMT_TxRequiredExample", activationConfig = { @ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Topic"), @ActivationConfigProperty(propertyName = "destination", propertyValue = "ndipiazza.pl.testing"), @ActivationConfigProperty(propertyName = "setupAttempts", propertyValue = "-1"), @ActivationConfigProperty(propertyName = "hA", propertyValue = "true"), @ActivationConfigProperty(propertyName = "clientID", propertyValue = "ndipiazza.pl.testing"), @ActivationConfigProperty(propertyName = "subscriptionName", propertyValue = "ndipiazza.pl.testing"), @ActivationConfigProperty(propertyName = "connectorClassName", propertyValue = "org.hornetq.core.remoting.impl.netty.NettyConnectorFactory,org.hornetq.core.remoting.impl.netty.NettyConnectorFactory"), @ActivationConfigProperty(propertyName = "connectionParameters", propertyValue = "host=127.0.0.1;port=5445,host=127.0.0.1;port=5545"), @ActivationConfigProperty(propertyName = "subscriptionDurability", propertyValue ="Durable")}) public class TestMDB implements MessageDrivenBean, MessageListener { MessageDrivenContext mContext = null; Context jndiContext = null; int count = 0; private static final long serialVersionUID = -8023536477112837766L; public void onMessage(Message message) { System.out.println("\n ************************* In onMessage of TestMDB : " + (++count)); } public void ejbRemove() throws EJBException { mContext = null; } public void setMessageDrivenContext(MessageDrivenContext messageDrivenContext) throws EJBException { mContext = messageDrivenContext; try { jndiContext = new InitialContext(); } catch (NamingException ne) { throw new EJBException(ne); } } }
So the MDB listens on both 127.0.0.1:5445 and 127.0.0.1:5545
We start up the cluster and the client then start sending the cluster some messages. You at that point correctly see the messages being consumed from the client from the first JMS server listed (127.0.0.1:5445).
Now we stop the node 127.0.0.1:5445, and instead of failing over to 127.0.0.1:5545, you get this:
Thread-29 (HornetQ-client-global-threads-7213879)) Notified of connection failure in xa recovery connectionFactory for provider ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl [initialConnectors=[org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=127-0-0-1, org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5545&host=127-0-0-1], discoveryGroupConfiguration=null], connectorConfig=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5545&host=localhost, backupConfig=null] will attempt reconnect on next pass: HornetQException[errorCode=4 message=The connection was disconnected because of server shutdown] at org.hornetq.core.client.impl.ClientSessionFactoryImpl$Channel0Handler$1.run(ClientSessionFactoryImpl.java:1467) [hornetq-core-2.2.13.Final.jar:] at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [hornetq-core-2.2.13.Final.jar:] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) [rt.jar:1.6.0_24] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [rt.jar:1.6.0_24] at java.lang.Thread.run(Unknown Source) [rt.jar:1.6.0_24]
This error message says: "will attempt reconnect on next pass" but the failover never occurs and the message processing will not resume until the client server redeploys the WAR file with the MDB.
Later, another error appears:
17:31:14,354 WARN [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016027: Local XARecoveryModule.xaRecovery got XA exception XAException.XAER_RMERR: javax.transaction.xa.XAException: Error trying to connect to any providers for xa recovery at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.getDelegate(HornetQXAResourceWrapper.java:275) [hornetq-jms-2.2.13.Final.jar:] at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.recover(HornetQXAResourceWrapper.java:77) [hornetq-jms-2.2.13.Final.jar:] at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecovery(XARecoveryModule.java:503) [jbossjts-4.16.2.Final.jar:] at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.resourceInitiatedRecoveryForRecoveryHelpers(XARecoveryModule.java:471) [jbossjts-4.16.2.Final.jar:] at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.bottomUpRecovery(XARecoveryModule.java:385) [jbossjts-4.16.2.Final.jar:] at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:166) [jbossjts-4.16.2.Final.jar:] at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:789) [jbossjts-4.16.2.Final.jar:] at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371) [jbossjts-4.16.2.Final.jar:] Caused by: java.lang.IllegalStateException: Cannot create session factory, server locator is closed (maybe it has been garbage collected) at org.hornetq.core.client.impl.ServerLocatorImpl.assertOpen(ServerLocatorImpl.java:1823) [hornetq-core-2.2.13.Final.jar:] at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:699) [hornetq-core-2.2.13.Final.jar:] at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connect(HornetQXAResourceWrapper.java:321) [hornetq-jms-2.2.13.Final.jar:] at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.getDelegate(HornetQXAResourceWrapper.java:251) [hornetq-jms-2.2.13.Final.jar:] ... 7 more
Even when i turn back on the node i shut down... it still won't start processing again. So not only does it not failover, it won't even reconnect to the existing node after it returns.
This error message eventually repeats.
So it looks like this is the next pass it was talking about. The error complains about this: Cannot create session factory, server locator is closed (maybe it has been garbage collected)
When I debug that line of code, The state is
ServerLocatorImpl$STATE (id=327) --> name "CLOSED" (id=331) , ordinal 1
Any ideas why this might happen?
-
hornetq-issue-files.zip 18.6 KB