Static cluster stops recieving messages
alexc099 Nov 23, 2011 2:33 PMWhile I'm fighting with the network people over UDP filtering I figured I should set up a static cluster so I can keep testing. My problem is that my cluster will receive messages for a while but will then just stop a couple of thousand messages in. I'm not seeing any error logging from HornetQ, my producer just starts waiting and my consumer gets nothing. Note that this only happens when I have static discovery set up. Broadcast discovery works just fine, other than the fact that my producer and consumer have to live on the same box because of my network issues.
I've been looking at the clustered-static-discovery example config and for the life of me I don't see anything out of the ordinary.
This is the config for server0:
<configuration xmlns="urn:hornetq"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:hornetq /schema/hornetq-configuration.xsd">
<clustered>true</clustered>
<shared-store>true</shared-store>
<persistence-enabled>true</persistence-enabled>
<failover-on-shutdown>true</failover-on-shutdown>
<allow-failback>true</allow-failback>
<jmx-management-enabled>true</jmx-management-enabled>
<journal-directory>/shared/hq/data/journal</journal-directory>
<paging-directory>/shared/hq/data/paging</paging-directory>
<bindings-directory>/shared/hq/data/bindings</bindings-directory>
<journal-min-files>10</journal-min-files>
<large-messages-directory>/shared/hq/data/large-messages</large-messages-directory>
<connectors>
<connector name="netty">
<factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
<param key="host" value="${hornetq.remoting.netty.host:qa29-vm}"/>
<param key="port" value="${hornetq.remoting.netty.port:5445}"/>
</connector>
<connector name="server1-connector">
<factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
<param key="host" value="qa30-vm"/>
<param key="port" value="5446"/>
</connector>
</connectors>
<acceptors>
<acceptor name="netty">
<factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>
<param key="host" value="${hornetq.remoting.netty.host:qa29-vm}"/>
<param key="port" value="${hornetq.remoting.netty.port:5445}"/>
</acceptor>
</acceptors>
<cluster-connections>
<cluster-connection name="my-cluster">
<address>jms</address>
<connector-ref>netty</connector-ref>
<retry-interval>500</retry-interval>
<use-duplicate-detection>true</use-duplicate-detection>
<forward-when-no-consumers>true</forward-when-no-consumers>
<max-hops>1</max-hops>
<static-connectors>
<connector-ref>server1-connector</connector-ref>
</static-connectors>
</cluster-connection>
</cluster-connections>
<security-settings>
<security-setting match="#">
<permission type="createNonDurableQueue" roles="guest"/>
<permission type="deleteNonDurableQueue" roles="guest"/>
<permission type="consume" roles="guest"/>
<permission type="send" roles="guest"/>
<permission type="createDurableQueue" roles="guest"/>
</security-setting>
</security-settings>
<address-settings>
<!--default for catch all-->
<address-setting match="#">
<dead-letter-address>jms.queue.DLQ</dead-letter-address>
<expiry-address>jms.queue.ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<max-size-bytes>10485760</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>BLOCK</address-full-policy>
</address-setting>
</address-settings>
</configuration>
Server1 has the same settings save host, port, a connector that points to server0, its corresponding reference in the cluster-connection and that its directories point to /shared/hq/data-2. Server3 and server4 (0 and 1's HA backup) have the <backup>true</backup> setting and I've tried them both with and without the cluster-connecton settings and even just not starting them none of that make a difference. My hornetq-jms.xml has this as a connection factory:
<connection-factory name="NettyConnectionFactory">
<xa>false</xa>
<ha>true</ha>
<connectors>
<connector-ref connector-name="netty"/>
</connectors>
<entries>
<entry name="/ConnectionFactory"/>
</entries>
<client-failure-check-period>60000</client-failure-check-period>
<!-- Pause 1 second between connect attempts -->
<retry-interval>1000</retry-interval>
<!-- Multiply subsequent reconnect pauses by this multiplier. This can be used to
implement an exponential back-off. For our purposes we just set to 1.0 so each reconnect
pause is the same length -->
<retry-interval-multiplier>1.0</retry-interval-multiplier>
<!-- Try reconnecting an unlimited number of times (-1 means "unlimited") -->
<reconnect-attempts>-1</reconnect-attempts>
</connection-factory>
My consumer uses Spring to get its connection factory with this config:
<util:constant id="QUEUE_CF"
static-field="org.hornetq.api.jms.JMSFactoryType.QUEUE_CF" />
<bean name="transportConfiguration"
class="org.hornetq.api.core.TransportConfiguration">
<constructor-arg
value="org.hornetq.core.remoting.impl.netty.NettyConnectorFactory" />
<constructor-arg>
<map key-type="java.lang.String" value-type="java.lang.Object">
<entry key="host" value="qa29-vm" />
<entry key="port" value="5445" />
</map>
</constructor-arg>
</bean>
<bean name="jmsFactory" class="org.hornetq.api.jms.HornetQJMSClient"
factory-method="createConnectionFactoryWithHA">
<constructor-arg index="0" ref="QUEUE_CF" />
<constructor-arg index="1" ref="transportConfiguration" />
<property name="reconnectAttempts" value="5"/>
<property name="connectionTTL" value="60000"/>
<property name="clientFailureCheckPeriod" value="30000"/>
</bean>
Edit: I also tried getting the connection factory through JNDI like so, but I'm still seeing the same issue. Connects, producer sends some messages then it all hangs.
<bean id="jndiTemplate" class="org.springframework.jndi.JndiTemplate">
<property name="environment">
<props>
<prop key="java.naming.factory.initial">org.jnp.interfaces.NamingContextFactory</prop>
<prop key="java.naming.provider.url">jnp://qa29-vm:1099</prop>
</props>
</property>
</bean>
<bean id="jmsFactory" class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiTemplate" ref="jndiTemplate"/>
<property name="jndiName" value="ConnectionFactory"/>
</bean>
Finally, my producer (JMeter) uses JNDI to get ConnectionFactory from jnp://qa29-vm:1099 using the org.jnp.interfaces.NamingContextFactory.
Like I said, the only difference between my working and non-working setup is the cluster discovery mode. Anyone got any ideas on what I messed up on?