11 Replies Latest reply on Jul 4, 2010 11:17 PM by mcleanl

    Cluster, channelBuffer error...

    mcleanl

      I appreciate that this is a bit random however my environment is throwing the following error:

       

       

      WARNING: removing consumer which did not handle a message, consumer=org.hornetq.core.server.cluster.impl.BridgeImpl@d4364b, message=Reference[533]:REL
      IABLE
      java.lang.NullPointerException
              at org.hornetq.core.buffers.impl.ResetLimitWrappedHornetQBuffer.<init>(ResetLimitWrappedHornetQBuffer.java:39)
              at org.hornetq.core.message.impl.MessageImpl.getBodyBuffer(MessageImpl.java:254)
              at org.hornetq.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:220)
              at org.hornetq.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:139)
              at org.hornetq.core.server.cluster.impl.BridgeImpl.handle(BridgeImpl.java:477)
              at org.hornetq.core.server.impl.QueueImpl.handle(QueueImpl.java:1361)
              at org.hornetq.core.server.impl.QueueImpl.deliver(QueueImpl.java:1153)
              at org.hornetq.core.server.impl.QueueImpl.access$700(QueueImpl.java:65)
              at org.hornetq.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:1567)
              at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:96)
              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
              at java.lang.Thread.run(Unknown Source)
      

       

      The outcome is that my cluster becomes unclustered and does not recluster (static setup as no multicast UDP). From the NPE thrown does this mean anything to anyone?

       

      I have looked at the codebase (src) and note that in the class ResetLimitWrappedHornetQBuffer on line 39 is where the ChannelBufferWrapper (i.e. super) is passed a reference to the underlying Netty ChannelBuffer (which in this case appears to be <null>)

       

       

      37 public ResetLimitWrappedHornetQBuffer(final int limit, final HornetQBuffer buffer, final MessageInternal message)
      38    {
      39       super(buffer.channelBuffer());
      

       

      Does anyone know why the netty ChannelBuffer would be null?

       

      Thank you,

      Luke.

        • 1. Re: Cluster, channelBuffer error...
          mcleanl

          Assuming some typr of buffering error so other things I have found....

           

          1) Turning on debugging appears to give no further information on this

           

          2) I changed my code so that UDP/Multicast cluster discovery is enabled instead of the static addresses and this makes no difference.

           

          3) It appears that the cluster is still connected however the consumer is removed (ignored?).

           

          4) Tried setting  consumer-window-size = 0. Same error

           

          5) Tried setting  consumer-window-size = -1. Same error

           

          6) Tried setting tcp-send-buffer-size and tcp-receive-buffer-size = 4915000 (default 32768), Same error

           

          OK, so I admit I have no idea whatsoever whether it is something I'm doing or some bug... but then no one else appears to be hitting this issue. I just don;t know enough about why the channelBuffer would be null....

          • 2. Re: Cluster, channelBuffer error...
            ataylor

            what version are you using?

             

            does this occur in trunk?

             

            can u provide a test case that we can run?

            • 3. Re: Cluster, channelBuffer error...
              timfox

              This looks like something that was fixed a while back. Always take a look in JIRA first, and make sure this happens with the latest version (you didn't say what version you are running).

               

              There's a wiki page on the HornetQ wiki "how to report an issue.

              • 4. Re: Cluster, channelBuffer error...
                mcleanl

                2.1.0.Final. (Mule 2.2.1) on Windows (Server 2003). I'll see what I can do to cobble together a test case.

                • 5. Re: Cluster, channelBuffer error...
                  mcleanl

                  OK, firstly I apologise that this post is going to be quite lengthy but I'll try to kepp it as short as possible. Secondly I apologise because I'm not feeling the 'buzz'... more like a 'bumbling' around in the dark...

                   

                  Including my code was just going to be too much touble for anyone to debug so I took a different tact and luckily found the same issues. Probably something with my understanding but I'd appreciate even a telling off...

                   

                  1) Start from a known good! 2.1.0.Final / Java 1.6.0.b07

                   

                  I ran the example 'clustered-queue'.

                   

                  Output

                   

                       [java] Sent message: This is text message 0
                       [java] Sent message: This is text message 1
                       [java] Sent message: This is text message 2
                       [java] Sent message: This is text message 3
                       [java] Sent message: This is text message 4
                       [java] Sent message: This is text message 5
                       [java] Sent message: This is text message 6
                       [java] Sent message: This is text message 7
                       [java] Sent message: This is text message 8
                       [java] Sent message: This is text message 9
                       [java] Got message: This is text message 0 from node 0
                       [java] Got message: This is text message 1 from node 1
                       [java] Got message: This is text message 2 from node 0
                       [java] Got message: This is text message 3 from node 1
                       [java] Got message: This is text message 4 from node 0
                       [java] Got message: This is text message 5 from node 1
                       [java] Got message: This is text message 6 from node 0
                       [java] Got message: This is text message 7 from node 1
                       [java] Got message: This is text message 8 from node 0
                       [java] Got message: This is text message 9 from node 1
                       [java] example complete
                       [java]
                       [java] #####################
                       [java] ###    SUCCESS!   ###
                       [java] #####################
                  

                   

                  There were a couple of insignificant errors saying Unable to validate user: HORNETQ.CLUSTER.ADMIN.USER

                   

                  Probably caused by not securing the cluster with a valid cluster user and password

                   

                  2) Copied the clustered-queue example and created a clustered-static example

                   

                  Following the manual I firstly edited both the hornetq-jms.xml file to point to both servers (0+1) to point to the two server connectors

                   

                   

                  <configuration xmlns="urn:hornetq"
                               xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                               xsi:schemaLocation="urn:hornetq /schema/hornetq-jms.xsd">
                  
                  <!--the connection factory used by the example-->
                       <connection-factory name="ConnectionFactory">
                            <connectors>
                                 <connector-ref connector-name="netty-connector0"/>
                                 <connector-ref connector-name="netty-connector1"/>
                            </connectors>
                            <entries>
                                 <entry name="ConnectionFactory"/>
                            </entries>
                       </connection-factory>
                  
                  <!--the queue used by the example-->
                       <queue name="exampleQueue">
                            <entry name="/queue/exampleQueue"/>
                       </queue>
                  
                  </configuration>
                  

                   

                  netty-connector0 will point to server0 and netty-connector1 will point to server1 as my cluster.

                   

                  The manual also shows how to wire up these two connectors in both the hornet-configuration.xml files.

                   

                  This is the server1 hornetq-configuration.xml (below) however the only difference between them is that in server0 hornetq-configuration.xml the acceptor port is 5445 rather than 5446

                   

                   

                  <configuration xmlns="urn:hornetq"
                               xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                               xsi:schemaLocation="urn:hornetq 
                  /schema/hornetq-configuration.xsd">
                  <clustered>true</clustered>
                  
                  <!-- Connectors -->
                  <connectors>
                  <connector name="netty-connector1">
                  <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
                  <param key="host" value="localhost"/>
                  <param key="port" value="5446"/>
                  </connector>
                  <connector name="netty-connector0">
                  <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
                  <param key="host" value="localhost"/>
                  <param key="port" value="5445"/>
                  </connector>
                  </connectors>
                  
                  <!-- Acceptors -->
                  <acceptors>
                  <acceptor name="netty-acceptor">
                  <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>
                  <param key="host" value="0.0.0.0"/>
                  <param key="port" value="5446"/>
                  </acceptor>
                  </acceptors>
                  
                  <!-- Clustering configuration -->
                  
                  <cluster-connections>
                  <cluster-connection name="va-cluster">
                  <address>jms</address>
                  <retry-interval>500</retry-interval>
                  <use-duplicate-detection>true</use-duplicate-detection>
                  <forward-when-no-consumers>true</forward-when-no-consumers>
                  <max-hops>1</max-hops>
                  <connector-ref connector-name="netty-connector0" />
                  <connector-ref connector-name="netty-connector1" />
                  </cluster-connection>
                  </cluster-connections>
                  
                  <!-- Other config -->
                  
                  <security-settings>
                  <!--security for example queue-->
                  <security-setting match="jms.queue.exampleQueue">
                  <permission type="createDurableQueue" roles="guest"/>
                  <permission type="deleteDurableQueue" roles="guest"/>
                  <permission type="createNonDurableQueue" roles="guest"/>
                  <permission type="deleteNonDurableQueue" roles="guest"/>
                  <permission type="consume" roles="guest"/>
                  <permission type="send" roles="guest"/>
                  </security-setting>
                  </security-settings>
                  
                  </configuration>
                  

                   

                  I run my new clustered-static example and get the following output

                   

                   

                  Build script found javac.
                  hit alternate
                  Using the following ant version from ..\..\..\tools\ant:
                  calling ..\..\..\tools\ant\bin\ant.bat -version
                  Apache Ant version 1.7.1 compiled on June 27 2008
                  calling ..\..\..\tools\ant\bin\ant.bat  -Dhornetq.run_script=true
                  Buildfile: build.xml
                  
                  run:
                  
                  init:
                       [mkdir] Created dir: 
                  C:\hornetq-2.1.0.Final\examples\jms\clustered-static\build
                       [mkdir] Created dir: 
                  C:\hornetq-2.1.0.Final\examples\jms\clustered-static\build\classes
                  
                  compile:
                        [echo] 
                  src.example.dir=C:\hornetq-2.1.0.Final\examples\jms\clustered-static\src
                       [javac] Compiling 6 source files to 
                  C:\hornetq-2.1.0.Final\examples\jms\clustered-static\build\classes
                  
                  runExample:
                        [java] serverProps = -XX:+UseParallelGC -Xms256M -Xmx256M 
                  -XX:+AggressiveOpts -XX:+UseFastAccessorMethods -Dcom.sun
                  .management.jmxremote 
                  -Djava.util.logging.config.file=C:\hornetq-2.1.0.Final\examples\common/config/logging.properties 
                  -Djava.naming.factory.initial=org.jnp.interfaces.NamingContextFactory 
                  -Djava.naming.factory.url.pkgs=org.jboss.naming:org
                  .jnp.interfaces
                        [java] 28/06/2010 10:01:31 org.hornetq.common.example.HornetQExample run
                        [java] INFO: hornetq.example.runServer is true
                        [java] 28/06/2010 10:01:31 org.hornetq.common.example.HornetQExample startServer
                        [java] INFO: starting server with config 'server0' logServerOutput true
                        [java] HornetQServer_0 err:[main] 10:01:33,046 WARNING [org.hornetq.core.deployers.impl.FileConfigurationParser]  A IO wasn't 
                  located on this platform, it will fall back to using pure Java NIO. If your platform is Linux, install LibAIO to enable the AIO journal
                        [java] HornetQServer_0 err:[main] 10:01:33,109 INFO [org.hornetq.core.server.impl.HornetQServerImpl]  live server is starting..
                        [java] HornetQServer_0 err:[main] 10:01:33,140 INFO [org.hornetq.core.persistence.impl.journal.JournalStorageManager]  Using NIO Journal
                        [java] HornetQServer_0 err:[main] 10:01:33,156 WARNING  [org.hornetq.core.server.impl.HornetQServerImpl]  Security risk! It has 
                  been detected that the cluster admin user and password have not been 
                  changed from the installation default. Please see the HornetQ user 
                  guide, cluster chapter, for instructions on how to do this.
                        [java] HornetQServer_0 err:[Thread-0 (group:HornetQ-server-threads31447311-27173235)] 10:01:34,718 INFO 
                  [org.hornetq.core.server.cluster.impl.BridgeImpl]  Connecting bridge sf.va-cluster.1 to its destination
                        [java] HornetQServer_0 err:[Thread-1 (group:HornetQ-server-threads31447311-27173235)] 10:01:34,750 INFO 
                  [org.hornetq.core.server.cluster.impl.BridgeImpl]  Connecting bridge sf.va-cluster.0 to its destination
                        [java] HornetQServer_0 err:[main] 10:01:34,796 INFO [org.hornetq.core.remoting.impl.netty.NettyAcceptor]  Started Netty 
                  Acceptor version 3.2.0.Final-r2292 0.0.0.0:5445 for CORE protocol
                        [java] HornetQServer_0 err:[main] 10:01:34,796 INFO [org.hornetq.core.server.impl.HornetQServerImpl]  HornetQ Server version 
                  2.1.0.Final (marimbondo, 118) started
                        [java] HornetQServer_0 out:STARTED::
                        [java] 28/06/2010 10:01:34 org.hornetq.common.example.HornetQExample startServer
                        [java] INFO: starting server with config 'server1' logServerOutput true
                        [java] HornetQServer_0 err:[Thread-1 (group:HornetQ-server-threads31447311-27173235)] 10:01:34,859 INFO 
                  [org.hornetq.core.server.cluster.impl.BridgeImpl]  Bridge sf.va-cluster.0 is connected to its destination
                        [java] HornetQServer_1 err:[main] 10:01:35,890 WARNING [org.hornetq.core.deployers.impl.FileConfigurationParser]  AIO wasn't 
                  located on this platform, it will fall back to using pure Java NIO. If your platform is Linux, install LibAIO to enable the AIO journal
                        [java] HornetQServer_1 err:[main] 10:01:35,953 INFO [org.hornetq.core.server.impl.HornetQServerImpl]  live server is starting..
                        [java] HornetQServer_1 err:[main] 10:01:35,984 INFO [org.hornetq.core.persistence.impl.journal.JournalStorageManager]  Using 
                  NIO Journal
                        [java] HornetQServer_1 err:[main] 10:01:36,000 WARNING [org.hornetq.core.server.impl.HornetQServerImpl]  Security risk! It has 
                  been detected that the cluster admin user and password have not been changed from the installation default.
                  Please see the HornetQ user guide, cluster chapter, for instructions on how to do this.
                        [java] HornetQServer_1 err:[Thread-0 (group:HornetQ-server-threads31447311-27173235)] 10:01:37,625 INFO 
                  [org.hornetq.core.server.cluster.impl.BridgeImpl]  Connecting bridge sf.va-cluster.1 to its destination
                        [java] HornetQServer_1 err:[Thread-1 (group:HornetQ-server-threads31447311-27173235)] 10:01:37,640 INFO 
                  [org.hornetq.core.server.cluster.impl.BridgeImpl]  Connecting bridge sf.va-cluster.0 to its destination
                        [java] HornetQServer_1 err:[main] 10:01:37,687 INFO [org.hornetq.core.remoting.impl.netty.NettyAcceptor]  Started Netty 
                  Acceptor version 3.2.0.Final-r2292 0.0.0.0:5446 for CORE protocol
                        [java] HornetQServer_1 err:[main] 10:01:37,703 INFO [org.hornetq.core.server.impl.HornetQServerImpl]  HornetQ Server version 
                  2.1.0.Final (marimbondo, 118) started
                        [java] HornetQServer_1 out:STARTED::
                        [java] 28/06/2010 10:01:37 org.hornetq.common.example.HornetQExample getContext
                        [java] INFO: using server0\client-jndi.properties for jndi
                        [java] HornetQServer_1 err:[Thread-1 (group:HornetQ-server-threads31447311-27173235)] 10:01:37,750 INFO 
                  [org.hornetq.core.server.cluster.impl.BridgeImpl]  Bridge sf.va-cluster.0 is connected to its destination
                        [java] HornetQServer_1 err:[Thread-0 (group:HornetQ-server-threads31447311-27173235)] 10:01:37,765 INFO 
                  [org.hornetq.core.server.cluster.impl.BridgeImpl]  Bridge sf.va-cluster.1 is connected to its destination
                        [java] HornetQServer_0 err:[Thread-0 (group:HornetQ-server-threads31447311-27173235)] 10:01:37,765 INFO 
                  [org.hornetq.core.server.cluster.impl.BridgeImpl]  Bridge sf.va-cluster.1 is connected to its destination
                        [java] 28/06/2010 10:01:37 org.hornetq.common.example.HornetQExample getContext
                        [java] INFO: using server1\client-jndi.properties for jndi
                        [java] HornetQServer_1 err:[Thread-2 (group:HornetQ-client-global-threads-15868406)] 10:01:38,109 SEVERE 
                  [org.hornetq.core.server.cluster.impl.ClusterConnectionImpl]  Failed to handle message
                        [java] HornetQServer_1 err:java.lang.IllegalStateException: Cannot 
                  find binding for jms.queue.exampleQueue8c716cac-8237-11df-a46b-0015c507a7c8
                        [java] HornetQServer_1 err:        at org.hornetq.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.doConsumerCreated(ClusterConnectionImpl.java:774)
                        [java] HornetQServer_1 err:        at org.hornetq.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.onMessage(ClusterConnectionImpl.java:568)
                        [java] HornetQServer_1 err:        at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:823)
                        [java] HornetQServer_1 err:        at org.hornetq.core.client.impl.ClientConsumerImpl.access$100(ClientConsumerImpl.java:46)
                        [java] HornetQServer_1 err:        at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:941)
                        [java] HornetQServer_1 err:        at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:96)
                        [java] HornetQServer_1 err:        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
                        [java] HornetQServer_1 err:        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                        [java] HornetQServer_1 err:        at java.lang.Thread.run(Thread.java:619)
                        [java] HornetQServer_1 err:[Thread-2 (group:HornetQ-client-global-threads-15868406)] 10:01:38,125 SEVERE 
                  [org.hornetq.core.server.cluster.impl.ClusterConnectionImpl]  Failed to handle message
                        [java] HornetQServer_1 err:java.lang.IllegalStateException: Cannot
                  find binding for jms.queue.exampleQueue8c716cac-8237-11df-a46b-0015c507a7c8
                        [java] HornetQServer_1 err:        at org.hornetq.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.doConsumerCreated(ClusterConnectionImpl.java:774)
                        [java] HornetQServer_1 err:        at org.hornetq.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.onMessage(ClusterConnectionImpl.java:568)
                        [java] HornetQServer_1 err:        at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:823)
                        [java] HornetQServer_1 err:        at org.hornetq.core.client.impl.ClientConsumerImpl.access$100(ClientConsumerImpl.java:46)
                        [java] HornetQServer_1 err:        at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:941)
                        [java] HornetQServer_1 err:        at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:96)
                        [java] HornetQServer_1 err:        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
                        [java] HornetQServer_1 err:        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                        [java] HornetQServer_1 err:        at java.lang.Thread.run(Thread.java:619)
                        [java] Sent message: This is text message 0
                        [java] Sent message: This is text message 1
                        [java] Sent message: This is text message 2
                        [java] Sent message: This is text message 3
                        [java] Sent message: This is text message 4
                        [java] Sent message: This is text message 5
                        [java] Sent message: This is text message 6
                        [java] Sent message: This is text message 7
                        [java] Sent message: This is text message 8
                        [java] Sent message: This is text message 9
                        [java] Got message: This is text message 0 from node 0
                        [java] Got message: This is text message 1 from node 1
                        [java] Got message: This is text message 3 from node 0
                        [java] Got message: This is text message 4 from node 1
                        [java] Got message: This is text message 6 from node 0
                        [java] Got message: This is text message 7 from node 1
                        [java] Got message: This is text message 9 from node 0
                        [java] HornetQServer_1 err:[Thread-2 (group:HornetQ-client-global-threads-15868406)] 10:01:44,390 SEVERE 
                  [org.hornetq.core.server.cluster.impl.ClusterConnectionImpl]  Failed to handle message
                        [java] HornetQServer_1 err:java.lang.IllegalStateException: Cannot
                  find binding for jms.queue.exampleQueue8c716cac-8237-11df-a46b-0015c507a7c8
                        [java] HornetQServer_1 err:        at org.hornetq.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.doConsumerClosed(ClusterConnectionImpl.java:828)
                        [java] HornetQServer_1 err:        at org.hornetq.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.onMessage(ClusterConnectionImpl.java:574)
                        [java] HornetQServer_1 err:        at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:823)
                        [java] HornetQServer_1 err:        at org.hornetq.core.client.impl.ClientConsumerImpl.access$100(ClientConsumerImpl.java:46)
                        [java] HornetQServer_1 err:        at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:941)
                        [java] HornetQServer_1 err:        at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:96)
                        [java] HornetQServer_1 err:        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
                        [java] HornetQServer_1 err:        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                        [java] HornetQServer_1 err:        at java.lang.Thread.run(Thread.java:619)
                        [java] HornetQServer_1 err:[Thread-2 (group:HornetQ-client-global-threads-15868406)] 10:01:44,406 SEVERE 
                  [org.hornetq.core.server.cluster.impl.ClusterConnectionImpl]  Failed to handle message
                        [java] HornetQServer_1 err:java.lang.IllegalStateException: Cannot
                  find binding for jms.queue.exampleQueue8c716cac-8237-11df-a46b-0015c507a7c8
                        [java] HornetQServer_1 err:        at org.hornetq.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.doConsumerClosed(ClusterConnectionImpl.java:828)
                        [java] HornetQServer_1 err:        at org.hornetq.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.onMessage(ClusterConnectionImpl.java:574)
                        [java] HornetQServer_1 err:        at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:823)
                        [java] HornetQServer_1 err:        at org.hornetq.core.client.impl.ClientConsumerImpl.access$100(ClientConsumerImpl.java:46)
                        [java] HornetQServer_1 err:        at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:941)
                        [java] HornetQServer_1 err:        at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:96)
                        [java] HornetQServer_1 err:        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
                        [java] HornetQServer_1 err:        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                        [java] HornetQServer_1 err:        at java.lang.Thread.run(Thread.java:619)
                        [java] java.lang.NullPointerException
                        [java]     at org.hornetq.jms.example.ClusteredStaticExample.runExample(ClusteredStaticExample.java:120)
                        [java]     at org.hornetq.common.example.HornetQExample.run(HornetQExample.java:73)
                        [java]     at org.hornetq.jms.example.ClusteredStaticExample.main(ClusteredStaticExample.java:36)
                        [java]
                        [java] #####################
                        [java] ###    FAILURE!   ###
                        [java] #####################
                        [java] Java Result: 1
                  
                  BUILD FAILED
                  C:\hornetq-2.1.0.Final\examples\jms\clustered-static\build.xml:22: The 
                  following error occurred while executing this lin
                  e:
                  C:\hornetq-2.1.0.Final\examples\common\build.xml:151: Example 
                  org.hornetq.jms.example.ClusteredStaticExample failed
                  
                  Total time: 13 seconds
                  
                  

                   

                  These Failed to handle message appear to be my problem (I think the same thing is happening in my development environment)

                   

                  Out of interest I ran the example again (without removing data or build directories), I get the same Failed to handle message error message, but sometimes I don't.  I get some runs resulting is SUCCESS... however when I look at the output there are very strange results with some messages doubling up and other messages disappearing.

                   

                   

                  [java] Sent message: This is text message 0
                        [java] Sent message: This is text message 1
                        [java] Sent message: This is text message 2
                        [java] Sent message: This is text message 3
                        [java] Sent message: This is text message 4
                        [java] Sent message: This is text message 5
                        [java] Sent message: This is text message 6
                        [java] Sent message: This is text message 7
                        [java] Sent message: This is text message 8
                        [java] Sent message: This is text message 9
                        [java] Got message: This is text message 2 from node 0
                        [java] Got message: This is text message 2 from node 1
                        [java] Got message: This is text message 5 from node 0
                        [java] Got message: This is text message 3 from node 1
                        [java] Got message: This is text message 8 from node 0
                        [java] Got message: This is text message 5 from node 1
                        [java] Got message: This is text message 0 from node 0
                        [java] Got message: This is text message 7 from node 1
                        [java] Got message: This is text message 1 from node 0
                        [java] Got message: This is text message 9 from node 1
                  
                        [java] Sent message: This is text message 0
                        [java] Sent message: This is text message 1
                        [java] Sent message: This is text message 2
                        [java] Sent message: This is text message 3
                        [java] Sent message: This is text message 4
                        [java] Sent message: This is text message 5
                        [java] Sent message: This is text message 6
                        [java] Sent message: This is text message 7
                        [java] Sent message: This is text message 8
                        [java] Sent message: This is text message 9
                        [java] Got message: This is text message 4 from node 0
                        [java] Got message: This is text message 2 from node 1
                        [java] Got message: This is text message 6 from node 0
                        [java] Got message: This is text message 3 from node 1
                        [java] Got message: This is text message 8 from node 0
                        [java] Got message: This is text message 6 from node 1
                        [java] Got message: This is text message 0 from node 0
                        [java] Got message: This is text message 8 from node 1
                        [java] Got message: This is text message 1 from node 0
                        [java] Got message: This is text message 9 from node 1
                  
                        [java] Sent message: This is text message 0
                        [java] Sent message: This is text message 1
                        [java] Sent message: This is text message 2
                        [java] Sent message: This is text message 3
                        [java] Sent message: This is text message 4
                        [java] Sent message: This is text message 5
                        [java] Sent message: This is text message 6
                        [java] Sent message: This is text message 7
                        [java] Sent message: This is text message 8
                        [java] Sent message: This is text message 9
                        [java] Got message: This is text message 4 from node 0
                        [java] Got message: This is text message 2 from node 1
                        [java] Got message: This is text message 5 from node 0
                        [java] Got message: This is text message 3 from node 1
                        [java] Got message: This is text message 7 from node 0
                        [java] Got message: This is text message 5 from node 1
                        [java] Got message: This is text message 0 from node 0
                        [java] Got message: This is text message 7 from node 1
                        [java] Got message: This is text message 1 from node 0
                        [java] Got message: This is text message 9 from node 1
                  

                  I have re-read the manual upteen times. I have tried all sorts of combinations and permutations.... but in the end I have to admit that I must be an idiot. I'm not understanding what is going on here or what to do to get it running with static connectors.

                   

                  I attach my example and thank anyone who can take the time to set me on the path to hornetq success.

                  • 6. Re: Cluster, channelBuffer error...
                    artp
                    This configuration won't work as I've discovered. When you start up the instance(ie server 0) that is configured for connector 0, you need to remove it from the cluster connections. In my example server0 and server 1 refer to connector0 and connector1, respectively.

                    From this
                    <cluster-connections>
                    <cluster-connection name="va-cluster">
                    <address>jms</address>
                    <retry-interval>500</retry-interval>
                    <use-duplicate-detection>true</use-duplicate-detection>
                    <forward-when-no-consumers>true</forward-when-no-consumers>
                    <max-hops>1</max-hops>
                    <connector-ref connector-name="netty-connector0" />
                    <connector-ref connector-name="netty-connector1" />
                    </cluster-connection>
                    </cluster-connections>

                     

                    To this on server0


                    <cluster-connections>
                    <cluster-connection name="va-cluster">
                    <address>jms</address>
                    <retry-interval>500</retry-interval>
                    <use-duplicate-detection>true</use-duplicate-detection>
                    <forward-when-no-consumers>true</forward-when-no-consumers>
                    <max-hops>1</max-hops>
                    <connector-ref connector-name="netty-connector1" />
                    </cluster-connection>
                    </cluster-connections>

                    Then do the reverse on server1

                    <cluster-connections>
                    <cluster-connection name="va-cluster">
                    <address>jms</address>
                    <retry-interval>500</retry-interval>
                    <use-duplicate-detection>true</use-duplicate-detection>
                    <forward-when-no-consumers>true</forward-when-no-consumers>
                    <max-hops>1</max-hops>
                    <connector-ref connector-name="netty-connector0" />
                    </cluster-connection>
                    </cluster-connections>

                    • 7. Re: Cluster, channelBuffer error...
                      timfox

                      All this stuff is changing/being improved as descrive in the thread on the dev forum (improvements to HA), so I'm kind of loathed to go deep debugging into this right now, as I am currently implementing the new look implementation.

                      • 8. Re: Cluster, channelBuffer error...
                        artp

                        Tim,

                         

                        The configuration changes I described should fix his problem.  Keep up the good work, HornetQ rocks!

                        • 9. Re: Cluster, channelBuffer error...
                          timfox

                          BTW, you mentioned you changed the port for sever-1 compared to server-0 - this implies to me that your "servers" are actually all on the same machine, since otherwise you wouldn't need to change the port.

                           

                          If your servers are all on the same machine you need to make sure their data directories aren't all pointing to the same directory on disk. Each server needs its own data directory, or all sorts of weird things will happen.

                          • 10. Re: Cluster, channelBuffer error...
                            mcleanl

                            Hi Tim,

                             

                            As I explained in my rather verbose explanation, I used an existing example and by design they use the same machine and different data directories. In real life they are on different machines and use the same port number.

                             

                            Thanks to Art's response I have a suggestion on how to fix this! I had tried the cross-over that he suggests before by it just looked so wrong. If that is the way to solve this in the interim that is good however that means that I now need to maintain a different configuration for every instance.  That is a disappointing overhead.

                             

                            What version are you hoping to implement your unspeakable refactoring?

                             

                            Regards,

                            Luke.

                            • 11. Re: Cluster, channelBuffer error...
                              mcleanl

                              (Still working with 2.1.0 Final)


                              One of my findings through my experimentation is that I had not understood the manual. I had read section 38.5 and had concluded that I needed to implement both section 38.5.1.1 Specifying List of Servers using JMS as well as 38.5.2 Specifying List of Servers to form a Cluster. Through my experimentation I have found that you can do one or the other but NOT both.


                              The suggestions in this thread and others mentioned removing the local host IP address from the <cluster-connections> and this worked to a degree (bridge connected but messages lost). I also found that I could also leave the local host IP connector entry there as long as I removed this from the <connectors> under the <connection-factory> element on the hormet-jms.xml file.


                              Either way bridges were created however messages were still lost. I ran the example I had built and only every third message was 'found'.  I'm sure that it must be one of the settings so I started removing anything that wasn't absolutely required. If I changed max-hops to 2 (from 1) then I got every second message. I kept incresing it and no more were found. The really strange thing happened when I set max-hops to 0 (I know I was getting desparate!).  All the messages were read... Does anyone know why? max-hops = 0 doesn't even appear to be a valid setting.

                               

                              All 10 messages are written to node 0, then messages are read from alternate nodes (0,1,0,1,0 etc).

                               

                                    [java] Sent message: This is text message 0
                                    [java] Sent message: This is text message 1
                                    [java] Sent message: This is text message 2
                                    [java] Sent message: This is text message 3
                                    [java] Sent message: This is text message 4
                                    [java] Sent message: This is text message 5
                                    [java] Sent message: This is text message 6
                                    [java] Sent message: This is text message 7
                                    [java] Sent message: This is text message 8
                                    [java] Sent message: This is text message 9
                                    [java] Got message: This is text message 0 from node 0
                                    [java] Got message: This is text message 1 from node 1
                                    [java] Got message: This is text message 2 from node 0
                                    [java] Got message: This is text message 3 from node 1
                                    [java] Got message: This is text message 4 from node 0
                                    [java] Got message: This is text message 5 from node 1
                                    [java] Got message: This is text message 6 from node 0
                                    [java] Got message: This is text message 7 from node 1
                                    [java] Got message: This is text message 8 from node 0
                                    [java] Got message: This is text message 9 from node 1
                                    [java] example complete
                                    [java]
                                    [java] #####################
                                    [java] ###    SUCCESS!   ###
                                    [java] #####################
                              

                               

                              I again attach the simple test case example, written in the same style as the examples included with the product.

                               

                              1) So that others can reproduce in a familiar environment; and,

                              2) So that people don't ask me for this to then do nothing with it.

                               

                              I apologize that the batch files are renamed build.bam and cleanup.bam (you will need to rename to bat to run). Our file scanners otherwise remove all batch files because they are apparently dangerous...

                               

                              Regards,

                              Luke.