12 Replies Latest reply on Jan 2, 2016 4:30 PM by igor.polevoy

    HornetQ throws server - HQ224045: Failed to run large message deliverer

    igor.polevoy

      Hi, there!

      We are using embedded HornetQ version 2.4.1.Final

      There are a few thousand messages just sitting under largemessages directory. They are not delivered, and I presume because of this exception

      I found these stack traces in our log files:

      ERROR [Thread-9 (HornetQ-server-HornetQServerImpl::serverUUID=228d6abf-6f78-11e5-9476-ab4e397e2e52-1420470414)]: server - HQ224045: Failed to run large message deliverer
      java.lang.UnsupportedOperationException
        at java.nio.ByteBuffer.array(ByteBuffer.java:994)
        at org.hornetq.core.server.impl.ServerConsumerImpl$LargeMessageDeliverer.deliver(ServerConsumerImpl.java:1090)
        at org.hornetq.core.server.impl.ServerConsumerImpl$2.run(ServerConsumerImpl.java:969)
        at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:104)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

      and this one:

      2015-12-29 03:02:02,658 ERROR [Thread-1 (HornetQ-client-global-threads-2138527112)]: client - HQ124003: Message Listener failed to prepare message for receipt
      HornetQLargeMessageException[errorType=LARGE_MESSAGE_ERROR_BODY message=HQ119028: Timeout waiting for LargeMessage Body]
        at org.hornetq.core.client.impl.LargeMessageControllerImpl.waitCompletion(LargeMessageControllerImpl.java:372)
        at org.hornetq.core.client.impl.LargeMessageControllerImpl.saveBuffer(LargeMessageControllerImpl.java:319)
        at org.hornetq.core.client.impl.ClientLargeMessageImpl.checkBuffer(ClientLargeMessageImpl.java:186)
        at org.hornetq.core.client.impl.ClientLargeMessageImpl.checkCompletion(ClientLargeMessageImpl.java:89)
        at org.hornetq.jms.client.HornetQMessage.doBeforeReceive(HornetQMessage.java:968)
        at org.hornetq.jms.client.HornetQTextMessage.doBeforeReceive(HornetQTextMessage.java:129)
        at org.hornetq.jms.client.JMSMessageListenerWrapper.onMessage(JMSMessageListenerWrapper.java:79)
        at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1116)
        at org.hornetq.core.client.impl.ClientConsumerImpl.access$500(ClientConsumerImpl.java:56)
        at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1251)
        at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:104)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

      The number of files in "largemessages' is growing, and I'm afraid that some data is not processed.

      Any help is much appreciated!

        • 1. Re: HornetQ throws server - HQ224045: Failed to run large message deliverer
          jbertram

          What JVM are you using in this environment?

          • 2. Re: HornetQ throws server - HQ224045: Failed to run large message deliverer
            igor.polevoy

            Hi, Justin. My environment:

             

            java version "1.8.0_66"

            Java(TM) SE Runtime Environment (build 1.8.0_66-b17)

            Java HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode)

             

            Linux sender5.host11.com 3.13.0-45-generic #74-Ubuntu SMP Tue Jan 13 19:36:28 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

            Distributor ID: Ubuntu

            Description: Ubuntu 14.04.3 LTS

            Release: 14.04

            Codename: trusty

            • 3. Re: HornetQ throws server - HQ224045: Failed to run large message deliverer
              jbertram

              It looks like this exception would be thrown if the body of the message is null for some reason.  What are the size of the files on disk?  Are they non-zero? 

               

              Are you able to reproduce the problem reliably?  If so, could you provide a test-case?

              • 4. Re: HornetQ throws server - HQ224045: Failed to run large message deliverer
                igor.polevoy

                Yes, there is one file with size 0.

                How was it created? Also, can I just kill it?

                Getting a test case would be tough,  as we have this embedded into our app.

                One other thing is: we do not want large messages. our biggest messages are just over 100k. So, we configured a minimum large message size to be 250000, but large message files are still created.

                 

                Sorry for so many questions, and thanks for advice!

                • 5. Re: HornetQ throws server - HQ224045: Failed to run large message deliverer
                  jbertram

                  I'm not sure how a large file with size zero would ever be created.  You could try shutting down the broker, removing the file, and then restarting the broker, but be sure to back up your data before you make any changes so that if problems occur you can restore.

                   

                  How is your min-large-message-size configured?  Did you do it programmatically or via XML?

                  • 6. Re: HornetQ throws server - HQ224045: Failed to run large message deliverer
                    igor.polevoy

                    Here is what we did:

                     

                    1. Reduced the sized of messages being sent into broker (50% of what we sent before)

                    2. Deleted all large messages from largemessages directory (backed them up)

                    3. Restarted the app and got the same exception:

                     

                    java.lang.UnsupportedOperationException

                      at java.nio.ByteBuffer.array(ByteBuffer.java:994)

                      at org.hornetq.core.server.impl.ServerConsumerImpl$LargeMessageDeliverer.deliver(ServerConsumerImpl.java:1090)

                      at org.hornetq.core.server.impl.ServerConsumerImpl$2.run(ServerConsumerImpl.java:969)

                      at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:104)

                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

                      at java.lang.Thread.run(Thread.java:745)

                     

                    Now, the broker created 11 files with 0 size

                     

                    drwxrwxr-x 2 pigeon pigeon 4096 Dec 30 14:23 ./

                    drwxr-xr-x 8 pigeon pigeon 4096 Dec 30 14:21 ../

                    -rw-rw-r-- 1 pigeon pigeon    0 Dec 30 14:22 12896597601.msg

                    -rw-rw-r-- 1 pigeon pigeon    0 Dec 30 14:22 19338070485.msg

                    -rw-rw-r-- 1 pigeon pigeon    0 Dec 30 14:22 19338070491.msg

                    -rw-rw-r-- 1 pigeon pigeon    0 Dec 30 14:22 19338070496.msg

                    -rw-rw-r-- 1 pigeon pigeon    0 Dec 30 14:22 19338070499.msg

                    -rw-rw-r-- 1 pigeon pigeon    0 Dec 30 14:22 19338070503.msg

                    -rw-rw-r-- 1 pigeon pigeon    0 Dec 30 14:22 19338070506.msg

                    -rw-rw-r-- 1 pigeon pigeon    0 Dec 30 14:22 19338070522.msg

                    -rw-rw-r-- 1 pigeon pigeon    0 Dec 30 14:22 19338070525.msg

                    -rw-rw-r-- 1 pigeon pigeon    0 Dec 30 14:23 19338070531.msg

                    -rw-rw-r-- 1 pigeon pigeon    0 Dec 30 14:22 9230422.msg

                     

                    Also, it shows that QueueControl#countMessages(null)); returns more than 18k messages in the queue, but they are not delivered. They are just sitting there.

                     

                    We set the min size of large messages programmatically:

                     

                    ConnectionFactoryConfiguration cfConfig = ....;

                    cfConfig.setMinLargeMessageSize(250000);

                     

                    Appreciate helping!

                    • 7. Re: HornetQ throws server - HQ224045: Failed to run large message deliverer
                      igor.polevoy

                      latest update:

                      I re-created entire directory where HornetQ stores any data, and restarted the process. It is working smoothly now. Apparently some data file somewhere had a poison pill.

                      I can re-sent all messages again, so no data loss. I "resolved" the problem, but still do not know what caused it and how to avoid in future,

                      • 8. Re: HornetQ throws server - HQ224045: Failed to run large message deliverer
                        igor.polevoy

                        I solved my immediate problem, but if you want to recreate this condition, and improve the product, I can make my data directory available to you.

                        All you need to do is point to this directory and watch it fail.

                        Let me know -

                         

                        tx

                        • 9. Re: HornetQ throws server - HQ224045: Failed to run large message deliverer
                          jbertram

                          Please attach your data directory at your convenience.

                           

                          Also, I recommend you check out Apache ActiveMQ Artemis.  The HornetQ code-base was donated to Apache over a year ago and has become a sub-project of ActiveMQ called Artemis.  All the APIs you're using for your embedded use-case should still be in tact, but a lot of work has been done in other areas.  The 1.2.0 release should be out soon.  To be clear, no further work will be done on HornetQ as such.  All work is being concentrated on Artemis.

                          • 10. Re: HornetQ throws server - HQ224045: Failed to run large message deliverer
                            igor.polevoy

                            I was unaware of this. Is Redhat dropping HornetQ? If so, I can certainly switch to Artemis. What is the roadmap for  messaging under Redhat?

                             

                            tx

                            • 11. Re: HornetQ throws server - HQ224045: Failed to run large message deliverer
                              jbertram

                              HornetQ is living on as Apache ActiveMQ Artemis.  As I indicated previously, the work which was previously concentrated on HornetQ has been relocated to Artemis.

                               

                              As far as a messaging road-map from Red Hat, I'd encourage you to discuss this with your Red Hat account representative.

                              • 12. Re: HornetQ throws server - HQ224045: Failed to run large message deliverer
                                igor.polevoy

                                Thanks for the follow up! We are looking to integrate Artemis now.