11 Replies Latest reply on Dec 7, 2012 3:15 PM by clebert.suconic

    Record is too large to store - Hornetq 2.2.13 with Jboss 7.1.1 community

    kperry

      I am a little confused on why we got the error "Record is too large to store". 

       

      Background

      We are using a request/response pattern using with some clustered, remote, named queues and some temporary queues.  Our workflow goes something like this:

      1. Webservice call creates a temporary queue
      2. Then it creates a message and sets the reply as the temporary queue
      3. Then the producer on the webservice sends the message to a remote cluster (we have 2 nodes on this remote cluster).
      4. A bunch of work is done on the remote machine via an mdb consumer on the named queue in the cluster
      5. Then a result object is sent back to the temporary queue 

       

      What Happened.

      1. When trying to send a message back to the temporary queue (see step 5 above), we received the "Record is too large to store" error message.
        • The message that appeared to cause this was large - the straight xml was 103429 bytes
      2. When trying to figure out what was happening, we switched over to our stage environment and the same thing happened in stage with almost the exact same size message

       

      Problems and Questions

      1. In this particular scenario, it caused our cluster to fail.  Why?
        • If we brought only node one up, then messages sailed on through (it didn't matter which one as long as both weren't up)
        • The only way we could resolve was to delete out the data directory on both nodes
      2. What scenarios will cause this error to occur?
        1. I have tried to duplicate in our stage environment, but I am unable to - even with messages very large - much larger than the one that appeared to cause this failure
      3. I thought large messages only put a pointer on the journal file?

       

      What we did.

      1. We increased the default setting for the journal-file-size from

      <journal-file-size>102400</journal-file-size>

      TO:

      <journal-file-size>10485760</journal-file-size>

       

      We haven't got the error since making this change.  Should the journal-file-size be larger by default - it looks like it is set to 10MB here?  https://access.redhat.com/knowledge/docs/en-US/JBoss_Enterprise_Application_Platform/5/html/HornetQ_User_Guide/configuring.message.journal.html

       

      Here is the stacktrace:

      11:59:09,824 ERROR [org.hornetq.core.protocol.core.ServerSessionPacketHandler] (Thread-12812 (HornetQ-remoting-threads-HornetQServerImpl::serverUUID=69b6be6b-29f0-11e2-b370-005056a2009c-1801090058-2138600341)) Caught unexpected exception: java.lang.IllegalArgumentException: Record is too large to store 102051
           at org.hornetq.core.journal.impl.JournalImpl.appendRecord(JournalImpl.java:2908) [hornetq-core-2.2.13.Final.jar:]
           at org.hornetq.core.journal.impl.JournalImpl.appendAddRecordTransactional(JournalImpl.java:1066) [hornetq-core-2.2.13.Final.jar:]
           at org.hornetq.core.persistence.impl.journal.JournalStorageManager.storeMessageTransactional(JournalStorageManager.java:645) [hornetq-core-2.2.13.Final.jar:]
           at org.hornetq.core.postoffice.impl.PostOfficeImpl.processRoute(PostOfficeImpl.java:1017) [hornetq-core-2.2.13.Final.jar:]
           at org.hornetq.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:678) [hornetq-core-2.2.13.Final.jar:]
           at org.hornetq.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:580) [hornetq-core-2.2.13.Final.jar:]
           at org.hornetq.core.server.impl.ServerSessionImpl.doSend(ServerSessionImpl.java:1437) [hornetq-core-2.2.13.Final.jar:]
           at org.hornetq.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:1164) [hornetq-core-2.2.13.Final.jar:]
           at org.hornetq.core.protocol.core.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:441) [hornetq-core-2.2.13.Final.jar:]
           at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:508) [hornetq-core-2.2.13.Final.jar:]
           at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:556) [hornetq-core-2.2.13.Final.jar:]
           at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:517) [hornetq-core-2.2.13.Final.jar:]
           at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:533) [hornetq-core-2.2.13.Final.jar:]
           at org.hornetq.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:166) [hornetq-core-2.2.13.Final.jar:]
           at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [hornetq-core-2.2.13.Final.jar:]
           at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_31]
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_31]
           at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_31]