HornetQ startup fails during initialization with null pointer exception if large number of page files are present.
labdhi Sep 7, 2011 3:48 AMI am using standalone HornetQ 2.2.5.Final version. paging is enabeled and schedule delivery feature is in use.
journal size is 400 MB and page file size is 10 MB.
using plain JMS transactions instead of JTA. application is multithreaded with dozens of producers and more than 50 concurrent consumers around 20 Queues
Spring is used to produce and consume the messages.
CachingConnectionFactory is used to wrap the HornetQQueueConnectionFactory and various JMS admin objects like connection, session and message producers are cached by Spring.
DefaultMessageListenerContainers provided by Spring are in use to pull the message fom HornetQ. at the time of pulling the message JMS session in use is transacted and transaction is a local JMS transaction. jta is not in use in short thruout.
ConsumerWindowSize is set to 0 on to ensure that only one message will get pulled at a time per consumer thread. as and when consumer thread is done with one message[either commit or rollback] it pulls another message from HornetQ.
HornetQ is running as standalone process on a different node. producers and consumers are remotely connecting to it.
incase of huge load HornetQ starts paging incoming messages to page files.
If we end up accumulating too many page files on the disk at that time another thing what i noticed is that HornetQ instance is not even coming up properly during restart of the process and failing during initialization itself. stack trace is given below for such kind of scenarios.
[main] 06:50:31,490 SEVERE [org.hornetq.core.server.impl.HornetQServerImpl] Failure in initialisation
- java.lang.NullPointerException
at org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl.getPageInfo(PageSubscriptionImpl.java:726)
at org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl.getPageInfo(PageSubscriptionImpl.java:712)
at org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl.processReload(PageSubscriptionImpl.java:655)
at org.hornetq.core.paging.cursor.impl.PageCursorProviderImpl.processReload(PageCursorProviderImpl.java:241)
at org.hornetq.core.paging.impl.PagingStoreImpl.processReload(PagingStoreImpl.java:355)
at org.hornetq.core.paging.impl.PagingManagerImpl.processReload(PagingManagerImpl.java:250)
at org.hornetq.core.persistence.impl.journal.JournalStorageManager.loadMessageJournal(JournalStorageManager.java:1229)
at org.hornetq.core.server.impl.HornetQServerImpl.loadJournals(HornetQServerImpl.java:1619)
at org.hornetq.core.server.impl.HornetQServerImpl.initialisePart2(HornetQServerImpl.java:1469)
at org.hornetq.core.server.impl.HornetQServerImpl.access$100(HornetQServerImpl.java:132)
at org.hornetq.core.server.impl.HornetQServerImpl$SharedStoreLiveActivation.run(HornetQServerImpl.java:356)
at org.hornetq.core.server.impl.HornetQServerImpl.start(HornetQServerImpl.java:570)
at org.hornetq.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:275)
at com.qpass.service.jms.HornetQService.main(HornetQService.java:51)
my hornetq-configuration.xml is given below.
<configuration xmlns="urn:hornetq"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:hornetq /schema/hornetq-configuration.xsd">
<acceptors>
<acceptor name="netty">
<factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>
<param key="host" value="${hostname}"/>
<param key="port" value="${livePort}"/>
<param key="use-nio" value="true"/>
</acceptor>
<acceptor name="netty_localhsot">
<factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>
<param key="host" value="localhost"/>
<param key="port" value="${livePort}"/>
<param key="use-nio" value="true"/>
</acceptor>
</acceptors>
<address-settings>
<!--default for catch all-->
<address-setting match="jms.queue.ConsumerUpdateRequestQueue">
<dead-letter-address>jms.queue.consumerupdate.ExceptionQueue</dead-letter-address>
<max-delivery-attempts>1</max-delivery-attempts>
<max-size-bytes>${queueDataInMemory}</max-size-bytes>
</address-setting>
<address-setting match="jms.queue.ConsumerUpdateRetryQueue">
<dead-letter-address>jms.queue.chargesandpayments.ExceptionQueue</dead-letter-address>
<max-delivery-attempts>1</max-delivery-attempts>
<max-size-bytes>${queueDataInMemory}</max-size-bytes>
</address-setting>
<address-setting match="jms.queue.EventDistributionQueue">
<dead-letter-address>jms.queue.notification.ExceptionQueue</dead-letter-address>
<max-delivery-attempts>1</max-delivery-attempts>
<max-size-bytes>${EDQ_queueDataInMemory}</max-size-bytes>
</address-setting>
<address-setting match="jms.queue.consumerupdate.DeadUpdateQueue">
<max-size-bytes>${queueDataInMemory}</max-size-bytes>
<max-delivery-attempts>-1</max-delivery-attempts>
<redelivery-delay>60000</redelivery-delay>
</address-setting>
<address-setting match="jms.queue.notification.DeadEventQueue">
<max-size-bytes>${queueDataInMemory}</max-size-bytes>
<max-delivery-attempts>-1</max-delivery-attempts>
<redelivery-delay>60000</redelivery-delay>
</address-setting>
<address-setting match="jms.queue.mail.DeadLetterQueue">
<max-size-bytes>${queueDataInMemory}</max-size-bytes>
<max-delivery-attempts>-1</max-delivery-attempts>
<redelivery-delay>60000</redelivery-delay>
</address-setting>
<address-setting match="jms.queue.chargesandpayments.DeadMessageQueue">
<max-size-bytes>${queueDataInMemory}</max-size-bytes>
<max-delivery-attempts>-1</max-delivery-attempts>
<redelivery-delay>60000</redelivery-delay>
</address-setting>
<address-setting match="jms.queue.chargesandpayments.DeadMessageQueues">
<max-size-bytes>${queueDataInMemory}</max-size-bytes>
<max-delivery-attempts>-1</max-delivery-attempts>
<redelivery-delay>60000</redelivery-delay>
</address-setting>
<address-setting match="jms.queue.osn.ExternalNotificationDeliveryDeadQueue">
<max-size-bytes>${queueDataInMemory}</max-size-bytes>
<max-delivery-attempts>-1</max-delivery-attempts>
<redelivery-delay>60000</redelivery-delay>
</address-setting>
<address-setting match="jms.queue.extnotification.ExternalNotificationRedeliveryDeadQueue">
<max-size-bytes>${queueDataInMemory}</max-size-bytes>
<max-delivery-attempts>-1</max-delivery-attempts>
<redelivery-delay>60000</redelivery-delay>
</address-setting>
<address-setting match="jms.queue.extnotification.ExternalNotificationRedeliveryQueue">
<max-size-bytes>${queueDataInMemory}</max-size-bytes>
<max-delivery-attempts>-1</max-delivery-attempts>
<redelivery-delay>60000</redelivery-delay>
</address-setting>
<address-setting match="jms.queue.HeuristicHazardsQueue">
<max-size-bytes>${queueDataInMemory}</max-size-bytes>
</address-setting>
<address-setting match="jms.queue.osn.ExternalNotificationDeliveryQueue">
<dead-letter-address>jms.queue.extnotification.ExternalNotificationRedeliveryQueue</dead-letter-address>
<max-delivery-attempts>1</max-delivery-attempts>
<max-size-bytes>${ENDQ_queueDataInMemory}</max-size-bytes>
</address-setting>
<address-setting match="jms.queue.BillingRequestQueue">
<max-size-bytes>${BRQ_queueDataInMemory}</max-size-bytes>
</address-setting>
<address-setting match="jms.queue.MailQueue">
<dead-letter-address>jms.queue.mail.ExceptionQueue</dead-letter-address>
<max-delivery-attempts>3</max-delivery-attempts>
<max-size-bytes>${queueDataInMemory}</max-size-bytes>
</address-setting>
<address-setting match="jms.queue.PaymentAdaptorRequestQueue">
<dead-letter-address>jms.queue.chargesandpayments.ExceptionQueue</dead-letter-address>
<max-delivery-attempts>-1</max-delivery-attempts>
<max-size-bytes>${queueDataInMemory}</max-size-bytes>
</address-setting>
<address-setting match="jms.queue.PaymentAdaptorResponseQueue">
<dead-letter-address>jms.queue.chargesandpayments.DeadMessageQueues</dead-letter-address>
<max-delivery-attempts>1</max-delivery-attempts>
<max-size-bytes>${PARQ_queueDataInMemory}</max-size-bytes>
</address-setting>
<address-setting match="jms.queue.PaymentAdaptorRetryQueue">
<dead-letter-address>jms.queue.chargesandpayments.DeadMessageQueues</dead-letter-address>
<max-delivery-attempts>1</max-delivery-attempts>
<max-size-bytes>${queueDataInMemory}</max-size-bytes>
</address-setting>
<address-setting match="jms.queue.notification.RedeliveryQueue">
<dead-letter-address>jms.queue.notification.ExceptionQueue</dead-letter-address>
<max-delivery-attempts>1</max-delivery-attempts>
<max-size-bytes>${queueDataInMemory}</max-size-bytes>
</address-setting>
<address-setting match="jms.queue.consumerupdate.ExceptionQueue">
<dead-letter-address>jms.queue.consumerupdate.DeadUpdateQueue</dead-letter-address>
<max-delivery-attempts>1</max-delivery-attempts>
<max-size-bytes>${queueDataInMemory}</max-size-bytes>
</address-setting>
<address-setting match="jms.queue.chargesandpayments.ExceptionQueue">
<dead-letter-address>jms.queue.chargesandpayments.DeadMessageQueue</dead-letter-address>
<max-delivery-attempts>1</max-delivery-attempts>
<max-size-bytes>${queueDataInMemory}</max-size-bytes>
</address-setting>
<address-setting match="jms.queue.extnotification.ExceptionQueue">
<dead-letter-address>jms.queue.extnotification.ExternalNotificationRedeliveryDeadQueue</dead-letter-address>
<max-delivery-attempts>1</max-delivery-attempts>
<max-size-bytes>${queueDataInMemory}</max-size-bytes>
</address-setting>
<address-setting match="jms.queue.mail.ExceptionQueue">
<dead-letter-address>jms.queue.mail.DeadLetterQueue</dead-letter-address>
<max-delivery-attempts>1</max-delivery-attempts>
<max-size-bytes>${queueDataInMemory}</max-size-bytes>
</address-setting>
<address-setting match="jms.queue.notification.ExceptionQueue">
<dead-letter-address>jms.queue.notification.DeadEventQueue</dead-letter-address>
<max-delivery-attempts>1</max-delivery-attempts>
<max-size-bytes>${queueDataInMemory}</max-size-bytes>
</address-setting>
${customHornetQCoreQueueConfiguration}
</address-settings>
<jmx-domain>com.qpass</jmx-domain>
<paging-directory>${data_dir}/paging</paging-directory>
<bindings-directory>${data_dir}/bindings</bindings-directory>
<journal-directory>${data_dir}/journal</journal-directory>
<large-messages-directory>${data_dir}/large-messages</large-messages-directory>
<journal-file-size>${journalSize}</journal-file-size>
<journal-min-files>3</journal-min-files>
<journal-type>NIO</journal-type>
<id-cache-size>1</id-cache-size>
<persist-id-cache>false</persist-id-cache>
<security-enabled>false</security-enabled>
<remoting-interceptors>
<class-name>com.qpass.service.jms.ScheduledDeliveryInterceptor</class-name>
</remoting-interceptors>
<message-counter-enabled>true</message-counter-enabled>
<persist-delivery-count-before-delivery>false</persist-delivery-count-before-delivery>
<message-counter-sample-period>300000</message-counter-sample-period>
<message-counter-max-day-history>2</message-counter-max-day-history>
<message-expiry-scan-period>600000</message-expiry-scan-period>
<message-expiry-thread-priority>1</message-expiry-thread-priority>
<memory-measure-interval>180000</memory-measure-interval>
${customHornetQCoreConfiguration}
</configuration>