6 Replies Latest reply on Oct 18, 2007 12:41 PM by niefeng

JMS msg get redelivered although setting retry to 0

niefeng Oct 9, 2007 6:23 AM

Hi all,

I have some problem on the JMS msg redelivery after the main node has been down abnormally. I would like to disable such a redelivery due to some business logic requirement.

My box has the following setup:
version: JBOSS 4.0.5 GA
OS: Sun Solaris 10
JRE: JRE 1.5.10

Deploy env:
Two Jboss server running at cluster mode.
JMS: HA-JMS

When the main node has been gone down unexpectedly, the JMS service is temporary unavailable to the second node (The handover of the JMS service to the second node will take some time).

I have observed the case that if the onMessage() of the MD Bean deployed on second node has been called prior the failure of the main node, and that MD Bean finished the processing when the handover process still on going, then a redeliver of that jms msg is kicked in due to the MD Bean hasn't ack that msg. This part is functioning as expected.

Snippet of the exception after the onMessage of the MD Bean has completed, but at that time the handover process still on going, and it is not able to send back the ACK:

[09:10:07-16:50:30,815] [ INFO:STDOUT] [JMS SessionPool Worker-5] ReceiveMessageListenerMDBean end onMessage
[09:10:07-16:50:30,834] [ WARN:org.jboss.tm.TransactionImpl] [JMS SessionPool Worker-5] XAException: tx=TransactionImpl:XidImpl[FormatId=257, GlobalId=apple/168, BranchQual=, localId=168] errorCode=XAER_RMERR
org.jboss.mq.SpyXAException: Resource manager error during prepare; - nested throwable: (org.jboss.mq.SpyJMSException: Cannot process a transaction; - nested throwable: (java.io.IOException: Client is not connected))
at org.jboss.mq.SpyXAException.getAsXAException(SpyXAException.java:72)
at org.jboss.mq.SpyXAResource.prepare(SpyXAResource.java:151)
at org.jboss.tm.TransactionImpl$Resource.prepare(TransactionImpl.java:2212)
at org.jboss.tm.TransactionImpl.prepareResources(TransactionImpl.java:1660)
at org.jboss.tm.TransactionImpl.commit(TransactionImpl.java:347)
at org.jboss.tm.TxManager.commit(TxManager.java:240)
at org.jboss.jms.asf.StdServerSession.onMessage(StdServerSession.java:351)
at org.jboss.mq.SpyMessageConsumer.sessionConsumerProcessMessage(SpyMessageConsumer.java:905)
at org.jboss.mq.SpyMessageConsumer.addMessage(SpyMessageConsumer.java:170)
at org.jboss.mq.SpySession.run(SpySession.java:323)
at org.jboss.jms.asf.StdServerSession.run(StdServerSession.java:194)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743)
at java.lang.Thread.run(Unknown Source)
Caused by: org.jboss.mq.SpyJMSException: Cannot process a transaction; - nested throwable: (java.io.IOException: Client is not connected)
at org.jboss.mq.SpyJMSException.getAsJMSException(SpyJMSException.java:72)
at org.jboss.mq.SpyJMSException.rethrowAsJMSException(SpyJMSException.java:57)
at org.jboss.mq.Connection.send(Connection.java:1114)
at org.jboss.mq.SpyXAResourceManager.prepare(SpyXAResourceManager.java:255)
at org.jboss.mq.SpyXAResource.prepare(SpyXAResource.java:147)
... 11 more
Caused by: java.io.IOException: Client is not connected
at org.jboss.mq.il.uil2.SocketManager.internalSendMessage(SocketManager.java:264)
at org.jboss.mq.il.uil2.SocketManager.sendMessage(SocketManager.java:220)
at org.jboss.mq.il.uil2.UILServerIL.transact(UILServerIL.java:368)
at org.jboss.mq.Connection.send(Connection.java:1110)
... 13 more

Due to some logic constraint, I need to disable the redelivery of the JMS for the above case. I have tried to disable such a redeliver in the standardjboss.xml as follow. I set the tag "MaxTimesRedelivered" to 0.

<invoker-proxy-binding>
 <name>message-driven-bean-with-delivered-once</name>
 <invoker-mbean>default</invoker-mbean>
 <proxy-factory>org.jboss.ejb.plugins.jms.JMSContainerInvoker</proxy-factory>
 <proxy-factory-config>
 <JMSProviderAdapterJNDI>DefaultJMSProvider</JMSProviderAdapterJNDI>
 <ServerSessionPoolFactoryJNDI>StdJMSPool</ServerSessionPoolFactoryJNDI>
 <CreateJBossMQDestination>true</CreateJBossMQDestination>
 <!-- WARN: Don't set this to zero until a bug in the pooled executor is

fixed -->
 <MinimumSize>1</MinimumSize>
 <!-- MaximumSize>15</MaximumSize -->
 <MaximumSize>50</MaximumSize>
 <KeepAliveMillis>30000</KeepAliveMillis>
 <MaxMessages>1</MaxMessages>
 <MDBConfig>
 <ReconnectIntervalSec>10</ReconnectIntervalSec>
 <DLQConfig>
 <DestinationQueue>queue/DLQ</DestinationQueue>
 <MaxTimesRedelivered>0</MaxTimesRedelivered>
 <TimeToLive>0</TimeToLive>
 </DLQConfig>
 </MDBConfig>
 </proxy-factory-config>
 </invoker-proxy-binding>

However, I still observed the redelivered from the queue to the MD Bean.

I also tried to ignore such a redelivery in the code level as follow

if(message.getJMSRedelivered())
 {
 infoLog("onMessage", "re receive msg from from/queue again. Stop processing");
 return;
 }

However, the flag "jmsRedelivered" is still set to false and the md bean still getting the jms msg that has been processed before.

Could I know is there anywhere I have mis-configured? Or is there any alternative I can adapt to disable the redelivery in the above case?

NOTE: The above two methods I mentioned is working fine if I do a messageCtxt.setRollback() in the MD BEAN.

1. Re: JMS msg get redelivered although setting retry to 0

niefeng Oct 11, 2007 5:36 AM (in response to niefeng)

Hi all,

Did I omit something in my description or the information I provided is not sufficient for the case?

Thanks for your help.
Actions
2. Re: JMS msg get redelivered although setting retry to 0

adrian.brock Oct 12, 2007 9:14 AM (in response to niefeng)

Show System.out.println(message) for both deliveries of the message.
Actions

3. Re: JMS msg get redelivered although setting retry to 0

niefeng Oct 13, 2007 4:46 AM (in response to niefeng)

Hi Adrian, thanks for your reply.

Below is the sysout of the jms message.

1st received msg

[13:10:07-16:16:22,461][JMS SessionPool Worker-19][ INFO][GN.APP.WORKFLOW][]Msg received: org.jboss.mq.SpyObjectMessage {
Header {
 jmsDestination : QUEUE.sendSignalQueue
 jmsDeliveryMode : 2
 jmsExpiration : 0
 jmsPriority : 4
 jmsMessageID : ID:240-119226338233616
 jmsTimeStamp : 1192263382336
 jmsCorrelationID: null
 jmsReplyTo : null
 jmsType : null
 jmsRedelivered : false
 jmsProperties : {}
 jmsPropReadWrite: false
 msgReadOnly : true
 producerClientId: ID:240
}

The redelivered message

[13:10:07-16:18:17,913][JMS SessionPool Worker-59][ INFO][GN.APP.WORKFLOW][]Msg received: org.jboss.mq.SpyObjectMessage {
Header {
 jmsDestination : QUEUE.sendSignalQueue
 jmsDeliveryMode : 2
 jmsExpiration : 0
 jmsPriority : 4
 jmsMessageID : ID:240-119226338233616
 jmsTimeStamp : 1192263382336
 jmsCorrelationID: null
 jmsReplyTo : null
 jmsType : null
 jmsRedelivered : false
 jmsProperties : {}
 jmsPropReadWrite: false
 msgReadOnly : true
 producerClientId: ID:240
}

4. Re: JMS msg get redelivered although setting retry to 0

adrian.brock Oct 17, 2007 9:26 AM (in response to niefeng)
Ok, I think I understand what the problem is.

The issue is that the failover node doesn't know that the message was already
delivered. The node (the original node) that did know has crashed.

The problem comes because the failover node uses the transaction log
to recover the state to the "last known good state" of the message
(which was in your case before the message was delivered to the MDB).

The only way I can see to fix the problem, would be to update the
redelivered status and redelivery count of the message in the database
before the message is delivered to the client (instead of when the server
receives the NACK/rollback - which it won't do if the server crashes
- it will do it if the problem is just the client loosing connectivity).

i.e. in BasicQueue
http://viewvc.jboss.org/cgi-bin/viewvc.cgi/jbossas/trunk/jbossmq/src/main/org/jboss/mq/server/BasicQueue.java?revision=61447&view=markup

This would involve moving the code in "nackMessage()"
try { ... // Set redelivered, vendor-specific flags message.invalidate(); // Update the persistent message outside the transaction // We want to know the message might have been delivered regardless if (message.isPersistent()) server.getPersistenceManager().update(message, null); } catch (JMSException e) { log.error("Error setting redelivered " + message, e); } }

To the setupMessageAcknowledgement() method instead (which is always
invoked before delivering the message).

But, this would obviously have a deterimental effect on performance
since it would always update persistent messages with the redelivered status
in the database regardless of whether it is was actually redelivered or not
(if it doesn't get redelivered, i.e. it is ACKed/committed, it will be deleted so the
early change in status is irrelevant).

Do you want to try that patch?
Actions
5. Re: JMS msg get redelivered although setting retry to 0

adrian.brock Oct 17, 2007 9:41 AM (in response to niefeng)

Bug report:
http://jira.jboss.com/jira/browse/JBAS-4870
Actions
6. Re: JMS msg get redelivered although setting retry to 0

niefeng Oct 18, 2007 12:41 PM (in response to niefeng)

Hi Adrian,

Thanks for the explanation on the problem. I will give it a try for that patch. Could I know where should I get it?

Will the solution you suggested (update the redelivery status prior the msg delivery to the client) be applied to the latest Jboss AS also? or it is a temporary solution to resolve my case?

In the jira, I saw the fix version will be in JBossAS-5.0.0.Beta3 and JBossAS-4.2.3.GA. Does it mean I need to update my current jboss to the latest version after the official patch make available? Or will you provide a seperate patch for JBoss 4.0.5 GA?
Actions

Go to original post