-
30. Re: XARecovery: Messaging integration with JBoss Transaction
Tim,
If the Recovery manager *somehow* gets hold of a XAResoruce reference, that'll be adequate (for now!) for recovery, am I correct in taking this direction?
(It is almost same as your JMSRecoverable class publishing to JNDI from which you intended to grab the XAResource and then call recovery)."timfox" wrote:
On recovery you need to "resurrect" the transaction instance from the database (since that is currently acting as our transaction log) and add it to the transaction repository. Whether this is done on startup or on demand is another question..
By adding to transaction repository, do you mean JBossTS object store? I can fetch the transaction from the database, but I am not sure whether we can merge with JBossTS (if I am not mis-reading your lines) -
31. Re: XARecovery: Messaging integration with JBoss Transaction
"mskonda" wrote:
Tim,
If the Recovery manager *somehow* gets hold of a XAResoruce reference, that'll be adequate (for now!) for recovery, am I correct in taking this direction?
This is the bit I still can't fathom: if the XAResource was enrolled with a JBossTS transaction, then there will be a reference to it within the transaction log already, assuming it failed at the "right time" during 2PC. If you start with a clean object store and run your test, then you SHOULD see a log entry for the transaction after the failure (and before you restart your system). If you don't see one, then one of the 6 possible options I mentioned before happened. -
32. Re: XARecovery: Messaging integration with JBoss Transaction
"mskonda" wrote:
Tim,
If the Recovery manager *somehow* gets hold of a XAResoruce reference, that'll be adequate (for now!) for recovery, am I correct in taking this direction?
(It is almost same as your JMSRecoverable class publishing to JNDI from which you intended to grab the XAResource and then call recovery).
Yes. For recovery all you need to do is provide the TM with an XAResource so it can call recover() on it."timfox" wrote:
On recovery you need to "resurrect" the transaction instance from the database (since that is currently acting as our transaction log) and add it to the transaction repository. Whether this is done on startup or on demand is another question..
By adding to transaction repository, do you mean JBossTS object store? I can fetch the transaction from the database, but I am not sure whether we can merge with JBossTS (if I am not mis-reading your lines)
Forget the JBoss TS Object Store - we are not using it as already explained.
For TransactionRepository I mean org.jboss.messaging.core.tx.TransctionRepository - this is the class that holds the JBoss Messaging local transactions.
It's probably best to "reconstitute" prepared txs from the db on node startup. -
33. Re: XARecovery: Messaging integration with JBoss Transaction
Most of the plumbing should be in place (although untested).
Just need to make sure the transaction repository is populated with prepared txs from the db on node recovery. -
34. Re: XARecovery: Messaging integration with JBoss Transaction
There is the issue that the xaresource needs to contact the possibly remote jms server in order to actually perform recovery.
I'm beginning to think that making it serializable as Mark suggests is the right way to go. The serializable XAResource knows the jms server to contact.
If the server hasn't been brought back up then the remote call will fail. (Not sure the best way to signal this to the transaction manager - just throw an exception from recover?).
I'm going to think about this some more tomorrow since I have quite bad jet lag right now :) -
35. Re: XARecovery: Messaging integration with JBoss Transaction
You only need to contact the remote server when the transaction manager instructs the XAResource to either commit or roll back. At that stage, you can throw an XAException with the right error code (XAER_RMFAIL) and the transaction service will simply try again later.
-
36. Re: XARecovery: Messaging integration with JBoss Transaction
This is what I think:
Application Server Scenario with JBossTS:
1. A client calls a transactional method. The method updates a Database and sends a message to a JMS Server in one tx
2. As we are using the JBossTS and implicitly the Transaction Manager (in case of a container managed tx), the enlistment of the resources takes place and will be logged by the tx manager in its own log (assuming the resource to adhere to XA Spec).
From TM's point of view, the JBossMessaging is another resource provider, so the enlistment, start, end, delistment of the resource will follow accordignly.
We can expect the TM to log the information about the participants and will use them during the recovery phase.
I think the missing piece is - can the JBossMessaging XAResource be resurrected by the transaction manager to call recovery on it? (by making this serializable might give enough information to the recovery module?)
Also, how does the TransactionManager treat non serialized XAResources? -
37. Re: XARecovery: Messaging integration with JBoss Transaction
Ok, finally I have found some time to read (skim) the JBoss Transactions documentation :)
It's my understanding that making the XAResource serializable is only useful for "transaction iniitiated" recovery - this is when the failure occurs after an entry has been written in the tx log.
It's possible that failure can occur before this - in which case JBoss TS has no XAresource to deserialize, so needs to call a XAResourceRecovery instance to get an XAResource instance and call recover() on it.
It seems to me that we cannot guarantee that failure occurs after a record has been written in the tx log, so transaction initiated recovery is no good for us.
(Aside - Actually I am trying to think of a situation when you can guarantee that failure only occurs after a record has been written in the tx log and therefore it's appropriate to use transaction initiated recovery - maybe when it's all in one vm?)
So, if we're going to use resource initiated recovery then we need to implement XAResourceRecovery which should be fairly straightforward.
In the case the jms server node is not up, then XAResourceRecovery needs to signal to the TM that the node is not available - not sure how we do this - we can't wait until we have obtained the XAresource instance and call commit/rollback on it since we can't obtain the instance if the server is not up. -
38. Re: XARecovery: Messaging integration with JBoss Transaction
"mskonda" wrote:
I think the missing piece is - can the JBossMessaging XAResource be resurrected by the transaction manager to call recovery on it? (by making this serializable might give enough information to the recovery module?)
Yes, this is what remains to be done.
We are currently discussing whether it's appropriate to make it serializable and let JBoss TS do "transaction initiated recovery", or implement the XAResourceRecovery factory and do "resource initiated recovery". -
39. Re: XARecovery: Messaging integration with JBoss Transaction
"mskonda" wrote:
Also, how does the TransactionManager treat non serialized XAResources?
For non-serializable XAResources, you need to register an appropriate XAResourceRecovery instance with the recovery sub-system. -
40. Re: XARecovery: Messaging integration with JBoss Transaction
"timfox" wrote:
Ok, finally I have found some time to read (skim) the JBoss Transactions documentation :)
It's my understanding that making the XAResource serializable is only useful for "transaction iniitiated" recovery - this is when the failure occurs after an entry has been written in the tx log.
If you are using distributed transactions with JBossTS, then you'll probably be using interposition (have a subordinate coordinator on the same machine as the XAResource). In which case, if the XAResource is serializable, it'll be within the log of the subordinate transaction coordinator and will be used for bottom-up recovery too.
It's possible that failure can occur before this - in which case JBoss TS has no XAresource to deserialize, so needs to call a XAResourceRecovery instance to get an XAResource instance and call recover() on it.
No. XAResourceRecovery is only used if the XAResource isn't serializable. We only try to write the information about participants after prepare, so failures before this point won't be influenced by the serializability (of lack of it) of the XAResource.
It seems to me that we cannot guarantee that failure occurs after a record has been written in the tx log, so transaction initiated recovery is no good for us.
I think there may be a misunderstanding of what transaction initated recovery is for. It happens when/if the coordinator process/node crashed after prepare and before the transaction has fully committed. At that point, the log entry will contain references to all of the participants (or subordinate coordinators) and the recovery subsystem will ask them all to commit. It'll keep doing this until all participants have committed, at which point the log will be deleted from the store.
(Aside - Actually I am trying to think of a situation when you can guarantee that failure only occurs after a record has been written in the tx log and therefore it's appropriate to use transaction initiated recovery - maybe when it's all in one vm?)
Hopefully the comments above fix this for you :-)
So, if we're going to use resource initiated recovery then we need to implement XAResourceRecovery which should be fairly straightforward.
You don't need to use resource initiated recovery at all. In fact, some implementations of transactions don't support it. They should, because it gives you better failure recovery (faster).
In the case the jms server node is not up, then XAResourceRecovery needs to signal to the TM that the node is not available - not sure how we do this - we can't wait until we have obtained the XAresource instance and call commit/rollback on it since we can't obtain the instance if the server is not up.
Nope. XAResourceRecovery instances don't do any signalling. They are called by the recovery subsystem when it needs to get a handle on a new XAResource instance because the old one (the one it had before the failure) is no longer available (it wasn't able to be serialized to the log). They are driven entirely by the recovery system. -
41. Re: XARecovery: Messaging integration with JBoss Transaction
"mark.little@jboss.com" wrote:
"mskonda" wrote:
Also, how does the TransactionManager treat non serialized XAResources?
For non-serializable XAResources, you need to register an appropriate XAResourceRecovery instance with the recovery sub-system.
If you make your XAResource serializable, then you don't need to do anything else :-) You don't need to worry about XAResourceRecovery at all in that case. Hence the reason I'd recommend it, if possible. -
42. Re: XARecovery: Messaging integration with JBoss Transaction
"timfox" wrote:
So, if we're going to use resource initiated recovery then we need to implement XAResourceRecovery which should be fairly straightforward.
In the case the jms server node is not up, then XAResourceRecovery needs to signal to the TM that the node is not available - not sure how we do this - we can't wait until we have obtained the XAresource instance and call commit/rollback on it since we can't obtain the instance if the server is not up.
I did implement this recovery mechanism (created a MessagingXARecovery class that implements XAResourceRecovery). However, there is one main issue:
I am trying to create the XAResoruce using the XA connection factory obtained from the JNDI in the getXAResource() method.
The problem is - the method getXAResource() (once the hasMoreResouces() returns true) is called prior to application sever complete boot up as the recovery precedes other tasks. Ths fails as JNDI server is not yet up and references have not been populated.
I am not sure whether we can postpone the recovery manager until for few minutes or until the server is completey up -
43. Re: XARecovery: Messaging integration with JBoss Transaction
We're not (currently) using distributed transactions.
We have something like the following:
Transaction Manager on node A (probably an app server) in which we enlist XAresources corresponding to jms servers on (potentially) different nodes e.g B, C
Effectively the XAResource enlist at node A is a remote proxy to the actual resource manager on node B or C (for example).
When prepare is issued on the XAresource this cause a remote call to node B or C, where the state is stored in the jboss messaging transaction log (current ly the db).
If the information about participants is only stored after prepare on node A, then it seems to me that if the node A can crash after the state has been stored on node B but before it has been logged on A.
In such a case I don't understand how we can use transaction initiated recovery as you suggest, since if we do so, then the transaction manager will never know to rollback the transaction (since it has no record).
What am I missing here? -
44. Re: XARecovery: Messaging integration with JBoss Transaction
"mskonda" wrote:
"timfox" wrote:
So, if we're going to use resource initiated recovery then we need to implement XAResourceRecovery which should be fairly straightforward.
In the case the jms server node is not up, then XAResourceRecovery needs to signal to the TM that the node is not available - not sure how we do this - we can't wait until we have obtained the XAresource instance and call commit/rollback on it since we can't obtain the instance if the server is not up.
I did implement this recovery mechanism (created a MessagingXARecovery class that implements XAResourceRecovery). However, there is one main issue:
I am trying to create the XAResoruce using the XA connection factory obtained from the JNDI in the getXAResource() method.
The problem is - the method getXAResource() (once the hasMoreResouces() returns true) is called prior to application sever complete boot up as the recovery precedes other tasks. Ths fails as JNDI server is not yet up and references have not been populated.
I am not sure whether we can postpone the recovery manager until for few minutes or until the server is completey up
Recovery runs periodically, so a failure initially shouldn't be a problem.