1 2 3 4 5 6 Previous Next 80 Replies Latest reply on Jan 27, 2010 4:51 PM by marklittle Go to original post
      • 75. Re: Jboss transaction recovery issue
        marklittle

        OK Sal, let's just concentrate on the DB2 case for now.

         

        So as we saw a while back, in this situation you have a stale XAResource and initial recovery fails with the warning about a non-serializable XAResource. As I explained, this goes away eventually because recovery gets a new XAResource to replace the old one. Do you know where that instance is coming from? Do you provide it through the XAResourceRecovery interface?

         

        The reason I ask is because it does seem from what you are saying that the subsequent instance is causing the problem: if it gives an RMFAIL error code then we're kicked into heuristic mode again when in fact we probably shouldn't be this time.

        • 76. Re: Jboss transaction recovery issue
          scarceller

          mark.little@jboss.com wrote:

           

          OK Sal, let's just concentrate on the DB2 case for now.

           

          So as we saw a while back, in this situation you have a stale XAResource and initial recovery fails with the warning about a non-serializable XAResource. As I explained, this goes away eventually because recovery gets a new XAResource to replace the old one. Do you know where that instance is coming from? Do you provide it through the XAResourceRecovery interface?

           

          The reason I ask is because it does seem from what you are saying that the subsequent instance is causing the problem: if it gives an RMFAIL error code then we're kicked into heuristic mode again when in fact we probably should be this time.

          Mark,

           

          I'm using the AppServerJDBCXARecovery.class that comes with V5 AS. I simply setup recovery for DB2 and Oracle via this class by adding XAResourceRecovery1 and XAResourceRecovery2 properties to the jbossts-properties.xml file. I know I have this setup correctly because it can do rollbacks on xids found in either DB that are not in the ObjectStore.

           

          If you look through the log I sent you'll see the issue is the 2nd time it tries to replay the commit during retry, it simply gets a XAER_RMFAIL and then I back where I started from with this xid marked as Hazard and the in-doubt still in DB2 in need of commit.

           

          The trouble here is that this xid is NOT in a Huristic state in DB2 it is in an in-doubt state waiting for the TM to tell it what to do. The TM simply never gets back to it to commit it.

          • 77. Re: Jboss transaction recovery issue
            scarceller

            Mark,

             

            Did another test for the DB2 in-doubts:

             

            • This time I pulled the network on DB2 and I had 5 in-doubts.
            • Left DB2 disconnected
            • Waited for all Trans to finish/timeout in the AS
            • Then simply did a gracefull shutdown on the AS
            • Check the 2 DBs and DB2 was short 5 orders when compared to Oracle but DB2 had 5 in-doubts so all is OK so far.
            • Still had DB2 disconnected
            • Re-started the AppServer, in this case the DB2 JDBC pool does NOT (and can not) have any stale connections
            • Waited for server to be fully started.
            • Then reconnected DB2
            • This time the TM AtomicRecovery successfully commited the 5 in-doubts!

             

            After doing this test I'm rather certain the issue is stale connections in the pool during TM recovery result in XAER_RMFAIL. This is not good because this causes recovery to bail out early when it should not. In this case the XAER_RMFAIL really should not set the xid aside as a hazard because the real issue is simply a stale connection.

             

            I can't spend any more time on this for a while now but I feel I now understand the issues.

             

            Thanks.

            • 78. Re: Jboss transaction recovery issue
              marklittle
              Sal, it does sounds like your assessment is right. We're making changes in the Xid implementation we use so that this case will be easier to detect in the future. In the meantime if you feel confident that all RMFAIL returns can be retried in your environment then you can always modify the TS codebase.
              • 79. Re: Jboss transaction recovery issue
                scarceller

                Here is the exception I get from the jbosstools GUI when I try to drill into the xid to get it's details.

                 

                --------------------------------------------------------

                2010-01-27 14:47:18,890 ERROR [STDERR] (AWT-EventQueue-0) java.lang.NullPointerException
                2010-01-27 14:47:18,890 ERROR [STDERR] (AWT-EventQueue-0)     at com.arjuna.ats.tools.objectstorebrowser.frames.BrowserFrame.updateTreePath(BrowserFrame.java:499)
                2010-01-27 14:47:18,890 ERROR [STDERR] (AWT-EventQueue-0)     at com.arjuna.ats.tools.objectstorebrowser.frames.BrowserFrame.valueChanged(BrowserFrame.java:450)
                2010-01-27 14:47:18,890 ERROR [STDERR] (AWT-EventQueue-0)     at javax.swing.JTree.fireValueChanged(Unknown Source)
                2010-01-27 14:47:18,890 ERROR [STDERR] (AWT-EventQueue-0)     at javax.swing.JTree$TreeSelectionRedirector.valueChanged(Unknown Source)
                2010-01-27 14:47:18,890 ERROR [STDERR] (AWT-EventQueue-0)     at javax.swing.tree.DefaultTreeSelectionModel.fireValueChanged(Unknown Source)
                2010-01-27 14:47:18,906 ERROR [STDERR] (AWT-EventQueue-0)     at javax.swing.tree.DefaultTreeSelectionModel.notifyPathChange(Unknown Source)
                2010-01-27 14:47:18,906 ERROR [STDERR] (AWT-EventQueue-0)     at javax.swing.tree.DefaultTreeSelectionModel.setSelectionPaths(Unknown Source)
                2010-01-27 14:47:18,906 ERROR [STDERR] (AWT-EventQueue-0)     at javax.swing.tree.DefaultTreeSelectionModel.setSelectionPath(Unknown Source)
                2010-01-27 14:47:18,906 ERROR [STDERR] (AWT-EventQueue-0)     at javax.swing.JTree.setSelectionPath(Unknown Source)
                2010-01-27 14:47:18,906 ERROR [STDERR] (AWT-EventQueue-0)     at javax.swing.plaf.basic.BasicTreeUI.selectPathForEvent(Unknown Source)
                2010-01-27 14:47:18,906 ERROR [STDERR] (AWT-EventQueue-0)     at javax.swing.plaf.basic.BasicTreeUI$Handler.handleSelection(Unknown Source)
                2010-01-27 14:47:18,906 ERROR [STDERR] (AWT-EventQueue-0)     at javax.swing.plaf.basic.BasicTreeUI$Handler.mousePressed(Unknown Source)
                2010-01-27 14:47:18,906 ERROR [STDERR] (AWT-EventQueue-0)     at java.awt.Component.processMouseEvent(Unknown Source)
                2010-01-27 14:47:18,906 ERROR [STDERR] (AWT-EventQueue-0)     at javax.swing.JComponent.processMouseEvent(Unknown Source)
                2010-01-27 14:47:18,906 ERROR [STDERR] (AWT-EventQueue-0)     at java.awt.Component.processEvent(Unknown Source)
                2010-01-27 14:47:18,906 ERROR [STDERR] (AWT-EventQueue-0)     at java.awt.Container.processEvent(Unknown Source)
                2010-01-27 14:47:18,906 ERROR [STDERR] (AWT-EventQueue-0)     at java.awt.Component.dispatchEventImpl(Unknown Source)
                2010-01-27 14:47:18,906 ERROR [STDERR] (AWT-EventQueue-0)     at java.awt.Container.dispatchEventImpl(Unknown Source)
                2010-01-27 14:47:18,906 ERROR [STDERR] (AWT-EventQueue-0)     at java.awt.Component.dispatchEvent(Unknown Source)
                2010-01-27 14:47:18,906 ERROR [STDERR] (AWT-EventQueue-0)     at java.awt.LightweightDispatcher.retargetMouseEvent(Unknown Source)
                2010-01-27 14:47:18,921 ERROR [STDERR] (AWT-EventQueue-0)     at java.awt.LightweightDispatcher.processMouseEvent(Unknown Source)
                2010-01-27 14:47:18,921 ERROR [STDERR] (AWT-EventQueue-0)     at java.awt.LightweightDispatcher.dispatchEvent(Unknown Source)
                2010-01-27 14:47:18,921 ERROR [STDERR] (AWT-EventQueue-0)     at java.awt.Container.dispatchEventImpl(Unknown Source)
                2010-01-27 14:47:18,921 ERROR [STDERR] (AWT-EventQueue-0)     at java.awt.Window.dispatchEventImpl(Unknown Source)
                2010-01-27 14:47:18,921 ERROR [STDERR] (AWT-EventQueue-0)     at java.awt.Component.dispatchEvent(Unknown Source)
                2010-01-27 14:47:18,921 ERROR [STDERR] (AWT-EventQueue-0)     at java.awt.EventQueue.dispatchEvent(Unknown Source)
                2010-01-27 14:47:18,921 ERROR [STDERR] (AWT-EventQueue-0)     at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
                2010-01-27 14:47:18,921 ERROR [STDERR] (AWT-EventQueue-0)     at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
                2010-01-27 14:47:18,921 ERROR [STDERR] (AWT-EventQueue-0)     at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
                2010-01-27 14:47:18,921 ERROR [STDERR] (AWT-EventQueue-0)     at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
                2010-01-27 14:47:18,921 ERROR [STDERR] (AWT-EventQueue-0)     at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
                2010-01-27 14:47:18,921 ERROR [STDERR] (AWT-EventQueue-0)     at java.awt.EventDispatchThread.run(Unknown Source)

                --------------------------------------------------------------

                 

                Any idea why I can't drill into the xid using the GUI?

                 

                Thanks.

                • 80. Re: Jboss transaction recovery issue
                  marklittle
                  Can you take this to a new forum entry so we can track it separately? Thanks.
                  1 2 3 4 5 6 Previous Next