3 Replies Latest reply on Jun 4, 2013 10:27 AM by jobame

    Transaction recovery failing

    jobame

      Hello,

       

      in our application on JBoss AS 7 we get an ORA-01591 error (Oracle 11g, datasource configuration see below) because a transaction lock is still held in the database. However,

      - we don't know how to find out why that lock is still held

      - the recovery by JBoss seems to fail.

       

      We googled and found a few links which didn't help us (e.g. https://community.jboss.org/wiki/TxNonSerializableXAResource or https://community.jboss.org/wiki/JBossTSRecovery). Some links only seem to point to a solution on the database side but the database/network had no problem during the given time. So, we assume that something can be done on JBoss side.

       

      How can this issue be solved?

       

      From the server.log

       

      29052013 13:10:08,188 WARN  [SqlExceptionHelper.java:logExceptions:143] SQL Error: 1591, SQLState: 72000

      [...]

      29052013 13:10:08,198 WARN  [TwoPhaseCoordinator.java:beforeCompletion:281] ARJUNA012125: TwoPhaseCoordinator.beforeCompletion - failed for SynchronizationImple< 0:ffff0a00218c:-13d832b:51a5e03b:8b9, org.hibernate.engine.transaction.synchronization.internal.RegisteredSynchronization@722d6636 >: javax.persistence.QueryTimeoutException: ORA-01591:

      lock held by in-doubt distributed transaction 3.1.138483.

      [...]

      29052013 13:11:09,692 WARN  [XAResourceRecord.java:restore_state:949] ARJUNA016037: Could not find new XAResource to use for recovering non-serializable XAResource XAResourceRecord < resource:null, txid:< formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff0a00218c:79e772b3:51a4525c:60, node_name=1, branch_uid=0:ffff0a00218c:79e772b3:51a4525c:cf, subordinatenodename=null, eis_name=java:/ds_crm24_oracle >, heuristic: TwoPhaseOutcome.FINISH_OK, product: Oracle/Oracle Database 11g Release 11.2.0.3.0 - 64bit Production, jndiName: java:/ds_crm24_oracle com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord@579cdf24 >

       

      Sometimes the following message is logged as well which according to https://community.jboss.org/wiki/TxNonSerializableXAResource appears because the resource wasn't Serializable. The solution would be "to provide an instance of a XAResourceRecovery implementation and tie it into the recovery process in order for this to occur". However, how is that done?

      29052013 10:44:08,378 WARN  [XAResourceRecord.java:topLevelCommit:527] ARJUNA016038: No XAResource to recover < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff0a00218c:79e772b3:51a4525c:60, node_name=1, branch_uid=0:ffff0a00218c:79e772b3:51a4525c:cf, subordinatenodename=null, eis_name=java:/ds_crm24_oracle >

       

      Datasource configuration

       

      <xa-datasource jta="true" jndi-name="java:/ds" pool-name="ab23Pool" enabled="true" use-java-context="true" use-ccm="true">
          <xa-datasource-property name="URL">
              jdbc:oracle:thin:@123.45.6.78:1521:AB23SAFO
          </xa-datasource-property>
          <xa-datasource-property name="User">
              SCHEMAAB
          </xa-datasource-property>
          <xa-datasource-property name="Password">
              PWDAB
          </xa-datasource-property>
          <driver>XAoracle</driver>
          <transaction-isolation>TRANSACTION_READ_COMMITTED</transaction-isolation>
          <xa-pool>
              <min-pool-size>10</min-pool-size>
              <max-pool-size>20</max-pool-size>
              <prefill>false</prefill>
              <use-strict-min>true</use-strict-min>
              <flush-strategy>FailingConnectionOnly</flush-strategy>
              <is-same-rm-override>false</is-same-rm-override>
              <interleaving>false</interleaving>
              <no-tx-separate-pools>true</no-tx-separate-pools>
              <pad-xid>false</pad-xid>
          </xa-pool>
          <validation>
              <valid-connection-checker class-name="org.jboss.jca.adapters.jdbc.extensions.oracle.OracleValidConnectionChecker"/>
              <stale-connection-checker class-name="org.jboss.jca.adapters.jdbc.extensions.oracle.OracleStaleConnectionChecker"/>
              <exception-sorter class-name="org.jboss.jca.adapters.jdbc.extensions.oracle.OracleExceptionSorter"/>
          </validation>
          <timeout>
              <set-tx-query-timeout>true</set-tx-query-timeout>
              <blocking-timeout-millis>30000</blocking-timeout-millis>
              <idle-timeout-minutes>30</idle-timeout-minutes>
              <query-timeout>120</query-timeout>
              <use-try-lock>60</use-try-lock>
          </timeout>
      </xa-datasource>
      
        • 1. Re: Transaction recovery failing
          jesper.pedersen

          Remove "User" and "Password" and add a <security> element. And upgrade to WildFly 8 Alpha1.

           

          If you search for JBoss EAP and Oracle XA I'm sure that you will find a best practice setup.

          • 2. Re: Transaction recovery failing
            jobame

            What would that change on the transaction side?

            • 3. Re: Transaction recovery failing
              jobame

              We found the banal cause of the lock: a developer had a session open with a "not-yet-committed" transaction *argh* When that issue was solved we had the dba remove the locks and everything was fine again. Some uncertainty still remains though.

               

              Regarding the occasionally logged message we shut down the server, deleted the directory $JBOSS_HOME/standalone/data/tx-object-store/ShadowNoFileLockStore/defaultStore and then restarted the server.