0 Replies Latest reply on May 24, 2010 9:30 AM by jbosse

    Bridge qos_once_and_only_once - XA Recovery - Lost Messages

    jbosse

      Hi all,

       

      i'm finding some trouble configuring XA Recovery and bridge with qos _once_and_only_once:

       

      In this example i've deployed a queue (queuea) on jbm1 bridged with a queue (queueb) on jbm2 using qos once_and_only_once.

       

      Following jbm 1.4.5 user guide, steps are :

       

      1) Source and destination are on different server -> need to enlist sending and consuming session in JTA Transaction

      2) Both supplied connection factories are XA

      3) Enable XA recovery with JBoss Transaction

       

       

      On jbm1 (ServerPeerID 49) , RemoteJMSProvider definition:

       

      <mbean code="org.jboss.jms.jndi.JMSProviderLoader" name="jboss.messaging:service=JMSProviderLoader,name=RemoteJMSProvider">
          <attribute name="ProviderName">RemoteJMSProvider</attribute>
          <attribute name="ProviderAdapterClass">org.jboss.jms.jndi.JNDIProviderAdapter</attribute>
          <!-- The combined connection factory -->
          <attribute name="FactoryRef">XAConnectionFactory</attribute>
          <!-- The queue connection factory -->
          <attribute name="QueueFactoryRef">XAConnectionFactory</attribute>
          <!-- The topic factory -->
          <attribute name="TopicFactoryRef">XAConnectionFactory</attribute>
          <attribute name="Properties">
             java.naming.factory.initial=org.jnp.interfaces.NamingContextFactory
             java.naming.factory.url.pkgs=org.jboss.naming:org.jnp.interfaces
             java.naming.provider.url=ip.address.jbm.2:1099
          </attribute>
        </mbean>

       

       

      queuea:

       

      <mbean code="org.jboss.jms.server.destination.QueueService" name="jboss.messaging.destination:service=Queue,name=queuea" xmbean-dd="xmdesc/Queue-xmbean.xml">
          <depends optional-attribute-name="ServerPeer">jboss.messaging:service=ServerPeer</depends>
          <depends>jboss.messaging:service=PostOffice</depends>
        </mbean>

       

       

      Bridge:

       

      <mbean code="org.jboss.jms.server.bridge.BridgeService" name="jboss.messaging:service=Bridge,name=queueab" xmbean-dd="xmdesc/Bridge-xmbean.xml">
          <depends optional-attribute-name="SourceProviderLoader">jboss.messaging:service=JMSProviderLoader,name=JMSProvider</depends>
          <depends optional-attribute-name="TargetProviderLoader">jboss.messaging:service=JMSProviderLoader,name=RemoteJMSProvider</depends>
          <attribute name="SourceDestinationLookup">/queue/queuea</attribute>
          <attribute name="TargetDestinationLookup">/queue/queueb</attribute>
          <attribute name="QualityOfServiceMode">2</attribute>
          <attribute name="MaxBatchSize">1</attribute>
          <attribute name="MaxBatchTime">1000</attribute>
          <attribute name="FailureRetryInterval">5000</attribute>
          <attribute name="MaxRetries">-1</attribute>
          <attribute name="AddMessageIDInHeader">false</attribute>
        </mbean>

       

      On jbm2 (ServerPeerID 51)  queueb:

       

      <mbean code="org.jboss.jms.server.destination.QueueService" name="jboss.messaging.destination:service=Queue,name=queueb" xmbean-dd="xmdesc/Queue-xmbean.xml">
          <depends optional-attribute-name="ServerPeer">jboss.messaging:service=ServerPeer</depends>
          <depends>jboss.messaging:service=PostOffice</depends>
        </mbean>

       

       

       

      Enable XA recovery with JBoss Transactions, added in jbossjta-properties.xml of jbm1 and jbm2:

       

      <property name="com.arjuna.ats.jta.recovery.XAResourceRecovery.JBMESSAGING1"
                 value="org.jboss.jms.server.recovery.MessagingXAResourceRecovery;java:/DefaultJMSProvider"/>

       

       

      Seems that sending messages to queuea and simulating a crash ( kill -9 pidjbm ) cause that some message are lost and not arrive on queueb.

       

      Step to reproduce:

       

      - started instances jbm1 and jbm2 (JBoss 4.2.3 with messaging 1.4.5, persistence on oracle db 11.2 ) on different linux server with queue and bridge definition

      - sending 1000 messaging on queuea with jmeter

      - while message are in transit from jbm1 to jbm2 , kill -9 jbm1

      - wait a couple of minutes

      - kill -9 jbm2

      - start  jbm1

      - start  jbm2

      - wait 10 minutes

      - looking on queueb via jmx-console there're 999 messages.

       

      On db seems there's a stuck transaction:

       

      SQL> select count(*) from jbm1.jbm_msg;

        COUNT(*)
      ----------
               1

      SQL> select count(*) from jbm1.jbm_msg_ref;

        COUNT(*)
      ----------
               0

      SQL> select count(*) from  jbm1.jbm_tx;

        COUNT(*)
      ----------
               0

      SQL>  select count(*) from  jbm2.jbm_msg;

        COUNT(*)
      ----------
            1000

      SQL> select count(*) from  jbm2.jbm_msg_ref;

        COUNT(*)
      ----------
            1000

      SQL> select count(*) from  jbm2.jbm_tx;

        COUNT(*)
      ----------
               1

       

       

      In jbm1 server.log see:

       

      2010-05-24 13:46:03,483 WARN  [com.arjuna.ats.jta.logging.loggerI18N] [com.arjuna.ats.internal.jta.resources.arjunacore.norecoveryxa] [com.arjuna.ats.interna
      l.jta.resources.arjunacore.norecoveryxa] Could not find new XAResource to use for recovering non-serializable XAResource < 131075, 29, 27, 1--3f578bcf:b0e5:4
      bfa6494:96a-3f578bcf:b0e5:4bfa6494:96b^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
      ^@^@^@^@^@^@^@^@^@^@^@^@^@ >

       

      or in another case:

       

      SQL> select count(*) from jbm1.jbm_msg;

        COUNT(*)
      ----------
               1

      SQL> select count(*) from jbm1.jbm_msg_ref;

        COUNT(*)
      ----------
               1

      SQL> select count(*) from jbm1.jbm_tx;

        COUNT(*)
      ----------
               1

      SQL> select count(*) from jbm2.jbm_msg;

        COUNT(*)
      ----------
            1000

      SQL> select count(*) from jbm2.jbm_msg_ref;

        COUNT(*)
      ----------
            1000

      SQL> select count(*) from jbm2.jbm_tx;

        COUNT(*)
      ----------
               1

       

      jbm1 server.log :

       

      2010-05-24 14:20:46,882 DEBUG [com.arjuna.ats.jta.logging.loggerI18N] [com.arjuna.ats.internal.jta.recovery.info.notrollback] Told not to rollback < 131075,
      1--3f578bcf:e8cc:4bfa6b9d:a2e, -3f578bcf:e8cc:4bfa6b9d:a2f >

       

      Same behaviour, send 1000 message on queuea, crash, 999 arrive on queueb.

       

      Seems that records remain forever on db  (no tansaction replication nor cleanup), suspect a configuration issue.

      Looking the steps on user guide and on the wiki http://community.jboss.org/wiki/JBMBridgeConfiguration i've not found the error.

       

      There're other steps to follow to configure bridge, qos once_and_only_once and xa recovery that i miss?

       

      Thanks in advance,

       

      Carlo