1 2 Previous Next 29 Replies Latest reply on Feb 12, 2014 12:02 PM by tomjenkinson Go to original post
      • 15. Re: Re: XAER_DUPID: The XID already exists (even with new mysql connector)
        gytis

        It looks like it was first introduced by you, Mark, and it was correct: FishEye: Diff - JBossTransactions/trunk/ArjunaCore/arjuna/classes/com/arjuna/ats/arjuna/xa/XID.java . Then it was updated by Jonathan twice: FishEye: Diff - JBossTransactions/trunk/ArjunaCore/arjuna/classes/com/arjuna/ats/arjuna/xa/XID.java and FishEye: Diff - JBossTransactions/trunk/ArjunaCore/arjuna/classes/com/arjuna/ats/arjuna/xa/XID.java . Last one is basically the same as it is now. Finally it was moved from ArjunaCore to ArjunaJTA.

        • 16. Re: Re: XAER_DUPID: The XID already exists (even with new mysql connector)
          marklittle

          The class was written by me (around 1999), so I do remember it. However, the condition within the createXid isn't something I recall.

          • 17. Re: XAER_DUPID: The XID already exists (even with new mysql connector)
            marklittle

            For instance, in version 4.2.3 it's much simpler (and the call to get_xid below was just to convert a Uid):

             

            private final Xid createXid (boolean branch, XAModifier theModifier)

                    {

                            Xid jtaXid = baseXid();

             

                            if (jtaXid != null)

                                    return jtaXid;

             

                            try

                            {

                                    com.arjuna.ats.arjuna.xa.XID jtsXid = _theTransaction.get_xid(branch);

                                    com.arjuna.ats.arjuna.xa.XID tempXid = new com.arjuna.ats.arjuna.xa.XID();

             

                                    tempXid.formatID = jtsXid.formatID;

                                    tempXid.gtrid_length = jtsXid.gtrid_length;

                                    tempXid.bqual_length = jtsXid.bqual_length;

                                    System.arraycopy(jtsXid.data, 0, tempXid.data, 0, tempXid.data.length);

             

              jtaXid = new com.arjuna.ats.jta.xa.XidImple(tempXid);

             

                                    if (theModifier != null)

                                    {

                                            try

                                            {

                                                    jtaXid = theModifier.createXid((com.arjuna.ats.jta.xa.XidImple) jtaXid);

                                            }

                                            catch (Exception e)

              {

                                                    e.printStackTrace();

                                            }

                            }

             

                                    return jtaXid;

              }

            catch (Exception e)

                            {

                                    e.printStackTrace();

                            }

             

                            return null;

                    }

            • 18. Re: Re: XAER_DUPID: The XID already exists (even with new mysql connector)
              marklittle

              Talk with Jonathan, because a quick scan through the history shows that a lot of this was changed when he added the EIS name support.

              • 19. Re: XAER_DUPID: The XID already exists (even with new mysql connector)
                tomjenkinson

                Just to confirm, my reading of the Nicholas' message was that point 3 was about the debug in XID::toString() which is what I was suggesting Nicolas raise a PR for.

                 

                I will chat to Paul and Jonathan about the createXid method.

                • 20. Re: XAER_DUPID: The XID already exists (even with new mysql connector)
                  marklittle

                  ok

                  • 21. Re: Re: Re: XAER_DUPID: The XID already exists (even with new mysql connector)
                    paul.robinson

                    Nicolas,

                    Tom Jenkinson wrote:

                     

                    4. We will need to get Paul Robinson and Gytis Trikleris to take a look at this, but conceptually something like "if supportedFormatIds.contains(formatID)" where supportFormatIds = new ArrayList(new Integer[{131077}, {131080}]); might work. That being said, its possible that bridging intentionally does not allow the branch to be changed in which case one of those guys will be able to suggest a way forward.

                     

                    We've done some investigation on this and now have an idea of what is going on. First of all, here's some background...

                     

                    XTS manages a mapping between the incoming WS-AT transaction and the JTA transaction used by the application. This mapping is done in the TXBridge. It is named as such due to it bridging WS-AT transactions to and from JTA transactions. In reality these two types of transactions are mostly independent with a mapping between them stored in the TXBridge. In the case of an incoming WS-AT transaction, this mapping is keyed on the transaction ID of the WS-AT transaction with an association to the JTA XID. The format of XID used here is BridgeDurableParticipant rather than JTA. Two reasons for this are; 1) it helps to distinguish record types at recovery time and B) it prevents branching, which cannot be done in this scenario due to a limitation in JCA.

                     

                    So, your first service creates the bridge mapping and begins a new subordinate JTA transaction with a new XID. this mapping is stored in a static map and re-used every time a request comes in under the same WS-AT transaction (in that server instance). This is the right thing to do. However, in your case, the second request is targetted at a different application. We think this is causing a request to Narayana to enlist the MySQL resource for a second time. This is quite normal, and Narayana has code in place to detect already enlisted resources, preventing them being re-enlisted under the same XID. However, detecting this does require a conversation with the resource driver, relying on it to recognise already enlisted resources. We have had problems with this in the past. It's also possible that there is a bug in the Narayana code around how we detect RMs already enlisted.

                     

                    Firstly, can you provide us with the configuration of your datasource(s)? If you have one per application, please provide both.

                     

                    Secondly, we don't yet have a reproducible test case to hand, so it would help if you could do a little debug for us. Here's what we suggest:

                     

                    Breakpoint com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple#isNewRM and when it is called, just prior to the second call to enlist the MySQL resource, check:

                     

                    1) Line 1446: Enumeration el = _resources.keys(); this should contain the details of the previously enlisted resource. If it's empty, we have a problem.

                    2) Line 1454: if (x.isSameRM(xaRes)); This should return 'true', but we suspect it may return false.

                    3) Similarly check what happens when dealing with the '_duplicateResources' collection.

                     

                    This should tell us if it's a bug in the Narayana or the MySQL driver code.

                     

                    Let us know your results and we'll investigate further,

                     

                    Thanks,

                     

                    Paul.

                    • 22. Re: Re: Re: XAER_DUPID: The XID already exists (even with new mysql connector)
                      nbarrera

                      Thanks Paul Robinson and Gytis Trikleris,

                       

                      Your explanation is really clear.

                       

                      I 'll be able to work on the debugging you suggest on Monday, I 'll also post the datasources configuration then as I 'm out of the office for the rest of the week.

                       

                      Thanks again,

                       

                      Nicolas.-

                      • 23. Re: Re: Re: Re: XAER_DUPID: The XID already exists (even with new mysql connector)
                        nbarrera

                        Hi Paul,

                         

                        I 've done my homework

                         

                        Firstly, can you provide us with the configuration of your datasource(s)? If you have one per application, please provide both.

                         

                         

                         

                        As listed in the WildFly admin console:       

                         

                        Name:

                            jta1

                           

                        JNDI:

                            java:/datasources/jta1

                           

                        XA Data Source Class:

                            com.mysql.jdbc.jdbc2.optional.MysqlXADataSource

                           

                        Is enabled?:

                            true

                           

                         

                        Driver:

                            jbweb_1.war_com.mysql.jdbc.Driver_5_1

                           

                         

                        Share Prepared Statements:

                            false

                           

                         

                        Statement Cache Size:

                           0

                         

                        Key   Value

                         

                        url      jdbc:mysql://localhost/jta1

                        ----

                         

                        Name:   

                             jta2

                           

                        JNDI:

                            java:/datasources/jta2

                           

                        XA Data Source Class:

                            com.mysql.jdbc.jdbc2.optional.MysqlXADataSource

                           

                        Is enabled?:

                            true

                           

                        Driver:

                            jbweb_1.war_com.mysql.jdbc.Driver_5_1

                           

                        Share Prepared Statements:

                            false

                           

                        Statement Cache Size:

                            0

                         

                        Key    Value

                         

                        url       jdbc:mysql://localhost/jta2

                         

                        --------

                         

                        Secondly, we don't yet have a reproducible test case to hand, so it would help if you could do a little debug for us. Here's what we suggest:

                         

                        Breakpoint com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple#isNewRM and when it is called, just prior to the second call to enlist the MySQL resource, check:

                         

                        1) Line 1446: Enumeration el = _resources.keys(); this should contain the details of the previously enlisted resource. If it's empty, we have a problem.

                        2) Line 1454: if (x.isSameRM(xaRes)); This should return 'true', but we suspect it may return false.

                        3) Similarly check what happens when dealing with the '_duplicateResources' collection.

                         

                        This should tell us if it's a bug in the Narayana or the MySQL driver code.

                         

                         

                        I 've done the debug session you suggested and here 're the results:

                         

                        1)

                        Enumeration el = _resources.keys();

                         

                        _resources.keys() was not empty, I 'll be posting eclipse's debugger info about variables values and stuff.

                         

                        _resources.png

                         

                        2)

                        if (x.isSameRM(xaRes))

                         

                        well, this is returning FALSE  so you can now confirm your suspects. I 'll also attach debugger details on screenshots of surrounding variables values.

                         

                        xaRes variable values

                        xaRes.png

                         

                        x variable values

                         

                        x.png

                         

                        3)

                        Similarly check what happens when dealing with the '_duplicateResources' collection.

                         

                        _duplicateResources is empty.

                         

                         

                         

                        Hope that helps somehow, I 'm looking forward to hear your insights,

                         

                        thanks!

                        • 24. Re: XAER_DUPID: The XID already exists (even with new mysql connector)
                          nbarrera

                          Nicolas Barrera escribió:

                          2) After upgrading mysql-connector, I could see a difference in the logs where now I was seeing the correct error message XAER_DPUID but the problem was that while serviceB was trying to make XA start and mysql said that the XID already existed jboss just threw the exception and didn't retry with another XID.

                           

                          The problem seemed to be that the error code was treated as failed XA code, and the connection was dropped instead of using it to retry a new XID.

                           

                          I had to modify the ironjacamar jdbc adapter method from the wildfly-8.0.0.CR1/modules/system/layers/base/org/jboss/ironjacamar/jdbcadapters/main/ironjacamar-jdbc-1.1.2.Final.jar jar file

                           

                          org.jboss.jca.adapters.jdbc.xa.XAManagedConnection.isFailedXA(int)

                          1. //old code 
                          2.  
                          3.    private boolean isFailedXA(int errorCode) 
                          4.    { 
                          5.       return !(errorCode >= XAException.XA_RBBASE && errorCode < XAException.XA_RBEND); 
                          6.    } 

                           

                          1. //modified code 
                          2.  
                          3.    private boolean isFailedXA(int errorCode) 
                          4.    { 
                          5.       return errorCode != XAException.XAER_DUPID && !(errorCode >= XAException.XA_RBBASE && errorCode < XAException.XA_RBEND); 
                          6.    } 

                           

                          1. //older code (from grepcode's version 1.1.0.Final http://grepcode.com/file/repository.jboss.org/nexus/content/repositories/releases/org.jboss.ironjacamar/ironjacamar-jdbc/1.1.0.Final/org/jboss/jca/adapters/jdbc/xa/XAManagedConnection.java#XAManagedConnection.isFailedXA%28int%29
                          2.  
                          3.    private boolean isFailedXA(int errorCode) 
                          4.    { 
                          5.       return (errorCode == XAException.XAER_RMERR || errorCode == XAException.XAER_RMFAIL);      
                          6.    } 

                           

                          With "old code" running XAER_DPUID would return true so the connection is discarded.

                           

                          With "modified code" XAER_DPUID would return false, so the connection is not discarded rather an XAException is thrown, that lets another object the oportuninty to recover from that exception by retrying.

                           

                          With "older code" XAER_DPUID would also return false, so.. I don't understand the reason for that change but that change had impact on the TransactionImple code.

                           

                          Tom Jenkinson escribió:

                           

                          ...

                          2. Sounds like a JCA bug, you could raise that over here: https://issues.jboss.org/browse/JBJCA
                          ...
                          Tom

                           

                          Hi Tom, I was thinking about this comment you made to me and I 'm not really sure if this is a bug in the ironjacamar project..., after all the problem could be that the mysql driver or xts code (this is currently being studied by Paul)  has a problem that generates this

                          scenario where a duplicate id is being used and ironjacamar could be reacting correctly by discarding a connection with that kind of error.

                           

                          I think I will first start a discussion on the ironjacamar forum before filing a JIRA for them, what is in discussion here is if the DPUID error is "fatal enough" to discard that connection knowing that discarding the connection will prevent the xts retry code from execution.

                           

                          I 've been searching in the ironjacamar changelog and found that changes in their code had a related JIRA issue:

                          [JBAS-3336] Destroy connections after a ResourceManager failure - JBoss Issue Tracker

                           

                          I 've also re tested my scenario with the old version of ironjacamar and currently, provided that I 've made other changes to the xts code mentioned above, as a new XID is used for the second webservice call no retries are needed and no changes to ironjacamar jdbc adapter are needed!

                           

                          I hope I was somehow clear, I think I did a quite messy post :S sorry for that.

                          • 25. Re: XAER_DUPID: The XID already exists (even with new mysql connector)
                            paul.robinson

                            Nicolas,

                             

                            I'm pretty sure I know what's going on now.

                             

                            You have two datasources registered for the same MySQL server. This might be a valid thing to do, I don't know. However, I do think it's confusing the 'isSameRM' implementation:

                             

                            When you checked "Enumeration el = _resources.keys();" you found the details of the first MySQL resource. Shortly after 'isSameRM' is called to compare that resource to the second MySQL resource. Despite it being the same physical MySQL resource, isSameRM is claiming that it's a different one, so we go ahead and try to call 'start' for a second time on that resource, which is invalid and causes the exception you see.

                             

                            As 'isSameRM' is implemented by the MySQL driver, I think that is where the issue is. Unless of course it's not valid to have two datasources for the same physical server. Even if the latter is the case, I think that maybe the MySQL driver should detect this.

                             

                            I think the next step is to:

                             

                            1) Try using the same datasource configuration for both applications (if you can for your use-case). If you configure it in the standalone.xml, it will be available to both applications. Even if this is not a long term solution for you, it would be nice to try it out in order to gain some knowledge about the issue.

                            2) Debug the isSameRM method in the MySQL driver to see why it's not detecting this dup resource, and then file an issue with MySQL.

                             

                            Paul.

                            • 26. Re: XAER_DUPID: The XID already exists (even with new mysql connector)
                              marklittle
                              • 27. Re: Re: XAER_DUPID: The XID already exists (even with new mysql connector)
                                nbarrera

                                Hi Paul,

                                 

                                1) Try using the same datasource configuration for both applications (if you can for your use-case). If you configure it in the standalone.xml, it will be available to both applications. Even if this is not a long term solution for you, it would be nice to try it out in order to gain some knowledge about the issue.

                                Sorry, I can't merge those two databases in the same database.

                                 

                                2) Debug the isSameRM method in the MySQL driver to see why it's not detecting this dup resource, and then file an issue with MySQL.

                                 

                                Paul.

                                 

                                I 've seen the isSameRM method in the mysql driver code and it seems to be asking if the two connections are the same. It's not surprising that the method returns false as in fact the two connections aren't equal. They differ in the database schema name.

                                 

                                public boolean isSameRM(XAResource xares) throws XAException {
                                    if (xares instanceof MysqlXAConnection) {
                                            return this.underlyingConnection.isSameResource(((MysqlXAConnection) xares).underlyingConnection); 
                                    } 
                                    return false;
                                }
                                

                                 

                                 

                                public boolean isSameResource(Connection otherConnection) {
                                    synchronized (getConnectionMutex()) {
                                        if (otherConnection == null) {
                                            return false;
                                        }
                                      
                                        boolean directCompare = true;
                                      
                                        String otherHost = ((ConnectionImpl)otherConnection).origHostToConnectTo;
                                        String otherOrigDatabase = ((ConnectionImpl)otherConnection).origDatabaseToConnectTo;
                                        String otherCurrentCatalog = ((ConnectionImpl)otherConnection).database;
                                      
                                        if (!nullSafeCompare(otherHost, this.origHostToConnectTo)) {
                                            directCompare = false;
                                        } else if (otherHost != null && otherHost.indexOf(',') == -1 &&
                                                otherHost.indexOf(':') == -1) {
                                            // need to check port numbers
                                            directCompare = (((ConnectionImpl)otherConnection).origPortToConnectTo ==
                                                this.origPortToConnectTo);
                                        }
                                      
                                        if (directCompare) {
                                            if (!nullSafeCompare(otherOrigDatabase, this.origDatabaseToConnectTo)) {
                                                directCompare = false;
                                            } else if (!nullSafeCompare(otherCurrentCatalog, this.database)) {
                                                directCompare = false;
                                            }
                                        }
                                
                                        if (directCompare) {
                                            return true;
                                        }
                                      
                                        // Has the user explicitly set a resourceId?
                                        String otherResourceId = ((ConnectionImpl)otherConnection).getResourceId();
                                        String myResourceId = getResourceId();
                                      
                                        if (otherResourceId != null || myResourceId != null) {
                                            directCompare = nullSafeCompare(otherResourceId, myResourceId);
                                          
                                            if (directCompare) {
                                                return true;
                                            }
                                        }
                                      
                                        return false;
                                    }
                                }
                                

                                 

                                 

                                Now, I don't know if this implementation is correct or not as I don't have that much jca knowledge. In fact I 'm confused because the first method is asking if the Resource Manager is the same, but in it's implementation only returns true when *resources* are the same! (this sounds weird).

                                Could you give me some thoughts on this implementation of isSameRM so I could go to the mysql forums with a little more background?

                                 

                                I 've been googling for jca resource managers but I haven't found much of it. Which would be the Resource Manager in my case? Should it be some  mysql driver object? Why would these two transactions (in different databases) share the same RM, is it because they use the same mysql rdbms server?

                                 

                                Provided that isNewRM returns a not null object the next step would be that TransactionImple executes a JOIN of the new transaction with the first one (using the same XID without problems)?

                                 

                                Finally, and regarding to the patches I 've posted before and applied to my code. As far as I 've understood from the link Mark Little suggested me to read nothing worse than a performance issue would happen if I use my patch (which generates a new xid for the second transaction). I mean, there isn't a risk of transactions inconsistency or data loss or something like that, isn't it?

                                 

                                thank you,

                                 

                                Nicolas.-

                                • 28. Re: Re: XAER_DUPID: The XID already exists (even with new mysql connector)
                                  nbarrera

                                  In case a XA START JOIN would be needed I guess mysql won't be able to handle it.

                                   

                                  from:

                                  http://dev.mysql.com/doc/refman/5.7/en/xa-restrictions.html

                                  For XA START, the JOIN and RESUME clauses are not supported.

                                  For XA END, the SUSPEND [FOR MIGRATE] clause is not supported.

                                   

                                   

                                  http://frankkieviet.blogspot.com.ar/2010/01/how-to-enlist-and-delist-xa-resources.html

                                   

                                   

                                  I guess I 'm in a trouble.

                                  • 29. Re: Re: XAER_DUPID: The XID already exists (even with new mysql connector)
                                    tomjenkinson

                                    Hi Nicolas,

                                     

                                    The restriction around creating multiple branches is imposed by Narayana itself in the bridging code. Normal JTA transactions do not have the limitation. Our bridging implementation uses existing code designed to work for standards based JCA inflow which does not allow the transaction manager to alter the branch qualifier as the API does not allow it to be aware of the "space" that it can utilize, we could look to modify this for your case.

                                     

                                    In the meantime, if you wanted to override the isSameRM behaviour you could wrap the XAResource yourself and return true, as you say though, we would call TMJOIN on it. As you say JOIN/RESUME are not supported, but whether they are silently ignored or an error is returned I don't know.

                                     

                                    You can do:

                                    xar =  mySQL.getXAResource();

                                    Xid xid = new XidImple();

                                    xar.start(TMSTART, xid);

                                    xar.start(TMJOIN, xid);

                                    And see what happens.

                                     

                                    One other thing I can think of is to override the .equals() operation too in order that we can drop out of the enlist without calling start to avoid that TMJOIN:

                                    narayana/ArjunaJTA/jta/classes/com/arjuna/ats/internal/jta/transaction/arjunacore/TransactionImple.java at master · jbos…

                                     

                                    The caveat: The impact of changing the behavior of the resource manager with both of those changes could be quite subtle and not-obvious so not recommended. I would suggest get the isSameRM, and equals changes to be made upstream.

                                     

                                    The other interesting point is that the webservices are possibly asynchronous? In which case if both services are invoked in parallel, can MySQL tolerate having mutliple threads operating on the same branch at the same time?

                                     

                                    Tom

                                    1 2 Previous Next