1 2 Previous Next 28 Replies Latest reply: Jun 14, 2007 1:23 AM by Umakanth Srinivasan RSS

DataSource failover

Adrian Brock Master

This relates to the following task:
http://jira.jboss.com/jira/browse/JBAS-1391

Rather than discuss the design in JIRA we can discuss it here.

  • 1. Re: DataSource failover
    Adrian Brock Master

    For the initial implementation, I would like to see the connectionURL
    remain sticky until it needs to failover.
    That will reduce the problem of inconsistency where the replicating database
    has some latency in its replication or does not fully support distributing locking
    (maybe only commit time leading to excessive lazy failures).

  • 2. Re: DataSource failover
    Adrian Brock Master

    For a full implementation, I would like to see this exposed as a policy
    such that the deployer can choose the failover policy
    (and potentially load balancing policy if the db supports it correctly).

    This will require not just checking for errors at connection construction time,
    but also tracking failures from matchManagedConnections (checkValidConnection)
    and runtime errors checkError/ExceptionSorter.
    The ExceptionSorter would need to know the difference between
    a FATAL error (the connection is broken) versus a FAILOVER error
    (the database is broken).

    Extra consistency could also be added by extending the ConnectionEventListener
    interface with a propriatory interface, adding

    void connectionFailoverOccurred(ConnectionEvent event);
    


    This would allow the adapter to signal that the connection should be closed
    once the application has finished with it, as opposed to the
    connectionErrorOccurred which signals immediate destruction.

    With this extra method, the pluggable policy could choose a "first available"
    type policy.
    e.g.
    1) main db server fails so JBoss fails over to secondary/backup server
    2) it remains sticky on that server until the main server is back
    3) it then hands out connections from the main server and tells the
    listeners (JBoss's connection manager/pool) that the connections to the
    backup server should be closed when the applications have finished
    with the current requests.

  • 3. Re: DataSource failover
    Alexey Loubyansky Master

    The testcase for this issue is added

    package org.jboss.test.jca.test;
    public class HAConnectionFactoryUnitTestCase
    


    It's based on the HSQLDB. Also I didn't mention that it requires checking the validity of the connection before it is given to a caller. In the testcase, I use check-valid-connection-sql for that.

  • 4. Re: DataSource failover
    Alexey Loubyansky Master

    What about the introduction of the new ha-local-tx-datasource element?
    We could probably do some XSL tricks (like check whether url-delimeter is present and connection-url actually contains more than one url) and avoid the introduction of a new element.

  • 5. Re: DataSource failover
    Adrian Brock Master

     

    "alex.loubyansky@jboss.com" wrote:
    What about the introduction of the new ha-local-tx-datasource element?
    We could probably do some XSL tricks (like check whether url-delimeter is present and connection-url actually contains more than one url) and avoid the introduction of a new element.


    Yes, but I want it to be experimental for now.
    i.e. the user chooses they are going to use it.
    It should be documentated as such.

    It should be trivial in future releases to make ha-local and local do the same thing,
    once we are confident it is working correctly.

  • 6. Re: DataSource failover
    Adrian Brock Master

     

    "alex.loubyansky@jboss.com" wrote:
    Also I didn't mention that it requires checking the validity of the connection before it is given to a caller. In the testcase, I use check-valid-connection-sql for that.


    This is true regardless of HA.
    JDBC has no mechanism to signal a broken connection asynchronously.
    The only way to avoid broken connections being given to the application
    (where it will find it is broken on first use) is to have a validity checker
    that checks the connecton is not broken before it is handed out from the pool.
    http://www.jboss.org/wiki/Wiki.jsp?page=WhatHappensToJBossIfIRestartTheDatabase

  • 7. Re: DataSource failover
    Alexey Loubyansky Master

    Now URLs are selected with sticky round-robin algorithm. This is done in URLSelector which is an inner class of HALocalManagedConnectionFactory.

    Should I be warried about thread-safeness of the whole implementation?

  • 8. Re: DataSource failover
    Alexey Loubyansky Master

    It seems to be logical to be able to specify user/password per URL instead of using the same user/pwd for every url in the list?

    Then, as for me, it would make sense to still configure the ha datasource as a collection of datasources. Not using all the elements that are usually used to configure a local datasource but a subset (since, e.g. jndi-name would be specified only for the ha one).

    The same for XA. If I am going to specify properties per connection, I would collect them under a single parent element.

    What do you think?

  • 9. Re: DataSource failover
    Adrian Brock Master

     

    "alex.loubyansky@jboss.com" wrote:

    Should I be warried about thread-safeness of the whole implementation?


    Yes, the pool allows multiple threads to be dealing with connection construction/validation
    at the same time, upto the max-size which is the basis for the semaphore.

  • 10. Re: DataSource failover
    Adrian Brock Master

     

    "alex.loubyansky@jboss.com" wrote:
    It seems to be logical to be able to specify user/password per URL instead of using the same user/pwd for every url in the list?

    Then, as for me, it would make sense to still configure the ha datasource as a collection of datasources. Not using all the elements that are usually used to configure a local datasource but a subset (since, e.g. jndi-name would be specified only for the ha one).

    The same for XA. If I am going to specify properties per connection, I would collect them under a single parent element.

    What do you think?


    Unfortunatley, the JCA spec does not allow for complicated javabean construction


    <!--
    The element config-property-type contains the fully qualified Java
    type of a configuration property as required by ManagedConnection-
    Factory instance.

    The following are the legal values of config-property-type:
    java.lang.Boolean, java.lang.String, java.lang.Integer,
    java.lang.Double, java.lang.Byte, java.lang.Short,
    java.lang.Long, java.lang.Float, java.lang.Character

    Example: <config-property-type>java.lang.String</config-property-type>
    -->
    <!ELEMENT config-property-type (#PCDATA)>


    Like I said before, perhaps a better approach is to plugin a strategy/policy
    where this could be better defined?

    Don't forget the user/password are only used when there is no Subject,
    i.e. the admin has configured JAAS for the logins
    In fact, this whole issue is very complicated and ill defined/supported in the spec.

  • 11. Re: DataSource failover
    Alexey Loubyansky Master

    I've got first HA XA tests passing. Here is the ha-xa-datasource configuration I used for testing

     <ha-xa-datasource>
     <jndi-name>XADerbyDS</jndi-name>
     <track-connection-by-tx/>
     <isSameRM-override-value>false</isSameRM-override-value>
    <!-- <xa-datasource-class>org.apache.derby.jdbc.EmbeddedXADataSource</xa-datasource-class> -->
     <xa-datasource-class>org.jboss.test.cmp2.advanced.ejb.DerbyXADataSource</xa-datasource-class>
     <xa-datasource-property name="URL">derby/hatest1|derby/hatest2</xa-datasource-property>
     <xa-datasource-property name="User">sa</xa-datasource-property>
     <xa-datasource-property name="Password"></xa-datasource-property>
     <xa-datasource-property name="CreateDatabase">create</xa-datasource-property>
    
     <url-delimeter>|</url-delimeter>
     <check-valid-connection-sql>VALUES CURRENT TIMESTAMP</check-valid-connection-sql>
    
     <!-- corresponding type-mapping in the standardjbosscmp-jdbc.xml (optional) -->
     <metadata>
     <type-mapping>Derby</type-mapping>
     </metadata>
     </ha-xa-datasource>
    


    As you can see, it's basically the same as in the local case, i.e. URL contains a list of URLs and url-delimeter specifies the delimeter.

    The test is not added to the testsuite since we don't have derby in thirdparty (yet).

    Implementation notes. XAManagedConnectionFactory creates XADataSources lazily, i.e. not at deployment time but on the first request. In the HA implementation, I create XADataSources at deployment. Is this ok? Any special reason for lazy instantiation except for "i don't use it, so i don't need it to be created"?

  • 12. Re: DataSource failover
    Alexey Loubyansky Master

    Adrian, what else should be done here for the 4.0.2 release?
    The XA HA stuff is only in HEAD currently.

  • 13. Re: DataSource failover
    Adrian Brock Master

    You can't just have

    <url-delimiter>

    You need to identify which of the xadatasource properties identifies the server,
    e.g. for mssql it is "ServerName"

    <url-property>ServerName</url-property>
    <url-delimiter>|</url-delimiter>
    


    This is not an issue for local since the connection url is always in the same config.

  • 14. Re: DataSource failover
    Adrian Brock Master

    The XADataSource should just be a factory, it should be the connections that are the
    heavy weight objects.
    I'd guess this is the usual "You don't pay for what you don't use"
    philosophy that JBoss normally does.

    It looks like the only change you made to the existing stuff was to make
    the xa props a protected field, so I don't see why it would be problem backporting
    to JBoss4 (assuming you didn't break the stylesheet :-).

    Do you have it documentated somewhere on the WIKI (as experimental)?
    http://www.jboss.org/wiki/Wiki.jsp?page=JBossJCA
    and add a subtask here:
    http://jira.jboss.com/jira/browse/JBAS-1597

1 2 Previous Next