1 2 Previous Next 28 Replies Latest reply on Jun 14, 2007 1:23 AM by kanth_seenu

    DataSource failover

      This relates to the following task:
      http://jira.jboss.com/jira/browse/JBAS-1391

      Rather than discuss the design in JIRA we can discuss it here.

        • 1. Re: DataSource failover

          For the initial implementation, I would like to see the connectionURL
          remain sticky until it needs to failover.
          That will reduce the problem of inconsistency where the replicating database
          has some latency in its replication or does not fully support distributing locking
          (maybe only commit time leading to excessive lazy failures).

          • 2. Re: DataSource failover

            For a full implementation, I would like to see this exposed as a policy
            such that the deployer can choose the failover policy
            (and potentially load balancing policy if the db supports it correctly).

            This will require not just checking for errors at connection construction time,
            but also tracking failures from matchManagedConnections (checkValidConnection)
            and runtime errors checkError/ExceptionSorter.
            The ExceptionSorter would need to know the difference between
            a FATAL error (the connection is broken) versus a FAILOVER error
            (the database is broken).

            Extra consistency could also be added by extending the ConnectionEventListener
            interface with a propriatory interface, adding

            void connectionFailoverOccurred(ConnectionEvent event);
            


            This would allow the adapter to signal that the connection should be closed
            once the application has finished with it, as opposed to the
            connectionErrorOccurred which signals immediate destruction.

            With this extra method, the pluggable policy could choose a "first available"
            type policy.
            e.g.
            1) main db server fails so JBoss fails over to secondary/backup server
            2) it remains sticky on that server until the main server is back
            3) it then hands out connections from the main server and tells the
            listeners (JBoss's connection manager/pool) that the connections to the
            backup server should be closed when the applications have finished
            with the current requests.

            • 3. Re: DataSource failover
              aloubyansky

              The testcase for this issue is added

              package org.jboss.test.jca.test;
              public class HAConnectionFactoryUnitTestCase
              


              It's based on the HSQLDB. Also I didn't mention that it requires checking the validity of the connection before it is given to a caller. In the testcase, I use check-valid-connection-sql for that.

              • 4. Re: DataSource failover
                aloubyansky

                What about the introduction of the new ha-local-tx-datasource element?
                We could probably do some XSL tricks (like check whether url-delimeter is present and connection-url actually contains more than one url) and avoid the introduction of a new element.

                • 5. Re: DataSource failover

                   

                  "alex.loubyansky@jboss.com" wrote:
                  What about the introduction of the new ha-local-tx-datasource element?
                  We could probably do some XSL tricks (like check whether url-delimeter is present and connection-url actually contains more than one url) and avoid the introduction of a new element.


                  Yes, but I want it to be experimental for now.
                  i.e. the user chooses they are going to use it.
                  It should be documentated as such.

                  It should be trivial in future releases to make ha-local and local do the same thing,
                  once we are confident it is working correctly.

                  • 6. Re: DataSource failover

                     

                    "alex.loubyansky@jboss.com" wrote:
                    Also I didn't mention that it requires checking the validity of the connection before it is given to a caller. In the testcase, I use check-valid-connection-sql for that.


                    This is true regardless of HA.
                    JDBC has no mechanism to signal a broken connection asynchronously.
                    The only way to avoid broken connections being given to the application
                    (where it will find it is broken on first use) is to have a validity checker
                    that checks the connecton is not broken before it is handed out from the pool.
                    http://www.jboss.org/wiki/Wiki.jsp?page=WhatHappensToJBossIfIRestartTheDatabase

                    • 7. Re: DataSource failover
                      aloubyansky

                      Now URLs are selected with sticky round-robin algorithm. This is done in URLSelector which is an inner class of HALocalManagedConnectionFactory.

                      Should I be warried about thread-safeness of the whole implementation?

                      • 8. Re: DataSource failover
                        aloubyansky

                        It seems to be logical to be able to specify user/password per URL instead of using the same user/pwd for every url in the list?

                        Then, as for me, it would make sense to still configure the ha datasource as a collection of datasources. Not using all the elements that are usually used to configure a local datasource but a subset (since, e.g. jndi-name would be specified only for the ha one).

                        The same for XA. If I am going to specify properties per connection, I would collect them under a single parent element.

                        What do you think?

                        • 9. Re: DataSource failover

                           

                          "alex.loubyansky@jboss.com" wrote:

                          Should I be warried about thread-safeness of the whole implementation?


                          Yes, the pool allows multiple threads to be dealing with connection construction/validation
                          at the same time, upto the max-size which is the basis for the semaphore.

                          • 10. Re: DataSource failover

                             

                            "alex.loubyansky@jboss.com" wrote:
                            It seems to be logical to be able to specify user/password per URL instead of using the same user/pwd for every url in the list?

                            Then, as for me, it would make sense to still configure the ha datasource as a collection of datasources. Not using all the elements that are usually used to configure a local datasource but a subset (since, e.g. jndi-name would be specified only for the ha one).

                            The same for XA. If I am going to specify properties per connection, I would collect them under a single parent element.

                            What do you think?


                            Unfortunatley, the JCA spec does not allow for complicated javabean construction


                            <!--
                            The element config-property-type contains the fully qualified Java
                            type of a configuration property as required by ManagedConnection-
                            Factory instance.

                            The following are the legal values of config-property-type:
                            java.lang.Boolean, java.lang.String, java.lang.Integer,
                            java.lang.Double, java.lang.Byte, java.lang.Short,
                            java.lang.Long, java.lang.Float, java.lang.Character

                            Example: <config-property-type>java.lang.String</config-property-type>
                            -->
                            <!ELEMENT config-property-type (#PCDATA)>


                            Like I said before, perhaps a better approach is to plugin a strategy/policy
                            where this could be better defined?

                            Don't forget the user/password are only used when there is no Subject,
                            i.e. the admin has configured JAAS for the logins
                            In fact, this whole issue is very complicated and ill defined/supported in the spec.

                            • 11. Re: DataSource failover
                              aloubyansky

                              I've got first HA XA tests passing. Here is the ha-xa-datasource configuration I used for testing

                               <ha-xa-datasource>
                               <jndi-name>XADerbyDS</jndi-name>
                               <track-connection-by-tx/>
                               <isSameRM-override-value>false</isSameRM-override-value>
                              <!-- <xa-datasource-class>org.apache.derby.jdbc.EmbeddedXADataSource</xa-datasource-class> -->
                               <xa-datasource-class>org.jboss.test.cmp2.advanced.ejb.DerbyXADataSource</xa-datasource-class>
                               <xa-datasource-property name="URL">derby/hatest1|derby/hatest2</xa-datasource-property>
                               <xa-datasource-property name="User">sa</xa-datasource-property>
                               <xa-datasource-property name="Password"></xa-datasource-property>
                               <xa-datasource-property name="CreateDatabase">create</xa-datasource-property>
                              
                               <url-delimeter>|</url-delimeter>
                               <check-valid-connection-sql>VALUES CURRENT TIMESTAMP</check-valid-connection-sql>
                              
                               <!-- corresponding type-mapping in the standardjbosscmp-jdbc.xml (optional) -->
                               <metadata>
                               <type-mapping>Derby</type-mapping>
                               </metadata>
                               </ha-xa-datasource>
                              


                              As you can see, it's basically the same as in the local case, i.e. URL contains a list of URLs and url-delimeter specifies the delimeter.

                              The test is not added to the testsuite since we don't have derby in thirdparty (yet).

                              Implementation notes. XAManagedConnectionFactory creates XADataSources lazily, i.e. not at deployment time but on the first request. In the HA implementation, I create XADataSources at deployment. Is this ok? Any special reason for lazy instantiation except for "i don't use it, so i don't need it to be created"?

                              • 12. Re: DataSource failover
                                aloubyansky

                                Adrian, what else should be done here for the 4.0.2 release?
                                The XA HA stuff is only in HEAD currently.

                                • 13. Re: DataSource failover

                                  You can't just have

                                  <url-delimiter>

                                  You need to identify which of the xadatasource properties identifies the server,
                                  e.g. for mssql it is "ServerName"

                                  <url-property>ServerName</url-property>
                                  <url-delimiter>|</url-delimiter>
                                  


                                  This is not an issue for local since the connection url is always in the same config.

                                  • 14. Re: DataSource failover

                                    The XADataSource should just be a factory, it should be the connections that are the
                                    heavy weight objects.
                                    I'd guess this is the usual "You don't pay for what you don't use"
                                    philosophy that JBoss normally does.

                                    It looks like the only change you made to the existing stuff was to make
                                    the xa props a protected field, so I don't see why it would be problem backporting
                                    to JBoss4 (assuming you didn't break the stylesheet :-).

                                    Do you have it documentated somewhere on the WIKI (as experimental)?
                                    http://www.jboss.org/wiki/Wiki.jsp?page=JBossJCA
                                    and add a subtask here:
                                    http://jira.jboss.com/jira/browse/JBAS-1597

                                    1 2 Previous Next