2 Replies Latest reply on Oct 25, 2011 12:57 PM by sannegrinovero

    dynamic reconfiguration

    jhalliday

      The configuration mechanism in JBossTS has evolved over the years, most recently to use a set of javabeans to hold the myriad config values. There are a dozen or so EnvironmentBeans, roughly one per module, and a central repository (BeanPopulator) that holds a singleton instance of each. The singletons are instantiated on demand with initial values populated from an embedded properties file or system properties. These defaults can then be overridden programatically, either by a bean wiring framework using xml (as in older JBossAS transaction-jboss-beans.xml) or in java code using the bean's setter methods (as in AS6 ArjunaTransactionManagerService etc). So far so good - this approach provides type safety for the values as well as compatibility with modern DI frameworks and is a big improvement over what we has before.

       

      It still has one significant shortcoming though: once the transaction system is initialized, any subsequent config changes are ignored. That would not be so bad, except that the initialization can happen very early. Specifically we have a lot of static initializers that read and cache config values at class load time. e.g.

       

      {code}

          public class SomeThing

          {

              private static final boolean XA_TRANSACTION_TIMEOUT_ENABLED;

       

              static

              {

                  XA_TRANSACTION_TIMEOUT_ENABLED = jtaPropertyManager.getJTAEnvironmentBean().isXaTransactionTimeoutEnabled();

              }

       

              public void doThing()

              {

                  XA_TRANSACTION_TIMEOUT_ENABLED ? thing() : otherThing();

              }

          }

       

          public class jtaPropertyManager

          {

              public static JTAEnvironmentBean getJTAEnvironmentBean()

              {

                  return BeanPopulator.getDefaultInstance(JTAEnvironmentBean.class);

              }

          }

       

          public class BeanPopulator

          {

              private static final ConcurrentMap<String, Object> beanInstances = new ConcurrentHashMap<String, Object>();

       

              public static <T> T getDefaultInstance(Class<T> beanClass) throws RuntimeException {

                  return (T) beanInstances.get( beanClass.getName() );

              }

          }

      {code}

       

      This provides very fast runtime execution, but a call to e.g.

       

      {code}jtaPropertyManager.getJTAEnvironmentBean().setXaTransactionTimeoutEnabled(true);{code}

       

      must come before the classload of SomeThing if it's going to have any effect.

       

      So what are the alternatives?

       

      Option A:

       

      {code}

          public class SomeThing

          {

              public void doThing()

              {

                  if(jtaPropertyManager.getJTAEnvironmentBean().isXaTransactionTimeoutEnabled())

                  {

                      thing()

                  }

                  else

                  {

                      otherThing();

                  }

              }

          }

      {code}

       

      looks attractive at first glance. However the system is now so highly tuned that the runtime overhead of the underlying hashmap lookup for

      {code}beanInstances.get( beanClass.getName() );{code} is going to cause significant performance impact, see e.g. JBTM-853.

       

      Additionally, usage patterns of the form

       

      {code}

              public class SomeThing

              {

                  public void doThing()

                  {

                      if(jtaPropertyManager.getJTAEnvironmentBean().isXaTransactionTimeoutEnabled())

                      {

                          thing()

                      }

       

                      unrelatedStuff();

       

                      if(jtaPropertyManager.getJTAEnvironmentBean().isXaTransactionTimeoutEnabled())

                      {

                          relatedThing();

                      }

                  }

              }

      {code}

       

      will need to be hunted down and replaced with

       

      {code}

              public class SomeThing

              {

                  public void doThing()

                  {

                      boolean cachedConfig = jtaPropertyManager.getJTAEnvironmentBean().isXaTransactionTimeoutEnabled();

       

                      if(cachedConfig)

                      {

                          thing()

                      }

       

                      unrelatedStuff();

       

                      if(cachedConfig)

                      {

                          relatedThing();

                      }

                  }

              }

      {code}

       

      including all the more complex cases where the usages are in the same call chain but not necessarily the same method, class or even package.

       

      Option B:

       

      {code}

              public class SomeThing

              {

                  private final ConfigBeanHolder config;

       

                  public SomeThing(ConfigBeanHolder config)

                  {

                      this.config = config;

                  }

       

                  public void doThing()

                  {

                      if(config.getJTAEnvironmentBean().isXaTransactionTimeoutEnabled())

                      {

                          thing()

                      }

       

                      unrelatedStuff();

       

                      if(config.getJTAEnvironmentBean().isXaTransactionTimeoutEnabled())

                      {

                          relatedThing();

                      }

                  }

              }

      {code}

       

      which eliminates the expensive hashmap operation and potentially allows us to do nifty stuff like running multiple different configurations in parallel e.g.

       

      {code}

              jtaThing = new SomeThing(jtaConfig)

              jtsThing = new SomeThing(jtsConfig)

             

              if( needDistributedTx ) {

                  useThing( jtsThing );

              } else {

                  useThing( jtaThing );

              }

      {code}

       

      However, we would need to retrofit the entire codebase with suitable constructors as well as hunting down the repeated usages of the same parameter as mentioned for the previous option.

       

      Option C:

       

      {code}

          public class SomeThing

          {

              private static boolean XA_TRANSACTION_TIMEOUT_ENABLED;

       

              static

              {

                  XA_TRANSACTION_TIMEOUT_ENABLED = jtaPropertyManager.getJTAEnvironmentBean().isXaTransactionTimeoutEnabled();

       

                  jtaPropertyManager.getJTAEnvironmentBean().registerPropertyChangeListener(

                      new PropertyChangeListener() {

                          public void propertyUpdated(String name, Object value) {

                              if("XA_TRANSACTION_TIMEOUT_ENABLED".equals(name) {

                                  XA_TRANSACTION_TIMEOUT_ENABLED = value;

                              }

                          }

                      }

              }

       

              public void doThing()

              {

                  XA_TRANSACTION_TIMEOUT_ENABLED ? thing() : otherThing();

              }

          }

      {code}

       

      which should be fairly quick at runtime but introduces a lot of boilerplate code and still suffers from some of the problems identified for other options.

       

       

      There are additional concerns that affect all options. For example, some properties cannot be considered in isolation as they interact with others:

       

      {code}

          public class SomeThing

          {

              private Object someProperty;

              private Object relatedProperty;

       

              public boolean isConfigValid()

              {

                  return someProperty.equals(relatedProperty);

              }

          }

      {code}

       

      maintaining validity in the face of mutability would require e.g.

       

      {code}

          public class SomeThing

          {

              private final ImmutableConfigHolder config;

             

              public void updateConfig(ImmutableConfigHolder replacementConfig) {

                  config = replacementConfig;

              }

             

              public boolean isConfigValid()

              {

                  return config.getSomeProperty().equals(config.getRelatedProperty());

              }

          }

      {code}

       

      which given the complex interrelationships between properties may bean we can't effectively decouple individual module's configs from one another any more and wind up with a single huge environment bean containing all the properties.

       

      Finally, we need to consider the problems that arise in trying to support a system that is reconfigurable. The content of the properties file is no longer sufficient to determine the runtime config. To diagnose or reproduce issues we'd additionally require either a comprehensive log of runtime property changes or individual error log messages to be retrofitted to contain a dump of any relevent config option values.

       

      The final option is a bit of a wildcard: don't change anything in the JBossTS codebase at all. Instead, use the JBossMC module system to effectively isolate multiple copies of the transaction system with different configs and hot swap between them to effect a config change. That's not without its problems, particularly where recovery is concerned, but they may turn out to be more tractable or at least less labor intensive than solutions that require wholesale audit and refactoring of the entire JBossTS codebase.  On the other hand if we plan to substantially refactor the codebase anyhow then making changes to the config system along the way may not be so bad.

       

      We can also potentially tackle the issue on a property by property basis, giving priority to those options for which dynamic reconfiguration has most benefit. Or go for a more limited objective of keeping the config static but delaying its effective freeze beyond classloading time, for example by requiring an explicit lifecycle with an instantiate/start step that snapshots the config values at that point and builds the system using them.

       

      Thoughts?

        • 1. Re: dynamic reconfiguration
          marklittle

          A variant of your second last option (priority to specific configuration properties): why not add a very simple event bus, whereby if a property is allowed to change dynamically, those components that want to be informed about the change have to register for a callback if it changes? (Basically cache invalidation.)

           

          I think that the number of options that should be dynamically reconfigurable is much less than the number of configuration options we have or are likely to ever have. So optimising for that approach would get my vote. And your last suggestion, of delaying the freeze point, may well be sufficient to do that too.

          • 2. Re: dynamic reconfiguration
            sannegrinovero

            While profiling HibernateOGM, the bottlenecks are currently in other projects code but while they are being solved the impact of these Map lookups in the transaction manager are getting more significant every day - currently consuming around 2% of total CPU time, I'd expect this ratio to grow soon as we solve other issues, so it would be nice to avoid the lookups as much as possible. Fixing JBTM-929 was a good improvement.

             

            About the eventbus, that seems interesting but the changed properties would need to set a memory barrier whose cost is paid for at each read, for example a volatile might become costly on non-x86 CPUs - is that something we should be concerned about?

             

            Hibernate 4 introduced an internal ConcurrentHashMap "ServiceRegistry" holding all the main components; this provides more flexibility but is introducing a new runtime cost: not significant when connecting to a slow JDBC database, more concerning when connecting to a data grid: often we have to cache references to components, losing the option to hot-swap them.

             

            Delaying the freeze point sounds nice to me, but I'm unfit to judge about the Transactions implementation internals. I've often used lazy static initialization to temporarily workaround performance problems, but it's hard to maintain and I've never suggested introducing it in a mission critical library.