14 Replies Latest reply on Mar 2, 2011 2:35 PM by dward

    Configuration Thoughts

    kcbabo

      Wanted to share some initial thoughts around configuration and get some feedback.  There are actual two different flavors of configuration:

       

      1) System configuration - similar to jbossesb-properties.xml and the like in JBoss ESB.  This is configuration of the runtime and doesn't change for any given service deployment.

       

      2) Application configuration - this is more like jboss-esb.xml in JBoss ESB.   The application configuration consists of service definitions, binding details, service artifacts, etc.  Basically this is the information required to get your deployed services up and running.  This information does change between deployments and as such the application configuration is supplied with each deployment.

       

      I'm going to focus on application configuration in this post.  There are three primary consumers of the application configuration:

      • Individual components get information on what they are supposed to do (provide a service, consume a service) and how (configuration details).  For example, a SOAP gateway would be told that it will be exposing a WS endpoint for a SwitchYard service along with a pointer to the WSDL to use.
      • The core runtime will read policy, transformation requirements, wiring, and other service metadata and register appropriate handlers. 
      • Clients wishing to browse/discover service metadata based on the application configuration.  Note that this will most likely be stored outside of the core runtime in a repository and is not something that is generated as part of the deploy/lifecycle workflow of the core runtime.

       

      With all that out of the way, let's get into details via examples. I'm going to use a YAML-style format here because it's concise and makes me feel cool.

       

      We want to define a "HelloWorld" service.  Our service is implemented as a CDI bean and conforms to the "Greeting" service interface.

       

      service :
          name : HelloWorld
          interface : org.switchyard.examples.Greeting
          implementation : bean
      

       

      The implementation value is simply the container that holds the service implementation logic.  I could define another service implemented in Spring.

       

      service :
          name : RandomNumberGenerator
          interface : org.switchyard.examples.LuckyLottoNumbers
          implementation : spring
      

       

      Now let's say I want to include lucky lottery numbers with my greeting.  I can do that by adding a reference to RandomNumberGenerator from HelloWorld.

       

      service :
          name : HelloWorld
          interface : org.switchyard.examples.Greeting
          implementation : bean
          reference : lottoNumbers
              target : RandomNumberGenerator
      

       

      I can expose my HelloWorld service by adding a reference binding.

       

      service :
          name : HelloWorld
          interface : org.switchyard.examples.Greeting
          implementation : bean
          reference : lottoNumbers
              target : RandomNumberGenerator
      

       

       

      Or I can associate a reference binding without changing the original service definition.

       

      reference :
          target : HelloWorld
          binding : soap
              address : http://localhost:8080/HowdyYall

       

      The above examples are pretty basic, but cover some of the basic use cases of the ESB.  For the first iteration, we need the ability to support the following:

      • Provide a service locally by specifiying a service + implementation
      • Expose a local service to external consumers using a reference binding
      • Expose a remote service locally by specifying a service + binding
      • Consume a service locally by specifying a reference + target

       

      If you have a passing familiarity with SCA, you will notice that this looks similar to a number of concepts in the SCA assembly model specification.   This is deliberate and I think there are a number of ways we can integrate nicely with SCA without providing full SCA runtime implementation.  One way is to serialize our configuration model as SCDL (SCA configuration language).  Anyway, we can talk more about that later.  For now, I just wanted to get a conversation started on the important pieces w/r/t configuration.  I plan to take Tom's CDI work and code up some examples and the associated config for those examples.

        • 1. Re: Configuration Thoughts
          tfennelly

          Hey Keith.

           

          So in the case of the following example (from above):

           

          service :
              name : HelloWorld
              interface : org.switchyard.examples.Greeting
              implementation : bean
              reference : lottoNumbers
                  target : RandomNumberGenerator

           

          You're exposing a remote "RandomNumberGenerator" service locally as "HelloWorld", right?  In these case, where will the lower level details of how the local HelloWorld service (Greeting impl) integrates with RandomNumberGenerator service (LuckyLottoNumbers impl) live?

           

          Will we be looking at an initial deployment "container" as part of this work i.e. something that reads SCDL (or whatever you decide to use) and hooks up the doimain, registers the services etc etc?  Of course you could continue doing it in code... just wondering.

          • 2. Re: Configuration Thoughts
            tfennelly

            Of course I'm also interested in how this info will be made available at runtime.  Example... the TransformHandler I'm experimenting with needs access to Service interface info to get operation type info (input/output types).

            • 3. Re: Configuration Thoughts
              kcbabo

              In that particular example, the implementation of the HelloWorld service is a consumer of another service.  Logically, it's identical to the CDI example that you created where you have a bean annotated with @Service and a field within that bean is using @Service as an injection point for a reference to another service.  There's nothing inherent in a reference that indicates the target service is remote.  If you do plan on invoking the service through a gateway (e.g. it's a partner's web service endpoint), then you can specify a binding directly on the reference itself or the reference points to a declared service with a binding definition.

               

              Here's an example of how a reference could specify the binding for a remote service directly:

               

              reference: OtherService
                  binding: soap
                      wsdlAddress: http://somedude.com:8080/coolservice?wsdl
              

               

              Here's an example where a reference points to a service definition which contains remote binding details:

              reference: OtherService
                  target: SomeDudesService
              
              serivce:
                  name: SomeDudesService
                  binding:
                      soap:
                          endpoint: http://localhost:8080/coolservice 
              


              One of the primary advantages of specifying references is that it establishes a clear dependency graph for deployed services.  We can use this to perform deploy-time validation, ordered lifecycle startup, etc.

               

              Yes, we will need some type of thin deployment wrapper that recognizes a switchyard deployment, parses the config, and hands off configuration to appropriate components.

              • 4. Re: Configuration Thoughts
                kcbabo
                Of course I'm also interested in how this info will be made available at runtime.  Example... the TransformHandler I'm experimenting with needs access to Service interface info to get operation type info (input/output types).

                 

                There needs to be a facility to query for the service information at runtime.  I need to give a little thought to what the API will look like to provide this information and to query for it.  I imagine it will be pretty straightforward.

                 

                Have you considered which options there might be for interface types?  Certainly Java interface is one, where the input/output messages are typed Java objects.  WSDL would be another one, where you provided with a schema for the input and output.  I wonder if we will have a third category where the service definition does not point to a WSDL; instead it would use a more ad hoc mechanism where it could define the input and output message types and nothing else.

                • 5. Re: Configuration Thoughts
                  tfennelly

                  I'm sure there's lots more, but just thinking of transformation requirements I think we'd need the ability to be able to get:

                   

                  1. A list of operations available on a service
                  2. Supported input payloads for a service operation.
                  3. Supported output payloads for a service operation.

                   

                  I think that's all I'd need for now.

                  • 6. Re: Configuration Thoughts
                    kcbabo

                    Yes, although maybe #1 is not really a question that transformation logic would ask.  Instead, it would probably pass the operation name in as a parameter to #2 and #3.    Yeah, it's a nit. :-)

                    • 7. Configuration Thoughts
                      kcbabo

                      Posted some more thoughts on configuration in the Articles space.

                       

                      http://community.jboss.org/wiki/SwitchYardApplicationConfiguration

                      • 8. Configuration Thoughts
                        burmanm

                        The Wiki talks about serialization format, could be it possible to interface this method to allow other implementations also? In that case, it would be possible to build for example a database based solution as well for the configuration, or replace the underlying metadata format later on if we find SCA wasn't suitable for something (I don't know SCA, just throwing up the idea).

                         

                        Just afraid that if the implementation is tightly tied to what SCA can do or can't do, it's next to impossible later on to allow other methods.

                        • 9. Re: Configuration Thoughts
                          kcbabo

                          Good point on the pluggability front, particularly because it formalizes the idea that the configuration model (i.e. the information we care about to configure services, components, ec.) is distinct from any particular serialization format.  SCA is an attractive serialization in my mind because it lines up with my view of how services should be defined and the relationships between them should be represented.  It also is a standard, which means it's one less proprietary/project-specific thing that users have to learn.  With all of that in mind, I don't think that we should "depend" on SCA.  The primary use of an SCA serialization would be for governance-related activities.

                           

                          The database idea is an interesting one, but I haven't really thought it through to be honest.  Would be interesting to experiment with it once we get the config API in place.

                          • 10. Re: Configuration Thoughts
                            dward

                            Given that SCDL seems to be a good representation of the structure we are aiming for, let's start with this example:

                             

                            SCDL

                            <composite>

                                <service name="M1AppService" promote="SimpleService">

                                    <binding.soap>

                                        <port>MyWebService/SOAPPort</port>

                                        <wsdl>service.wsdl</wsdl>

                                    </binding.soap>

                                </service>

                                <component name="SimpleService">

                                    <implementation.bean class="org.switchyard.example.m1app.SimpleBean"/>

                                    <service name="SimpleService">

                                        <interface.java interface="org.switchyard.example.m1app.SimpleService"/>

                                    </service>

                                    <reference name="anotherService">

                                        <interface.java interface="org.switchyard.example.m1app.AnotherService"/>

                                    </reference>

                                </component>

                                <component name="AnotherService">

                                    <implementation.bean class="org.switchyard.example.m1app.AnotherBean"/>

                                    <service name="AnotherService">

                                        <interface.java interface="org.switchyard.example.m1app.AnotherService"/>

                                    </service>

                                </component>

                            </composite>

                             

                            Again, I know we don't want to be locked into SCA for SwitchYard's canonical model, but something close to that is as good a first (actually second) stab as any. So, I am proposing this API, and I will follow it with an explanation:

                             

                            API

                            package org.switchyard.config;

                             

                            public interface Model {}

                             

                            public interface ExternalServiceModel extends Model {

                                public QName getName();

                                public ComponentModel getComponent();

                                public List<BindingModel> getBindings();

                            }

                             

                            public interface ComponentModel extends Model {

                                public QName getName();

                                public ImplementationModel getImplementation();

                                public List<InternalServiceModel> getInternalServices();

                                public List<ReferenceModel> getReferences();

                            }

                             

                            public interface ImplementationModel extends Model {

                                public ImplementationType getType();

                            }

                             

                            public enum ImplementationType {

                                BEAN;

                            }

                             

                            public interface InternalServiceModel extends Model {

                                public QName getName();

                                public InterfaceModel getInterface();

                            }

                             

                            public interface InterfaceModel extends Model {

                                public InterfaceType getType();

                                public org.switchyard.metadata.ServiceInterface getMetadata();

                            }

                             

                            public enum InterfaceType {

                                JAVA;

                            }

                             

                            public interface ReferenceModel extends Model {

                                public QName getName();

                                public QName getTarget();

                                public BindingModel getBinding();

                            }

                             

                            public interface BindingModel extends Model {

                                public BindingType getType();

                            }

                             

                            public enum BindingType {

                                SOAP;

                            }


                             

                            Explanation:

                            1. An ExternalServiceModel represents an exposed/promoted service. In the XML, this would be the "M1AppService".
                            2. An InternalServiceModel represents a service nested within a component. In the XML, this would be the "SimpleService", "anotherService" and "AnotherService".
                            3. The various enums line up with the technology section of certain elements. For example, <implementation.bean/> is an ImplementationModel with ImplementationType.BEAN, <interface.java/> is an InterfaceModel with InterfaceType.JAVA and <binding.soap/> is a BindingModel with BindingType.SOAP.  The point of the enums is so that "handlers" of the configuration have known types to key off of.
                            4. An InterfaceModel exposes ServiceInterface metadata.
                            5. The reason ImplementationModel, InterfaceModel and BindingModel are so "plain" is that implementations are free to expose whatever properties are required.  The canonical interfaces don't specify any properties, the implementations will.  How does a config "handler" get at these properties?  Well, when they request a config, they can request it per a particular model namespace, and thus know exactly what they can cast the interface to.  This builds on the "Dynamic Model Builder" pattern pioneered by Smooks: http://goo.gl/lRsru
                            6. The root Model interface is notably empty. Initially, this is because there is a ModelBuilder mechanism I'm working on that has to deal with something common.  In the future, every Model might expose a DOM Element, but I'm not 100% sure on liking that now or not.
                            7. Regarding the ModelBuilder, that isn't shown here, but is the way that one can build up a configuration model in java code, without needing to start with config files.  The ModelBuilder will be able to dynamically create the right implementations of the Models based on namespace, again something I'm building on top of the "Dynamic Model Builder" work.

                             

                             

                            Comments welcome, although I am going ahead and working on the implementation in the meantime...

                            • 11. Configuration Thoughts
                              kcbabo


                              Thanks for taking a first swing at the API.  I agree with your comments around SCDL/SCA.  If nothing else, SCDL represents a good starting point as it lines up pretty well with the abstract configuration model we had in mind.

                               

                              A few pieces of feedback on the interfaces:

                               

                              Providing enumerations for interface, binding, and implementation type might not be a good idea, as the set of types is not static.  String seems like a reasonable representation for the time being.

                               

                              Is there a root level interface representing the entire configuration?  When you parse a SwitchYard configuration into a configuration model, what do you get back?  I'm thinking of a top-level API that has things like: getComponents, getServices, etc.

                               

                              I'm assuming that Model will have some sort of mechanism to pull out the specific configuration for a binding, implementation, etc.  I think this will be facilitated by the model handler you mentioned, but just want to make sure. 

                               

                              Tough call between having a single ServiceModel and separating it out into InternalServiceModel and ExternalServiceModel.  We have to start somewhere and an initial implementation will give us an opportunity to play around with it to see if it works or smells.

                              • 12. Re: Configuration Thoughts
                                dward

                                Providing enumerations for interface, binding, and implementation type might not be a good idea, as the set of types is not static.  String seems like a reasonable representation for the time being.

                                 

                                Okay. I've changed them to Strings.

                                 

                                Is there a root level interface representing the entire configuration?  When you parse a SwitchYard configuration into a configuration model, what do you get back?  I'm thinking of a top-level API that has things like: getComponents, getServices, etc.

                                Yeah, I hit that too. I've added a CompositeModel.

                                 

                                I'm assuming that Model will have some sort of mechanism to pull out the specific configuration for a binding, implementation, etc.  I think this will be facilitated by the model handler you mentioned, but just want to make sure.

                                Yes; absolutely. The handler is the way of pulling in a specific implementation that can be cast, however, it's not necessary. I've added a getConfig():Config to the base Model interface, which gives you access to all configuration elements provided, without being specific to DOM. (There is a DOMConfig implementation, of course).

                                 

                                Tough call between having a single ServiceModel and separating it out into InternalServiceModel and ExternalServiceModel.  We have to start somewhere and an initial implementation will give us an opportunity to play around with it to see if it works or smells.

                                Agreed. I'm moving forward with the internal/external split for now, though.

                                 

                                Thanks for the feedback.

                                • 13. Re: Configuration Thoughts
                                  dward

                                  I wanted to give an update on the Configuration work.  All requirements are functional now - the only things left are design/implementation review by Keith, usability for the soap component by Magesh, usability for maven plugin or tooling by Tom C. and/or Brian, and possibly review of the dynamic model building by Tom F.  I would like to get this work pushed into upstream master ASAP, so please hit me up with any questions as soon as you have them.

                                   

                                  Okay, now for a quick overview.  You can pull down the work from here:

                                  https://github.com/errantepiphany/core/tree/configuration

                                   

                                  All the config work is in it's own new maven sub-project called core-config:

                                  https://github.com/errantepiphany/core/tree/configuration/core/config

                                   

                                  There are 3 different java packages: org.switchyard.config, org.switchyard.config.model, and org.switchyard.config.model.v1.  I will explain the most important classes within each of these packages here next.

                                   

                                  org.switcyhard.config

                                  1. Configuration: Probably the second-most important interface in the project.  It abstracts all manipulation of attributes, it's value, and it's children.  It can also easily dump it's data to an OutputStream or Writer.
                                  2. DOMConfiguration: The only implementation of Configuration.  When you work with the Configuration interface, you are ignorantly manipulating a DOM tree.  When you dump out this implementation's data, it simply outputs it's XML structure.
                                  3. Configurations: First, this is a factory for Configurations.  However more interestingly it can merge Configurations.  The purpose of this is for when we want to merge a user-provided configuration file with a Configuration that has been built up in some other way (either manually, by a tool, or by a wrapper Model - but we'll get to that in a bit).
                                  4. ConfigurationResource: This utility class makes it easy to read in a configuration from a URL, File, InputStream, Reader, InputSource, classpath resource, etc.
                                  5. Descriptor: A class that wraps and maps Properties that are used by the pluggable model marshalling mechanism that will be described below.
                                  6. There are other classes in this package, but not as important.

                                   

                                  org.switchyard.config.model

                                  1. Model: Arguably the most important interface in the project, and is the root interface for all config models we represent.
                                    • Each Model wraps a Configuration.
                                    • Sub-interfaces of Model (as will be iterated below) provide helper methods that set attributes and child Models (actually their wrapped Configurations), to the wrapped Configuration.
                                    • Most often, you will not have to deal with Configurations.  You will deal directly with the Model interfaces, and let them do the work of manipulating the wrapped Configurations for you.
                                    • A Model also provides a way to dump out it's data to an OutputStream or Writer, however it just delegates to the wrapped Configuration, of course.
                                  2. Models: A factory for Models, however, like Configurations, it also provides a merging mechanism.  It of course delegates to the underlying Configurations.
                                  3. ModelResource: This utility class makes it easy to read in a model from a URL, File, InputStream, Reader, InputSource, classpath resource, etc.  And once again, you can rightfully assume it will use a ConfigurationResource under the hood.
                                  4. ModelMarshaller: This is the extendable Model mechanism which is used to allow changing versions of Models over time, using the Descriptor interface described above.  Even within the same configuration file, you can, via namespaces, contain varying versions of the Model configurations.  This idea was gratefully stolen from Tom Fennelly's work.
                                  5. There are other classes in this package, but not as important.

                                   

                                   

                                  org.switchyard.config.model.composite

                                  1. CompositeModel, ExternalServiceModel, BindingModel, ComponentModel, ImplementationModel, InternalServiceModel, InterfaceModel, ReferneceModel: These are all interfaces which expose the absolute required properties of our configuration model.  The accessor methods of the implementations of these interfaces use the helper methods in the abstract BaseModel, which as mentioned above, manipulates the wrapped Configuration objects.

                                   

                                  org.switchyard.config.model.composite.v1

                                  1. V1CompositeModel, V1ExternalServiceModel, V1BindingModel, V1ComponentModel, V1ImplementationModel, V1InternalServiceModel, V1InterfaceModel, V1ReferneceModel: These are the first-version implementations of the root interfaces.
                                  2. V1ModelMarshaller: An implementation of ModelMarshaller which will construct the afformentioned V1 models during deserialization, and output them during serialization.  It, by default, delegates to the BaseModelMarshaller, which delegates to the BaseModel, which delegates to the underlying DOMConfiguration for dumping out the data.  This can be overriden of course if custom serialization (or deserialization) is required.  It's important to note, however, that a user of the API never works directly with a ModelMarshaller.  It's more of an SPI.

                                   

                                  There are several JUnit tests in ConfigurationTests and ModelTests.  ConfigurationTests is for unit testing the underlying configuration mechanism.  ModelTests is for unit testing the model-handling mechanism.  Because of this, if you are looking for examples of how to do things as an end-user/developer would, I would suggest focusing on ModelTests.

                                   

                                       Keith, look at ModelTests.testMerge() to see how easy it is to automagically merge Models.  That test case actually does the exact merge that came from your PDF requirements example you sent me.


                                       Magesh, look at ModelTests.testReadCustomViaModel() to see how you can add and extend configuration elements - for example, if you want to read or write <port> and <wsdl> elements (which would be the new ones) underneath a <biding.soap>.  You can also ad-hoc access the underlying Configuration.

                                   

                                       Tom F., look at ModelTests.testReadExtended() to see how dynamic model building works.

                                   

                                  For people who don't want to pull down the source and dig through it yourselves, I will put some example code here:

                                   

                                  Example Code

                                  ModelResource mr = new ModelResource();

                                  CompositeModel composite = (CompositeModel)mr.pull("/path/to/config.xml");

                                  ExternalServiceModel externalService = composite.getServices().get(0);

                                  BindingModel binding = (BindingModel)externalService.getBindings().get(0);

                                  Assert.assertTrue(binding instanceof SOAPBindingModel);

                                  SOAPBindingModel soap_binding = (SOAPBindingModel)externalService.getBindings().get(0);

                                  Assert.assertEquals("soap", binding.getType());

                                  PortModel port = soap_binding.getPort();

                                  Assert.assertEquals("MyWebService/SOAPPort", port.getPort());

                                  Assert.assertEquals(true, port.isSecure());

                                  WSDLModel wsdl = soap_binding.getWSDL();

                                  Assert.assertEquals("service.wsdl", wsdl.getLocation());

                                  Assert.assertEquals("foobar", wsdl.getDescription());

                                  composite.write(System.out);

                                   

                                  I'll be happy to answer any questions.  But in the meantime, I have to add javadoc comments and wrestle with checkstyle.

                                  • 14. Configuration Thoughts
                                    dward

                                    Important Update: The Configuration/Model work has been finalized for M1.  (Not to say things won't change after...)  However, there have been some changes to the above.  For the latest structure/details, please see here:

                                     

                                    SwitchYard Configuration and Model APIs

                                     

                                    SwitchYard Application Configuration