Smooks Editor: Template Generation

One thing we would like to be able to do in the Smooks Editor is to automatically generate templates (FreeMarker, XSLT etc) for a target message definition (XSD, CSV fields, EDI Mapping Model etc).  Smooks uses templating technologies (e.g. FreeMarker, XSLT etc) as a way of generating Smooks Editor - Input and Output Message Types in different formats (aka "transformation").

 

To this end, we are working on an API that the editor will be able to use to generate templates.  Initially, we are working on generation of FreeMarker templates, but we are keeping other templating solutions in mind too e.g. XSLT.

 

 

The Goals

  1. Have a "standard" mechanism (within the Editor) of mapping to Smooks Editor - Input and Output Message Types.  Hopefully this will make it easier to add support for new output data formats, without having to reimplement all the mapping functionality in the core of the Editor. 
  2. Have a consistent user experience for mapping data onto Smooks Editor - Input and Output Message Types.
  3. Hide the underlying templating technology from the core Editor code.  This should make it easier to add support for new templating technologies e.g. in the next version of the editor (if we have XSLT templating support) using FreeMarker or XSLT should simply be a case of using the XSLT TemplateBuilder (Vs the FreeMarker TemplateBuilder) from the point of view of the Editor.
  4. Hide the underlying templating technology from the User.  We don't want the user to have to know how to write FreeMarker or XSLT.  They jut need to know the data and what the mappings need to be.  Again, in the next version of the editor (if we have XSLT templating support) using FreeMarker or XSLT should simply be a Radio Button selection from the point of view of the User.

The Basic Concept

The basic concept is that the Editor will:

 

  1. Capture the Smooks Editor - Input and Output Message Types from the user e.g. message schema for the output message.  We should be able to reuse the same Wizards/Forms used to capture message type/format info for Input (Source) messages.  Record these additional Editor specific details on the template configuration (note, Smooks runtime does not ned this additional metadata).  This type metadata will basically be the same info captured from the user for input messages of the same type.
  2. Construct a ModelBuilder instance based on the message type and parameters supplied by the user:
    • XML: An XSDModelBuilder from an XSD.
    • XML: An XMLModelBuilder from a sample XML message.
    • CSV: A CSVModelBuilder from a set of CSV fields + other CSV message type params.
    • EDI: An EDIModelBuilder from an EDI Mapping Model (like an XSD for an EDI message).
  3. From the DOM Model produced by the ModelBuilder instance, construct a TemplateBuilder instance for the target template type:   
    • XML via XSD (FreeMarker): XSDFreeMarkerTemplateBuilder.
    • XML via XSD (XSLT): XSDXSLTTemplateBuilder.
    • CSV (FreeMarker): CSVFreeMarkerTemplateBuilder.
    • etc....
  4. Take the same DOM Model and "draw" it as the target model in the templating Editor/View.  So, no matter what the target data format is, we draw it on the screen as a hierarchical (tree) model that we can map onto.
  5. As the user maps from the source model to the target model (from the ModelBuilder), the editor adds src->target "Mappings" to the TemplateBuilder instance.  The TemplateBuilder validates the mappings, returning the Mapping instance, if the mapping was accepted by the TemplateBuilder.
  6. Once done, the editor calls TemplateBuilder.buildTemplate() and the TemplateBuilder builds an appropriate template for the target model, based on the added Mapping instances.

 

Working with existing configuration should be very similar.  In this case, the message type information should already be present on the templating configuration (e.g. <ftl:freemarker>).  If not present on the config, the Editor should capture the parameters from the user.  The Editor should then:

 

  1. Construct a ModelBuilder instance (see #2 above).
  2. Construct a TemplateBuilder instance (see #3 above).  This is where the twist comes... this time, the TemplateBuilder constructor should also be supplied with the existing template text from the configuration being read.  The TemplateBuilder instance needs to parse the supplied template and, using the supplied model, extract the Mapping list and add them to the TemplateBuilder's list of Mappings.
  3. Draw the model in the Editor View (see #4 above).
  4. Get the mappings list from the TemplateBuilder (List<Mapping>) and draw the mappings.
  5. See #5 and #6 above.

 

We'll see specific examples of this in the coming sections.

 

The Message Model

In order for the mapping process to work in a "generic" way (from the point of view of the core editor code), we need a hierarchical representation that will work for any target message format (XML, CSV, EDI etc).  We've chosen the standard W3C DOM API for this (standard part of the JDK).

 

It is the responsibility of the ModelBuilder implementation to create the DOM Model that the editor will use to "draw" the message model seen by the user.  The following sections show how to create ModelBuilder and TemplateBuilder instances for specific message format + templating technology combinations (e.g. CSV by FreeMarker).  The key thing to remember here is that, from the point of view of the Editor's core mapping code, the message models all "look" the same i.e. they are DOM trees, annotated with information such as node cardinality (minOccurs/maxOccurs) and other information.

 

So the Editor needs to capture some basic information regarding the type of message the user want to generating through the mapping process.  With this informtion, the Editor should know which TemplateBuilder API components to use i.e. which ModelBuilder implementation + which TemplateBuilder implementation.

 

Once the Editor has constructed the appropriate ModelBuilder instance, it can get the basic model that needs to be displayed:

 

Document modelToDisplay = modelBuilder.buildModel();

 

It then needs to "draw" the model on the canvas as a standard tree like structure, by iterating down through the DOM tree.  Each node in the view needs to remember which DOM model node it is associated with, such that, when the user maps from the Source model (e.g. the Java Bindings) onto the templating message model, the Editor is able to notify the TemplateBuilder instance that such a mapping has taken place (similar process when a mapping is removed).

 

Model Annotations

As already stated, the model (created by the ModelBuilder instance) will have annotations that tell the Editor some specific details about nodes in the model, and hence what type of mappings can be performed etc

 

These annotations (elements and attributes) are identified by a specific namespace:

 

 

Obviously the Editor should not display any elements or attributes from this namespace (the reserved namespace).  The Editor can check if a model element (or attribute) is in this namespace by calling the ModelBuilder.isInReservedNamespace(Node) method.

minOccurs and maxOccurs

It's obvious enough what these annotations are for.  They will be visible in the model as follows (serialized representation of a CSV example):

 

<csv-record xmlns:smk="http://www.jboss.org/xsd/tools/smooks" smk:maxOccurs="-1" smk:minOccurs="0">
    <firstname smk:maxOccurs="1" smk:minOccurs="1"/>
    <lastname smk:maxOccurs="1" smk:minOccurs="1"/>
    <country smk:maxOccurs="1" smk:minOccurs="1"/>
</csv-record>

 

From an Editors perspective, it needs to make sure only Collection sources are mapped onto nodes that have a maxOccurs > 1, or equal to -1 (unbound - see <csv-record> above).  By this, we mean that if the source is a Java Object model (Java Mappings/Bindings), the only thing the user is allowed to map onto such a node on the target templating model is something like a List.  More on Collection Mappings in the folloing section.

 

To reiterate... the idea here is that the Editor just deals with astandardized, hierarchical representation/model of the target message.  It doesn't know (or need to know) that the target message format is actually XML, or EDI, or CSV, or whatever.  It's the responsibility of the TemplateBuilder instance to work out how the mappings made in the editor are reflected in the generated template.

 

<compositor> Element

This is a special element used in the model where there is a constraint on a series of elements e.g. only one of the specified child elements can be present in a valid message of that type (ala <choice> in an XSD).

 

In the following example, note the <smk:compositor> element enclosing the <QuantityValue> and <QuantityRange> elements.  The minOccurs/maxOccurs attributes on the <smk:compositor> element tell you that only one of these elements is permitted in a valid message of this type.

 

<QuantityLeftToRecieve smk:maxOccurs="1" smk:minOccurs="1">
    <smk:compositor smk:maxOccurs="1" smk:minOccurs="1" type="choice">
        <QuantityValue SignificanceCoded="#optional" SignificanceCodedOther="#optional" smk:maxOccurs="1" smk:minOccurs="1" xmlns="rrn:org.xcbl:schemas/xcbl/v4_0/core/core.xsd"/>
        <QuantityRange smk:maxOccurs="1" smk:minOccurs="1" xmlns="rrn:org.xcbl:schemas/xcbl/v4_0/core/core.xsd">
            <MinimumValue SignificanceCoded="#optional" SignificanceCodedOther="#optional" smk:maxOccurs="1" smk:minOccurs="1"/>
            <MaximumValue SignificanceCoded="#optional" SignificanceCodedOther="#optional" smk:maxOccurs="1" smk:minOccurs="1"/>
        </QuantityRange>
    </smk:compositor>
    <UnitOfMeasurement smk:maxOccurs="1" smk:minOccurs="1" xmlns="rrn:org.xcbl:schemas/xcbl/v4_0/core/core.xsd">
        <UOMCoded smk:maxOccurs="1" smk:minOccurs="1"/>
        <UOMCodedOther smk:maxOccurs="1" smk:minOccurs="0"/>
    </UnitOfMeasurement>
</QuantityLeftToRecieve>

 

As already stated, the Editor should not display any elements from the reserved namespace.

 

See Hiding and Showing Model Nodes.

 

Mappings

 

Two types of mappings are currently supported (and I think that is all that's required).  We discuss these in the following subsections:

 

Collection Mappings

This type of mapping is performed when mapping onto a node what has a "maxOccurs" value greater than 1, or equal to -1 (unbounded).

 

The Editor needs to enforce some simple mapping rules here i.e.

 

  1. That the user is only able to map a Collection source onto such a node.  E.g. when mapping from the Java Mappings (data in the Bean Context), the user is mapping from an appropriate Java Collection type e.g. a java.util.List type.
  2. That Collection Mappings are made before mappings onto nodes inside the collection node.  If this is not done, the TemplateBuilder will generate an Exception, so it would be good for the Editor to avoid this.

 

A Collection mapping is made onto the TemplateBuilder instance by calling the TemplateBuilder.addCollectionMapping() method.  We'll look at this in more detail in the sections dealing with specific use cases.

 

Value Mapping

This type of mapping is performed then mapping a specific data value from the source model (e.g. the Java Mappings) onto the target templating model.  It is performed by using the TemplateBuilder.addValueMapping() method.  We'll look at this in more detail in the sections dealing with specific use cases.

 

Hiding and Showing Model Nodes

As mappings are made in the Editor, it should make calls to the TemplateBuilder.addCollectionMapping() and TemplateBuilder.addValueMapping() methods (as outlined above).  These methods return a Mapping instance.  The Editor needs to store the Mapping instance against the physical mapping, and use it in the call to the TemplateBuilder.removeMapping(Mapping) method if the user deletes that mapping.

 

The Mapping instance contains, among other things, a "hideNodes" node list (List<Node>).  The Editor should check this list after calling the add methods on the TemplateBuilder, "hiding" the visual representation of the Nodes (Elements and Attributes) found in the List.  On the other side of this equation, the TemplateBuilder.removeMapping(Mapping) returns a node list (List<Node>), which are a List of hidden Nodes that need to be made visible in the Editor view.  The Nodes in this list may or may-not be the same as the Nodes in the hideNodes list, depending on other mappings that have been made onto the model.

 

So why are we hiding and showing nodes?  Some message schema allow optional fragments (see <compositor> element).  The most notable example of this are XML Schema <choice> definitions e.g.:

 

<complexType name="animal">
     <choice minOccurs="1" maxOccurs="1">
          <element name="dog" type="tns:dog"></element>
          <element name="person" type="pref:person"></element>
          <element name="cat" type="tns:cat"></element>
     </choice>
</complexType>

 

Message fragments conforming to the above "animal" type permit a single instance of either a <dog>, <person> or <cat> sub-fragment within elements of type "animal".

 

In the above case, the initial model created by the ModelBuilder instance will "show" all three sub-fragments (<dog>, <person> and <cat>).  If the user maps onto one of these sub-fragments, the Mapping instance returned by either of the TemplateBuilder add methods will list the other sub-fragments in the hideNodes list.  A call to the TemplateBuilder.removeMapping(Mapping) method will return a list of these same sub-fragments, as long as no other mappings are "active".

 

The bootom line, from the point of view of the core Editor code, is to hide the nodes listed in the Mapping instance returned from the TemplateBuilder add methods, and show/unhide the nodes returned from the call to the TemplateBuilder.removeMapping(Mapping) methodIt is the responsibility of the TemplatBuilder instance to worry about what nodes need to be hidden or shown.

CSVFreeMarkerTemplateBuilder

This TemplateBuilder implemenation will build a FreeMarker template for constructing a CSV message from a set of src to target mappings.

 

Creating the 1st Config

We use the CSVModelBuilder class to build a DOM Model (of the target message) to be displayed by the editor.  So, to use this, the Editor first needs to capture the target CSV field names from the user.  It then constructs the Template Builder as follows:

 

CSVModelBuilder modelBuilder = new CSVModelBuilder("firstname", "lastname", "country");

 

Once the ModelBuilder instance is created, the editor can use it to build the TemplateBuilder instance for the target template type (FreeMarker, XSLT etc).  At the moment we only have a FreeMarker TemplateBuilder implementation:

 

TemplateBuilder templateBuilder = new CSVFreeMarkerTemplateBuilder(modelBuilder, ',', '\"');

 

And the rest of the code is the same, no matter what the target data format and template type.  The Editor gets the model from the TemplateBuilder instance and "draws" the target model in the editor:

 

Document model = templateBuilder.getModel();

// Use the DOM model to draw the target model in the editor...

 

The target model for this CSV target message should look something like (a cleaner version of):

 

<csv-record>
    <firstname>
    <lastname>
    <country>

 

See Balsamiq Flash of the proposed mechanism for captuing the target message details.

 

Then, as the user makes mappings from the source to target models (e.g. from the Java Bean Context beans to the target model), the Editor will add Mappings to the TemplateBuilder instance.  There are 2 types of mappings:

 

  1. Collection Mapping: These mappings tell the template builder how to map from the source onto a collection of data in the target model.  The Elements in the target message DOM Model contains data attributes specifying node cardinality (i.e. minOccurs/maxOccurs), so the editor can see the collections in the model.  In the target CSV model being used here, the <csv-record> node would require a collection mapping since it represents the collection in the target CSV message.
  2. Value Mapping: These mappings tell the template builder how to map a source value onto the target model.  A Value Mapping cannot be added if a parent Collection Mapping has not been made on the model.

 

A Collection Mapping would look as follows:

 

builder.addCollectionMapping(srcModelPath, targetModelNode, collectionItemName);

 

And a Value Mapping:

 

builder.addValueMapping(srcModelPath, targetModelNode);

 

Both samples of code (immediately above) show a srcModelPath argument in the method calls.  In the case of a mapping from the Bean Context, this patrh will be the Java Object Graph path e.g. "person.firstName".  The targetModelNode is the target model node onto which the source data is to be bound.

 

And then, to build the actual template:

 

String template = builder.buildTemplate();

 

The key point of note here is that the Editor code knows nothing about the target data format, or the templating technology being used.  The editor's responsibility is purely that of displaying the model in a clean structured way (as a tree), capturing the drag & drop mappings made by the user and adding them to the TemplateBuilder instance.


The resulting Smooks FreeMarker configuration should look as follows:

 

<ftl:freemarker applyOnElement="#document">
    <ftl:template>
<![CDATA[<#list people as person>
"${person.fname}","${person.lname}","${person.address.country}"
</#list>]]></ftl:template>
    <param name="messageType">CSV</param>
    <param name="csvFields">firstname,lastname,country</param>
    <param name="separatorChar">,</param>
    <param name="quoteChar">"</param>
</ftl:freemarker>

 

Processing an existing Template Config

The editor obviously needs to be able to open existing template configurations.  Lets use the configuration we created above:

 

<ftl:freemarker applyOnElement="#document">
    <ftl:template>
<![CDATA[<#list people as person>
"${person.fname}","${person.lname}","${person.address.country}"
</#list>]]></ftl:template>
    <param name="messageType">CSV</param>
    <param name="csvFields">firstname,lastname,country</param>
    <param name="separatorChar">,</param>
    <param name="quoteChar">"</param>
</ftl:freemarker>

 

From this configuration, the Editor can see that it needs to use the CSVModelBuilder and CSVFreeMarkerTemplateBuilder classes (we could create a Factory class for this).  The process is nearly the same as when creating the initial configuration, accept this time, the Editor:

 

  1. Doesn't need to ask the user for message type information.  We stored it on the config as <param>s. 
  2. Constructs the CSVToFreemarkerTemplateBuilder instance with the additional "template" parameter.
  3. Gets the intitial set of mappings (List<Mapping>) from the CSVToFreemarkerTemplateBuilder instance.

 

CSVModelBuilder modelBuilder = new CSVModelBuilder("firstname", "lastname", "country");
TemplateBuilder builder = new CSVFreeMarkerTemplateBuilder(modelBuilder, ',', '\"', template);
List<Mapping> mappings = builder.getMappings();

// Draw the model and the initial mappings...

 

XMLFreeMarkerTemplateBuilder

This TemplateBuilder implemenation will build a FreeMarker template for constructing an XML message from a set of src to target mappings.

 

A number of options exist wrt creating the model for an XML message:

 

  1. XSD:  For this we have the XSDModelBuilder.
  2. XML Sample: TODO - need to create an XMLModelBuilder for 3.1.0.GA.
  3. WSDL: TODO - need to create WSDLModelBuilder post 3.1.0.GA.
  4. RELAX NG etc: TODO - Post 3.1.0.GA

XSDModelBuilder

As it's name should suggest, this ModelBuilder implementation builds a model (that can be used by the Editor) from an XML Schema (XSD).  It uses the Eclipse Schema Infoset Model API for processing the XML Schemas.

Creating the 1st Config

We use the XSDModelBuilder class to build a DOM Model (of the target message) to be displayed by the editor.  So, to use this, the Editor first needs to capture the target XSD from the user.  It then constructs the Template Builder as follows:

 

ModelBuilder modelBuilder = new XSDModelBuilder(org.eclipse.emf.common.util.URI xsdFile);

 

Once the ModelBuilder instance is created, the editor can use it to build the TemplateBuilder instance for the target template type (FreeMarker, XSLT etc).  At the moment we only have a FreeMarker TemplateBuilder implementation:

 

TemplateBuilder templateBuilder = new XMLFreeMarkerTemplateBuilder(modelBuilder);

 

After that, from the point of view of the core Editor code, the process is exactly the same as that of the CSVFreeMarkerTemplateBuilder, outlined above.  In fact, the process is the same, no matter what the TemplateBuilder implementation is.

 

Once the mappings are made in the Editor and the template has been created, the <ftl:freemarker> config should look something like:

 

<ftl:freemarker applyOnElement="#document">
    <ftl:template>
<![CDATA[<?xml version="1.0" encoding="UTF-8"?>
<ns0:creature xmlns:ns0="http://www.example.org/creature/">
    <cat>${cat?string}</cat>   
</ns0:creature>]]></ftl:template>
    <param name="messageType">XML</param>
    <param name="modelSrcType">XSD</param>
    <param name="modelSrc">... path to XSD file ...</param>
    <param name="omitXMLDecl">false</param>
</ftl:freemarker>

 

Note the "omitXMLDecl" parameter.  This parameter (default=false) tells the XMLFreeMarkerTemplateBuilder whether or not to omit the standard XML declaration from the generated XML template.  See the XMLFreeMarkerTemplateBuilder.setOmitXMLDeclaration(boolean) method.

 

Processing an existing Template Config

The editor obviously needs to be able to open existing template configurations.  Lets use the configuration we created above:

 

<ftl:freemarker applyOnElement="#document">
    <ftl:template>
<![CDATA[<?xml version="1.0" encoding="UTF-8"?>
<ns0:creature xmlns:ns0="http://www.example.org/creature/">
    <cat>${cat?string}</cat>   
</ns0:creature>]]></ftl:template>
    <param name="messageType">XML</param>
    <param name="modelSrcType">XSD</param>
    <param name="modelSrc">... path to XSD file ...</param>
    <param name="omitXMLDecl">false</param>
</ftl:freemarker>

 

From this configuration, the Editor can see that it needs to use the XSDModelBuilder and XMLFreeMarkerTemplateBuilder classes (we could create a Factory class for this).  The process is nearly the same as when creating the initial configuration, accept this time, the Editor:

 

  1. Doesn't need to ask the user for messageType etc information.  We stored it on the config as <param>s. 
  2. Constructs the XMLFreeMarkerTemplateBuilder instance with the additional "template" parameter.
  3. Gets the intitial set of mappings (List<Mapping>) from the XMLFreemarkerTemplateBuilder instance.

 

ModelBuilder modelBuilder = new XSDModelBuilder(xsdURI);
TemplateBuilder builder = new XMLFreeMarkerTemplateBuilder(modelBuilder, template);
List<Mapping> mappings = builder.getMappings();

// Draw the model and the initial mappings...

 

XMLModelBuilder

ModelBuilder from an XML Sample message.

 

TODO

WSDLModelBuilder

ModelBuilder from a WSDL.

 

TODO