JMS Adapter

You are on vacation, your hairdryer expects 240V and has a UK 3-pin plug, but the socket is 110V with a US 2-pin plug. You need a lightweight travel adapter.

You have a JMS application which sends and receives messages in one format, but the application you want to talk to receives and sends messages in another format. You need a lightweight JMS Adapter.

Typical solutions

The point-to-point adapter pattern is typically used to solve this problem :-

Note: The word "adapter" is also sometimes also used to describe adapting between interfaces/transports (eg: JDBC to JMS), but here we are using the word to describe the adaptation of the data.

Sometimes enterprises implement a decoupled model with pub-adapters and sub-adapters, and a standard intermediate data representation :-

Adapters typically implement the VETO standard industry pattern, sometimes taken to mean :-

In the Java JMS and XML world, these typically resolve to :-

Sometimes adapters implement VETRO, in which the R stands for Routing.

Products such as Apache Camel, OpenAdapter and Mule give you frameworks for implementing this. Enterprises sometimes build their own.

What is JMS Adapter

JMS Adapter is a JMS implementation which you configure to use an underlying JMS implementation (eg: ActiveMQ, Apache Qpid, WebSphere MQ, Tibco EMS). The application is then configured to use JMS Adapter. As a result, it can mediate any message as it is sent or received.

It has the following features :-

Things it doesn't do include :-

The two patterns shown earlier now can be revised. In the point to point case, we could have either :-

or

And in the pub-sub case :-

Its not unusual for enterprises to have hundreds of adapters. A large proportion of these (maybe half) might be expected to be simple enough that they can be implemented in JMS Adapter. The others might need a more complex solution, or perhaps use JMS Adapter for part of their implementation, and then draw upon external components which just do the missing parts (such as sequencing and routing). The resulting reduction in complexity of development, number of JMS destinations, number of running processes to manage and hardware needed to host adapters could be very significant.

Configuring JMS Adapter into the application

Application classpath

Add nyangau-jmsadapter.jar to the application classpath. JMS Adapter itself has no dependencies on any other 3rd party .jar files.

However, if you will be using the <mapdb/> data mapping feature to query a database to enrich your data, you'll need to ensure the vendors JDBC implementation is on the classpath also.

If you've written any extensions to JMS Adapter that use any other 3rd party libraries, you'll need to ensure they are on the classpath too.

JNDI Setup

A normal application will use the following sequence of events :-

  1. Application reads its configuration to determine the
  2. Application uses the InitialContextFactory to connect to JNDI to obtain the underlying JMS implementations administered objects, which implement things like QueueConnectionFactory and Queue interfaces, and contain information like service hostnames, ports, security information, connection and retry parameters etc..
  3. Application uses returned objects to invoke the JMS provider.

We want to interpose JMS Adapter. If we are able to change the JNDI names in the applications configuration, and we are able to populate additional objects into JNDI, then we want :-

Or, if we can't change the JNDI names in the applications configuration, or we are not able to populate additional objects into JNDI (as would be the case with the tibjmsnaming JNDI subset provided as a part of Tibco EMS), we want :-

The following sequence of events now occurs :-

  1. Application reads its configuration, but this time it gets back different values
  2. Application connects to (a possibly different) JNDI and looks up JMS Adapter administered objects (using possibly different JNDI names).
  3. Application uses the returned objects to invoke JMS Adapter.
  4. JMS Adapter administered objects contain JMS Adapter uses this information to fetch the underlying JMS implementations administered objects.
  5. JMS Adapter uses the returned underlying objects to invoke the underlying JMS provider. Of course, it also mediates messages on the way through.

Setting up JNDI for JMS Adapter therefore requires populating one JNDI with details of how to reach (possibly a different) JNDI, and the details of the entries in there to use. JNDI providers typically have quite a bit of variation in the parameters needed, especially once you start to consider security and SSL related parameters. Therefore it is hard to write one tool which is able to do this, so instead we show some sample code.

In the sample, the client application would normally use application message formats in application specific queues, and some adapter would convert between this and a standard services message formats in standard service specific queues. Here, we configure the application to use JMS Adapter to translate application messages to standard messages, and put them in standard queues, and vice versa :-

//
// SetupJNDI.java - Register things in JNDI to support App.java sample code
//

import java.util.*;
import javax.jms.*;
import javax.naming.*;
// import com.tibco.tibjms.naming.*;
import nyangau.jmsadapter.*;

public class SetupJNDI
  {
  public static void main(String[] args)
    {
    try
      {
      // The JNDI we're going to store Jmsa AOs into
      InitialContext ic = new InitialContext(
        new Hashtable<String,Object>()
          {{
          // We're going to use the cheap and cheerful filesystem based JNDI
          put(Context.INITIAL_CONTEXT_FACTORY, "com.sun.jndi.fscontext.RefFSContextFactory");
          put(Context.PROVIDER_URL, "file:fs");
/*
          // Other vendor unique parameters used to talk to JNDI,
          // such as LDAP specific or SSL specific ones, would be added here
          put(...);
*/        
          }}
        );

      // The properties used to reach the underlying JNDI implementation
      Hashtable<String,Object> ht = new Hashtable<String,Object>()
        {{
        // We're hosting queues on a local Tibco EMS and using its JNDI
        put(Context.INITIAL_CONTEXT_FACTORY, "com.tibco.tibjms.naming.TibjmsInitialContextFactory");
        put(Context.PROVIDER_URL, "tcp://localhost:7222");
/*
        // Other vendor unique parameters used to talk to JNDI,
        // such as SSL ones, would be added here
        put(Context.SECURITY_PRINCIPAL, "admin");
        put(Context.SECURITY_CREDENTIALS, "password");
        put(TibjmsContext.SSL_ENABLE_VERIFY_HOST, new Boolean(false));
        put(TibjmsContext.SSL_ENABLE_VERIFY_HOST_NAME, new Boolean(false));
        put(TibjmsContext.SSL_TRUSTED_CERTIFICATES,
          new Vector<String>()
            {{
            add("root1.cert.pem");
            add("root2.cert.pem");
            }}
          );
*/
        }};

      // We're going to need a QCF
      ic.rebind(
        "qcf",
        new JmsaQueueConnectionFactory(
          ht,
          new Hashtable<String,Object>()
            {{
            put(Jmsa.UNDERLYING_CONNECTION_FACTORY, "QueueConnectionFactory");
            put(Jmsa.CONFIG, "file:appstd.xml");
            put(Jmsa.PROPERTIES, "file:appstd.properties");
            }}
          )
        );

      // Lets create a Q for deals
      ic.rebind(
        "appdealq",
        new JmsaQueue(
          ht,
          new Hashtable<String,Object>()
            {{
            put(Jmsa.UNDERLYING_DESTINATION, "stddealq");
            }}
          )
        );

      // Lets create a Q for confirmations
      ic.rebind(
        "appconfirmq",
        new JmsaQueue(
          ht,
          new Hashtable<String,Object>()
            {{
            put(Jmsa.UNDERLYING_DESTINATION, "stdconfirmq");
            }}
          )
        );

      }
    catch ( NamingException e )
      {
      e.printStackTrace();
      }
    }
  }

After running this :-

Application can refer to and on its behalf, JMS Adapter will access
com.sun.jndi.fscontext.RefFSContextFactory
file:fs
com.tibco.tibjms.naming.TibjmsInitialContextFactory
tcp://localhost:7222
JNDI name qcf JNDI name QueueConnectionFactory
JNDI name appdealq JNDI name stddealq
JNDI name appconfirmq JNDI name stdconfirmq

Note in particular that the constructors for the JmsaQueueConnectionFactory and JmsaQueue take two Map<String,Object>. The entries from both are combined. They are seperate to make it easier to write programs like the one above. You can prepare one Map describing how to reach the underlying JNDI (called ht in the example), and use this repeatedly, whilst having other Maps to describe each underyling connection factory or destination.

Note Hashtable<String,Object> implements Map<String,Object>.

The XML configuration

Here is a configuration that does nothing :-

<?xml version="1.0"?>
<jmsadapter>

  <!-- JMS Adapter configuration for adapter ${ja_name} -->

  <!-- Various data map elements go here.
       The element names all start with map, such as mapinline.
       A data map is something which maps Strings to Strings,
       and various mechanisms to do this are provided. -->

  <!-- Various flow elements go here.
       Each flow contains a sequence of flow steps,
       each of which is an action to perform on the message or variables. -->

  <!-- Various queue and topic elements go here.
       Each identifies a destination by underlying JMS provider destination name,
       and associates it with flow(s) -->

</jmsadapter>

In the JNDI setup, the location of the configuration is specified using the Jmsa.CONFIG property. Note that it is a URL, so the configuration can come from a file, from a web server, from within a .jar file, etc.. In a Production setup, its quite likely that the developer will have delivered their JMS Adapter configuration as a set of files inside a .jar file, because the XML configuration is likely to refer to other files, such as XSD schema, XSLT transformations, data mapping properties etc..).

Every JMS Adapter connection factory must have a configuration. Also, note that each JMS Adapter connection factory can have its own seperate JMS Adapter configuration file.

After reading the configuration into memory, but before parsing it into a DOM structure, every ${variable} sequence is expanded. Even ${variable} sequences within XML comments are expanded. Each variable is first considered as a property, and failing that as a system property, next the default is used (if specified), and failing that JMS Adapter throws an exception. The syntax ${variable:default} is used to define the default value.

The full detail of what goes in the XML configuration is covered in the following sections.

The properties

Here is a simple set of properties :-

ja_name=My First JMS Adapter

In the JNDI setup, the location of this file is specified using the Jmsa.PROPERTIES property. Note that it is a URL, so the properties can come from a file, from a web server, from within a .jar file, etc.. In a Production setup, its quite likely that the properties would be a file or fetched from a central web server.

Properties are optional, but as above, if the XML references a ${variable} and this does not exist in the properties, or as a system property, and a default is not given, JMS Adapter will throw an exception.

The intent is that developers will prepare the XML file, using ${variable} placeholders for JDBC URLs, Web Service URLs, queue names, etc.., and the deployer will use a properties file to map these to the specific set of resources being used in UAT, Production environments.

Queues and Topics to be mediated

You identify which queues and topics will be mediated using <queue/> and <topic/> elements in the XML configuration.

To ensure that when the application sends to the stddealq, the message is processed by the AppDeal_to_StdDeal flow, you'd write :-

<queue name="stddealq" out="AppDeal_to_StdDeal"/>

To ensure that when the application receives a message from the stdconfirmq, the message is processed by the StdConfirmation_to_AppConfirmation flow, you'd write :-

<queue name="stdconfirmq" in="StdConfirmation_to_AppConfirmation"/>

The destination names are the names in the underlying JMS implementation. They are not JNDI names.

Temporary destinations

The <queue/> elements shown so far explicitly name the queue. However, when implementing a request-reply interaction pattern, the client can create a temporary queue for the reply to be sent to. The outbound request may need converting, and the reply may need converting back too.

If we have JMS Adapter configured into the client, then this can be acheived like this :-

<queue name="requestq"
       out="ClientRequest_to_ServerRequest"
       outreply="ServerResponse_to_ClientResponse"/>

Under the covers, when sending the request, JMS Adapter spots the use of a temporary destination in the JMSReplyTo, and creates a temporary association :-

<queue name="__tempq12345"
       in="ServerResponse_to_ClientResponse"/>
__tempq12345 is an example of the name of a temporary queue.

This only lives for as long as the temporary queue.

Its also possible that JMS Adapter might be being used in the server, perhaps because the server is new and needs to understand an old style of request. The server might use :-

<queue name="requestq"
       in="OldRequest_to_NewRequest"
       inreply="NewResponse_to_OldResponse"/>

Under the covers, when JMS Adapter receives a message from the requestq, it creates a temporary association :-

<queue name="__replyq12345"
       out="NewResponse_to_OldResponse"/>
__replyq12345 is an example of the name of the reply to queue.

However, there is a problem: JMS Adapter cannot know whether the reply queue is temporary or not, and cannot know when it gets deleted. So it cannot know when to forget this temporary association. Therefore, over time, this could represent a memory leak. JMS Adapter only remembers the 1000 most recent of these, and when the 1001st is added, forgets the 1st. This is obviously not a 100% reliable way of handling the problem.

As a result, we say that request-reply is best mediated at source.

Note: Everything described here for queues works equally for topics.

Flows

A flow is a sequence of flow steps, which are executed one by one. Each performs some processing action, such as manipulating part of the message, validating, enriching, transforming, etc..

A flow context is passed along the flow, and each step updates part of it.

Any flow with a name may be referred to by a <queue/> or <topic/> element. Flows should typically be named to reflect what they do. So, TextMessageMxML_to_MapMessageFpML might be a good name for a flow that converted a Murex TextMessage to a FpML MapMessage.

An simpler example flow might look like this :-

<flow name="TextMessage_to_MapMessage">
  <bodyget/>
  <bodytype type="MapMessage"/>
  <bodyset item="textNode"/>
</flow>

Flows have a special eflow="exceptionflowname" attribute which is described in the exception handling section.

Flow Context

The flow context contains :-

The JMS Message is actually an instance of the JMS Adapter JmsaMessage class, or derivation thereof. As a result, it is possible to make changes to the message structure and content that would be impossible when operating on vendor supplied implementations of the JMS Message interface. Specifically :-

Each variable can be of any Java Object :-

At the beginning of processing, the flow context contains the message, and no variables are defined. After processing, the message is passed on, and the flow context discarded.

Flow steps

Flow steps are the commands that make up a flow.

All steps have a name="stepname" attribute, which is optional. If specified, then any diagnostics that are given include the step name you specify, otherwise a machine generated identifier is displayed. In the tables below the name attribute is not shown.

Any step that has a var="varname" attribute will default to v0 if not specified.

Any step that has a destvar="varname" attribute will default to the same as the var="varname" attribute if not specified.

Flow steps - Message property access

Element Action
<propget
    prop="propname"
    var="varname"/>
Fetch JMS Message property propname and store it in variable varname. If the JMS Property is not defined, then the variable becomes Null. A variable being Null is distinct from the variable not having been defined.
<propset
    var="varname"
    prop="propname"/>
Store variable varname into JMS Message property propname. An exception is thrown if the variable is not defined, or is Null. Also, an exception will be thrown if the variable is not Boolean, Byte, Short, Integer, Long, Float, Double or String.
<propdel
    prop="propname"/>
Delete JMS Message property propname. No exception is thrown if the property already does not exist.

Flow Steps - Message body access

Element Action
<bodyget
    reset="true|false"
    type="UTF|
          Boolean|
          Character|
          Byte|
          Short|
          Integer|
          Long|
          Float|
          Double|
          byte[]"
    item="itemname"
    var="varname"/>
Read message body into variable varname.
  • When reading from a BytesMessage then type is used to decide what to read from it (default UTF, resulting in a String), and reset is used to decide whether to reset to the start of the start of the message (default true). Note there is currently the limitation that if reading byte[] from a BytesMessage, reset must be true.
  • When reading from a MapMessage, item must be specified (else an exception is thrown), and identifies which item in the map to read.
  • When reading from a StreamMessage, reset is used to decide whether to reset to the start of the message (default true). If the end of the stream is reached, the result is Null.
  • When reading from an ObjectMessage, if the object in the message is null, the result is Null.
  • When attempting to read from a plain old Message, ie: a message with headers and properties, but no body (ie: no payload), an exception is thrown.
<bodyset
    var="varname"
    clear="true|false"
    item="itemname"/>
Write variable varname into the the message body. If clear="true" (the default), the message body is cleared first. You might not want to do this if you were assigning several items into a MapMessage, or assembling a stream of objects in an StreamMessage, etc.
  • When writing to a BytesMessage, if the variable is not byte[], Boolean, Byte, Character, Short, Integer, Long, Float, Double or String, then an exception is thrown. When writing a String, its written in UTF form.
  • When writing to a TextMessage, if the variable is not a String, then its .toString() method is used.
  • When writing to a MapMessage, item must be specified (else an exception is thrown), and identifies which item in the map to write to. If the variable is not Boolean, Character, Byte, Short, Integer, Long, Float, Double, String or byte[], then an exception is thrown.
  • When writing to a StreamMessage, if the variable is not byte[], Boolean, Byte, Character, Short, Integer, Long, Float, Double or String, then an exception is thrown.
<bodydel
    item="itemname"/>
If a MapMessage and item is specified, then clear that item from the map. Otherwise clear the whole message body.
<bodytype
    type="BytesMessage|
          TextMessage|
          MapMessage|
          StreamMessage|
          ObjectMessage|
          Message"/>
Convert the JMS message to the indicated type, clearing out the message body. JMS message headers and properties are preserved. Note: In JMS, a plain Message is a message with headers and properties, but no body (ie: no payload).

Flow steps - Variable manipulation

Element Action
<varset
    var="varname"
    value="somevalue"/>
Set variable varname to String value somevalue.
<vardef
    var="varname"
    value="somevalue"/>
If variable varname is not defined or is Null, set it to String value somevalue.
<vardel
    var="varname"/>
Delete variable varname. No exception is thrown if the variable already does not exist.
<vartype
    var="varname"
    destvar="destvarname"
    type="String|
          Document|
          Boolean|
          Character|
          Byte|
          Short|
          Integer|
          Long|
          Float|
          Double|
          byte[]|
          Object"
    encoding="UTF-8|
              UTF-16|
              ISO8859-1|
              XML|
              ... other"
    coalescing="true|false"
    expandentityreferences
      ="true|false"
    ignoringcomments
      ="true|false">
  <outputproperty
      name="propname"
      value="propvalue"/>
  ... more outputproperties
</vartype>
Convert variable varname to type type, storing the result in destvarname. This is a general purpose tool for converting between object types. For details of what conversions are possible, and what attributes are used, see the Legal <vartype/> conversions below.
<varmap
    var="varname"
    destvar="destvarname"
    xpath="xpathexp"
    nsctx="ns1=uri1 ns2=uri2 ..."
    regexp="javaregexp"
    group="capturinggroup"
    map="datamapname"/>
If varname is a undefined, or Null, throw an exception. Selects part of the varname variable :-
  • If varname is a Document and the xpath attribute is specified, then select parts of the document using the XPath expression. Note that the XPath expression can use namespace prefixes which are defined in the optional nsctx attribute. Beware: after mapping, both varname and destvarname will point to the same Document, so both are mapped..
  • If varname is a String and the regexp attribute is specified (and optionally the group attribute, default 0), then select parts of the string using the regular expression.
  • Otherwise, throw an exception.

Attempts to map it using the data map datamapname. There must be a data map defined with the indicated name at the top level of the XML configuration. For a description of data maps, see later in this document.

If afterwards, there are still unmapped values, then an exception is thrown. To help with diagnosis, the text of the exception will include the first 10 unmapped values.

<varselect
    var="varname"
    destvar="destvarname"
    xpath="xpathexp"
    nsctx="ns1=uri1 ns2=uri2 ..."
    regexp="javaregexp"
    group="capturinggroup"/>
If varname is a undefined, or Null, throw an exception. Selects part of the varname variable :-
  • If varname is a Document and the xpath attribute is specified, then select parts of the document using the XPath expression. Note that the XPath expression can use namespace prefixes which are defined in the optional nsctx attribute.
  • If varname is a String and the regexp attribute is specified (and optionally the group attribute, default 0), then select parts of the string using the regular expression.
  • Otherwise, throw an exception.
Legal <vartype/> conversions

The <vartype/> mechanism attempts to "do the right thing" and give maximum flexibility in conversion between object types :-

ConvertFrom
StringDocumentBooleanByte CharacterShort IntegerLong Float Doublebyte[] ObjectException
ToString y y,d2 y,2s y,2s y,2s y,2s y,2s y,2s y,2s y,2s y,enc y,2s y,stack
Document y,2d y            y,enc,2d  
Boolean y   y y y y y y y y     
Byte y   y y y y y y y y     
Character y   y y y y y y y y     
Short y   y y y y y y y y     
Integer y   y y y y y y y y     
Long y   y y y y y y y y     
Float y   y y y y y y y y     
Double y   y y y y y y y y     
byte[] y,enc y,d2,enc           y y,ser  
Object               y,deser   

Key :-

<xmltransform/> (which is a step described later in this document) does XSLT from DOM source to DOM destination, and as a result it seems that <xsl:output indent="yes"/> doesn't work. Therefore, so if you want nicely formatted XML, you typically find yourself handling the indentation when the DOM is serialised to textual form, using <vartype type="String"/> and a nested <outputproperty name="indent" value="yes"/> instead. Update: I also find it necessary to use a <outputproperty name="{http://xml.apache.org/xslt}indent-amount" value="4"/> or similar setting.

Note that the encoding="XML" has a special meaning. It is intended to cope with the way in which XML is encoded, with or without BOMs, encoded using UTF-8 by default, or using another encoding :-

The XML character set encoding is handled in this way so as to seperate it from the process of parsing. This makes it possible to do certain kinds of processing of XML data without having to parse it (which could be quicker), and it also makes it possible to extract certain parts of an XML document, even if it is incomplete, corrupt, or otherwise doesn't parse cleanly.

Flow steps - XML specific manipulations

Element Action
<xmlvalidate
    var="varname"
    schematype="schemaType"
    schema="urlOfXSD"/>
Validate variable varname against schema urlOfXSD. Throws an exception if variable varname is not defined, or is Null, or is not a Document. Throws an exception if the document fails validation. schematype defaults to the string equivelent of of XMLConstants.W3C_XML_SCHEMA_NS_URI, which means the schema is WXS XSD. In theory, if you have the relevant 3rd party libraries available, you could use RELAX-NG and other schema types too.
<xmltransform
    var="varname"
    destvar="destvarname"
    xslt="urlOfXSLT">
  <param
      name="paramname"
      var="paramvarname"/>
  ... more params
</xmltransform>
XSLT transform variable varname, storing the result in variable destvarname. Throws an exception if variable varname is not defined, is Null, or is not a Document. Also throws an exception if the XSLT transformation fails. Each <param/> element results in a parameter being passed to the XSLT transformer, whose name is paramname and whose value is the value of the variable paramvarname. Throws an exception if any variable paramvarname is not defined, or is not of type String.

Flow steps - Control flow

Element Action
<call
    flow="nestedflowname"/>
Call the indicated nested flow, and return back to this one. If the nested flow definition has a eflow attribute, then it is ignored. Any exception raised during the processing of the nested flow is still processed by the parent eflow. See the exception handling section for details.
<switch
    var="varname">
  <case
      regexp="pattern"
      flow="nestedflowname"/>
  ... more cases
</switch>
Fetch the variable varname. If not defined, Null, or not a String, then throw an exception. Step through each <case/>, attempting to match against the regular expression. If a match is found, call the nested flow, and do not consider any subsequent <case/>s. If no match is found, no nested flow is called, and no exception is thrown. Any exception raised during the processing of the nested flow is still processed by the parent eflow. The matching is for a complete match, ie: the regular expression has to match the entire value, not just a portion of it. There is no <default flow="defaultflow"/>, this can be acheived using <case regexp=".*" flow="defaultflow"/>.
<throw
    cause="Too complicated"/>
Throws an exception with the indicated cause in the text of the exception.

Flow steps - Debugging

Element Action
<debug
    logfile="filename"
    logmessage="true|false"
    logvars="true|false"
    vars="v1,v2,v3"/>
    
Logs the state of the flow context. If logmessage is specified, the log is appended to the indicated file, else it is displayed to standard output. If logmessage="true" (default is false), a textual representation of the message is output. If logvars="true" (default is false), a textual representation of the variables is output. If vars is specified, only the named variables are output, else all of them are.

As an example, if you had the following flow :-

<flow name="AppDeal_to_StdDeal">
  <bodyget item="appNode"/>
  <vartype type="Document"/>
  <debug name="after-parsing" logvars="true" logmessage="true"/>
  ... the rest omitted
</flow>

Then the following could be output :-

=== FlowStep after-parsing
Message propertes:
Message body (MapMessage):
  item "appNode" = String "<?xml version="1.0"?>
<AppDeal>
  <trade>1234</trade>
  <stock>Gold</stock>
  <amount>1000</amount>
  <country>UK</country>
  <site>London</site>
</AppDeal>
"
Variables:
  var "v0" = Document
        <?xml...
        <AppDeal>
        - "
  "
        - <trade>
        - - "1234"
        - </trade>
        - "
  "
        - <stock>
        - - "Gold"
        - </stock>
        - "
  "
        - <amount>
        - - "1000"
        - </amount>
        - "
  "
        - <country>
        - - "UK"
        - </country>
        - "
  "
        - <site>
        - - "London"
        - </site>
        - "
"
        </AppDeal>

The textual rendition of the message and variables is not intended to be suitable for cut-n-pasting into a file and processing further. Its intended to facilitate problem diagnosis. For this reason, the string representation is known as the "diagnostic form". This is why types are shown, and strings "quoted".

Flow Steps - User defined

I'm not likely to be able to second guess every conceivable thing a customer might like to do with the JMS Message or variables. So there is an extension mechanism.

Element Action
<stepuser
    class="className"
    ... other attributes >
  ... other nested elements
</stepuser>
Allow the user to add their own step which does whatever they need. className is loaded, and its constructor called with the DOM org.w3c.dom.Node corresponding to the <stepuser/>, and the implementation then reads its configuration. At runtime, the process method is called passing the FlowContext, and this method can then manipulate the message and variables as it sees fit.

A simple user defined step :-

//
// FlowStepCompare.java - Check two variables are the same
//

package mypackage;

import org.w3c.dom.Element;
import org.w3c.dom.NodeList;
import org.w3c.dom.NamedNodeMap;
import org.w3c.dom.Node;
import org.w3c.dom.Text;
import javax.jms.JMSException;
import nyangau.jmsadapter.*;

public class FlowStepCompare extends FlowStep
  {
  protected String var1;
  protected String var2;
  protected String destvar;
  public FlowStepCompare(Node eStep)
    throws FlowProcessorException
    {
    super(eStep);
    NamedNodeMap nnmStep = eStep.getAttributes();
    Node aVar1 = nnmStep.getNamedItem("var1");
    var1 = ( aVar1 != null ) ? aVar1.getNodeValue() : "v0";
    Node aVar2 = nnmStep.getNamedItem("var2");
    var2 = ( aVar2 != null ) ? aVar2.getNodeValue() : "v0";
    Node aDestVar = nnmStep.getNamedItem("destvar");
    if ( aDestVar == null )
      throw new FlowProcessorException("missing destvar attribute in step "+name);
    destvar = aDestVar.getNodeValue();
    }
  public void process(FlowContext c)
    throws FlowProcessorException, JMSException
    {
    Object o1 = c.getVar(var1);
    if ( o1 == null )
      throw new FlowProcessorException(var1+" is undefined in step "+name);
    if ( o1 instanceof FlowNullValue )
      throw new FlowProcessorException(var1+" is Null in step "+name);
    Object o2 = c.getVar(var2);
    if ( o2 == null )
      throw new FlowProcessorException(var2+" is undefined in step "+name);
    if ( o2 instanceof FlowNullValue )
      throw new FlowProcessorException(var2+" is Null in step "+name);
    if ( o1.toString().equals( o2.toString() ) )
      c.setVar(destvar, "same");
    else
      c.setVar(destvar, "different");
    }
  }

The jmsadapter.xsd used to validate JMS Adapter XML configurations is designed to use <xsd:any/> and <xsd:anyAttribute/> so as to allow you to put your own elements and attributes in the XML configuration file. In this case, to use the step, you'd use :-

<stepuser classname="mypackage.FlowStepCompare"
          var1="v1" var2="v2" destvar="comparisonResult"/>

Remember to put your .class or .jar file on the classpath.

Data maps

Data maps are things which map input String values to output String values.

In one message format, the US Dollar might appear as $ and in another message format, it might be USD. Data maps provide a mechanism for mapping from one form to the other.

All data maps can have a name="datamapname" attribute. In the examples below, the name attribute is shown where it makes sense, ie: for top level data maps, not for nested ones. Any named top level data map can referenced by a <varmap/> step.

Data maps - To a default value

The following is a data map which maps every unmapped input value to the output value UNKNOWN.

<mapdefault name="to_Unknown" dest="UNKNOWN"/>

Doesn't seem very useful on its own, but becomes more useful when used at the end of a <maplist/>, see below.

Data maps - Using inline data

The following is a data map which performs a set of fixed mappings, which are explicitly stated in the XML configuration :-

<mapinline name="CurrencySymbol_to_CurrencyCode">
  <maplet src="$" dest="USD"/>
  <maplet src="&#163;" dest="GBP"/>
</mapinline>

Data maps - Using properties data

Here is a data map which reads properties from a URL whenever its asked to perform a mapping :-

<mapprops name="CurrencySymbol_to_CurrencyCode"
          properties="file:CurrencySymbol_to_CurrencyCode.properties"/>

This variant reads properties in XML format :-

<mapprops name="CurrencySymbol_to_CurrencyCode"
          xml="file:CurrencySymbol_to_CurrencyCode.xml"/>

Both properties and xml attributes can be given, in which case it reads the properties followed by XML properties.

Other URLs (such as those that use http: and jar: protocols) can also be used.

Reading an entire file (or URL) every time you want to satisfy a mapping request only makes sense if the data is constantly changing. To avoid this, you can use <mapcache/>, described later. Having just read the entire file (or URL), this data map returns all the mappings from the file, giving <mapcache/> the opportunity to cache it all.

Data maps - Using a database query

Sometimes mapping data exists inside database tables.

Imagine ELEMENT_TABLE contains :-

NAMESYMBOL
GoldAu
SilverAg
LeadPb

The following data map can be used :-

<mapdb name="ElementName_to_ElementSymbol"
       driver="oracle.jdbc.OracleDriver"
       url="jdbc:oracle:thin@127.0.0.1@1521:XE"
       userid="scott" password="tiger"
       scolumns="NAME" dcolumn="SYMBOL" table="ELEMENT_TABLE"/>

If asked to map Gold and Silver, the data map creates a query of the form :-

SELECT "NAME","SYMBOL" FROM ELEMENT_TABLE WHERE
  ( "NAME"='Gold'  ) OR
  ( "NAME"='Silver')

The database replies with matching rows, and the data map uses them.

A more complex kind of mapping may be appropriate, in which the items to be mapped actually span multiple columns. Consider the CSL_TABLE table :-

countrysitelocation
UKLondonLONDON
USParisPARIS_TEXAS
FRParisPARIS_FRANCE

This could have been set up using the following SQL, which is a part of the example shipped with JMS Adapter, and is helpfully included in the example/setupCslTable.sql file :-

-- Set up the CSL_TABLE used by the <mapdb> mapping

CREATE TABLE CSL_TABLE
	(
	"country"  varchar(10),
	"site"     varchar(10),
	"location" varchar(20)
	) ;

INSERT INTO CSL_TABLE VALUES ( 'UK', 'London', 'LONDON'       ) ;
INSERT INTO CSL_TABLE VALUES ( 'US', 'Paris' , 'PARIS_TEXAS'  ) ;
INSERT INTO CSL_TABLE VALUES ( 'FR', 'Paris' , 'PARIS_FRANCE' ) ;

SELECT * FROM CSL_TABLE ;

-- End

The following data map can be used :-

<mapdb name="CountrySite_to_Location"
       driver="oracle.jdbc.OracleDriver"
       url="jdbc:oracle:thin@127.0.0.1@1521:XE"
       userid="scott" password="tiger"
       split="," scolumns="country,site" dcolumn="location" table="CSL_TABLE"/>

Note the use of split="," to identify how to split the value to be mapped into peices, and the also note that the scolumns attribute identifies a number of columns. scolumns is always seperated by , characters, regardless of the split attribute (which applies to the data to be mapped only).

If asked to map UK,London and US,Paris, the data map creates a query of the form :-

SELECT "country","site","location" FROM CSL_TABLE WHERE
  ( "country"='UK' AND "site"='London' ) OR
  ( "country"='US' AND "site"='Paris'  )

Databases may temporarily be unreachable or unavailable. The previous example may be written with the default behaviour shown explicitly. ie: No retry occurs, and if we can't reach the database an exception is thrown :-

<mapdb name="CountrySite_to_Location"
       driver="oracle.jdbc.OracleDriver"
       url="jdbc:oracle:thin@127.0.0.1@1521:XE"
       userid="scott" password="tiger"
       split="," scolumns="country,site" dcolumn="location" table="CSL_TABLE"
       retrycount="0" retrydelay="0" retryfail="throw"/>

Here is a different strategy, in which 10 retries are performed (11 tries in total) with a 100ms delay between each, and if we still can't reach the database, execution proceeds without throwing an exception, presumably in the hope that a later mapping in a <maplist/> will be able to fill in the missing gaps :-

<mapdb name="CountrySite_to_Location"
       driver="oracle.jdbc.OracleDriver"
       url="jdbc:oracle:thin@127.0.0.1@1521:XE"
       userid="scott" password="tiger"
       split="," scolumns="country,site" dcolumn="location" table="CSL_TABLE"
       retrycount="10" retrydelay="100" retryfail="proceed"/>

Data maps - Using a web service

WSDL for a JMS Adapter Mapper web service has been defined. Such a web service accepts a mapping name and a list of source values to map, and returns the list with an even number of elements, comprised of source1, destination1, source2, destination2, ... Only successfully mapped source values have corresponding source, destination pairs in the output.

If the web service wants to indicate an error condition, it returns a list comprised of one element only - the error text. An obvious error condition is if the mapping name isn't understood. In this case, an exception is thrown, including the error text.

Here is an example of calling such a service :-

<mapws name="ElementName_to_ElementSymbol"
       wsdl="http://localhost:8888/ElementMapper?WSDL"
       mapping="Name_to_Symbol"/>

If the mapping attribute is omitted, it defaults to the empty string. The intent of the mapping name is to allow the web service to be able to perform a variety of mappings, and the client (in this case JMS Adapter) select among them.

The namespace used in the WSDL file can also be specified using the namespace attribute, although the default of http://www.nyangau.org/jmsadapter/mapper is normally fine. The service name in the WSDL file can also be specified using the service attribute, although the default of MapperService is normally fine. The port name in the WSDL file can also be specified using the port attribute, although the default of MapperPort is normally fine.

Web services may temporarily be unreachable or unavailable. The previous example may be written with the default behaviour shown explicitly. ie: No retry occurs, and if we can't reach the web service an exception is thrown :-

<mapws name="ElementName_to_ElementSymbol"
       wsdl="http://localhost:8888/ElementMapper?WSDL"
       mapping="Name_to_Symbol"
       retrycount="0" retrydelay="0" retryfail="throw"/>

Here is a different strategy, in which 10 retries are performed (11 tries in total) with a 100ms delay between each, and if we still can't reach the web service, execution proceeds without throwing an exception, presumably in the hope that a later mapping in a <maplist/> will be able to fill in the missing gaps :-

<mapws name="ElementName_to_ElementSymbol"
       wsdl="http://localhost:8888/ElementMapper?WSDL"
       mapping="Name_to_Symbol"
       retrycount="10" retrydelay="100" retryfail="proceed"/>

Implementing the web service using JAX-WS

You can use JAX-WS to build a Mapper web service. A simple implementation built using JAX-WS looks like this :-

//
// Mapper.java - Mapper Web Service
//
// This implementation exists as a template to generate WSDL from,
// and model to create others from.
// It maps between element names and symbols.
// It uses an ArrayOfString class so as to be sure the wrapped document literal
// SOAP has an element enclosing the elements of the lists of strings.
//

package nyangau.jmsadapter.mapper;

import java.util.Set;
import java.util.Map;
import java.util.HashMap;
import java.util.List;
import java.util.ArrayList;
import javax.jws.WebService;
import javax.jws.WebMethod;
import javax.jws.WebParam;
import javax.jws.WebResult;
import javax.xml.ws.Endpoint;

@WebService(
  targetNamespace="http://www.nyangau.org/jmsadapter/mapper",
  serviceName="MapperService"
  )
public class Mapper
  {
  protected Map<String,String> mNameToSymbol = new HashMap<String,String>()
    {{
    put("Gold"  , "Au");
    put("Silver", "Ag");
    }};
  protected Map<String,String> mSymbolToName = new HashMap<String,String>()
    {{
    put("Au", "Gold"  );
    put("Ag", "Silver");
    }};
  @WebMethod()
  public @WebResult(targetNamespace="http://www.nyangau.org/jmsadapter/mapper", name="mapResult") ArrayOfString map(
    @WebParam(targetNamespace="http://www.nyangau.org/jmsadapter/mapper", name="mapName") String mapName,
    @WebParam(targetNamespace="http://www.nyangau.org/jmsadapter/mapper", name="unmapped") ArrayOfString unmapped
    )
    {
    List<String> mapped = new ArrayList<String>();
    Map<String,String> m;
    if ( mapName.equals("Name_to_Symbol") )
      m = mNameToSymbol;
    else if ( mapName.equals("Symbol_to_Name") )
      m = mSymbolToName;
    else
      // Failures are indicated by returning a list of length 1
      {
      mapped.add("bad mapName: "+mapName);
      return new ArrayOfString( mapped );
      }
    for ( String u : unmapped.getList() )
      {
      String s = m.get(u);
      if ( s != null )
        {
        mapped.add(u);
        mapped.add(s);
        }
      }
    return new ArrayOfString( mapped );
    }
  public static void main(String[] args)
    {
    Endpoint.publish("http://localhost:8888/ElementMapper", new Mapper());
    }
  }

It uses a helper class to ensure that in the wrapped document literal SOAP messages, the lists of strings are enclosed in a parent element. This is not needed when working purely in Java with JAX-WS, but becomes important when trying to interoperate between Java and .NET.

//
// ArrayOfString.java - Ensure lists of strings have an enclosing element
//

package nyangau.jmsadapter.mapper;

import java.util.List;
import java.util.ArrayList;
import javax.xml.bind.annotation.XmlElement;

public class ArrayOfString
  {
  @XmlElement(namespace="http://www.nyangau.org/jmsadapter/mapper", name="string")
  protected List<String> list;
  public ArrayOfString() { list = new ArrayList<String>(); }
  public ArrayOfString(List<String> l) { list = l; }
  public List<String> getList() { return list; }
  }

Note the way in which I force the namespace to my chosen value everywhere I can. This appears to be necessary to get consistent WSDL from both the wsgen command and from the URL ending in ?WSDL served by the running web service, and also makes it easier when trying to interoperate with .NET.

The sample can be run on the command line (for testing purposes), and whilst running, its WSDL is available at http://localhost:8888/ElementMapper?WSDL.

Important: JMS Adapter uses web service client classes which are wsimported from the WSDL which was wsgenerated from the above sample code.

Implementing the web service using .NET using ASMX

A simple .NET ASMX based implementation should look like this :-

<%@ WebService Language="C#" class="Nyangau.JmsAdapter.Mapper.MapperService" %>

using System;
using System.Collections.Generics;
using System.Web.Services;

namespace Nyangau.JmsAdapter.Mapper
  {
  [WebService (Namespace="http://www.nyangau.org/jmsadapter/mapper")]
  public class MapperService : WebService
    {
    [WebMethod]
    public List<string> map(string mapName, List<string> unmapped)
      {
      List<string> l = new List<string>();
      foreach ( String s in unmapped )
        if ( s.Equals("Gold") )
          {
          l.Add(s);
          l.Add("Au");
          }
      return l;
      }
    }
  }

Ok, so this doesn't do a very good job of mapping - the point here is to illustrate all the scaffolding needed around the task at hand.

This would be put into a file ElementMapper.asmx and placed into a suitable IIS website or virtual directory.

Its WSDL can be fetched by appending ?WSDL to the end of the URL, eg: http://somehost/somepath/ElementMapper.asmx?WSDL. You do need to use this WSDL, as not only does it have the correct service URL, it also has a SOAPAction in it which .NET web services need (and is not present in JAX-WS WSDL).

It is also necessary to specify port="MapperServiceSoap" in the <mapws/> element, as the .NET ASMX WSDL names its ports this way.

ASMX based web services aren't strategic, and have been superceded by WCF based web services. Shame, they're certainly the simplest.

This sample code is supplied for reference in the JMS Adapter package, in the mapper.asmx subdirectory.

Implementing the web service using .NET using WCF

A simple .NET WCF based web service has an interface :-

//
// IMapperService.cs - Service Interface
//

using System;
using System.Collections.Generic;
using System.Runtime.Serialization;
using System.ServiceModel;
using System.Text;

namespace WcfMapper
    {
    [ServiceContract(Namespace="http://www.nyangau.org/jmsadapter/mapper")]
    public interface IMapperService
        {
        [OperationContract]
        ArrayOfString map(string mapName, ArrayOfString unmapped);
        }
    [CollectionDataContract(ItemName="string", Namespace="http://www.nyangau.org/jmsadapter/mapper")]
    public class ArrayOfString : List<string> { }
    }

Note the way the [CollectionDataContract] is used to ensure that when lists of strings are converted to XML, they have elements called string, so as to match the JAX-WS code and ASMX .NET examples above.

It also has an implementation of the interface :-

//
// MapperService.cs - Service Implementation
//

using System;
using System.Collections.Generic;
using System.Runtime.Serialization;
using System.ServiceModel;
using System.Text;

namespace WcfMapper
    {
    [ServiceBehavior(Namespace="http://www.nyangau.org/jmsadapter/mapper")]
    public class MapperService : IMapperService
        {
        public ArrayOfString map(string mapName, ArrayOfString unmapped)
            {
            ArrayOfString l = new ArrayOfString();
            foreach (String s in unmapped)
                if ( s.Equals("Gold") )
                    {
                    l.Add(s);
                    l.Add("Au");
                    }
            return l;
            }
        }
    }

Metadata is needed in the form of an App.config file :-

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <system.web>
    <compilation debug="true" />
  </system.web>
  <system.serviceModel>
    <services>
      <service name="WcfMapper.MapperService" behaviorConfiguration="WcfMapper.MapperServiceBehavior">
        <host>
          <baseAddresses>
            <add baseAddress="http://windowshost/MapperService/" />
          </baseAddresses>
        </host>
        <endpoint address="" binding="basicHttpBinding" contract="WcfMapper.IMapperService"
                  bindingNamespace="http://www.nyangau.org/jmsadapter/mapper">
          <identity>
            <dns value="windowshost"/>
          </identity>
        </endpoint>
        <endpoint address="mex" binding="mexHttpBinding" contract="IMetadataExchange"/>
      </service>
    </services>
    <behaviors>
      <serviceBehaviors>
        <behavior name="WcfMapper.MapperServiceBehavior">
          <serviceMetadata httpGetEnabled="True"/>
          <serviceDebug includeExceptionDetailInFaults="True" />
        </behavior>
      </serviceBehaviors>
    </behaviors>
  </system.serviceModel>
</configuration>

Note the use of bindingNamespace to complete the job of ensuring everything is in the right namespace. Its not enough to use attributes in the C# code. This attribute isn't present in a newly created Visual Studio WCF project.

Its WSDL can be fetched by appending ?WSDL to the end of the URL, eg: http://windowshost/MapperService/?WSDL. Notice that httpGetEnabled="True", so that the WSDL is exposed. Again, this attribute isn't present in a newly created Visual Studio WCF project.

It is also necessary to specify port="BasicHttpBinding_IMapperService" in the <mapws/> element, as the .NET WCF WSDL names its ports this way.

This sample code is supplied for reference in the JMS Adapter package, in the mapper.wcf subdirectory.

Data maps - Mapping and caching returned results

Some mappings are expensive to perform. Perhaps they read files or from URLs, or make queries to external services, such as databases or web services, perhaps they do difficult computations. So you may want to cache the results and avoid remapping the same value for a while (or maybe even indefinitely).

This here is a data map which caches data returned :-

<mapcache name="ElementName_to_ElementSymbol" ttl="60000" size="1000">
  <mapws wsdl="http://localhost:8888/ElementMapper?WSDL"
         mapping="Name_to_Symbol"/>
</mapcache>

Any mapped value returned from the inner mapping is kept for at most 60000ms. Note that this is 60000ms from the moment the value is returned from the inner mapping, not from the moment the inner mapping is invoked. If ttl is not specified, returned mappings are not expired.

At most 1000 returned values are kept in the cache. If size is not specified, the default is 100.

When the inner data map is invoked, it can return more mappings than it is asked for, and the cache will retain everything that is returned.

So in the following example, if asked to map "Monday" and "Tuesday", and later asked to map "Wednesday", the file is only read once. This is because the size is large enough to keep all the answers and there is no ttl defined :-

<mapcache name="EnglishDayNames_to_FrenchDayNames" size="7">
  <mapprops xml="file:EnglishDayNames_to_FrenchDayNames.xml"/>
</mapcache>

Data maps - Mapping using a list of data maps

Mappings can be assembled to make compound maps using <maplist/>, eg:

<maplist name="tryLocalThenLookupInDatabase">
  <mapprops properties="file:CountryNames_to_CountryCodes.properties"/>
  <mapdb driver="oracle.jdbc.OracleDriver"
         url="jdbc:oracle:thin@127.0.0.1@1521:XE"
         userid="scott" password="tiger"
         scolumns="cty_name" dcolumn="cty_code" table="CTY_TABLE"/>
  <mapdefault dest="XX"/>
</maplist>

The above example has a local file which maps United Kingdom to UK, France to FR etc.. However, if after this properties file has been searched, any remaining values still remain unmapped, it consults the database, and if after that there are unmapped values they are mapped to XX. Note how <mapdefault/> ensures the mapping as a whole cannot fail, and no exception will be thrown

Note that simple mappings, <mapcache/> and <maplist/> can be combined in a variety of ways. You can construct maps that consult a variety of sources, and cache the results from each of them for varying lengths of time.

Data maps - User defined maps

I'm not likely to be able to second guess every conceivable mapping a customer might like to do. So there is an extension mechanism.

<mapuser
    class="className"
    ... other attributes >
  ... other nested elements
</mapuser>

This allows the user to add their own data map which does whatever they need. The className is loaded, and its constructor called with the DOM org.w3c.dom.Node corresponding to the <mapuser/>, and the implementation then reads its configuration. At runtime, the map method is called passing a set of strings to map, and a map to populate with mappings.

The rule is that any mapping that is successfully performed must be removed from the unmapped set, and the mapping added to the mapped map. Additional mappings may be added to the the mapped map, in the hope there is a caching layer above to retain them.

A somewhat contrived simple user defined mapping :-

//
// DataMapToLowerCase.java - Map to lower case
//

package mypackage;

import java.util.Set;
import java.util.Map;
import org.w3c.dom.Element;
import org.w3c.dom.NodeList;
import org.w3c.dom.NamedNodeMap;
import org.w3c.dom.Node;
import org.w3c.dom.Text;
import nyangau.jmsadapter.*;

public class DataMapToLowerCase extends DataMap
  {
  public DataMapToLowerCase(Node eDataMap)
    throws FlowProcessorException
    {
    super(eDataMap);
    }
  public void map(Set<String> unmapped, Map<String,String> mapped)
    throws FlowProcessorException
    {
    for ( String s : unmapped )
      {
      mapped.put(s, s.toLowerCase());
      unmapped.remove(s);
      }
    }
  }

The jmsadapter.xsd used to validate JMS Adapter XML configurations is designed to use <xsd:any/> and <xsd:anyAttribute/> so as to allow you to put your own elements and attributes in the XML configuration file. In this case, to use the step, you'd use :-

<mapuser classname="mypackage.DataMapToLowerCase"/>

When writing your own data map, there is usually no need to implement any kind of caching, as you can always wrap a <mapcache/> around it.

Remember to put your .class or .jar file on the classpath.

There should be limited requirements to write user defined data maps, given that the ability to invoke web services is built-in.

Data maps - Tuple Mapping

The data mapping framework delivered with JMS Adapter only really understands the most common case of mapping a single value to a single value.

Sometimes you can want to map a single value to multiple output values, eg: a "location" to a tuple comprising a "country" and the "site" within the country. This should be handled by creating a mapping from location to country, and a mapping from location to site :-

For
  location -> ( country , site )
    "PARIS_FRANCE" -> ( "FR" , "Paris" )
    "PARIS_TEXAS"  -> ( "US" , "Paris" )
Use
  location -> country
    "PARIS_FRANCE" -> "FR"
    "PARIS_TEXAS"  -> "US"
  location -> site
    "PARIS_FRANCE" -> "Paris"
    "PARIS_TEXAS"  -> "Paris"

You might also want to map a number of values to a single value, eg: mapping a tuple comprised of "country" and "site" to a "location". To handle this, you should use XSLT to put the input values next to each other, seperated by a seperator such as a comma. The seperator should not be something occuring naturally in either of the values. This should not be difficult to find, as you have the whole Unicode character set at your disposal.

For
  ( country , site ) -> location
    ( "FR" , "Paris" ) -> "PARIS_FRANCE"
    ( "US" , "Paris" ) -> "PARIS_TEXAS"
Use
  country,site -> location
    "FR,Paris" -> "PARIS_FRANCE"
    "US,Paris" -> "PARIS_TEXAS"

A similar approach can be used for 1:3, 3:1, 2:2 and N:M mapping requirements.

Exception handling

As you will have seen, there are many reasons why the mediation of a message could fail, and exception being thrown.

JMS Adapter must have a strategy for handling this.

We don't have the option of letting an exception be thrown to the caller as a Java Exception. The caller isn't expecting anyone to be mediating messages, and so this would be an unwelcome surprise.

Traditionally, an adapter sends unprocessable messages to a dead letter queue, or a message hospital. Hopefully a human being gets to look at these, possibly fix them up and/or replay them :-

However, JMS Adapter has no such luxury. If asked to send a message, it had better send a message. If asked to receive a message, it had better return one. This is required so as to preserve all the nice message commit and any XA semantics that may be in force.

So we adopt a different approach. We rely on the fact an application must have a strategy of its own for dealing with bad/poisonous messages. Typically this would be to send the message to a dead letter queue, or a message hospital.

When we fail to mediate a message, all we need to do is create a deliberately bad message, and pass it onward. For a JMS Adapter attached to a publisher, this looks like this :-

And to a subscriber, it looks like this :-

This is why flows have an eflow attribute, which identifies an exception flow.

It is the job of the exception flow to create a deliberately bad message. eg: If the application expects a TextMessage, consider sending it a MapMessage. eg: If it expects a JMS property called "trade-id", don't give it one.

The bad message should ideally contain the identity of the message. This is an important point - whoever gets to investigate the message is going to want to know which message it was, so the main flow should strive to fish this information out of the message being mediated as soon as possible. As the failure could happen at various points throughout the flow, the message itself could be in a partially converted state when the failure happens. The message identity might be in the "TradeNumber" JMS property, or have to be XPath-ed out of the payload. The main flow should aim to store the message identity in the special _JMSAdapter_message_id variable as a String, as soon as possible.

The bad message should also contain some diagnostic information. At the point the exception flow is invoked, the following special variables will be set up :-

Pass the diagnostic information in JMS headers, and perhaps the payload, in such a way that the receiving application won't expect and interpret them. Here is an example :-

<flow name="StdConfirmation_to_AppConfirmation" eflow="make_poison_message">
  <propget prop="trade" var="_JMSAdapter_message_id"/>
  ... rest omitted
</flow>

<flow name="make_poison_message">
  <bodytype type="MapMessage"/>
  <varset value="JMS Adapter ${ja_name} couldn't process a message" var="info"/>
  <propset var="info" prop="BAD_explanation"/>
  <vardef var="_JMSAdapter_flow_message_id" value="unknown"/>
  <propset var="_JMSAdapter_flow_message_id" prop="BAD_message_id"/>
  <bodyset var="_JMSAdapter_flow_name" item="BAD_flow_name"/>
  <vartype var="_JMSAdapter_flow_exception" type="String"/> 
  <bodyset var="_JMSAdapter_flow_exception" item="BAD_flow_exception" clear="false"/>
  <bodyset var="_JMSAdapter_flow_context" item="BAD_flow_context" clear="false"/>
</flow>

Its quite likely you'd develop one exception flow and refer to it from lots of flows.

If no eflow attribute is present, then the message is converted to a raw JMS Message with no payload, and the previously described _JMSAdapter_* variables are copied to JMS message properties of the same name.

Exception flows raising exceptions

Of course, you could get an exception running the exception flow. This might happen during development, but an exception flow should really be written not to do this in Production.

Notice the use of <vardef/> in the example above to avoid failing if _JMSAdapter_flow_message_id isn't set yet when the exception flow is invoked. This is an example of defensive programming.

If an exception flow was not robust enough, and did throw an exception, then the message is converted to a raw JMS Message with no payload, and any of the previously described _JMSAdapter_* variables which are still defined are copied to JMS message properties of the same name.

In addition, the following JMS message properties are set up :-

At least thats something to go on, although its obviously better to defensively write the exception flow in the first place.

The eflow attribute on an exception flow is ignored.

Command line

JMS Adapter can be run on the command line. It includes a little script called run.sh which basically just puts enough on the classpath, and looks like this :-

#!/bin/ksh

d=`dirname $0`
java -cp $d/nyangau-jmsadapter.jar:$d/jms.jar nyangau.jmsadapter.JmsaMain "$@"

If you've written your own flow step or data map classes, you'd need these on the classpath too.

Version

You can check the version of JMS Adapter :-

$ ./run.sh version
JMS Adapter 0.7

Validate configuration

You can validate your JMS Adapter XML configuration. The location of the XML configuration should be specified as a URL, ie: if its a file, it needs the file: prefix. If the XML configuration refers to ${properties}, you'll need to specify the URL of the properties too. If the XML configuration refers to other resources, such as XSD files, XSLT files, properties files, etc.., and these use relative file URLs, make sure you run JMS Adapter from the right directory. If the XML configuration refers to web service WSDL, be sure it is accessible. No output means everything is ok, exception stack traces indicate problems :-

$ cd example
$ ../run.sh validate file:appstd.xml file:appstd.properties

If you refer to a ${property} but don't define it, or perhaps you forget to pass the URL of the properties, you could see :-

$ ../run.sh validate file:appstd.xml
nyangau.jmsadapter.FlowProcessorException: can't expand ${ja_name} as property or system property
        at nyangau.jmsadapter.FlowHelper.substituteProps(FlowHelper.java:322)
        at nyangau.jmsadapter.FlowHelper.substituteProps(FlowHelper.java:334)
        at nyangau.jmsadapter.FlowProcessor.<init>(FlowProcessor.java:176)
        at nyangau.jmsadapter.JmsaMain.main(JmsaMain.java:51)

If web service WSDL wasn't available, you might see something like this :-

$ ../run.sh validate file:appstd.xml file:appstd.properties
nyangau.jmsadapter.FlowProcessorException: WebServiceException in data map ElementName_to_ElementSymbol
        at nyangau.jmsadapter.DataMapWebService.<init>(DataMapWebService.java:65)
        at nyangau.jmsadapter.DataMap.newDataMap(DataMap.java:62)
        at nyangau.jmsadapter.FlowProcessor.<init>(FlowProcessor.java:252)
        at nyangau.jmsadapter.JmsaMain.main(JmsaMain.java:51)
Caused by: javax.xml.ws.WebServiceException: Failed to access the WSDL at: http://localhost:8888/ElementMapper?WSDL. It failed with: 
        Connection refused.
... snip

The XML configuration is validated using a file called jmsadapter.xsd, which is included as a resource inside the nyangau-jmsadapter.jar. This validation process catches a very large proportion of illegal configurations. If your configuration passes this test, there is a very low chance an exception will be thrown due to a configuration error when the application using JMS Adapter is run.

Standalone testing

It can be handy to be able to work on flows without having any JMS provider or application handy.

If you look at a given flow, it might typically have a structure like this :-

<flow name="somename">
  ... fetch parts of the inbound message to variables
  ... process the variables (validate, enrich, transform, ...etc)
  ... store variables into the outbound message
</flow>

If you comment out the first part, you can do things like this :-

$ cd example
$ ./run_app_debug.sh 

Which is coded like this :-

#!/bin/sh

d=`dirname $0`/..
java \
-DCSLMAP=FILE \
-cp $d/nyangau-jmsadapter.jar:$d/jms.jar:ojdbc14.jar \
nyangau.jmsadapter.JmsaMain \
flow \
file:appstd.xml \
file:appstd.properties \
AppDeal_to_StdDeal \
v0=:Hello file:AppDeal.xml

The Flow Processor is initialised, using the XML configuration and properties supplied. If there are no properties, pass -. The indicated flow is located (eg: AppDeal_to_StdDeal).

Next a Flow Context is prepared containing a plain old Message with no body, and no variables defined. The remaining arguments are iterated over. Any argument with an = is treated as a variable assignment. The variable is assigned the text from the URL indicated. An argument without an = causes the message to become a TextMessage with content from the URL indicated.

If a URL starts with : then its a special case - the value is the text immediately following. Kind of like a "here document", but for URLs.

The context is displayed, the flow executed, then the context re-displayed.

The above example could produce :-

$ cd example
$ ../run_app_debug.sh
=== BEFORE AppDeal_to_StdDeal
Message propertes:
Message body (TextMessage):
  text = "<?xml version="1.0"?>
<AppDeal>
  <trade>1234</trade>
  <stock>Gold</stock>
  <amount>1000</amount>
  <country>UK</country>
  <site>London</site>
</AppDeal>
"
Variables:
  var "v0" = String "Hello"

Debugging output in the middle not shown, then ...

=== AFTER AppDeal_to_StdDeal
Message propertes:
  prop "trade" = String "1234"
  prop "Comment" = String "Transformed by JMS Adapter"
Message body (TextMessage):
  text = "<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<StdDeal>
    <stock>Au</stock>
    <amount>1000</amount>
    <where location="LONDON"/>
</StdDeal>
"
Variables:
  var "v0" = String "Transformed by JMS Adapter"
  var "_JMSAdapter_message_id" = String "1234"

The example above is lucky in that the flow can just as easily work with a TextMessages as it can a MapMessages. This might not always be the case, hence the comment above about commenting out the logic which reads the variables from the message.

Examples

Validating the message

Sometimes we only want to be sure the message being sent is valid. The message is a TextMessage, and needs to be validated against a particular schema :-

<flow name="BytesMessage_to_TextMessage" eflow="make_poison_message">
  <bodyget/>
  <vartype type="Document"/>
  <xmlvalidate schema="file:AppDeal.xsd"/>
</flow>

An eflow attribute is used as the message text may not parse or validate cleanly.

Changing the body type of the message

In this example, the message is a BytesMessage and the payload is XML. This means we should infer the character encoding using the XML character encoding rules, so we use encoding="XML". We want a TextMessage :-

<flow name="BytesMessage_to_TextMessage" eflow="make_poison_message">
  <bodyget/>
  <vartype type="String" encoding="XML"/>
  <bodytype type="TextMessage"/>
  <bodyset/>
</flow>

An eflow attribute is used as we could have difficulty decoding the bytes into characters using the designated character encoding. eg: the XML could start with :-

<?xml encoding="something-silly"?>

Content based flow processing

Note I am careful not to say "content based routing", as JMS Adapter never dynamically picks which underlying destination to publish or subscribe to. Here we consider what happens if more than one message type flows through a given queue or topic. Consider that "buys" and "sells" are sent through the same queue, and we can tell which is which by the top level document node name :-

<flow name="SenderBuyOrSell_to_ReceiverBuyOrSell">
  <bodyget/>
  <vartype type="Document"/>
  <varselect xpath="name(/*)" destvar="DOCUMENT_NODE"/>
  <switch var="DOCUMENT_NODE">
    <case regexp="SenderBuy" flow="SenderBuy_to_ReceiverBuy"/>
    <case regexp="SenderSell" flow="SenderSell_to_ReceiverSell"/>
  </switch>
  <bodyset/>
</flow>

A more defensive approach might be to add the following limb :-

    <case regexp=".*" flow="strange_sender_message"/>

and the following flow :-

<flow name="strange_sender_message">
  <throw cause="not a SenderBuy or SenderSell"/>
</flow>

Other XPath expressions could be used to make decisions off of other parts of the message.

The example included with JMS Adapter

You have seen some of this already, some of its artifacts have already been used to illustrate points earlier in this document.

The example is included in the example subdirectory.

Scenario

A standard service expects standard deals on its input deal queue, and it sends standard confirmations on its output confirmation queue. We have an application which generates deals in an application specific deal format and expects confirmations back in an application specific format.

The problem here is that the application sends something very different to what the standard service expects, and the standard service replies with something very different to what the application expects. An ideal opportunity for JMS Adapter to do its magic.

In order to convert between application specific and standard service specific formats, there are structural differences in the messages and there are data elements/attributes which need to be mapped using an external web service and an external database table.

The database is not not shown in the picture, as its use is optional. We can persuade the example not to use it, and to use local file data instead.

Here is a pictorial representation of the example :-

The JMS Adapter administered objects set Jmsa.CONFIG to file:appstd.xml. The XML configuration is named this because it converts between application and standard messages.

XSD schemas are supplied describing the 4 different message formats. The application reads a sample valid application deal from a file and sends it. The standard service also works by reading a sample valid standard confirmation message from a file. Its the "Blue Peter" approach, ie: "here is one I prepared earlier". In other words, the application and standard service don't actually go to great lengths to assemble and process the messages they send and receive, thats not really the point of the example - the point is to show the conversion in the middle.

The application sends an application deal in the appNode item of a MapMessage, and it looks like this :-

<?xml version="1.0"?>
<AppDeal>
  <trade>1234</trade>
  <stock>Gold</stock>
  <amount>1000</amount>
  <country>UK</country>
  <site>London</site>
</AppDeal>

The standard service expects a standard deal in the stdNode item of a MapMessage, and it looks like this :-

<?xml version="1.0"?>
<StdDeal>
  <stock>Au</stock>
  <amount>1000</amount>
  <where location="LONDON"/>
</StdDeal>

Immediately we can see that we have a lot of work to do :-

In the reverse direction, the standard service sends a standard confirmation, which is a BytesMessage (String payload written encoded using UTF-8), and has the trade number in a JMS message property called trade. The application expects a TextMessage, and the trade number is in the payload.

Data maps

Within the XML configuration (appstd.xml) you will notice there are two implementations of the mappings between (country,site) and location. One uses database lookups and the other uses local file data. You pick which version by setting a system property, to either -DCSLMAP=DB or -DCSLMAP=FILE, and if you don't set the property, the default is to use the database. If you specify some other value, you'll find the XML configuration fails validation. If you use the file implementation, you don't need the database set up, which makes it easier to run the sample. You do still need the JDBC .jar file on the classpath. If you use the database implementation, you'd need to use SQL*Plus to run the example/setupCslTable.sql script.

The CountrySite_to_Location_FILE data map uses <maplist/> to demonstrate looking in a number of places.

The mappings between element names and element symbols are implemented as web service queries.

Flows

To ensure that the application deal is mediated on the way out, the configuration contains :-

<queue name="stddealq" out="AppDeal_to_StdDeal"/>

The flow contains the following (some debug related items removed) :-

<flow name="AppDeal_to_StdDeal" eflow="make_poison_message">
  <bodyget item="appNode"/>
  <bodydel item="appNode"/>
  <vartype type="Document"/>
  <xmlvalidate schema="file:AppDeal.xsd"/>
  <varselect xpath="//trade" destvar="_JMSAdapter_message_id"/>
  <xmltransform xslt="file:AppDeal_to_StdDeal.xsl"/>
  <varmap xpath="//@location" map="CountrySite_to_Location_${CSLMAP:DB}"/>
  <xmlvalidate schema="file:StdDeal.xsd"/>
  <vartype type="String">
    <outputproperty name="indent" value="yes"/>
    <outputproperty name="{http://xml.apache.org/xslt}indent-amount" value="4"/>
  </vartype>
  <varmap regexp="&lt;stock&gt;([A-Za-z]+)&lt;/stock&gt;" group="1"
          map="ElementName_to_ElementSymbol"/>
  <bodyset item="stdNode"/>
  <propset var="_JMSAdapter_message_id" prop="trade"/>
  <varset value="Transformed by JMS Adapter"/>
  <propset prop="Comment"/>
</flow>

Talking this through :-

I'm not going to show the full confirmation flow, as it is similar, but in reverse. Here are the parts that show how the trade number in the JMS message property becomes a part of the payload (using the XSLT parameter mechanism) :-

<flow name="StdConfirmation_to_AppConfirmation" eflow="make_poison_message">
  <propget prop="trade" var="_JMSAdapter_message_id"/>
  <propdel prop="trade"/>
  ... snip
  <xmltransform xslt="file:StdConfirmation_to_AppConfirmation.xsl">
    <param name="trade" var="_JMSAdapter_message_id"/>
  </xmltransform>
  ... snip
</flow>

The other significant difference is the use of <bodytype type="TextMessage"/> to convert the message from BytesMessage to TextMessage.

Exceptions

The exception flow is basically the same as the example given in the Exception handling section of this document. In the diagram, the exception handling is indicated as red arrows leading to an unhappy face. Basically the strategy used by the application and standard service is to display a message on the standard output where the user will see it. It is only a demo.

Consider the AppDeal_to_StdDeal flow shown above, as there is scope to do better. Observe that we don't know the trade ID until after we've parsed and validated the message, and perhaps the problem is that it is not well-formed XML. Perhaps there is scope to insert the following immediately after the payload is got using <bodyget/> :-

  <varselect regexp="&lt;trade&gt;([0-9]+)&lt;/trade&gt;" group="1"
             destvar="_JMSAdapter_message_id"/>

If the XML isn't well-formed, we might capture something meaningful. Of course, might also accidentlally catch matching text within an XML comment. If the XML is well-formed, and validates, we still do the proper XPath later, so we'll be sure to be no worse off.

Configuration

The diagram shows a pictoral summary of the configuration files used, connected to JMS Adapter using grey arrows. In the example, all the configuration files are referred to using relative file: URLs. This is great for development and testing. When it comes to production deployment it would probably be better to collect all of them together (except appstd.properties) in to appstd.jar and use URLs like this :-

jar:appstd.jar!/AppDeal.xsd

If it were ever necessary to add some user defined flow steps or data maps, you'd include their .class files in there too.

Running the sample

Prerequisites :-

Run the application by running run_app.sh. This application is configured to use JMS Adapter, and it knows the queues by the JNDI names appdealq and appconfirmq. The objects it finds in the local filesystem JNDI implementation are JMS Adapter administered objects, that point to the underlying queues in the underlying Tibco EMS. This application sends a deal in an "application defined deal" format, and expects an "application defined confirmation" format back.

Revision history

VersionDateComments
0.4  First public release.
0.52010-05-01 Added support for namespace prefixes in XPath expressions.
0.62010-06-16 Added Windows .bat files.
0.72014-09-04 Added example/setupCslTable.sql.
New java version.
Added Xalan indent-amount property.
Added script to debug example.
0.82014-11-16 Protect against SQL injection.
future...Optimise/pool JAXP assets, such as parsers and transformers.
Optimise/pool JDBC connections, as used by <mapdb/>.
Implement a Flow Step, perhaps called <vareval/> to do expression evaluation.
Selector transformation.

Legal stuff

I wrote all this code in my own time on my own equipment. I used public non-confidential information to do so. I hereby place all this code into the public domain. Feel free to do whatever you like with it. No copyright / no royalties / no guarantees / no problem. Caveat Emptor! Anyone offering ideas/code must be happy with the above.

Summary

JMS Adapter represents a very lightweight way to mediate data as it is published or subscribed, without having to deploy seperate integration components.

With its relatively simple XML configuration, a large proportion of mediation scenarios can be relatively easily described. Extension mechanisms exist allowing demanding users to enhance/extend the framework to meet demanding requirements.

Download from http://www.nyangau.org/jmsadapter/jmsadapter.zip.


This page maintained by Andy Key
andy.z.key@googlemail.com