JMS Adapter

You are on vacation, your hairdryer expects 240V and has a UK 3-pin plug, but the socket is 110V with a US 2-pin plug. You need a lightweight travel adapter.

You have a JMS application which sends and receives messages in one format, but the application you want to talk to receives and sends messages in another format. You need a lightweight JMS Adapter.

Typical solutions

The point-to-point adapter pattern is typically used to solve this problem :-

Note: The word "adapter" is also sometimes also used to describe adapting between interfaces/transports (eg: JDBC to JMS), but here we are using the word to describe the adaptation of the data.

Sometimes enterprises implement a decoupled model with pub-adapters and sub-adapters, and a standard intermediate data representation :-

Adapters typically implement the VETO standard industry pattern, sometimes taken to mean :-

• Validate - validate message content
• Enrich - perhaps looking up additional data based on data in the message
• Transform - transform the format, structure or content of the message
• Operate - a generic catch all for other kinds of processing such as normalization, canonicalization, data masking, etc...

In the Java JMS and XML world, these typically resolve to :-

• Validation against DTD or XSD schema
• Enrichment using static mappings, and data found by consulting web services or by performing database queries
• Transformation using XSLT or marshalling XML to objects, mapping fields in Java, and marshalling to new XML
• Operation using home written Java code

Sometimes adapters implement VETRO, in which the R stands for Routing.

Products such as Apache Camel, OpenAdapter and Mule give you frameworks for implementing this. Enterprises sometimes build their own.

What is JMS Adapter

JMS Adapter is a JMS implementation which you configure to use an underlying JMS implementation (eg: ActiveMQ, Apache Qpid, WebSphere MQ, Tibco EMS). The application is then configured to use JMS Adapter. As a result, it can mediate any message as it is sent or received.

It has the following features :-

• Manipulation of message properties and all JMS body message types
• Conversions between different data types, such as byte[], String, org.w3c.dom.Document, Integer, ...etc., with appropriate control over format conversions. This includes parsing of XML text to DOM tree and back again, and control over character encoding where appropriate.
• XSD validation
• Enrichment, specifically mapping of data using data maps, which can be static mapping tables, database lookups, web service calls, or combinations of these, optionally with caching.
• XSLT transformation.
• User extension points to allow arbitrary user written operations (aka flow steps) and enrichments (aka data maps).
• A strategy for exception handling resulting from trying to process poisonous/bad messages.
• It handles request-reply scenarios correctly in most cases, even though temporary destinations may be in use.
• Configured with a simple XML configuration. In most cases, the developer needn't write any Java.
• Configuration can use a set of properties, which you might treat like a deployment descriptor. eg: XML configuration has developer produced configuration, properties hold choices of JDBC URL etc..
• All configuration assets are located by URL, allowing storing on the filesystem, fetching from a central configuration web server, bundling in .jar files, etc..
• Simple deployment. No dependencies on any other 3rd party .jar files, thus reducing the probability of version clashes. Obviously, if you want to enrich by talking to a database, you'll need the vendors JDBC implementation on the classpath.
• No seperate adapter processes required, as the processing happens as a part of the JMS client process.
• Works on Java 6 or later. The data map which calls a web service makes use of the JAX-WS built into Java 6. It may be possible to use Java 5, if you can find a standalone JAX-WS implementation (and things it depends upon, such as JAXB), and add these to the classpath.

Things it doesn't do include :-

• DTD validation - Often people prefer XSD anyway (or are willing/able to use tooling to convert DTDs to XSDs in advance), or maybe Schematron generated XSLT based.
• Transformation using object mapping. Nothing stopping anyone writing a plugin that uses JAXB etc... themselves.
• Split, Join, Routing - as it is a thin JMS shim, if the client does a JMS send or publish, JMS Adapter must be sure to do a send or publish to the underlying JMS in exactly the same way. When considering the JMS Adapter layer, one message in implies one message out.
• Resequencing/Reordering - JMS Adapter changes nothing about the ordering of messages.
• Anything vendor specific - If you're application uses vendor specific features, such as the Tibco EMS "commit an individual message" feature, then JMS Adapter is not for you. JMS Adapter presents the standard JMS 1.1 API, and no more. You need to be able to configure the InitialContextFactory (class and) URL used to find administered objects in JNDI, so that the application finds and uses the JMS Adapter JMS implementation. The application should not code to vendor specific classes. The BeJUG JMS Rules give a good set of best practices.
• Selector Transformation - In principle there needs to be a way of transforming selectors, but this is not implemented at present. @@@ TBD.

The two patterns shown earlier now can be revised. In the point to point case, we could have either :-

or

And in the pub-sub case :-

Its not unusual for enterprises to have hundreds of adapters. A large proportion of these (maybe half) might be expected to be simple enough that they can be implemented in JMS Adapter. The others might need a more complex solution, or perhaps use JMS Adapter for part of their implementation, and then draw upon external components which just do the missing parts (such as sequencing and routing). The resulting reduction in complexity of development, number of JMS destinations, number of running processes to manage and hardware needed to host adapters could be very significant.

Configuring JMS Adapter into the application

Application classpath

Add nyangau-jmsadapter.jar to the application classpath. JMS Adapter itself has no dependencies on any other 3rd party .jar files.

However, if you will be using the <mapdb/> data mapping feature to query a database to enrich your data, you'll need to ensure the vendors JDBC implementation is on the classpath also.

If you've written any extensions to JMS Adapter that use any other 3rd party libraries, you'll need to ensure they are on the classpath too.

JNDI Setup

A normal application will use the following sequence of events :-

1. Application reads its configuration to determine the
   • Context.INITIAL_CONTEXT_FACTORY
   • Context.PROVIDER_URL
   • ... other JNDI provider specific properties needed when constructing the InitialContextFactory, such as SSL parameters
   • JNDI names of connection factories and destinations.
2. Application uses the InitialContextFactory to connect to JNDI to obtain the underlying JMS implementations administered objects, which implement things like QueueConnectionFactory and Queue interfaces, and contain information like service hostnames, ports, security information, connection and retry parameters etc..
3. Application uses returned objects to invoke the JMS provider.

We want to interpose JMS Adapter. If we are able to change the JNDI names in the applications configuration, and we are able to populate additional objects into JNDI, then we want :-

Or, if we can't change the JNDI names in the applications configuration, or we are not able to populate additional objects into JNDI (as would be the case with the tibjmsnaming JNDI subset provided as a part of Tibco EMS), we want :-

The following sequence of events now occurs :-

1. Application reads its configuration, but this time it gets back different values
2. Application connects to (a possibly different) JNDI and looks up JMS Adapter administered objects (using possibly different JNDI names).
3. Application uses the returned objects to invoke JMS Adapter.
4. JMS Adapter administered objects contain
   • Context.INITIAL_CONTEXT_FACTORY of the (possibly different) underlying JNDI
   • Context.PROVIDER_URL of the (possibly different) underlying JNDI
   • ... other underlying JNDI provider specific properties needed when constructing the InitialContextFactory
   • (possibly different) JNDI names of connection factories and destinations.
   • In the case of a connection factory, the administered object also contains Jmsa.UNDERYLING_CONNECTION_FACTORY, Jmsa.CONFIG and possibly also a Jmsa.PROPERTIES property.
   • In the case of a destination, the administered object also contains Jmsa.UNDERYLING_DESTINATION.
   JMS Adapter uses this information to fetch the underlying JMS implementations administered objects.
5. JMS Adapter uses the returned underlying objects to invoke the underlying JMS provider. Of course, it also mediates messages on the way through.

Setting up JNDI for JMS Adapter therefore requires populating one JNDI with details of how to reach (possibly a different) JNDI, and the details of the entries in there to use. JNDI providers typically have quite a bit of variation in the parameters needed, especially once you start to consider security and SSL related parameters. Therefore it is hard to write one tool which is able to do this, so instead we show some sample code.

In the sample, the client application would normally use application message formats in application specific queues, and some adapter would convert between this and a standard services message formats in standard service specific queues. Here, we configure the application to use JMS Adapter to translate application messages to standard messages, and put them in standard queues, and vice versa :-

//
// SetupJNDI.java - Register things in JNDI to support App.java sample code
//

import java.util.*;
import javax.jms.*;
import javax.naming.*;
// import com.tibco.tibjms.naming.*;
import nyangau.jmsadapter.*;

public class SetupJNDI
  {
  public static void main(String[] args)
    {
    try
      {
      // The JNDI we're going to store Jmsa AOs into
      InitialContext ic = new InitialContext(
        new Hashtable<String,Object>()
          {{
          // We're going to use the cheap and cheerful filesystem based JNDI
          put(Context.INITIAL_CONTEXT_FACTORY, "com.sun.jndi.fscontext.RefFSContextFactory");
          put(Context.PROVIDER_URL, "file:fs");
/*
          // Other vendor unique parameters used to talk to JNDI,
          // such as LDAP specific or SSL specific ones, would be added here
          put(...);
*/        
          }}
        );

      // The properties used to reach the underlying JNDI implementation
      Hashtable<String,Object> ht = new Hashtable<String,Object>()
        {{
        // We're hosting queues on a local Tibco EMS and using its JNDI
        put(Context.INITIAL_CONTEXT_FACTORY, "com.tibco.tibjms.naming.TibjmsInitialContextFactory");
        put(Context.PROVIDER_URL, "tcp://localhost:7222");
/*
        // Other vendor unique parameters used to talk to JNDI,
        // such as SSL ones, would be added here
        put(Context.SECURITY_PRINCIPAL, "admin");
        put(Context.SECURITY_CREDENTIALS, "password");
        put(TibjmsContext.SSL_ENABLE_VERIFY_HOST, new Boolean(false));
        put(TibjmsContext.SSL_ENABLE_VERIFY_HOST_NAME, new Boolean(false));
        put(TibjmsContext.SSL_TRUSTED_CERTIFICATES,
          new Vector<String>()
            {{
            add("root1.cert.pem");
            add("root2.cert.pem");
            }}
          );
*/
        }};

      // We're going to need a QCF
      ic.rebind(
        "qcf",
        new JmsaQueueConnectionFactory(
          ht,
          new Hashtable<String,Object>()
            {{
            put(Jmsa.UNDERLYING_CONNECTION_FACTORY, "QueueConnectionFactory");
            put(Jmsa.CONFIG, "file:appstd.xml");
            put(Jmsa.PROPERTIES, "file:appstd.properties");
            }}
          )
        );

      // Lets create a Q for deals
      ic.rebind(
        "appdealq",
        new JmsaQueue(
          ht,
          new Hashtable<String,Object>()
            {{
            put(Jmsa.UNDERLYING_DESTINATION, "stddealq");
            }}
          )
        );

      // Lets create a Q for confirmations
      ic.rebind(
        "appconfirmq",
        new JmsaQueue(
          ht,
          new Hashtable<String,Object>()
            {{
            put(Jmsa.UNDERLYING_DESTINATION, "stdconfirmq");
            }}
          )
        );

      }
    catch ( NamingException e )
      {
      e.printStackTrace();
      }
    }
  }

After running this :-

Application can refer to	and on its behalf, JMS Adapter will access
com.sun.jndi.fscontext.RefFSContextFactory file:fs	com.tibco.tibjms.naming.TibjmsInitialContextFactory tcp://localhost:7222
JNDI name qcf	JNDI name QueueConnectionFactory
JNDI name appdealq	JNDI name stddealq
JNDI name appconfirmq	JNDI name stdconfirmq

Note in particular that the constructors for the JmsaQueueConnectionFactory and JmsaQueue take two Map<String,Object>. The entries from both are combined. They are seperate to make it easier to write programs like the one above. You can prepare one Map describing how to reach the underlying JNDI (called ht in the example), and use this repeatedly, whilst having other Maps to describe each underyling connection factory or destination.

Note Hashtable<String,Object> implements Map<String,Object>.

The XML configuration

Here is a configuration that does nothing :-

<?xml version="1.0"?>
<jmsadapter>

  <!-- JMS Adapter configuration for adapter ${ja_name} -->

  <!-- Various data map elements go here.
       The element names all start with map, such as mapinline.
       A data map is something which maps Strings to Strings,
       and various mechanisms to do this are provided. -->

  <!-- Various flow elements go here.
       Each flow contains a sequence of flow steps,
       each of which is an action to perform on the message or variables. -->

  <!-- Various queue and topic elements go here.
       Each identifies a destination by underlying JMS provider destination name,
       and associates it with flow(s) -->

</jmsadapter>

In the JNDI setup, the location of the configuration is specified using the Jmsa.CONFIG property. Note that it is a URL, so the configuration can come from a file, from a web server, from within a .jar file, etc.. In a Production setup, its quite likely that the developer will have delivered their JMS Adapter configuration as a set of files inside a .jar file, because the XML configuration is likely to refer to other files, such as XSD schema, XSLT transformations, data mapping properties etc..).

Every JMS Adapter connection factory must have a configuration. Also, note that each JMS Adapter connection factory can have its own seperate JMS Adapter configuration file.

After reading the configuration into memory, but before parsing it into a DOM structure, every ${variable} sequence is expanded. Even ${variable} sequences within XML comments are expanded. Each variable is first considered as a property, and failing that as a system property, next the default is used (if specified), and failing that JMS Adapter throws an exception. The syntax ${variable:default} is used to define the default value.

The full detail of what goes in the XML configuration is covered in the following sections.

The properties

Here is a simple set of properties :-

ja_name=My First JMS Adapter

In the JNDI setup, the location of this file is specified using the Jmsa.PROPERTIES property. Note that it is a URL, so the properties can come from a file, from a web server, from within a .jar file, etc.. In a Production setup, its quite likely that the properties would be a file or fetched from a central web server.

Properties are optional, but as above, if the XML references a ${variable} and this does not exist in the properties, or as a system property, and a default is not given, JMS Adapter will throw an exception.

The intent is that developers will prepare the XML file, using ${variable} placeholders for JDBC URLs, Web Service URLs, queue names, etc.., and the deployer will use a properties file to map these to the specific set of resources being used in UAT, Production environments.

Queues and Topics to be mediated

You identify which queues and topics will be mediated using <queue/> and <topic/> elements in the XML configuration.

To ensure that when the application sends to the stddealq, the message is processed by the AppDeal_to_StdDeal flow, you'd write :-

<queue name="stddealq" out="AppDeal_to_StdDeal"/>

To ensure that when the application receives a message from the stdconfirmq, the message is processed by the StdConfirmation_to_AppConfirmation flow, you'd write :-

<queue name="stdconfirmq" in="StdConfirmation_to_AppConfirmation"/>

The destination names are the names in the underlying JMS implementation. They are not JNDI names.

Temporary destinations

The <queue/> elements shown so far explicitly name the queue. However, when implementing a request-reply interaction pattern, the client can create a temporary queue for the reply to be sent to. The outbound request may need converting, and the reply may need converting back too.

If we have JMS Adapter configured into the client, then this can be acheived like this :-

<queue name="requestq"
       out="ClientRequest_to_ServerRequest"
       outreply="ServerResponse_to_ClientResponse"/>

Under the covers, when sending the request, JMS Adapter spots the use of a temporary destination in the JMSReplyTo, and creates a temporary association :-

<queue name="__tempq12345"
       in="ServerResponse_to_ClientResponse"/>

This only lives for as long as the temporary queue.

Its also possible that JMS Adapter might be being used in the server, perhaps because the server is new and needs to understand an old style of request. The server might use :-

<queue name="requestq"
       in="OldRequest_to_NewRequest"
       inreply="NewResponse_to_OldResponse"/>

Under the covers, when JMS Adapter receives a message from the requestq, it creates a temporary association :-

<queue name="__replyq12345"
       out="NewResponse_to_OldResponse"/>

However, there is a problem: JMS Adapter cannot know whether the reply queue is temporary or not, and cannot know when it gets deleted. So it cannot know when to forget this temporary association. Therefore, over time, this could represent a memory leak. JMS Adapter only remembers the 1000 most recent of these, and when the 1001st is added, forgets the 1st. This is obviously not a 100% reliable way of handling the problem.

As a result, we say that request-reply is best mediated at source.

Note: Everything described here for queues works equally for topics.

Flows

A flow is a sequence of flow steps, which are executed one by one. Each performs some processing action, such as manipulating part of the message, validating, enriching, transforming, etc..

A flow context is passed along the flow, and each step updates part of it.

Any flow with a name may be referred to by a <queue/> or <topic/> element. Flows should typically be named to reflect what they do. So, TextMessageMxML_to_MapMessageFpML might be a good name for a flow that converted a Murex TextMessage to a FpML MapMessage.

An simpler example flow might look like this :-

<flow name="TextMessage_to_MapMessage">
  <bodyget/>
  <bodytype type="MapMessage"/>
  <bodyset item="textNode"/>
</flow>

Flows have a special eflow="exceptionflowname" attribute which is described in the exception handling section.

Flow Context

The flow context contains :-

• The JMS Message
• A set of named variables

The JMS Message is actually an instance of the JMS Adapter JmsaMessage class, or derivation thereof. As a result, it is possible to make changes to the message structure and content that would be impossible when operating on vendor supplied implementations of the JMS Message interface. Specifically :-

• the ability to change the body type of the message (eg: from TextMessage to MapMessage)
• the ability to modify properties and body contents, even when technically they are supposed to be read-only
• the ability to clear an individual JMS property of a Message
• the ability to clear an individual item in a MapMessage

Each variable can be of any Java Object :-

• a special Null value (implemented internally as an instance of FlowNullValue class, not as null)
• String
• Document (ie: org.w3c.dom.Document)
• Boolean
• Byte
• Character
• Short
• Integer
• Long
• Float
• Double
• byte[]
• Object (ie: any Java object)

At the beginning of processing, the flow context contains the message, and no variables are defined. After processing, the message is passed on, and the flow context discarded.

Flow steps

Flow steps are the commands that make up a flow.

All steps have a name="stepname" attribute, which is optional. If specified, then any diagnostics that are given include the step name you specify, otherwise a machine generated identifier is displayed. In the tables below the name attribute is not shown.

Any step that has a var="varname" attribute will default to v0 if not specified.

Any step that has a destvar="varname" attribute will default to the same as the var="varname" attribute if not specified.

Flow steps - Message property access

<propget
    prop="propname"
    var="varname"/>

<propset
    var="varname"
    prop="propname"/>

<propdel
    prop="propname"/>

Flow Steps - Message body access

<bodyget
    reset="true|false"
    type="UTF|
          Boolean|
          Character|
          Byte|
          Short|
          Integer|
          Long|
          Float|
          Double|
          byte[]"
    item="itemname"
    var="varname"/>

<bodyset
    var="varname"
    clear="true|false"
    item="itemname"/>

<bodydel
    item="itemname"/>

<bodytype
    type="BytesMessage|
          TextMessage|
          MapMessage|
          StreamMessage|
          ObjectMessage|
          Message"/>

Flow steps - Variable manipulation

<varset
    var="varname"
    value="somevalue"/>

<vardef
    var="varname"
    value="somevalue"/>

<vardel
    var="varname"/>

<vartype
    var="varname"
    destvar="destvarname"
    type="String|
          Document|
          Boolean|
          Character|
          Byte|
          Short|
          Integer|
          Long|
          Float|
          Double|
          byte[]|
          Object"
    encoding="UTF-8|
              UTF-16|
              ISO8859-1|
              XML|
              ... other"
    coalescing="true|false"
    expandentityreferences
      ="true|false"
    ignoringcomments
      ="true|false">
  <outputproperty
      name="propname"
      value="propvalue"/>
  ... more outputproperties
</vartype>

<varmap
    var="varname"
    destvar="destvarname"
    xpath="xpathexp"
    nsctx="ns1=uri1 ns2=uri2 ..."
    regexp="javaregexp"
    group="capturinggroup"
    map="datamapname"/>

<varselect
    var="varname"
    destvar="destvarname"
    xpath="xpathexp"
    nsctx="ns1=uri1 ns2=uri2 ..."
    regexp="javaregexp"
    group="capturinggroup"/>

The <vartype/> mechanism attempts to "do the right thing" and give maximum flexibility in conversion between object types :-

Convert		From
		String	Document	Boolean	Byte	Character	Short	Integer	Long	Float	Double	byte[]	Object	Exception
To	String	y	y,d2	y,2s	y,2s	y,2s	y,2s	y,2s	y,2s	y,2s	y,2s	y,enc	y,2s	y,stack
	Document	y,2d	y									y,enc,2d
	Boolean	y		y	y	y	y	y	y	y	y
	Byte	y		y	y	y	y	y	y	y	y
	Character	y		y	y	y	y	y	y	y	y
	Short	y		y	y	y	y	y	y	y	y
	Integer	y		y	y	y	y	y	y	y	y
	Long	y		y	y	y	y	y	y	y	y
	Float	y		y	y	y	y	y	y	y	y
	Double	y		y	y	y	y	y	y	y	y
	byte[]	y,enc	y,d2,enc									y	y,ser
	Object											y,deser

Key :-

• y = yes
• 2s = uses .toString()
• d2 = Document serialisation process uses <outputproperty/> elements
• 2d = Document parsing process uses coalescing attribute (default true), expandentityreferences attribute (default true) and ignoringcomments attribute (default false)
• enc = uses encoding attribute (default UTF-8)
• ser = if object is Serializable then serializes to byte[]
• deser = attempts to deserialize byte[] to object
• stack = uses .printStackTrace()

<xmltransform/> (which is a step described later in this document) does XSLT from DOM source to DOM destination, and as a result it seems that <xsl:output indent="yes"/> doesn't work. Therefore, so if you want nicely formatted XML, you typically find yourself handling the indentation when the DOM is serialised to textual form, using <vartype type="String"/> and a nested <outputproperty name="indent" value="yes"/> instead. Update: I also find it necessary to use a <outputproperty name="{http://xml.apache.org/xslt}indent-amount" value="4"/> or similar setting.

Note that the encoding="XML" has a special meaning. It is intended to cope with the way in which XML is encoded, with or without BOMs, encoded using UTF-8 by default, or using another encoding :-

• When converting from byte[] to String, look for the 0xef 0xbb 0xbf BOM marker and deduce UTF-8, look for the 0xfe 0xff BOM marker and deduce UTF-16BE, look for the 0xff 0xfe BOM marker and deduce UTF-16LE, else if the first few bytes match <?xml scan the first few bytes looking for the encoding attribute, and if found use that, else fall back to UTF-8.
• When converting from String to byte[], if the first few characters match <?xml scan the first few characters looking for the encoding attribute, and if found use that, else fall back to UTF-8. If UTF-16 is used, then the output is the 0xfe 0xff BOM followed by UTF-16BE data.

The XML character set encoding is handled in this way so as to seperate it from the process of parsing. This makes it possible to do certain kinds of processing of XML data without having to parse it (which could be quicker), and it also makes it possible to extract certain parts of an XML document, even if it is incomplete, corrupt, or otherwise doesn't parse cleanly.

Flow steps - XML specific manipulations

<xmlvalidate
    var="varname"
    schematype="schemaType"
    schema="urlOfXSD"/>

<xmltransform
    var="varname"
    destvar="destvarname"
    xslt="urlOfXSLT">
  <param
      name="paramname"
      var="paramvarname"/>
  ... more params
</xmltransform>

Flow steps - Control flow

<call
    flow="nestedflowname"/>

<switch
    var="varname">
  <case
      regexp="pattern"
      flow="nestedflowname"/>
  ... more cases
</switch>

<throw
    cause="Too complicated"/>

<debug
    logfile="filename"
    logmessage="true|false"
    logvars="true|false"
    vars="v1,v2,v3"/>
    

As an example, if you had the following flow :-

<flow name="AppDeal_to_StdDeal">
  <bodyget item="appNode"/>
  <vartype type="Document"/>
  <debug name="after-parsing" logvars="true" logmessage="true"/>
  ... the rest omitted
</flow>

Then the following could be output :-

=== FlowStep after-parsing
Message propertes:
Message body (MapMessage):
  item "appNode" = String "<?xml version="1.0"?>
<AppDeal>
  <trade>1234</trade>
  <stock>Gold</stock>
  <amount>1000</amount>
  <country>UK</country>
  <site>London</site>
</AppDeal>
"
Variables:
  var "v0" = Document
        <?xml...
        <AppDeal>
        - "
  "
        - <trade>
        - - "1234"
        - </trade>
        - "
  "
        - <stock>
        - - "Gold"
        - </stock>
        - "
  "
        - <amount>
        - - "1000"
        - </amount>
        - "
  "
        - <country>
        - - "UK"
        - </country>
        - "
  "
        - <site>
        - - "London"
        - </site>
        - "
"
        </AppDeal>

The textual rendition of the message and variables is not intended to be suitable for cut-n-pasting into a file and processing further. Its intended to facilitate problem diagnosis. For this reason, the string representation is known as the "diagnostic form". This is why types are shown, and strings "quoted".

Flow Steps - User defined

I'm not likely to be able to second guess every conceivable thing a customer might like to do with the JMS Message or variables. So there is an extension mechanism.

Element	Action
<stepuser class="className" ... other attributes > ... other nested elements </stepuser>	Allow the user to add their own step which does whatever they need. className is loaded, and its constructor called with the DOM org.w3c.dom.Node corresponding to the <stepuser/>, and the implementation then reads its configuration. At runtime, the process method is called passing the FlowContext, and this method can then manipulate the message and variables as it sees fit.

A simple user defined step :-

//
// FlowStepCompare.java - Check two variables are the same
//

package mypackage;

import org.w3c.dom.Element;
import org.w3c.dom.NodeList;
import org.w3c.dom.NamedNodeMap;
import org.w3c.dom.Node;
import org.w3c.dom.Text;
import javax.jms.JMSException;
import nyangau.jmsadapter.*;

public class FlowStepCompare extends FlowStep
  {
  protected String var1;
  protected String var2;
  protected String destvar;
  public FlowStepCompare(Node eStep)
    throws FlowProcessorException
    {
    super(eStep);
    NamedNodeMap nnmStep = eStep.getAttributes();
    Node aVar1 = nnmStep.getNamedItem("var1");
    var1 = ( aVar1 != null ) ? aVar1.getNodeValue() : "v0";
    Node aVar2 = nnmStep.getNamedItem("var2");
    var2 = ( aVar2 != null ) ? aVar2.getNodeValue() : "v0";
    Node aDestVar = nnmStep.getNamedItem("destvar");
    if ( aDestVar == null )
      throw new FlowProcessorException("missing destvar attribute in step "+name);
    destvar = aDestVar.getNodeValue();
    }
  public void process(FlowContext c)
    throws FlowProcessorException, JMSException
    {
    Object o1 = c.getVar(var1);
    if ( o1 == null )
      throw new FlowProcessorException(var1+" is undefined in step "+name);
    if ( o1 instanceof FlowNullValue )
      throw new FlowProcessorException(var1+" is Null in step "+name);
    Object o2 = c.getVar(var2);
    if ( o2 == null )
      throw new FlowProcessorException(var2+" is undefined in step "+name);
    if ( o2 instanceof FlowNullValue )
      throw new FlowProcessorException(var2+" is Null in step "+name);
    if ( o1.toString().equals( o2.toString() ) )
      c.setVar(destvar, "same");
    else
      c.setVar(destvar, "different");
    }
  }

The jmsadapter.xsd used to validate JMS Adapter XML configurations is designed to use <xsd:any/> and <xsd:anyAttribute/> so as to allow you to put your own elements and attributes in the XML configuration file. In this case, to use the step, you'd use :-

<stepuser classname="mypackage.FlowStepCompare"
          var1="v1" var2="v2" destvar="comparisonResult"/>

Remember to put your .class or .jar file on the classpath.

Data maps

Data maps are things which map input String values to output String values.

In one message format, the US Dollar might appear as $ and in another message format, it might be USD. Data maps provide a mechanism for mapping from one form to the other.

All data maps can have a name="datamapname" attribute. In the examples below, the name attribute is shown where it makes sense, ie: for top level data maps, not for nested ones. Any named top level data map can referenced by a <varmap/> step.

Data maps - To a default value

The following is a data map which maps every unmapped input value to the output value UNKNOWN.

<mapdefault name="to_Unknown" dest="UNKNOWN"/>

Doesn't seem very useful on its own, but becomes more useful when used at the end of a <maplist/>, see below.

Data maps - Using inline data

The following is a data map which performs a set of fixed mappings, which are explicitly stated in the XML configuration :-

<mapinline name="CurrencySymbol_to_CurrencyCode">
  <maplet src="$" dest="USD"/>
  <maplet src="&#163;" dest="GBP"/>
</mapinline>

Data maps - Using properties data

Here is a data map which reads properties from a URL whenever its asked to perform a mapping :-

<mapprops name="CurrencySymbol_to_CurrencyCode"
          properties="file:CurrencySymbol_to_CurrencyCode.properties"/>

This variant reads properties in XML format :-

<mapprops name="CurrencySymbol_to_CurrencyCode"
          xml="file:CurrencySymbol_to_CurrencyCode.xml"/>

Both properties and xml attributes can be given, in which case it reads the properties followed by XML properties.

Other URLs (such as those that use http: and jar: protocols) can also be used.

Reading an entire file (or URL) every time you want to satisfy a mapping request only makes sense if the data is constantly changing. To avoid this, you can use <mapcache/>, described later. Having just read the entire file (or URL), this data map returns all the mappings from the file, giving <mapcache/> the opportunity to cache it all.

Data maps - Using a database query

Sometimes mapping data exists inside database tables.

Imagine ELEMENT_TABLE contains :-

NAME	SYMBOL
Gold	Au
Silver	Ag
Lead	Pb

The following data map can be used :-

<mapdb name="ElementName_to_ElementSymbol"
       driver="oracle.jdbc.OracleDriver"
       url="jdbc:oracle:thin@127.0.0.1@1521:XE"
       userid="scott" password="tiger"
       scolumns="NAME" dcolumn="SYMBOL" table="ELEMENT_TABLE"/>

If asked to map Gold and Silver, the data map creates a query of the form :-

SELECT "NAME","SYMBOL" FROM ELEMENT_TABLE WHERE
  ( "NAME"='Gold'  ) OR
  ( "NAME"='Silver')

The database replies with matching rows, and the data map uses them.

A more complex kind of mapping may be appropriate, in which the items to be mapped actually span multiple columns. Consider the CSL_TABLE table :-

country	site	location
UK	London	LONDON
US	Paris	PARIS_TEXAS
FR	Paris	PARIS_FRANCE

This could have been set up using the following SQL, which is a part of the example shipped with JMS Adapter, and is helpfully included in the example/setupCslTable.sql file :-

-- Set up the CSL_TABLE used by the <mapdb> mapping

CREATE TABLE CSL_TABLE
	(
	"country"  varchar(10),
	"site"     varchar(10),
	"location" varchar(20)
	) ;

INSERT INTO CSL_TABLE VALUES ( 'UK', 'London', 'LONDON'       ) ;
INSERT INTO CSL_TABLE VALUES ( 'US', 'Paris' , 'PARIS_TEXAS'  ) ;
INSERT INTO CSL_TABLE VALUES ( 'FR', 'Paris' , 'PARIS_FRANCE' ) ;

SELECT * FROM CSL_TABLE ;

-- End

The following data map can be used :-

<mapdb name="CountrySite_to_Location"
       driver="oracle.jdbc.OracleDriver"
       url="jdbc:oracle:thin@127.0.0.1@1521:XE"
       userid="scott" password="tiger"
       split="," scolumns="country,site" dcolumn="location" table="CSL_TABLE"/>

Note the use of split="," to identify how to split the value to be mapped into peices, and the also note that the scolumns attribute identifies a number of columns. scolumns is always seperated by , characters, regardless of the split attribute (which applies to the data to be mapped only).

If asked to map UK,London and US,Paris, the data map creates a query of the form :-

SELECT "country","site","location" FROM CSL_TABLE WHERE
  ( "country"='UK' AND "site"='London' ) OR
  ( "country"='US' AND "site"='Paris'  )

Databases may temporarily be unreachable or unavailable. The previous example may be written with the default behaviour shown explicitly. ie: No retry occurs, and if we can't reach the database an exception is thrown :-

<mapdb name="CountrySite_to_Location"
       driver="oracle.jdbc.OracleDriver"
       url="jdbc:oracle:thin@127.0.0.1@1521:XE"
       userid="scott" password="tiger"
       split="," scolumns="country,site" dcolumn="location" table="CSL_TABLE"
       retrycount="0" retrydelay="0" retryfail="throw"/>

Here is a different strategy, in which 10 retries are performed (11 tries in total) with a 100ms delay between each, and if we still can't reach the database, execution proceeds without throwing an exception, presumably in the hope that a later mapping in a <maplist/> will be able to fill in the missing gaps :-

<mapdb name="CountrySite_to_Location"
       driver="oracle.jdbc.OracleDriver"
       url="jdbc:oracle:thin@127.0.0.1@1521:XE"
       userid="scott" password="tiger"
       split="," scolumns="country,site" dcolumn="location" table="CSL_TABLE"
       retrycount="10" retrydelay="100" retryfail="proceed"/>

Data maps - Using a web service

WSDL for a JMS Adapter Mapper web service has been defined. Such a web service accepts a mapping name and a list of source values to map, and returns the list with an even number of elements, comprised of source1, destination1, source2, destination2, ... Only successfully mapped source values have corresponding source, destination pairs in the output.

If the web service wants to indicate an error condition, it returns a list comprised of one element only - the error text. An obvious error condition is if the mapping name isn't understood. In this case, an exception is thrown, including the error text.

Here is an example of calling such a service :-

<mapws name="ElementName_to_ElementSymbol"
       wsdl="http://localhost:8888/ElementMapper?WSDL"
       mapping="Name_to_Symbol"/>

If the mapping attribute is omitted, it defaults to the empty string. The intent of the mapping name is to allow the web service to be able to perform a variety of mappings, and the client (in this case JMS Adapter) select among them.

The namespace used in the WSDL file can also be specified using the namespace attribute, although the default of http://www.nyangau.org/jmsadapter/mapper is normally fine. The service name in the WSDL file can also be specified using the service attribute, although the default of MapperService is normally fine. The port name in the WSDL file can also be specified using the port attribute, although the default of MapperPort is normally fine.

Web services may temporarily be unreachable or unavailable. The previous example may be written with the default behaviour shown explicitly. ie: No retry occurs, and if we can't reach the web service an exception is thrown :-

<mapws name="ElementName_to_ElementSymbol"
       wsdl="http://localhost:8888/ElementMapper?WSDL"
       mapping="Name_to_Symbol"
       retrycount="0" retrydelay="0" retryfail="throw"/>

Here is a different strategy, in which 10 retries are performed (11 tries in total) with a 100ms delay between each, and if we still can't reach the web service, execution proceeds without throwing an exception, presumably in the hope that a later mapping in a <maplist/> will be able to fill in the missing gaps :-

<mapws name="ElementName_to_ElementSymbol"
       wsdl="http://localhost:8888/ElementMapper?WSDL"
       mapping="Name_to_Symbol"
       retrycount="10" retrydelay="100" retryfail="proceed"/>

Implementing the web service using JAX-WS

You can use JAX-WS to build a Mapper web service. A simple implementation built using JAX-WS looks like this :-

//
// Mapper.java - Mapper Web Service
//
// This implementation exists as a template to generate WSDL from,
// and model to create others from.
// It maps between element names and symbols.
// It uses an ArrayOfString class so as to be sure the wrapped document literal
// SOAP has an element enclosing the elements of the lists of strings.
//

package nyangau.jmsadapter.mapper;

import java.util.Set;
import java.util.Map;
import java.util.HashMap;
import java.util.List;
import java.util.ArrayList;
import javax.jws.WebService;
import javax.jws.WebMethod;
import javax.jws.WebParam;
import javax.jws.WebResult;
import javax.xml.ws.Endpoint;

@WebService(
  targetNamespace="http://www.nyangau.org/jmsadapter/mapper",
  serviceName="MapperService"
  )
public class Mapper
  {
  protected Map<String,String> mNameToSymbol = new HashMap<String,String>()
    {{
    put("Gold"  , "Au");
    put("Silver", "Ag");
    }};
  protected Map<String,String> mSymbolToName = new HashMap<String,String>()
    {{
    put("Au", "Gold"  );
    put("Ag", "Silver");
    }};
  @WebMethod()
  public @WebResult(targetNamespace="http://www.nyangau.org/jmsadapter/mapper", name="mapResult") ArrayOfString map(
    @WebParam(targetNamespace="http://www.nyangau.org/jmsadapter/mapper", name="mapName") String mapName,
    @WebParam(targetNamespace="http://www.nyangau.org/jmsadapter/mapper", name="unmapped") ArrayOfString unmapped
    )
    {
    List<String> mapped = new ArrayList<String>();
    Map<String,String> m;
    if ( mapName.equals("Name_to_Symbol") )
      m = mNameToSymbol;
    else if ( mapName.equals("Symbol_to_Name") )
      m = mSymbolToName;
    else
      // Failures are indicated by returning a list of length 1
      {
      mapped.add("bad mapName: "+mapName);
      return new ArrayOfString( mapped );
      }
    for ( String u : unmapped.getList() )
      {
      String s = m.get(u);
      if ( s != null )
        {
        mapped.add(u);
        mapped.add(s);
        }
      }
    return new ArrayOfString( mapped );
    }
  public static void main(String[] args)
    {
    Endpoint.publish("http://localhost:8888/ElementMapper", new Mapper());
    }
  }

It uses a helper class to ensure that in the wrapped document literal SOAP messages, the lists of strings are enclosed in a parent element. This is not needed when working purely in Java with JAX-WS, but becomes important when trying to interoperate between Java and .NET.

//
// ArrayOfString.java - Ensure lists of strings have an enclosing element
//

package nyangau.jmsadapter.mapper;

import java.util.List;
import java.util.ArrayList;
import javax.xml.bind.annotation.XmlElement;

public class ArrayOfString
  {
  @XmlElement(namespace="http://www.nyangau.org/jmsadapter/mapper", name="string")
  protected List<String> list;
  public ArrayOfString() { list = new ArrayList<String>(); }
  public ArrayOfString(List<String> l) { list = l; }
  public List<String> getList() { return list; }
  }

Note the way in which I force the namespace to my chosen value everywhere I can. This appears to be necessary to get consistent WSDL from both the wsgen command and from the URL ending in ?WSDL served by the running web service, and also makes it easier when trying to interoperate with .NET.

The sample can be run on the command line (for testing purposes), and whilst running, its WSDL is available at http://localhost:8888/ElementMapper?WSDL.

Important: JMS Adapter uses web service client classes which are wsimported from the WSDL which was wsgenerated from the above sample code.

Implementing the web service using .NET using ASMX

A simple .NET ASMX based implementation should look like this :-

<%@ WebService Language="C#" class="Nyangau.JmsAdapter.Mapper.MapperService" %>

using System;
using System.Collections.Generics;
using System.Web.Services;

namespace Nyangau.JmsAdapter.Mapper
  {
  [WebService (Namespace="http://www.nyangau.org/jmsadapter/mapper")]
  public class MapperService : WebService
    {
    [WebMethod]
    public List<string> map(string mapName, List<string> unmapped)
      {
      List<string> l = new List<string>();
      foreach ( String s in unmapped )
        if ( s.Equals("Gold") )
          {
          l.Add(s);
          l.Add("Au");
          }
      return l;
      }
    }
  }

Ok, so this doesn't do a very good job of mapping - the point here is to illustrate all the scaffolding needed around the task at hand.

This would be put into a file ElementMapper.asmx and placed into a suitable IIS website or virtual directory.

Its WSDL can be fetched by appending ?WSDL to the end of the URL, eg: http://somehost/somepath/ElementMapper.asmx?WSDL. You do need to use this WSDL, as not only does it have the correct service URL, it also has a SOAPAction in it which .NET web services need (and is not present in JAX-WS WSDL).

It is also necessary to specify port="MapperServiceSoap" in the <mapws/> element, as the .NET ASMX WSDL names its ports this way.

ASMX based web services aren't strategic, and have been superceded by WCF based web services. Shame, they're certainly the simplest.

This sample code is supplied for reference in the JMS Adapter package, in the mapper.asmx subdirectory.

Implementing the web service using .NET using WCF

A simple .NET WCF based web service has an interface :-

//
// IMapperService.cs - Service Interface
//

using System;
using System.Collections.Generic;
using System.Runtime.Serialization;
using System.ServiceModel;
using System.Text;

namespace WcfMapper
    {
    [ServiceContract(Namespace="http://www.nyangau.org/jmsadapter/mapper")]
    public interface IMapperService
        {
        [OperationContract]
        ArrayOfString map(string mapName, ArrayOfString unmapped);
        }
    [CollectionDataContract(ItemName="string", Namespace="http://www.nyangau.org/jmsadapter/mapper")]
    public class ArrayOfString : List<string> { }
    }

Note the way the [CollectionDataContract] is used to ensure that when lists of strings are converted to XML, they have elements called string, so as to match the JAX-WS code and ASMX .NET examples above.

It also has an implementation of the interface :-

//
// MapperService.cs - Service Implementation
//

using System;
using System.Collections.Generic;
using System.Runtime.Serialization;
using System.ServiceModel;
using System.Text;

namespace WcfMapper
    {
    [ServiceBehavior(Namespace="http://www.nyangau.org/jmsadapter/mapper")]
    public class MapperService : IMapperService
        {
        public ArrayOfString map(string mapName, ArrayOfString unmapped)
            {
            ArrayOfString l = new ArrayOfString();
            foreach (String s in unmapped)
                if ( s.Equals("Gold") )
                    {
                    l.Add(s);
                    l.Add("Au");
                    }
            return l;
            }
        }
    }

Metadata is needed in the form of an App.config file :-

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <system.web>
    <compilation debug="true" />
  </system.web>
  <system.serviceModel>
    <services>
      <service name="WcfMapper.MapperService" behaviorConfiguration="WcfMapper.MapperServiceBehavior">
        <host>
          <baseAddresses>
            <add baseAddress="http://windowshost/MapperService/" />
          </baseAddresses>
        </host>
        <endpoint address="" binding="basicHttpBinding" contract="WcfMapper.IMapperService"
                  bindingNamespace="http://www.nyangau.org/jmsadapter/mapper">
          <identity>
            <dns value="windowshost"/>
          </identity>
        </endpoint>
        <endpoint address="mex" binding="mexHttpBinding" contract="IMetadataExchange"/>
      </service>
    </services>
    <behaviors>
      <serviceBehaviors>
        <behavior name="WcfMapper.MapperServiceBehavior">
          <serviceMetadata httpGetEnabled="True"/>
          <serviceDebug includeExceptionDetailInFaults="True" />
        </behavior>
      </serviceBehaviors>
    </behaviors>
  </system.serviceModel>
</configuration>

Note the use of bindingNamespace to complete the job of ensuring everything is in the right namespace. Its not enough to use attributes in the C# code. This attribute isn't present in a newly created Visual Studio WCF project.

Its WSDL can be fetched by appending ?WSDL to the end of the URL, eg: http://windowshost/MapperService/?WSDL. Notice that httpGetEnabled="True", so that the WSDL is exposed. Again, this attribute isn't present in a newly created Visual Studio WCF project.

It is also necessary to specify port="BasicHttpBinding_IMapperService" in the <mapws/> element, as the .NET WCF WSDL names its ports this way.

This sample code is supplied for reference in the JMS Adapter package, in the mapper.wcf subdirectory.

Data maps - Mapping and caching returned results

Some mappings are expensive to perform. Perhaps they read files or from URLs, or make queries to external services, such as databases or web services, perhaps they do difficult computations. So you may want to cache the results and avoid remapping the same value for a while (or maybe even indefinitely).

This here is a data map which caches data returned :-

<mapcache name="ElementName_to_ElementSymbol" ttl="60000" size="1000">
  <mapws wsdl="http://localhost:8888/ElementMapper?WSDL"
         mapping="Name_to_Symbol"/>
</mapcache>

Any mapped value returned from the inner mapping is kept for at most 60000ms. Note that this is 60000ms from the moment the value is returned from the inner mapping, not from the moment the inner mapping is invoked. If ttl is not specified, returned mappings are not expired.

At most 1000 returned values are kept in the cache. If size is not specified, the default is 100.

When the inner data map is invoked, it can return more mappings than it is asked for, and the cache will retain everything that is returned.

So in the following example, if asked to map "Monday" and "Tuesday", and later asked to map "Wednesday", the file is only read once. This is because the size is large enough to keep all the answers and there is no ttl defined :-

<mapcache name="EnglishDayNames_to_FrenchDayNames" size="7">
  <mapprops xml="file:EnglishDayNames_to_FrenchDayNames.xml"/>
</mapcache>

Data maps - Mapping using a list of data maps

Mappings can be assembled to make compound maps using <maplist/>, eg:

<maplist name="tryLocalThenLookupInDatabase">
  <mapprops properties="file:CountryNames_to_CountryCodes.properties"/>
  <mapdb driver="oracle.jdbc.OracleDriver"
         url="jdbc:oracle:thin@127.0.0.1@1521:XE"
         userid="scott" password="tiger"
         scolumns="cty_name" dcolumn="cty_code" table="CTY_TABLE"/>
  <mapdefault dest="XX"/>
</maplist>

The above example has a local file which maps United Kingdom to UK, France to FR etc.. However, if after this properties file has been searched, any remaining values still remain unmapped, it consults the database, and if after that there are unmapped values they are mapped to XX. Note how <mapdefault/> ensures the mapping as a whole cannot fail, and no exception will be thrown

Note that simple mappings, <mapcache/> and <maplist/> can be combined in a variety of ways. You can construct maps that consult a variety of sources, and cache the results from each of them for varying lengths of time.

Data maps - User defined maps

I'm not likely to be able to second guess every conceivable mapping a customer might like to do. So there is an extension mechanism.

<mapuser
    class="className"
    ... other attributes >
  ... other nested elements
</mapuser>

This allows the user to add their own data map which does whatever they need. The className is loaded, and its constructor called with the DOM org.w3c.dom.Node corresponding to the <mapuser/>, and the implementation then reads its configuration. At runtime, the map method is called passing a set of strings to map, and a map to populate with mappings.

The rule is that any mapping that is successfully performed must be removed from the unmapped set, and the mapping added to the mapped map. Additional mappings may be added to the the mapped map, in the hope there is a caching layer above to retain them.

A somewhat contrived simple user defined mapping :-

//
// DataMapToLowerCase.java - Map to lower case
//

package mypackage;

import java.util.Set;
import java.util.Map;
import org.w3c.dom.Element;
import org.w3c.dom.NodeList;
import org.w3c.dom.NamedNodeMap;
import org.w3c.dom.Node;
import org.w3c.dom.Text;
import nyangau.jmsadapter.*;

public class DataMapToLowerCase extends DataMap
  {
  public DataMapToLowerCase(Node eDataMap)
    throws FlowProcessorException
    {
    super(eDataMap);
    }
  public void map(Set<String> unmapped, Map<String,String> mapped)
    throws FlowProcessorException
    {
    for ( String s : unmapped )
      {
      mapped.put(s, s.toLowerCase());
      unmapped.remove(s);
      }
    }
  }

The jmsadapter.xsd used to validate JMS Adapter XML configurations is designed to use <xsd:any/> and <xsd:anyAttribute/> so as to allow you to put your own elements and attributes in the XML configuration file. In this case, to use the step, you'd use :-

<mapuser classname="mypackage.DataMapToLowerCase"/>

When writing your own data map, there is usually no need to implement any kind of caching, as you can always wrap a <mapcache/> around it.

Remember to put your .class or .jar file on the classpath.

There should be limited requirements to write user defined data maps, given that the ability to invoke web services is built-in.

Data maps - Tuple Mapping

The data mapping framework delivered with JMS Adapter only really understands the most common case of mapping a single value to a single value.

Sometimes you can want to map a single value to multiple output values, eg: a "location" to a tuple comprising a "country" and the "site" within the country. This should be handled by creating a mapping from location to country, and a mapping from location to site :-

For
  location -> ( country , site )
    "PARIS_FRANCE" -> ( "FR" , "Paris" )
    "PARIS_TEXAS"  -> ( "US" , "Paris" )
Use
  location -> country
    "PARIS_FRANCE" -> "FR"
    "PARIS_TEXAS"  -> "US"
  location -> site
    "PARIS_FRANCE" -> "Paris"
    "PARIS_TEXAS"  -> "Paris"

You might also want to map a number of values to a single value, eg: mapping a tuple comprised of "country" and "site" to a "location". To handle this, you should use XSLT to put the input values next to each other, seperated by a seperator such as a comma. The seperator should not be something occuring naturally in either of the values. This should not be difficult to find, as you have the whole Unicode character set at your disposal.

For
  ( country , site ) -> location
    ( "FR" , "Paris" ) -> "PARIS_FRANCE"
    ( "US" , "Paris" ) -> "PARIS_TEXAS"
Use
  country,site -> location
    "FR,Paris" -> "PARIS_FRANCE"
    "US,Paris" -> "PARIS_TEXAS"

A similar approach can be used for 1:3, 3:1, 2:2 and N:M mapping requirements.

Exception handling

As you will have seen, there are many reasons why the mediation of a message could fail, and exception being thrown.

JMS Adapter must have a strategy for handling this.

We don't have the option of letting an exception be thrown to the caller as a Java Exception. The caller isn't expecting anyone to be mediating messages, and so this would be an unwelcome surprise.

Traditionally, an adapter sends unprocessable messages to a dead letter queue, or a message hospital. Hopefully a human being gets to look at these, possibly fix them up and/or replay them :-

However, JMS Adapter has no such luxury. If asked to send a message, it had better send a message. If asked to receive a message, it had better return one. This is required so as to preserve all the nice message commit and any XA semantics that may be in force.

So we adopt a different approach. We rely on the fact an application must have a strategy of its own for dealing with bad/poisonous messages. Typically this would be to send the message to a dead letter queue, or a message hospital.

When we fail to mediate a message, all we need to do is create a deliberately bad message, and pass it onward. For a JMS Adapter attached to a publisher, this looks like this :-

And to a subscriber, it looks like this :-

This is why flows have an eflow attribute, which identifies an exception flow.

It is the job of the exception flow to create a deliberately bad message. eg: If the application expects a TextMessage, consider sending it a MapMessage. eg: If it expects a JMS property called "trade-id", don't give it one.

The bad message should ideally contain the identity of the message. This is an important point - whoever gets to investigate the message is going to want to know which message it was, so the main flow should strive to fish this information out of the message being mediated as soon as possible. As the failure could happen at various points throughout the flow, the message itself could be in a partially converted state when the failure happens. The message identity might be in the "TradeNumber" JMS property, or have to be XPath-ed out of the payload. The main flow should aim to store the message identity in the special _JMSAdapter_message_id variable as a String, as soon as possible.

The bad message should also contain some diagnostic information. At the point the exception flow is invoked, the following special variables will be set up :-

• _JMSAdapter_message_id, a String, (if the main flow managed to set it up prior to the point of failure)
• _JMSAdapter_flow_name, a String, the name of the top level flow running, at the point of failure.
• _JMSAdapter_flow_context, a String, the diagnostic form of the message context, showing the message content and all variables, at the point of failure. The diagnostic form is the same output that the <debug/> flow step outputs.
• _JMSAdapter_flow_exception, an Exception, detailing what exactly went wrong. If converted to a String using <vartype var="_JMSAdapter_flow_exception" type="String"/> then the result is a textual representation of the stack trace. This is the only reason Exception can be converted to String in the conversions table.

Pass the diagnostic information in JMS headers, and perhaps the payload, in such a way that the receiving application won't expect and interpret them. Here is an example :-

<flow name="StdConfirmation_to_AppConfirmation" eflow="make_poison_message">
  <propget prop="trade" var="_JMSAdapter_message_id"/>
  ... rest omitted
</flow>

<flow name="make_poison_message">
  <bodytype type="MapMessage"/>
  <varset value="JMS Adapter ${ja_name} couldn't process a message" var="info"/>
  <propset var="info" prop="BAD_explanation"/>
  <vardef var="_JMSAdapter_flow_message_id" value="unknown"/>
  <propset var="_JMSAdapter_flow_message_id" prop="BAD_message_id"/>
  <bodyset var="_JMSAdapter_flow_name" item="BAD_flow_name"/>
  <vartype var="_JMSAdapter_flow_exception" type="String"/> 
  <bodyset var="_JMSAdapter_flow_exception" item="BAD_flow_exception" clear="false"/>
  <bodyset var="_JMSAdapter_flow_context" item="BAD_flow_context" clear="false"/>
</flow>

Its quite likely you'd develop one exception flow and refer to it from lots of flows.

If no eflow attribute is present, then the message is converted to a raw JMS Message with no payload, and the previously described _JMSAdapter_* variables are copied to JMS message properties of the same name.

Exception flows raising exceptions

Of course, you could get an exception running the exception flow. This might happen during development, but an exception flow should really be written not to do this in Production.

Notice the use of <vardef/> in the example above to avoid failing if _JMSAdapter_flow_message_id isn't set yet when the exception flow is invoked. This is an example of defensive programming.

If an exception flow was not robust enough, and did throw an exception, then the message is converted to a raw JMS Message with no payload, and any of the previously described _JMSAdapter_* variables which are still defined are copied to JMS message properties of the same name.

In addition, the following JMS message properties are set up :-

• _JMSAdapter_eflow_name the name of the exception flow.
• _JMSAdapter_eflow_context the diagnostic form of the message context, showing the message content and all variables, at the point of failure.
• _JMSAdapter_eflow_exception detailing what exactly went wrong in the exception flow.

At least thats something to go on, although its obviously better to defensively write the exception flow in the first place.

The eflow attribute on an exception flow is ignored.

Command line

JMS Adapter can be run on the command line. It includes a little script called run.sh which basically just puts enough on the classpath, and looks like this :-

#!/bin/ksh

d=`dirname $0`
java -cp $d/nyangau-jmsadapter.jar:$d/jms.jar nyangau.jmsadapter.JmsaMain "$@"

If you've written your own flow step or data map classes, you'd need these on the classpath too.

Version

You can check the version of JMS Adapter :-

$ ./run.sh version
JMS Adapter 0.7

Validate configuration

You can validate your JMS Adapter XML configuration. The location of the XML configuration should be specified as a URL, ie: if its a file, it needs the file: prefix. If the XML configuration refers to ${properties}, you'll need to specify the URL of the properties too. If the XML configuration refers to other resources, such as XSD files, XSLT files, properties files, etc.., and these use relative file URLs, make sure you run JMS Adapter from the right directory. If the XML configuration refers to web service WSDL, be sure it is accessible. No output means everything is ok, exception stack traces indicate problems :-

$ cd example
$ ../run.sh validate file:appstd.xml file:appstd.properties

If you refer to a ${property} but don't define it, or perhaps you forget to pass the URL of the properties, you could see :-

$ ../run.sh validate file:appstd.xml
nyangau.jmsadapter.FlowProcessorException: can't expand ${ja_name} as property or system property
        at nyangau.jmsadapter.FlowHelper.substituteProps(FlowHelper.java:322)
        at nyangau.jmsadapter.FlowHelper.substituteProps(FlowHelper.java:334)
        at nyangau.jmsadapter.FlowProcessor.<init>(FlowProcessor.java:176)
        at nyangau.jmsadapter.JmsaMain.main(JmsaMain.java:51)

If web service WSDL wasn't available, you might see something like this :-

$ ../run.sh validate file:appstd.xml file:appstd.properties
nyangau.jmsadapter.FlowProcessorException: WebServiceException in data map ElementName_to_ElementSymbol
        at nyangau.jmsadapter.DataMapWebService.<init>(DataMapWebService.java:65)
        at nyangau.jmsadapter.DataMap.newDataMap(DataMap.java:62)
        at nyangau.jmsadapter.FlowProcessor.<init>(FlowProcessor.java:252)
        at nyangau.jmsadapter.JmsaMain.main(JmsaMain.java:51)
Caused by: javax.xml.ws.WebServiceException: Failed to access the WSDL at: http://localhost:8888/ElementMapper?WSDL. It failed with: 
        Connection refused.
... snip

The XML configuration is validated using a file called jmsadapter.xsd, which is included as a resource inside the nyangau-jmsadapter.jar. This validation process catches a very large proportion of illegal configurations. If your configuration passes this test, there is a very low chance an exception will be thrown due to a configuration error when the application using JMS Adapter is run.

Standalone testing

It can be handy to be able to work on flows without having any JMS provider or application handy.

If you look at a given flow, it might typically have a structure like this :-

<flow name="somename">
  ... fetch parts of the inbound message to variables
  ... process the variables (validate, enrich, transform, ...etc)
  ... store variables into the outbound message
</flow>

If you comment out the first part, you can do things like this :-

$ cd example
$ ./run_app_debug.sh 

Which is coded like this :-

#!/bin/sh

d=`dirname $0`/..
java \
-DCSLMAP=FILE \
-cp $d/nyangau-jmsadapter.jar:$d/jms.jar:ojdbc14.jar \
nyangau.jmsadapter.JmsaMain \
flow \
file:appstd.xml \
file:appstd.properties \
AppDeal_to_StdDeal \
v0=:Hello file:AppDeal.xml

The Flow Processor is initialised, using the XML configuration and properties supplied. If there are no properties, pass -. The indicated flow is located (eg: AppDeal_to_StdDeal).

Next a Flow Context is prepared containing a plain old Message with no body, and no variables defined. The remaining arguments are iterated over. Any argument with an = is treated as a variable assignment. The variable is assigned the text from the URL indicated. An argument without an = causes the message to become a TextMessage with content from the URL indicated.

If a URL starts with : then its a special case - the value is the text immediately following. Kind of like a "here document", but for URLs.

The context is displayed, the flow executed, then the context re-displayed.

The above example could produce :-

$ cd example
$ ../run_app_debug.sh
=== BEFORE AppDeal_to_StdDeal
Message propertes:
Message body (TextMessage):
  text = "<?xml version="1.0"?>
<AppDeal>
  <trade>1234</trade>
  <stock>Gold</stock>
  <amount>1000</amount>
  <country>UK</country>
  <site>London</site>
</AppDeal>
"
Variables:
  var "v0" = String "Hello"

Debugging output in the middle not shown, then ...

=== AFTER AppDeal_to_StdDeal
Message propertes:
  prop "trade" = String "1234"
  prop "Comment" = String "Transformed by JMS Adapter"
Message body (TextMessage):
  text = "<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<StdDeal>
    <stock>Au</stock>
    <amount>1000</amount>
    <where location="LONDON"/>
</StdDeal>
"
Variables:
  var "v0" = String "Transformed by JMS Adapter"
  var "_JMSAdapter_message_id" = String "1234"

The example above is lucky in that the flow can just as easily work with a TextMessages as it can a MapMessages. This might not always be the case, hence the comment above about commenting out the logic which reads the variables from the message.

Examples

Validating the message

Sometimes we only want to be sure the message being sent is valid. The message is a TextMessage, and needs to be validated against a particular schema :-

<flow name="BytesMessage_to_TextMessage" eflow="make_poison_message">
  <bodyget/>
  <vartype type="Document"/>
  <xmlvalidate schema="file:AppDeal.xsd"/>
</flow>

An eflow attribute is used as the message text may not parse or validate cleanly.

Changing the body type of the message

In this example, the message is a BytesMessage and the payload is XML. This means we should infer the character encoding using the XML character encoding rules, so we use encoding="XML". We want a TextMessage :-

<flow name="BytesMessage_to_TextMessage" eflow="make_poison_message">
  <bodyget/>
  <vartype type="String" encoding="XML"/>
  <bodytype type="TextMessage"/>
  <bodyset/>
</flow>

An eflow attribute is used as we could have difficulty decoding the bytes into characters using the designated character encoding. eg: the XML could start with :-

<?xml encoding="something-silly"?>

Content based flow processing

Note I am careful not to say "content based routing", as JMS Adapter never dynamically picks which underlying destination to publish or subscribe to. Here we consider what happens if more than one message type flows through a given queue or topic. Consider that "buys" and "sells" are sent through the same queue, and we can tell which is which by the top level document node name :-

<flow name="SenderBuyOrSell_to_ReceiverBuyOrSell">
  <bodyget/>
  <vartype type="Document"/>
  <varselect xpath="name(/*)" destvar="DOCUMENT_NODE"/>
  <switch var="DOCUMENT_NODE">
    <case regexp="SenderBuy" flow="SenderBuy_to_ReceiverBuy"/>
    <case regexp="SenderSell" flow="SenderSell_to_ReceiverSell"/>
  </switch>
  <bodyset/>
</flow>

A more defensive approach might be to add the following limb :-

    <case regexp=".*" flow="strange_sender_message"/>

and the following flow :-

<flow name="strange_sender_message">
  <throw cause="not a SenderBuy or SenderSell"/>
</flow>

Other XPath expressions could be used to make decisions off of other parts of the message.

The example included with JMS Adapter

You have seen some of this already, some of its artifacts have already been used to illustrate points earlier in this document.

The example is included in the example subdirectory.

Scenario

A standard service expects standard deals on its input deal queue, and it sends standard confirmations on its output confirmation queue. We have an application which generates deals in an application specific deal format and expects confirmations back in an application specific format.

The problem here is that the application sends something very different to what the standard service expects, and the standard service replies with something very different to what the application expects. An ideal opportunity for JMS Adapter to do its magic.

In order to convert between application specific and standard service specific formats, there are structural differences in the messages and there are data elements/attributes which need to be mapped using an external web service and an external database table.

The database is not not shown in the picture, as its use is optional. We can persuade the example not to use it, and to use local file data instead.

Here is a pictorial representation of the example :-

The JMS Adapter administered objects set Jmsa.CONFIG to file:appstd.xml. The XML configuration is named this because it converts between application and standard messages.

XSD schemas are supplied describing the 4 different message formats. The application reads a sample valid application deal from a file and sends it. The standard service also works by reading a sample valid standard confirmation message from a file. Its the "Blue Peter" approach, ie: "here is one I prepared earlier". In other words, the application and standard service don't actually go to great lengths to assemble and process the messages they send and receive, thats not really the point of the example - the point is to show the conversion in the middle.

The application sends an application deal in the appNode item of a MapMessage, and it looks like this :-

<?xml version="1.0"?>
<AppDeal>
  <trade>1234</trade>
  <stock>Gold</stock>
  <amount>1000</amount>
  <country>UK</country>
  <site>London</site>
</AppDeal>

The standard service expects a standard deal in the stdNode item of a MapMessage, and it looks like this :-

<?xml version="1.0"?>
<StdDeal>
  <stock>Au</stock>
  <amount>1000</amount>
  <where location="LONDON"/>
</StdDeal>

Immediately we can see that we have a lot of work to do :-

• The payload will need to be moved from appNode to stdNode.
• The structure of the message is different, so we'll need to use XSLT to convert it.
• In the application deal the commodity being traded is identified by element name, in the standard deal its by element symbol, so we'll need to use data mapping to convert.
• The (country,site) tuple is mapped to a location. Its a 2:1 map, as (FR,Paris) would map to PARIS_FRANCE and (US,Paris) would map to PARIS_TEXAS.
• The trade number isn't present in the payload of the standard deal. In fact, the standard service expects it to be present in a JMS message property called trade.

In the reverse direction, the standard service sends a standard confirmation, which is a BytesMessage (String payload written encoded using UTF-8), and has the trade number in a JMS message property called trade. The application expects a TextMessage, and the trade number is in the payload.

Data maps

Within the XML configuration (appstd.xml) you will notice there are two implementations of the mappings between (country,site) and location. One uses database lookups and the other uses local file data. You pick which version by setting a system property, to either -DCSLMAP=DB or -DCSLMAP=FILE, and if you don't set the property, the default is to use the database. If you specify some other value, you'll find the XML configuration fails validation. If you use the file implementation, you don't need the database set up, which makes it easier to run the sample. You do still need the JDBC .jar file on the classpath. If you use the database implementation, you'd need to use SQL*Plus to run the example/setupCslTable.sql script.

The CountrySite_to_Location_FILE data map uses <maplist/> to demonstrate looking in a number of places.

The mappings between element names and element symbols are implemented as web service queries.

Flows

To ensure that the application deal is mediated on the way out, the configuration contains :-

<queue name="stddealq" out="AppDeal_to_StdDeal"/>

The flow contains the following (some debug related items removed) :-

<flow name="AppDeal_to_StdDeal" eflow="make_poison_message">
  <bodyget item="appNode"/>
  <bodydel item="appNode"/>
  <vartype type="Document"/>
  <xmlvalidate schema="file:AppDeal.xsd"/>
  <varselect xpath="//trade" destvar="_JMSAdapter_message_id"/>
  <xmltransform xslt="file:AppDeal_to_StdDeal.xsl"/>
  <varmap xpath="//@location" map="CountrySite_to_Location_${CSLMAP:DB}"/>
  <xmlvalidate schema="file:StdDeal.xsd"/>
  <vartype type="String">
    <outputproperty name="indent" value="yes"/>
    <outputproperty name="{http://xml.apache.org/xslt}indent-amount" value="4"/>
  </vartype>
  <varmap regexp="&lt;stock&gt;([A-Za-z]+)&lt;/stock&gt;" group="1"
          map="ElementName_to_ElementSymbol"/>
  <bodyset item="stdNode"/>
  <propset var="_JMSAdapter_message_id" prop="trade"/>
  <varset value="Transformed by JMS Adapter"/>
  <propset prop="Comment"/>
</flow>

Talking this through :-

• The payload is extracted from the appNode item in the MapMessage, then that item is deleted from the message (after all, that item won't be present in the final message).
• Its parsed to an XML DOM.
• Its validated against the application deal XSD.
• The trade ID is XPathed out, and is put in the the special _JMSAdapter_message_id variable so that if an exception occurs trying to mediate the message, we have it handy.
• The structure of the message is transformed using XSLT.
• In the new message, the location attribute (of the where element) has the country and site seperated by a comma. This is mapped using <varmap/>. XPath is used to select the attribute, and a data map is identified to do the mapping.
• The resulting message is validated again, this time against the standard deal XSD.
• The message is converted to a String, hopefully with reasonable indentation.
• Then we map the element name to element symbol. First, notice the really ugly way that angle brackets are given in &entity; form in the regular expression, which is necessary according to XML rules. Really this is a bad way and bad place in the flow to map what is really XML data. It is shown here to demonstrate that mapping can select parts of a String using a regular expression. It also shows the full regular expression matching stuff around the text to be mapped, and the capturing group being used to say what part of the match is to be mapped. Normally the data mapping would be done while the message was still in DOM form, using XPath to select the text to be mapped, and before the final XSD validation happened.
• The message is then set into the stdNode.
• The trade ID is set into the trade message property.
• A comment is added to the message.

I'm not going to show the full confirmation flow, as it is similar, but in reverse. Here are the parts that show how the trade number in the JMS message property becomes a part of the payload (using the XSLT parameter mechanism) :-

<flow name="StdConfirmation_to_AppConfirmation" eflow="make_poison_message">
  <propget prop="trade" var="_JMSAdapter_message_id"/>
  <propdel prop="trade"/>
  ... snip
  <xmltransform xslt="file:StdConfirmation_to_AppConfirmation.xsl">
    <param name="trade" var="_JMSAdapter_message_id"/>
  </xmltransform>
  ... snip
</flow>

The other significant difference is the use of <bodytype type="TextMessage"/> to convert the message from BytesMessage to TextMessage.

Exceptions

The exception flow is basically the same as the example given in the Exception handling section of this document. In the diagram, the exception handling is indicated as red arrows leading to an unhappy face. Basically the strategy used by the application and standard service is to display a message on the standard output where the user will see it. It is only a demo.

Consider the AppDeal_to_StdDeal flow shown above, as there is scope to do better. Observe that we don't know the trade ID until after we've parsed and validated the message, and perhaps the problem is that it is not well-formed XML. Perhaps there is scope to insert the following immediately after the payload is got using <bodyget/> :-

  <varselect regexp="&lt;trade&gt;([0-9]+)&lt;/trade&gt;" group="1"
             destvar="_JMSAdapter_message_id"/>

If the XML isn't well-formed, we might capture something meaningful. Of course, might also accidentlally catch matching text within an XML comment. If the XML is well-formed, and validates, we still do the proper XPath later, so we'll be sure to be no worse off.

Configuration

The diagram shows a pictoral summary of the configuration files used, connected to JMS Adapter using grey arrows. In the example, all the configuration files are referred to using relative file: URLs. This is great for development and testing. When it comes to production deployment it would probably be better to collect all of them together (except appstd.properties) in to appstd.jar and use URLs like this :-

jar:appstd.jar!/AppDeal.xsd

If it were ever necessary to add some user defined flow steps or data maps, you'd include their .class files in there too.

Running the sample

Prerequisites :-

• Tibco EMS (a JMS implementation) running, listening on port 7222, with no security or passwords set up. It should have two queues defined: stddealq and stdconfirmq. If you want to use another JMS provider, or different queue names, fine, just edit the properties in SetupJNDI.java and Std.java and rebuild.sh and rerun_setup.sh. You'll need to ensure the supplied .sh files point to the right install directory too.
• The Element Mapper web service running. This is supplied in the mapper subdirectory, and is run using run.sh. Once running, its WSDL is visible at http://localhost:8888/ElementMapper?WSDL.
• Oracle XE running (a JDBC implementation) running. This should include a table called CSL_TABLE mapping between (country,site) and location. If you want to use another JDBC implementation, fine, just edit the appstd.xml configuration file. Alternatively, you can pass -DCSLMAP=FILE to run_app.sh.
• The standard service must be running. Start it by running run_std.sh. This service is a normal JMS application, directly using Tibco EMS. It expects to receive a "standard deal" on the stddealq queue, and in response it will send a "standard confirmation" on the stdconfirmq.

Run the application by running run_app.sh. This application is configured to use JMS Adapter, and it knows the queues by the JNDI names appdealq and appconfirmq. The objects it finds in the local filesystem JNDI implementation are JMS Adapter administered objects, that point to the underlying queues in the underlying Tibco EMS. This application sends a deal in an "application defined deal" format, and expects an "application defined confirmation" format back.

Revision history

I wrote all this code in my own time on my own equipment. I used public non-confidential information to do so. I hereby place all this code into the public domain. Feel free to do whatever you like with it. No copyright / no royalties / no guarantees / no problem. Caveat Emptor! Anyone offering ideas/code must be happy with the above.

Summary

JMS Adapter represents a very lightweight way to mediate data as it is published or subscribed, without having to deploy seperate integration components.

With its relatively simple XML configuration, a large proportion of mediation scenarios can be relatively easily described. Extension mechanisms exist allowing demanding users to enhance/extend the framework to meet demanding requirements.

Download from http://www.nyangau.org/jmsadapter/jmsadapter.zip.