Use of integration patterns in batch scenarios

The book Enterprise Integration Patterns – Designing, Building and Deploying Messaging Solutionsby Gregor Hohpe and Bobby Woolf (Addison-Wesley) describes a number of patterns for enterprise integration, focusing mainly on messaging concepts. It is interesting to note that many of these patterns can be applied to the batch scenarios as well, especially since the batch processing landscape has changed in the last several years from a traditional sequential processing style into a multi-threaded processing style involving integration with several internal and external systems.

Not so long ago, I used a couple of these patterns while recommending a solution for the architecture for the batch processes in a project. Various batch processes in the project consisted of multiple processing steps and adhered to the following two styles of execution:

• Per-record multi-threaded execution – Every selected record is assigned to a thread in a thread pool and goes through the various steps sequentially. The job ends when all threads join.
• Per-step multi-threaded execution: All selected records execute a given step concurrently via threads in a thread pool. However, these records will go to the next step only after all records finish the previous step.

(In the actual scenarios in the project, the steps were again subdivided into tasks. For the sake of simplicity in this article, I am considering each step as the lowest unit of work). The steps in the batch process were time-consuming. Without putting much extra burden on each step, I had to find a way to extract some information when each step finishes and use this information for purposes such as checkpoints, logging and monitoring.

The approach chosen was to use a wire-tapping mechanism to receive messages from every step with the required information, and then use the recipient list pattern to forward the information to the registered listeners. The Wire Tap is an integration pattern widely used in message-based integration architecture to capture a copy of the messages while they are being forwarded to the ultimate destination. Similarly, the Recipient List pattern is used in the integration domain to route messages to a number of statically or dynamically specified recipients.

The above diagram shows the batch execution styles (a) and (b) modified by incorporating these patterns. From an implementation point of view, a decorator pattern was used to add to each step the responsibility to send messages to the wire tap at the end of the processing.

Interestingly, several other integration patterns may be used in batch architecture. For example:

• Routers allow the data to choose the processing step to be used, based on certain properties or rules.
• Translaters allow the data to be transformed, as required by a given processing step.
• Splitters allow the data to be split and fed into various processing steps concurrently.
• Aggregators allow the data from each concurrent processing step to be combined, for the use of subsequent steps.

But in all these cases, the patterns are used in a data-centric fashion rather than message-centric, mainly because the batch processing tends to be data-focussed, whereas the integration is more message-oriented.