The Last Six Integration Patterns You'll Ever Need

IntegrationWorks

Integration architecture can be an intimidating topic to dive into. There is a plethora of different technologies, processes, platforms and patterns advertised to meet your every integration need. So, how do we break out of analysis paralysis and begin on our rewarding journey of integration architecture so we can give the devs something useful to do?

Believe it or not, and feel free not to, I won’t take it personally, there are just three super-patterns, which can model any integration scenario you can think of. Further, these three super-patterns each break down into two sub-patterns, giving us - if my math is correct - six concrete integration patterns.

Data vs Metadata

Before we enumerate the different super-patterns and sub-patterns, we need to talk about data and metadata. The flow of data, metadata or a combination of both, compose each pattern outlined in this article, so it is important to understand the distinction between the two.

Put simply, data is content, metadata is context. The term “meta” implies self-reference. For example, consider an image file. The data is the image itself, the bits and bytes which a photo program will translate to generate a picture. The metadata in this example is data about the data - information about the image such as the size, resolution, encoding format and colour model.

Introducing Data and Metadata Messaging Integration Terminology (DaMMIT)

Until today, there was no accepted standard for differentiating between data and metadata in architectural diagrams. This article proposes a new notation to do just this.

  1. A solid circle at the tail of an arrow identifies the flow of data.

With these patterns, it becomes a much simpler task of identification and modelling. Each sub-task maps naturally to technologies that fit the pattern.

Screen Shot 2021-02-15 at 10.08.11 am

2. A hollow circle at the tail of an arrow identifies the flow of metadata.

Screen Shot 2021-02-15 at 10.09.09 am

This notation is compatible with other standards, which deal with the head or shaft of the arrow. For example, we draw an asynchronous flow of data as:

Screen Shot 2021-02-15 at 10.09.54 am

Or a response of metadata as:

Screen Shot 2021-02-15 at 10.10.43 am

** DaMMIT also maps nicely to sequence diagrams and identifies what sort of data is being transferred at each step. **

Pull Based Integration

The first - and arguably the most prevalent - integration super pattern is that of Pull Based Integration. We use the pull-based interaction patterns when a consumer is requesting a response from a single provider. Data or metadata is retrieved without creating a residual obligation.

This super-pattern tends to map to synchronous request-response technologies and processes such as web services and APIs.

Pull-based integration is composed of two sub-patterns: query-response and request-confirm.

Query-Response

The query-response pattern is a query of data held by the provider using metadata to identify or filter the returned data.

Screen Shot 2021-02-15 at 10.11.46 am

The use-cases for this pattern are many and varied. Any GET verb on a RESTful API probably follows this pattern. For example, consider a “Get Account Details” call by a mobile application. The app sends metadata of the account (e.g. the account ID) to the provider which returns the account data (name, address etc.).

Request-Confirm

The request-confirm pattern is a query of the state of the data held by the provider. The provider returns metadata on the supplied data.

Screen Shot 2021-02-15 at 10.12.27 am

The use-cases for the request-confirm pattern are more nuanced and rarer than that of the query-response pattern. As an example, consider a form-validation service. The consumer would send the completed form to the provider, which would validate the form and return metadata describing its acceptability.

Push Based Integration

Pushed-based integration is when the provider sends data or metadata to one or more external consumers. The two patterns for providing push-based interaction are: notification and information distribution.

The pushed-based pattern maps best to asynchronous technologies such as event streaming.

Notification

A notification is a fire-and-forget pattern that is used to perform notifications about data changes. This pattern should be used when a provider is notifying an interested consumer that an event has taken place on their data.

Screen Shot 2021-02-15 at 10.13.23 am

An example use-case for the notification pattern is a delivery status update of a courier company. Metadata on a package (e.g. its delivery status) is sent to consumers to notify them of a state change of the data.

Information Distribution

The information distribution is the second variation of the push-based integration super pattern. It is an informal information exchange between parties to perform data synchronisation. We should use this pattern when a provider has changed data that a subscriber has registered an interest to keep synchronised.

Screen Shot 2021-02-15 at 10.14.08 am

An example use-case for this pattern is the distribution of a schedule. A large conference may publish a schedule of lectures available to conference goers through an application. Schedule changes can trigger a push of the data from the provider (conference organiser) to the consumers (conference attendees).

Information Exchange

The Information Exchange super pattern is commonly associated with more complex integrations. The request-response and commercial transaction sub-patterns form the information exchange super-pattern.

Enterprise Application Integration, or Enterprise Service Bus topologies typically implement the information exchange patterns.

Request-Response

The request-response pattern is a set of related data exchanges between systems for a single workflow, with data flowing in both directions.

Screen Shot 2021-02-15 at 10.14.58 am

This pattern is common in the UBL service interactions. For example, the quotation process involves the customer sending data pertaining to the quotation to the supplier, who responds with data of the quotation itself.

Commercial Transaction

This pattern implements a transaction which requires a form of non-repudiation. The term non-repudiation refers to an assertion which cannot be disputed. There are two main types of non-repudiation in integration architecture - non-repudiation of receipt, and non-repudiation of origin.

Non-repudiation of receipt means that a system sending a message has a guarantee that the destination system has in fact received it; and has concrete evidence to refute any contrasting claim (“I never got that message!”) made by the receiving system. Typically, this is achieved by requiring a receipt on delivery sent by the receiving system back to the sender, which the sender then retains to prove delivery was successful.

Non-repudiation of origin means that the receiving system can be sure that the message originated where they expected and that it has not been altered in transit. This is achieved by payload encryption and digital signatures and refutes any contrasting “I never sent that!” or “That’s not what I sent!” claims made by the sender.

Screen Shot 2021-02-15 at 10.15.41 am

The use-cases for this pattern are somewhat tautological - commercial transactions. Payments or other legally binding transactions between systems will require this pattern of integration.

Conclusion

Integration architecture can be intimidating. However, if we keep in mind these simple building-blocks of super-patterns and sub-patterns, and if we simplify our interactions as the flow of data and metadata between systems, then we can model even large or complex applications consistently and understandably.

 

If you’d like to know more about our work with any customer, please contact us here:

Ready to talk to us?

WHERE ARE WE?

 

NEW ZEALAND

Auckland
  • Level 6, Infosys House 79 Queen Street Auckland Central 1010
Wellington
  • Level 12, State Insurance Tower, 1 Willis Street, Wellington, 6011
  • +64 4 499 0930

AUSTRALIA

Brisbane
  • Level 10, 300 Ann Street, Brisbane, QLD 4000
  • +61 418 730 398
Sydney
  • Level 10, 20 Martin Place, Sydney, NSW 2000
  • +61 2 9051 2546
Melbourne
  • Level 2, Riverside Quay, 1 Southbank Boulevard, Southbank, VIC 3006
  • +61 3 9982 4433

UNITED KINGDOM

Dundee
  • Level 131 Albert Square, Meadowside, Dundee, DD1 1DJ
  • +44 7880 443 009