Maximising Your APIs with a Digital Integration Hub

APIs make the digital world go round. According to Gartner, they are an essential element of every digital strategy. They enable systems and applications to interact, connecting disparate data sources and bridging access protocols. APIs can unlock your organisation’s digital assets and allow them to be consumed in a controlled manner.

However, simply exposing your systems and enterprise applications via an API layer presents some formidable challenges, for example:

  • How do you shield your backend systems from being overrun with numerous requests?
  • How do you avoid an unmanageable mess of tightly-coupled point-to-point connections?
  • How do you ensure that consumers only have access to the data they should?

The answer may be a Digital Integration Hub (DIH).

The Problem with a Traditional API Integration Architecture

Modern enterprises will typically have invested in a well-conceived integration architecture for their APIs, introducing multiple layers that serve the data through a request-based integration layer:

clip_image002Image from "Turbocharge Your API Platform with a Digital Integration Hub – Gartner (G00360082, Jul 2018)

The primary role of the integration platform is to field requests from the APIs and deliver data from the source systems. Depending on the complexity of the systems, the underlying data structures, and the expected load from the API consumers, this solution may work reasonably well.

The problem is that even though the integration layer abstracts the underlying systems from the consumers, there is still a degree of coupling:

  • Backend systems ultimately bear the burden of responding to API requests – even if asynchronously.
  • Changes to the backend systems will necessarily impact the APIs and, consequently, the consumers.
  • The data returned by the APIs is typically bound to the data structures of the backend systems as opposed to a non-proprietary business model.
  • Whilst the API Gateway may perform coarse level authorisation, it is unlikely to filter data at a granular level.

In this conventional model, the last two points can only be addressed at the cost of highly complex API service logic – hence the term “macroservices” used by Gartner.

One might represent this coupling as a collection of meshed gears, whereby moving any one will force movement in the others:


Digital Integration Hub to the Rescue

The problems mentioned above can be resolved by introducing a High-Performance Data Store (HPDS) which sits between the integration and the services layer:

clip_image006Image from "Turbocharge Your API Platform with a Digital Integration Hub – Gartner (G00360082, Jul 2018)

The main purpose of this layer is to create an aggregated replica of the system of record data, fed via the integration layer which ideally operates in an event-based fashion (for example, Change Data Capture). This enables several significant improvements:

  • The APIs can deliver a much more responsive user experience as they draw from the HPDS instead of competing with the core processing functions of the target System of Record (SoR).
  • The SoRs can concentrate resources on their primary activities rather than serving API requests.
  • Complete decoupling of the APIs from the SoRs affords potential 24/7 availability, as well as shielding consumers from replacement of a legacy system.
  • The heavy lifting of normalising and aggregating data can be delegated to the HPDS instead of the microservices.
  • Reporting and analytics can also draw from the HPDS, providing business insight whilst again relieving the load on the primary SoRs.

The HPDS can be implemented in many different ways; for example, it may include a data lake, a data warehouse, an MDM, etc. The key is that it is highly performant, and a good metadata management solution would be essential to support a domain-driven design.

The integration layer should also support multiple patterns, although these will vary according to the organisation’s requirements. Example patterns would include:

  • Event brokering / messaging
  • Extract Transform & Load (ETL)
  • Change Data Capture (CDC)
  • Integration Patterns (ESB, iPaaS)
  • Stream processing (Spark, Flint)

Example Architecture

There are few (if any) products that will give you a full Digital Integration Hub out of the box. In all likelihood, it requires an assembly of multiple products, possibly from different vendors.

That said, Microsoft provides an Azure DIH Accelerator to facilitate building a DIH architecture in the cloud. Hosted in GitHub, it provides a pre-configured development environment with a sample application, as well as build and deployment automation.

The architecture is fairly basic, but it is a starting point which supports injection of your own business logic, as well as swapping out components (for example, substituting a SQL Database for the provided PostgreSQL database).

clip_image008Image from Microsoft, DIH Accelerator on GitHub

In this sample, the front-end APIs are implemented via API Gateway and Azure Functions, the integration layer is comprised of Logic Apps and Event Grid, and a data layer consists of a PosgreSQL database. It also provides monitoring via App Insights and Log Analytics. Although not included in the sample, Azure Synapse can easily be bolted on to provide analytics.


A Digital Integration Hub is not a simple architecture by any means and would typically be suited to a large mature enterprise that is going through a digital transformation. Some of the key challenges include:

  • Complexity of rolling out a high-performance data management technology (e.g. NoSQL DBMS or in-memory data grid)
  • Supporting bidirectional, event-driven synchronization between the HPDS and SoR applications
  • Designing a canonical data model for the DIH business entities that supports multiple channels
  • Implementing appropriate metadata management to support discovery and introspection of data entities and relationships represented across multiple data sources
  • Designing, building, and managing the complex distributed architecture of a DIH

Any organisation that aspires to setting up a DIH would do well to follow these recommendations:

  • Determine if the organisation needs a DIH and if it has the skills to support it.
  • Know its data and consumer requirements.
  • Understand the integration patterns that will be required.
  • Design its APIs to be abstracted from the underlying systems (“API First” approach).
  • Consider using a technology partner for implementation.


API-based access to disparate data services is costly from a performance and maintenance perspective. A Digital Integration Hub enables enhanced performance in accessing your organisational data, while ensuring effective protection of your backend systems. A DIH architecture also provides increased scalability, greater flexibility, and better insights.

Hairy Causes: A Movember to Remember

clip_image002You may notice a trend during the month of November where lots of normally clean-shaven blokes are suddenly sporting (often embarrassingly hideous) attempts at growing a moustache. There’s a good chance that they are making a spectacle of themselves for a worthy cause. Over the past sixteen years, the Movember Foundation has raised over a billion dollars globally to promote awareness of men’s health issues, including prostate cancer, testicular cancer, depression and suicide prevention. Last year, participants in Australia alone raised $29 million. More than 75% of the money raised is used to fund at least 1250 men’s health projects, including Beyond Blue and the Prostate Cancer Foundation of Australia.

For the past 12 years, I too have participated in one way or another in Movember, either by growing a mo’ myself or sponsoring others. This year I have joined the official Deloitte Platform Engineering (DPE) Movember team. My motivation has always come in part from my family’s history of prostate cancer: my dad and two of my three brothers have endured this disease. This year, I have even more inspiration, as I recently extended my family’s history via my own battle with prostate cancer. I will share a bit of my story below. Read more of this post

Life as a User Group Leader

Microsoft MVPs are recognised for their voluntary contributions to the technical community. There are many types of eligible contributions, but one of my more notable ones was serving as a user group leader. This is a significant undertaking, and in this post I hope to outline some of the aspects of the commitment and also some lessons I’ve learned over my 14 years of fulfilling this duty.

My Experience

In 2005, I was asked by Microsoft to start the Brisbane BizTalk User Group. The motivation came through working for one of several organisations that adopted BizTalk Server to handle critical enterprise integration processes. As a newbie to the product, I was heavily reliant on the help I received from the very few experts around Australia and the world, including Bill Chesnut, Mick Badran, and several other MVPs who blogged about their experience. With so little available knowledge and experience in Brisbane, Microsoft’s Geoff Clarke decided it would be a great idea to start a user group. It was a daunting challenge and Geoff had to twist my arm a little… but I was encouraged when over 30 people turned up at the first meetup, proving that I wasn’t alone in my struggles. I also had lots of support from Microsoft and my colleagues, and the group met monthly for years to follow.

Then in 2014, I was asked to take the reigns for the Brisbane Azure User Group, which had been established by Paul Bouwer about a year or two earlier. When Paul earned his “blue card” and became a Microsoft employee that year, he felt it was inappropriate for him to continue leading the group and that a community member would be more appropriate for the role. Again, I reluctantly agreed on the condition that I had at least two co-organisers to help. One of these gentlemen (Damien Berry) remains a co-organiser to this day.

I’ve also ran the Global Azure Bootcamp in Brisbane for four years, and the Global Integration Bootcamp for a couple of years as well.

Read more of this post

How to Explain Messaging Patterns to your Grandmother

First of all, I’d like to apologise to all grandmothers out there… I mean you no disrespect. It’s just meant to be a catchy title, really. I know grandmothers who are smarter than most of us.

A couple of months ago I had the privilege of speaking at the API Days event in Melbourne. My topic was on Building Event-Driven Integration Architectures, and within that talk I felt a need to compare events to messages, as Clement Vasters did so eloquently in his presentation at INTEGRATE 2018. In a slight divergence within that talk I highlighted three common messaging patterns using a pizza based analogy. Given the time constraint that segment was compressed into less than a minute, but I thought it might be valuable enough to put in a blog post.


Photo courtesy of

Read more of this post

Mexia Named in BRW Top Starters List!

Mexia Consulting has ranked 71st in BRW‘s list of Australia’s Top Starters for 2013! Congrats to my fearless leaders Mat Coleman and Dean Robertson who’ve led our team to great achievements over the past couple of years. Proud to be a member of this awesome enterprise integration team!!

Mexia Projects Director Mat Coleman and Technical Director Dean Robertson accepting the award from BRW

By the way, Mexia is hiring! Immediate positions available in Brisbane & Melbourne.  If you’ve got BizTalk or enterprise integration skills using the Microsoft stack, please contact us ASAP.


Well, after almost 13 years of service in the IT industry, I’ve finally decided to have my own blog! Not that I haven’t published before – I’ve posted several times on my employer’s website here, and  also published a number of presentations and webcasts as the leader of the Brisbane BizTalk User Group. But finally, I have my own personal soapbox! 🙂  I hope you enjoy whatever you may read here.

John Glisson - Geek of the Cloth

Thoughts on integration, technology and what-not...

Prashant BizTalk And Azure Integration Blogs

My Integration Experiences - BizTalk And Azure Integration


THINK: It's not illegal....yet.....

Abdul Rafay's BizTalk Blog

My experiences with BizTalk related to architecture, development and performance in my enterprise.

BizTalk musings

Issues, patterns and useful tips for BizTalk development


Enterprise Applicaiton Integration and SOA 2.0

Connected Pawns

Mainly BizTalk & Little Chess

Adventures inside the Message Box

BizTalk, Azure, and other tools in the Microsoft stack - Johann Cooper


Talk, talk and more talk about BizTalk

Richard Seroter's Architecture Musings

Blog Featuring Code, Thoughts, and Experiences with Software and Services

Sandro Pereira BizTalk Blog

My notes about BizTalk Server 2004, 2006, 2006 R2, 2009, 2010, 2013 and now also Windows Azure BizTalk Services.

BizTalk Events

Calendar of BizTalk events all over the world!

Mind Over Messaging

Musings on BizTalk, Azure, and Enterprise Integration News

The latest news on and the WordPress community.