The integration imperative
Systems integration is one of the most significant and persistent challenges faced by enterprise. Organisations all over the world are struggling to effectively liberate, control and govern their data. They’re dealing with monolithic architectures, legacy systems, data silos, inconsistencies, inefficiencies and duplication.
Manual integrations between systems involve a huge amount of effort. You take a data dump from one system, manually manipulate the data into a compatible format before uploading it to the new system.
This approach has left organisations with no data governance, poor information security and significant cost inefficiencies. This method is entirely manual, so it opens the door to huge duplication of effort and over time creates a mess of inefficient and insecure ways of managing data.
Numerous point to point integrations are often developed to get two systems talking to each other. By creating these 1:1 relationships, developers are able to build an automated process to manage data integration between two distinct points.
But because the integrations aren’t strategic and don’t scale, they also lead to poor data governance, duplication of effort, gross inefficiency and cost. With new integrations occurring often without visibility, control or the opportunity to evolve over time, this only adds to the complexity of the enterprise integration picture.
While most organisations are somewhere in the midst of these approaches, experimenting with new technologies and getting into the nitty-gritty of application integration and data liberation, a solution has been building traction: the integration platform.
What is an integration platform?
An integration platform is a solution that allows the development of integrations, strategies and API’s for various systems. They provide centralised access to core data together with an operating model to allow you to continuously evolve the services and integrations on the platform.
Establishing an integration platform reduces the complex maintenance and communication channels generated by point to point integrations and decouples the connections between source and destination by providing a platform for integration. Integrations can be deployed with minimal changes to existing applications and are kept secure by the policies and SLAs from your chosen provider.
A number of iPaaS products have risen to prominence, providing this platform essentially “as a service” and has been successful in helping organisations accelerate their integrations. But they bring their own host of challenges into the enterprise integration picture and aren’t always the best fit if you’re embarking on a new strategy today.
We’ve supported a lot of integration programmes over the years, with recent clients including Heathrow and University of Nottingham, and despite the popularity of iPaaS products, we’ve seen time and time again that building a cloud-native integration platform is often the best way to integrate systems and accelerate your transformation.
But how do you know which approach is the right fit for your business?
Platform selection strategy
It’s vital when you’re selecting your integration platform that you understand your current IT landscape, your forward IT strategy and the larger scope of what needs to be achieved.
When assessing platforms for their effectiveness and impact you must take into account your existing system landscape, integration approaches, developer skill sets, security, monitoring, advanced services such as IoT or stream analytics as well as wider operational and support structures. A ‘perfect’ solution in isolation that would require large scale changes in your workforce or that does not complement existing and future systems is probably not fit for purpose.
The cost of an integration platform is a lot more than just the cost of the licences. You must also consider the cost of the skilled people required to develop, maintain and support the integrations. Some products emphasise lightweight development approaches, however, it requires strong technical expertise to produce good results.
WHAT DO WE WANT OUR ARCHITECTURE TO LOOK LIKE?
Ensuring you have a clearlydefined architecture is crucial, especially when it comes to integration. A well-defined architecture enables us to select the appropriate tools for fast feature delivery and has a significant impact on the success of your future integrations.
It can also dictate the performance, maintainability, and future-proofing of your platform.
Best practice architecture for an integration platform should encompass 4 distinct layers: systems of record (SoR), integrations, services and channels.
SYSTEMS OF RECORD (SORS)
No doubt within your business you will have a number of core systems that you want to be able to integrate. These are the long-standing, foundational systems that your organisation may have relied upon for many years. Only by liberating the data from these systems, whether on-premises or hosted by a 3rd party, will you be able to take full advantage of the data within your business and use it in ways that were previously impossible.
These systems of record and the data within them are often business-critical and integral to the success of your organisation. Therefore, you’ll want to treat them with care and be able to standardise and control the development and deployment of these integrations wherever possible.
Does your platform of choice allow for best practices such as continuous integration/ continuous delivery, Infrastructure as Code (IaC), version control and deployment pipelines? Does it integrate with your existing tooling? Is an infrastructure as code approach available to aid environment creation?
These questions are important as with standardisation and automation, comes repeatable and predictable development. You want to avoid being bogged down in custom development for each integration or having to learn a new toolset from scratch before you can begin new integrations.
Ultimately, you want to be able to seamlessly talk to your systems of record, extracting a range of data from these systems to be ingested into your integration platform and liberated for use by other applications and services.
INTEGRATION PLATFORM
Once you’ve identified the relevant data from the systems of record, we’re into the integration layer, the platform itself and the ETL engine (extract, transform, load) which is at the centre of any integration platform. The ETL engine is responsible for ingesting, cleansing, blending and transforming data into standardised or controlled formats and structures, ready to be surfaced, queried or analysed.
Matching, merging and changing data is critical to the success of your integration platform. Your ETL engine should provide these, along with other transformation abilities, which allow modifications to the data. Tools like Data Factory and Logic Apps from Azure, make this available with simple drag and drop functionality. You should also look for tools that provide you with logs and analytics, to monitor and optimise your ETL processes.
Whatever the makeup of your ETL engine, whether it’s Azure components or an in-built engine from an iPaaS platform, it should enable you to enforce data quality and data governance standards. Therefore datasets from individual systems can be used together, and prepare the data to be leveraged by the service level or surfaced to users at the channel level.
SERVICES
With your data transformed, you are ready to make that data available to the service layer.
This layer is essentially responsible for exposing the data to your various user-facing channels.
It is the surface area of your integration platform where managed access to data can take place.
Data published from here should be standardised and viewed through the lens of your organisational operations (not based on SoR constraints and peculiarities). You may also wish to segment your data for granular access from integration platform consumers. This will be led by a well-formed and enforced data architecture and governance strategy.
The services layer should provide you with the ability to expose the data as RESTful APIs, for controlled access to transformed datasets, and through a service bus, for exposing more ‘hot’ or event/ change-feed based data.
To be a truly integrated system the platform should connect to your pre-existing organisational identity management (such as Azure Active Directory). This helps facilitate access control, increase security and data governance while, at the same time, ensuring consistency with the rest of your IT landscape.
With your service layer configured, you are able to expose your data to your channels and use it to improve the experiences you provide to your users.
CHANNELS
How do your users engage with you? Through web-based systems? A mobile app? A physical interaction?
Analysis of the expected user channels will inform your decisions about the earlier layers of your architecture. Specific use cases may require customised or aggregated APIs at the top level of a multi-tier architecture and/or have unique requirements around the user experience.
Therefore your platform should allow for a “layering strategy”, with representation and aggregation of multiple SoR data sets and unique APIs for different applications, while minimising duplication. Lowlevel system APIs can present data more closely resembling that of the underlying SoR, midlevel APIs provide filtered views and capabilities on top of the low-level APIs.
Consumer or top-level APIs then provide the final level of aggregation and tailoring of data to specific usecases providing a seamless experience for your end-user.
Platform strategy: key considerations
When selecting your approach to building your platform these are the questions you need to ask yourself:
DEVELOPER TOOLS
Does the platform provide its own developer tools or offer support for existing tools (such as Visual Studio or Eclipse)?
Can integrations be source controlled to allow for greater audit and control over development?
Do the platform-tools integrate with wider development requirements and your existing development tools?
DEPLOYMENT TOOLS
Is an Infrastructure as Code approach available to aid environment creation and disaster recovery?
Can an automated release pipeline manage all aspects of deployment or are manual runbooks required? Is this available on or through your existing deployment tools?
Have you thought about the costs of development and test environments?
Are your team confident in being able to deploy and manage multiple environments?
OPERATIONS
Can integrations be monitored at a granular level?
Are both high-level management dashboards and low-level data and alerts available?
Can this monitoring integrate with your existing operations tools?
Can the platform be configured to auto scale or failover in response to demand changes or issues?
DEVELOPMENT CAPABILITY
Are the existing development team experienced in the development workflow or do they have applicable skills that crossover and reduce the learning curve?
SECURITY
Does your chosen platform work with your existing identity provider?
Does the platform allow internal and external identities access to different ‘parts’ of the integration?
Does the platform fit within your existing security strategy and infrastructure?
Does the platform have clear lines between public and private services?
CAPITAL COST
Is there a large cost of entry?
Is that cost fixed over a long period?
Can the costs be managed by changes of use?
ONGOING COST
Ongoing costs include recurring license fees and consumption based pricing models.
We can break costs down into:
- Data ingress/egress (data in and out of your platform)
- PAYG (consumption) (only pay for the services you consume)
- Monthly subscription (a subscription fee which covers specific packages)
- Reservations (opportunities to save costs)
The problem with iPaaS
‘Out of the box’ iPaaS offerings can initially seem like an attractive solution to tackle the challenge of enterprise integration.
However, although iPaaS promises straightforward deployment and maintenance of integration flows without the need for hardware or middleware, it is often the case that an iPaaS solution can impinge an organisation’s ability to get maximum value.
Cost models are rarely transparent, you’ll pay an expensive annual subscription often with ‘hidden’ costs from hosting and other per integration costs. This makes it difficult to calculate the true/ total cost of ownership and to predict and optimise costs as you seek to evolve your integration platform capability.
Because of the nature of iPaaS - specifically, the ‘aaS’, utilising an iPaaS product effectively often requires specialist developer and tooling knowledge, meaning you’re having to up-skill your developers and even potentially build a specialist iPaaS team within your business - costing you more time and resource. Whereas with a cloud-native approach, the skills your developers glean can be used across a whole range of work-streams and projects.
Some platforms boast several features to increase ease of use such as drag and drop UIs to allow non-developers to use the product, however in doing so you’re potentially risking a loss of standard practice and increasing the risk of security breaches by having non-IT teams developing integrations.
IPAAS PRODUCTS
Disadvantages:
- Expensive annual licensing often with ‘hidden’ costs from hosting/ per integration costs and lack of transparency
- Requires specialist non-transferable skills for development within the platform and adding integrations
- Challenging to scale to multiple systems with unique requirements
- High risk of vendor ‘lock-in’ with little control over increased licensing changes
- Increased security risk through the democratisation of integration to non-IT teams
- Often dependent on cloud-native architectures to complete complex integrations
Advantages:
- Provides components required for an integration platform in a SaaS environment
- Access data in one place and set rules for how that data is organised and accessed
- Highly available architecture with guaranteed up-time
Cloud-native integration opportunity
Chances are you have already adopted a cloud computing platform, or are intending to at some point in the near future. With IT strategy aligning more and more closely with the cloud provider of choice, it is imperative that the wealth of opportunities and tools they make available form part of the integration conversation; If your using the cloud already for your identity, data and productivity needs, you should consider a solution that can make full use of that, without introducing further platform divergence and data siloing.
Going cloud-native offers everything you need to build out a highly performant, secure and cost-effective integration platform, and will allow you to ensure the applications and data at the centre of your business are accessible to each other.
It provides a complete and integrated development environment and deployment framework, along with reusable components, architecture patterns, development tools, and other services designed to accelerate and simplify the entire application life-cycle.
The cloud provides not only the data-integration components but full platform flexibility with the entire cloud feature set and fully integrated security and monitoring available to you.
By using the wealth of tools, services, and software components offered by your cloud platform provider, not only can you build out a best in class integration platform, organisations can reduce their software engineering workload and develop new system integrations quickly and cost effectively.
Plus, your cloud provider manages your underlying infrastructure, freeing your in-house teams to focus on evolving the platform, instead of getting bogged down with infrastructure management.
Cloud-native offers a wealth of options for cost management and optimisation - a key element for reducing total cost of ownership. It allows organisations to granularly manage costs at a per integration level and optimise integrations as they evolve.
It also gives you a unique platform for modernising existing and legacy systems – a key element of digital transformation. It allows organisations to free themselves from architectures that can’t support rapid change and shake off technical debt and duplication, built up in legacy systems.
As we’ll show you in the next chapter, choosing a cloud-native approach can help solve many of the challenges enterprise integration presents, which explains why 80% of development is expected take place on cloud platforms using microservices and cloud functions by 2021.
CLOUD NATIVE INTEGRATION PLATFORM
Disadvantages
- While you can solve complex problems with bespoke solutions, this may require additional cloud expertise from a partner or in-house developers
- Costs can be granularly controlled, but that may mean investing time upfront to ensure your cost management strategy is appropriate for your scaling needs and budget
- Can be built with existing components available from your cloud provider
- Maintain and build on internal knowledge held by your cloud team
- Move away from on-premises solutions and away from managing infrastructure • Low cost of entry with consumption-based pricing
- Facilitates better data ownership & governance
- Tightly integrate with existing cloud infrastructure and systems
- Allows for granular cost reporting and optimisation at a per integration level
5 big reasons to build cloud-native integration platform
1. MOVE AWAY FROM ON-PREMISES SOLUTIONS AND FREE YOURSELF FROM MANAGING INFRASTRUCTURE
Transformation is not new. A recent survey of IT leaders revealed that 93% of organisations are planning or currently engaged in a digital transformation programme. With the support of the Azure Cloud Adoption Framework, organisations have been taking a structured approach to their transformations, namely PaaS > IaaS > On-Premises infrastructure.
By using a cloud-native approach to build your integration platform, you’re solving any hosting problems you might have. There is no need for on-premises solutions or virtual machines in the cloud (IaaS) and you can avoid the often complex management of the SaaS iPaaS model.
You avoid all the expense and complexity of buying and managing a new product in your ecosystem, looking after the underlying infrastructure or specific development tools and resources. Your teams are free to customise and evolve your integration capability - your cloud platform provider manages
2. LOW COST OF ENTRY & TOTAL COST OF OWNERSHIP
When you choose to build your cloud-native integration platform, you’re avoiding several economic barriers to entry. There’s no software subscription costs or hidden fees to be considered, no costly onboarding/ implementation process or singular training required.
Your costs are transparent and easily reportable. Costs are made up of standing charges, which are predictable monthly costs associated with the various service stock-keeping units (SKU) representing the service’s performance tier; Consumption, meaning you only pay for the compute and storage you actually use; and data ingress/egress, the cost associated with the transport of data in and out of the cloud.
Alongside the cost of entry, going cloud-native can reduce the cost of ownership in many ways. Automated scaling can dial up the cost only when necessary, reducing the IT team overhead of manual intervention. Additionally, test and temporary environments can be spun up and down as needed. By choosing a cloud-native approach, you’re also reducing the amount of bespoking required per integration, allowing you to expand and scale the platform with ease, without attracting additional costs.
3. SAVE DEVELOPMENT TIME AND RESOURCE
With no more time spent on provisioning, configuring and managing infrastructure or learning the numerous nuances of an iPaaS product, your development teams can stay focused on delivering applications.
By using your cloud platform to integrate, your engineers get to the tools they’re already familiar with while integrations with DevOps tools makes deployment simpler, getting more done in less time.
Cloud-native can provide a broader set of compatible capabilities that go beyond pure iPaaS features allowing for a deeper and better tailored integration capability.
Pre-coded software components – from key integration services such as API Management (APIM) or Data Factory in Azure and customisable data transformation pipelines like mapping data flows – mean there’s less engineering to do in the first place.
For example, you could spend weeks on configuring API Gateways or identity aspects for an iPaaS product, with your resources taken up creating custom code. But because going cloud-native requires less coding and works seamlessly with cloud tools already in your business such as Identity Management, your teams are free to focus on adding value to your business. If you choose a cloud provider like Microsoft, you can also take advantage of incredibly tight integration with project management, build, release and test pipelines such as Azure DevOps.
4. BETTER SECURITY, DATA OPERATIONS AND DATA HANDLING
With GDPR in full effect, all organisations are having to take careful steps to ensure that the data they store and process is safe and secure. Whilst it’s true that lots of iPaaS products have security features, by choosing cloud-native you’re able to benefit from a host of data control and data security measures.
Most cloud platforms provide security of data both in transit and at rest, secure identity management with tools such as Azure Active Directory, secret management in Azure Key Vaults all whilst benefiting from the security of the Azure development and deployments tools (Azure DevOps) you’re already using, all backed up by your cloud providers SLAs.
Not only that, you’re able to benefit from data logs and metrics, build automated alerts for data governance protocols and intelligent threat detection.
5. GRANULAR COST MANAGEMENT WITH OPTIMISATION AND REPORTING AT A PER INTEGRATION LEVEL
One of the most documented benefits of adopting a cloud platform such as Azure is cost management. Your integration platform can also benefit from your providers full set of cloud cost management capabilities.
By using these cost management tools, you’re able to manage and optimise your cloud costs whilst maximising the potential of your platform. You have access to an always-on management facility which allows you to accurately plan and control your expenses with advanced cost analytics but you can also proactively optimise your costs by taking advantage of reservations, rightsizing or removing idle resources.
You can track resource usage and manage costs across your entire platform with a single unified view and drill down to a per integration level giving you access to rich operational and financial insights, improving your decision making and helping you manage and reduce the total cost of ownership.
Success Stories
University of Nottingham - BUILDING AN INTEGRATION PLATFORM FROM SCRATCH
Like many leading enterprises, the University of Nottingham (UoN) is like a ‘small city’ with its own unique operational and organisational priorities (managing learning, research, catering, communication infrastructure, transport, real estate, personnel, finance and more). In addition, UoN was facing many IT integration challenges in common with large corporate organisations.
UoN wanted to evolve their approach to integration to centralise and better coordinate University data within and between systems. We built a bespoke integration platform in Microsoft Azure, enabling UoN to unlock and integrate data from a range of critical systems of record (SoRs), improving their student/employee experiences and driving their digital transformation.
With help from our cloud experts, UoN has been able to completely transform the way that data is integrated within the university. By providing a secure and highly performant platform they have liberated and democratised data allowing them to better-leverage information from across the organisation.
We’re continuing to work in partnership with UoN, to help them build an integration Centre of Excellence allowing them to take ownership of future integrations and accelerate their digital transformation. We’re also helping them to modernise their infrastructure as well as designing and developing state of the art digital applications.
Gridserve - INTEGRATING CLOUD-NATIVE TECH WITH 3RD PARTY SYSTEMS
Gridserve is the UK’s answer to Tesla. Providing cutting edge sustainable energy solutions for critical power infrastructure, including electric forecourts, solar farms and telecom towers. Their cloud-native architecture, freedom from legacy systems and IaaS or on-premises solutions allows them to benefit from scalable and elastic infrastructure and simple cloudnative integrations between their systems.
However, in order to activate their vision of electric forecourts across the country, they had to integrate a number of different systems and bi-directional data feeds from various retail and clean energy suppliers.
Dootrix built an integration platform and ETL engine in Microsoft Azure, enabling Gridserve to extract and integrate data from a range of 3rd party systems. The platform brings together, transforms and processes data from multiple organisations, IT systems and applications to synchronize retail, point of sale and electronic vehicle charging data.
The integration platform and ETL play a critical role in day to day operations. By using Azure, Dootrix are able to provide Gridserve with a high level of scalability, stability and security. By implementing a data lake strategy, we’re allowing them to further leverage their data to experiment with data science and analytics to create future business intelligence functions.
Heathrow - LIBERATING DATA FROM LEGACY SYSTEMS
Over the course of the past 5 years, Heathrow has prioritised a cloud-native IT strategy to enable digital transformation across their business, leveraging the latest innovations and technology from Microsoft Azure to migrate and modernise their applications and platforms.
We guided the development of a software solution that used Azure API Management and Azure Service Bus to create a standardised approach for liberating a wealth of data from Heathrow’s enterprise onpremises data store.
The solution allows Heathrow to extract concise streams of information in a normative format to better make use of the vast array of data they expose. We were able to help Heathrow reduce connection/ integration costs by 250% and reduce development time from 3 months to two weeks per integration.
By leveraging a cloud-native integration approach, we managed the creation of the Passenger Information Hub, a solution that ingests a range of critical data from Heathrow’s enterprise data store and surfaces valuable customer insights.
Using API Management, Azure Service Bus as well as Cosmos DB and Power BI to visualise that data and enable Heathrow’s partner airlines to increase revenue and performance from Heathrow Services.
Ofcom - LAYING THE FOUNDATION FOR BETTER INTEGRATIONS
Ofcom is the government approved regulatory and competition authority for the broadcasting, telecommunications and postal industries in the UK. They provide a range of advisory and practical services that allow consumers access to the latest broadband availability data to inform their purchase decisions.
Our work with Ofcom has been focused around the development of highly performant APIs for their organisation. We’ve laid the foundation for a best in class integration approach, extracting and transforming data from a range of service providers.
Their service provides essential information on whether a network provider offers competitive broadband or mobile services at an individual location.
Built using Azure API Management (APIM) and multiple data services, our integration solution exposes a huge dataset of over 3m records with near-instant worldwide response. Because of our cloud expertise and approach, we were able to deliver the initial solution in only 4 weeks.
Integrate and transform faster with cloud-native
Enterprises across the UK are waking up to the fact that selecting a cloud-native approach to their enterprise integration can give them a critical edge over their competitors by supporting faster, more costeffective integrations and transform their digital capability.
Going cloud-native can help you solve the underlying challenges of enabling and accelerating digital transformation, giving you a platform to:
- Move away from on-premises solutions and free yourself from managing infrastructure
- Reduced cost of entry and total cost of ownership
- Save development time and resource
- Improve information security and data governance
- Intelligently manage, report and optimise costs at a per integration level
