Integration Is Not a Connection, It Is the City's Transport Plan

When I was a child, my favourite games were nearly always management and simulation titles. In SimCity I built cities, ran them as mayor and tried to keep the people living in them as happy as I possibly could. SimCity may well be the game I spent the most hours on as a boy. That is probably why Dinosaur Polo Club's Mini Motorways and Mini Metro have caught my attention again lately. What I noticed, though, is that I now play these two games with a very different eye. I always say it: once you get involved with CRM — once you build software in this field or deliver solutions to organisations — the way you look at life changes. From the way you plan your day to the way you tidy a cupboard at home, you start to see processes, relationships, gaps and room for improvement everywhere. When I met Siebel Systems in 2001 and delivered my first project, and then when Microsoft CRM 1.0 — what we know today as Microsoft Dynamics 365 — entered my life in 2003, I began to understand much better what it means to build a holistic solution for an organisation. I saw that a corporate structure, just like a city, is made up of different districts, processes, people and the roads that connect those pieces to each other. Playing city-building and city-management games today, I can see something else more clearly: in these games, the decisions that make the city grow also make it steadily more complex. Every new road, connection or fix that looks practical at first can turn into the system's biggest bottleneck later, if it is put in place without accounting for growth. The same thing happens in enterprise projects built with the same mindset. That is why I wanted to describe the resemblance between the cities we build in games and the data architectures and integrations we create in enterprise projects. Because whether you are building a city or a CDP and CRM ecosystem, every step you take today directly determines the traffic, the growth and the management capacity you will have tomorrow.

How do you build an architecture that meets today's needs while managing tomorrow's data traffic in CDP and CRM projects?

While a city's population is still small, laying roads wherever you like is easy and fun. With a few neighbourhoods, a limited number of vehicles and low traffic, almost any road works. When two districts need a connection, a new one is opened; when a third district grows, it gets a road of its own. In the short term the problem looks solved. Once the city starts to grow, the picture changes. New residential areas appear, the population rises, business districts multiply and people's movement habits shift. Roads that used to look adequate become bottlenecks. Roadworks in one district affect traffic across the whole city. The absence of alternative routes, badly designed central junctions and traffic lights, and a public transport plan drawn up too late all make growth expensive and painful. Integration projects go through something very similar. Setting up the data flow between ERP and CRM, getting forms from the website to the sales team or attaching contact-centre records to the customer profile can look like nothing more than a few technical connections at the start. But as the organisation grows, those connections become the company's data transport system. This is why integration is not simply a matter of “making two applications talk”. Integration architecture is a long-term city plan that determines which system data leaves from, which routes it travels, where it is enriched, who will use it and how it will be governed in a growing organisation. In CDP and CRM projects in particular, drawing up that plan at the very beginning is critical.

Every district on the map is a system, every road a data flow

Let us look at the map you see in Mini Motorways not merely as a city map, but as a representation of an enterprise technology ecosystem. The different districts on the map can stand for applications such as CRM, ERP, e-commerce, the mobile app, the contact centre, the loyalty system, marketing automation, service management and the data warehouse. The small markers are customers, transactions or touchpoints; the roads are the data flows between these systems. Let the pink arterial roads represent high-volume, critical integrations, and the side streets the lower-intensity operational connections. Bridges and junctions are the central architectural components: API management, the integration platform, the event streaming layer or the common data model. The analogy shows us something important: a road that works does not mean the transport system has been designed correctly. In the same way, the fact that an integration is carrying data today does not show that the architecture is ready for the future. The real question is this: will this structure still be manageable when the number of applications goes from five to twenty-five, the number of customers from a hundred thousand to ten million, and the number of users from fifty to two thousand? Integration architecture has to answer that question before the project begins.

The most common mistake: opening a new road for every new need

In most organisations, integrations are built as needs come up. The website is connected to CRM. Then customer data is transferred from ERP to CRM. Later the mobile app starts pulling data from CRM. When a marketing platform is added, two new connections appear between it and CRM. After a while the data warehouse, the contact centre and customer service applications join the same network. In a small setup with two or three applications this model can deliver quick results. But as the number of applications rises, the number of connections grows exponentially. In an environment where ten systems exchange data with each other in various directions, it is not only the applications that become interdependent: data formats, identity matching, error handling, security rules and business processes do too. What the organisation ends up with is a tangled road network on the map: - The same customer is held under different identities in different systems. - Changing one field affects several integrations. - It becomes unclear which system a piece of data came from. - Finding where faulty records originate gets harder. - When one system stops temporarily, problems cascade. - Bringing a new application live can take months. - Knowledge of the integrations stays tied to particular people or suppliers.

At this stage the problem is not the quality of the individual roads. The problem is that the roads were built without a shared transport policy — in other words, without thinking about the future. That future begins with anticipating growth and estimating how much the number of vehicles using a given road may increase. The consultants you work with, the city planners in this picture, matter enormously here — that hardly needs saying. Working with expert and creative consultants is what takes you to a structure that lasts.

CRM and CDP play different roles in the same city

To build the right architecture you have to separate the jobs CRM and CDP do. CRM is the operational memory of the relationship you have with the customer. Sales opportunities, quotes, meetings, service requests, tasks and the actions of account managers are all managed there. A CRM solution such as Microsoft Dynamics 365 can sit at the centre of sales, customer service and relationship management processes. A CDP, on the other hand, aims to build a unified view of the customer by bringing together the traces they leave across different channels. Web behaviour, mobile app activity, e-commerce transactions, campaign responses, loyalty movements and other contact data can be used inside the CDP for identity resolution, segmentation and activation. Put simply: - CRM answers the question “what relationship are we running with this customer?” - CDP answers the question “how well do we know this customer across every touchpoint?” These two structures are not alternatives to each other. Designed properly, they feed one another. The CDP can pass the customer's behaviour and preference signals to CRM. The sales, service or agent interactions that happen in CRM can in turn enrich the customer profile in the CDP. For that relationship to be healthy, though, you have to decide in advance which data will be produced in which system and which system is the authority for which data domain. Otherwise the argument “which is the correct customer record?” starts up between CRM, CDP, ERP and the marketing platform.

The first decision is not technology, it is data ownership

The starting point of integration architecture is not the tool you will use. The first decision is who owns the data. Which system manages the customer's legal entity name? Where is the master record for communication consent held? Is the current address taken from CRM or from ERP? Which application owns the loyalty points? How is a customer's unique identity created? How do you tell that records created in different channels belong to the same person? Integrations built before these questions are answered merely carry the uncertainty from one system to another. For every significant data set there should be a master record or master data source — a single source of truth. Other systems may use, enrich or copy that data for their own operational needs. But which one is the master must be beyond dispute. For example: - The master source for financial customer and invoice data can be ERP. - The master source for sales opportunities and account manager activity can be Microsoft Dynamics 365. - Behavioural events and the unified profile can be managed in the CDP. - Communication consents, depending on regulation and process design, can be held in a dedicated consent management system. - Customer engagement or experience layers such as 1Page Platform can present data from different sources to the user in a single context. This distribution does not have to be identical in every organisation. What matters is that the roles and the data architecture have been decided.

Architecture should not be built for today's data volume alone

An integration that carries a few thousand records a day at the start may have to handle millions of behavioural events two years later. That is why architectural design should assess not only the current data volume but the direction data growth is taking. The questions below should be answered with forecasts covering at least three and preferably five years: - How much can the total number of customers and prospects grow? - What level can the daily number of transactions and events reach? - How many events per second may arrive from web, mobile and physical channels? - How much historical data will be retained? - How quickly does data need to be updated? - Which processes can run in real time and which in batch? - During campaigns or peak seasons, how many times normal volume can be reached? - How many different sources will feed a single customer profile? It is worth stressing here that real-time integration is not the right answer in every case. A sales representative may need to see the customer's latest service request immediately. Monthly financial classification data, by contrast, may be fine transferred by a batch process running overnight. Trying to move every piece of data in real time increases cost and complexity. Moving everything in batches slows the customer experience down. The right architecture matches the need for speed to the business value of the data.

One of the most valuable project experiences I can offer as an example here is the Starbucks project: reaching more than 7 million customers in seven years, being able to complete a transaction in milliseconds on the channels the customer is using, and dissolving the queue that builds at the till with the right actions. Over the time we worked on that project, all of it was, I think, proof that we had drawn up a sound city plan.

The variety of applications matters as much as their number

Estimating how many applications will be in use in the future is not enough on its own. Where those applications will run and who will manage them has to be planned too. Over time the enterprise ecosystem can turn into a mixture of: - cloud-based enterprise applications, - legacy systems running on-premise, - SaaS solutions bought by individual business units, - partner and dealer platforms, - mobile applications, - custom-built microservices, - data warehouses and analytics environments, - artificial intelligence services, - physical store and field systems. As this spread grows, it becomes harder not just to connect things but to keep those connections secure and observable. Which service is accessing which data? How is authorisation handled? When a customer asks to be deleted or anonymised, which systems does the data have to be removed from? How will API usage limits be managed? When a data flow stops, who will notice? Integration architecture therefore has to cover API management, authentication, authorisation, encryption, logging, error monitoring and reprocessing from the outset. Security is not a barrier added afterwards; it is the traffic rules designed while the roads are being built.

Growth in departments and users is part of the architecture

A CRM or CDP project does not only grow in data and system scale. The organisation grows too. Where CRM is used at the start by a sales team of ten, customer service, marketing, field operations, finance, management, dealers and business partners may all join over time. A setup for fifty users can reach hundreds, even thousands. Every department looks at the customer through a different window: - the sales team wants to see opportunities and commercial potential, - the marketing team segments and campaign responses, - customer service requests and satisfaction, - the finance team risk and payment status, - operations teams the delivery or service process. These needs rest on a shared view of the customer; but not everyone needs access to all of the data. The future number of departments and users therefore directly shapes the role model, access rights, screen design, reporting structure and licence planning. Integration architecture must deliver data not only to the right system, but to the right user with the right permissions and the right context.

What are the basic principles of a solid integration architecture?

Preparing for the future does not mean building the biggest and most expensive infrastructure on day one. But the decisions that stop the architecture from having to be replaced wholesale once growth arrives do have to be taken early. A few basic principles help here. Technological coupling A change in one system should not directly affect all the others. Applications should communicate as far as possible through defined APIs, events or shared integration services. A common data dictionary Concepts such as “customer”, “active customer”, “prospect”, “order” and “interaction” need shared definitions. The same field meaning different things in different departments has to be prevented. Service/API-first design When a new capability is built, the current screen or application should not be the only thing in mind. How that capability will be used by other applications in the future has to be planned as well. Event-based communication Events such as a customer placing an order, updating their profile or joining a campaign can be pushed to the relevant systems in a controlled way. That saves every application from constantly polling the source system. Observability Whether an integration is working should not be something only the technical team checks by hand. Delays, errors, failed records and changes in volume must all be visible. Taking an end-of-day view and running health checks at set intervals will also increase the reliability of the system. Replayability When a system is temporarily unreachable, data must not be lost. Failed operations should be stored safely and reprocessed once the problem is resolved. Queueing structures — and offline capability, especially for mobile applications — become critical here. Security and privacy Data access rights, masking, encryption, retention periods and deletion processes should be natural parts of the architecture.

How should an integration roadmap be prepared?

The architectural vision can be long term; the implementation plan should proceed in stages. Step one: draw the map The first stage identifies the system inventory, the data sources, the existing connections and the critical business processes. Which system owns which data is made explicit. By the end of this period the organisation should hold at least the following: - a map of current and target systems, - the critical data flows, - a data ownership matrix, - a common customer identity approach, - security and authorisation principles, - volume and performance forecasts, - a prioritised list of integrations. Step two: build the backbone The second stage sets the API and integration standards. Identity matching, error handling, monitoring and logging mechanisms go live. A handful of flows with the highest business value are implemented end to end. The aim is not to connect every system at once, but to create the reliable backbone that later connections will build on. Step three: widen the CDP and CRM use cases The customer profile is enriched; sales, service and marketing scenarios are brought into the system. Department-level views and permissions are matured. Step four and beyond: scale and model As data volume, application count and the user base grow, capacity tests, cost optimisation and architectural reviews become routine. An integration guideline model is operated so that new integrations meet the same standards. A roadmap should be treated not as a fixed project plan but as a city plan that is updated as the organisation grows.

Do not measure success by the number of connections

The success of an integration programme cannot be measured by how many systems have been connected. A large number of connections can just as easily indicate uncontrolled growth as good architecture. More meaningful measures are these: - the time it takes to connect a new application to the ecosystem, - the rate of faulty or lost records, - the delay in updating the customer profile, - the proportion of customers that cannot be deduplicated, - the need for manual data correction, - the time it takes to detect an integration outage, - the time it takes for new departments to join the system, - how clearly the ownership and source of data fields is documented, - the number of connections affected by a change in one system. A healthy architecture makes growth invisible. If complexity is not rising at the same rate as the number of applications, users and records, you have built on the right foundation.

In conclusion: tomorrow's traffic is hidden in today's architectural decisions

CDP and CRM projects are most often kicked off around screens, features and short-term integration lists. Yet the thing that determines the long-term success of these projects is the data transport system behind them. An organisation connecting only a handful of applications today can, within a few years, become a structure that takes data from dozens of applications, processes millions of customer events and serves thousands of users across different departments. If that possibility is not foreseen at the start of the project, every new connection adds both investment and operational load. Foreseen and planned properly, every new application becomes a natural plug-and-play part of the existing architecture.