In reality, data integration is just another word for integrating systems. The main task of data integration is to secure a flow of data between different systems (for example an ERP system and a CRM system), each system dealing with the data with whatever business logic that is built into them. Very often, data integration transfers operational data from one system to the other, for example: Customers, Contacts, Items, Quotes, Orders, Invoices or Sales history.
But the scope of data integration can also include other types of data, for example either traditional master data (such as posting groups and other metadata that secure a consistent system setup and reporting) or it can basically be any table or field that you choose to map between the different systems. In reality, there are no limits to which data that can be relevant for data integration, it only depends on your system landscape and the business processes you are following.
How do you setup a data integration project?
Some years back, it was quite common to handle data integration by building direct batch interfaces where data was extracted to a file that was sent to the target system and imported also via a batch job. It required a lot more work upfront, both in terms of programming and in testing. Some of the pitfalls of that method were that 1) you had to figure out how to deal with incorrect data and 2) you had to be able to keep the systems in sync afterwards. If the systems remain untouched, then this could run for years. Now, when it comes to more agile systems like ERP (for example Microsoft Dynamics) and CRM (for example Salesforce) systems, it is more cost-efficient and a lot more robust to use a data integration platform where most of the work is reduced to simple a configuration and mapping of data.
Start by defining business processes and data integration scope
Firstly, you have to get your business processes in place and capture the requirements to build the scope. Now, there can be a lot of thinking around this, but often companies can benefit from best practice workflows so that they don’t have to build from scratch. For instance, it is logical that a customer in your ERP system comes from a lead in your CRM system that was converted into a customer. In this case, the data integration flow would be as follows: a customer gets created in the ERP system at a triggered event in the CRM system, for example the first time they order something. Most often, the data transfers are analyzed entity by entity for let’s say customers, contacts, items, etc... Therefore, in the planning you need to define which data is needed to flow between the systems and when.
Then Connect the systems via your data integration platform
Secondly, you have to connect the 2 systems that you want to integrate via the data integration platform: The system designs (meaning tables, fields, and other relevant information for the integration) are read and stored in order for the users to do the mappings correctly and with the right information. After that, the data has to be sequenced the right way in the data transfers. For example, if you are transferring an order, it makes sense to transfer items first because there might be a new item on the order. Without that item there would be a broken reference. It is also during this phase that you decide technically which system that has priority over the other (s) for bi-directional tables in case a piece of data has been updated in both systems (master and slave logic). When the data transfer is setup, you will start testing it, typically starting with a few records, until everything is ready. This is often the time where you realize that your existing data is not precisely as good and consistent as you thought it was, which drives a need for some data cleaning. You can also be in a situation where data has been created manually in both systems (for example, customers without a clear reference), which means that if you just started to integrate the systems you would get duplicates. This, of course, is not what you want. To prevent that, you will either have to make a batch job or a query that can update this automatically, or you will have to manually look them up and add an identifier that clearly links the two records together. It is a very simple thing to do, but if you have a lot of customers it can be time consuming to fix.
Finally, Sync the systems
Thirdly, when everything is tested, and initial data transfer is made, you have to bring the two systems in sync. You don’t have to do this as a big bang, you can take one logical area at a time and bring it live before moving on to the next. This way, your end users will be able to realize the benefit of the data integration faster, which is good for user adoption generally.
When everything is completed, you can start relaxing and switch to support mode, which is a very smooth process on a data integration platform environment because if something for whatever reason is not working as it should, the people in charge will receive an email notification with the relevant information and a direct link to the data transfer that has an error, for investigation and resolving. While these few steps might have appeared overwhelming and somewhat scary some years back, the task can be today a lot simpler and smoother as you can use data integration platforms (some even with pre-configured to pre-defined end-points) to ease your data integration project. Many of our customers are using the RapidiOnline integration platform to quickly and easily connect their Salesforce and Microsoft Dynamics systems.
About the author
With over 25 years’ experience in strategically propelling businesses forward, Henning is considered a business development entrepreneur with a passion for transforming businesses, sales and marketing operations through out-of-the-box thinking, concepts building and process automation to improve overall performance and scalability.