Data integration projects can be complex. Most data integration providers have done a lot to simplify the data integration process, but you can go far with a solid plan. How can you create a good data integration project plan?
Let's start by defining what we actually mean by data integration. Data integration means the integration of business systems. The primary purpose of this is to make sure data can flow securely and easily between two different systems, such as an ERP and a CRM. Each system will have its own processes, but will deal with data about the same people or things. Data integration tends to share information about customers, contacts, items, quotes, orders, invoices and sales history.
Other forms of data can also be integrated - there are no limits. You might want to integrate master data, such as posting groups or other metadata that secure a consistent system setup and reporting. Any data that you want to map between two systems can be integrated.
Setting up System integration project Steps
In the past, data integration was often done using direct batch interfaces. This meant that data was extracted to a file and then sent to the target system. At the same time, it was imported (via a batch job). This method was pretty labour intensive, especially at the programming and testing stage. It also had a couple of major faults. It was difficult to work out how to deal with incorrect data, and once the systems were integrated, they had to be kept in sync.
Most businesses now use agile systems such as Microsoft Dynamics and Salesforce. With these kinds of systems, it makes much more sense to use a data integration platform that simply configures and maps data. You need a proper data integration plan with detailed data integration process flows. But it does not have to be over-complicated. In fact, there are three system integration project steps with this kind of systems.
STEP 1: Define your business processes and data integration scope
To integrate, you need to understand what your processes actually are and what kind of integrations you’d benefit from.
For example, usually a customer in your ERP system will have come from your CRM system. They got into the CRM system by being converted by the CRM from a lead.
This would result in this data integration flow:
- Trigger event in the CRM system (such as first order)
- Customer created in ERP system.
Data transfers are usually analyzed to establish which data needs to be transferred, such as contact details or order history. When planning your data integration, you need to define which data you want to flow, and when.
STEP 2: Connect your systems via a data integration platform
The two systems being integrated need to be connected. System designs (that is, tables, fields and other relevant information) are read and stored to allow data to be mapped.
Then, data needs to be sequenced so that transfers happen in the right order. If transferring an order, for example, you will probably want to transfer the items ordered first, as there may be a new item in the order. Not getting the sequencing right can lead to broken references.
You’ll also need to decide which of the two systems being integrated has priority if a piece of data has been updated in both systems.
Once set up, you’ll need to test your data transfer. Generally, it’s best to start the testing process with just a few records. Often, you’ll see that your data isn’t as high-quality as you expected, and you’ll need to do some data cleaning before you move on to testing the rest.
Sometimes, you’ll have data that has been entered manually into both your systems, leading to duplicates. If this is the case, there’s a simple fix: create a batch job or query that can update duplicated data automatically. Alternatively, you can look up duplicates manually and link them with an identifier, but this is a slow process.
STEP 3: Sync your systems
Once testing and transfer is done, it’s time to sync your systems. This is usually best done in stages, making each area live as you go. This means you’ll start benefiting from integration more quickly.
When you’ve finished syncing and everything is working as it should, it’s time to switch to support mode. This is generally easy to manage with a data integration platform, which should send an email notification when something goes wrong, along with all the information they need to start fixing it.
Confused? Give us a call. We’ll talk you through how RapidiOnline can help you create a data integration project plan that does the hard work for you.
