One of the biggest challenges of data integration (for example CRM and ERP integration) is to have full transparency on the data transfers. This is absolutely essential, especially if something goes wrong. When building a data integration solution, one of the most common mistakes is to forget to consider what happens when an error occurs. If you have tried to build up your own custom interface between systems, you probably know that already.
When an error happens (and it always does), you might find that you have planned for the system to transfer data but not to handle a data or connection error.
Your systems might be setup to provide you with some light information about what went wrong. However, it is still up to you to figure out how to solve the problem. More importantly, it is up to you to figure out how to bring back both systems in sync.
It might not be a major problem if you are only syncing a few fields from one table. If you are transferring all information related to a customer or an order, the task is somewhat bigger and trickier unless your system does it for you.
The very first step in any data integration error handling is to figure out what your data integration errors are due to. The most typical errors either come from poor data connections. They could also come from bad or missing data in the source system causing validation errors in the target system.
Sometimes, errors can simply be due to a new system update either in the source system or in the target system. It is actually very rare that an error is caused by the integrating system itself.
Read also: 5 keys to successful data integration
Organizations are using a growing number of systems to collect precious customer data. CRM, SCM and ERP systems and data warehousing systems are used to gather and store data. As the data amount increases, we are referring to it as Big Data. This could result in data silos.
To obtain high data quality and have a unified view of their customer data, businesses need to start combining data from different sources. Consolidating data from disparate sources or data warehouses requires data integration tools.
From that moment on, businesses need to consider data storage, data access, data virtualization and data transfer. Data integration systems use different data transfer methods. For example, some use the "ETL" procedure that extracts, transforms and loads data in order to transfer it.
Many businesses use data integration platforms such as the RapidiOnline data integration solution to solve their data integration needs. This gives them the transparency and control they need to handle errors in data transfers.
Data integration platforms provide you with the required data mappings and migrations between standard fields. This allows you to minimize the number of data errors.
A data integration platform like RapidiOnline provides a log that contains information about all transfers, whether successful or not. Many systems simply display an error message in the data integration system when an error occurs. This does not provide you with all the information you need
While it is important to get informed about the problem when it occurs, it is even more critical to know how to solve the problem. Receiving information about the additional action needed to prevent new errors of the same type is vital.
RapidiOnline gives you all the information you need to make sure your system is always running. It helps you identify errors, diagnose then, solve them and prevent them from reoccurring.
Read also: Digital transformation cannot be successful without data integration
There are different ways to facilitate the handling of data integration errors.
The RapidiOnline solution provides an automated error escalation and handling system. It automatically sends out an email to the integration administrator(s) when an error occurs. The message contains the exact error message from the system that has created this error and a direct link to the data transfer that contains the error.
This makes it easy to investigate a problem. When you detect a data transfer error, all can simply correct the data in the source system. It is then automatically transferred the next time a data transfer for this table is scheduled to run. Alternatively, you can run the transfer again manually.
Some integration solutions display their own error messages. These are created by the data integration and not by the actual source or target system. From a programming perspective. It is a simpler way to deal with error handling.
But it does not provide the end-user with precise information solve the problem. Therefore, utilizing error messages from the source and target systems should be the preferred way.
When you have a setup with direct batch interfaces between two systems, if the batch is not running, you will often not receive any messages. Therefore, you will assume that everything is running fine. However, you can never be certain that it is the case. This type of system makes it difficult to have reliable error-free data.
When you select a data integration tool, the most important part is not to know whether it can offer real time data transfers or not. Most systems offer real time transfers. Most often companies do not even need real time integration.
What you really need to make sure is that the solution monitors the schedules and sends you notifications if they don’t run for a certain amount of time. Then you will never have to worry again about solving data integration errors again.
Read also: How do you measure the success of your data integration project?
Henning Lund
Your business is 10 steps away from perfectly integrated data systems. Learn about key preparation, best practise and more in our data integration handbook.
