What is the best way to handle data integration errors?
By Henning Lund - September 01, 2016
One of the biggest challenges when you are transferring data from one system to another (for example CRM and ERP) is to have full transparency on the data transfers. This is absolutely essential, especially if something goes wrong. If you have tried to build up your own custom interface between systems, you probably know that one of the most common mistakes is to forget to consider what happens when an error occurs. And when an error happens (because it always does), you might find that you have planned for the system to import data and not to handle a data or connection error. Then what? Your systems might be setup to provide you with some additional information in regards to what went wrong, but it is still up to you to figure out how to solve the problem, and more importantly, it is up to you to figure out how to bring back both systems in sync. Now, it might not be a major problem if you are only syncing a few fields from one table, but if you are transferring all information related to a customer or an order… then the task is somewhat bigger and trickier unless your system does it for you!
Where does the error come from?
First thing is to figure out what your data integration errors are due to. In my experience, the most typical errors either come from poor data connections, bad or missing data in the source system causing validation errors in the target system, or errors can simply be due to a new system update either in the source system or in the target system. It is actually very rare that an error is caused by the integrating system itself.
One of the biggest benefits of working with a data integration platform such as the RapidiOnline data integration solution (as opposed to most custom-built interfaces), is that you get exactly the transparency and control that you need – and you get it out of the box. This allows you to minimize the number of data errors, simply because the required data mappings and migrations between standard fields are included in the integration templates that come out of the box, ready to use. You can also ask a more experienced consultant to provide you with additional insights in regards to where you can benefit from a harmonization of field validation rules. If you go this way, remember that the golden rule is that any required validation happens where data is born. The data integration platform provides a log that contains information about all transfers, whether they were successful, or if they have contained an error. Just having an error message in a system does not help much because the real task is first to get informed about the problem, and then more importantly to figure out how to solve the problem and if some additional action is needed in order to prevent new errors of the same type.
There are different ways to facilitate the handling of data integration errors. For example, the RapidiOnline solution provides an automated error escalation and handling system which in its basic form shoots out an email to the integration administrator(s) when an error has occurred. It contains the exact error message from the system that has created this error and a direct link to the data transfer that contains the error. This makes it very fast to investigate a problem and figure out what is wrong. When you detect a data error, all you need to do is correct the data in the source system and then it is automatically transferred the next time a data transfer for this table is scheduled to run. And, if you want, you can of course always run the transfer again manually to check that the data is correct. Some integration solutions have built up their own error messaging system. In some cases, it means that the error messages that you get are determined by the data integration and not by the actual source or target system. From a programming perspective, it is a very simple way to deal with error handling. But for the end-user, who needs as precise information as possible to solve the problem, it can be a really painful way of working. It is actually to avoid that kind of situations that I prefer utilizing error messages from the source and target systems directly in our data integration solutions. That makes resolving errors much easier!
Last piece of advice: remember that, when you have a setup with direct batch interfaces between two systems, if the batch is not running, you will often not receive any messages and therefore, you will assume that everything is running fine – and you will be wrong. Make sure to choose a data integration solution that monitors the schedules and sends you notifications if they don’t run for a certain amount of time. Then you will never have to worry again about unattended errors in your system integration.
With over 25 years’ experience in strategically propelling businesses forward, Henning is considered a business development entrepreneur with a passion for transforming businesses, sales and marketing operations through out-of-the-box thinking, concepts building and process automation to improve overall performance and scalability.