More and more, companies are using an array of specialized systems (ERP, CRM, and any other system) to follow and support the digitization, globalization and general decentralization of business processes. This can create silos of data, which would seriously jeopardize the investment in these systems. These applications will only add value to the entire organization if they are aligned and used the same data, i.e. if they are integrated.
This is why, today, one of the most critical things to the success of companies today is the ability to integrate data from multiple sources. Having a robust system integration has become the key to building a strong enterprise architecture. But what is a robust data integration system?
1. Your data integration solution must be future proof
- I T IS SCALABLE: In order to be the robust foundation your company will strive and grow upon, it needs to be able to scale. Is your solution be able to support your company as it grows? Will it still be able to perform if the number of transfers between systems grows exponentially? Can you add new transfers to support new business processes? Often, custom-made integrations are a perfect fit for your current situation, but are rarely considering future growth scenarios. Pre-configured data integration platforms such as RapidiOnline are usually built to support the needs of small companies as well as large enterprises, and will typically be a reliable platform as your business booms.
- IT IS EASY TO UPGRADE: Just because your integration solution works with the current versions of the integrated system, it does not mean that it will keep functioning as new versions of the integrated systems are released. If you are building your own custom-made integration, your solution is at risk to encounter problems next time one of the integrated systems is updated or comes up with a new version… and fixing this could be a difficult, time-consuming and costly task – also because it will disrupt your business. Pre-configured data integration platforms vendors, on the other hand, usually work in close collaboration with the companies producing the systems they integrate. For example, as a Microsoft Dynamics partner and a Salesforce partner, Rapidi has a close collaboration with both Microsoft and Salesforce to make sure that we are always informed in due time about changes that are critical to our integration solution and can guarantee a system that works with future versions of the systems as well.
- IT CAN TAKE ON ADDITIONAL INTEGRATIONS: Many of the companies that have integrated their CRM and ERP will find themselves investing in more specialized applications that they would benefit greatly in integrating with wither their ERP or their CRM. And when you come to that point, your first question will be “can I use my integration platform to integrate with this new application”? Now, if you are using a custom-made integration, the chance that it was made generic enough to integrate with any new application is very slim. Pre-configured data integration platforms, on the other hand, are built to support a large variety of systems and integration end-points. Take RapidiOnline, for example. Although it comes out of the box with a standardized Salesforce-Microsoft Dynamics integration, it is also a strong platform that can support pretty much any other system. This is highly valued by our customers. A good example of that is Berling Media. They started by integrating Salesforce with Microsoft Dynamics AX using RapidiOnline. A few years later, they came back to us to integrate with Publit, a print-on-demand service. No doubt that it was a great benefit for them to have a future-proof integration platform.
Read also: 3 easy steps to plan your data integration project
2. Your data integration solution should not put your data at risk
What happens when you integrate two systems? Does the transfer of data from one system to the other expose and potentially endanger the security of your data? How does data integration impact data security? Can data integration make your company more vulnerable to a data security breach? I’ve seen many different solutions to integrations, and to be honest most of them with basically no encryption of data. Often plain .csv text files or XML files are sent by mail or via a FTP server for download and import to the other system(s). These files are also be stored on removable devices such as a usb-stick or memory card, and even though it’s a simple solution, it has a lot of constrains not only in regards to security but also in regards to data error handling when such occur.
What is a robust and secure system?
Data integration does not systematically imply a potential data security issue, especially these days where integrations can be handled via SSL encrypted data transfers between two systems with no staging of data on an ftp server or any other media. Data is read and written directly to and from the databases via the provided API’s. In reality, most data security breaches are made possible by traditional systems with security technology and security operations that fail to keep attackers outside.
At RapidiOnline, we have had focus on data integration for the past many years and data security has always been top of our minds. The way we enable secure data integration is in reality quite simple. The RapidiOnline Services run with a technology called the RapidiConnector that ensures compressed and secure data transfer between RapidiOnline and different on-premise systems. And of course, to ensure efficiency and security the data is compressed and encrypted.
The RapidiConnector
Now, the clever thing about the RapidiConnector is that it resides in your on-premise network connecting outbound to the central RapidiOnline Service and enabling two-way communication between RapidiOnline and your legacy systems. This has the great advantage that you don’t have to change anything in your network or firewalls in order to deploy RapidiOnline integration. So, the connection from the RapidiConnector is outbound only to RapidiOnline. No inbound ports have to be opened in your firewall. That way, we can make sure that your internal systems stay secured and protected.
The communication between RapidiOnline and the RapidiConnector is using a proprietary protocol which is highly efficient in terms of maximum reduction of the total amount of data and the number of packages sent and received. The protocol first joins e.g. 50 records in 1 package thereby stripping off all meta information to only send the relevant data. Furthermore the package is compressed before sending. This results in the most efficient data transfer. This method can reduce the amount of data needed to be sent by a factor of up to 50. This means e.g. that instead of sending 1000 bytes for each customer record, the RapidiConnector only needs to send 20 bytes. This dramatically reduces both the transfer time and needed bandwidth.
Read also: Should data integration be so hard?
3. Your data integration solution should let you easily solve data integration errors
One of the biggest challenges when you are transferring data from one system to another (for example CRM and ERP) is to have full transparency on the data transfers. This is absolutely essential, especially if something goes wrong. If you have tried to build up your own custom interface between systems, you probably know that one of the most common mistakes is to forget to consider what happens when an error occurs. And when an error happens (because it always does), you might find that you have planned for the system to import data and not to handle a data or connection error.
Then what? Your systems might be setup to provide you with some additional information in regards to what went wrong, but it is still up to you to figure out how to solve the problem, and more importantly, it is up to you to figure out how to bring back both systems in sync. Now, it might not be a major problem if you are only syncing a few fields from one table, but if you are transferring all information related to a customer or an order… then the task is somewhat bigger and trickier unless your system does it for you!
- Start by finding out where the error comes from
First thing is to figure out what your data integration errors are due to. In my experience, the most typical errors either come from poor data connections, bad or missing data in the source system causing validation errors in the target system, or errors can simply be due to a new system update either in the source system or in the target system. It is actually very rare that an error is caused by the integrating system itself.
- Error handling with a data integration platform
One of the biggest benefits of working with a data integration platform such as the RapidiOnline data integration solution (as opposed to most custom-built interfaces), is that you get exactly the transparency and control that you need – and you get it out of the box. This allows you to minimize the number of data errors, simply because the required data mappings and migrations between standard fields are included in the integration templates that come out of the box, ready to use.
You can also ask a more experienced consultant to provide you with additional insights in regards to where you can benefit from a harmonization of field validation rules. If you go this way, remember that the golden rule is that any required validation happens where data is born. The data integration platform provides a log that contains information about all transfers, whether they were successful, or if they have contained an error. Just having an error message in a system does not help much because the real task is first to get informed about the problem, and then more importantly to figure out how to solve the problem and if some additional action is needed in order to prevent new errors of the same type.
- Make the error handling process easier
There are different ways to facilitate the handling of data integration errors. For example, the RapidiOnline solution provides an automated error escalation and handling system which in its basic form shoots out an email to the integration administrator(s) when an error has occurred. It contains the exact error message from the system that has created this error and a direct link to the data transfer that contains the error. This makes it very fast to investigate a problem and figure out what is wrong. When you detect a data error, all you need to do is correct the data in the source system and then it is automatically transferred the next time a data transfer for this table is scheduled to run. And, if you want, you can of course always run the transfer again manually to check that the data is correct.
Some integration solutions have built up their own error messaging system. In some cases, it means that the error messages that you get are determined by the data integration and not by the actual source or target system. From a programming perspective, it is a very simple way to deal with error handling. But for the end-user, who needs as precise information as possible to solve the problem, it can be a really painful way of working. It is actually to avoid that kind of situations that I prefer utilizing error messages from the source and target systems directly in our data integration solutions. That makes resolving errors much easier!
Remember that, when you have a setup with direct batch interfaces between two systems, if the batch is not running, you will often not receive any messages and therefore, you will assume that everything is running fine – and you will be wrong. Make sure to choose a data integration solution that monitors the schedules and sends you notifications if they don’t run for a certain amount of time. Then you will never have to worry again about unattended errors in your system integration.
Read also: How data integration makes life easier
Learn more about our robust data integration solutions
These are only some of the criteria that will help you select a robust data integration solution. You can probably add others to the list, but this should help you get started. Rapidi offers robust solutions for data integration and replication and migration - delivered both as on-demand and on premise solutions. With more than 20 years of experience and unique in-depth knowledge in data integration and data replication, we have a strong focus on developing intuitive, flexible and user-friendly interfaces that can be setup in no time and customized to match your specific needs. Get in touch with us or download the below whitepaper to learn more.
