By Henning Lund - May 19, 2017
More and more, companies are using an array of specialized systems (ERP, CRM, and any other system) to follow and support the digitization, globalization and general decentralization of business processes. This can create silos of data, which would seriously jeopardize the investment in these systems. These applications will only add value to the entire organization if they are aligned and used the same data, i.e. if they are integrated.
This is why, today, one of the most critical things to the success of companies today is the ability to integrate data from multiple sources. Having a robust system integration has become the key to building a strong enterprise architecture. But what is a robust data integration system?
What happens when you integrate two systems? Does the transfer of data from one system to the other expose and potentially endanger the security of your data? How does data integration impact data security? Can data integration make your company more vulnerable to a data security breach? I’ve seen many different solutions to integrations, and to be honest most of them with basically no encryption of data. Often plain .csv text files or XML files are sent by mail or via a FTP server for download and import to the other system(s). These files are also be stored on removable devices such as a usb-stick or memory card, and even though it’s a simple solution, it has a lot of constrains not only in regards to security but also in regards to data error handling when such occur.
Data integration does not systematically imply a potential data security issue, especially these days where integrations can be handled via SSL encrypted data transfers between two systems with no staging of data on an ftp server or any other media. Data is read and written directly to and from the databases via the provided API’s. In reality, most data security breaches are made possible by traditional systems with security technology and security operations that fail to keep attackers outside.
At RapidiOnline, we have had focus on data integration for the past many years and data security has always been top of our minds. The way we enable secure data integration is in reality quite simple. The RapidiOnline Services run with a technology called the RapidiConnector that ensures compressed and secure data transfer between RapidiOnline and different on-premise systems. And of course, to ensure efficiency and security the data is compressed and encrypted.
Now, the clever thing about the RapidiConnector is that it resides in your on-premise network connecting outbound to the central RapidiOnline Service and enabling two-way communication between RapidiOnline and your legacy systems. This has the great advantage that you don’t have to change anything in your network or firewalls in order to deploy RapidiOnline integration. So, the connection from the RapidiConnector is outbound only to RapidiOnline. No inbound ports have to be opened in your firewall. That way, we can make sure that your internal systems stay secured and protected.
The communication between RapidiOnline and the RapidiConnector is using a proprietary protocol which is highly efficient in terms of maximum reduction of the total amount of data and the number of packages sent and received. The protocol first joins e.g. 50 records in 1 package thereby stripping off all meta information to only send the relevant data. Furthermore the package is compressed before sending. This results in the most efficient data transfer. This method can reduce the amount of data needed to be sent by a factor of up to 50. This means e.g. that instead of sending 1000 bytes for each customer record, the RapidiConnector only needs to send 20 bytes. This dramatically reduces both the transfer time and needed bandwidth.
Read also: Should data integration be so hard?
One of the biggest challenges when you are transferring data from one system to another (for example CRM and ERP) is to have full transparency on the data transfers. This is absolutely essential, especially if something goes wrong. If you have tried to build up your own custom interface between systems, you probably know that one of the most common mistakes is to forget to consider what happens when an error occurs. And when an error happens (because it always does), you might find that you have planned for the system to import data and not to handle a data or connection error.
Then what? Your systems might be setup to provide you with some additional information in regards to what went wrong, but it is still up to you to figure out how to solve the problem, and more importantly, it is up to you to figure out how to bring back both systems in sync. Now, it might not be a major problem if you are only syncing a few fields from one table, but if you are transferring all information related to a customer or an order… then the task is somewhat bigger and trickier unless your system does it for you!
Remember that, when you have a setup with direct batch interfaces between two systems, if the batch is not running, you will often not receive any messages and therefore, you will assume that everything is running fine – and you will be wrong. Make sure to choose a data integration solution that monitors the schedules and sends you notifications if they don’t run for a certain amount of time. Then you will never have to worry again about unattended errors in your system integration.
Read also: How data integration makes life easier
These are only some of the criteria that will help you select a robust data integration solution. You can probably add others to the list, but this should help you get started. Rapidi offers robust solutions for data integration and replication and migration - delivered both as on-demand and on premise solutions. With more than 20 years of experience and unique in-depth knowledge in data integration and data replication, we have a strong focus on developing intuitive, flexible and user-friendly interfaces that can be setup in no time and customized to match your specific needs. Get in touch with us or download the below whitepaper to learn more.