Real-time data integration is being increasingly used by companies to improve their business and better serve their customers. Traditional batch systems are being converted to real-time systems that process transactions as they occur. The advantages include improved knowledge of how business is performing and offering services that outperform competitors. An architecture with real-time information and on-demand processing can have a huge payback for certain business scenarios.

Batch Processing

In batch processing, a large group of transactions are collected and the data is processed by a program during a single run. With the large volume of work to do, the process must run when resources are less busy, usually overnight. Programmers have traditionally written programs for batch processing, so it's well understood and easy to implement. But it postpones access to data, requires close supervision, and risks down time. You miss out on the knowledge of your business until after it's happened. Because any problem can delay the entire process, you need support personnel to closely monitor while it runs. And if there's a problem, you risk even further delay of data and some systems may not operate correctly for users.

Real-Time Processing

In real-time processing, small groups of transactions are processed on demand. Instead of a single run, there is data that arrives continuously to be processed by a program. The load of processing data is spread across the day when transactions occur. The advantage of real-time processing is that it increases access to data, runs with fewer resources, and improves uptime. With continuous integration of data, you have knowledge of your business as transactions occur. With small groups of transactions to process, it can be run periodically during the day without impacting interactive users. If an error occurs, it can be handled immediately by staff who are already on site, and the system has a larger window of time to recover. It is more difficult to design real-time processing and it includes some additional overhead, so not every system should be architected as real-time.

When to Use Real-Time

While systems with real-time processing require more effort to design and implement, the benefit to your business can be huge. Let's look at the areas where real-time processing has the biggest impact.

Core to Business

Data that is critical to business operations should be real-time because it is your most valuable data. You want to gain every advantage over competitors at your core. For instance, a retailer should have immediate access at central office to their sales data from stores. Once the core data is real-time, other systems can make use of it and new systems become possible. For example, the retailer runs a promotion and watches sales performance as customers respond to the incentive. The response can be analyzed in real-time and the promotion can be adjusted instantly to maximize sales. Inventory systems can be updated real-time, making it possible to check inventory by location from the call center or website. Loyalty customers can have receipts emailed to them immediately after purchase. New aspects of your business open up that weren't there before.

Research shows the trend is towards real-time data. In a study of 21 large companies across a range of industries, including banking, finance, and retail, research firm Market Clarity found that most of the companies had moved to delivering real-time information feeds. The report found that applications requiring real-time access to data were mainly core to their business.

Supporting Customers Online

Customer self-service and ecommerce websites should be real-time because it's the best way to service customers who've come to expect it. Gone are the days of waiting to check your balance or see a transaction the next day. Now, you get an alert when a spouse spends too much at the department store. You place an order and receive email updates within minutes as the status changes from received to payment authorized to shipped. A website with real-time data has an advantage that attracts customers in search of instant gratification. Batch processing doesn't make as much sense on the web where access is available all the time from anywhere.

Enabling Business

We've seen that real-time integration keeps your business competitive and enables new kinds of services. Most companies are incorporating real-time integration into their core systems where it has the most impact. On the web, customers expect self-service and ecommerce sites to have interaction with real-time data. As IT managers look at strategies for real-time data integration, they will consider data replication, messaging, web services, and event-driven architectures. As competition increases and customers expect more, we'll see these strategies deployed more often.

Eric Long
Author: Eric Long

Eric is a software developer and technology enthusiast with a background in developing custom applications for Information Technology. As a long-time Linux user, he strongly believes in open source. He focuses on product solutions and spreading the word for JumpMind.