Bridging the Integration GapBy David A. Kelly
Organizations eliminate data integration bottlenecks and keep information in sync with Oracle data integration solutions.
Cloud computing, the data volume facing most organizations, and the need for real-time data and reporting are all driving a rethink of data integration solutions.
“As organizations move to elastic and highly scalable environments, dynamic data movement for synchronizing or replicating databases becomes very important,” says Carl Olofson, research vice president of database management and data integration software research at International Data Corporation (IDC). “Organizations typically have huge amounts of data in their internal systems that are not aligned and can’t be used easily in a data integration project. That’s why it’s critical for organizations to think about data integration strategically, not project by project.”
That’s just one of the reasons why the old standby of doing data integration internally or creating one-off data integration solutions doesn’t work anymore. According to Olofson, in nearly all cases, internal development of data integration solutions is at the least a bad choice, and it could even be a disastrous choice, because the results can be so limiting. “You won’t be able to extend it, you can’t use it as part of a strategic initiative, and you have to maintain it for all time. It’s going to be an ongoing cost,” says Olofson.
“We’ve expanded the realm of business intelligence beyond just the business analysts who are doing long-term planning to include a wide range of business users who are making tactical decisions,” Olofson says. “And these people are more interested in up-to-the-minute data.”
That’s where Oracle Data Integrator and Oracle GoldenGate—strategic data integration solutions from Oracle—come in.
Oracle Data Integration Solutions
Oracle Data Integrator is a high-performance, batch-oriented data integration solution for heterogeneous data integration. It provides the ability to move data across different systems and transform it as necessary, regardless of whether it’s coming from or going to an Oracle-based system.
According to Alok Pareek, vice president of product management for data integration at Oracle, Oracle Data Integrator performs because it doesn’t use a midtier server like other data integration solutions. “Oracle Data Integrator takes a next-generation data integration approach that extracts the data from the source system, loads it into the target system, and then transforms the data,” says Pareek. “Thus you don’t have the performance cost of midtier server processing, and you get to leverage the computing power of the database or data warehouse.”
Oracle GoldenGate is Oracle’s solution for dynamic data integration needs—where organizations need to have up-to-the-minute information to make business decisions or want new ways of offloading expensive legacy systems. “Oracle GoldenGate provides high-performance data integration with transactional integrity across heterogeneous systems,” says Pareek. “It can scale to very high volumes.”
While Oracle Data Integrator approaches data integration on a table-by-table basis, Oracle GoldenGate is designed to keep transactional data consistent as it moves data from a source to target system. “With Oracle GoldenGate you can have applications that are deployed against your target system that continue to read or analyze data even as information is being updated,” says Pareek.
High-Performance Data Integration
When it comes to cloud-based applications and services, data replication and integration might not be the first things that come to mind. But they are key for John Martin, COO of Denver, Colorado-based IQNavigator.
“Data replication and integration are critical for us as a SaaS [software as a service] provider, especially with the types of processes we automate for our clients,” says Martin.
IQNavigator is a leading provider of services procurement and extended workforce solutions. Global 2,000 companies use IQNavigator to help clients manage workforce and services procurement in real-time, through IQNavigator’s SaaS applications. “Our solution handles the sourcing of services, from identifying the right supplier to negotiating and recording the contract to payments for services,” says Martin. “We can even track the workers onsite that are delivering those services and reconcile them against the contracts.”
IQNavigator has to undertake an average of 10 different business-critical integrations with each client’s internal systems. From master data transactions to personnel information to time cards, it all needs to be integrated in real time with IQNavigator’s applications.
IQNavigator is now using two Exadata Database Machine X2-2 systems from Oracle as the foundation of its next-generation online transaction processing (OLTP) applications and real-time reporting services. From enabling IQNavigator to migrate to Oracle Exadata without downtime in a phased model to synchronization in the Oracle Exadata production environment with standby systems, Oracle GoldenGate is IQNavigator’s key to ensuring accurate and consistent data.
IQNavigator decided on the combination of Oracle Exadata and Oracle GoldenGate because with more than 3 million users—from many Global 2000 companies—spread across 113 countries and thousands of suppliers, it needed a high-performance, integrated system.
“We need to be able to process financial transactions with 100 percent transactional integrity while processing close to US$10 billion annually,” says Martin. “The scalability that’s inherent in Oracle Database and the Oracle Exadata platform is a huge benefit for our clients.”
With the new architecture based on Oracle Exadata and Oracle GoldenGate, IQNavigator expects to be able to improve system throughput and increase performance of complex queries by an average of 10 times. An added benefit is the new real-time reporting capabilities from IQNavigator’s transactional system, which will enable the company to deliver a whole new class of real-time services to its customers.
“For high-volume global clients, real-time reporting is extremely important,” says Martin. “Gaining faster access to real-time data opens up the door to adding new types of services and capabilities for our clients.”
Real-Time Data for Real-Time Business
Few industries move faster than telecommunications, especially when it comes to mobile services. Staying competitive in that industry requires an IT and business information platform that can deliver near-real-time data to business analysts, marketing, sales, and customer support. Take the case of Avea, a leading provider of communications services to more than 12 million customers in Turkey.
“Rapid replication is critical in some areas, such as our mobile number portability systems,” says Özlem Topçakan, business intelligence manager at Avea. “If there’s any latency, we need to be able to take immediate action.”
As with most mobile phone companies, new customer acquisition is critical. But servicing new customers effectively requires business users and managers that have the most-accurate and -up-to-date information available.
Avea needed to create a new data warehouse with a high-performance daily extract, transform, and load (ETL) process and the ability to provide high-performance user queries. One of the challenges it faced was an ETL processing window that was increasing day by day. It needed a new architecture that would meet expanding business requirements.
The company redesigned its systems so that production systems were replicated using Oracle Data Integrator to an operational data store (ODS), which was then used to feed the data warehouse running on Oracle Exadata. In addition, the ODS became the primary source for online operational reports, which avoids putting a query load on the production systems.
Avea developed an innovative solution that uses Oracle GoldenGate and Oracle Data Integrator to feed its data warehouse. With the new solution, each day 500 million rows of data are moved from source systems into the data warehouse.
Avea uses Oracle Data Integrator for its ETL operations, and it uses Oracle GoldenGate to replicate data to its ODS, which allows Avea’s business users to access data in near-real time.
“We manage all our ETL windows within Oracle Data Integrator. We use Oracle GoldenGate for replicating the data from production systems to the ODS,” says Topçakan. “The ODS is used for online reports using interactive reporting tools.”
Topçakan further explains that Avea automatically manages latency on its operational data store environment through Oracle Data Integrator. At the start of the ETL process, Oracle Data Integrator checks the Oracle GoldenGate logs for latency and then automatically switches source systems if needed.
“Most of the time we replicate from the production systems to the ODS in just three seconds, so business users can reach the data extremely quickly,” says Topçakan.
When Avea moved its data warehouse to Oracle Exadata from IBM p5 servers, the ETL performance improved by four times. Prior to migration, the batch processing window had been 12 hours, and now on Oracle Exadata it’s down to 3.5 hours. At the same time, the performance improved by an average of 10 times.
Now that Avea has an operational data store, it can refresh its data warehouse multiple times a day, if needed. “That’s a prerequisite for a near-real-time data warehouse, so it’s a big plus for us,” says Topçakan.
Prior to implementing Oracle Data Integrator, Avea used a combination of tools and utilities to do its data movement. “Before Oracle Data Integrator, we had to look at separate UNIX jobs, individual ETL tools, and different kinds of systems,” says Topçakan. “Now, we can manage all our data integration through Oracle Data Integrator.”
Using Oracle Data Integrator to manage data integration has also simplified the development and management process for Avea. “Oracle Data Integrator is a very easy tool for development,” says Topçakan. “It’s easy to learn, easy to develop with, and easy to share information with other team members.”
Strategic Data Integration
When it comes to data integration, doing nothing isn’t a viable strategy. For most organizations, simply addressing the continuous growth of data volumes will require a re-evaluation of data strategy regardless of the effectiveness of an organization’s current solution.
That’s why now is a good time for organizations to take a step beyond viewing data integration as a utility and recognizing it as something more.
“It’s important to remember that data integration isn’t just plumbing,” says IDC’s Olofson. “Data integration is really the underlying technology for a more mature and unified enterprise information management environment. Everyone needs to be thinking about data integration strategically.”
David A. Kelly (davidakelly.com) is a business, technology, and travel writer who lives in West Newton, Massachusetts.Send us your comments