DOWNLOAD Oracle Fusion Middleware

TAGS Business Intelligence, BPEL, ESB, Data Integration, SOA, SOA Integration, All Architect Articles

SOA Integration in the Legacy Environment

by Tom Laszewski and Jason Williamson

Organizations with legacy applications don't have to throw the baby out with the bathwater when undertaking new initiatives.

Published October 2008

Introduction

There is an increasing drive for organizations to move away from monolithic, static applications to Service-oriented Architecture (SOA) and the promise of agile, cost-effective, future-proof services. That is certainly the case among the many Fortune 500 companies with which the authors are involved in legacy modernization efforts. Monolithic systems form the core of nearly all of these companies.

But while services and the notions of agility and portability form the basis of our discussions of new application development initiatives with these organizations, legacy applications are the elephant in the room. What does SOA mean for organizations that have applications that are years, even decades old, applications that are maintained with a dwindling workforce and costly to change? These are issues that every IT manager must deal with.

The simple fact is that these monolithic systems are at the heart of banking, transportation, and manufacturing sectors around the world. But organizations do not have to throw the baby out with the bathwater when it comes to new initiatives. This article provides several examples that illustrate how Oracle technologies can be used to integrate legacy environments.

SOA in Context

The term SOA has been used, perhaps overused, to the point that it has lost its currency among decision makers. So before we dive into our use cases, let's take a moment to define what SOA means in terms of the legacy mainframe environment and legacy modernization.

Think of SOA as a framework for building applications, rather than as a specific set of technologies or protocols. SOA is a way of providing computing resources by exposing discrete business processes for use by consumers. More than a set of protocols and technologies—such as SOAP and WSDL or Java/RMI—SOA is a blueprint for building applications. Web services are one option for implementing this framework. Other technologies that can be leveraged to employ a service-oriented architecture include Business Activity Monitoring (BAM), Business Process Execution Language (BPEL), Enterprise Service Bus (ESB), and Business Process Management (BPM) engines. Technically, FTP is a SOA interface!

Integrating SOA into a Legacy Framework

The fundamental idea of enabling a legacy application starts with identifying the core building blocks and access points in the mainframe. There are three main points of integration into the legacy mainframe:

1. Presentation Layer: The goal here is to provide some level of service to the consumer to improve the user experience and extend access to other applications through the front end.
   
   
2. Business Layer: Simply stated, this is a procedure call wrapped in a SOA service. As illustrated in the hands-on case study below, this can really open up a legacy application for reuse and agility. But this task is far from trivial. As with any legacy application (mainframe or otherwise), business processes are often not discrete, standalone services. After years of development, the typical code base ends up resembling a big ball of string. It can be tough to determine where a process starts and where it ends.
   
   
3. Data Layer: In this instance, a SOA service or call can be made to a legacy data store or file system. This allows users to use native data calls, yet access relational or non-relational data stores. With data integration, enterprises can support fully-transactional, bidirectional, and SQL-based access to any data store on the mainframe.

SOA Integration: Top Four Scenarios and Oracle Solutions

Technical journals and IT analysts have made the point that SOA is neither a product nor solution, but a journey. In that same sense, modernization is just such a journey, but one for which some additional packing is necessary, so to speak. This modernization journey will take unexpected twists and turns and visit unanticipated places as business objectives, key personnel, and technologies change.

The information presented here is based on factors and situations we have found to be common among the customers and partners we have assisted on their modernization journeys. The specifics of each situation and environment -- business drivers, strategic direction, tactical solutions under consideration, or biggest pain point ? may vary. But it is our hope that by the end of this section any questions about when, where, and how to start will have answered.

Oracle Products Included in the Solution

The Oracle Fusion Middleware family of products contains all the necessary components for a Service-Oriented Legacy Service Bus/SOA Integration architecture. But before we get started, let's cover why the selected products make the most sense, and explain why other products were omitted.

Why Web services?
Web services are the foundation of communication in this context, as well as the results of the processing of mainframe artifacts. Without Web services, a Legacy Service Bus/Legacy SOA Integration is impossible. Enough said.

Why Oracle Legacy Adapters?
We refer to the Oracle Legacy Adapters as the Legacy Service Engine. These adapters allow a seamless flow of information from the legacy environment to the open systems environment.

Why Oracle BPEL Process Manager?
Business process orchestration and human workflow are essential to the communication between the Legacy Service Bus and the mainframe, and between internal and external services. To align business with IT, applications must become more process-driven. In today's world, workflow engines place the necessary tasks in workflow inboxes, eliminating the need for users to remember activities that must be performed.

Oracle BPEL Process Manager allows SOA service implementations, human interaction, and system workflow to be orchestrated quickly and easily using graphical drag-and-drop techniques that allow direct user involvement in system orchestration, thereby reducing the amount of custom code required, and increasing application agility.

Why Oracle Enterprise Service Bus?
BPEL provides business and human workflow for business services. However, a number of services are generic to many applications. A service bus provides these generic services by implementing standard facilities, such as messaging or integration capabilities, often by encapsulating different technologies that implement the generic services, making them appear as one.

Oracle ESB provides a messaging and integration layer for directly connecting applications and web services through a common infrastructure. Oracle ESB combines an asynchronous messaging backbone with intelligent message transformation and routing to ensure that messages are passed reliably. The mainframe is known for its quality of service and reliability.

Oracle ESB makes use of Oracle Application Server's grid capabilities to allow the formation of an ESB grid that crosses multiple platforms.

Why Oracle Enterprise Manager (OEM)?
As discussed earlier in this article, there is no need for a specialized Web Services Manager or Web Services Registry at this point in the modernization journey. However, it makes sense to make use of the technology in the Oracle Enterprise Manager product to manage the SOA middle-tier and perhaps at some point the Oracle Database Grid. The SOA Management Pack also provides a means to monitor Web services from an end-user perspective and for request-based monitoring. Administrators can define SOAP tests for one or more partner links of a BPEL process, any hosted web service, or an external service.

Why Oracle Identity Management?
Every IT environment needs some form of security. The SOA Legacy Integration journey must begin with security that is strong, but not overwhelming to implement and maintain. Identity Management is a broad topic when it comes to Oracle. It includes everything from single sign-on to provisioning, federated identity, directory services, certificate authority, and more. In the immediate term, we will use Oracle Internet Directory (remember, this is called OID and is an LDAP compliant directory) and Oracle Enterprise Single Sign-On to allow simultaneous login to all applications (web services).

Why Oracle WebCenter Portal?
A portal provides a single web-based entry point into a variety of applications. Using a portal allows an organization to create a unique user experience that combines many of the functional components of the SOA applications into a single point of access.

Why Oracle Data Integrator?
Oracle Data Integrator (ODI) allows users to extract data from many legacy and non-legacy sources?including those on the mainframe?using web services that are part of the LSB application server. ODI can load and transform the data into a number of data sources. Flat file information sharing is often a cornerstone of legacy information integration. Replacing this custom processing with a product such as ODI centralizes flat file processing and makes it more agile and adaptable. It also makes it much easier to enhance the system for new flat file types or to make changes to current structures.

Why Oracle Business Activity Monitoring (BAM)?
Modern systems do not require a report, query, or lengthy batch process to complete before providing information on business activity. Instead, real-time information is viewed via graphical dashboards that can send out alerts through a variety of communication mechanisms when user-defined thresholds of importance are met. Oracle (BAM) will be used to provide this real time feedback for web services, BPEL, and ESB processes.

Why Oracle Business Intelligence?
Instead of relying on hard copy reports, real-time business intelligence and data mining tools can be used to slice and dice data to determine key performance indicators and discover data trends -- without having to wait for IT. This increases the overall agility of the organization by allowing end users to react quickly and directly to ongoing business changes.

Scenario One: Enterprise Information Integration

Also known as: Data Integration, file sharing, file messaging.

Problem

The current infrastructure for information integration is fragile, expensive, and difficult to maintain.

The problem is often characterized by issues with no common work-flow approach, a lack of data quality and data profiling capabilities, customized transformation logic unique to each data feed, lack of real time monitoring capability, and an inability to quickly add new data feeds.

Context

Data needs to be shared with new systems developed on open systems and with other mainframe systems, both internal and external to the organization. Most legacy systems have incoming or outgoing data feeds.

Forces

Stand-alone applications and organizations are things of the past. The need now is for information-sharing between applications and businesses.

Solution

The solution includes Oracle ESB, Oracle BPEL, Oracle Legacy Adapters, Oracle E-Business Suite Adapters, and Oracle OEM, as depicted below:

Scenario One Summary

An important objective of Legacy SOA Integration is to avoid disruption of the current business processing and legacy system. This solution holds true to this objective by leaving the data feed unchanged.

Performing even minor changes to data feeds can be problematic:

• If a third party or even an internal organization outside your control is involved, it can take months to complete the changes
  
  
• Even internally owned data feeds impact the source systems, current processing, and target systems, so even a small change can have a ripple effect.

The current data feed will remain intact, will download via FTP to a directory, and will most likely remain batched. This solution involves technologies such as Legacy Adapters and Oracle Messaging because they can be adapted when changes to business processes are made.

Oracle ESB
The Oracle ESB will use the file or FTP adapters to read the flat file and then transform that file to a common (canonical) XML file format. Based on the source of the data feed, the message will be routed to the appropriate Oracle BPEL process.

Oracle BPEL
This is where the previously mentioned workflow and processing come into play:

• Oracle BPEL will call a Java or Web service process to perform any validation processing. This validation processing can call out to the Oracle database to validate information based on the data in the database.
  
  
• After validation, file type-specific processing takes place. This involves the application of business rules to the input data file. This business processing can call the Oracle Rules engine

Common Error Processing
Validation and/or business rule processing errors will be passed to an error-handling route. The BPEL worklist will be populated, so a human may correct the problem file or records.

Data persistence Web service
The data will be persisted in the Oracle database, IMS database, and/or the Oracle E-Business Suite.

Scenario Two: Web Enablement

Also known as: Screen Scraping, Re-interfacing.

Problem

Customer support people, sales representatives, customers, and partners would like to access the system through the Web. However, no single interface is available to allow updates to both the legacy system and the Oracle system.

Old green-screen technologies have many limitations, not the least of which is that they are not very intuitive. Access to several screen or systems is necessary in order to access essential information, and there is no point-and-click. In addition, users often have to go to multiple systems to either query or update the same or similar data.

Context

Users want their data now, wherever they are, and at any time of the day or night. Users also want legacy systems and new Oracle environments to work together.

A business process that requires a user to query and then update multiple systems slows everything down, and in many cases may cause data inconsistencies.

Forces

Technologies that provide better application interfaces have been in the marketplace for years. But of even greater significance is the entry into the workforce of a generation of explicitly web-savvy users, and their ranks continue to grow.

Solution

This solution includes Oracle WebCenter, JSF/JSP, Oracle Legacy Adapters, and Oracle OEM, as depicted below:

Scenario Two: Summary

The key to this scenario is the interface. Therefore, Oracle WebCenter and/or JSP/JSF will be used. For the first iteration of this scenario, JSP and/or JSF can be used to simplify development and speed deployment. A more sophisticated alternative would use JSF to develop JSR-168 portlets, which would then be deployed using Oracle WebCenter.

Scenario Three: Report Off-Load Using Data Migration

Also known as: Data migration, Legacy Operational Data Store, Reporting Modernization.

Problem

IT Perspective: The legacy reporting infrastructure costs millions to run and has resulted in a six-month backlog of report requests.

User Perspective: Even with 100 green-bar reports there is insufficient information on which to make business decisions.

Context

Users need access to information in a variety of formats and dimensions. They also need to be able to easily create ad hoc and what-if scenarios.

It is not uncommon for a mainframe-centric organization to rely solely on spreadsheets for strategic sales forecasting.

Forces

Generating reports on the mainframe is expensive, and places obstacles between business users and the information they need to make decisions. In a typical organization users will create their own reports using Excel, SQL, and other desktop tools, often resulting in duplicated data throughout the enterprise.

Solution

This solution includes Oracle Data Integrator, Oracle BPEL, Oracle Legacy Adapters, Oracle BAM, Oracle BI Suite and Oracle OEM.

Scenario Three Summary

This summary will explain the rationale behind the selection of Oracle products used in this scenario.

Oracle BAM
In Oracle BAM, both the end-user decision makers and IT management see the real time flow of information into the reporting system. End-user decision makers can make real time decisions based on the most current information. IT management can get immediate alerts if a data load is running slow, or on the amount of data being loaded each minute, or on the average response time of ad hoc user reports.

Oracle BPEL
Oracle BPEL is used to orchestrate the flow of information into the Oracle reporting database. Oracle BPEL can be used to schedule data loads based on the time of arrival, or on the arrival of a specific file, or by constantly looking for the data extracts to load.

Oracle Data Integrator (ODI)
Oracle Data Integrator provides a fast method to bulk load data into the Oracle reporting database. It also provides access to data transformation, data cleansing, and data management services. As ODI is fully web service-enabled, any ODI component can be consumed by Oracle BPEL, Oracle ESB, or any other web service-enabled tool or product.

Oracle BI
The preceding products are infrastructure products used to support the end objective: moving reporting off the mainframe to make it accessible to all users. Oracle BI tools can complement that effort by reducing the proliferation of Excel spreadsheets and customer SQL reports on user desktops.

Scenario Four: End-to-End SOA

Also known as: Software as a Service (SaaS), Legacy SOA Integration.

Problem
The legacy system is a black box. Getting information into it is painful, and getting information out is worse. It provides no visibility into the business processes on which the business depends.

Context
The mainframe system does not reflect how business is done today. The legacy system is difficult to maintain, resists enhancement, and presents an obstacle to rolling out new services/product offerings to internal and external customers.

Forces
The user community demands real-time information processing with immediately available results. Information integration interfaces into the system change every week, and new trading partners want to interface with the business in days, not months. The system interface needs to be personalized in order to allow role-based access to appropriate information:

• Full access for internal power users.
• Sales people need access to pertinent sales data.
• Customers need access to data related to orders.
• Executives need real-time, up-to-the-minute business insight.

Solution

This solution includes Oracle WebCenter, Oracle PBM, Oracle BPEL, Oracle Legacy Adapters, Oracle BAM, Oracle ESB, Oracle OID and SSO, and Oracle OEM.

Scenario Four Summary

As with the previous scenario, this summary will explain the rationale behind the selection of the Oracle products used in this scenario.

Oracle WebCenter
Oracle WebCenter provides a quick and easy way to provide personalized views into the system. WebCenter also provides integration to OID and SSO for user authentication and authorization, as well as the ability to verify security across all parts of the system. Gone are the days of logging into multiple systems. Gone also are the days of looking between two green-screens to verify data. Now all the data is in one web dashboard.

Oracle BAM
BAM plays a big role here as the amount of processing increases. BAM makes it easier to monitor all the business process and services in one screen.

Oracle ESB
Just as the ESB was used elsewhere in this article to help migrate data from other systems and provide an integration bus, it is used here to accept two different input file formats and process them in a single workflow.

Oracle OID and SSO
OID is used across all of the SOA products (BAM, ESB, and BPEL) to provide a common security repository. SSO makes life easier by allowing access to all applications via a single login. Users logging in on one application can be simultaneously verified across all applications.

SOA Integration: Final Summary

This article provides an overview of how a legacy environment can be integrated into current and future initiatives. The examples and techniques presented here represent only a few of the possibilities. The authors provide a more detailed, hands-on approach in the book Oracle Modernization Solutions (Packt Publishing, ISBN: 1847194648).