Blueprint for Successful SOA Integration

by Dain Hansen
03/12/2008

Abstract

SOA Integration has recently emerged as the de facto standard for successful IT integration; it leverages the benefits of Service Oriented Architecture (SOA) to solve one of the most fundamental challenges IT is facing today. Some architects mistake SOA Integration for the inclusion of an enterprise service bus or BPEL along with some adapters. There's more to it. This article will discuss how SOA Integration can be defined, what it solves, what to look for, and some points to think about for your IT organization. It also points out some of the most common mistakes, such as the Accidental Integration Architecture pattern.

What Is SOA Integration?

• Is SOA the same as Enterprise Service Bus? - No.
• Does SOA only integrate SOAP, Web Services? - No.
• Does it simply connect your SOA into mine? - No, but nice try.

Put rather simply, gaps exist in your IT. These gaps—which are between people, processes, and applications—can take a toll on the effectiveness of your business. If you are good at your job, this is probably nothing new for you; you face it every day. Companies like yours have spent vast IT budgets attempting to plug these gaps.

These gaps may manifest themselves on either the IT side or the business side of the organization, as illustrated in Figure 1.

Attempts to solve these integration challenges using code, EAI techniques, or piece-meal approaches haven't produced tangible business results—for example, upgrading an existing PeopleSoft instance to a newer version. All the business logic has to be redone; it may use a different interface or data format.

SOA offers an alternative. One of the lessons learned in IT is that SOA helps to drive IT innovation, keeping things flexible while reducing risk and cutting costs.

Applying SOA to Integration

SOA and integration together comprise a solution that transforms your brittle IT systems, applications, and data sources into highly flexible reusable services.

SOA integration capitalizes on the lessons learned in Service-Oriented Architecture to make better, more innovative use of IT assets by transforming them into services that can be shared and reused across the enterprise. As part of its core capabilities, SOA integration provides a framework for enabling services, data, and events, and connecting them to better align with business requirements. It supports key integration patterns that allow IT managers and IT architects to aggregate, orchestrate, and mediate these services. This can accelerate IT responsiveness to constant changes in business requirements.

Accidental Integration Architecture

Historically, integration started as a pattern to build better transparency between consumers and producers of information across multiple channels. You can think of EAI as one of the earliest forms of integration focused around application integration, ERP modernization, and B2B (a specific type of multi-enterprise integration). As SOA was recently introduced, integration evolved to meet other, more specific needs such as service mediation for web services. On the journey to integration, enterprises also recognize that data services as well as real-time events needed to be treated uniquely within other integration scenarios.

All these integration patterns are "correct," but, unfortunately, many companies struggled to adopt all of them or simply favored one but not another. Figure 2 is an example of the Accidental Integration Architecture pattern that emerged simply as a means to solve multiple types of integration between consumers and producers. It was an accidental architecture focused on trying to consolidate the integration challenge.

Despite the fact that integration bridges could be built across these types of patterns, this solution model isn't ideal. Simply put, it doesn't scale and has an unnecessarily high total cost of ownership. This is primarily because the point solutions for the various integration patterns are not integrated. It has holes in both security and management, as well as several redundancies in how information is passed back and forth between the producer and the consumer of the services.

SOA Integration Solution

To avoid the pitfalls of an Accidental Integration Architecture, we'll want to start leveraging common standards-based re-usable components to better unify the solution. You probably are thinking now Hey, isn't that one of the lessons learned from SOA" Let's look at a re-use example within Integration: components such as connectivity, security, mediation can be thought of as re-usable shared-services. Each shared component may require specializations in tooling, run-time optimizations, and at the same time should be unified across the platform. For example, the run-time optimization you need for managing a data query in a data service is different than for managing a stateless message flow. Each pattern requires unique processing, optimizations, and tooling.

In this example described in Figure 3, SOA Integration behaves as a completely integrated solution. Components for service integration, process integration, service orchestration, data services, connectivity, and unified tooling, each work together to provide the necessary integration patterns needed for abstracting multiple integration scenarios.

Now let's look at these shared components in detail within the context of the SOA Integration solution.

Next Generation ESBs & Orchestration. ESBs grew out of the need to implement specific patterns for integration - especially mediation - and turn those patterns into a fabric or back-plane for SOA. This approach was an important first step for successful SOA. ESBs alone are extremely versatile and flexible at handling high volumes and service mediation cases. However, early ESBs were limited in their capacity to handle the heavy lifting required to perform application integration, process integration, Java or .NET integration, or anything task that required going beyond the boundaries of service mediation patterns.

A few ESBs jumped to include these capabilities. These EAI-ESBs offered sluggish performance, or simply became too complex for a single user to fathom. Other ESBs could not scale to address the additional integration challenges.

In order to participate as part of a larger SOA solution, ESBs must be able to scale across multiple projects. To deliver on this requirement, the next generation of ESBs need more than simple service mediation and routing. They need the kind of advanced optimization in both mediation and orchestration that will allow truly enterprise-wide deployment. (Figure 4)

The importance of orchestration as a shared component gives ESBs the flexibility for success in either paradigm. Large volumes of stateless messages can be processed through mediation only, whereas, in some cases, an orchestration step is required. This shouldn't require separate hops between modules, rather it should be an optimized pathway from the ESB as if it still 'was' the ESB. These next generation ESBs give the best of both worlds. They work both independently in mediation-only mode, as well as, part of more complex orchestration cases in an overall SOA Integration solution.

Data Services. Recently, Information as a Service (IaaS) models have emerged that focus on data integration and on optimizing the exposure of information contained in data from multiple disparate sources. This technique has revolutionized the integration of data across the enterprise. Data Architects know that data requirements are unique and separate from those of services or even events. For example, data requires modeling, and must be secure, private, and protected flows into and out of multiple systems. In addition, federated query requirements and read/write requirements present a much more complex paradigm than simply routing a service from one application to the next.

Data service integration patterns may be unique, but they definitely share components with the SOA Integration solution. For example, data services might need to access the same ERP system that a process would, so why not re-use these key SOA based connectivity and ESB capabilities for accessing data systems across multiple federated projects? For this reason, data services behave as a shared component in the SOA Integration solution. They make use of other elements in tooling, service integration, process integration and connectivity as a way to simplify complex information-centric integration patterns.

SOA-based Connectivity. It always comes down to the last mile, doesn't it? Solving integration patterns effectively requires an efficient, secure, and reliable flow of data into and out of multiple sources. In the context of SOA Integration, this capability must be shared across integration solutions, and not confined to an ESB or a BPM. Connectivity must be a shared reusable component.

For example, in a data-services-centric architecture, data might need to traverse traditional data sources as well as ERP systems. Performing each of these ERP integrations through a separate adapter will result in end-to-end reliability, security, and quality of service (QoS) issues. In the ideal solution architecture, the connectivity itself is a fully integral part of the SOA Integration solution, which can leverage the same tooling to expand the Integration flow to other components.

In Figure 5, an ESB includes connectivity as a reusable component that can be plugged into multiple packaged applications. Out of the box schemas are included for ease of mapping between the ESB and the packaged applications directly. The advantage of this architecture is that integrations between these applications and services are bi-directional; can leverage end-to-end SOA security paradigms; as well as ensure that these messages behave with the right prescribed Quality of Service.

Connectivity is no longer about adapters - connectivity is a completely agile component of SOA for enterprises to the benefits of a Service Oriented Architecture.

Event Processing & Event Driven SOA. Event Processing has also arrived on the scene to complement SOA and constitute Event Driven SOA. Event-Driven SOA combines SOA's request-response and EDA's (Event-Driven Architecture) event publish-subscribe paradigms. Supporting events and services within SOA allows designers to map the application design to the business problem, which typically consists of both events and requests/responses. An optimized business needs to handle both services and events to enable extreme transaction processing (XTP). This results in extreme agility - enabling the enterprise to conduct business at light speed. Figure F highlights a business scenario where Event Processing and SOA come together to solve the problem of lost baggage during airline travel. In this example, events from the various operational area 'clouds' converge into a complex event processing engine. The ESB acts both as a source and sink for events. Based on certain event processing rules, 'lost bag' events are detected and handed over to BPM and BAM—components integrated into the application following SOA principles—for further processing.

SOA Integration leverages Event Processing for mission critical, high volume, complex transactions that require real-time (or near real time) response. Events both complex and simple must be correlated, processed, filtered, and sometimes optimized as part of a stateless activity in an integration pattern. These unique event-processing requirements must be part of a SOA Integration Architecture. Combined they give enterprises the agility they are looking for along with the mission-critical foundation they need to run their business.

Unified SOA Tooling. One of the most notable components with SOA Integration solutions is the advancement of the tooling, especially the Eclipse standard.This has tremendously impacted SOA Integration, especially because of a large developer and architecture community. SOA Integration solutions need to take advantage of unified tooling that can be leveraged for the solution to be effective. The tooling in SOA Integration needs to seamlessly interoperate with SOA systems, such as SOA Integration and SOA Governance as well as open-source plugins or 3^rd party plugins. These are the existing tools that are already used by the enterprise, and SOA Integration needs to adapt to those existing tools, rather than the other way around. For example, a service needs to be checked into a source control management system. These tools exist out of the box today in Eclipse - so why re-invent this with SOA Integration?

Figure 7 describes an example of how ESB routing configuration is defined as part of an Eclipse environment. Common components within the eclipse designer can be leveraged for other types of process or data integration; additionally development tools or other Java or .NET extensibility tools are all available through this same environment.

SOA Integration Solution Architecture

The components in the SOA Integration solution must work together as a solution to provide effective end-to-end integration capabilities. While individual components may require specialized tooling, run-time optimization, or management, the various components must be unified across the platform. For example, the run-time optimization required to manage a data query is different from that required to manage a stateless message flow. Put them together and SOA Integration (Figure H) is able to connect services, data, events, and custom apps to any Business Process, Web 2.0-based interaction portal, or any other consuming element.

Lessons Learned in SOA applied to the Solution Architecture

If it is one thing that SOA has taught us, it is the need for re-use. We've just looked at how leveraging common components within the SOA Integration solution gives us the most well-rounded capabilities for the overall solution. Beyond that lesson, we'll look at three other important lessons learned in SOA:

• Start Governance early: SOA Integration and SOA Governance need to work together to give you the benefits of agility, cost-reduction, and reduced risk.
• Don't do SOA in an IT vacuum: With SOA and BPM ensure that business processes are optimized to realize the benefit of SOA for the entire organization.
• Start small, but think holistically: Small projects are a great starting point, but services quickly develop into composite services. These composite services need to be managed, designed, and deployed centrally, as well as, work together with the SOA Integration solution.

Start SOA Governance Early for better Visibility

One of the key lessons learned from SOA: start SOA Governance early! This precept can be applied to the SOA Integration Solution Architecture as well. Effective governance provides visibility into and control of your implementations. The same is true for integration implementations. SOA Governance is crucial to successful SOA Integration.

Governance is essential to ensuring that your SOA has been implemented as intended and that it runs as intended and continues to meet business needs. Getting better visibility into and control of your SOA system (a large part of which is Integration-centric) requires a purpose-built architecture that combines Integration and Governance.In that context, Integration patterns require more than simply monitoring services through an ESB. The emergence of SOA Governance is driving a shift away from a narrow focus on the governance of integration solutions alone to a more holistic focus on governance across the all elements of the SOA and throughout the entire SOA lifecycle.

For example, to restrict data access, a policy for fine-grained entitlements might be implemented for a data access service. This has its own unique enforcement and policy management requirements for enforcing the entitlement on a particular service.

Patterns like these described in Figure 9 are emerging as the standard for successful SOA Governance. When SOA Integration and SOA Governance work together, they give the right control and visibility needed for successful SOA implementations.

Don't do SOA in an IT vacuum

The convergence of SOA and BPM has become one of the most talked about subjects in the industry - and one of the most confusing. Let's try to eliminate some of that confusion by first focusing on the influence of BPM on SOA architectures.

We can't talk about IT integrating ERP systems without also considering how that integration will play with the business analysts or LOB who actually need that data or services for their business processes. These two often contradictory elements need to work together as part of a fully integrated solution. When SOA and BPM play well together we have seen enormous benefits. Not only is there alignment between the various roles in IT and the LOB, but the processes are implemented in an optimized way that both camps can successfully manage. Let's look at one example of how SOA & BPM can work together and provide complimentary benefits.

Once a business process model is implemented and automated across the SOA Integration, optimization occurs when runtime feedback is returned to the business analyst via integrated business-activity monitoring. This lets business users see where process improvement needs to occur in real time. Once improvements are identified, the business analyst can update both the models and the business and the development cycle begin anew. True business transformation and optimization are realized through this iterative business-integration cycle.

To see these types of benefits SOA and BPM needs to be integrated in such a way as to give multiple users in the organization common unified tools to share metadata, share governance and management information, and ultimately optimize the interoperability between a business processes and how those processes are translated to the back-end integration.

Figure 10 describes one example of how that type of process can be described from a business analyst view point. That process can be shared into a repository; the respective metadata (not necessarily the process view) can be shared among architects, developers, and operations. For example, the ERP entry point for the back-end process can be implemented by an architect, a developer can build a transformation map, and the operations people can specify an SLA policy.

When BPM is combined with the power of SOA Integration enterprises reap enormous benefits.

Start small, but think holistically

The concept of building larger service composites (multiple complex services without boundaries) is nothing new. However, trends are emerging that are shaping the creation, management, and deployment of composite services.

Enter SCA, the Service Component Architecture standard. This important standard specifies an abstract model that architects can leverage for developing, managing, and modeling composites. These composites are enterprise-wide, free of any limitations imposed by project boundaries or the scope of an integration scenario. These integration services between ESBs, BPMs, and Data Service Platforms constitute one component within a composite flow. Other entry-points can originate from Web 2.0 portals or even simple SOAP-based JMS events.

The importance of managing these composites and providing tools for them is extremely critical for the success of complex SOA Integration projects. In this example in Figure 11, an architect models a composite service flow within the Eclipse environment. The entry points and dependencies are identified which helps to ensure that the actual implementation matches the intended implementation. This also helps ensure that the lifecycle of complex SOA integration implementations can be managed successfully where hand-offs between multiple roles can be better governed if these dependencies are centrally managed.

Putting It all Together

The combination of SOA governance, BPM, and composite services adds up to state-of-the art capabilities for integrating any type of service, data, message, or event. Adopting a holistic approach to SOA governance, management, and security will provide essential visibility and control, and allow business processes to be tied into integration services. Those services can then be shared with teams across the enterprise.

What to Look for in SOA Integration?

SOA integration fulfills a great many purposes for your enterprise. But choose your architecture pattern carefully and ensure that your solution architecture meets the forward-looking requirements of your business. Below are some key questions you should ask when considering a SOA integration solution:

• Does my ESB leverage reusable components for connectivity, security, and orchestration?
• How are composite services managed? Are they standards-based? Do they work only with integration or can they go beyond?
• Is there a SOA governance solution optimized for the integration solution?
• Is it SOA integration or simply integration?
• How does BPM fit together with the SOA integration solution?
• What types of connectivity options are offered—an adapter approach or a SOA approach?
• Are there common or shared components in the architecture? Can tooling modules be plugged in?
• Will you need to start completely over when you think about data or events?
• Does the SOA integration solution have capabilities for enterprise wide deployments?
• Does the SOA integration solution scale beyond the enterprise?

Last words

Enterprises aren't simple, nor are the IT infrastructures that support them. A fully-integrated SOA Integration Solution simplifies the burden and gets you back to the business at hand: keeping your business competitive.

At BEA, SOA integration is not just about a single product, a single technology, or a single integration standard. We believe that an effective SOA integration solution simplifies complex enterprise-wide integration challenges by providing flexible infrastructure that can be deployed as a suite or individually. BEA's SOA integration portfolio is designed to ensure the enterprise is at the center of integration efforts, and not consumed by complexity.

References

Supporting References

• Rethinking SOA and Integration - some analysis by Lorraine Lawson

Dain Hansen is Director, Product Marketing Manager for SOA Integration at BEA. He is currently working on projects in SOA, including: SOA Management, Governance and has long running experience with ESB technology.