Mastering SOA

 

Part 3: Orchestrating into End-to-End Processes


by Joan Lawson and Dave Shaffer

 

Build agile and standard business processes that orchestrate services into end-to-end business flows.

 

Downloads for this article:
 Oracle SOA Suite and Services Registry


Published February 2007

The previous installments in this series described how to leverage the latest standards and architectural approaches to service-enable back-end applications and to publish those services on an enterprise service bus (ESB). But all those services are useful only in the context of the clients and business processes that access them. This installment focuses on how to build agile and standard business processes that orchestrate those services into end-to-end business flows.

A business process is a series of steps required to implement a business function. The business process may include system interactions and/or human interactions, within an enterprise or extending across corporate boundaries. Some enterprises merely document their business processes, for consistency or regulatory purposes. However, the past several years have seen an increase in the number of organizations that use IT systems to execute and monitor their business processes in an automated fashion. We believe that one of the key drivers behind this accelerating adoption of business process management (BPM) is the emergence of broadly accepted standards such as WS-BPEL for implementing and executing business processes.

 

Figure 1
Figure 1 Example BPEL process for order processing


In this installment of "Mastering SOA," we will describe how to meet the challenge of business process orchestration using these standards, incorporating experiences from both the Oracle vendor perspective and Monster Worldwide, with several successful Oracle Enterprise Service Bus and Oracle BPEL Process Manager service-oriented architecture (SOA) implementations in production.

Monster Worldwide

Monster Worldwide has determined that BPEL is an important standard because SOA and BPEL offer the required agility and extensibility to be a global presence while meeting the local business requirements of customers. In addition, BPEL is driven by a standardization committee supported by several IT industry players, thereby minimizing the risk of vendor lock-in that would result from choosing a proprietary integration platform.

With Monster Worldwide's growing inventory of applications, it is necessary to interconnect these applications using Web services and SOA. It publishes services by exposing interfaces to existing systems in a standard fashion, leveraging Web services standards such as SOAP, WSDL, and XML Schema. Monster Worldwide then orchestrates these services into end-to-end process flows. BPEL is the industry standard for such orchestration.

In this article, you will find examples of how Monster is implementing SOA and BPEL, along with tips and tricks gleaned from SOA implementations at other Oracle customers. You will also learn about the features of the BPEL standard, how it fits with the other standards in the SOA space, and how the Oracle SOA Suite provides an infrastructure that implements these standards.

BPEL Basics

BPEL defines an XML-based language for specifying the behavior of business processes that are based on Web services. The first public BPEL specification was released in July 2002, as BPEL4WS version 1.0, combining IBM's WebServices Flow Language (WSFL) and Microsoft's XLANG specification. Version 1.1 was released in 2003, collaborated on by many industry vendors, and version 2.0 of the WS-BPEL standard is expected to be released from the OASIS standards body in early 2007.

The BPEL standard enables both interoperability and portability for an organization's business processes to a degree not previously possible. Interoperability enables business processes operating on different business process engines to interact with each other and with services published from many vendor and open source platforms. This is possible because BPEL is layered on top of the lower levels of the Web services stack—namely, SOAP, UDDI, WS-Addressing, WSDL, XML, XML Schema, and others. Interoperability in the arena of business processes has been possible for several years; however, the BPEL and Web services standards take it to the next level. But beyond interoperability, the concept of portability for business processes is truly something new. The process portability offered by BPEL ensures that business processes can be defined using tools that support the standard and then run on any compliant systems. Today, BPEL engines are available from large platform vendors such as Oracle and IBM as well as from smaller niche vendors and open source companies.

Choreography and Orchestration

Various activities are executed to implement the runtime lifecycle of a business process. BPEL is a Web services orchestration language, distinct from process choreography. Process choreography, as the term is commonly used in the IT world, describes the interplay of various trading partners to implement a multiorganization business function. For example, in the supply chain space, the fulfillment of a product purchase may involve the exchange of purchase orders, advance shipping notices, and money between two or more companies. Choreography does not describe how each company would conduct its operations, only how the different companies would interface with each other.

Process orchestration is when a central process coordinates the execution of different Web services operations. The central conductor (as in an orchestra) is aware of the overall goal of the orchestration, the operations involved, and the order of the operation invocation. This centralized management allows Web services to be added or removed without each being aware of its effect on others, as well as compensatory processes to be implemented in case of faults and exceptions.

Process orchestration and process choreography are similar only to the degree that they describe service interfaces and interactions. For the purposes of this discussion, we will focus on process orchestration. When you use BPEL for orchestration, the BPEL language provides a standard for controlling the overall sequence of operations, invoking services and executing on a BPEL server.

One of the most complex integration efforts at Monster Worldwide was the development of an order-to-invoice integration from Oracle's Siebel Customer Relationship Management (Siebel CRM) to Oracle E-Business Suite. The requirements consisted of splitting the original order message (a Siebel-generated object that consisted of both order and customer data) into two messages, transforming data in both messages, and conducting significant enrichment of the order data before sending the message into Oracle E-Business Suite. In addition, it was necessary to create the customer profile in Oracle E-Business Suite before posting the customer order. A series of BPEL processes were constructed to coordinate the execution of these steps. Once the Siebel order object is received, the first service splits the object into two messages, customer and order. The orchestration then enhances the order and customer data in parallel. When the order data has been transformed, it is then passed to the service that exercises the entire order data enhancement. When complete, the orchestration passes the customer message to Oracle E-Business Suite and awaits a successful response before passing the order message along. This orchestration assures Monster that all activity happens in the expected sequence and assures us that, if a fault is encountered during the process, proper compensation occurs.

BPEL Framework

BPEL is a powerful language that allows the development of complex business processes using Web services components. A BPEL process can specify which Web services should be invoked and the sequence in which they are invoked. A BPEL server tracks the business processes that are involved in a transaction; ensures that the steps are executed in the correct order; and manages transactions, compensation, and exceptions.

BPEL supports all the control flow structures you would expect from any orchestration language: loops and conditionals, variables, assigning data values, defining fault handlers, and much more. The BPEL receive activity is used to define the events or messages that will create an instance of a business process when a client invokes it for the first time through a service interface. The process will then invoke other services, which are the activities in the defined business process sequence. A response will typically be generated (reply).

Along the way, the BPEL process may have need to manipulate data variables (assign), handle faults and exceptions (throw), or wait for a set period of time. Finally, the process can end (terminate) at any time.

The element variable defines all messages and data used throughout the process description. Variables are used for every message sent to the involved services or received from them. They are utilized to handle the process execution data in the same manner as in any programming language.

Structural activities may include a BPEL orchestration that performs activities sequentially (sequence) or that performs activities in parallel (flow). The following are some of the most important control flow activities in BPEL:

  • switch - choose a route based on a condition
  • while - wait for a specific condition to be satisfied
  • pick - wait for one of several messages to be received

BPEL can define the execution of business processes based on conditional behaviors. For example, Monster is separated into regions: North America, Europe, and Asia Pacific. Based on the originating region of the order, the BPEL processes execute regional-specific logic.

Similarly, Monster has defined the order-to-invoice BPEL flow to process the customer and order in parallel, while processing the order services sequentially.

BPEL processes orchestrate services (partnerLinks), which define the interfaces to the parties with which the process interacts. These partnerLinks include both services that are invoked by the BPEL process and the interface through which clients invoke the BPEL process itself. Flexible binding of BPEL processes to the physical service endpoints can be done at design, deployment, or runtime. For the services described above, the Siebel and Oracle E-Business Suite adapters are our key partnerLinks.

BPEL has rich support for asynchronous events. A BPEL process can wait to receive a message, or request, from a client (the same BPEL receive activity as used to initiate a new process instance) at any time. The BPEL engine will handle the overhead of persisting the state of the process until the message comes in and also correlating the message to a particular instance of the process, through either standard WS-Addressing or custom correlation mechanisms.

BPEL processes themselves can expose synchronous or asynchronous interfaces to their clients. Synchronous BPEL process operations block the client until the process returns a response to the client. Asynchronous BPEL interfaces can be one-way ("fire and forget") or return a response to a client via a callback. Asynchronous processes are more commonly used for longer-running operations, and synchronous processes are more commonly used for shorter-running operations.

At Monster, the order-to-invoice process as described above is a long-running, asynchronous process. In addition to the logic described above, the invoice number of the invoice created in Oracle E-Business Suite is returned via a callback to Siebel, and if an exception occurs, the exception handling process includes human workflow.

Exceptions and Compensation Handling

Often, the effort to implement the positive use-case for a business process (when everything goes right) takes only a small amount of the total effort of designing and implementing the business process. The majority of time is spent in defining all the appropriate exception handling logic to build a robust and resilient process that does reasonable things under all expected conditions (and ideally even under unexpected conditions). BPEL provides rich exception handling capabilities so that developers can define, in a hierarchical try/catch sort of pattern, what to do in case of exceptions and faults during the execution of a process.

However, beyond exception handling, some processes require more transactional behavior. In the world of short-lived flows and services, standard transaction models are sufficient, using XA or ACID transaction models where resources can be locked for the duration of a transaction and rolled back if necessary.

However, in a business process orchestration world, where services may be owned by external organizations and processes and services interfaces may take minutes, hours, or even days to execute, a new transaction model is required. These are called long-running transactions or compensating transactions. While the world is still waiting for true standards in this arena that allow transaction coordinators to handle compensating transactions automatically, the BPEL language provides support for this capability through a feature called compensation handlers.

Compensation handlers can explicitly undo work that was completed successfully prior to some error condition arising. Each invocation activity can, as needed, be paired with a compensation activity that undoes it. If a fault occurs, the compensation handlers for the already completed work within the scope of a failing activity are used to undo the completed work.

The downside to the compensation handler approach is that it requires the process designer to explicitly think about how to compensate each activity. On the positive side, the BPEL engine will then track all the activities at runtime and execute only those compensation handlers for the previously completed steps. Also, the BPEL engine will snapshot data during runtime so that the compensation handlers have access to the data that was current at the time the original activity completed. Finally, this approach means that the services themselves don't need to have any explicit support for compensating transactions—a good thing, because there are as yet no standards such as XA and JTA for compensating transactions.

BPEL 2.0

BPEL 2.0 is the evolution of BPEL 1.1 as a public OASIS standard. BPEL 2.0 provides some significant enhancements, such as:

  • New activity types—if-then-else, repeatUntil, validate, forEach, and extensionActivity
  • Completion condition in forEach activity
  • Variable initialization
  • XSLT for variable transformations—new XPath extension function bpws:doXslTransform
  • Simplified XPath access to variable data—XPath variable syntax $variable[.part]/location
  • XML Schema variables in Web services activities—for WS-I doc literal style service interactions
  • Locally declared messageExchange—internal correlation of receive and reply activities Clarification of abstract processes

Looking even beyond BPEL 2.0, areas where we see additional evolution for BPEL and related standards include explicit support for human workflow as well as for subprocesses. (For more information about BPEL 2.0, see this Technical White Paper from "The New SOA Standards" series on OTN.)

Human Workflow

BPEL was designed for interactions between services that are modeled as Web services. This means that it doesn't say anything about human interactions and manual workflow steps. However, because BPEL has rich support for asynchronous services, it is possible to implement a human workflow service that makes people and manual steps look like any other asynchronous service from the perspective of the process. The advantage to this approach is that the BPEL process can stay 100% standard and yet both manual and automated systems interactions can be orchestrated easily. This approach is one that Oracle has taken with Oracle BPEL Process Manager's Human Workflow services.

With this approach, user interactions can range from simple scenarios, such as a manual approval step in a process, to a complex scenario, where data must be entered by the user before the process can continue. There is now an active standardization effort underway, called BPEL4People, which aims to standardize this methodology for incorporating human tasks into BPEL processes.

Business Rules

Business rules are a way to abstract specific actions or policies of a business into an externalized description. Using a development approach that externalizes frequently changing business logic into business rules, rather than embedding them in services and process flows, allows a business to remain agile in the description of its operations, definitions, and constraints. Businesses are able to respond to change more efficiently because the rules are centralized in one repository, managing them separately from core business processes.

Here are some example scenarios for when agility needs can be met by abstracting business rules:

  • Business logic that changes regularly or rapidly (for example, pricing rules and promotions)
  • Complex decision logic that would be painful to implement with languages such as BPEL or Java
  • Regulatory requirements

Monster Worldwide determined that the Oracle SOA Suite was the right solution in large part due to the tight integration of its component products. One of those components is the Business Rules Engine, which when used in conjunction with BPEL allows the compartmentalization of our business rules. For example, Monster uses business rules for the tax computations of sales order line items. Monster's rules also include business-specific rules such as those that describe where jobs and résumés need to be posted on the job search site.

There are two primary architectural approaches for including a business rules engine in an SOA. Business rules can be implemented in a utility service layer. These rules are generically designed and provide decision services that can be leveraged from many places. Business rules can also be implemented as services with very specific business behavior. Whether a utility service or a business service, when business rules are included as part of an orchestrated process, they can dictate the logic and flow behind the processes in a much more flexible fashion than if they were hard-coded in a procedural programming language. We recommend as a best practice that organizations think explicitly about where business rules may be appropriate for their business processes and use a business rule product to define and execute those rules with business user—friendly editing interfaces available when appropriate.

Design Patterns

A design pattern is a "solution to a problem in context"; that is, it represents a high-quality solution to a recurring problem in design. Design patterns have gotten a great deal of attention in IT over the past several years, and we are now starting to see the emergence of service orchestration design patterns. Here we describe several such patterns.

Error Hospital. This pattern provides exception handling as a service. To maximize process simplicity and performance, the business processes are designed as simple asynchronous or even short-lived synchronous processes. When there is an exception, messages are published to an error hospital process that then provides the necessary exception management, including human services if needed.

Many service orchestration patterns are implemented in orchestration engines such as Oracle BPEL Process Manager:

  • Composite Service. Combines the functionality of multiple services into a composite service and makes it available to consumers interested in it as a whole. BPEL supports this by automatically making every process available as a service that can be used as a building block to build higher-level, compound flows.
  • Orchestration Engine. Specifies the control and data flow associated with an orchestration explicitly, using a standard language such as BPEL that can run on an orchestration engine. The orchestration engine can provide infrastructure services such as audit trails, persistence of long-running flows, common exception handling patterns, and so on.
  • Service Compensation. Provides transactional support in a manner that is aligned with loose coupling, stateless services, and long-running processes.
  • Orchestration Instance. Creates and allows users to independently observe and manage concurrent instances of a business process.

There are a number of design patterns that Monster uses in the architecture of our integration middleware.

  • Guaranteed Delivery
    • How can a message sender make sure its message will be delivered to the relevant process in the BPEL orchestration engine, even if the messaging system fails?
    • Transactionally persist the messages to a data store before passing the message onto the BPEL orchestration engine. Each time, delete the message that was persisted prior to the previous process.
  • Message History and Audit Logs
    • How can we analyze and debug the flow of messages in a loosely coupled system?
    • Attach a message history to the message that lists all applications or components that the message has passed through since origination.
  • Splitter
    • How can we process a message if it contains multiple elements, each of which may be processed differently?
    • Break out the composite message into a series of individual messages, each containing data related to one item.
  • Message Store
    • How can we report against message information without disturbing the loosely coupled and transient nature of a messaging system?
    • Capture information about each message in a central location, separate from the transactional data stores of the underlying orchestration engine platform.
  • Asynchronous Request Reply
    • When an application sends a message, how can it get a response from the server?
    • Send a pair of request-reply messages, each on its own channel and correlated via message body data or WS-Addressing.
  • Aggregator
    • How can we combine the results of individual but related messages so that they can be processed as a whole?
    • Use a stateful filter to collect and store individual messages until it receives a complete set of related messages. Then, publish a single message distilled from the individual messages.

Testing Orchestrated BPEL Processes

Testing is both critical and complex when it involves long-running, asynchronous processes that orchestrate many services. To fully test a process, it is necessary to confirm the following:

  • The service meets functional requirements. Each service, individually, needs to meet the defined functional requirements. In addition, each process also needs to be individually tested against functional and quality-of-service requirements.
  • The process simulates all possible interactions between services with all possible return values, including exceptions, timeouts, retries, and so on. Testing should be done that will include all possible positive and negative use-cases together with the possible exception handling options. The process will scale in production. At Monster we believe that performance testing should simulate at least 200% of expected maximum transaction volume.
  • Modifications to shared services do not expose shared vulnerabilities. Ensure that changes in services do not harm the applications that use them.

Conclusion

Over the past few years, we have seen business process orchestration standards such as BPEL, and technology platforms such as Oracle SOA Suite, emerge and mature. As these standards and platforms have been adopted, a set of best practices and design patterns is beginning to emerge. We hope that articles such as those in this series will serve as a catalyst to encourage more customers to share their ideas and knowledge.

In this article, we have discussed a number of business process orchestration features and requirements that are common to many modern business applications: long-running processes and human workflow, reliable communication, use of Web services, process persistence, and compensation handling.

The strength of new standards such as BPEL and Web services is in their abstractions and platform infrastructure, which inherently provide these capabilities so that IT organizations can write less plumbing code. With these tools in hand, we believe that enterprises will be able to achieve more robust and flexible business process automation and monitoring than was previously possible.


Joan Lawson Joan Lawson is Director of Global Integration at Monster Worldwide. Her career spans individual contributor and management roles in Information Technology, Software Engineering, and Information Architecture.
Dave Shaffer Dave Shaffer s Senior Director of Product Management for Oracle. He has held consulting, product management, and software development roles over the last 15 years at a range of technology companies, including Collaxa, Apple Computer, NeXT Software, and Integrated Computer Solutions.

Send us your comments