Part 3: Orchestrating into End-to-End Processes
by Joan Lawson and Dave Shaffer
Build agile and standard business processes that orchestrate services into end-to-end business flows.
Published February 2007
The previous installments in this series described how to leverage the latest standards and architectural approaches to service-enable back-end applications and to publish those services on an enterprise service bus (ESB). But all those services are useful only in the context of the clients and business processes that access them. This installment focuses on how to build agile and standard business processes that orchestrate those services into end-to-end business flows.
A business process is a series of steps required to implement a business function. The business process may include system interactions and/or human interactions, within an enterprise or extending across corporate boundaries. Some enterprises merely document their business processes, for consistency or regulatory purposes. However, the past several years have seen an increase in the number of organizations that use IT systems to execute and monitor their business processes in an automated fashion. We believe that one of the key drivers behind this accelerating adoption of business process management (BPM) is the emergence of broadly accepted standards such as WS-BPEL for implementing and executing business processes.
Figure 1 Example BPEL process for order processing
In this installment of "Mastering SOA," we will describe how to meet the challenge of business process orchestration using these standards, incorporating experiences from both the Oracle vendor perspective and Monster Worldwide, with several successful Oracle Enterprise Service Bus and Oracle BPEL Process Manager service-oriented architecture (SOA) implementations in production.
Monster Worldwide has determined that BPEL is an important standard because SOA and BPEL offer the required agility and extensibility to be a global presence while meeting the local business requirements of customers. In addition, BPEL is driven by a standardization committee supported by several IT industry players, thereby minimizing the risk of vendor lock-in that would result from choosing a proprietary integration platform.
With Monster Worldwide's growing inventory of applications, it is necessary to interconnect these applications using Web services and SOA. It publishes services by exposing interfaces to existing systems in a standard fashion, leveraging Web services standards such as SOAP, WSDL, and XML Schema. Monster Worldwide then orchestrates these services into end-to-end process flows. BPEL is the industry standard for such orchestration.
In this article, you will find examples of how Monster is implementing SOA and BPEL, along with tips and tricks gleaned from SOA implementations at other Oracle customers. You will also learn about the features of the BPEL standard, how it fits with the other standards in the SOA space, and how the Oracle SOA Suite provides an infrastructure that implements these standards.
BPEL defines an XML-based language for specifying the behavior of business processes that are based on Web services. The first public BPEL specification was released in July 2002, as BPEL4WS version 1.0, combining IBM's WebServices Flow Language (WSFL) and Microsoft's XLANG specification. Version 1.1 was released in 2003, collaborated on by many industry vendors, and version 2.0 of the WS-BPEL standard is expected to be released from the OASIS standards body in early 2007.
The BPEL standard enables both interoperability and portability for an organization's business processes to a degree not previously possible. Interoperability enables business processes operating on different business process engines to interact with each other and with services published from many vendor and open source platforms. This is possible because BPEL is layered on top of the lower levels of the Web services stack—namely, SOAP, UDDI, WS-Addressing, WSDL, XML, XML Schema, and others. Interoperability in the arena of business processes has been possible for several years; however, the BPEL and Web services standards take it to the next level. But beyond interoperability, the concept of portability for business processes is truly something new. The process portability offered by BPEL ensures that business processes can be defined using tools that support the standard and then run on any compliant systems. Today, BPEL engines are available from large platform vendors such as Oracle and IBM as well as from smaller niche vendors and open source companies.
Choreography and Orchestration
Various activities are executed to implement the runtime lifecycle of a business process. BPEL is a Web services orchestration language, distinct from process choreography. Process choreography, as the term is commonly used in the IT world, describes the interplay of various trading partners to implement a multiorganization business function. For example, in the supply chain space, the fulfillment of a product purchase may involve the exchange of purchase orders, advance shipping notices, and money between two or more companies. Choreography does not describe how each company would conduct its operations, only how the different companies would interface with each other.
Process orchestration is when a central process coordinates the execution of different Web services operations. The central conductor (as in an orchestra) is aware of the overall goal of the orchestration, the operations involved, and the order of the operation invocation. This centralized management allows Web services to be added or removed without each being aware of its effect on others, as well as compensatory processes to be implemented in case of faults and exceptions.
Process orchestration and process choreography are similar only to the degree that they describe service interfaces and interactions. For the purposes of this discussion, we will focus on process orchestration. When you use BPEL for orchestration, the BPEL language provides a standard for controlling the overall sequence of operations, invoking services and executing on a BPEL server.
One of the most complex integration efforts at Monster Worldwide was the development of an order-to-invoice integration from Oracle's Siebel Customer Relationship Management (Siebel CRM) to Oracle E-Business Suite. The requirements consisted of splitting the original order message (a Siebel-generated object that consisted of both order and customer data) into two messages, transforming data in both messages, and conducting significant enrichment of the order data before sending the message into Oracle E-Business Suite. In addition, it was necessary to create the customer profile in Oracle E-Business Suite before posting the customer order. A series of BPEL processes were constructed to coordinate the execution of these steps. Once the Siebel order object is received, the first service splits the object into two messages, customer and order. The orchestration then enhances the order and customer data in parallel. When the order data has been transformed, it is then passed to the service that exercises the entire order data enhancement. When complete, the orchestration passes the customer message to Oracle E-Business Suite and awaits a successful response before passing the order message along. This orchestration assures Monster that all activity happens in the expected sequence and assures us that, if a fault is encountered during the process, proper compensation occurs.
BPEL is a powerful language that allows the development of complex business processes using Web services components. A BPEL process can specify which Web services should be invoked and the sequence in which they are invoked. A BPEL server tracks the business processes that are involved in a transaction; ensures that the steps are executed in the correct order; and manages transactions, compensation, and exceptions.
BPEL supports all the control flow structures you would expect from any orchestration language: loops and conditionals, variables, assigning data values, defining fault handlers, and much more. The BPEL receive activity is used to define the events or messages that will create an instance of a business process when a client invokes it for the first time through a service interface. The process will then invoke other services, which are the activities in the defined business process sequence. A response will typically be generated (reply).
Along the way, the BPEL process may have need to manipulate data variables (assign), handle faults and exceptions (throw), or wait for a set period of time. Finally, the process can end (terminate) at any time.
The element variable defines all messages and data used throughout the process description. Variables are used for every message sent to the involved services or received from them. They are utilized to handle the process execution data in the same manner as in any programming language.
Structural activities may include a BPEL orchestration that performs activities sequentially (sequence) or that performs activities in parallel (flow). The following are some of the most important control flow activities in BPEL:
BPEL can define the execution of business processes based on conditional behaviors. For example, Monster is separated into regions: North America, Europe, and Asia Pacific. Based on the originating region of the order, the BPEL processes execute regional-specific logic.
Similarly, Monster has defined the order-to-invoice BPEL flow to process the customer and order in parallel, while processing the order services sequentially.
BPEL processes orchestrate services (partnerLinks), which define the interfaces to the parties with which the process interacts. These partnerLinks include both services that are invoked by the BPEL process and the interface through which clients invoke the BPEL process itself. Flexible binding of BPEL processes to the physical service endpoints can be done at design, deployment, or runtime. For the services described above, the Siebel and Oracle E-Business Suite adapters are our key partnerLinks.
BPEL has rich support for asynchronous events. A BPEL process can wait to receive a message, or request, from a client (the same BPEL receive activity as used to initiate a new process instance) at any time. The BPEL engine will handle the overhead of persisting the state of the process until the message comes in and also correlating the message to a particular instance of the process, through either standard WS-Addressing or custom correlation mechanisms.
BPEL processes themselves can expose synchronous or asynchronous interfaces to their clients. Synchronous BPEL process operations block the client until the process returns a response to the client. Asynchronous BPEL interfaces can be one-way ("fire and forget") or return a response to a client via a callback. Asynchronous processes are more commonly used for longer-running operations, and synchronous processes are more commonly used for shorter-running operations.
At Monster, the order-to-invoice process as described above is a long-running, asynchronous process. In addition to the logic described above, the invoice number of the invoice created in Oracle E-Business Suite is returned via a callback to Siebel, and if an exception occurs, the exception handling process includes human workflow.
Exceptions and Compensation Handling
Often, the effort to implement the positive use-case for a business process (when everything goes right) takes only a small amount of the total effort of designing and implementing the business process. The majority of time is spent in defining all the appropriate exception handling logic to build a robust and resilient process that does reasonable things under all expected conditions (and ideally even under unexpected conditions). BPEL provides rich exception handling capabilities so that developers can define, in a hierarchical try/catch sort of pattern, what to do in case of exceptions and faults during the execution of a process.
However, beyond exception handling, some processes require more transactional behavior. In the world of short-lived flows and services, standard transaction models are sufficient, using XA or ACID transaction models where resources can be locked for the duration of a transaction and rolled back if necessary.
However, in a business process orchestration world, where services may be owned by external organizations and processes and services interfaces may take minutes, hours, or even days to execute, a new transaction model is required. These are called long-running transactions or compensating transactions. While the world is still waiting for true standards in this arena that allow transaction coordinators to handle compensating transactions automatically, the BPEL language provides support for this capability through a feature called compensation handlers.
Compensation handlers can explicitly undo work that was completed successfully prior to some error condition arising. Each invocation activity can, as needed, be paired with a compensation activity that undoes it. If a fault occurs, the compensation handlers for the already completed work within the scope of a failing activity are used to undo the completed work.
The downside to the compensation handler approach is that it requires the process designer to explicitly think about how to compensate each activity. On the positive side, the BPEL engine will then track all the activities at runtime and execute only those compensation handlers for the previously completed steps. Also, the BPEL engine will snapshot data during runtime so that the compensation handlers have access to the data that was current at the time the original activity completed. Finally, this approach means that the services themselves don't need to have any explicit support for compensating transactions—a good thing, because there are as yet no standards such as XA and JTA for compensating transactions.
BPEL 2.0 is the evolution of BPEL 1.1 as a public OASIS standard. BPEL 2.0 provides some significant enhancements, such as:
Looking even beyond BPEL 2.0, areas where we see additional evolution for BPEL and related standards include explicit support for human workflow as well as for subprocesses. (For more information about BPEL 2.0, see this Technical White Paper from "The New SOA Standards" series on OTN.)
BPEL was designed for interactions between services that are modeled as Web services. This means that it doesn't say anything about human interactions and manual workflow steps. However, because BPEL has rich support for asynchronous services, it is possible to implement a human workflow service that makes people and manual steps look like any other asynchronous service from the perspective of the process. The advantage to this approach is that the BPEL process can stay 100% standard and yet both manual and automated systems interactions can be orchestrated easily. This approach is one that Oracle has taken with Oracle BPEL Process Manager's Human Workflow services.
With this approach, user interactions can range from simple scenarios, such as a manual approval step in a process, to a complex scenario, where data must be entered by the user before the process can continue. There is now an active standardization effort underway, called BPEL4People, which aims to standardize this methodology for incorporating human tasks into BPEL processes.
Business rules are a way to abstract specific actions or policies of a business into an externalized description. Using a development approach that externalizes frequently changing business logic into business rules, rather than embedding them in services and process flows, allows a business to remain agile in the description of its operations, definitions, and constraints. Businesses are able to respond to change more efficiently because the rules are centralized in one repository, managing them separately from core business processes.
Here are some example scenarios for when agility needs can be met by abstracting business rules:
Monster Worldwide determined that the Oracle SOA Suite was the right solution in large part due to the tight integration of its component products. One of those components is the Business Rules Engine, which when used in conjunction with BPEL allows the compartmentalization of our business rules. For example, Monster uses business rules for the tax computations of sales order line items. Monster's rules also include business-specific rules such as those that describe where jobs and résumés need to be posted on the job search site.
There are two primary architectural approaches for including a business rules engine in an SOA. Business rules can be implemented in a utility service layer. These rules are generically designed and provide decision services that can be leveraged from many places. Business rules can also be implemented as services with very specific business behavior. Whether a utility service or a business service, when business rules are included as part of an orchestrated process, they can dictate the logic and flow behind the processes in a much more flexible fashion than if they were hard-coded in a procedural programming language. We recommend as a best practice that organizations think explicitly about where business rules may be appropriate for their business processes and use a business rule product to define and execute those rules with business user—friendly editing interfaces available when appropriate.
A design pattern is a "solution to a problem in context"; that is, it represents a high-quality solution to a recurring problem in design. Design patterns have gotten a great deal of attention in IT over the past several years, and we are now starting to see the emergence of service orchestration design patterns. Here we describe several such patterns.
Error Hospital. This pattern provides exception handling as a service. To maximize process simplicity and performance, the business processes are designed as simple asynchronous or even short-lived synchronous processes. When there is an exception, messages are published to an error hospital process that then provides the necessary exception management, including human services if needed.
Many service orchestration patterns are implemented in orchestration engines such as Oracle BPEL Process Manager:
There are a number of design patterns that Monster uses in the architecture of our integration middleware.
Testing Orchestrated BPEL Processes
Testing is both critical and complex when it involves long-running, asynchronous processes that orchestrate many services. To fully test a process, it is necessary to confirm the following:
Over the past few years, we have seen business process orchestration standards such as BPEL, and technology platforms such as Oracle SOA Suite, emerge and mature. As these standards and platforms have been adopted, a set of best practices and design patterns is beginning to emerge. We hope that articles such as those in this series will serve as a catalyst to encourage more customers to share their ideas and knowledge.
In this article, we have discussed a number of business process orchestration features and requirements that are common to many modern business applications: long-running processes and human workflow, reliable communication, use of Web services, process persistence, and compensation handling.
The strength of new standards such as BPEL and Web services is in their abstractions and platform infrastructure, which inherently provide these capabilities so that IT organizations can write less plumbing code. With these tools in hand, we believe that enterprises will be able to achieve more robust and flexible business process automation and monitoring than was previously possible.