SOA Best Practices: The BPEL Cookbook

Building a Web Services Network with BPEL
by Yves Coene and The Hoa Nguyen

A case study in how the European Space Agency used BPEL scopes, BPEL domains, and the Oracle BPEL Process Manager API to build a partner-friendly Web services network.

 Review complete BPEL Cookbook index

Downloads for this article:
 Oracle BPEL Process Manager and Designer

Buoyed by maturing Web service standards, more and more organizations are using Web services in a collaborative environment. BPEL is fast becoming the platform for orchestrating these Web services for inter-enterprise collaboration. BPEL offers the compelling benefits of a standards-based approach and loosely-coupled process integration to companies building an online marketplace or collaborative network.

Yet the exciting new capabilities offered by Web services carry some risk. In many cases, partner relationships break down or integration costs skyrocket if certain technical and administrative challenges are not addressed at design time:

  • Partners must agree well in advance to conduct business according to specific criteria. Transport protocol, purpose of the interaction, message format, and business constraints have to be communicated clearly.
  • Joining the network has to be an easy process; collaborative networks become successful mainly through growth.
  • Users must easily find business services at runtime, or the promise of services-oriented architecture (SOA) is largely lost. (Service repositories are useful for this purpose.) If developers cannot readily find and reuse services, the services essentially don't exist.
  • Partners should have the ability to monitor Web services in real-time. End users should be able to track the progress of a specific order, and trading partners diagnose a specific bottleneck within a business process.
These challenges are exacerbated when a collaborative network operates in a hosted environment. In that model, partners expose the functionality provided by their legacy applications into a Web service. This Web service is published into a centralized repository. The host is responsible for orchestrating the complex business processes, which in turn, leverage partner Web services.

In this installment of The BPEL Cookbook, I will explain the architectural considerations associated with these challenges, using a European Space Agency (ESA) project on which our team from Spacebel s.a. was involved, as a case study. We will also discuss how this project leveraged BPEL scopes, BPEL domains, and the Oracle BPEL Process Manager API to build a "partner friendly" collaborative network.

Overview of the ESA Network

The ESA has embarked on a strategic initiative to create a BPEL-driven collaborative network of service providers based entirely on open standards. This network, referred to as Service Support Environment (SSE) network, combines Earth Observation (EO) and Geographic Information System (GIS) services from third parties to provide value-added composite services. SSE is a growing network currently including 20+ partners spread across nine different countries.

As shown in Figure 1, SSE is a simple realization of BPEL-enabled network. ESA acts as an intermediary broker that supports process-based collaboration between different partners on Web services standards such as SOAP, WSDL, WS-Addressing, WS-Inspection, and others. This network operates in a hub-and-spoke topology: Service providers use Oracle BPEL Designer to contribute different types of Earth observation and GIS services into a centralized repository, thereby creating an ever-expanding catalog of services.

Figure 1 SSE architecture

SSE provides the necessary infrastructure to

  • Host and manage the central repository that serves as a catalog of available services
  • Register and search services inside the central directory
  • Execute the short-lived and long-lived business process inside the Oracle BPEL engine
  • Enable partners to monitor the execution of Web services using the Oracle BPEL Process Manager console
End users request specific services by browsing the catalog of available services. Upon request, SSE invokes the associated business process. This business process calls the Web service (which runs at the service provider location) to complete the request.

SSE allows synchronous as well as asynchronous models of interaction. ESA has leveraged the Oracle BPEL Process Manager API extensively to offer utmost flexibility and ease-of-use experience to service providers and end users.

Designing a Web Services Network

Open standards are changing the rules of integration. BPEL offers a process-centric approach to cross-enterprise integration whereby partner interactions are defined using BPEL process flows. This blend of SOA with BPEL provides a unique opportunity to build a loosely-coupled collaborative network.

Hub and spoke (the approach taken with SSE) is the widely used network topology wherein one organization establishes a connection with various partners. Alternatively, the network can adopt a unilateral peer-to-peer model. In this case, every partner provides a platform for Web services security and provisioning.

Now, let's examine the four areas of network design:

  • Setting up the interface relationship
  • Simplifying partner enablement
  • Creating a centralized service registry
  • Empowering partners and end users with self-service monitoring
Setting up the interface relationship. Collaborative network design begins with defining the rules of engagement. These rules specify the messages exchanged within a business process, the sequence in which these messages are exchanged, and the physical attributes of this message. In order to communicate properly, all partners have to be able to answer the following questions.
  • Purpose of the interaction—Is this request for a quote or purchase order?
  • Message format—How is the message encoded?
  • Vocabulary—How should our messages be structured so that other parties can understand and process them?
  • Business constraints—How soon should we respond to a request?
  • Communication channel—Should our messages be encrypted?
To help partners get those answers, ESA publicly published an Interface Control Document to define these terms. This document formalizes the technical integration rules that were established, refined, and validated during several ESA-funded projects. Message-based SOAP over HTTP or HTTPS for secure communication are the protocols for communication between the SSE server and service providers. (As of this writing we are analyzing the use of WS-Security.) Web Services Definition Language (WSDL) is the only interface contract binding all entities; the service provider has to create a WSDL file that describes its SOAP interface and makes it available for other partners. Some of the information contained in the WSDL file is fixed but the following information has to be provided:
  • Selected operations according to the selected interaction model (search, RFQ, order)
  • Physical location of the services
  • Import of service XSD Schema
To facilitate comparison of services, consistency, and message translations, ESA mandates the use of XSD Schema to express the XML payloads. SSE also requires the use of a master XSLT document to ensure consistency in the presentation layer. The template stylesheet has to be imported in each service as follows:
                               
<xsl:stylesheet version="1.0"
                                
xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:oi="http://www.esa.int/oi"> <!-- Import statements --> <xsl:import href="./SSE.xsl"/> <!-- Apply the template for the root element from SSE standard template --> <xsl:template match="/"> <xsl:apply-imports/> </xsl:template> ...
xsl:apply-imports processes the root node using the template rules imported from the SSE stylesheet. When registering the service, the service provider provides the URL of the XML Schema, WSDL, and XSLT file. The SSE imposes the use of document-style SOAP. This approach allows for detailed specification of service-in and output data and validation of incoming and outgoing messages using XML Schema.

The approach adopted by ESA offers a quick-turnaround solution for companies looking to start a Web services network with a limited number of partners. As the network size grows, new variants of message formats, communication rules, security, and transport mechanisms will need to be introduced.

Simplifying Partner enablement. The growth of any network relies on the ease with which new partners can join and participate. The following factors will have a large impact on this process.

  • Integration with hub—How do partners create and submit their Web services? Can the hub support different transport protocols?
  • Process management—Is there a proper delineation between different partner processes? Can a partner modify its business process without affecting the reliability of the entire network? Can processes be generated on the fly using metadata definition?
SSE has been successful in forging stronger ties with its partners by simplifying the technical demands placed on service providers. ESA accomplishes this goal by distributing partner connectivity software, partitioning the development platform, and automatically generating process flows on behalf of the partners.

For example, to speed up the integration process, ESA distributes SSE Toolbox, a free toolkit that acts as an interface between SSE and service providers' existing systems. SSE Toolbox, which is based on the Sun Java Web Services Developer Pack, provides an XML scripting language supporting various back-end integration mechanisms such as FTP, file exchange, JDBC, calling a Java API, HTTP, and so on. It also automatically generates the WSDL file that is required to register the service.

Different partners contribute multiple services into the central repository. Partitioning the development and deployment environment shields partners from each other's changes. SSE effectively leverages partitioning by using BPEL domains within the Oracle BPEL Process Manager. BPEL domains allow a developer or administrator to partition a single instance of the Oracle BPEL Process Manager into multiple virtual BPEL sandboxes. A BPEL domain is identified by an ID and protected by a password. When a service provider registers on SSE, the Oracle BPEL Process Manager API is invoked to automatically create a BPEL domain to hold service definition files.

Here's an example of a createDomain method:

                               
/**
     * Create new domain space for a service provider to hold her/his services workflow
     * definitions files in
     *
     * @param domainName The Id to identify the domain
     * @param password The password used to login to the corresponding domain
     * @exception RemoteException System communication error
     * @exception WorkflowException Thrown if any error happens on the server
         that prevent the delete
     *
     */
    public void createDomain(String domainName, String password)
        throws RemoteException, WorkflowException{
      if(ml.isDebugEnabled()) ml.debug("Enter createDomain
                                
(domain = " + domainName + " password = " + password); /** * check if the being created domain exist? */ try { Locator locator = new Locator(domainName, password); ml.info("Stop creating domain: " + domainName + " because it has already existed."); throw new WorkflowException("1019"); } catch (com.oracle.bpel.client.ServerException e) { ; } try { //obtain the domain admin password from the system configuration SSE.properties file String domainAdminPassword = SystemConfigurationInfo.getProperty (WorkflowConstant.BPEL_DOMAIN_ADMIN_PASSWORD); ServerAuth auth = ServerAuthFactory.authenticate( domainAdminPassword, "localhost" ); if(ml.isDebugEnabled()) ml.debug("obtain authentication ok"); // Create server object ... this is our service interface // Server server = new Server( auth ); // Domain id is "newDomain", the password is "myPassword" if(ml.isDebugEnabled()) ml.debug("create server instance ok"); // Map domainProperties = new HashMap(); domainProperties.put( Configuration.DATASOURCE_JNDI, SystemConfigurationInfo.getProperty (CommonConstant.DEFAULT_BPELDOMAIN_DS_JNDI)); domainProperties.put( Configuration.TX_DATASOURCE_JNDI, SystemConfigurationInfo.getProperty (CommonConstant.DEFAULT_BPELDOMAIN_DS_JNDI)); if(ml.isDebugEnabled()) ml.debug("create domain - ds jndi property key/value: " + Configuration.DATASOURCE_JNDI + "/" + SystemConfigurationInfo.getProperty (CommonConstant.DEFAULT_BPELDOMAIN_DS_JNDI)); if(ml.isDebugEnabled()) ml.debug("create domain - tx_ds jndi property key/value: " + Configuration.TX_DATASOURCE_JNDI + "/" + SystemConfigurationInfo.getProperty (CommonConstant.DEFAULT_BPELDOMAIN_DS_JNDI)); server.createDomain(domainName, password, domainProperties); if(ml.isDebugEnabled()) ml.debug("Enter createDomain (domain = " + domainName + " password = " + password); } catch( com.oracle.bpel.client.ServerException se ){ ml.error(se.getMessage()); if(ml.isDebugEnabled()) se.printStackTrace(); throw new WorkflowException("1018", se.getCause()); } }
In the implementation of the runBuildScript method below, the Oracle bpelc function is accessed via an Ant build script. The method runBuildScript invokes an Ant project file that in turn invokes bpelc to compile and deploy the service provider's BPEL process.
                               
/**
     * execute the ant script to build an Oracle BPEL process that implements the workflow.
     * The script also deploys the workflow to the service domains.
     * All input information is provided under the props at the input param.
     * @param props Contain all necessary properties used to build/deploy the workflow BPEL process
     * @throws WorkflowException
     */
    private void runBuildScript(String buildFilename, Properties props) throws WorkflowException{
        if(ml.isDebugEnabled())ml.debug("Enter runBuildScript(buildFileName = " + buildFilename +
        ", properties = ...");
        try {
          Project project = new Project();
          project.init();
          File buildFile = new File(buildFilename);

          if (!buildFile.exists()) throw new WorkflowException("1015");
          if(ml.isDebugEnabled()) ml.debug("ant build file: " + buildFile.getAbsolutePath());

          ProjectHelper.configureProject(project, buildFile);
          //prepare logger for the project build
          PrintStream out = System.out;
          BuildLogger logger = new DefaultLogger();
          logger.setMessageOutputLevel(Project.MSG_DEBUG);
          logger.setOutputPrintStream(out);
          logger.setErrorPrintStream(out);
          project.addBuildListener(logger);

          //set project properties
          Enumeration keys = props.keys();
          while(keys.hasMoreElements()){
            String key = keys.nextElement().toString();
            project.setProperty(key, props.getProperty(key));

          }
          // test
          //excute default target
          project.executeTarget(project.getDefaultTarget());
                  if(ml.isDebugEnabled())ml.debug("Exit runBuildScript
                                        (buildFileName = " + buildFilename +
                                  ", properties = ...");
        }
        catch (Exception ex) {
          ml.error(ex.getMessage());
          if(ml.isDebugEnabled()) ex.printStackTrace();
          throw new WorkflowException("1002",ex.getCause());
        }
  }

                            
In your Web services network design, you should consider using BPEL domains to segregate the process design and deployment platform for all involved parties. Here are some possible applications:
  • Partition a single Oracle BPEL Process Manager instance into a multi-developer environment. In this case the domain ID usually identifies the developer owning that domain.
  • Partition a single Oracle BPEL Process Manager instance into development and QA environments. In that case the domain IDs might be "test" and "QA."
  • Partition a single Oracle BPEL Process Manager instance into an environment that can be used by multiple departments or partners. In these cases, the domain ids are the names of the departments or partners.
Creating a central service registry. After you have defined the network relationships, partners are free to join and contribute their services. The ability to publish and search these services, of course, is one of the underpinnings of an SOA platform.

In this process, partners publish their Web services to a central registry where all information about the service is managed. This central framework promotes service reusability and minimizes the effort and time required to locate a service. Lack of a central registry leads to inconsistency and chaos. In turn, it degrades the flexibility and openness of the network.

The SSE network leverages a central Web services repository heavily; partners use Web Services Inspection Language (WSIL) for discovery of available services. Service providers have the ability to reuse existing Web services (offered by other providers within the network) by selecting respective WSDL files. This task is accomplished by adding the WS-Inspection URL http://services.eoportal.org/inspection.wsil in the Oracle BPEL Designer tool configuration file UDDIProviderList.xml, as shown below.

<provider>
  <description>ESA SSE Portal</description>
  <type>wsil</type>
  <inquiryURL>
     http://services.eoportal.org/inspection.wsil
  </inquiryURL>
</provider>
The Oracle BPEL Designer at the service provider location connects to the SSE server and discovers the available services and their WSDL files using the WS-Inspection protocol. After connecting to the WS-Inspection server, the list of all available services is displayed, as shown in Figure 2.

Figure 2 List of available SSE services

For each service, a corresponding WSDL file and short textual description is provided. A partner link for this service is added by selecting the WSDL file. When the process flow has been built, it is deployed on the SSE portal using the BPEL console. (In its next release, SSE will integrate UDDI registry with Oracle BPEL. When a new service is registered, it will automatically be registered in this UDDI registry as well. This integration will facilitate service discovery by external tools, which now is only possible if these tools support WS-Inspection.)

Although you can build SOA and achieve many of its benefits without establishing a service repository, a repository is indispensable in the long term. Architecture can cope without a repository if the scope of the service is just one project. However, most enterprise scenarios are characterized by variety of services and partners, with most of them being in constant flux.

Providing self-service monitoring. In a collaborative network where multiple parties participate in a business process, monitoring the execution of a business process is crucial so that key performance indicators around business processes can be tied to service-level agreements. (For example, one of the key requirements to join the network could be to acknowledge the request for quote within two hours.) Monitoring of business processes can offer key insight into how long a specific business process instance takes to execute, why delays occur, and how the problem could be rectified for future transactions.

This level of diagnosis should be offered to the partners and the end users. By creating a self-service environment, partners and end users have the ability to track their individual processes and, in turn, alleviate the problems detected by a centralized monitoring framework.

The Oracle BPEL Console can be used to monitor and debug business processes (see Figure 3). ESA and service providers leverage BPEL console to track how many BPEL process instances are running and completed, the average duration of a process instance (to get detailed breakdown of time consumption within the process), and the process instance audit trail in text format, allowing partners to view intermediate results.

Figure 3 Monitoring business processes with Oracle BPEL Console

In addition, end users ordering a specific service have the ability to track the state of their order. This information is presented outside the BPEL Console using the Oracle BPEL Process Manager API. Service providers are encouraged to use meaningful scope names in their BPEL processes; at run-time, the portal extracts the name of the current BPEL "scope" using the IInstanceHandle.getStatus() API to present this progress information to the end-user.

Scopes are hierarchically organized parts into which a complex business process can be divided. They provide behavioral contexts for activities. By using meaningful scope names in BPEL process, it is possible for partners to track the status of "short-lived" and "long-lived" business processes.

For example, see the implementation of method getOrderSubstatus below. This method allows ESS partners to get the current status of the BPEL process instance using the name of the scope being executed in the bpel file.

                               
* call the Oracle BPEL API to get current status of the workflow instance, corresponding to
 * the ordered supplied at the input
 * @param ordered The order identified
 * @param workflowId The workflow name (or Id) that is processing the order stage.
 * Normally, there are two stages of an order: send(process)rfq and send(process) order
 * @param domain The domain that the order workflow belongs to
 * @param password The password used to login to the workflow domain
 * @return the current status of the workflow instance - particularly,
 * it is the name of the currentactive scope in the workflow BPEL file.
 * @throws RemoteException
 * @throws WorkflowException
 */
public String getOrderSubstatus(String ordered, String workflowId, String domain,
                                 String password)
        throws RemoteException, WorkflowException
 {
                if(ml.isDebugEnabled()) ml.debug("Enter getOrderSubstatus(ordered = " + ordered
                         + ", workflowId = " + workflowId + ", domain = " + domain + ",
                password = " + password);
                String status = "";
                try {

                        Locator locator = new Locator(domain, password);
                        IinstanceHandle instance = locator.lookupInstance(ordered + "_" + workflowId);

                        if( instance.isComplete() )  status = "Completed";
                        else status = instance.getStatus();

                        return status;
                } catch (com.oracle.bpel.client.ServerException e) {
                        // TODO Auto-generated catch block

                        ml.error(e.getMessage());
                        if(ml.isDebugEnabled()) e.printStackTrace();
                        throw new WorkflowException("1016", e.getCause());
                }

}

                            
Conclusion

The ESA has built its entire collaborative network with agility in mind. Interface relationships have been flexibly defined to facilitate easier adoption and rapid evolution. Its use of BPEL domains to offer an independent workspace offers massive flexibility to the service providers; providers can modify their business processes without affecting the stability of the network. BPEL scopes have enabled ESA to track the status of the in-flight processes at the micro level. BPEL process flows are generated automatically on the behalf of service providers. All these factors cumulatively contribute towards making the network more partner-friendly and minimize the upfront investment by the partner.

However, there are other areas on network design—such as distributed transaction management, security, and trading partner management—that we have not addressed here. As B2B networks grow in size, BPEL can take the lead in orchestrating private processes. For example, the Oracle Integration B2B product, which is interoperable with Oracle BPEL Process Manager, addresses public process choreography, handshake protocol support (RosettaNet, ebXML, EDI, HIPAA), message formats handling (MIME, SMIME, AS2, XMLDSig), and trading partner management (nonrepudiation, service level agreements, partner agreements).


Yves Coene Yves Coene currently works for Spacebel s.a. in Brussels as Project Manager. He has 15 years of experience in aerospace software projects such as Ariane 5, International Space Station, F16 MLU, and various other projects for the European Space Agency. Since 2001, he and his team have been responsible for the SSE project for ESA in Frascati, Italy.

The Hoa Ngyuen The Hoa Nguyen currently works for the SDC subsidiary of Spacebel s.a. in Brussels as senior software engineer. His main interests are J2EE, Web services, and workflow development with BPEL. Since 2001, he has been one of the lead engineers of the SSE project team at Spacebel and is also in charge of SSE software releases and on-site SSE software installations at ESA.

Send us your comments