Event-Driven SOA: Events Meet Services

by Guido Schmutz

Achieve extreme loose coupling within a Service-Oriented Architecture by using event-driven interactions.

Published July 2011

Introduction

This article explores the difference between a request-driven and an event-driven solution for the same problem and discusses the pros and cons of each approach. The article also illustrates a mix of event-driven combined with request-driven interactions. The case is presented in a small sample scenario based on a car rental company.

In modern software solutions, three basic types of interactions can be identified and defined [1]:

  • Time-driven - an agent, or a group of agents, initiates an interaction at a specified time. The interaction completes upon reaching the specified time.
  • Request-driven - a client requests a service from the server and waits to receive a reply from the server. The interaction is initiated by the client and completes when the server replies. The reply can be a result message or just an acknowledgment that the server has received and processed the request.
  • Event-driven - an agent initiates an interaction by creating an object describing an event. In general an event object does not specify which agents should read (receive) the event object, when they should read it, or what they should do upon reading it. The set of participants is open-ended, as are the time when the interaction terminates and what the agents will do upon reading the event.

In a traditional Service-Oriented Architecture (SOA) request-driven interaction styles are common and widely used. A SOAP-based web service that a consumer invokes is clearly a form of request-driven interaction.

Time-driven interactions are used when scheduling mechanisms are in place. In a SOA this can be achieved, for instance, in a BPEL environment using the various OnAlarm features on a Pick Activity or on a Process/Scope level.

Event-driven interactions are less common, although they can be a very helpful instrument for architects in order to increase the flexibility and agility of a service-oriented solution. This, in turn, is important in order to reach one of the core goals of a SOA: to increase business and IT alignment.

The use case of the sample scenario

The sample scenario presented here is based on a fictitious car rental company, RYLC Inc. The company has launched an effort to modernize its existing IT system to one based on a Service-Oriented Architecture. The next step on their roadmap is to create a new centralized Customer Management System. RYLC has selected Oracle SOA Suite 11g as its SOA platform.

The new Customer Management System will offer two service capabilities, one for creating a new customer and one for retrieving an existing customer. The capability for adding a new customer involves the execution of a business process, as multiple steps need to be fulfilled, such as checking the creditability of the customer, rating the customer, and finally inserting the customer data into the customer database. RYLC plans to use BPEL for the automation of the business process in order to get the necessary flexibility and align the process with the changes in the business.

RYLC's existing Accounting System must be informed when a new customer has been accepted by the Customer Management System. That will be accomplished using the Accounting System's existing SOAP-based web service interface, which will be added to the steps of the business process.

Implementation using request-driven interactions

Figure 1 shows the existing Accounting System on the right and the new Customer Management System on the left. These two systems belong to separate domains, according to the Domain Inventory Pattern [2].

schmutz-soa-eda-fig01
Figure 1: Integration Architecture Diagram showing the new Customer Management linked to the existing Accounting

Figure 2 shows the Service Component Architecture (SCA) Composite diagram with part of the implementation of the Customer Management System. On the left we can see that the two capabilities have been merged into one service and exposed. The CustomerManagement Mediator component represents the central entry point (applying the Official Endpoint Pattern [2]).

The "Create New Customer" capability is implemented by the NewCustomer BPEL component. This BPEL component references an external service offered by the Accounting System as well as the Customer Entity Service that wraps the operations on the Customer entity.

schmutz-soa-eda-fig02
Figure 2: SCA Composite with the implementation of the Customer Management system

Figure 3 shows the implementation details of the NewCustomer BPEL component. We can see that the logic is organized into different scopes. The first generates the unique ID of the customer. The second defines the logic for processing the customer data (i.e. check creditability, assign rating, invoke the Customer Entity Service to insert the customer) . Finally, the last scope (outlined in red) invokes the "Add New Customer" web service on the Accounting System.

schmutz-soa-eda-fig03
Figure 3: Implementation of the New Customer BPEL component

The Accounting System uses its own data model for representing a customer, which is different from the model used in the Customer Management System. Therefore we need to transform the data prior to invoking the service on the Accounting System (applying the Data Model Transformation Design Pattern [2]). This is done in BPEL using the improved Assign Activity, available since Oracle SOA Suite 11.1.1.4., which offers a much easier way to map multiple values than earlier versions of Oracle SOA Suite. The resulting transformation is shown in Figure 4.

schmutz-soa-eda-fig04
Figure 4: Applying the Data Model Transformation Design Pattern [2] by a BPEL Assign to map the different customer representation

However, BPEL is normally the wrong approach to do doing huge Data Model Transformations. A better approach is shown in Figure 5, where an additional Mediator component IntegrateAccountSytem is placed between the BPEL component and the external reference. The Mediator component will be responsible for the necessary match of integration logic between the Customer Management system and the Accounting system, such as the Data Model Transformation in this case.

schmutz-soa-eda-fig05
Figure 5: Adding an additional Mediator component to wrap the necessary integration logic

The Mediator component uses XSL transformation to map from one data model to another. The graphical representation of the necessary transformation in JDeveloper is shown in Figure 6.

schmutz-soa-eda-fig06
Figure 6: XSL Style sheet for the customer transformation from the customer to the accounting domain

Change in business and its IT implications

After six months, RYLC decides to add a new Car Rental System to the application landscape. This system, a commercial off-the-shelf solution (COTS), also needs to know about the new customers added to the Customer Management System. This can be accomplished using the out-of-the-box SOAP-based Web Service interface offered by the Car Rental System. Figure 7 shows the new Car Rental System in green, with the invocation of the "Create New Customer" web service from the Customer Management System.

schmutz-soa-eda-fig07
Figure 7: Integration Architecture Diagram showing the new Off-the-Shelf System added to the picture

In order to integrate the new Car Rental System with the Customer Management System we have to change the SCA composite and add the invocation of the Web Service to the BPEL orchestration.

As shown in Figure 8, we first have to add the external reference to the Car Rental System and then use an additional Mediator component to integrate it with the existing BPEL component.

schmutz-soa-eda-fig08
Figure 8: Integrate the second system with an additional Mediator component and link it to the BPEL component

The Mediator component will use its own XSL transformation to handle the difference in the data model for the customer data in the Car Rental System. Figure 9 shows that the Car Rental System needs less information than the Accounting System (compare to Figure 6).

schmutz-soa-eda-fig09
Figure 9: XSL transformation to the XML data model used by the Car Rental System

Now we can enhance the BPEL process logic to include the call to the Car Rental System, more precisely, the Mediator component wrapping it. As shown in Figure 10, an additional scope InformCarRentalSystem has been added.

schmutz-soa-eda-fig10
Figure 10: Adding the additional step to the orchestration

Adding the invocation of the additional system to the Customer Management System wasn't that difficult. Thanks to the flexible nature of BPEL, together with the usage of a Data Model Transformation in a Mediator component, it can be done in a declarative manner.

Table 1 evaluates the positive and negative effects of the business extension and its implications for the technical solution.

positive Declarative nature of BPEL makes the additional service invocation an easy task.
negative We have to change the customer domain whenever a new system needs to know about a new customer
  • This makes governance more difficult.
  • As we communicate across Domain Inventory boundaries, we probably have to involve people from different teams. They have to be available and must synchronize their road-maps, which might be difficult.
negative Customer domain needs to know about the different formats used by the two other systems/domains in order to implement the necessary Data Model Transformation.
  • Responsibility for the mapping lies on the Customer Management System side
  • If the service contract on one of the other systems changes, a change in the Customer Management System is also necessary.
negative While the Accounting System is integrated through a standard SOAP-based web service, from a Customer Management System perspective, the two systems are not as loosely coupled as one might wish:
  • If the Accounting System is unavailable, the Customer Management System will have to take care of the exception handling. In case of a recoverable problem, such as short network interruptions, one possible solution might be a simple retry configured through the fault policy framework.
  • If the Accounting System has different uptimes, we might have to add some sort of persistency on the Customer Management System to prevent message loss.
  • When communicating across Domain Inventories we should strive for maximizing loose coupling.
Table 1: Evaluation of request-driven integration of the two other systems


How can we fix the negative effects of the solution presented so far? Wouldn't it be better if the Customer Management System didn't have to inform all the interested parties about new customers? What if we could just signal that a new customer has been successfully added to the system at the end of the NewCustomer business process?

The next section answers these questions using an event-driven interaction style.

Better Decoupling through an event-driven interaction style

Oracle SOA Suite 11g includes the Event Delivery Network (EDN), a mechanism that simplifies communication through events, Figure 11 shows the EDN in the center, with the potential producers on the left and the consumers on the right. EDN can run either on top of an Oracle database or a JMS queue.

schmutz-soa-eda-fig11
Figure 11: Event Delivery Network (EDN) - from Oracle Event Driven Reference Architecture [4]

Figure 12 shows the revised integration architecture from Figure 1. The different systems/domains no longer communicate directly with each other. The Event Delivery Network is now part of the architecture and further decouples the different systems/domains. Communication is accomplished through a publish/subscribe mechanism, by which the Customer Management System publishes a NewCustomer business event whenever a customer is successfully added to the system. All interested systems can subscribe to that business event and be informed immediately after the event is signaled (implementation of the Event Driven Messaging Design Pattern [2]).

schmutz-soa-eda-fig12
Figure 12: Integration Architecture diagram showing the Event Delivery Network (EDN) between the different systems/domains.

In order to work with the Event Delivery Network, we first have to create the business event, i.e. the contract between the publisher and the EDN. No WSDL is needed for the business events in EDN. Events are defined using the Event Definition Language (EDL), an XML dialect proprietary to Oracle SOA Suite. There is no accepted standard (such as WSDL) for events, but an extension for event processing has been added to SCA specification [5].

Figure 13 shows the graphical representation of an event in JDeveloper, and Listing 1 shows the corresponding XML document.

schmutz-soa-eda-fig13
Figure 13: Graphical definition of the NewCustomer business event in JDeveloper

An event has a name and content defined by an XML Schema element. Our NewCustomer business event uses the Customer XML Schema element, so an event will always transport all information about a customer. As a result, our event is rather coarse grained. We will discuss later the granularity of events and why it might be a good idea to limit the amount of information sent as the payload of an event object.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<definitions xmlns="http://schemas.oracle.com/events/edl"
      targetNamespace="http://schemas.oracle.com/events/edl/CustomerEvent">
    <schema-import namespace="http://www.mybank.org/Customer" 
                   location="xsd/customer.xsd"/>
    <event-definition name="NewCustomer">
        <content xmlns:ns1="http://www.mybank.org/Customer"  
                 element="ns1:Customer"/>
    </event-definition>
</definitions>

Listing 1: XML definition of the business event in the Event Definition Language (EDL)


With the NewCustomer business event in place we can change the SCA composite so that it no longer communicates directly with the two other systems/domains. As shown in Figure 14, the two external references and the two mediators from the previous solution have been removed (compare to Figure 8). The BPEL component has been changed so that it signals the NewCustomer event, as indicated by the special flash icon on the right side of the BPEL component.

schmutz-soa-eda-fig14
Figure 14: SCA Composite with the BPEL component directly signaling the NewCustomer business event.

The signaling of an event from within a BPEL process is done through a (BPEL) Invoke activity: instead of invoking a service through a partner link based on a WSDL, we signal the previously defined business event, as shown in Figure 15.

schmutz-soa-eda-fig15
Figure 15: The NewCustomer business event is signaled by using an Invoke activity with the corresponding event.

There is still a need for an Assign activity setupEvent (shown in Figure 15, right before the Invoke activity) which sets up the payload of the business event as shown in Figure 16.

schmutz-soa-eda-fig16
Figure 16: Using an Assign activity to specify the payload of the business event (using the variable specified in the Invoke)

Alternatively, a Business Event can also be signaled using (again) a Mediator component, as shown in Figure 17. The flash icon on the right side of the Mediator again indicates the signaling of the NewCustomer event.

schmutz-soa-eda-fig17
Figure 17: Business Event could also be published by a Mediator component

As shown in Figure 18, the BPEL process invokes the service operation on the Mediator component, delegating the event publication to the Mediator component.

schmutz-soa-eda-fig18
Figure 18: BPEL component is invoking the service on the Mediator to publish the business event

This is all that is needed on the Customer Management System. We either signal an event directly in the BPEL process, or use an additional Mediator component. This also works if no consumers/subscribers are yet registered for the Business Event — the BPEL Process will signal the NewCustomer event to EDN, regardless.

Let's now add the consumers of the Business Event. For that we create two new SCA composites, one for the Accounting domain and one for the Car Rental domain.

Figure 19 shows the SCA composite for the Accounting domain with a Mediator component and the external reference to the Accounting System. The Mediator component has subscribed to the NewCustomer event, which is again shown by the flash icon, now on the left side of the Mediator.

schmutz-soa-eda-fig19
Figure 19: Mediator with the subscription for the NewCustomer Business Event

Upon subscribing to an Event the following properties have to be defined: the event name, the consistency, the run-as-publisher, and (optionally) a filter (see Figure 20).

schmutz-soa-eda-fig20
Figure 20: Properties of the subscription

Filter criteria can be defined to act on the contents of subscribed events. In our case the filter field is empty, so we subscribe to all NewCustomer events.

The consistency level defines the Quality of Service (QoS) of the event delivery. Table 2 documents and describes the three possible levels.

one and only one An event is delivered to the subscriber in its own global (that is, JTA) transaction. Any changes made by the subscriber within that transaction are committed after the event processing is complete. If the subscriber fails, the transaction is rolled back. Failed events are retried a configured number of times.
guaranteed Events are delivered to the subscriber asynchronously without a global transaction. The subscriber can choose to create its own local transaction for processing, but it is committed independently of the rest of the event processing. The event is guaranteed to be handed to the subscriber, but because there is no global transaction, there is a possibility that a system failure can cause an event to be delivered more than once. If the subscriber throws an exception (or fails in any way), the exception is logged, but the event is not resent.
immediate Events are delivered to the subscriber in the same global transaction and same thread as the publisher. The publish call does not return until all immediate subscribers have completed processing. If any subscribers throw an exception, no additional subscribers are invoked and an exception is thrown to the publisher. The transaction is rolled back in case of any error during immediate processing.

Table 2: Different levels of delivery consistency [3]

By using "one and only one" together with the right error handling strategy we can guarantee that an event is always processed successfully, either directly or through multiple retries. The major consequence of this choice is that it is no longer the burden of the Customer Domain to make sure that the notifications are successfully delivered. But it is the responsibility of each subscribing domain to make sure that the business event is successfully processed. If a given system cannot guarantee the same availability than the Oracle SOA Suite platform, it might be necessary to additionally store the business event before sending it to the system (i.e. in a JMS queue).

The SCA composite for the Rental domain is done in the same way. Figure 21 shows the three different SCA composites, one per domain, decoupled through the Event Delivery Network. An event signaled to the EDN is transported transparently to all parties subscribed to such an event.

schmutz-soa-eda-fig21
Figure 21: different SCA composites decoupled through the Event Delivery Network (EDN)

So what have we achieved with this event-driven interaction? Table 3 evaluates the pros and cons of the event-driven solution.

positive The Customer Management System is highly decoupled from the other systems:
  • it no longer has to inform all interested other systems via messaging that a new customer has been added
  • it does not have to care about situations where an interested system is not available
positive The Customer Management System no longer has to transform the representation of the customer to all the formats of all the other systems. It delivers the customer in its format in the payload of the event and the systems subscribing to the event are then responsible to transform it to the format needed.
negative Event-driven interactions are one-way only and can therefore not replace request-response interactions.
negative Subscribers are not durable; the Event Delivery Network (EDN) delivers a signaled business event only to a SCA composite (if it is up and running).
negative We have no fine control about what information is sent to which receivers. The entire customer representation is sent as a payload of the event. From a Customer Management perspective we no longer know the consumers, which represents consumers, which represent a loss of control. Gaining control is a question of the granularity of the event.
Table 3: Evaluation of request-driven invocation of surrounding systems


Granularity of business events

Similar to the issue of granularity when designing service capabilities, we also need to think about the granularity of events. What kind of information should the business event contain? Do we need the complete customer representation with all its information (such as addresses)? Or do we pass only the amount of information necessary for the event subscriber (if there is any) to request further information on demand? This question can impact both the performance of the system and the security of the data.

From a Customer Domain perspective we might not be interested in publishing all the information about a customer in the Business Event. If we send only the ID of the customer we are able to offer special service contracts according to the different consumer categories/security level s (applying the Concurrent Contract Design Pattern [2]), which a subscriber can then use to get additional information.

Figure 22 shows the change in the integration architecture if we send only the Customer ID in the event payload. The red arrows from the Accounting and Car Rental System are the service calls to the Retrieve Customer capability for the enrichment of the event information.

schmutz-soa-eda-fig22
Figure 22: Integration Architecture with an enrichment of the missing information

Listing 2 shows the event definition with the new Business Event NewCustomerMinimal which only uses the Customer ID in the payload.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<definitions xmlns="http://schemas.oracle.com/events/edl" 
    targetNamespace="http://schemas.oracle.com/events/edl/CustomerEvent">
    <schema-import namespace="http://www.mybank.org/Customer" 
                   location="xsd/customer.xsd"/>
    <event-definition name="NewCustomerMinimal">
        <content xmlns:ns1="http://www.mybank.org/Customer" 
                 element="ns1:ID"/>
    </event-definition>
</definitions>

Listing 2: XML definition of the NewCustomerMinimal Business Event in the Event Definition Language (EDL)


Figure 23 shows the subscription change in the SCA composite for the Accounting System. If we subscribe to the NewCustomerMinimal event, we first have to call the Customer Management System to get the rest of the customer information through the RetrieveCustomerByID operation. After that we can call the Accounting System as we did before.

schmutz-soa-eda-fig23
Figure 23: Subscription to NewCustomerMinimal event, enrichment and then invoking the Accounting system

Figure 24 shows the Routing Rules of the Mediator component. After a Business Event is received, the data of the event is transformed and then sent to the RetrieveCustomerByID operation. The result of this request (i.e. the customer information) is then transformed to the format accepted by the AddNewCustomer operation of the Accounting System.

schmutz-soa-eda-fig24
Figure 24: Routing Rules of the Mediator component showing the invocation of the RetrieveCustomerByID operation for the enrichment

Figure 25 shows the transformation, which is similar to that used in the previous steps (compare to Figure 6).

schmutz-soa-eda-fig25
Figure 25: Transformation of the enriched information to the format of the Accounting system

Figure 26 shows the SCA Composite of the Customer Management System with the implementation of the RetrieveCustomerByID operation and the delegation to CustomerEntityService to retrieve the data.

schmutz-soa-eda-fig26
Figure 26: SCA composite of the Customer Management System implementing the RetrieveCustomerByID operation by using the CustomerEntityService

This process also uses a request-driven interaction in order to do the enrichment and call the RetrieveCustomerByID operation? However, this time the request is initiated by the Accounting Domain rather than the Customer Management System. So the stronger coupling is on the consuming side of the event, in order to retrieve more information about the event. Thus, the coupling is inverted, and we basically mix event-driven with request-driven interactions to control system coupling.

Conclusion

This article presents two solutions for implementing the requirement to inform interested parties about the creation of a new customer in the Customer Management System.

The first solution illustrates a traditional service-oriented approach using a request-driven interaction. The systems involved are rather tightly coupled, which may lead to a set of problems in the sequel.

Replacing the request-driven approach with an event-driven interaction style, as illustrated in the second solution, helped to decouple the different systems and domains, leading to a more flexible and agile architecture.

Of course, you should not replace all your request-driven interactions with event-driven ones. Loose coupling is especially important at the boundary of the systems, i.e. between the service domains of your Domain Inventories, as Figure 27 shows.

schmutz-soa-eda-fig27
Figure 27: Decoupling Service Domains through events

References

[1] Mani Chandy and Roy Schulte, Event Processing - Designing IT Systems for Agile Companies (McGraw-Hill, 2010).

[2] Thomas Erl, et.al. SOA Design Patterns (Prentice Hall PTR, 2009).

[3] Antony Reynolds and Matt Wright, Oracle Fusion Middleware Developer's Guide for Oracle SOA Suite 11g Release 1, (Packt, 2010), Chapter 36 "Using Business Events and the Event Delivery Network."

[4] Oracle Reference Architecture - Event Driven Architecture Foundation, 2010. Part of the IT Strategies from Oracle Document Library.

[5] SCA Service Component Architecture Assembly Model Specification - Extensions for Event Processing and Pub/Sub, 2009.



Guido Schmutz is an Oracle ACE Director and Technology Manager for SOA and Emerging Trends at Trivadis, an independent IT consulting company based in Switzerland. He is a co-author of Service-Oriented Architecture: An Integration Blueprint   (Packt Publishing, 2010). He frequently writes on his blog at http://guidoschmutz.wordpress.com.