Building Intelligent Processes with Insight-Driven Agility

by Matt Miller and Mark Simpson

Combining SOA and BI for Organizational Evolution

Part of the Oracle Fusion Middleware Patterns series.

Published August 2009

 Oracle SOA Suite
 Oracle Business Intelligence

Business processes are part of the DNA of an organization. They enable the organization to deliver service to its customers, executing the business model in a consistent way. As organizations change, their business processes have to change with them. Agility at this process level is essential — the ability to quickly adapt to changes in market conditions, regulatory requirements, or business models. Service Oriented Architecture (SOA) forms the foundation of process agility.

SOA enables business processes to be abstracted out of the underlying application systems, yet work with the functionality of these internal and external assets to deliver more agile processes. Separating out the business rules and making them available as services for the business process to consume is a second key enabler for agility in SOA.

Learning to exploit this agility in the most effective way and manage the resulting business change poses an additional challenge to an organization. This business change is driven by strategic decision-making, by evolving and refining the business model, and by the tactical necessity to react to both internal and external unforeseen factors. Business Insight delivered through a Business Intelligence (BI) toolset surfaces strategic and tactical business metrics with context to help make those decisions on how best to exploit the agility provided by the underlying SOA infrastructure.

Business Processes that are supported by SOA allow the organization to directly execute the business model and better support business change; BI provides measures that inform decisions for strategic and tactical change within an organization. Combining SOA with BI allows you to act on those measures, changing processes, services and rules to target identified improvement goals. SOA and BI are natural partners for a changing organization.

Business Intelligence meets Service Oriented Architecture

BI has for a long time been associated with SOA as a complimentary approach. However, this association has often only been focused at the boundaries of each - using SOA Data Services and Events to provide information for a BI Reporting schema or utilising BI tools to report on the usage and effectiveness of Business Services. The complementary nature of the two approaches goes even farther, and substantial benefits can be realised by weaving BI and SOA together to deliver business solutions that are both agile and highly insightful.

Business Patterns and Architecture for combining BI with SOA

This article focuses on three key patterns for combining these approaches to support Business Agility and meet the requirements of a changing business landscape. These patterns will form the basis of the use case introduced in the case study.

  1. Actionable Business Insight, using information in BI Dashboards to initiate SOA services that will take actions, such as moving loan profiles from one product to another if the dashboards suggest that a particular product is unsubscribed. The key enabler for this business pattern is the identification of thresholds, and building them into your BI Dashboards as guided alerts to identify business exceptions. These exceptions could be issues, such as an overloaded call centre, or they could be opportunities, such as identifying that a given call centre is performing exceptionally well for a particular type of product.

    The first stage of actionable insight is to provide alerts and context to users to identify the need for actions, and then present links that will allow users to take those actions. These links will invoke business services that could be a simple data update, such as changing the default call centre, or they could be services that execute a business process, such as a call centre product transfer process. These business services could also change business rule parameters, such as reducing the rating that indicates that a customer is seen as high risk if the BI metric for current risk exposure goes above a certain threshold.

    The second stage of actionable insight is to automate the triggering of an action. In this case a dashboard is only used to support the automatic decision to invoke a business service that runs a process or changes a rule. The dashboard may be issued to a manager to review and monitor the results of the automated change, or it may just be recorded as a snapshot to feed process improvement or internal and external process audits.

    Figure 1
    Figure 1: BI Metrics initiating a Business Service to execute a process
    This architecture pattern identifies thresholds for the business metric and invokes services that will execute a business process to exploit positive threshold breaches and mitigate the damage done from negative threshold breaches. This architecture pattern turns an informative BI metric into an actionable BI metric by combining it with SOA services.

  2. Insight Driven Process, allowing SOA driven business processes to introspect analytics through BI Services to determine the path through which a process should be taken. For example, if a BI Service shows that a particular branch is busy and not hitting conversion targets, the business process could route new loan applications to a branch with the capacity to fulfill the requests and generate a track record for the conversion.

    While the first pattern changes rules and process paths reactively at runtime based on an authorised BI alert, this pattern is built in at design time, and the BI metric is available to the process modeler as a way to guide the path through the process. At design time the business analyst considers multiple versions of the process - such as a high risk customer acquisition process and a risk averse version of the same business process.

    Figure 2
    Figure 2: SOA Process accessing a Business Metric through a BI Service to determine which path to take
    The second architecture pattern allows BI metrics to be used in a business process or decision service, or to be routed directly to a stakeholder as part of the execution of a process instance. This architecture pattern turns a narrowly focused process that is concerned only with the entity data flowing through it into a contextually aware process capable of reacting automatically to changes in the organizational situation.

  3. Context Aware Decisions, using business rules with organization-wide contextual BI information to drive business processes. This allows rules to take into account the wider state of the organization whilst making decisions on particular process instances.

    The isolated execution of rules based only on the data within the current instance is not how decisions are normally made within an organization. Business context is as important within decisions as the data contained within the individual instance of an entity. Decisions on whether to loan money are not just based on the applicant's age or ability to pay, but also on wider business metrics, such as current availability to corporate credit, risk profiles, projected interest rates, etc. Rules based on a narrow view of the instance alone are not enough to provide the real organizational agility.

    An addition to this pattern lies in the introduction of human workflow into an automated process. Stakeholders are required to interact with an automated process because they have some expert knowledge that cannot be automated as a business rule decision. This expert knowledge can be greatly enriched by the delivery of BI data to support the human decision. Therefore at any human touch point within an automated process, contextual BI dashboards should be present in addition to information regarding the current task instance.

    Figure 2
    Figure 3: SOA Process accessing a Business Metric through a BI Service to determine which path to take
    This architecture pattern allows Business Rules to operate on the insight gathered from the BI metric, allowing facts to be based on current organization and market context.

Technology Requirements for combining BI and SOA

In order for technology to support these architectural patterns, your BI tool must support actions and must be able to deliver BI services to an external consumer. Your SOA toolset must be able to consume these BI services, embed BI data and dashboards into any human workflow elements, and be able to access these BI services as facts within a rule engine or decision service.

Figure 4:Technology Requirements for BI and SOA Architecture Patterns

Building Insight, Actions, and Context into Processes

A few key principles must be adhered to when including decisions that are based on the wider business context when building processes that can react to immediate issues and/or opportunities:

  1. Process Design is an essential element, and must be modeled if you are going to use business context to dictate the path through a process or to react to alerts that are raised within the execution of a process. Your modeling will take on extra dimensions. For instance, you may model two routes through a process, corresponding to high or low risk exposure.
  2. Business Activity Monitoring is a complementary event-driven approach to identifying and taking immediate actions based on real time business occurrences. It is crucial in steering the business in the right direction. BAM and BI are necessary components in a complete solution for the provision of insight-driven business processes. BAM is intended to deal with the current state of operation, enabling a short leadtime in reacting to issues or opportunities. BI takes a more analytical approach to guiding business processes or adding current enterprise context to individual instances of a business process. To draw a sailing auto-pilot analogy, BI helps determine the path/time to the next waypoint based on average speed, distance, and past trips. BAM, however,will refine the current path based on large waves, sudden wind changes, and any immediate obstacles.
  3. Understand and model the actions you want to take on the BI insight. A metric is much more effective if thresholds and actions are defined. When modeling BI metrics it is important to think about the boundaries that will change the meaning of this metric (high, low, etc.) and then determine the business actions, normally defined as a process or rule change that should be performed to realign this metric. This adds key new dimensions to BI modeling.
  4. A consistent Data Model is required, and will be used by the business process, the business rules and the BI/BAM elements. Combining BI and SOA means that you are sharing data between transactional and analytical systems. Common data models must be produced to support this. The data model will take account of metrics, thresholds, alerts, actions, and transactional data, with alerts and actions performing the role of mediator between the analytical world and the transactional world.
  5. Appreciate the different types of information services and source data from the most suitable type. Data for a business process can be provided via a number of different categories of data services. For instance, transactional data services where the data is current and reflective of the current process; business metrics which are normally analytical views of the data summarized through a relevant dimension; or complex decision services, where the data is the result of business rules. Choosing how to implement the data service at design time is key to the successful combination of BI and SOA. That decisions includes determining which data sources are most relevant to be based on BI metrics and which should be retrieved from transaction data services.

Case Study - Insightful Processes for Vehicle Re-marketing at Motability Operations

The Vehicle Remarketing initiative at Motability Operations offers an excellent illustration of the value of combining agile processes with business insight.

Motability Operations is a not-for-profit public company that runs the Motability Car Scheme. The largest fleet operator in the UK and the biggest supplier of used cars to the trade, Motability Operations is constantly challenged to create innovative, cost effective and efficient ways to market vehicles for sale as these vehicles reach the termination of their leases.

As vehicles approach the end of their leases it is essential to push them to the most suitable sales channel, whether the channel is a Direct Sale Website, a Broker Network, an Auction Site, or a Consumer Campaign. In a highly competitive and saturated market it is essential that the marketing channel and pricing model reflect customer needs and evolve as the market changes. Motability Operations must have insight into the performance of the vehicles on these channels and of the current market conditions in order to avoid any market saturation whilst ensuring that the syndication and pricing of cars across the channels reflect buying behavior.

Business Scenario for Insight Driven Processes

Syndication Process and Rule Change based on Business Insight into Sales Channel and Market Performance

The business scenario looks to handle the market segmentation of cars by using BI and SOA to manage the mix of vehicle models and prices across the channels. The scenario uses insight provided by BI Dashboards to trigger alerts that will automatically make changes to the Business Process to rectify issues if the market is becoming saturated with a specific vehicle model. Extending this model, the business process is further enhanced to interrogate the BI objects containing information regarding the current performance of the sales channels in order to better inform its decisions and flows. This extra market information gives insight into how busy the channels are, the current volume of a certain car model residing on the channel, or information into the sales performance on the channel.

Figure 5 illustrates an insight alert that is triggered on the dashboard because the volume of one of the channels is falling below a pre-determined threshold. The alert guides the user to an action that will rectify the problem, in this case a rule change to allocate cars of a slightly worse condition to this channel in order to achieve an increase in volume.

Figure 5: Dashboard alert to trigger a business service that will alter
allocation rules in order to restore a channel to proper capacity.

Authorization is required to enable such a change to business policy. A workflow task is raised to notify a user that a rule change has been suggested based on Channel Insight. In order allow the user to make the most intelligent and informed decision, relevant BI metrics are delivered as part of the task. These metrics give the user the wider context around sales performance, current pricing against market average, and consumer behaviour on the channels. The user can then use the alert information with these additional measures to determine whether to accept the automatic change to the rules and process routing. This change cycle, incorporating business insight into the rule-changing process, will rectify the problem of low volumes on a given channel without affecting connected sales and profit metrics.

Figure 6: The task of authorizing a change to vehicle allocation rules is supplemented
with key contextual BI via the dashboard to allow the user to make a more informed approval

The next step is to automate the loop from business insight to business change. This can be done by automatically switching between different business process paths and business rule sets, as determined by the current market situation. If the business insight indicates that it is a quiet period for selling vehicles and that stock levels are high, an event could trigger an automated business change to switch to a low sale price business process and supporting rule set.

The Business Process for allocating cars to channels can utilise data sourced from a BI service to determine if the current market conditions and channel performance suit the particular instance of the vehicle the process is routing. This extra level of insight could result in the termination of the business process that was routing the car to that particular channel. In our business scenario we might see an example in which there are too many cars of a specific model on an Auction channel, saturating that channel and affecting sales volume. The Business Process would interrogate the BI metrics to determine the current channel stock situation and sales performance before syndicating the vehicles. Any vehicles determined to be unsuitable for the channel at this time would then be redistributed to the most relevant channel or queued until market conditions change.

Adding Business Rules to determine which channel is suitable based on the gathered insight enables automated redistribution, ensuring that the car is allocated to the channel that is most likely to produce the best price and quickest sale.

Figure 7

Figure 7 shows the BPEL process that calls a BI Service to gather the current market performance. The process will have already determined the channels the vehicle is suitable for; this suitability is getting further enriched with the current situation of the channel. Using Business Rules based on this market insight enables the allocation process to take the channel situation into account when routing the car.
Figure 8
Figure 8

Figure 8 shows a dashboard view of the cars for sale on each channel. The same information is presented to BPEL as Channel Insight through a BI service. The data indicates an abundance of cars of a particular model (Vauxhall Astra), resulting in low sales performance.
Figure 9

Figure 9 shows how any new Astras allocated to this channel are disallowed. As a result, the vehicles flow down a different path of the BPEL process. A further reallocation process could be implemented, which would be invoked when the Channel Insight indicates that normal levels of the vehicle type have been restored. The vehicles could then be re-routed using BPEL and Business Rules.

Solution Architecture for Vehicle Remarketing

The solution architecture for the business scenarios presented here is based on Oracle SOA Suite and Oracle BI Enterprise Edition (OBIEE), with the following components:

  • BPEL (Business Process Execution Language) Process Manager for the automation of business processes for marketing the vehicles and for the Human Workflow element to authorize syndication changes.
  • Interactive BI Dashboards for visibility of the current channel and market conditions, with a focus on the BI Delivers tool to produce alerts, delivering contextual dashboards to users and initiating web services to start business processes and services to act on the information.
  • Oracle Business Rules to provide agility by allowing immediate changes in the channel allocation policy that will take action on the alerts, without requiring an an IT change cycle.
  • Oracle ESB to provide mediation capability to switch between business processes and rulesets dependant on the data provided by the dashboards.
  • Web Services access into the Enterprise Information Model to make the dashboard information directly accessible by the business process, rules and services.

Figure 10: Solution Architecture for Insight-Driven Vehicle Remarketing

In order to illustrate this architecture, let's walk through the solution components required for delivering the business scenario, describing the interaction between the components and any relevant technical implementation elements. The white numerals below correspond to the white numerals in Figure 10.

1. The business scenario starts with a guided alert within a dashboard. This alert informs the user that action must be taked to change an allocation rule in order to restore a channel that has fallen below normal capacity.

When a threshold is exceeded, a "Channel Over Capacity" iBot (an alert based on user specified conditions) is generated from the BI Dashboard. This iBot uses a BI Delivers action to invoke the Channel Allocation Rule Change BPEL process.

Figure 11

Figure 11: The BI Dashboard triggers a Business Process

2. As part of the alert-driven change of rules to route more vehicles to a particular channel, a level of human authorization has been built into the solution. The BPEL flow that is invoked to deal with the Channel Allocation Rule Change will gather further information on the current channel performance, identify the most suitable modification to the rules, and then would raise a task via theHuman Tasklist Workspace for an analyst to authorize the rule change.

The task will include information relating to the rule change and will also present the analyst with contextual dashboards that provide all of the required information in one place. This eliminates the need to search for information through multiple screens or queries in Business Applications, Process Management Engines, and BI tools.

Figure 12
Figure 12: Human Interaction within the Business Process via the BI Dashboard

3. In order to allow a business process to access the BI Metrics that will allow current channel volumes to directly affect the process, BPEL accesses BI Web Services using Oracle ESB to mediate the request and decouple the BPEL process from the BI Web Service call. This decoupling ensures that the Business Process in BPEL is shielded from any change in the structure of the BI objects.

The Channel Allocation BPEL flow will utilise a "Current Vehicle Saturation" web service to provide information from a Channel Summary BI object that shows the current sales performance for a channel before determining whether more vehicles of a specific type should be syndicated to that channel.

Figure 13

Figure 13: Web Service Access to BI Insight from the Business Process

4. In order to fully automate and close the solution loop, continuous alteration of rules can be implemented by using programmatic rule APIs to modify the structure, parameters, and boundary conditions of the business rules. This allows the the solution to evolve , as dictated by Business Insight.

An alternate solution to this continuous approach to rule change is to build Java rule facts (data objects that are asserted when a rule is executed) within the Business Rule definitions that map directly to BI objects provided by the Web Services. Therefore the rules will be based on facts, such as "lowChannelUtilization," that will hand off the collection of the data that determines whether the channel is utilized by the BI objects.

Figure 14

Figure 14: Business Rules Introduced for Insightful and Intelligent Automated Decisions

Solution Benefits for Motability Operations

This article has described how the combination of BI and SOA allowed Motability Operations to address the challenge of selling each vehicle on the right channel at the right time for the right price. This goal was achieved through:

  • Threshold Breaches alerts in Dashboards, triggering the execution of business services to rectify issues like the market saturation of a vehicle type.
  • Business Processes using BI insight to ensure the correct flow based on market and consumer behavior, such as the popularity of a certain vehicle on one channel.
  • Automated Business rules that make decisions based on business insight, along with transactional data to enable channel allocation and pricing rules that take market performance into consideration.

The architecture of this solution provides the ability to react to market changes and ensure that decisions on where to market vehicles are based not only on the vehicle details but also on the insight gathered from channel performance. This provides a wider market context in which to make vehicle routing decisions, and also allows closer alignment with how the business wants to operate.

When other factors --, such as cost of the sale of the cars for each channel, or the cost of transferring cars -- are taken into account and built into the solution, a comprehensive insight-driven allocation process is produced that can implement an optimal and cost-saving business strategy for marketing the vehicles.


Process agility with insight: a powerful approach for business solutions

The case study presented here illustrates the power of combining SOA with insight from BI. It also illustrates how your BI Dashboards can be greatly enhanced through actions that will invoke services and process. In such a solution there are a number of proven business and implementation patterns that will help with architecting the solution.

If agility is required within processes, it is the combination of SOA, Process Automation, Business Intelligence, and Business Rules that will provide the solution. This is a very compelling story to take to any Business Architect who wants to get the most out his/her processes and support any impending business change.

The authors wish to express their gratitude for Neela Chaudhari's assistance in preparing this article.

Matt Miller Matt Miller is Head of Business Analysis and Testing at Motability Operations. He is also responsible for delivery of the Vehicle Remarketing technology project detailed in the article. Throughout his career Matt has worked in wide variety or technical roles with several large media companies, including IPC Media, Associated Newspapers and EMI Music.
Mark Simpson Mark Simpson Oracle ACE Director is an Oracle ACE Director specializing in SOA and Middleware, and leads the SOA technology practice for Griffiths Waite. Mark has been an advisor on the deployment of SOA solutions at a host of leading organizations in the UK, and has led implementations based on Oracle Technology, including the first production Oracle SOA implementation in the UK and the first production Oracle Business Activity Monitoring implementation in the world.