Invoice Processing in a Service-Oriented Environment

Explore a process integration scenario implemented using Oracle Enterprise Service Bus, step by step.

by Ronald van Luttikhuizen

Published March 2008

Foo Inc. is a telephone company. Due to new legislation, greater customer demands, mergers, and fiercer competition, its profits have gradually declined over the last few years. Foo hires the business consultancy firm “U-Build & We-Advise Inc.” to help turn the tide. The outcome of the audit shows that better business-IT alignment can resolve the problems by resulting in both more-flexible business and more-flexible IT. The report furthermore states that an architecture based on Service-Oriented Architecture (SOA) can achieve these goals.

The firm decides to implement an SOA, and that enterprise architecture will be used as a means to structure and streamline this effort. An enterprise architecture team is set up, which reports directly to Foo’s corporate level. To avoid becoming an ivory tower and producing unnecessary paperwork, the team decides to model enterprise architecture on a need-only basis, thereby producing tangible results early. A meet-in-the-middle approach is used: while modeling business processes in a top-down fashion, the required and already provided services are inventoried.

Early in the project, a proof-of-concept is started—not only to get used to the new SOA technology stack but also to gain the trust of Foo’s more-skeptical employees. Ineffective and problematic activities are dealt with first. In this case, the problem concerns invoice processing for Acme Inc., Foo’s preferred supplier for consumer electronics. These products are used for promotional activities. Handling these invoices is error-prone because they are all sent by fax and processed manually. As it turns out, Acme is more than willing to cooperate in this project because it will cut costs for both companies.

This concept of an ecosystem in which organizations are helping each other based on mutual benefits is one of the cornerstones of service enablement.

Architecture

From a business perspective, invoice processing is a service, among others, used in the “Purchase-to-Pay” (procurement) business process. This process is depicted in the following Business Process Modeling Notation (BPMN) diagram.

Figure 1 The “Purchase-to-Pay” business process

As the figure shows, the process involves the following steps:

  1. A Foo employee makes an internal purchase request for some products.
  2. The order is placed at a preferred supplier.
  3. When the products are received, the supplier is paid and the requester is notified.

The activity “Pay Supplier” itself is a subprocess, depicted below.

Figure 2: The “Pay Supplier” subprocess

The process is initiated when an invoice is received. The invoice is then automatically validated. In the case of a valid invoice, the invoice is paid. Otherwise, a human workflow starts in which the invoice will be manually processed.

As the diagram shows, the “Pay Supplier” subprocess makes use of a service to receive invoices. In the case of Acme invoices, a service “Receive Acme Invoice” is needed.

If we drill down to this level of service design and implementation, we cross the border between process and technical architecture. Instead of BPMN, a more technical modeling notation, Unified Modeling Language (UML), is used.

The UML activity diagram below models the behavior of the “Receive Acme Invoice” service.

Figure 3 The “Receive Acme Invoice” service

When we look at the invoice service more specifically, we can define the following actions:

  • Files from Acme Inc. containing invoice information are read and archived.
  • The invoice data is translated from Acme’s format into Foo’s canonical format.
  • The transformed data is inserted into Foo’s invoice database.
  • Meanwhile an event is published indicating that a new invoice from Acme has been received. Every service and/or process that is interested in such an event can subscribe to it. This promotes loose coupling and asynchronicity, which leads to event-driven architecture (EDA). (See Lucas Jellema’s article “ Building Event-Driven Architecture with an Enterprise Service Bus” for more information on implementing an EDA.)

The activity diagram is split into an Acme-specific part (from start activity to transform activity) and a generic part that applies to all types of incoming invoices. This promotes both design-time and runtime reuse.

Why SOA?

Although SOA, like all new paradigms, is no universal silver bullet, a correct usage of SOA in the right situations provides numerous benefits. This case shows some of the promises delivered by service enablement:

  • Business-IT alignment and business value
    By using their business processes as a starting point for service development, Foo makes sure that software deliverables have concrete and direct business value that can easily be traced back to the business and its processes. Both business and IT stakeholders have a clear view of the architecture through the use of appropriate views and diagrams on each level, from business processes to subprocesses to services to implementation.
  • Flexibility and agility
    Services encapsulate functionality as building blocks with a clear contract (in other words, interface) without its users needing to know implementation details. As long as the contract remains the same, a service can easily be replaced without affecting related business processes. Concretely, if Foo has a better service for processing invoices, it can readily be plugged in to replace the existing service. Also, if the underlying implementation changes—for example, by not persisting data in the database but making it part of the process instance—the effects are only local to the service.
  • Reusability
    Runtime reusability is achieved through defining services as building blocks for processes and using open standards for ease of connectivity. Concretely, the same invoice service can also be easily used and invoked from other processes.

Why ESB?

As stated in the beginning of this article, the invoice processing service will be implemented using Oracle ESB, which facilitates implementation of more low-level services within an SOA and EDA based on open standards (XPath, XSLT, SOAP, JMS, JCA, and so on). ESBs are particularly well suited for virtualization, routing, and data transformation, thereby separating these concerns from the design, implementation, and execution of business logic and processes, which in the Oracle stack is typically done with BPA and BPEL tooling. In this use case, we primarily need adapter, transformation, and routing capabilities rather than business and process logic.

The position of the previously named components in an overall, high-level Oracle SOA platform is depicted below.

Figure 4 The Oracle SOA platform

Implementation

From a technical point of view, the invoice process is not entirely straightforward to implement; the remainder of this article will outline the particular steps needed to get it working. In short, this involves:

  • Reading files using an inbound file adapter
  • Routing and transforming invoice data using a routing service and XSL
  • Persisting data into master-detail tables using a database adapter and database sequencing

Event publishing is beyond the scope of this article.

Let’s get started with the implementation. Note that all sources can be found in the project files zip.

Preliminary Setup: Oracle Database, Oracle SOA Suite, and Oracle JDeveloper

The preliminary setup is documented in the accompanying setup guide.

Some interesting things to note about the installation and configuration procedures are:

  • It is important to know what settings are stored where. To illustrate this, we’ll have a look at database-related configuration. Global information is configured on the Oracle Application Server-level in connection factory settings: connection settings, username and password, XA-enablement (global transactions), use of sequencing, and so on. Mapping-specific (table-specific) settings however are configured within the ESB project using configuration files: database sequence names, whether or not to cache data for a particular table, etc. Knowing this makes bug fixing a lot easier. 
  • Pay attention to JNDI names you have specified during adapter configuration. These values must match those specified later on in JDeveloper. If these do not match, a new adapter connection factory using default values will be created runtime. This e.g. means that a database adapter will not use native sequencing.
  • The passwords for some adapter connection factories (such as the JMS adapter) default to welcome1 during installation, not to your oc4jadmin password. Make sure you change the password in case your oc4jadmin password is not welcome1.
  • Contrary to a native data source, a managed data source supports global transactions. When configuring the Database Adapter connection factory, the xADataSourceName is configured, not dataSourceName.

The ESB Project, Systems, and Service Groups

Now that the infrastructure setup is completed, we can actually start building the ESB flow that processes Acme invoices.

  1. In Oracle JDeveloper, create a new application workspace to hold your application. Right-click Applications in the Applications Navigator. Select New Application… and enter InvoiceDe mo as the application name. Accept the defaults and click OK to create the application. Select Cancel in the Create Project dialog box.

  2. Figure 5 Creating a new application in Oracle JDeveloper

  3. In the Applications Navigator, right-click the newly created application and select New Project…. Select General → Projects → ESB Project from the New Gallery dialog box. Enter InvoiceESB as the project name, accept all other defaults, and click OK. Oracle JDeveloper will create a new ESB project containing a jpr and an esb file. The esb file, named InvoiceESB.esb, is automatically opened in the ESB Designer.

  4. Figure 6 Creating a new ESB project in Oracle JDeveloper

    The next step involves structuring the future services in our ESB project. For this purpose, systems and service groups are used. Systems are mandatory top-level organizational units, and service groups are optional suborganizational units. This means that a service is either part of another service group or that a service directly belongs to a system.

    A system is a representation of a single application, proxy for an application, or a technical system. In this example, ESB systems are used to divide our enterprise into functional domains such as “Finance” or “Customers.”

    A service group can be used to further organize services much like Java packages or file system directories. A service group bundles a set of related services.

  5. Click the Create System/Group icon in the top of the ESB Designer. This will open the Create ESB System or Service Group dialog box. Select System, enter Finance as the name, and possibly add a description. Click OK to create the new ESB system definition.

  6. Figure 7 The Create System/Service Group option in Oracle JDeveloper’s ESB Designer

    Figure 8 Creating the ESB System “Finance” in Oracle JDeveloper

  7. Again, open the Create ESB System or Service Group dialog box. This time, select Service Group. This will show an additional option that allows you to select the system of which the new service group will be part. If the containing system is not yet set to Finance, then do so by clicking the flashlight icon. This opens the ESB Service Group Browser.

  8. Figure 9 The ESB Service Group Browser in Oracle JDeveloper

    Select the system Finance that was created in the previous step and click OK. Back in the Create ESB System or Service Group dialog box, enter Invoice as name, possibly provide a description, and click OK to create the new service group.

    Figure 10 Creating the “Invoice” service group in Oracle JDeveloper as part of the “Finance” system

You have now defined the structure in which the ESB project will be deployed. As a result, the Finance.esbsys and Finance_Invoice.esbgrp files are added to the project.

File Adapter Service

It’s time to create the actual services and adapters. Because all data flowing through the ESB is in XML format, we’ll first need to create an XML Schema describing the Acme invoice data (in other words, the data entering our ESB flow).

  1. Create a public_html directory in the InvoiceESB project directory and copy AcmeInvoice.xsd (located in the zip file) to this directory. By default, XML files are stored in an ESB project’s root directory, but for overview we’ll place these files in a separate directory.
  2. Create an inbound file adapter that will poll for new Acme invoice files. Make sure the ESB Designer is open. From the Component Palette on the right, select Adapter Services and drag the File Adapter onto the ESB Designer window. This will open the Create File Adapter Service dialog box.

  3. Figure 11 Creating an inbound file adapter

    Enter ReadAcmeInvoice as the service name and optionally add a description. Make sure that Finance.Invoice is selected as containing System/Group. If it is not already selected, use the flashlight icon to open the ESB Systems/Groups browser and select Finance → Invoice.

  4. Click the Configure adapter service wsdl icon, next to WSDL File, to open the Adapter Configuration Wizard and continue with the adapter service configuration.

  5. Figure 12 Step 3 of the inbound file Adapter Configuration Wizard

    In this wizard, use the following settings:

    1. Step 1: Accept the defaults.
    2. Step 2: Operation Type: Read File.
    3. Step 2: Operation Name: ReadInvoice.
    4. Step 3: Directory Names are Specified as: Logical Name.
    5. Step 3: Directory for Incoming Files (logical name): acmeInvoiceIn.
    6. Step 3: Select Archive processed files.
    7. Step 3: Archive Directory for Processed Files (logical name): acmeInvoiceArchive.
    8. Step 3: Select Delete files after successful retrieval.
    9. Step 4: Select File Wildcards as the name patterns specification.
    10. Step 4: Include Files with Name Pattern: AcmeInvoice_*.xml.
    11. Step 5: Select a polling frequency of 5 seconds and a minimum file age of 5 seconds.
    12. Step 6: Click browse and select Project Schema Files → AcmeInvoice.xsd → invoice. Note that this is the schema from the previous step. The example input file used later to initiate and test the ESB flow must be valid according to this XSD element.

    Figure 13 Selecting the XSD element for inbound Acme invoices

    A routing service for this inbound file adapter, called ReadAcmeInvoice_RS, is automatically added to the project when the file adapter wizard is completed. This adapter provides routing and transformation capabilities.

    Figure 14 The ESB Designer after creating the inbound file adapter service

    However, before continuing with the routing service, we’ll first complete the file adapter configuration by providing values for the logical names:

  6. Open the file adapter from the ESB Designer by double-clicking it. Expand the Endpoint Properties and use the green + icon to add a value. In the Endpoint Property Chooser dialog box, select acmeInvoiceIn. In the value column, enter a location on your file system from which the example input files will be read—for example, C:/Temp/Acme. This must be an existing directory.

  7. Figure 15 The Endpoint Property Chooser dialog box

  8. Repeat this step to provide a value for acmeInvoiceArchive—for example, C:/Temp/Acme/Archive. Processed files will be stored in this directory.

  9. Figure 16 The routing service after providing values for the inbound file adapter service properties

    You have completed the adapter configuration for reading Acme invoice files. Save all files by clicking the Save All icon.

Database Adapter Service

We continue by creating a database adapter that will insert invoice data into the master-detail tables.

  1. Make sure the ESB Designer is open. From the Component Palette on the right, select Adapter Services and drag the Database Adapter onto the ESB Designer window. This will open the Create Database Adapter Service dialog box.
  2. In this window, enter InsertAcmeInvoice as the service name and possibly add a description. Make sure that Finance.Invoice is selected as containing System/Group. If it is not already selected, use the flashlight icon to select the appropriate system and service group.

  3. Figure 17 Creating a database adapter

  4. Click the Configure adapter service wsdl icon, next to WSDL File, to open the Adapter Configuration Wizard and continue with the adapter service configuration.

  5. Figure 18 Step 2 of the database Adapter Configuration Wizard

    Figure 19 Step 3 of the database Adapter Configuration Wizard

    Figure 20 Importing master and detail invoice tables

    In this wizard, use the following settings:

    1. Step 1: Accept the defaults.
    2. Step 2: In the Connection drop-down list, select the invoiceDemo database connection you created earlier.
    3. Step 2: Make sure the JNDI name exactly matches the JNDI location you specified during the Oracle Application Server database adapter configuration. This should be eis/DB/invoiceDemo.
    4. Step 3: Select Perform an Operation on a Table as the operation type. Select Insert Only and deselect all other options.
    5. Step 4: Click Import Tables… to open a dialog box in which you can select the invoice tables. Click Query to get an overview of all tables in the INVOICE_DEMO schema (there should be only two tables). Select both the INVOICES and INVOICE_LINES tables and shuffle them to the selected widget. Back in the Select table dialog box, select INVOICES as the root database table because this is the master table.
    6. Step 5: The master-detail relationship should be visible in this window as two relationships between INVOICES and INVOICE_LINES; a 1:1 and 1:M(any) relationship. Finish the wizard by clicking Next and Finish.

    Some postconfiguration steps are required to use database sequences. Because database adapters are implemented using Oracle TopLink, we need to configure TopLink-specific settings. The adapter uses Oracle TopLink’s XML-Object (XO) and Object-Relational (OR) mapping capabilities to transform XML into SQL statements. When the following sequence settings are configured, the database adapter will automatically populate primary and foreign keys at runtime when inserting data in the invoice tables.

  6. Select the InsertAcmeInvoice TopLink map by navigating to Application Sources → TopLink → InsertAcmeInvoice in the Applications Navigator. Double-click it to open the map and change the following values:
    1. Select Use Native Sequencing.
    2. Set Preallocation Size to 1; this value must equal the increment by size of the database sequences.

    Figure 21 Modify the Oracle TopLink map to enable native sequencing

  8. Select the Invoices descriptor in the Structure window by expanding the InsertAcmeInvoice node. Change the following:
    1. Select Use Sequencing.
    2. Enter INVOICES_SEQ in the sequence Name field.
    3. The Table and Field values should refer to the INVOICES table and its primary key ID.

    Figure 22 Configure sequence settings for the Invoices TopLink descriptor

  10. Repeat the previous step for the InvoiceLines descriptor. This time enter INVOICE_LINES_SEQ as the sequence name.

  11. Figure 23 Configure sequence settings for the InvoiceLines TopLink descriptor

  12. Save all files in the ESB project by clicking Save All.

  13. If you compile the project by right-clicking it in the Applications Navigator and selecting Rebuild, you might get the warning TopLink found problems in InsertAcmeInvoice. You can resolve this warning by expanding the Invoices descriptor of the InsertAcmeInvoice map in the Structure window, selecting the invoiceLinesCollection mapping, and deselecting Use Method Accessing.

    Figure 24 Deselecting the “Use Method Accessing” option

    The warning should be resolved if you rebuild the ESB project.

Important:
Sometimes the TopLink sequence information might not be generated correctly. This can result in runtime errors indicating that no primary keys are created.

To verify whether this is the case, open the InsertAcmeInvoice_toplink_mappings.xml file located in the project’s root directory, in an external text editor. Search for the <toplink:sequencing> element. This element should appear twice—once for every invoice descriptor/table (INVOICES and INVOICE_LINES). If no such elements can be found, do the following:

  1. Open the InsertAcmeInvoice.xml file located in the src/META-INF directory.
  2. Search for the toplink:sequencing element within the <opm:alias> Invoices</opm:alias> element. This element is located between the toplink:descriptor-type and toplink:caching elements.
  3. Copy the entire toplink:sequencing element from the InsertAcmeInvoice.xml file to the exact same location in the InsertAcmeInvoice_toplink_mappings.xml file.
  4. Repeat this step for the toplink:sequencing element within the <opm:alias> InvoiceLines</opm:alias> element.
  5. Save the modified InsertAcmeInvoice_toplink_mappings.xml file.

…
<opm:alias>InvoiceLines</opm:alias>
…
<toplink:descriptor-type>independent</toplink:descriptor-type>
<toplink:sequencing>
  <toplink:sequence-name>INVOICE_LINES_SEQ</toplink:sequence-name>
  <toplink:sequence-field table="INVOICE_LINES" name="ID" xsi:type="opm:column"/>
</toplink:sequencing>
<toplink:caching>
…
<opm:alias>Invoices</opm:alias>
…
<toplink:descriptor-type>independent</toplink:descriptor-type>
<toplink:sequencing>
  <toplink:sequence-name>INVOICES_SEQ</toplink:sequence-name>
  <toplink:sequence-field table="INVOICES" name="ID" xsi:type="opm:column"/>
</toplink:sequencing>
…

Routing Service

As already mentioned, Oracle JDeveloper automatically created a routing service for the inbound file adapter. However, this routing service does not contain any routing or transformation logic yet. Let’s create this functionality to connect the inbound file adapter to the outbound database adapter.

  1. Open the ReadAcmeInvoice_RS routing service by double-clicking it in the ESB Designer. The routing service has a single operation, ReadInvoice, with no routing rules. (Note that the option Invocable from an external service is selected by default. This means that during deployment of the ESB project a Web service will be created that, if executed, invokes this routing service. This web service is listed in Oracle Application Server Enterprise Manager if you navigate to the Web Services tab of the OC4J instance in which the ESB is deployed. This feature can be used for testing purposes. If you do not require the routing service to be externally invocable you can deselect the Invocable from an external service option.)
  2. Add a routing rule by clicking the green + icon to the right of the ReadInvoice operation. The Browse Target Service Operation dialog box opens.

  3. Figure 25 The routing service with no routing rules for the ReadInvoice operation

  4. Select the insert service of the database adapter we created earlier as target.

  5. Figure 26 Selecting the insert service of the InsertAcmeInvoice adapter as the target service

    The Routing Service window shows that the insert is synchronously invoked after an invoice file is read by the file adapter.

    Figure 27 The routing service including a routing rule for the ReadInvoice operation

    You will need to create a transformation rule because the XML format of the inbound invoice files differs from the XML format expected by the database adapter.

  6. Click on the XSL mapping icon next to <<Transformation Map>> to create a transformation (XSL) file. In the dialog box, select Create New Mapper File, accept the default name invoice_To_InvoicesCollection.xsl, and click OK. This will open the XSLT mapper tool. You’ll see the Acme XML format on the left-hand side and the Oracle TopLink database XML format on the right-hand side.

  7. Figure 28 The XSLT mapper tool in Oracle JDeveloper

    The XSLT mapper enables you to visually define mappings from input elements to output elements by allowing drag-and-drop actions. The mapper also provides auto-complete functionality for input and output elements that (partially) have the same name.

  8. Try the auto-complete functionality by dragging the imp1:invoice element from the left onto the top:Invoices element on the right. The Auto Map Preferences dialog box appears.

  9. Figure 29 Auto-mapping input and output XML files by using the XSLT mapper tool in Oracle JDeveloper

    Leave the defaults of the mapper unchanged and click OK. As you can see, almost all elements are mapped. Only the following two modifications are needed:

    1. Remove the mapping from /imp1:invoice/imp1:invoiceLines/imp1:invoiceLine/imp1:productId to top:InvoicesCollection/top:Invoices/top:invoiceLinesCollection/top:InvoiceLines/top:id because the ID column of INVOICE_LINES is automatically populated by a database sequence. You can remove this mapping by selecting it and deleting it in the Mapper window.
    2. Supply a value for top:supplierName by selecting it, right-clicking on it, and choosing Set Text…. Enter Acme Inc. as the value.

    The resulting mapping should look like this:

    Figure 30 Final mapping

The project is finished and ready to be deployed to the ESB server and taken for a spin by testing it with a sample Acme invoice file.

Deployment

Deploy the ESB project to Oracle Application Server by right-clicking the project in the Applications Navigator and selecting Register with ESB. Oracle JDeveloper will inform you that the Finance system, Invoice service group, and adapter and routing services are created on the server.

Figure 31 Registering the ESB project with Oracle ESB

Figure 32 Registration successful

Open to the ESB Control to view the registered ESB flow. Select Start → Programs → Oracle – <SOA Suite instance name> → Oracle ESB → ESB Console. This will open the ESB Control in a browser window.

Figure 33 Oracle Application Server ESB Control

In the Services overview, drill down to view the Finance system, the Invoice service group, and the services that you have just created and deployed. (The ESB Control is documented in the Oracle Enterprise Service Bus Developer’s Guide .)

Testing

You can initiate and test the ESB flow by copying the AcmeInvoice_20071230.xml test input file from the InvoiceDemo/InvoiceESB/resources/input directory in the zip file to the inbound directory specified as the acmeInvoiceIn endpoint property of the inbound file adapter.

As a result, the Acme invoice data from the XML file is inserted into the invoice database tables. Verify this by looking in the database. Note that the primary and foreign keys are automatically populated.

Also view the completed instance in the ESB console.

Figure 34 An instance of the ESB flow completed successfully

You have designed and implemented a real-life integration scenario and tested it using an example input file.

Conclusion

Based on a real-life scenario, this article demonstrates how enterprise architecture can be used to identify and model services required from a business perspective. These services can then be used in the composition and execution of processes. This approach guarantees that services provide concrete business value and can be reused.

With this article, you have implemented such a service using Oracle ESB by composing multiple, already available adapters and routing services into a flow used for processing invoices. This is done based on open standards such as Web services and various other Java and XML-related standards.

Future additions for reading invoices from other suppliers can easily be integrated by merely extending this ESB flow. Only inbound and transformation services need to be added while functionality for persisting data and publishing events is already in place.

Next Steps

In our case study, we have further integrated this invoice service into the overall SOA environment by implementing another service whose purpose is to process all new invoices—not only Acme invoices. This service subscribes to new invoice events published by the read invoice service; validates the invoice; and, based on the outcome, automatically pays the supplier or starts a human task. This flow is universal for all types of incoming invoices. The supplier-specific logic is part of the validate invoice service. The above scenario thus implements the remainder of the “Pay Supplier” subprocess, as shown in Figure 2.

Some other, more general, issues to keep in mind:

  • Increase robustness by adding error-handling functionality.
  • Use Oracle ESB’s custom Ant tasks to automate deployment of ESB flows.
  • Think about high-availability, clustering, and security.
  • Complete the ESB System definition for Finance in the ESB Control; provide values for  Notification Details, Virtual Host, Port, and so on.

Ronald van Luttikhuizen ( ronald.vanluttikhuizen@approach-alliance.nl) is an architect at Approach Alliance ( www.approach-alliance.com), a Netherlands-based information and communications technology consultancy focusing on SOA and business intelligence.