File transfer has been one of the most commonly used technologies across the world ever since it became practical to connect computers together with some mode of communication. Companies of all shapes and sizes have relied heavily in the past on this technology for the exchange of file-based information between different IT systems. In spite of the increasing importance of message-oriented middleware and the promises of service-oriented architecture (SOA), data exchange in the modern enterprise uses the simple process of transferring files. Recent government regulations such as HIPAA and Sarbanes-Oxley are calling for every process to be documented, auditable, and accountable, including those business processes that incorporate or leverage FTP. Therefore, it's becoming increasingly important for an organization's strategy to have enhanced file transfer capabilities as part of the overall integration strategy.
This article discusses how managed file transfer capabilities can be enabled in BEA WebLogic Platform for business-to-business (B2B) integration, and discusses some of the issues involved in handling large files for transformation.
File transfer has been used by companies almost since computing began. The simple concept formed a cornerstone of many companies' IT implementations. File transfer refers to technology provided to allow files to be passed from one application to another, with the other application running either on the same computing platform or on another system entirely. The logic customarily used is that as business applications are driven by files of data, which they process, then a way of linking two parts of the business together is to exchange files data the receiving application can process.
Traditional file transfer implementations generally have used a batch-style process. Information is gathered in one application/location/system and then passed to a different, possibly remote application for processing. For example, in a payroll processing system, time sheet information is gathered from all departments and locations within the company and batched together in data files, which are then transmitted to the corporate mainframe. There they are received and processed by the company payroll application, where payments are scheduled and pay-slip information is generated. This type of facility fulfilled a very basic level of application and system integration. However, it had some severe limitations. A number of these stemmed from the fact that when file transfer technology was created, most of the usage was in a batch style of operations. The new SEC-compliance requirement to Sarbanes-Oxley calls for every process to be documented, auditable, and accountable, including those business processes that incorporate or leverage FTP. It is becoming increasingly important for the organization's strategy to have managed file transfer capabilities as an integral part of overall integration strategy.
Until recently, there has been a single, standard response to the difficulties associated with bringing existing technologies into the SOA world—use one of the Web service enablement tools available to create a new interface to the existing system. Typical SOA architecture would look somewhat like Figure 1, where the service provider exposes a well-known interface, and the services are invoked either using SOAP- or REST-style messaging.
Figure 1. SOA using Web services
However, this SOA architecture may not always be possible or desirable, usually for one of two reasons:
When we consider existing transport protocols, in particular the business processes they support, we are confronted with a similar situation. In reality, many of these are still batch based, even to the extent of using FTP or email to distribute non-XML documents such as Excel files. Rather than disrupting the entire organization, an SOA implementation must accommodate and integrate seamlessly—without necessarily enforcing change on the business processes involved. Keeping these issues in mind, we'll discuss how WebLogic Platform helps incorporate SOA-based managed file transfer capabilities in the existing batch-based file transfer mechanisms.
Let's invent a business story from which we can create a sample file transfer-based SOA and explore how the file transfer-based SOA solution can be enabled in WebLogic Platform. This example demonstrates the features WebLogic Platform provides to implement a file transfer-based SOA solution. At times we have deliberately introduced simplifications to ease understanding in what should have otherwise been a real-world example.
Take the case of GlobalTx Inc., one of the top brokerage firms in the U.S. and a leading provider of operations support and trading services to trading partners. Its traditional paper- and phone-based system couldn't scale to meet demand. Therefore the company decided to implement an SOA-based B2B integration project to enable trading partners to place orders directly into the order manager system (OMS). One of the project goals was to support secure transfer of files containing these orders, encoded as per the FIX (Financial Information eXchange) specification over FTP. Other capabilities envisioned included the automation of file handling, the ability to handle large files for transformation, and efficient file management with versioning capabilities to prevent data duplication or loss of data. Figure 2 gives a snapshot of how one can implement SOA-based managed file transfer capabilities for GlobalTx. In the following sections, we will refer to this figure again to see how the GlobalTx requirements are made possible in WebLogic Platform by employing its SOA-based managed file transfer capabilities.
When you're building WebLogic Platform applications, Java controls provide a convenient way to incorporate access to resources and encapsulate business logic. Controls are essentially lightweight, metadata-driven component models. For more information on their usage, refer to the documentation. WebLogic Workshop has a lot of built-in controls accessible through the Data Palette present in the right-hand bottom corner. File control is one of the integration controls we will discuss. A file control can be created in a few simple steps as described in the documentation. But the basic file transfer functionalities provided by this file control by themselves do not meet the requirement of GlobalTx's proposed enterprise-level, SOA-based B2B integration.
For the file transfer to have any value in today's business environment, the following quality of service and flexibility measures are required:
The rest of this article examines each of these requirements in turn, and looks at how GlobalTx can implement its own SOA-based managed file transfer solution using the features provided by WebLogic Platform. We will detail the steps necessary to build a WebLogic Platform application that utilizes both WebLogic Integration (such as file control, file event generator, timer service, and message transformation) and WebLogic Portal features (such as content management) that meet our requirements. We will discuss the use of a special tool called Format Builder, available with WebLogic Platform, for defining transformation mappings from non-XML to XML. The only third-party component we will use will be for secured file transfer that can be built as a custom Java control.