SOA: Are We Reinventing the Wheel?
Pages: 1, 2

SOA and Distributed Technologies: Lessons Learned

From the discussion of CORBA and J2EE, you see that both these technologies provide a way to build SOA. There have been several successful deployments of these technologies across different industry verticals. However, if you were to evaluate reuse at an enterprise level even within those enterprises that have deployed these technologies, the level of reuse is not where it should be. It is not uncommon to come across enterprises in which  dozens of "services" do exactly the same thing from a business standpoint—validate credit cards, look up employees, and so on.

Why does this duplication exist? The duplication  is due to two major reasons:

  • Heterogeneity or multi-middleware issues: A typical enterprise usually will have multiple applications, tools, and technologies. The same is true for middleware: The Java camp may use J2EE; the Microsoft camp DCOM, and so on. What this means from the perspective of building services is that there are multiple ways to describe your interface (IDL for CORBA, MIDL for DCOM, Java for J2EE). Also, since each of these technologies defines its own transport protocol, there are multiple ways to consume these services. Though attempts were made to build "bridges" that tried to span the differences in programming models as well as the transport protocol between these technologies, the actual user experience was very poor: unintelligible mapping (try a Java-to-IDL mapping to get an idea), poor performance of such bridges, interoperability problems with vendors blaming each other. In short, neither the way to define nor the way to consume services is universal, which hinders interoperability and therefore reuse.
  • Complexity of these technologies: These technologies are all complex to learn; you need to learn a new API and a new programming model to be able to use these technologies. Building services is very coding intensive with these technologies, particularly CORBA. J2EE has done a better job in this regard, with support for a more declarative approach to building services. However, even J2EE is not for the faint of heart with its collection of more than a dozen APIs that a programmer has to master to be effective. The other part of complexity is in managing the lifecycle of these services, for example, deploying the service, versioning, retiring the services. These technologies provide poor support for this aspect with J2EE doing a slightly better job at it, having the benefit of experience with CORBA and DCOM. However, the question of effectively managing these services remains a point of much pain (and cost) in every enterprise.

So though these distributed component technologies provide explicit support to build SOA and are based on industry-accepted standards, these two factors above have hindered reuse at an enterprise level. Note that the actual situation is much worse because along with these standards-based technologies, an enterprise may have systems based on proprietary middleware technologies such as MQ and TIBCO. All these systems, both standards-based and those based on proprietary technologies, should be considered as an enterprise's IT assets since much investment has gone into building these systems. Functionality already within these existing systems must be exposed as reusable services from which new systems can be easily composed. Experience with CORBA and J2EE has taught us that this will not be solved by introducing yet another set of APIs or yet another transport protocol. What we need to do is to standardize on the message format and use transport protocols already prevalent in an enterprise such as MQ and IIOP. You will see in the next section how Web services address this requirement.

Web Services to the Rescue?

Web services, which are based on many of the same technologies as the Internet and the Web, address many of the issues with distributed component models. I assume that you already have a basic understanding of Web services. For those who want a quick primer, refer to the references at the end of this article. In this section, I examine how Web services address the limitations of distributed component models such as CORBA and J2EE.

  • Web services use HTTP as their transport protocol. HTTP is already widely deployed and understood, which not only helps in interoperability but also benefits from the operational efficiency of using an existing infrastructure. Note that Web services are actually independent of the transport protocol; HTTP is just one of the transports that can be used. You can just as easily use Web service with the transport protocols already prevalent in an enterprise such as MQ and IIOP.
  • Usage of XML: Web services use XML for describing the interface as well as the message that needs to be exchanged between the service and its consumer. You can use Web Services Description Language (WSDL) to describe the interface to a Web service. Though on the surface WSDL may look similar to IDL or Java interfaces, WSDL is a lot more powerful and has constructs that supports loose coupling of services. For example, WSDL supports multiple type systems so you can use it to describe services that use the type system of a mainframe system you may already have in place. It also supports multiple protocols and transports, so you could, for example, indicate that a particular service is available over JMS instead of HTTP. Finally, with WSDL, it is possible to define the operational behavior of a Web service in an abstract way and then bind it to specific ways to access it at different network end points.
  • Promise of reduced complexity: One of the reasons that the distributed component technologies are complex is that they dictate their own programming model, creating the need to learn yet another set of APIs. Web services, in contrast, do not dictate any new programming model; you can continue working with whatever environment you are already familiar with, and this may include J2EE or CORBA. This also means that you can use any programming language of your choice—C++, Java, Perl. Web services is strictly an interfacing and messaging technology.

As you can see from the above discussion, Web services do indeed solve many of the problems associated with distributed component models. Web services are the best fit for deploying an SOA.

We aren't quite there yet, however. As I alluded to earlier, when discussing the complexity associated with distributed component models, one of the major contributors to complexity is the challenge of supporting the entire lifecycle of a service, which involves not only building it but also deploying, versioning, securing, and managing it. We also need to broaden the definition of what a service is, to embrace what an enterprise has even if it is not based on Web services technologies such as SOAP and WSDL. This would include things such as integrations built out of messaging solutions like .NET, MQ Series, TIBCO, and legacy and packaged applications. To enable true reuse, we need to break down the silos that exist within an enterprise. We also need to make discovery and usage of these services a lot easier—it should move out the realm of the programmer to operational staff and even to the business analyst.

Service Infrastructure: The Emerging Solution

We see that the industry continues to evolve based on the experiences and lessons learned collectively. The above discussion should not be taken to mean that older technologies don't have any value or place in the enterprise. We still need object-oriented programming languages such as Java, distributed component technologies, and messaging systems to build new systems. As mentioned in the last section, the challenge is to enable reuse across existing technology silos by treating the functionality within these systems as services and enabling the creation of new composite applications that leverage these services. Though it is certainly possible to hand-build this with existing technologies, it is neither easy nor cost effective. Just as we have J2EE as the infrastructure to build applications, we need a similar infrastructure to compose, consume, and manage the lifecycle of services that span an enterprise. In short, we need service infrastructure (see Figure 1). The essential characteristics of such a service infrastructure should include:

  • Multi-middleware support: CORBA, J2EE, MQ, Tuxedo—these are all used in an enterprise for integration. The applications built on these products and technologies have themselves become a silo, so within an organization you may have multiple "domains" that don't talk to each other. Service infrastructure should allow an enterprise to expose the useful assets in these silos at the right level of granularity. Exposing such services should be made easy by having tools that discover the metadata in these systems and automatically create services out of them. All such services should be exposed in a uniform way, abstracted from the technology that it is based on; for example, I should be able to build a new composite application that leverages billing functionality based on MQ and call processing based on Tuxedo without having to learn anything about either MQ or Tuxedo.
  • Support for multiple transports: JMS, MQ, FTP, HTTP, and so on are all prevalent in an enterprise today and will be present in the foreseeable future. Service infrastructure should not only support these but also provide a way to extend them with any proprietary protocol or transport an enterprise might have. Additionally, the service infrastructure should do protocol switching directly, without forcing the services to use a common protocol like HTTP. This is important for some high-performance applications where the overhead of converting to either a canonical message or using a protocol like HTTP may not be acceptable.
  • Service publishing and discovery: Services have to be discovered to be able to be reused. Some form of registry is needed to discover existing services (the discovery may be based on some form of query or browsing) and to register or publish new services. Additionally, policies around these services may need to be defined and registered; such policies could describe things like cost of using the service and the security characteristics.
  • Service runtime and lifecycle management: This refers to monitoring of services for availability, failover of services, versioning of services, or retiring services. In short, superior operational support must be provided. In previous generations of integration and middleware products, management and operational support almost seemed like an afterthought. This caused considerable pain and cost to most organizations. Service infrastructure should make operational considerations a core offering. Another important aspect here is that since services could be reused in multiple applications, non-availability of a service can affect all applications that depend on it. Therefore special ways to handle versioning and upgrading of services should be provided.
  • Support for service composition: This refers to building new composite applications using existing services. Since the services themselves are application neutral and may be used in contexts that the original designer of the service may not have envisioned, support for transformation of messages, routing based on the content of the message, and introduction of some simple conditional logic must be provided.
  • Service security: Security obviously is a big concern, so controlling access to services based on some policy, auditing of usage,  triggering of alarms, and so on, must be provided. Also, since services can span applications that may have their own security model, integration with these security models must be supported to provide end-to-end security.
  • Ease of use: The service infrastructure should not itself introduce a new programming model or API. It should be highly declarative in nature with little or no coding needed. This should be provided with highly intuitive visual tools.

Figure 1
Figure 1. Essential elements of a service infrastructure

Are We There Yet?

Service infrastructure goes a long way in enabling a SOA and promoting reuse. This represents the culmination of collective learning by the industry to enable reuse and make building systems less complex. This will no doubt continue to evolve based on experience deploying new applications that leverage such an infrastructure. Many companies are legitimately concerned about the number of Web services standards, some of which are driven by various competing vendor camps. More vendor cooperation is needed not only to quickly converge on some important standards in the Web services and security arena, but also to address actual testing and validation so that interoperability of products is not left as an exercise for the customer.

I listed a long set of requirements for the service infrastructure. The good news is that several products on the market already support a majority of these features. However, it is very important to recognize that your enterprise will not have a service infrastructure that delivers value in terms of flexibility and agility just by virtue of buying a product that claims to support these features. SOA and the service infrastructure that enables SOA can be transformational and therefore needs changes in organization, process governance, and culture. The IT organization should enforce architectural consistency and align itself more closely with its business stakeholders to get support and funding as it undergoes the transformation. Processes should be defined as to how services are delivered and  how to manage the interdependencies of the participants in a SOA. Some services reused by most applications may be horizontal and generic; others may be specialized to a few select applications. A governance model that covers how these services are funded and developed is important. Finally, the culture should encourage and reward reuse of existing services instead of building everything from scratch.

If all this sounds daunting, it should—it is a daunting task. However, there are several pieces of promising developments. First, the Web Service Interoperability (WS-I) group is specifically charted to promote Web services interoperability across different implementations. Vendors do interoperability testing based on the profiles developed by WS-I. Though this may seem like an obvious thing to do, it was not the case with either CORBA or J2EE. It was worse with proprietary middleware and EAI products. The second piece of good news is that the tools on which to build a service infrastructure have become sophisticated yet easy to use. BEA's AquaLogic Service Bus product, for example, is entirely configuration-driven—almost all tasks can be done with a graphical user interface. Finally, SOA has already been deployed in many organizations where the benefits in terms of faster application development and cost benefits have been realized. The more mature IT organizations have captured such metrics and best practices around their implementations to learn and to demonstrate to their businesses how agile IT can be a competitive differentiator.

References

Nick Simha is the manager for the Systems Engineering group supporting partners at BEA. His interests include distributed computing, applicability of technology to solve business problems and Service Oriented Architecture. He holds a M.S. from the University of Missouri.