Rethinking Connectivity

by Quinton Wall
04/16/2008

Abstract

Businesses demand more from information technology departments than ever before. IT must find new ways to meet this demand—often with reduced budgets and limited resources. One such approach to achieving this seemingly insurmountable task is to invest in modern methodologies and strategies such as Service Orientated Architecture. Connectivity, a critical aspect of any IT solution, has received little attention of late.

This article presents a broader view of how organizations may approach connectivity through the definition of four dimensions: internal, external, industry, and hosted connectivity.

By focusing on these four dimensions and rethinking how to approach connectivity, organizations may be better prepared to capitalize on opportunities, modernization, and rationalization of connectivity points, and can establish a solid foundation for future growth through emerging trends.

Introduction

In recent years Service Orientated Architecture (SOA) has taken center stage as the predominate method for exposing and enabling existing systems as reusable services. In addition, SOA has been widely embraced for the construction of new applications in a standards-based, loosely coupled approach. Like many new technologies, there is a learning curve to productivity, and SOA has been no different. Gartner often refers to this stage of a technology's lifecycle as the "Trough of Disillusionment," where reality may not be living up to expectations.

Gartner now suspects that SOA is beginning to enter the "Slope of Enlightenment," which results in operational improvements and realization of promised benefits. Organizations are beginning to see return on their investment and repeatable success. Much of this success may be attributed to growing technical experience and maturation of SOA tooling, especially around management and governance. In the author's opinion, this success may be limited in longevity if organizations are unable to achieve success in the most basic aspect of SOA—that being enablement of existing assets, in particular through connectivity.

Wait. Before you fall off your chair and begin composing a response to my apparent lack of knowledge of a little piece of SOA tooling called an Enterprise Service Bus (ESB), it is important for me to set some terminology and, in particular, get on the path to thinking about connectivity a little differently.

So let's start with a definition:

According to dictionary.com, connectivity is "the ability to make and maintain a connection between two or more end points." Sounds logical enough, but the key words here are make and maintain. To make and maintain a connection, a number of aspects are involved, specifically items such as security, protocols, transports, and standards. These aspects influence but do not define the broader context on how the connection is used.

The definition above and, in particular, the distinction between make and maintain vs. how, is a fundamental difference that must be understood. This is where connectivity is germane to technologies such as ESB that do define the how through mediation, transformation, consumption, and enforcement. Organizations need to start to rethink connectivity in order to evolve beyond current paradigms.

Rethinking Connectivity definition
Figure 1: Rethinking connectivity definition


Four Dimensions of Connectivity

We've arrived at an understanding of how connectivity, as an abstract concept, is important to organizations looking to ensure longevity of SOA and modernization initiatives. It is helpful to identify four dimensions of connectivity. These dimensions are:

Dimension Name Description
1
Internal Connectivity Connectivity to internal systems including packaged applications
2
External Connectivity Business-to-business or business-to-consumer connectivity beyond enterprise boundaries and often leveraging trading partners. Multi-enterprise integration
3
Industry Connectivity Industry-specific connectivity such as SWIFT
4
Hosted Connectivity On-demand connectivity covering software as a service (SAAS) and the emerging potential of hosted internal connectivity

Although each dimension is distinct, many mature enterprises require a combination of all four in order to complete business requirements. One such example may be a payments gateway or a trading clearing house that requires external connectivity to other banks, internal connectivity to their payments and accounting systems, and SWIFT-formatted messages that are send to the SWIFT network. Although still in its early stages of broad adoption, hosted connectivity may also play a role in payment gateways. The remainder of this article will describe each of these categories in detail and show how they may be used in a modern connectivity solution.

As Carl Sagan once said, "You must know the past to understand the future." Figure 2 provides a high-level view of a typical enterprise and how they have traditionally addressed connectivity. In particular, consolidation of the connectivity dimensions described above is often sorely neglected, with duplicated and redundant solutions. This is especially true for B2B initiatives, which have been so prolific in their unsuccessful efforts to consolidate their connectivity solutions that a new term— multi-enterprise integration—was coined to describe the issue.

In my experience, SOA initiatives have focused primarily on consolidation of internal activities but have yet to expand the scope to the broader dimensions described above. Any modern approach to connectivity must ensure a consistent solution regardless of what dimension an organization requires. Without a holistic approach to connectivity, organizations are likely to struggle to truly achieve true effectiveness across their initiatives.

Traditional View of Connectivity
Figure 2: A traditional view of connectivity (click the image for a larger version)


Dimension 1: Internal Connectivity

In general, SOA initiatives have focused on internal connectivity activities where new and existing systems are exposed through standards-based interfaces, primarily Web services. SOA provides a unique approach to reducing proprietary connections that often proliferated through an organization as a result of EAI implementations and the attempt to morph vendor products into modern SOA solutions. For the most part, the mind shift from propriety to standards-based, composable services has been successful through ESB implementations and modernized EAI tooling. Current emphasis of these activities is now on repeating this success by expanding to enterprise-wide adoption. Although a long way from complete, the shift for internal connectivity is well understood. Where it has struggled however, is in the realm of packaged applications—those stalwart survivors of the mega-application era that very often are the lifeblood of an organization now matter how much we want to deny it.

So, if packaged applications are so critical to organizations, why has connectivity been mostly ignored? The reality is that many people believe that the problem has been solved through a word that raises fear in many IT organizations: adapters.

Traditionally, adapters provide last-mile connectivity to legacy applications including packaged applications such as ERPs and CRMs. It is important to recognize that adapters still play an important part in internal connectivity as adapter vendors provide the ecosystem for the long-tail economics of supporting multiple application versions, proprietary interfaces that upstream providers such as BEA, IBM, and Oracle rely on to go to market and confidently state, "Yes, we can connect to that." However, the majority of middleware vendors recognize the importance of such a statement but understand that the adapter business is not a viable business to be in due to the long-tail economics of support mentioned above. (One exception to this is Tibco which continues to invest in developing proprietary adapters.)

Almost all larger vendors OEM these adapters to ensure they solve connectivity to legacy and packaged applications but problems remain in this approach. In particular adapters struggle to address security requirements, performance issues and the ability decouple the implementation from packaged application change. This last point in particular is of critical importance as analysts (Forrester, 2007) suggest that over the next few years approximately 40 percent of organizations will undertake either a major or minor upgrade of their packaged applications.

So, do adapters solve internal connectivity? In my opinion, the answer is a resounding "No!" In the era of SOA and the importance of standards-based, composable architectures, architects and analysts must recognize the importance of connectivity as a horizontal layer that affects every aspect of an organization's infrastructure.

If this thinking is correct, then traditional adapters, with their point-based solution that offers little in terms of service orientation, are orthogonal to the needs of a service orientated environment. Internal connectivity must now be composable, reusable services that can be managed and monitored just as any other service.

Where does this lead packaged application connectivity then? While most connectivity offerings are certainly adapter-based, adapters are being phased out by Web services, we will see generic Web services being replaced by a different approach to service-oriented connectivity, an approach that builds on the ESB as the SOA backplane. ERP connectivity to major packaged applications such as SAP, Siebel, Peoplesoft, and Oracle Financials will be provided as highly performant native transports in ESBs; in essence, existing infrastructure is enriched and reused at an infrastructure level. BEA SmartConnect is one example where this has already happened, and others will follow suit. What this shift means is that all of the inherent benefits of an ESB-based solution such as reliability, scalability, and security propagation are brought to bear on solving the shortcomings of traditional adapters.

Such a decoupled-native approach demonstrates the importance of an organic connectivity layer that clearly separates the make and maintain from the how aspects you begin to be able to identify much broader notions of reuse than we have previously seen. In the past, reuse referred to one service being used in multiple places, but now we are able to leverage the ESB implementation, enrich existing infrastructure by implementing native transports and reuse this entire infrastructure for other uses previously delegated to proprietary, troublesome point solutions otherwise known as adapters.

Still think adapters completely solve internal connectivity?

As stated above, I would expect that proprietary adapters that have already begun to be replaced by Web services will shift toward ESB enrichment-based approaches that not only address the adapter issues described above but also address those limitations that are presented through the use of Web services such as performance, transactions. What we begin to see is the notion of composable services built on infrastructure assets to provide an increased, demonstrable return on investment.

Dimension 2: External Connectivity

When most people think of external connectivity, the answers are usually focused on B2B/EDI based data exchanges that required dedicated trading partner management consoles and protocols. Very often this connectivity grew out of EAI-based solutions that expanded their protocol support so they could handle the make and maintain aspects of these connections. Just as adapter-based solutions resulted in very little reuse, external connectivity was no different. Many organizations already have multiple B2B solutions based on siloed business need. There may be RosettaNet-based connectivity or perhaps even modern B2B solutions built on Web services that certainly solve the need. Sure, I get that, but the fact is, the problem is very similar to internal connectivity, with the solution even more similar.

I touched on this above, but the inevitable direction of standalone, external connectivity solutions is to wire these solutions together with the technology equivalent of duct tape, better know as Web services. The result of such endeavors will be termed multi-enterprise integration. While not entirely detrimental to an organization, especially considering that analysts suggest that B2B and multi-enterprise spending is estimated to rise approximately 50 percent by 2011 (Gartner, 2006), such an approach is likely to run into the same problems as adapters: Performance will suffer, security will be a nightmare, and management will require the introduction of SOA-based tools that focus on Web service management but will fall short of deeper analysis of message payloads and formats. Think for a moment how difficult it will be to track down an error in a RosettaNet or ebXML schema as it is transformed from one Web service to the next as the transaction moves from one B2B implementation to the next. Then consider the additional complexity when talking about long running transactions that may take days or even weeks to complete in their entirety!

So what is the answer?

Not surprisingly, the answer seems to be a connectivity layer built on an ESB backplane that is already supported and understood within the organization. This connectivity layer would be very similar to the internal connectivity layer described above, and is quite possibly an extension to the internal connectivity layer with the enrichment of specific transports designed for handling the make and maintains requirements of external connectivity. Remember: At this stage, an organization has already solved the how through their investment in ESB-related technologies and are now simply reusing and extending this environment. The result of such activity is a multi-enterprise system that already supports internal connectivity requirements and has been logically extended through partnerships with key vendors to provide a single solution for connectivity, regardless of the source.

To date I have not seen a solution that takes such a logical extension approach, although SAAS solutions certainly focus on the problem from a similar angle. However, just as rethinking internal connectivity begins to provide infrastructure asset-based service reuse, I would expect niche vendors who currently offer comprehensive external connectivity support (such as EDIFECS that offer EDI-based standards support) to begin to embrace the notion of an external connectivity layer and start offering these infrastructure asset services and enrichment through native transports built upon an ESB foundation.

Dimension 3: Industry Connectivity

Although very similar to external connectivity in that it interacts with business partners external to an organization, industry connectivity refers to those connectivity scenarios where industry-specific formats are critical to making and maintaining connections. A perfect example of this is within the financial industry, which often requires interactions with SWIFT networks for payment gateways. Messages need to be formatted in a particular fashion and are often interacting with proprietary systems that are de facto standards, for better or for worse. The similarity doesn't stop there with regard to external connectivity where organizations currently have multiple point solutions depending on the industry requirements, with very little reuse at the infrastructure asset service level.

As organizations mature I would expect the need for industry connectivity to become a differentiator that vendors offer to demonstrate their connectivity solutions. This is already apparent with BEA's Financial Services Edition, a SWIFT-certified solution leveraging ALSB but enriched to graphically support message construction and streamlined access to the SWIFT network. Further, I predict that the efficiency of pure Web service connectivity will begin to be questioned as we see a return to specific, optimized connectivity solutions. Industry-focused standards and transports will undoubtedly continue to be critical to any organization investigating a layered connectivity solution.

Dimension 4: Hosted Connectivity

I touched briefly on software as a service (SAAS) and believe we are seeing the convergence of external and industry connectivity—mainly under the guise of sales force automation through providers such as Salesforce.com. SAAS offers hosted solutions to connectivity that introduces a fourth dimension that is certainly compelling and important but in my opinion does not really address infrastructure asset service reuse. On the surface this may seem a negative, but when it comes to hosted connectivity dimensions, reuse is no longer as relevant as it has been in the past. Why do you need reuse when you no longer develop the solution? I have asked this question in the past: Which comes first—reuse or complexity? The important answer to this is to ensure that the solution you develop is as simple as possible and promotes reuse at all levels, including at the connectivity and infrastructure asset level. SAAS expands this model to include hosted asset reuse.

So perhaps no reuse is not too bad?

What makes hosted connectivity even more interesting in terms of its evolution is when you consider MuleSource's intentions in this space as well as recent acquisitions such as Cape Clear by WorkDay, an on-demand enterprise service provider. Providing services around hosted CRM solutions is certainly brewing here. When you consider the potential of hosted connectivity, including hosting internal connectivity (on-demand ESBs are certainly a reality when you consider WorkDay, Cape Clear, and, particularly, Cisco's purchase of Reactivity which provides appliance-based connectivity as part of the network infrastructure), rationalizing infrastructure asset-level service reuse and composition becomes increasingly important, as component-level reuse may no longer be as relevant.

Putting It All Together

We have defined four dimensions of connectivity and the evolving need to broaden discussions beyond individual dimensions to a more holistic approach. Figure 3 puts all of these pieces together in an approach that uses common foundational elements such as Enterprise Service Bus and enriches the infrastructure with additional capabilities to optimize each dimension. As organizations begin to introduce internal connectivity as a way of enabling existing assets extending to multi-enterprise integration (external + internal connectivity) the level of composition and integration asset reuse rises in parallel. Such efforts directly empower organizational drivers such as competitive time to market and business agility. Although a great amount of uncertainty exists about the fourth dimension—hosted connectivity—organizations that invest in rationalizing their connectivity via a layered approach are better equipped to leverage emerging opportunities that will present themselves through the maturity of hosted connectivity solutions.

In addition to the growing number of hosted connectivity opportunities, it must be recognized that no connectivity solution will ever be able to support all connection types. As such, a complete connectivity story must provide the ability to leverage adapters where appropriate and so easily create custom transports—preferably through a rich supportive development environment, typically supported through some form of connectivity software development kit (SCA). Connectivity should be open, flexible, and able to evolve; approaching it as a logical dimensions is the key to success.

Four Dimensions of Connectivity
Figure 3: Four dimensions of connectivity


Conclusion

In summary, as organizations evolve beyond redundant point solutions and recognize the shortcomings of traditional adapter approaches, an understanding of the four dimensions of connectivity is critical to ongoing success. Service Orientated Architecture and the introduction of the ESB as a SOA backplane provide strong opportunities for organizations to evolve their notion of reuse from component level to broader infrastructure assets. By enriching the ESB connectivity ecosystem, the introduction of logical layers or dimensions becomes a seamless extension of current service-oriented strategies.

Going forward, organizations must focus on rethinking their connectivity strategies to take advantage of emerging opportunities such as multi-enterprise integration and software as a service. The result of such a shift will enable organizations to leverage connectivity as a competitive advantage. Products such as SmartConnect and Financial Services Edition are an indication of where the industry is headed with feature-rich, extensible ESBs that provide the foundational connectivity building blocks in the form of native transports in favor of one-size-fits-all Web services. In the end, without a modern solution to connectivity, as a colleague of mine once said, "Everything else is just lipstick on a pig!"

References

Quinton Wall is a Sr. Product Marketing Manager for Integration at BEA where he is responsible for articulating the strategic vision and direction of the products such as WebLogic Integration and AquaLogic Integration.