As Published In September/October 2004

Southwest's Strategy for Success: Consolidate!
By Kelli Wiseth

As with its fleet of aircraft, Southwest Airlines takes a single-platform approach to its database, standardizing on Oracle9i RAC.

As the first commercial U.S. airline to compete on price, Southwest Airlines has brought many innovations to air travel since its first Boeing 737 left the tarmac, in 1971. Its no-seat-assignment boarding process, for example, provides efficiency on the ground, helping Southwest achieve quick turnaround and a superior on-time record. Another innovation and boon to efficiency was Southwest's introduction of ticketless travel, systemwide.

Southwest uses cost efficiency strategically, aggressively finding more-efficient ways of doing things. For instance, this year Southwest has begun equipping all new planes and retrofitting its fleet with "blended winglets," an aviation technology (a wingtip extension) that can improve takeoff performance, reduce engine maintenance costs, and has an estimated fuel burn savings of 3 to 4 percent per aircraft.

Four Keys to Successful Consolidation Whether you're consolidating servers, database instances, applications, or other IT assets, following these guidelines will help improve the chances of success. 1. Start with support from your executive management. As Don Lovett, BearingPoint managing director, points out, consolidation projects have all the trappings of large-scale ERP implementations, and as such, they require executive sponsorship, adequate resources, realistic budgets, and specific schedules, with milestones and well-defined deliverables. "Executive management support and governance are always key ingredients in any type of consolidation project," advises Lovett. Southwest DBAs acknowledge that executive support was necessary to the success of their consolidation project. "We have very committed management, completely backing our efforts," says Murugesan Paramasivam. "That made the entire experience easier." 2. Engage the process owners. It's important to recognize that instance consolidations, while seeming to be a technical problem, are also every bit a business issue, and the business process owners must be directly engaged in the consolidation effort. It's not uncommon, according to Lovett, for different divisions, on different instances, to have divergent processes. For example, an organization might have three different divisions on different instances, and over the years, the processes have grown apart. Lovett recommends that instance consolidation projects begin with a discovery phase, where various business leaders get together and understand how their processes have diverged and bring them back together in more of a common framework. 3. Plan for performance, and test, test, test. Performance planning, hardware testing, and software testing are very important. Identify the target goal for your consolidated instance, identify the metrics you will use to evaluate and test, and evaluate on an ongoing basis as you migrate instance data to the consolidated instance. BearingPoint's Lovett recommends that, before you begin the project, you consider performance by defining objectives for the instance. For example, how many users will it support? What will the service levels be? How much do you expect it to grow? Identify the performance metrics you will use to effectively evaluate the consolidated instance. As you move processes and applications to the newly consolidated instance, test to ensure that the functionality is as expected. And Lovett suggests creating solid test scripts and automating as much as possible. 4. Define proper objectives and appropriate time frames. The time that any given consolidation project will take depends on several factors, such as type of consolidation, the size of the organization, and the number of IT assets. Nonetheless, according to BearingPoint's Lovett, most organizations will want to see some tangible result in a three- to six-month time frame. "That result may be the completion in the first three to six months of a solid design with a well-defined execution plan, with an additional year to a year and a half to execute that plan. Or, for a smaller company, a three- to six-month time frame may result in a fully realized consolidated instance. For a large, global organization, the complete process may be a two- to three-year journey." Regardless, it's important to be realistic and set the right expectations and resource commitments. "Aggressive time frames are fine," says Lovett, "but just make sure they're realistic, so you're not disappointed."

At the heart of Southwest's success is its single-platform strategy: Its fleet consists exclusively of aircraft from the Boeing 737 line.

In Southwest's case, a common fleet significantly simplifies scheduling, operations, and maintenance. Training costs for pilots, ground crew, and mechanics are lower, because there's only a single aircraft to learn. Purchasing, provisioning, and other operations are also vastly simplified, thereby lowering costs.

Now, Southwest is planning on saving by consolidating in another area: the data center. The company began a database consolidation effort last year to create a common platform to support its enterprise applications.

Southwest isn't alone in looking to achieve savings from IT consolidation. In general, according to Gartner, Inc., data, "There has been a strong trend toward server consolidation since 1997, led by enterprises in the U.S., Canada, and Western Europe." Between 1998 and 2001, for example, the number of organizations with a server consolidation under way grew from 30 percent to 69 percent, according to a Gartner survey of its clients.

Rewards of Consolidation

Organizations are looking to consolidate servers, storage, network, applications, data, or database instances for a number of reasons, but cost containment and better utilization are often at the top of the list, according to John R. Phelps, research vice president at Gartner, Inc. Phelps focuses on server consolidation, and from that perspective, he defines three broad types or stages of server consolidation—logical consolidation, physical consolidation, and rationalized consolidation (or, simply, "rationalization").

Logical consolidation occurs at an organizational level, when all the enterprise servers are put under the control of a central IT group. "Logical consolidation doesn't reduce the number of servers you have—it just means that the organization puts all of its IT assets under the control of a central, controlling organization," says Phelps. The controlling organization can then start enforcing standards, implementing hardware and software asset management, and putting other best practices into place, Phelps explains. The next stage is physical consolidation, which aims to reduce the number of places where servers are located. Says Phelps, "It entails the colocation of multiple platforms at fewer locations."

After physical consolidation, organizations ultimately want to achieve "rationalization, in which you actually start reducing the number of servers by implementing multiple applications on fewer, more powerful platforms, often through partitioning or workload management," says Phelps.

Another type of consolidation project that can bring great benefits to organizations is database instance consolidation, says Don Lovett, managing director at BearingPoint, one of the world's largest business consulting and systems integration firms and an Oracle alliance business partner. With instance consolidation, organizations can gain "greater visibility into processes," says Lovett.

Instance consolidation is especially beneficial to global organizations, according to Lovett. "Global organizations that have different systems and processes in many parts of the world stand to gain more-effective, more-efficient business processes with a consolidated global instance," says Lovett.

In addition to achieving cost containment, organizations that consolidate instances want to be able "to support better data and transaction transparency," says Lovett. For example, many organizations want to consolidate database instances as a proactive response to the Sarbanes-Oxley requirements, HIPAA (Health Insurance Portability and Accountability Act of 1996), and product safety regulations, Lovett explains.

According to Lovett, divergent instances may exist in many organizations for historical reasons (legislative requirements at the time) or lack of ability to scale, for example. An instance consolidation initiative can give organizations the opportunity "to explore how putting everything together can conceivably improve processes," says Lovett.

"Although it's a journey that has a few bumps in the road, it's well worth doing. The people who have come out the other end to the planned state are enjoying lower IT costs and more-efficient and -effective business processes. They have a much better ability to train their people, because they have common practices. It certainly helps meet the goals of a global organization. These global firms traditionally had a lot of challenges in different parts of the world, having different systems and different processes. So the whole vision of a consolidated global instance is worth pursuing," says Lovett. "Twenty to thirty percent savings on IT costs and a similar amount of reduction in training expenses for most instance consolidation efforts is what we'd expect to see."

Lovett recommends "crafting consolidation initiatives as business projects supported by technology" for optimal return on investment. "We've seen people miss some of the ingredients of success when they approach consolidation strictly as a technical problem," says Lovett.

Consolidation starts with the "creation of an end-state vision. This is rapidly followed by a discovery phase—identifying existing assets—and an execution phase, where you try to realize that end-state vision through a variety of prototyping, testing, and data migration and conversion techniques," says Lovett.

"It's all about getting one database information source that can reside on a single physical hardware device, or with tools such as Oracle's Real Application Clusters environment, creating a virtual database made up of multiple pieces of hardware," says Lovett. "But the key is the single global database—that's what enables the consolidation to really provide benefits."

But, as suggested in the Oracle Information Architecture (see "The Oracle Information Architecture"), consolidation and standardization apply not only to servers or databases but to business processes as well. According to Oracle Senior Director of Technology Marketing George Demarest, "The more you can standardize common business processes and consolidate skill sets, the less complicated and costly it is to run your company. As we can see from Southwest's business model, this holds true beyond IT operations."

Southwest Consolidates on a Single Platform

The Oracle Information Architecture Companies such as Southwest have learned the benefits of standardization and system consolidation, and according to George Demarest, Oracle technology marketing senior director, it's an idea that Oracle has championed for years. Demarest points out that Oracle itself has consolidated many systems and adopted standards for its internal applications, platforms, systems, and procedures and now plans to share what it has learned with its customers. "Oracle has greatly benefited from a standardized, consolidated technology infrastructure, using our own products and e-business best practices," explains Demarest. "This marriage of technology and operations is the basis for the Oracle Information Architecture." Oracle Information Architecture defines the entire lifecycle of information, from the collection and protection of raw data to the smooth interoperation of enterprise applications, to the delivery of timely business intelligence that determines the value of information assets in the modern enterprise. These guiding principles and the technologies that enable them include Business processes: automated e-business operations based on consistent, timely information Data hub: an information-quality service that creates, maintains, and synchronizes standard enterprise definitions for entities such as "customers," "orders," and "employees" for all applications Development framework: a collaborative development environment that supports consistent application deployment,/span> Enterprise management: a centralized view of your entire system for management and optimization Grid infrastructure: an enterprise computing framework that delivers efficient, predictable, utility-like information processing across the organization Information access: consistent access to all enterprise information, applications, and collaboration from any device

At its headquarters, in Dallas, Texas, Southwest runs some 400 enterprise applications that support the gamut of its operational, financial, and administrative activities—everything from aircraft maintenance management systems to accounting; finance; business intelligence; data marts; and ground operations, including crew scheduling, reservations, and ticketing.

As an example, Southwest's service parts management (SPM) system is an extensive supply chain management system that facilitates provisioning and other activities among more than 1,500 parts and service suppliers related to repair and maintenance of Southwest's fleet. outhwest's portfolio of enterprise applications runs on several database management systems, including Adabas, IDMS, Informix, Oracle, SQL Server, and Teradata. According to Lionel Reynolds, manager of database administration at Southwest, "One of the primary motivations for our consolidation project, long-term, is to reduce the DBMS footprint at Southwest." For example, bringing all Oracle-based applications to a single release simplifies patching, maintenance, and support.

The consolidation effort started last fall, by migrating all the Oracle-based applications—about 25 percent of the 400—onto one central release of Oracle. Prior to the start of the consolidation effort, "We were running everything from Oracle8i Release 8.1.5, Release 8.1.6, and Release 8.1.7 and 8.1.7.4 to Oracle9i Release 9.2.0.3," says Murugesan Paramasivam, senior Oracle DBA at Southwest.

Selecting Oracle9i Real Application Clusters (RAC) was an easy decision for Reynolds and his team. "We decided to move to the Oracle RAC platform because our Interactive Marketing group had been very successful in implementing Oracle9i RAC for the Rapid Rewards and southwest.com system," says Reynolds. "Having witnessed that group's success with Oracle9i RAC, we just took the idea and went with it."

The approximately 100 Oracle-based applications that were running across various releases are now all running on Oracle9i RAC, Release 9.2.0.4. The team plans to roll out Oracle9i Release 9.2.0.6 by the end of this year, and then after that, Oracle 10g in the first quarter of 2005. Longer-term, the vision is to run an enterprise grid that supports all Southwest applications.

In addition to reducing the number of Oracle database releases, consolidating on Oracle9i RAC also provides much-needed high availability for Southwest's critical applications and was another primary motivation for the consolidation project, according to Reynolds.

Although Southwest's consolidation onto Oracle9i RAC is too new to provide any quantifiable data, such as return on investment (ROI) or performance statistics, Reynolds, Paramasivam, and others have witnessed firsthand the benefit of Oracle's clustering technology during a recent outage. According to Paramasivam, "One of the servers went down, and nobody noticed. We didn't receive one call from the system's end users."

In the past, continues Paramasivam, when a server node failed, it would take three to four minutes—or longer—for the database to come back up on the backup node. Now, with Oracle9i RAC, says Paramasivam, "you're talking seconds."

Southwest Senior Oracle DBA Tem Youngblood concurs: "When the server crashed, the UNIX engineering team called us, anticipating issues, but "the database connections failed over seamlessly onto the other nodes in the cluster."

How They Did It

Migrating close to 100 applications to a new environment required thoughtful planning and rigorous attention to detail. According to BearingPoint's Lovett, the typical consolidation project "has a lot of the characteristics of a large-scale enterprise resource planning [ERP] project" and requires the same planning, support, and execution (see "Four Keys to Successful Consolidation").

Southwest Airlines Company www.southwest.com Dallas, Texas, USA The largest U.S. carrier in terms of scheduled domestic departures, Southwest Airlines Company is the leading low-fare, high-customer-satisfaction airline in the U.S., serving short- and medium-haul city pairs (Oakland-Los Angeles and Oakland-Nashville, for example). Southwest operates 405 Boeing 737 aircraft and provides service to 61 airports in 31 states throughout the U.S., with Philadelphia, Pennsylvania, as the most recent addition to this roster. Southwest has one of the lowest operating cost structures in the domestic airline industry and consistently offers low fares, nonetheless enjoying more than three decades of profitability. Southwest by the Numbers Aircraft: Boeing 737-200, Boeing 737-300, Boeing 737-500, Boeing 737-700 Fleet: 405 aircraft Average age of aircraft: 9.6 years Flights per day: 2,800 Aircraft utilization: 7.2 flights per day, or 12 hours per day Employees: 34,000 Revenue passengers carried in 2003: 65,673,945 Number of enterprisewide applications: 400 Applications migrated to Oracle9i Real Application Clusters (RAC): 100 (as of June 2004) Oracle releases prior to consolidation: Oracle8i Release 8.1.5, 8.1.6, 8.1.7, and 8.1.7.4 and Oracle 9i Release 9.2.0.3 Oracle release after consolidation: Oracle9i Release 9.2.0.4 (planning Release 9.2.0.6 for year's end; planning for Oracle 10g in Q1 2005) Months to consolidate: approximately 8 (September 2003-May 2004) DBAs supporting Oracle: 6 Non-Oracle databases planned for migration to Oracle: 5 or more Size of Oracle instance(s) running on Oracle RAC: Range from 20GB to 1TB Hardware, OS, and clusterware details: 4 Sun Fire V480s, each with 4 UltraSPARC III Cu processors and 16GB memory 4 Sun Fire V880s, each with 4 UltraSPARC III Cu processors and 8GB memory Sun Solaris 8 Fujitsu PRIMECLUSTER V4.1A NetApp Filer FAS960 (storage) NetApp Snapshot Technology (for database backups) Oracle Education from any device

According to Youngblood, "We treated the consolidation process just as we would any new development project, by first enlisting the participation of the stakeholders—the business application people whose application we wanted to migrate—as well as the other teams we would be working with throughout the consolidation process."

Working together, the DBA team and the application team would run some jobs to shake out the issues and make sure everything looked OK, passing the application to the software application testing (SAT) team.

Once the application passed the SAT team's acceptance tests, it would go into production. Testing the Oracle RAC component involved shutting down nodes and the database to evaluate the application's responses.

"Every application had to pass through this series of tests individually and then in the aggregate," adds Paramasivam, "to ensure that they would continue to meet all expectations." And, Youngblood adds, "each DBA took a group of application teams; therefore, in parallel, we were able to accomplish this task in a reasonable amount of time."

Educating Stakeholders

Another important part of the process was educating all stakeholders about the benefits of Oracle9i RAC, as well as about the responsibilities that the respective application teams would have to assume in order to ensure that their applications migrated successfully to the new Oracle9i RAC environment. For example, an important part of the education process included telling the application developers about how to take advantage of transparent application failover (TAF) and other features of Oracle9i RAC, says Youngblood.

In many cases, application developers had little to do, but in some cases, some amount of minimal code tweaking was necessary to ensure that the application, when deployed on Oracle9i RAC, would respond gracefully to outages. Part of the reason, opines Paramasivam, is that the Southwest application portfolio for Oracle-based applications comprises a wide range of technologies. "We have all kinds of applications, ranging from in-house custom-built J2EE-based, to C++/XA, to PL/SQL, and in addition to that, we also have various third-party applications," all of which typically require different database connection methods.

For example, an application that used a hard-coded connection string to make a connection to the database application prior to migration to Oracle9i RAC would likely need some minor modification to be able to reconnect to a new node after a failure.

Nonetheless, says Reynolds, "Overall, we were very pleased with the ease of the process of migrating to the Oracle9i RAC environment."

Some of the bumps in the road were due to factors such as third-party applications that had not yet been certified for Oracle9i RAC. Says Reynolds, "Several vendors had to come on-site during this process to certify their application for Oracle9i RAC." This is one thing to keep in mind if your organization plans a major consolidation. In some cases, vendors simply hadn't had the capability to certify on an Oracle RAC cluster, so coming on-site to Southwest gave them that chance.

Looking Ahead

Southwest's consolidation of the approximately 100 Oracle-based applications took about eight months from start to finish. The schedule was aggressive, given that it was done with the six senior Oracle DBAs, who also had to maintain an operational steady state as well as manage new projects during the migration process. Reynolds credits his team of six senior DBAs—Sree Bammidipati, Alex Hwang, Kent Schneberger, Guillermo Solano, Murugesan Paramasivam, and Tem Youngblood—and UNIX engineers Zachary Lawson and Jason Norman with making the first phase of the consolidation such a success.

Next Steps DOWNLOAD Oracle Database 10g Real Application Clusters LEARN more about Oracle Real Application Clusters

Over the next several years, Southwest will continue to migrate the remaining applications to Oracle RAC. First on tap is the Crew Solutions, Southwest's enterprisewide scheduling system that lets flight personnel (pilots and attendants) bid for flight routes and shifts, schedule vacations, and conduct other operations-scheduling activities, taking into account union rules and other important business considerations. According to Reynolds, this application is one of the most critical to Southwest's day-to-day operations.

Migration for the other applications will be driven by the business application owners, and the DBAs who support the other, non-Oracle databases will be trained on Oracle, according to Reynolds. "Those DBAs bring a lot of expertise from those other environments, so we're going to retrain them in the Oracle environment," says Reynolds. The endgame is to have all the Southwest enterprise applications running on Oracle RAC clusters. "Oracle is the strategic database for Southwest," says Reynolds. "Just as with our fleet of Boeing 737s, the Southwest philosophy is to pick one model, one platform, to gain economies of scale."