Lessons from Going Global at Oracle
by Bret Fuller

Bret Fuller, Senior Vice President of Applications IT at Oracle, describes how Oracle consolidated its worldwide e-business systems (more than 70 instances) into a Global Single Instance, while standardizing its business processes.

It all started with a simple question posed by Larry Ellison some six years ago: "How many employees work for Oracle?"

Back then, the answer to that question required pulling data from over 70 countries, aggregating it, and coming up with a total—and by the time the process had concluded, the number had changed. The lesson was clear: having so many disparate systems—and not just for Human Resources (HR), but for Financials, Consulting, and Customer Relationship Management (CRM) as well—was not the way to get accurate and timely information within a global corporation.

And so began Oracle's journey to develop a Global Single Instance (GSI) from over 70 database instances, consolidating all its worldwide business data into a single global view. Today, using our global HR system, Larry can get a daily update of how many people work for Oracle (and in which countries), and can garner similar global data for Financials, Consulting, and CRM across all Oracle lines of business. Furthermore, the Oracle GSI Project has enabled Oracle to standardize its business processes and consolidate its services into four regional centers (in Ireland, Australia, India, and the U.S.), providing shared services for Oracle's global business far more efficiently and cheaply than before.

In this article, I'll focus on some of the business, process, and technology challenges Oracle faced—and the solutions it developed—during its six-year-long GSI consolidation project. The lessons we learned are applicable to many global firms that need better oversight of their worldwide business, even as they strive to reduce their IT and service costs. We provide business managers with the rationale for globally standardizing their business processes, even as we lay out an implementation map for those senior system architects, DBAs, integrators, and developers who must take on the hard work of consolidating the disparate systems.

From the start, it's all about setting the right goals. Any global company embarking on a database/application consolidation project has to first decide what level of consolidation it wants to pursue—and if the scope of the various aspects of the project will be regional or global.

In our view, there are four possible levels for any consolidation, each of which builds on the level before. The first level involves moving and aggregating all your separate database instances into a single data center (either regional or global). The benefits of a Level One consolidation are reduced facility and headcount costs for system administration, along with better system service and availability.

That said, we must be clear: No real informational value is achieved by mere data center aggregation. For that, you need to take it a step further by consolidating your separate instances into a single database. This is the second level, an achievement we call a “technical consolidation.” With a Level Two consolidation, you can achieve improved access and better information from your data, even as you further reduce your hardware costs.

Beyond that, superior information can only be obtained from your single database instance if you take it to a third level by standardizing your business processes as much as possible. Having standard global business processes ensures “apples-to-apples” comparisons across all lines of business, no matter what country your data originates from. And a Level Three consolidation not only reduces processing and transaction costs, it increases your ability to leverage best practices across your company.

Finally, when you've standardized your business processes—and instantiate them in your regional or global instance—you now have the opportunity to provide shared services from regional or global service centers. Instead of having many service representatives located in each country doing a variety of tasks, you can now have a much fewer number of service experts more tightly focused on supporting the standardized processes across the different countries. For example, now that Oracle has a single standard way to enter orders, employees in the shared service center for the Europe, Middle East, and Africa (EMEA) region can enter the orders for France, Italy, Spain, Portugal, and so on, and ensure that when he or she enters the order, the order is entered correctly.

Oracle chose a Level Four consolidation as the ultimate target for its GSI Project, though we sequenced aspects of the project differently in EMEA and Latin America, as described below. The end result, however, is that Oracle now has a GSI housed in Austin for HR, Financials, Consulting, and CRM—and all our business processes have been globally standardized. In the end, we also chose regional service centers for our finance services, building ones in Dublin, Sidney, Bangalore, and Rocklin, Calif., because we felt such a strategy better served each of the regions.

Other Strategic Decisions

When you've determined the level at which the consolidation will take place, a comprehensive strategy must be developed for getting from here to there. Remember, Oracle's ultimate goal was a global consolidation, but we did most of the work regionally first. We started our consolidation efforts with the different instances already residing in the regional data centers—two for EMEA (which we merged), one for Asia-Pacific, one for Latin America, and one for the U.S.—and used those centers to execute the consolidations.

In Latin America, for example, we built a Level Two technical consolidation, where the 10 separate country databases were consolidated into a single regional database instance without changing any business processes first. This purely technical consolidation enabled the project to move ahead rapidly, initially saving several months in the process, but it also meant we had to do another round later when we standardized the business processes and developed the shared services center for Latin America.

Sequencing it this way had other drawbacks too. Not only did our initial strategy prevent us from evaluating beforehand the amount and quality of the data we did bring forward, but we found that revisiting the process consumed some of the time we'd initially gained.

By contrast, in EMEA we targeted a Level Four consolidation from the start, front-loading the business process standardization effort. This made the consolidation effort more difficult, but it also allowed EMEA to advance more directly toward establishing a regional shared service center—all the way to Level Four. We achieved not only the cost savings of the technical approach on the data center side, but a similar savings as well on the facilities and administration (F&A) side. In addition, by standardizing the business processes as we went along, the consolidated instance provided much more valuable data for managing our business up front. And as companywide initiatives have been rolled out (including new e-business ones), the changes they've required have been the same across all operations and therefore quicker and easier to implement.

Business Challenges and Solutions

To begin, we set up a GSI steering committee to oversee the whole project—a team consisting of executives from the business, IT, and development. To achieve agreement on global processes and prioritization, we then set up an organizational structure with global representation by region. For each business process we wanted to standardize, we had a global process owner on the business as well as the IT side. In addition, we named local business and IT representatives from each region who were responsible for ensuring the needs of each country within the region were addressed. The business leaders took the lead by identifying their business requirements and the IT representatives provided guidance on the capabilities of the systems, helping the business focus on realistic improvements in processes—and when need be, recommending generic improvements in the applications. These workgroups then developed the new global processes for each flow. If the group was unable to resolve an issue, it was raised to our GSI steering committee for resolution.

We tailored our efforts to globally standardize all business processes to how we actually use our applications to meet our business requirements, striving as we went along to implement processes that could be addressed by standard versions of the applications. We began by defining roughly 12 business flows, which cut across application modules, and eventually wound up defining nearly 100 standardized processes, a few examples being HR, Commissions, Support Renewal, Procure to Pay, and Quote to Cash. To parse it further for just one process: in Quote to Cash we identified all the functions in the flow, from Quoting, Contract, Order Entry, and Product Shipment, to Invoicing, Collection, and posting to GL. We then defined best practices for each function, balancing global priorities with localization needs. Note that Oracle E-Business Applications have been globalized for world currencies and localized for language and many country-specific legal and financial requirements.

The technology and data issues in a global consolidation project are numerous, but I'll focus here on the processes required for managing these issues, not the details themselves. The most difficult areas involve setup, data migration and conversion, data quality and integrity, and issues related to performance.

Identifying the system setup required by a set of agreed upon global business processes is a key component of the project. We broke up the setup requirements into two buckets—global and local, to reflect the two logical requirements. The global setup is a configuration that all organizations operating within the instance need to adhere to. The local setups are those setups that are necessary for doing business in a specific legal entity or country (for example, payment by EFT in the U.S. vs. payment by bank transfer in Germany). Both these setups can only be determined after the business processes have been established. Establishing the setup involves working with the business owners, but in general, it's IT's responsibility to take the business process definitions and turn them into a configuration for the applications setup. Examples of setup included the assignment of flexfields and their validations as well as the GL Chart of Accounts definitions.

A complicating factor of these setups is that they are directly related to the processes, which in turn are subject to change. However, success in a consolidation project requires that proper change control be implemented over the setup. We implemented version control on top of our business processes. Therefore, any major change to processes and/or setup requires a change to our standard single instance implementation model. The important point here is that change is unavoidable, but needs to managed, documented, and controlled.

Once you have arrived at your “standard” setup, you can actually work on mapping each of the source instances to be moved to the target instance. We used spreadsheets to compare and analyze the data structures, transaction types, primary and foreign key mappings, and so on. For every migration, we did three test migrations: an initial technical test, one for a conference room pilot (to verify global processes for the local situation), and one user acceptance test (for transaction and capacity testing). We then did the final production migration. During the UAT, we got as many users on the system as we could for performance testing and to ensure that users' queries were scaling well enough for daily use. ERP test migrations were also done after month-end close, usually in the second week of the month, to give us two weeks to fix any problems before the next month-end period.

Another issue to be resolved during data migration was how much data to actually migrate. The consolidation project provides an opportunity to review the amount of history to bring into the global single instance. In the GSI project, we decided to standardize on bringing in all the data to begin with. This allowed the consolidation project to move ahead without taking the time to analyze what we could actually achieve and remove.

Lessons Learned The achievements of our GSI Project were fourfold: We consolidated our disparate E-Business systems into a Global Single Instance. We globally standardized our business processes. We standardized our application footprint. We consolidated our services into regional shared service centers. It was a long journey, but we learned some lessons along the way. First, begin your project by identifying the scope of your consolidation. Given the key driver of better information, implement as many standard business processes as possible. Leverage the best practices in use worldwide for everyone, but also take the opportunity to re-think your business. The resulting improvements in process across your company will provide returns much greater than those achieved by simply consolidating database instances. Create a global team comprising executive sponsors, key business users, and IT staff and give them the authority to drive business change within your company. Lay out your project plan so that it builds on experience and allows the organization to be successful. Be aggressive but sensible, particularly as you load new data into your consolidated instance, and allow time to address both unforeseen issues and new process opportunities. In order to be successful, you must leverage your business community as much as possible. These are the people who will reap the benefits provided by the more valuable information gained from a consolidated system. However, it is important that you take the time to explain the priorities and set appropriate expectations with this group. Standard processes will definitely be an improvement across the company, but some groups may have to give up some things for the benefit of others. The tradeoff on IT gains vs. productivity and informational gains is a difficult balance to achieve, but is critical to the overall success of the final outcome, and if executed successfully, your company will benefit in a profound way.

In addition to deciding upon the amount of data to be brought into the consolidated instance, you also have the opportunity to focus on data quality and integrity during the migration process. The quality of information that can be attained from the consolidated instance is directly related to the quality of data brought in from the separate instance. In our process, we asked each country to do a measure of data cleaning beforehand, setting time limits on the process, knowing that further cleaning could be done once the migration was accomplished. We didn't hold up the migration if the data wasn't cleansed. Thus, each company undertaking a consolidation project must look at the quality of the data in the individual instances and determine the appropriate level of resource to apply to cleaning up this data if necessary beforehand. Of course, one of the major areas to focus on is customer information. Here's a good application of the 90/10 Rule: If you clean up the 10% that represent your biggest customers, you'll get 90% of the benefit in terms of information.

Performance and the Grid

Finally, one of the key areas of concern for anyone doing a consolidation project is performance. There are three types of performance issues that you need to be aware of and plan for. The first one, of course, is the system performance of the working consolidated instances. Care must be taken to ensure that the performance required by the business can be met by the consolidated instance. Here the UAT is crucial.

All the standard areas of concern need to be researched and addressed to a satisfactory level. As you add more and more data and increase the load, you must plan for steadily increasing requirements for CPU and memory, disk space, concurrent usage and peak loads, and network bandwidth. It also means managing user expectations and retraining users to run more efficient queries.

The second area of concern for performance was the actual migration scripts themselves, and the bandwidth of the data migration streams piped through such Oracle tools as Import/Export and SQL Loader. We had to address questions like “How many instances could we move into a regional instance in any given time period?” and “How long would a regional instance have to be offline to accomplish the move?” We moved up to three instances into a consolidated instance in a given month. This limitation led to us running parallel regional consolidations, which then were combined into two global instances that we then combined into the final GSI. We would have liked to move all instances into the GSI database once our global setup was established. This would have saved us from moving some instances twice—once into the regional consolidation , followed by the move of regional consolidation instance into the GSI.

However, due to our aggressive schedule, we needed more than one “funnel” to accomplish all the instance migrations within the timeframe of our project. We therefore decided to continue to consolidate on two instances in the latter phases—the EMEA instance and the instance in the U.S., which then became the GSI. Midway through 2003, we migrated the EMEA instance into the U.S. global instance. And in October 2004, we migrated the global CRM system (which we'd already developed in 2001) into GSI. We now have a truly global E-Business Suite single instance for all our Financials, Consulting, HR, and CRM.

Finally, of course, the performance of the resulting consolidated GSI must be ensured. Here's where Oracle's emerging vision of a “commodity grid” came into play. We devised a highly scalable architecture for the Austin data center, built on a midtier farm of 50 commodity 2650 Dell Dual Processor boxes running Linux—utilizing their fast processors and load-balanced for specific application functions across our “BigIP” Gigabit Ethernet Intranet. This midtier is connected to a back-end cluster of four Sun 12K multi-processor servers running Oracle Real Application Clusters (RAC) on the Oracle9i Database. The Linux boxes route the processing by application and process type to the proper back-end node: for example, Node 1 does Order Processing, Node 2 does all the Self Service, node 3 does Oracle Financials, and Node 4 does HR and Payroll, though these loads can be shifted as need be.

A single application code tree of the E-Business Suite is installed on a Network Appliance Filer Cluster 600, running on top of Oracle Application Server—thus reducing the footprint of our systems and simplifying any patching and upgrades. The Linux boxes simply service this code tree, load-balanced among the processes using the Concurrent Manager. The single code tree services all the Linux boxes. In turn, all the database data—6TB of it—is spread across an EMC DMX 3000 disk array. (See figure below.) Finally, the whole GSI system is backed up in Colorado Springs, Colo., for disaster recovery purposes.

Global Systems, Today and Tomorrow