As Published In July/August 2006

Get Control
By David Baum

The trend is inescapable: Networks get faster, computers get more powerful, clustering technologies get more effective, and storage devices get more efficient. These advances pave the way for software applications that are more capable and intelligent, automating a diverse array of business functions. Yet for many IT managers, quantitative improvements in the IT infrastructure don't make any qualitative difference where it matters most: saving time and energy in provisioning, managing, and monitoring critical information systems.

Enterprise applications built around a service-oriented architecture (SOA) are further changing the nature of system management in areas such as configuration management, service-level management, application performance, and infrastructure management. Highly modular programs can take advantage of virtualized

The Oracle Enterprise Manager 10g Universe Download pdf

system resources in a grid infrastructure, making the mapping between services and infrastructure components more complex than it was in the days of monolithic applications. Having the ability to manage this complexity and measure end-user experience in the context of the many working parts of an SOA environment is critical.

This virtual world of SOA and grid computing promises tremendous flexibility for deploying and enhancing business applications, but management solutions must evolve in tandem to achieve comparable advances for IT professionals.

"Many of today's emerging applications are being created to run in highly complex multitier environments where changes at any level can impact application performance," confirms Mary Johnston Turner, a vice president at Boston-based technology research firm Summit Strategies. "These environments are both flexible and dynamic," Turner adds, "but because the tiers are so tightly interconnected, managing them has become much more complex."

Oracle is helping companies manage the multitier complexity of their IT environments. "Our investment in Oracle Enterprise Manager complements our strong technology and application products, allowing customers to improve service quality and adopt new technology innovations while controlling management costs," says Oracle President Charles Phillips. "Oracle Grid Control is the centerpiece of this solution and a key enabler in the adoption of grid computing and service-oriented architectures."

Adopting the Grid

Snapshots Advance America www.advanceamerica.net Location: Spartanburg, South Carolina Industry: Financial services Number of employees: 6,500 Oracle products and services: Oracle Database 10g, Oracle RAC, Oracle Clusterware, Oracle Automatic Storage Management, Oracle Automatic Database Diagnostic Monitor, Oracle Enterprise Manager 10g Grid Control, and Oracle RAC Pack consulting services Pfizer www.pfizer.com Location: Morris Plains, New Jersey Industry: Pharmaceuticals Number of employees: 115,000 Oracle products and services: Oracle8i Database, Oracle9i Database, Oracle Database 10g, Oracle RAC, Oracle Enterprise Manager 10g Grid Control, Oracle Diagnostic Pack, Oracle Tuning Pack Replacements www.replacementsltd.com Location: Greensboro, North Carolina Industry: Retail Number of employees: 550 Oracle products and services: Oracle Database 10g, Oracle RAC, Oracle Enterprise Manager 10g Grid Control, Oracle RAC Pack consulting services Starwood Hotels and Resorts Worldwide www.starwoodhotels.com Location: White Plains, New York Industry: Travel, guest services Number of employees: 145,000 Oracle products and services: Oracle9i Database, Oracle Database 10g, Oracle Enterprise Manager 10g Release 2

Starwood Hotels and Resorts Worldwide, with approximately 850 properties in more than 95 countries, depends on a production grid environment to run its business. Starwood's Oracle enterprise grid hosts its reservation system, guest loyalty system, Web systems, and call center environment. "These information systems encompass our entire business, so it is critical that we establish very reliable OS, storage management, and clustering capabilities," says Bill Camp, vice president of enterprise systems at Starwood.

Starwood is basing its online environment and underlying database on Oracle Database 10g Release 2. The company has a 5TB Oracle database running on a three-node cluster orchestrated with Oracle Real Application Clusters (Oracle RAC). Starwood's application portfolio includes distribution, content management, and marketing, in addition to the reservation and customer-loyalty systems, Web sites, and call center. The hospitality chain processes billions of dollars in bookings per year through its Oracle-based information systems. Oracle Enterprise Manager helps make it possible.

"Our goal is 99.999 percent availability, which is one of the reasons we have made Oracle RAC and Oracle Enterprise Manager software our standard going forward," says Arup Nanda, director of database engineering at Starwood. "We use the Oracle Grid Control software within Oracle Enterprise Manager for automatic grid monitoring, space management, performance monitoring, and alerting DBAs. It gives us a single point of focus for our Oracle-based systems."

Starwood finds Oracle Enterprise Manager 10g Release 2 to be ideal for managing a grid-based infrastructure because it reveals so much information in the browser, especially cluster-interconnect information from Oracle RAC. "The SGA-attach feature of the [Oracle] Enterprise Manager interface allows us to get to the database and retrieve key performance numbers from the SGA directly, without logging in and using a SQL interface," Nanda explains. "This allows us to resolve issues quickly. For us, it was one of the key reasons to implement Oracle Enterprise Manager 10g [Release] 2."

SOA keeps the system dynamic. "We're adopting a service-oriented architecture based on Oracle technology because it is much more flexible adapting to changing business requirements," Camp says. "Having made that decision, it was natural to go with Oracle's management tools as well."

Finding better ways to manage information—easily, securely, and at the lowest possible cost—was a key design goal for Oracle Database 10g. Oracle Enterprise Manager 10g takes advantage of instrumentation and data collection technologies built into the internals of Oracle Database 10g, Oracle Fusion Middleware, and Oracle E-Business Suite and couples that with end-to-end service-level management capabilities to continually monitor the status and health of any application.

"The self-managing capabilities of Oracle Database 10g fully automate or significantly simplify many routine functions, enabling database administrators to easily optimize performance, manage backups, and perform other day-to-day tasks faster than ever before," says Moe Fardoost, director of marketing for Oracle Enterprise Manager. "In complex environments containing many moving parts, with constantly changing workloads, automated management is essential to maintaining costs."

These comprehensive system management capabilities have been essential to Advance America, the largest provider of payday cash-advance services in the United States. The company operates more than 2,600 cash-advance centers in 37 states and is adding about 400 centers per year.

Several years ago, Advance America implemented an enterprise software application called the eAdvantage system to handle critical business functions such as collecting customer information, processing new cash advances, and servicing existing ones. Initially this application was based on a distributed Sybase environment, with 2,000+ branch databases replicating information to a master corporate database on a nightly basis. As the company grew, the complexity of this environment caused delays that affected internal and external operations.

"Even with all servers working at maximum capacity, we no longer had a sufficient window for processing all this transactional data overnight," recalls David Toothman, CIO at Advance America. "Our distributed database environment could no longer sustain the business, let alone handle growth."

Advance America needed an infrastructure that would expand and scale as business grew—without requiring a corresponding increase in IT management cost and complexity. Toothman and his colleagues adopted Oracle Database 10g, Oracle RAC, and Oracle Enterprise Manager 10g. Together with Oracle Consulting, they implemented a grid-computing environment that effectively operates as a single system.

"One of the key benefits of Oracle Database 10g Release 2 Real Application Clusters is its ability to pass information to the application server tier about the load on and availability of particular nodes," says Randy Hietter, director of product management for Oracle RAC. "The middle tier can intelligently route connection requests to the least busy node. In that way, the middle tier and database tier are functioning as a cooperative computing grid, dynamically exchanging information about their status so as to maximize the utilization and performance of grid resources."

Thanks to Oracle RAC and Oracle Enterprise Manager, Advance America's Oracle grid is reliable and can be expanded by adding nodes to meet expanding workloads. "The new system architecture has proven to be much more reliable by eliminating single points of failure," Toothman says. "In six months of production processing, we have had zero unplanned downtime for the database. The environment is also much more scalable, giving us the ability to add capacity on demand in a very cost-effective way."

At Advance America, consolidation and management capabilities make a difference on the front lines: where people and systems interact. According to Sanjay Bamba, the company's director of database services, consolidating their infrastructure to a clustered database system has significantly reduced management costs. His IT staff can perform information loads and roll-ups about two hours faster with the consolidated grid system—a 25 percent improvement. Meanwhile, at the cash-advance centers, daily reports are produced more quickly and are available three to four hours earlier—a 50 percent improvement.

"Instead of maintaining more than 2,000 decentralized databases—with their recurring upgrade, support, and management costs—requests from each cash-advance center are directed to a service running on all four nodes of the Oracle RAC cluster," Bamba says. "This allows us to adjust the resources available to each group of clients as the workload fluctuates. It's working beautifully."

Advance America has discovered another advantage of deploying Oracle Enterprise Manager software: Oracle Clusterware eliminates the need for costly third-party software. "Oracle Grid Control provides us with a single solution for monitoring the whole cluster—a centralized place to look at the health of our system," says Bamba. "We don't have to consult multiple utilities to determine if things are working well. We use it for performance analysis, job scheduling, resource monitoring, and to keep an eye on our physical-standby environment. With [Oracle Enterprise Manager 10g] Release 2, we're able to monitor our environment through new service and system dashboards, and everyone gets what they need: service-level information for business users and system information for administrators. We gain economies of scale as we bring more activities under the Oracle Enterprise Manager umbrella," he continues. "In a multitier computing environment, the Oracle software allows us to do more with less."

While numerous third-party management tools are available, few of them can supply the aggregated information you need to make intelligent decisions—particularly when multiple servers and databases are involved. To help system administrators gain a thorough understanding of database management in Oracle grid environments, Oracle has introduced Oracle Diagnostic Pack and Oracle Tuning Pack, specialized management utilities that complement and extend Oracle Enterprise Manager Grid Control software.

"We have to provide 24/7 support for our databases, and we needed a tool that would let us know if there is a problem, so we could be on top of it right away," says Susan Szot, manager of the Corporate Information Technology Metro DBA team at Pfizer, the world's largest research-based pharmaceutical company. "We tried several kinds of monitoring tools, with various degrees of success. [Oracle] Enterprise Manager Grid Control had the most promise, so we started a pilot. Now we use the software to monitor all of our databases, in conjunction with the Oracle Diagnostics Pack and Oracle Tuning Pack."

Szot and her team of 15 DBAs use Oracle Enterprise Manager Grid Control 10g to manage 30 terabytes of Oracle data housed in 900 Oracle databases on an Oracle grid consisting of 250 servers—a mixture of Sun Solaris and Microsoft Windows computers.

Pfizer primarily uses Oracle9i Database and is moving toward Oracle Database 10g for all its production systems. Formerly, they depended on third-party tools to manage these information systems, but Szot says the software didn't scale well and lacked insight into the Oracle product stack. As a result, Pfizer had to use several software products to monitor and maintain its IT systems.

Today, Oracle Enterprise Manager Grid Control gives Pfizer one central management utility for streamlining and automating a wide variety of tasks. According to Bala Palayam, a DBA team lead at Pfizer, Oracle Diagnostic Pack is especially useful for troubleshooting. It includes a diagnostic engine built into the Oracle Database kernel, enabling Oracle Enterprise Manager to continually collect metrics and information about the performance of the database environment. If a user complains about performance, a couple of clicks in a GUI-based diagnostic console often reveal the problem.

"In the past, we couldn't tell if we had a problem with the database, the network, or something else, so we had to dispatch a team of people to figure out what was going on," says Palayam. "It might take all day. Now, Oracle Enterprise Manager Grid Control can tell us if something happened with the database before we get all these other people involved. For example, if a user drops an index, which causes a runaway query, we can pinpoint the problem in 10 minutes."

All of these issues become more pronounced in complex environments, since Pfizer is managing not just individual servers but a shared virtualized infrastructure. "Gradually, we are learning to use Oracle Enterprise Manager to manage and monitor not just the database but the underlying infrastructure," Szot says.

Pfizer uses the Oracle Tuning Pack to improve application performance via better SQL query management. "If a SQL query is chewing up a lot of CPU time, Oracle Enterprise Manager can often identify why," says Palayam. "For example, it can reveal a hanging process and even kill the process right away to free up the database. The culprit might be the number of sessions or how particular processes are being executed."

While Oracle Enterprise Manager can take corrective action when needed, Pfizer prefers to be alerted to problems and advised about solutions—then fix the actual problems themselves. The exception comes with analyzing database schemas and tables—a tedious process when you are dealing with 900 databases. "Before, we'd have to set up dozens of individual jobs to do that," Szot says. "Now we can automate it all through Oracle Enterprise Manager. What used to be a 30-hour job can now be accomplished in just two or three hours."

Oracle Enterprise Manager Grid Control also improves security with respect to Sarbanes-Oxley compliance. "Because we have all of our databases hooked to Oracle Enterprise Manager, we have a centralized way of running security scripts," Szot says.

Palayam appreciates Oracle Enterprise Manager's Web-based architecture, since it means Pfizer does not have to install so much desktop software to manage the grid. "All you do is point administrators to a URL to access all of the management functions," he explains. "It is downward-compatible, so we can use it to monitor Oracle Database 10g, Oracle9i Database, Oracle8i Database, and even to monitor Microsoft SQL Server databases."

Pfizer has integrated Oracle Enterprise Manager Grid Control with Oracle Recovery Manager (Oracle RMAN), which streamlines the process of scheduling database backups. "Oracle management utilities are helping us to centralize security, backup, monitoring, and maintenance activities, as well as to integrate our backup strategy with our monitoring strategy," Szot says. "We've only scratched the surface, yet already this software is solving our current problems and getting us ready for the future. We are less reliant on third-party management tools and we've eliminated some third-party tools from our environment. I think it's only going to get better."

Flexibility with Oracle RAC

Plugging In to Oracle Enterprise Manager To reduce IT costs, many companies are recognizing the importance of acquiring management and diagnostic tools that offer a holistic view—of servers, storage devices, operating systems, application servers, and databases—that is mapped directly to their core applications. Oracle is leading the charge to integrate management with Oracle Enterprise Manager 10g Release 2, which includes plug-ins and connectors to monitor and manage the entire application stack, from custom software modules to infrastructure elements. "The Oracle Enterprise Manager plug-ins dramatically extend Oracle's ability to monitor data from other systems," says Mary Johnston Turner, a vice president at Summit Strategies in Boston. "Customers can detect and resolve issues throughout an Oracle grid as well as monitor and manage non-Oracle products." Companies use the plug-ins to integrate F5 load balancers, NetApp filers, EMC storage, Checkpoint and Juniper firewalls, non-Oracle middleware, the Microsoft .NET Framework, and the Microsoft Windows Server System, including the SQL Server database. Oracle Enterprise Manager connectors enable the bidirectional exchange of information between Oracle Enterprise Manager and other management solutions that customers may have in their environments. For example, Starwood has plans to integrate Oracle Enterprise Manager 10g with HP OpenView, its corporate monitoring standard. According to Moe Fardoost, director of marketing for Oracle Enterprise Manager, the plug-ins and connectors enable a single administrator to perform complex management tasks, such as provisioning extra resources to meet increased demand, even in a heterogeneous environment. "Oracle has used open industry standards such as Web services and XML to simplify the process of integrating other management systems," he explains. "Through plug-ins and connectors, Oracle Enterprise Manager can be applied to a single database system or an entire grid environment. This allows customers to monitor multiple business applications and management utilities as easily as if they were managing one application running on one computer."

Just as the added functionality of Oracle Diagnostics and Tuning Packs extend the core capabilities of Oracle Enterprise Manager, Oracle RAC enhances database configurations. With Oracle RAC, Oracle Database 10g runs on two or more servers in a cluster that concurrently access a single shared database. This configuration extends flexibility and scalability to the business applications that use Oracle databases, since administrators can add additional server and storage resources as needed. This configuration is also more reliable: If one server or storage device encounters a problem, the other devices in the cluster will continue to carry the load.

"[Oracle] RAC addresses unplanned outages, with extensive data protection and disaster recovery capabilities, which helps us sustain business continuity," says Starwood's Nanda.

This architecture was appealing to Replacements, a specialty retailer with an inventory of 10 million pieces of tableware in more than 200,000 patterns. The company has been using Oracle technology since 1994 to power its inventory system. In 2005, Replacements deployed an enterprise grid infrastructure comprised of Oracle 10g software.

"After extensive testing, we determined that Oracle RAC is a more reliable and robust structure than a single-instance failover model, as we had been using before," says Jim Meredith, IT manager at Replacements. "And because we use commodity hardware, the equipment costs us less as well."

Replacements uses Oracle Enterprise Manager 10g to automate DBA tasks, and the Web console supplies insight into every aspect of service performance, so they can more easily troubleshoot problems. The IT staff is also using Oracle Automatic Storage Management to simplify storage management activities, which has eliminated the need for third-party cluster file system software.

"All of the storage for the database is controlled through Oracle Enterprise Manager and related utilities," says Meredith. "For example, database backups are controlled and managed through Oracle Recovery Manager. We also use the paging and notification facilities within Oracle Enterprise Manager to detect potential problems, such as if resource use exceeds a certain threshold. If configurable thresholds such as 'run queue length' or 'session count' are exceeded, Oracle Enterprise Manager can bring up another instance of the service on another node. It works very well."

An Inside View

Summit Strategies' Turner believes that there's a reason Oracle Enterprise Manager is so effective within grid computing installations. "Oracle's strategy is to manage from the inside of the application—mapping deep knowledge of its core system components with real performance of end-user business services at the level of discrete business services," she says. "Oracle has a good perspective on what's going on with those business services and is gradually extending Oracle Enterprise Manager to encompass multiple layers of the infrastructure. Other system management tools have more of an 'outside-looking-in' perspective on the application and database environments."

In other words, as a vendor that supplies not only the database but also many mature enterprise applications and associated middleware, Oracle has an intimate view of the entire application stack. According to Dave Pearson, senior director of Oracle architecture, this unique insight into the infrastructure allows Oracle Enterprise Manager 10g Release 2 to provide richer end-to-end application management for all application environments.

"Once you define services, either at the database or the application level, you can set policies that establish expected service levels, measure those service levels from an end-user perspective, rapidly isolate problematic issues through the entire stack, and respond by adjusting resources as appropriate," he explains.

Managing Change

While these new management capabilities represent a huge advancement for the industry, teaching and learning new procedures can be challenging in many corporate cultures. Still, customers who have used the browser-based interface in Oracle Enterprise Manager claim that it makes routine tasks and training easier and more efficient.

"Training DBAs is extremely easy with the Web interface of Oracle Enterprise Manager Grid Control," says Starwood's Nanda. "Rather than having to work with a dozen tools and learn their commands, you can do it all through a single interface."

Having a cohesive view of the IT environment becomes more important as customers adopt virtualized grids in which physical resources enable a logical set of IT services. Oracle Enterprise Manager treats IT resources as virtual components that can be transparently mapped onto business services, provisioned, monitored, and managed. "Graphical service topologies and service dashboards continually display pertinent information to both IT managers and business owners, so they can monitor application services and the quality of service being delivered to end users, rather than focusing only on the performance of discrete components," Oracle's Fardoost says.

Next Steps READ more about Oracle Enterprise Manager plug-ins and connectors Oracle Grid Computing READ Oracle Grid Computing customer ROI studies

Advance America's Toothman believes that approaching the management discipline at this level "ups the ante for administrators" and that their jobs become correspondingly more interesting.

"As more of our routine management tasks are automated by Oracle software, our IT staff is able to take on more-challenging tasks for the business," he says. "Oracle Enterprise Manager makes it easier to understand the grid environment through the topology viewer and other mappings."

Summit Strategies' Turner believes that the technology professionals who confront these management challenges will find more-interesting work. "Gradually we're going to see a need for higher-caliber IT professionals who can think at these higher levels of abstraction."

These changes in IT management approach are already happening as businesses migrate toward a more-automated IT environment. Replacements has discovered these productivity benefits firsthand in its work with Oracle Enterprise Manager. "We've transferred much of the administrative load off of our UNIX/Linux administrator," says Meredith. "Similarly, our DBA is doing a lot more high-level managing and a lot less mundane work. With Oracle Enterprise Manager, he can see all the databases and all the servers through one interface, so he knows at a glance if there is a resource bottleneck, such as if a server hits a memory or CPU threshold or end-user response time gets a little bit slow."

Turner sees this pattern repeatedly. "The journey toward management automation requires enterprises to plan and operate IT differently than they might have done in the past," she says.

"In an automated, policy-driven world, service-level agreements need to be defined in the context of end-to-end business processes rather than physical hardware characteristics, like percentage of uptime," she says. "Similarly, IT staff members need to rethink their roles and responsibilities as they move away from manual problem intervention and start focusing on automation through more-comprehensive management software such as Oracle Enterprise Manager."

But the successes reported here—and many others just like them—are motivating many companies to get started now.