Oracle Takes In Some Sun

Oracle IT basks in great results with Sun hardware in the data center.

by Monica Mehta, August 2011

What are the three most important benefits good hardware can deliver to a business? Savings, performance, and flexibility. Businesses need servers, storage, and backup solutions that reduce costs and deliver more value—especially in terms of space and energy consumption. Solid hardware provides the support for business processes to run quickly and reliably, even as it serves multiple purposes and environments.

Since the acquisition of Sun Microsystems in 2010, Oracle’s information technology department has deployed Sun hardware on a grand scale, optimizing databases, applications, and hardware to work together. One year later, the results have been remarkable. In some cases, the IT department has seen 75 percent cost savings and faster performance from databases and applications in a broad range of environments, supporting more than 21,000 developers, 100,000 employees, 4.5 million Oracle On Demand end users, and 350,000 Oracle University students.

The lessons learned have been immensely valuable for both Oracle staff and customers. “In many cases, we are building targeted, engineered solutions, bringing the best of the hardware together with the best of our software,” says Craig Yappert, senior director of Oracle Global IT.

A Unified Infrastructure

Oracle’s IT department consists of three divisions: Oracle Product Development IT, Oracle Commercial IT (which serves Oracle On Demand customers), and Oracle Global IT (which manages the shared infrastructure across the three entities).

Oracle’s Product Development IT (Oracle PDIT) division oversees massive data stores, supporting the development of products and services. Oracle PDIT’s infrastructure empowers more than 21,000 software engineers to develop Oracle database, middleware, and application products on a daily basis. Oracle PDIT also runs all the internal back-office business applications, oracle.com, Oracle Support services, and company e-mail and collaborative tools.

Oracle PDIT staff has already converted 50 percent, or 5,000 terabytes, of its storage infrastructure to Oracle’s Sun unified storage—specifically, the Sun ZFS Storage Appliance—to support development and testing of applications, virtual machines, instance cloning, and virtualization projects.

Also, the combination of Oracle’s Sun server and related hardware with Oracle software, as with Oracle Exadata and Oracle Exalogic machines, is used in both engineering and running back-office applications. And in the future, says Campbell Webb, vice president of Oracle PDIT, the hardware will serve as the platform for all of Oracle PDIT’s back-office applications. Oracle PDIT currently has 29 Oracle Exadata machines running back-office applications—with implementation of another 23 planned for 2011—processing collaborative applications such as e-mail and instant messaging, content management, business intelligence, customer relationship management, and customer support.

Additionally, Sun ZFS Storage Appliance products helped Oracle PDIT reduce costs by 75 percent, replacing every four storage devices from NetApp or EMC with a single Sun ZFS Storage Appliance. This has reduced power consumption from roughly 20 kilowatts to only 4 to 5 kilowatts, and it has reduced the area the equipment occupies from four floor tiles to one, making way for additional growth.

Sun ZFS Storage Appliance products also delivered dramatic improvements, increasing capacity by 50 percent. The compute server farm, home to more than 5,000 systems in 12 clusters, has seen as much as six times the performance. Now, Oracle PDIT gets roughly 15 percent more compute hours from the same capacity of servers. Designed with higher upper bounds of storage density, Oracle’s Sun unified storage solutions do not have limitations in terms of capacity or cloning.

“We can execute 15 percent more workload in a 24-hour period,” says Webb. “The storage eliminates our code testing backlog, effectively making our applications run faster and more smoothly.”

These increases also apply to Oracle’s Sun servers. Similar to the Sun ZFS Storage Appliance, Sun Fire servers provide a material performance boost compared to Oracle PDIT’s legacy infrastructure. The servers are typically eight times faster than technology purchased three years ago and have four times the capacity. Refresh initiatives over the last year have realized 80 percent gains in both space and power savings, providing much-needed capacity for new engineering and back-office initiatives.

The new Sun Fire X4170 M2 servers are the perfect platform for Oracle PDIT’s highly virtualized infrastructure. With 144 GB of RAM and two Intel Xeon X5670 processors each, the new Sun Fire servers enable Oracle PDIT to support 18 smaller virtual engineering environments for development and testing, or 8 larger virtual performance testing environments on a single physical server. In some cases, the division might have a combination of both. Provisioning of both physical and virtual environments is managed by Oracle PDIT’s own purpose-built DevOps application, which has allowed staff to scale to 75,000 environments in the development operation alone.

“This level of automation provided by DevOps has allowed Oracle PDIT to satisfy the ever-changing business requirements of Oracle’s engineering community, creating competitive advantage by reducing the time it takes Oracle to bring new products into the market,” says Webb.

Lessons Learned

Oracle PDIT runs all flavors of software from the various Oracle product lines on its storage—database, middleware, and applications. As a result, Oracle customers reap the benefits of mature products, tested internally on taxing workloads.

“We aggressively adopt new technology offerings and assume the risk because we can afford to do so, providing critical feedback to our engineering teams,” says Webb. “At the end of the day, the customers benefit the most with a mature product, optimally tuned for the complete hardware and software stack.”

Oracle’s Sun unified storage solutions also contain software features that simplify handling and management. For instance in standard environments, switching storage systems and migrating data from one device to another can be laborious and costly. But Sun ZFS Storage Appliance’s Shadow Migration feature automates migration of referenced data from legacy platforms, so Oracle PDIT staff can easily migrate data without service interruption. This transparent, seamless transfer of data has limited downtime during the switching process.

A resident analytics tool helps Oracle PDIT staff understand workload specifics in-depth, allocating hardware configurations to maximize the resource utilization. Webb says his team can identify bottlenecks immediately and react quickly using far more information than they’ve had from any other vendor. “The analytics let you drill down to a very low level to find out how the system is running,” says Webb. “This is extremely relevant for storage where you have hundreds or thousands of machines accessing storage. It translates into performance improvements, which means you can run more jobs faster and more efficiently.”

In the coming years, Oracle PDIT plans to replace 305 existing storage units with 123 Sun storage devices. Webb expects Oracle’s Exadata Database Machine X2-8 to have a major impact on reporting and business intelligence, allowing users to perform data processes in real time. He says having the most-recent information available to the user base at any time allows them to make more-informed decisions much more quickly. “They don’t have to wait 8 hours or 24 hours until the next set of data comes in and gets updated to run their reports. The data they see is effectively live,” he says.

The adoption of Smart Flash Cache—an Oracle Database feature that extends memory area size and caching to a large flash memory storage device—adds flexibility in locating data, says Webb, dramatically increasing I/O rates. “We’re seeing remarkable changes in applications performance by changing out the underlying infrastructure and making these new technologies available through the software layers,” he says.

Commercial IT Benefit

Oracle’s Commercial IT (Oracle CIT) division is the backbone of Oracle On Demand, Oracle’s hosting and management services. This IT infrastructure supports 4.5 million Oracle On Demand end users whose businesses rely on subscriptions to remotely managed Oracle technology and applications.

For the IT pros inside Oracle CIT, that means maintaining vast amounts of reliable servers, storage, and backup—and the mission-critical data that courses through them. Roughly 50 percent of Oracle CIT runs on Sun hardware—most prevalently Sun servers, Oracle’s Sun unified storage solutions, StorageTek SL48 tape library, and Oracle Exadata machines.

Oracle CIT staff have standardized both their own infrastructure and commercial customers’ storage on Sun solutions. Approximately 3 petabytes of data is now on Sun storage, which includes Oracle’s Sun unified storage solutions and Sun fibre channel storage. Oracle CIT’s hosting services include data backup, which means Oracle CIT backs up close to 1.5 PB of data per week. Oracle CIT also offers lifecycle management to customers with Oracle Exadata and Oracle Exalogic solutions, from purchasing to maintenance and support.

Over the next year, Oracle CIT expects to see the same 75 percent cost reduction from its use of Sun storage that Oracle PDIT has achieved. “We expect the compression deduplication ratios to be significantly better than what we were achieving with other vendors’ products,” says Ajay Srivastava, vice president of service engineering in Oracle CIT.

Advanced features in Oracle’s Sun unified storage solutions bring many benefits to Oracle On Demand customers. Storage systems are optimized to use flash cache, which increases application speed and reduces latency. The Sun ZFS Storage Appliance Shadow Migration feature has made transitioning data from other storage devices simpler and less expensive. Srivastava also sees benefits in duplication, replication compression, and advanced monitoring. “These features have allowed us to replace multiple storage devices from other vendors with a single unified storage system,” he says.

Oracle CIT uses StorageTek tape storage to back up all customer data. Tape backup is vital to maintaining Oracle On Demand’s service-level agreements, which guarantee customers backup of their production and nonproduction data for a specified amount of time. “StorageTek tape storage products provide greater flexibility and security for complex and heterogeneous environments managed by Oracle On Demand,” says Srivastava.

Sun servers have been standard hardware for Oracle On Demand services for more than five years. More than 3,000 Sun servers support Oracle CIT’s multitiered architectures, in both virtualized and nonvirtualized environments. “Sun servers have been very stable and high-performing,” says Srivastava. “There is greater reliability and increased uptime, which means higher availability to our customers.”

Because Oracle Exadata is a complete package of servers, storage, networking, and software that is massively scalable, secure, and redundant, it meets customers’ specific high-performance requirements. The benefits of Oracle Exadata for Oracle On Demand customers include high levels of performance with very low-latency response times. Srivastava says this has helped Oracle On Demand meet the most-demanding customer loads, while maintaining customer satisfaction.

“Customers see unparalleled price performance with Oracle Exadata,” says Oracle CIO Mark Sunday. “They have the servers, storage, networking, and database engineered together, and purchased at a significantly lower cost. The performance is so great, it’s enabling them to do things they’ve never done before.”

One Platform for All

Overall, Oracle’s IT department is seeing significant benefits from using its Sun hardware in the data center. It is also working closely with Oracle hardware engineering and development teams to optimize the hardware and software to work well together, creating a more cost-effective, efficient, and reliable system both internally and for Oracle customers.

Webb says Oracle has spent millions on storage in the past year alone, so getting the most out of that investment is critical to the success of IT at the company. Continuing into 2011, Webb will be deploying Oracle Fusion Customer Relationship Management for Oracle’s sales force, completely on Oracle Exadata and Oracle Exalogic. This combination of next-generation applications with Oracle’s next-generation hardware promises to be a major boon for the company. “My vision is really a simple one,” says Webb. “When you walk down the aisle of an Oracle data center, all you’ll see is Oracle Exadata and Exalogic.”

Sunday says the hardware work being done inside Oracle IT will ultimately pay major dividends for customers. “We are able to leverage what we are doing at Oracle to continue to drive advancements in our products,” he says. “These advancements help improve performance, scalability, reliability, and maintainability, and reduce total cost of ownership.”

For More Information

Oracle IT: Best-Practices Laboratory
Oracle’s Sun Servers and Storage
Oracle Next-Generation Data Center


Monica Mehta is a frequent contributor to Profit.