Inside Oracle Database 10g Release 2 Beta Week Testing
By Bill Burke & Byron Pearce

Once again, the "Dirty Dozen" pushes a beta release to its limits, and learns some interesting things in the process

If you are an OTN member of long standing, you may recall that in 2003, Oracle and the International Oracle Users Group (IOUG) collaborated to launch the first-ever beta migration test at Oracle HQ in Redwood Shores, Calif. The idea was to assemble a crack team of one dozen DBAs, transport them to a lab with production databases in tow, and give the beta a rigorous "road test" over the course of an entire week.

This project was extremely successful for IOUG as well as for the Oracle Database product management team; working together, we uncovered issues that probably only would have been noticed after the fact in real-world deployments—making for a better product and user experience.

In February 2005, Oracle approached the user community during the International Oracle Users Council (IOUC) meeting about participating in a similar week-long event to test Oracle Oracle Database 10g Release 2. This time, international Oracle user groups would be included to support enhanced testing of new features for globalization support. Consequently, members of the Asia Pacific Oracle Users Group (APOUG) and European Oracle Users Group (EOUG) joined others from the IOUG to form a multinational team of 15 members, some of whom have nearly a quarter century of Oracle experience in a broad array of areas, from Oracle Real Application Clusters (RAC), to unstructured content management, to application development. (See sidebar for the identities of this event's "Dirty Dozen.")

The Mission

Our mission would, be once again, to bring as many real, production databases as we could carry into the lab and then successfully upgrade them to Oracle Database 10g Release 2, testing as much of the technology stack as possible in the process. The goal was not only to hammer the database, but also to challenge the Oracle developers in attendance with real-world production problems and scenarios.

The Process

The process had four main components:

1. Upgrade production databases from a previous version to Oracle Database 10g Release 2 using a supported method (inline, leap-frog, export/import, data extraction/data load, etc.)
2. Exercise features and functionality of the new release, and perform comparative performance testing.
3. Collaborate with Oracle development teams to identify and define bugs or other issues.
4. Provide constructive feedback on the overall upgrade process, documentation, Oracle Universal Installer (OUI), Database Upgrade Assistant (DBUA), Database Configuration Assistant (DBCA), and other utilities.

The approach for this event was to encompass as much of the new Oracle Database 10g Release 2 core technology stack as possible by testing on single instance and RAC configurations. In addition, consideration was given to the type of upgrade testing to be performed, the feature functionality of the existing database(s) which would be included for testing, and the upgrade paths that would be utilized in the actual upgrade process. Several of the original participants from the first event were also present to provide a base of veteran testers.

The Platforms

Testing was performed on Red Hat Linux Enterprise Server running on Dell Intel machines and Solaris running on Sun SPARC platforms. Each machine had a limited amount of internal storage with a Network Appliance Filer providing additional storage, as well as iSCSI storage for Oracle Cluster File System (OCFS) and Automatic Storage Management (ASM) utilization on the RAC clusters. Engineers from Network Appliance were on-site to support the devices and provide an overview of the technology behind them to those who were interested. The OS and initial Oracle versions were pre-installed to save time in getting the initial databases up and running.

The Week in Review

Once again, Debbie Migliore and the Beta Program Office staff from Oracle hosted the event at a brand-new lab specifically built for this type of event.

With the exception of a Solaris notebook that died in transit, all the databases were installed and up and running quickly on their new "home" systems under their original version. Targeted upgrades included:

1. Single-instance upgrade from Oracle8i to Oracle Database 10g Release 2 on Solaris
2. Single-instance upgrade from Oracle9i to Oracle Database 10g Release 2 on Linux and Solaris
3. Single-instance upgrade from Oracle Database 10g Release 1 to Oracle Database 10g Release 2 on Linux and Solaris
4. Oracle RAC 2-node cluster from Oracle9i to Oracle Database 10g Release 2 on Linux and Solaris
5. Oracle RAC 2-node cluster from Oracle Database 10g Release 1 to Oracle Database 10g Release 2 on Linux and Solaris

For example, several of the testers and product managers had the opportunity to discuss the requirements for a single installation of Oracle Cluster Ready Services (CRS) on Oracle RAC. Assuming that CRS for Oracle Database 10g Release 1 had been installed according to OFA standards (and the Oracle Database 10g Release 1 RAC Installation Guide), then $ORACLE_HOME would look something like this:

/u01/app/oracle/product/10.1.0/db_1 

/u01/app/oracle/product/10.1.0/crs_1

Without this collaboration between the testers and the Oracle RAC team, this discrepancy may not have been uncovered and communicated to Oracle in as nearly a timely fashion.

The Dirty Dozen's Favorite Features

While a discussion of all the new features of Oracle Database 10g Release 2 requires a series of articles—and in fact, you can read such a series on OTN—we wanted to at least provide the test team's collective list (in no particular order) of favorite features here.

Oracle Automatic Storage Management Enhancements. Oracle Automatic Storage Management (ASM), introduced with Oracle Database 10g Release 1, provides the ability to intelligently stripe and mirror data across storage devices, including raw, physical disks. The initial release addressed many of the performance and redundancy considerations of storage management, such as mirroring disks and writing directly to the raw device to achieve better performance and eliminate filesystem overhead. However, management of the storage had to take place via the SQL*Plus interface, and it was a bit unwieldy to have to construct often elaborate queries and subqueries to view simple information about the storage subsystem configuration.

With Oracle Database 10g Release 2, Oracle ASM now has a command-line interface called asmcmd that offers a Unix-like interface to the files stored in ASM. With asmcmd, the gulf between usability and manageability has been bridged and the majority of common tasks can be performed through this interface.

Several other challenges with ASM have been resolved in the newest release—among them, the ability to easily move files into and out of the Oracle ASM instance using basic tools such as FTP. This capability simplifies conversion to ASM by providing a crucial ability to effectively use Transportable Tablespaces for that purpose.

Oracle Secure Backup. Backups have always been a sticky subject between the DBAs and the various systems teams. Sure, RMAN performs the backup to disk, but what assurance is there that it actually made the final journey to tape?

With Oracle Database 10g Release 2, you can resolve this issue with the implementation of Oracle Secure Backup (OSB). This tool provides Oracle with a full media manager, capable of managing tapes and libraries, that performs database as well as OS backups of local or remote systems. OSB integrates with RMAN and the Flashback Recovery Area (FRA) in Oracle Database 10g, and can be monitored and scheduled via Oracle Enterprise Manager. With OSB, a truly lights-out backup system can be put in place, and a simple yet extremely informative report in Enterprise Manager can be viewed to determine the status of recent backups.

OSB has the potential to greatly streamline the backup process (especially in smaller shops where an enterprise backup solution may not already be in place). As tester Leng Kaing puts it, "At last, we no longer have to point fingers about who's problem it is when the backup fails. It's all done by Oracle, so there's a central point of contact for all of our inquiries!"

Oracle RAC Enhancements. While several improvements have been made to RAC, the one that got the most attention was the introduction of a pre-installation tool called Cluster Verify Utility (CVU). By executing CVU prior to installation, all the components necessary for a clean installation of the Oracle RAC cluster are checked and success/failure reports are generated. Thus the DBA can review the status of a given set of hosts and address any outstanding issues before starting the installation. Given how vital a correct installation is to the long-term success of an Oracle RAC project, this is a welcome and much-needed tool.

One quibble with Oracle RAC 10g Release 1 is the fact that both the Cluster Repository (OCR) and the Voting Disk cannot be multiplexed by the database. This limitation can lead to a built-in single point of failure unless mirroring of the disk containing these resources can be done via the hardware. Fortunately, that issue has been resolved, and multiple locations for OCR and Voting can be selected during installation.

Finally, the new concept of organizing work into services—and integrating them with Oracle RAC by allowing resources to be managed as a unit—earned high marks from the testers. These services include things like TCP/IP timeout management, load-balancing based on various metrics (including run-time/performance-based balancing among nodes), and the ability to set service thresholds and send alerts if those thresholds are not met.

XML Improvements. Previous versions of Oracle Database include many XML features, but Oracle Database 10g Release 2 offers a much improved ability to manage and process XML data. For example, in Oracle XML DB, the Xpath Rewrite can now handle many more complicated operations and yield results several orders of magnitude faster without any changes to code. Oracle XML features also now include the ability to add and remove nodes from a document. Now it is also possible to manage Oracle XML DB via Oracle Enterprise Manager and to have transportable tablespaces containing XMLType data.

This list simply doesn't do adequate justice to the positive feedback shared collectively with Oracle's development managers; there are many more features that truly shine in Oracle Oracle Database 10g Release 2 that we simply don't have space to cover here. (Again, Arup Nanda's "Oracle Database 10g: Top Features for DBAs" Release 2 Addendum offers a much more complete and detailed list.) You will find that the enhancements, new features, and improved install/upgrade utilities make a tremendous difference in the ease of transitioning to this new release.

What About Performance?

New features are great for the DBAs in the trenches. But how well will the database perform under the new version?

We documented several internal performance enhancements, including:

• Asynchronous commits
• Automatic tuning of db_file_multiblock_read_count
• Up to 10x improvement in sort performance
• Up to 3x improvement in aggregate performance

Happy Migrating

Once again, the commitment from all involved to create a technically challenging environment made taking a week out of everyone's saturated business schedule more than worthwhile.

Where workarounds were needed, development was side by side with the team as the challenges were undertaken. The new documentation was functionality tested from scratch, which pointed up a number of opportunities for clarification and streamlining of process.

At the end of the week—and after more than 600 person-hours of in-depth testing—our findings were unanimous: If the handful of issues we considered critical are addressed prior to the products release, Oracle Database 10g Release 2 will be a significant evolution in the Oracle Database 10g product line.

Special thanks to everyone who participated in the IOUC testing event, but especially to Sal Bingham, Leng Kaing, Charles Kim, Daniel Liu, Mahesh Pujari, and Murali Vallath for providing valuable input and examples for this article.