DOWNLOAD Oracle Database 10g

TAGS rac, dba, clustering, All

Putting Oracle RAC to the Test

Read the results of the Oracle RAC SIG's punishing trial of Oracle Real Application Clusters.

Published November 2006

It is now the fourth year since the creation of the Oracle Real Application Clusters Special Interest Group (RAC SIG). Its purpose is to enhance the experience of RAC customers by providing a forum for technical interchange.

With the RAC SIG's continuous focus toward helping improve Grid Control and the RAC product, recently the RAC SIG participated in an "all hands on deck" testing event at Oracle HQ in Redwood Shores, Calif. We joined several of Oracle's largest and best-known customers. During this event, the Oracle RAC developers who wrote the product were on hand as we "ripped" through the product to find any deficiencies they could fix. The event consisted of 15 RAC users ("hackers") from around the world in one room for four solid days, with tools and clusters provided, with one focus: "Test until we could break it and then get the developers to fix it."

While Oracle has done these types of events several times with every major release of Oracle products; this is probably the first where the focus was entirely on RAC-related enhancements and the new tools introduced to help the RAC administrators. In this article, we'll provide the testing results for some these features and tools.

By the way, the Independent Oracle Users Group (IOUG) is another great resource for RAC users. Joining the IOUG will not only give you some great RAC/Grid best practices, information and resources, but will also get you a discount to the next IOUG event, Collaborate '07, in Las Vegas.

Provisioning

When a node is to be added to the cluster, DBAs used to have to go through heaps and bounces to apply all the required components and patches. And then install all the required Oracle binaries before the node is part of the cluster. Now this feature is built into Enterprise Manager Grid Control. You can now add one or more nodes as a member of an existing cluster by deploying a pre-patched image and

2006 "RAC Hackers" at Oracle HQ

Alagu Muthukumaran and Kumar Yechuri, Wells Fargo Alon Spiegel, Xpert-One Amit Kumar, Webex Charles Kim, Fidelity National Financial Dan Norris, IT Convergence Daniel Liu, First American Real Estate Solutions Logan McLeod and Madan Gopal, Dell Murali Vallath, Summersky Enterprises Rich Niemiec, TUSC Sanjay Shetty, MFG Systems Corp Satish Mehta, Yahoo Satya Krishnaswamy, Hewlett Packard Sergio Del Rio, Templates 4 Business Inc

configuring them. Enterprise Manager gives the option to either source that image from an existing node or from a repository of software images, called the Software Library. The idea is that one would install the software on a test node and patch it to the right level, test it, and then approve it (gold image) to be distributed over the enterprise. In Oracle's gold image creation step, one can point to an installed ORACLE_HOME (gold image) on an existing host (that has to run the Enterprise Manager Agent) and then Enterprise Manager will zip it up and store it in the Software Library.

DBAs would recollect that in 10.2.0.2 EM, one could create or extend a cluster seamlessly in few simple steps:

1. Agent deployment
2. Oracle Clusterware Deployment (CRS) and Configuration
3. ASM deployment (optional)
4. RAC software deployment
5. Add instance ("Database creation" for new cluster/"Instance addition" for extension case)

All these steps are also exposed through EMCLI (command line interface), which makes it possible for integrators to call them directly from other scripts. For example, you can use the metrics in the EM repository (the EM schema is open and published) to add a RAC node if the CPU load reaches beyond a threshold.

We ran some POCs (Proof of Concept) on this feature during the RAC Test Session. The team tested this feature by adding a third node to the existing clusters as part of the testing. The team then provided the list of errors and bugs to be fixed followed by making suggestions on where this could be improved.

The session was preceded by a roadmap presentation where it was mentioned that provisioning a node would be a simple single step operation in the coming version. Enterprise manager will be smart enough to automatically figure out the existing components in the new node (in case of a previously unfinished installation) and incrementally provision the additional components.

EM version 10.2.0.1 functionality required some patches and workarounds, which are listed in this white paper.

Step 1: Agent Deployment. This is where you push the agent to the new node of the cluster. Here is an example of one of the main screens where the source is listed and the New Agent Installation is specified. This is after the source is picked from a list and credentials are verified:

Step 2: Oracle Clusterware Deployment. Here we are cloning from the first node to the third node. Enterprise Manager pre-populates the wizard with the post cloning scripts that need to be run and can automatically execute them if superuser credentials are set up on the target node with sudo (a command that allows the user to execute a command as the superuser or other user). Here is one of the important screens in the seven-step wizard showing the source and destination for the CRS clone:

Step 3: ASM Deployment (optional). If you are using ASM, here is one of the important screens in the seven-step wizard showing the source and destination for the ASM clone:

This screen shows how the job gets setup in the scheduler where it can be monitored and retried in case of failure:

Step 4: RAC Deployment. Now we are ready to clone the Database Home (not the instance yet). We are going to clone the Database Home of the first node to the third node. This is a seven-step wizard. Here again, the Enterprise manager can automatically run the root.sh script if sudo is set up and perform the post-cloning configuration of the database and ASM software. This time I also show the opening step where all "source" locations are shown and then I skip to step three where the source and destination (as with the other clones we did earlier) are both displayed. There are also steps that will validate your credentials.

This screen shows how the job gets setup in the scheduler where it can be monitored:

Under the Administration Tab, we'll choose the Add Instance Link, supply some credentials and then add the instance as shown below:

There are some review and confirmation screens and then the job is scheduled for your viewing:

The provisioning feature will help DBAs to cut down the software and instance deployment time by a huge margin. Since tested images are pre-patched and provisioned, it cuts down the painstaking steps to patch individual nodes and ensure all of them are identical.

Clusterware for Third-Party Applications

In Oracle Database 10g Release 2, the clusterware was made into a generic product—it can now be used with non-Oracle applications to provide high availability. During our stay at the Oracle HQ, Oracle demonstrated this feature and through practical hands on session allowed us to learn this feature.

Here is the Oracle Clusterware solution and a diagram of the RAC implementation showing the high availability (HA) solution that it creates:

Now let's take this outside the Oracle Database. The Clusterware API provides options to integrate third party applications into the Oracle Clusterware allowing the application to take advantage of the clusterware stack and provide high availability; by migrating applications or components of applications from one server to another should the server fail. While the details of this are beyond the scope of this article, there are three key controls that are available: crs_start, crs_stop, and check will invoke the agent. The framework is a simple scripting language. You may, for instance want to invoke the apache command apachectl -k start for crs_start:

You may, for examplr, want to invoke the apache command apachectl -k stop for crs_stop (once coded, the command is run at will):

You may also want to invoke various commands that check whether various things are running ("Is the httpd process running?" or "Can I request a Web page?"). You can put this in the check command:

De-install Utilities

To help the RAC database administrators during the testing phase of using RAC and to de-install the product binaries with ease, Oracle will introduce de-install utilities in a later version of 10g Release 2. This is one of the most requested tools that I've heard customers ask for as a way to quickly and cleanly de-install and de-configure an entire RAC installation.

This product de-configures and de-installs the Oracle clusterware AND Oracle Homes. One of the nicest things about it is that it works with a failed configuration of Oracle RAC and Clusterware homes, not just successful ones. It also downgrades a failed upgrade of the Oracle databases.

We performed extensive testing of the product and found this to be one of the best new offerings that Oracle had for DBAs to use during the testing phase of a new implementation. Here's what the tool does:

• Discovers current state of installation and configuration
• Interacts with the user to confirm discovered data
• Gets user inputs for any undiscovered data
• Saves user inputs to a file for later use
• Cleans installation and configuration data files
• Removes Oracle software

Failed installation and configuration state discovery (Oracle RAC Home)

Cleanup of installation and configuration data (Oracle RAC Home)

Failed installation and configuration state discovery (Oracle CRS Home)

Cleanup of installation and configuration data (Oracle CRS Home)

And here is a sample log from the de-install and de-configuration tool:

		[oracle@bpo45 logs]$ more deinstall_deconfig.out2

############ CLUSTERWARE/RAC DEINSTALL DECONFIG TOOL START ############


######################## CHECK OPERATION START ########################
Install check configuration START

Checking for existance of the Oracle home location /rac01/app/oracle/product/10.2.0/db_2
Checking for existance of central inventory location /rac01/app/oracle/oraInventory
Checking for existance of the Oracle clusterware home /rac01/crs/oracle/product/10/crs
The following nodes are part of this cluster: bpo45,bpo46

Install check configuration END

Network Configuration check config START
Network de-configuration trace file location: /rac01/app/oracle/deconfig/clusterdeconfig/logs/netdc_check.log.
Some locally configured listeners were discovered in node bpo45 and they were stopped
Some locally configured listeners were discovered in node bpo45 and they were stopped
Some locally configured listeners were discovered in node bpo46 and they were stopped
Some locally configured listeners were discovered in node bpo46 and they were stopped

Specify the list of listener prefixes that are configured in this Oracle home []:

Specify the list of listener prefixes to be migrated to another database or ASM home of 
version Oracle Database 10g or above. This ensures that clients can continue to connect to other
 Oracle instances after the migration. Specify "." if you do not want to migrate any listener prefixes. 
 Listeners that you do not specify here will be de-configured. []:
Network Configuration check config END

Database Check Configuration START

Database de-configuration trace file location '/rac01/app/oracle/deconfig/clusterdeconfig/logs/assistantsdc_check51104.log
'

Specify the list of database names that are configured in this Oracle home [orclF]:

###### For ASM Instance ######
Specify the list of nodes on which ASM is configured []:

###### For Database 'orclF' ######

Specify the nodes on which this database has instances [bpo45, bpo46]: 

Specify the instance names [orclF1, orclF2]: 

Specify the local instance name on node bpo45  [orclF1]: 

Specify whether this was an upgraded database. The de-configuration tool will attempt to downgrade 
the database to the lower version if it is an upgraded database [n]: 

Specify the storage type used by the Database ASM|CFS|RAW [ASM]:

Specify the Oracle home where the ASM instance is configured [/rac01/app/oracle/product/10.2.0/db_1]:

Specify the local ASM instance on the node [+ASM1]:

Specify the list of disk_groups used for the database files, including the flash recovery area if configured [DATA,DEMO]:

Specify the list of directories if any database files exist on a shared file system. 
If 'orclF' subdirectory is found, then it will be deleted. Otherwise, the specified directory will be deleted. 
Alternatively, you can specify list of database files with full path [+DATA/ORCLF]:

Specify the flash recovery area location, if it is configured on the file system. If 'orclF' subdirectory is found,
then it will be deleted. []:
Specify the database dump destination directory [/rac01/app/oracle/product/10.2.0/db_2/admin/orclF]:
Specify the archive log destination directories []: Database Check Configuration END
Enterprise Manager Configuration Assistant START
Checking configuration for database orclF
Enterprise Manager Configuration Assistant END

######################### CHECK OPERATION END #########################


############################### SUMMARY ###############################
Oracle Home selected for de-install is: /rac01/app/oracle/product/10.2.0/db_2
Inventory Location where the Oracle home registered is: /rac01/app/oracle/oraInventory
Oracle Clusterware Home is: /rac01/crs/oracle/product/10/crs
The cluster node(s) on which the Oracle home exists are: bpo45,bpo46
Stopping of local listener LISTENER_BPO37 succeeded.
Stopping of local listener LISTENER_BPO37 succeeded.
The following databases were selected for de-configuration : orclF
Database unique name : orclF
Storage used : ASM
Will update the Enterprise Manager configuration for the following database(s): orclF
No Enterprise Manager ASM targets to update
No Enterprise Manager listener targets to migrate Log file location is /rac01/app/oracle/deconfig/clusterdeconfig/logs/deinstall_deconfig.out2
Error file location is /rac01/app/oracle/deconfig/clusterdeconfig/logs/deinstall_deconfig.err2

############# CLUSTERWARE/RAC DEINSTALL DECONFIG TOOL END #############

Oracle Enterprise Manager 10g Grid Control Features

With Oracle Enterprise Manager 10g Grid Control, you now have a tremendous product at your side. This section will look at a few of the screens that you can use to monitor the grid. But, we warn you, there are so many great screens and tools that could never do the product justice.

One of the best screens is one that we saw earlier in the article: It's the screen to click on a cluster and see whether the nodes are up or down as well as see the individual nodes. Here is the cluster ioug showing six nodes that are all up. To get to this screen, we just went to the Targets tab and clicked on the ioug cluster.

Move down the page a bit, you can see the instances (all using ASM) that are associated with this cluster.

If you click on the Topology tab, you can see the topology for all six instances (each instance is on a separate node, so there are also six separate nodes. Notice that my mouse is over one of the instances and additional information about the instance is provided.

If you click on the Performance tab and then click onto the CPU Used chart, you can see performance all nodes in the "ioug" cluster, each in a different color.

And there's much more to this tool including interconnect information and global block transfer information. It is a great tool for monitoring the grid.

Special thanks to Angelo Pruscino, Sohan DeMel, Erik Peterson, Debbie Migliore, Sudip Datta, Philip Newlan, Jagan Athreya, Anil Khilani, Bharat Paliwal,Ron Weiss, Ara Shakian, Prabhaker Gongloor (GP), Barb Lundhild, Kirk McGowan, Carol Colrain, Stefan Pommerenk, Troy Anthony, Sue Tang, Joshua Solomin, Eunhei Jang, Supratim Chowdhury, Sourav Mukherjee and Khethavath Singh (KP)

Rich Niemiec is the CEO of The Ultimate Software Consultants (TUSC), an Oracle Certified Advantage Partner. He is also the author of the Oracle Press best-selling book Oracle9i Performance Tuning Tips and Techniques. Rich has been giving lectures and presentations on Oracle for the past 19 years and is the past President of the IOUG and the current President Midwest Oracle Users Group (MOUG) and VP of the RAC SIG.