Build Your Own Oracle RAC 10g Release 2 Cluster on Linux and FireWire (Continued)

For development and testing only; production deployments will not be supported!

Perform the following installation procedures on only one node in the cluster! The Oracle Clusterware software will be installed to all other nodes in the cluster by the Oracle Universal Installer.

You are now ready to install the "cluster" part of the environment - the Oracle Clusterware. In the previous section, you downloaded and extracted the install files for Oracle Clusterware to linux1 in the directory /u01/app/oracle/orainstall/clusterware. This is the only node from which you need to perform the install.

During the installation of Oracle Clusterware, you will be asked for the nodes involved and to configure in the RAC cluster. Once the actual installation starts, it will copy the required software to all nodes using the remote access we configured in the section Section 13 ("Configure RAC Nodes for Remote Access").

So, what exactly is the Oracle Clusterware responsible for? It contains all of the cluster and database configuration metadata along with several system management features for RAC. It allows the DBA to register and invite an Oracle instance (or instances) to the cluster. During normal operation, Oracle Clusterware will send messages (via a special ping operation) to all nodes configured in the cluster, often called the "heartbeat." If the heartbeat fails for any of the nodes, it checks with the Oracle Clusterware configuration files (on the shared disk) to distinguish between a real node failure and a network failure.

After installing Oracle Clusterware, the Oracle Universal Installer (OUI) used to install the Oracle 10g database software (next section) will automatically recognize these nodes. Like the Oracle Clusterware install you will be performing in this section, the Oracle Database 10g software only needs to be run from one node. The OUI will copy the software packages to all nodes configured in the RAC cluster.

Oracle Clusterware Shared Files

The two shared files (actually file groups) used by Oracle Clusterware will be stored on the Oracle Cluster File System, Release 2 (OFCS2) we created earlier. The two shared Oracle Clusterware file groups are:

• Oracle Cluster Registry (OCR)
  • File 1 : /u02/oradata/orcl/OCRFile
  • File 2 : /u02/oradata/orcl/OCRFile_mirror
  • Size : (2 * 100MB) = 200M

• CRS Voting Disk
  • File 1 : /u02/oradata/orcl/CSSFile
  • File 2 : /u02/oradata/orcl/CSSFile_mirror1
  • File 3 : /u02/oradata/orcl/CSSFile_mirror2
  • Size : (3 * 20MB) = 60MB

Note: It is not possible to use Automatic Storage Management (ASM) for the two shared Oracle Clusterware files: Oracle Cluster Registry (OCR) or the CRS Voting Disk files. The problem is that these files need to be in place and accessible BEFORE any Oracle instances can be started. For ASM to be available, the ASM instance would need to be run first.

Also note that the two shared files could be stored on the OCFS2, shared RAW devices, or another vendor's clustered file system.

Verifying Terminal Shell Environment

Verify Server and Enable X Server Access

# hostname
linux1
# xhost +
access control disabled, clients can connect from any host

Login as the oracle User Account and Set DISPLAY (if necessary)

# su - oracle
$ # IF YOU ARE USING A REMOTE CLIENT TO CONNECT TO THE
$ # NODE PERFORMING THE INSTALL
$ DISPLAY=<your local workstation>:0.0
$ export DISPLAY

Verify Remote Access / User Equivalence

Verify you are able to run the Secure Shell commands (ssh or scp) or the Remote Shell commands (rsh and rcp) on the Linux server you will be running the Oracle Universal Installer from against all other Linux servers in the cluster without being prompted for a password.

When using the secure shell method, user equivalence will need to be enabled on any new terminal shell session before attempting to run the OUI. To enable user equivalence for the current terminal shell session, perform the following steps remembering to enter the pass phrase for each key that you generated when prompted:

$ exec /usr/bin/ssh-agent $SHELL
$ /usr/bin/ssh-add
Enter passphrase for /u01/app/oracle/.ssh/id_rsa: xxxxx
Identity added: /u01/app/oracle/.ssh/id_rsa (/u01/app/oracle/.ssh/id_rsa)
Identity added: /u01/app/oracle/.ssh/id_dsa (/u01/app/oracle/.ssh/id_dsa)
$ ssh linux1 "date;hostname"
Wed Sep 13 18:27:17 EDT 2006
linux1
$ ssh linux2 "date;hostname"
Wed Sep 13 18:28:00 EDT 2006
linux2

When using the remote shell method, user equivalence is generally defined in the /etc/hosts.equiv file for the oracle user account and is enabled on all new terminal shell sessions:

$ rsh linux1 "date;hostname"
Tue Sep 12 12:37:17 EDT 2006
linux1
$ rsh linux2 "date;hostname"
Tue Sep 12 12:38:00 EDT 2006
linux2

Installing Cluster Ready Services

clssnmDiskPingMonitorThread: voting device access hanging (45010 miliseconds)

With 10.1.0.2, the timeout value was calculated as follows:

time_in_secs > CSS misscount, then EXIT

Starting with 10.1.0.3, the formula was changed to:

disktimeout_in_secs = MAX((3 * CSS misscount)/4, CSS misscount - 15)

This change was motivated mainly to allow for a faster cluster reconfiguration in case of node failure. With the default CSS misscount value of 60 in 10.1.0.2, we would have to wait at least 60 seconds for a timeout, where this same default value of 60 can be shaved by 15 seconds to 45 seconds starting with 10.1.0.3.

OK, so why all the talk about CSS misscount? As I mentioned earlier, I would often have the database creation process fail (or other high I/O loads to the system) from the Oracle Clusterware crashing. The high I/O would cause lengthy timeouts for CSS while attempting to query the voting disk. When the calculated timeout was exceeded, Oracle Clusterware crashed. This problem is likely to occur as the FireWire drives you are using are not blazing-fast. The slower the drive, the more often this will occur.

Well, the good news is that you can modify the CSS misscount value from its default value of 60 (for Linux) to allow for lengthier timeouts. For the drives you have been using with this guide, you can get away with a CSS misscount value of 360. Although I haven't been able to verify it, I believe the CSS misscount can be set as large as 600.

So how do you modify the default value for CSS misscount? Well, there are several ways. The easiest way is to modify the root.sh for Oracle Clusterware before running it on each node in the cluster. (The instructions for modifying the root.sh script for Oracle Clusterware can be found here.)

If Oracle Clusterware is already installed, you can still modify the CSS misscount value using the $ORA_CRS_HOME/bin/crsctl command. (The instructions for verifying and modifying the CSS misscount using crsctl can be found in the section "Verify Oracle Clusterware / CSS misscount value".)

Perform the following tasks to install the Oracle Clusterware:

$ cd ~oracle
$ /u01/app/oracle/orainstall/clusterware/runInstaller -ignoreSysPrereqs
      

Verify Oracle Clusterware / CSS misscount value

In the section "CSS Timeout Computation in 10g RAC 10.1.0.3", I mentioned the need to modify the CSS misscount value from its default value of 60 to 360 (or higher). Within that section I explained how to accomplish that by modifying the root.sh script before running it on each node in the cluster. If you were not able to modify the CSS misscount value within the root.sh script, you can still perform this action by using the $ORA_CRS_HOME/bin/crsctl program. For example, to obtain the current value for CSS misscount, use the following:

$ORA_CRS_HOME/bin/crsctl get css misscount
360

• Start only one node in the cluster. For my example, I would shutdown linux2 and startup only linux1.
• From the one node (linux1), login as the root user account and type:

  $ORA_CRS_HOME/bin/crsctl set css misscount 360

• Reboot the single node (linux1).
• Start all other nodes in the cluster.

Verify Oracle Clusterware Installation

After the installation of Oracle Clusterware, you can run through several tests to verify the install was successful. Run the following commands on all nodes in the RAC cluster.

Check cluster nodes

$ /u01/app/oracle/product/crs/bin/olsnodes -n
linux1 1
linux2 2

$ ls -l /etc/init.d/init.*
-r-xr-xr-x 1 root root 1951 Oct 4 14:21 /etc/init.d/init.crs*
-r-xr-xr-x 1 root root 4714 Oct 4 14:21 /etc/init.d/init.crsd*
-r-xr-xr-x 1 root root 35394 Oct 4 14:21 /etc/init.d/init.cssd*
-r-xr-xr-x 1 root root 3190 Oct 4 14:21 /etc/init.d/init.evmd*

Perform the following installation procedures on only one node in the cluster! The Oracle database software will be installed to all other nodes in the cluster by the Oracle Universal Installer.

After successfully installing the Oracle Clusterware software, the next step is to install the Oracle Database 10g Release 2 (10.2.0.1.0) with RAC.

For the purpose of this example, you will forgo the "Create Database" option when installing the software. You will, instead, create the database using the Database Creation Assistant (DBCA) after the install.

Like the Oracle Clusterware install (previous section), the Oracle10g database software only needs to be run from one node. The OUI will copy the software packages to all nodes configured in the RAC cluster.

Verifying Terminal Shell Environment

Login as the oracle User Account and Set DISPLAY (if necessary)

# su - oracle
$ # IF YOU ARE USING A REMOTE CLIENT TO CONNECT TO THE
$ # NODE PERFORMING THE INSTALL
$ DISPLAY=<your local workstation>:0.0
$ export DISPLAY

Verify Remote Access / User Equivalence

Verify you are able to run the Secure Shell commands (ssh or scp) or the Remote Shell commands (rsh and rcp) on the Linux server you will be running the Oracle Universal Installer from against all other Linux servers in the cluster without being prompted for a password.

When using the secure shell method, user equivalence will need to be enabled on any new terminal shell session before attempting to run the OUI. To enable user equivalence for the current terminal shell session, perform the following steps remembering to enter the pass phrase for each key that you generated when prompted:

$ exec /usr/bin/ssh-agent $SHELL
$ /usr/bin/ssh-add
Enter passphrase for /u01/app/oracle/.ssh/id_rsa: xxxxx
Identity added: /u01/app/oracle/.ssh/id_rsa (/u01/app/oracle/.ssh/id_rsa)
Identity added: /u01/app/oracle/.ssh/id_dsa (/u01/app/oracle/.ssh/id_dsa)
$ ssh linux1 "date;hostname"
Wed Sep 13 18:27:17 EDT 2006
linux1
$ ssh linux2 "date;hostname"
Wed Sep 13 18:28:00 EDT 2006
linux2

When using the remote shell method, user equivalence is generally defined in the /etc/hosts.equiv file for the oracle user account and is enabled on all new terminal shell sessions:

$ rsh linux1 "date;hostname"
Tue Sep 12 12:37:17 EDT 2006
linux1
$ rsh linux2 "date;hostname"
Tue Sep 12 12:38:00 EDT 2006
linux2

Install Oracle Database 10g Release 2 Software

Install the Oracle Database 10g Release 2 software with the following:

$ cd ~oracle
$ /u01/app/oracle/orainstall/database/runInstaller -ignoreSysPrereqs
      

Perform the following installation procedures from only one node in the cluster! The Oracle Database 10g Companion CD software will be installed to all other nodes in the cluster by the Oracle Universal Installer.

After successfully installing the Oracle Database software, the next step is to install the Oracle Database 10g Release 2 Companion CD software (10.2.0.1.0).

Please keep in mind that this is an optional step. For the purpose of this guide, my testing database will often make use of the Java Virtual Machine (Java VM) and Oracle interMedia and therefore will require the installation of the Oracle Database 10g Companion CD. The type of installation to perform will be the Oracle Database 10g Products installation type.

This installation type includes the Natively Compiled Java Libraries (NCOMP) files to improve Java performance. If you do not install the NCOMP files, the ORA-29558:JAccelerator (NCOMP) not installed error occurs when a database that uses Java VM is upgraded to the patch release.

Verifying Terminal Shell Environment

Login as the oracle User Account and Set DISPLAY (if necessary)

# su - oracle
$ # IF YOU ARE USING A REMOTE CLIENT TO CONNECT TO THE
$ # NODE PERFORMING THE INSTALL
$ DISPLAY=<your local workstation>:0.0
$ export DISPLAY

Verify Remote Access / User Equivalence

Verify you are able to run the Secure Shell commands (ssh or scp) or the Remote Shell commands (rsh and rcp) on the Linux server you will be running the Oracle Universal Installer from against all other Linux servers in the cluster without being prompted for a password.

When using the secure shell method, user equivalence will need to be enabled on any new terminal shell session before attempting to run the OUI. To enable user equivalence for the current terminal shell session, perform the following steps remembering to enter the pass phrase for each key that you generated when prompted:

$ exec /usr/bin/ssh-agent $SHELL
$ /usr/bin/ssh-add
Enter passphrase for /u01/app/oracle/.ssh/id_rsa: xxxxx
Identity added: /u01/app/oracle/.ssh/id_rsa (/u01/app/oracle/.ssh/id_rsa)
Identity added: /u01/app/oracle/.ssh/id_dsa (/u01/app/oracle/.ssh/id_dsa)
$ ssh linux1 "date;hostname"
Wed Sep 13 18:27:17 EDT 2006
linux1
$ ssh linux2 "date;hostname"
Wed Sep 13 18:28:00 EDT 2006
linux2

When using the remote shell method, user equivalence is generally defined in the /etc/hosts.equiv file for the oracle user account and is enabled on all new terminal shell sessions:

$ rsh linux1 "date;hostname"
Tue Sep 12 12:37:17 EDT 2006
linux1
$ rsh linux2 "date;hostname"
Tue Sep 12 12:38:00 EDT 2006
linux2

Install Companion CD Software

Install the Companion CD software with the following:

$ cd ~oracle
$ /u01/app/oracle/orainstall/companion/runInstaller -ignoreSysPrereqs
      

Perform the following configuration procedures from only one node in the cluster! The Network Configuration Assistant (NETCA) will setup the TNS listener in a clustered configuration on all nodes in the cluster.

The DBCA requires the Oracle TNS Listener process to be configured and running on all nodes in the RAC cluster before it can create the clustered database.

The process of creating the TNS listener only needs to be performed on one node in the cluster. All changes will be made and replicated to all nodes in the cluster. On one of the nodes (I will be using linux1) bring up the NETCA and run through the process of creating a new TNS listener process and also configure the node for local access.

Verifying Terminal Shell Environment

Login as the oracle User Account and Set DISPLAY (if necessary)

# su - oracle
$ # IF YOU ARE USING A REMOTE CLIENT TO CONNECT TO THE
$ # NODE PERFORMING THE INSTALL
$ DISPLAY=<your local workstation>:0.0
$ export DISPLAY

Verify Remote Access / User Equivalence

Verify you are able to run the Secure Shell commands (ssh or scp) or the Remote Shell commands (rsh and rcp) on the Linux server you will be running the Oracle Universal Installer from against all other Linux servers in the cluster without being prompted for a password.

When using the secure shell method, user equivalence will need to be enabled on any new terminal shell session before attempting to run the OUI. To enable user equivalence for the current terminal shell session, perform the following steps remembering to enter the pass phrase for each key that you generated when prompted:

$ exec /usr/bin/ssh-agent $SHELL
$ /usr/bin/ssh-add
Enter passphrase for /u01/app/oracle/.ssh/id_rsa: xxxxx
Identity added: /u01/app/oracle/.ssh/id_rsa (/u01/app/oracle/.ssh/id_rsa)
Identity added: /u01/app/oracle/.ssh/id_dsa (/u01/app/oracle/.ssh/id_dsa)
$ ssh linux1 "date;hostname"
Wed Sep 13 18:27:17 EDT 2006
linux1
$ ssh linux2 "date;hostname"
Wed Sep 13 18:28:00 EDT 2006
linux2

When using the remote shell method, user equivalence is generally defined in the /etc/hosts.equiv file for the oracle user account and is enabled on all new terminal shell sessions:

$ rsh linux1 "date;hostname"
Tue Sep 12 12:37:17 EDT 2006
linux1
$ rsh linux2 "date;hostname"
Tue Sep 12 12:38:00 EDT 2006
linux2

Run the Network Configuration Assistant

To start the NETCA, run the following GUI utility as the oracle user account:

$ netca &
      

Screen Name	Response
Select the Type of Oracle Net Services Configuration	Select Cluster Configuration
Select the nodes to configure	Select all of the nodes: linux1 and linux2.
Type of Configuration	Select Listener configuration.
Listener Configuration - Next 6 Screens	The following screens are now like any other normal listener configuration. You can simply accept the default parameters for the next six screens:    What do you want to do: Add    Listener name: LISTENER    Selected protocols: TCP    Port number: 1521    Configure another listener: No    Listener configuration complete! [ Next ] You will be returned to this Welcome (Type of Configuration) Screen.
Type of Configuration	Select Naming Methods configuration.
Naming Methods Configuration	The following screens are:    Selected Naming Methods: Local Naming    Naming Methods configuration complete! [ Next ] You will be returned to this Welcome (Type of Configuration) Screen.
Type of Configuration	Click Finish to exit the NETCA.

The Oracle TNS listener process should now be running on all nodes in the RAC cluster:

$ hostname
linux1
$ ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}'
LISTENER_LINUX1
=====================
$ hostname
linux2
$ ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}'
LISTENER_LINUX2

The database creation process should only be performed from one node in the cluster!

We will use the DBCA to create the clustered database.

Before executing the DBCA, make sure that $ORACLE_HOME and $PATH are set appropriately for the $ORACLE_BASE/product/10.2.0/db_1 environment.

You should also verify that all services we have installed up to this point (Oracle TNS listener, Oracle Clusterware processes, etc.) are running before attempting to start the clustered database creation process.

Verifying Terminal Shell Environment

Login as the oracle User Account and Set DISPLAY (if necessary)

# su - oracle
$ # IF YOU ARE USING A REMOTE CLIENT TO CONNECT TO THE
$ # NODE PERFORMING THE INSTALL
$ DISPLAY=<your local workstation>:0.0
$ export DISPLAY

Verify Remote Access / User Equivalence

Verify you are able to run the Secure Shell commands (ssh or scp) or the Remote Shell commands (rsh and rcp) on the Linux server you will be running the Oracle Universal Installer from against all other Linux servers in the cluster without being prompted for a password.

When using the secure shell method, user equivalence will need to be enabled on any new terminal shell session before attempting to run the OUI. To enable user equivalence for the current terminal shell session, perform the following steps remembering to enter the pass phrase for each key that you generated when prompted:

$ exec /usr/bin/ssh-agent $SHELL
$ /usr/bin/ssh-add
Enter passphrase for /u01/app/oracle/.ssh/id_rsa: xxxxx
Identity added: /u01/app/oracle/.ssh/id_rsa (/u01/app/oracle/.ssh/id_rsa)
Identity added: /u01/app/oracle/.ssh/id_dsa (/u01/app/oracle/.ssh/id_dsa)
$ ssh linux1 "date;hostname"
Wed Sep 13 18:27:17 EDT 2006
linux1
$ ssh linux2 "date;hostname"
Wed Sep 13 18:28:00 EDT 2006
linux2

When using the remote shell method, user equivalence is generally defined in the /etc/hosts.equiv file for the oracle user account and is enabled on all new terminal shell sessions:

$ rsh linux1 "date;hostname"
Tue Sep 12 12:37:17 EDT 2006
linux1
$ rsh linux2 "date;hostname"
Tue Sep 12 12:38:00 EDT 2006
linux2

Create the Clustered Database

To start the database creation process, run the following:

$ dbca &
      

When the DBCA has completed, you will have a fully functional Oracle RAC cluster running!

Create the orcltest Service

During the creation of the Oracle clustered database, you added a service named orcltest that will be used to connect to the database with TAF enabled. During several of my installs, the service was added to the tnsnames.ora, but was never updated as a service for each Oracle instance.

Use the following to verify the orcltest service was successfully added:

SQL> show parameter service
NAME TYPE VALUE
-------------------- ----------- --------------------------------
service_names string orcl.idevelopment.info, orcltest

SQL> show parameter service
NAME TYPE VALUE
-------------------- ----------- --------------------------
service_names string orcl.idevelopment.info
SQL> alter system set service_names = 
2 'orcl.idevelopment.info, orcltest.idevelopment.info' scope=both;

Ensure that the TNS networking files are configured on all nodes in the cluster!

listener.ora

We already covered how to create a TNS listener configuration file (listener.ora) for a clustered environment in Section 22. The listener.ora file should be properly configured and no modifications should be needed.

For clarity, I have included a copy of the listener.ora file from my node linux1 in this guide's support files. I've also included a copy of my tnsnames.ora file that was configured by Oracle and can be used for testing the Transparent Application Failover (TAF). This file should already be configured on each node in the RAC cluster.

You can include any of these entries on other client machines that need access to the clustered database.

Connecting to Clustered Database From an External Client

This is an optional step, but I like to perform it in order to verify my TNS files are configured correctly. Use another machine (i.e. a Windows machine connected to the network) that has Oracle installed (either 9i or 10g) and add the TNS entries (in the tnsnames.ora) from either of the nodes in the cluster that were created for the clustered database.

Then try to connect to the clustered database using all available service names defined in the tnsnames.ora file:

C:\> sqlplus system/manager@orcl2
C:\> sqlplus system/manager@orcl1
C:\> sqlplus system/manager@orcltest
C:\> sqlplus system/manager@orcl
      

When creating the clustered database, we left all tablespaces set to their default size. If you are using a large drive for the shared storage, you may want to make a sizable testing database.

Below are several optional SQL commands for modifying and creating all tablespaces for the test database. Please keep in mind that the database file names (OMF files) used in this example may differ from what Oracle creates for your environment. The following query can be used to determine the file names for your environment:

SQL> select tablespace_name, file_name
2 from dba_data_files
3 union
4 select tablespace_name, file_name
5 from dba_temp_files;
TABLESPACE_NAME FILE_NAME
--------------- --------------------------------------------------
EXAMPLE +ORCL_DATA1/orcl/datafile/example.257.570913311
INDX +ORCL_DATA1/orcl/datafile/indx.270.570920045
SYSAUX +ORCL_DATA1/orcl/datafile/sysaux.260.570913287
SYSTEM +ORCL_DATA1/orcl/datafile/system.262.570913215
TEMP +ORCL_DATA1/orcl/tempfile/temp.258.570913303
UNDOTBS1 +ORCL_DATA1/orcl/datafile/undotbs1.261.570913263
UNDOTBS2 +ORCL_DATA1/orcl/datafile/undotbs2.265.570913331
USERS +ORCL_DATA1/orcl/datafile/users.264.570913355

$ sqlplus "/ as sysdba"
SQL> create user scott identified by tiger default tablespace users;
SQL> grant dba, resource, connect to scott;
SQL> alter database datafile '+ORCL_DATA1/orcl/datafile/users.264.570913355' resize 1024m;
SQL> alter tablespace users add datafile '+ORCL_DATA1' size 1024m autoextend off;
SQL> create tablespace indx datafile '+ORCL_DATA1' size 1024m
2 autoextend on next 50m maxsize unlimited
3 extent management local autoallocate
4 segment space management auto;
SQL> alter database datafile '+ORCL_DATA1/orcl/datafile/system.262.570913215' resize 800m;
SQL> alter database datafile '+ORCL_DATA1/orcl/datafile/sysaux.260.570913287' resize 500m;
SQL> alter tablespace undotbs1 add datafile '+ORCL_DATA1' size 1024m
2 autoextend on next 50m maxsize 2048m;
SQL> alter tablespace undotbs2 add datafile '+ORCL_DATA1' size 1024m
2 autoextend on next 50m maxsize 2048m;
SQL> alter database tempfile '+ORCL_DATA1/orcl/tempfile/temp.258.570913303' resize 1024m;
      

Status Tablespace Name TS Type Ext. Mgt. Seg. Mgt. Tablespace Size Used (in bytes) Pct. Used
--------- --------------- ------------ ---------- --------- ------------------ ------------------ ---------
ONLINE UNDOTBS1 UNDO LOCAL MANUAL 1,283,457,024 85,065,728 7
ONLINE SYSAUX PERMANENT LOCAL AUTO 524,288,000 275,906,560 53
ONLINE USERS PERMANENT LOCAL AUTO 2,147,483,648 131,072 0
ONLINE SYSTEM PERMANENT LOCAL MANUAL 838,860,800 500,301,824 60
ONLINE EXAMPLE PERMANENT LOCAL AUTO 157,286,400 83,820,544 53
ONLINE INDX PERMANENT LOCAL AUTO 1,073,741,824 65,536 0
ONLINE UNDOTBS2 UNDO LOCAL MANUAL 1,283,457,024 3,801,088 0
ONLINE TEMP TEMPORARY LOCAL MANUAL 1,073,741,824 27,262,976 3
 ------------------ ------------------ ---------
avg 22
sum 8,382,316,544 976,355,328

8 rows selected.

The following RAC verification checks should be performed on all nodes in the cluster! For this guide, we will perform these checks only from linux1.

This section provides several srvctl commands and SQL queries you can use to validate your Oracle RAC 10g configuration.

There are five node-level tasks defined for SRVCTL:

• Adding and deleting node-level applications
• Setting and unsetting the environment for node-level applications
• Administering node applications
• Administering ASM instances
• Starting and stopping a group of programs that includes virtual IP addresses, listeners, Oracle Notification Services, and Oracle Enterprise Manager agents (for maintenance purposes).

Status of all instances and services

$ srvctl status database -d orcl
Instance orcl1 is running on node linux1
Instance orcl2 is running on node linux2

Status of a single instance

$ srvctl status instance -d orcl -i orcl2
Instance orcl2 is running on node linux2

Status of a named service globally across the database

$ srvctl status service -d orcl -s orcltest
Service orcltest is running on instance(s) orcl2, orcl1

Status of node applications on a particular node

$ srvctl status nodeapps -n linux1
VIP is running on node: linux1
GSD is running on node: linux1
Listener is running on node: linux1
ONS daemon is running on node: linux1

Status of an ASM instance

$ srvctl status asm -n linux1
ASM instance +ASM1 is running on node linux1.

List all configured databases

$ srvctl config database
orcl

Display configuration for our RAC database

$ srvctl config database -d orcl
linux1 orcl1 /u01/app/oracle/product/10.2.0/db_1
linux2 orcl2 /u01/app/oracle/product/10.2.0/db_1

Display all services for the specified cluster database

$ srvctl config service -d orcl
orcltest PREF: orcl2 orcl1 AVAIL:

Display the configuration for node applications - (VIP, GSD, ONS, Listener)

$ srvctl config nodeapps -n linux1 -a -g -s -l
VIP exists.: /linux1-vip/192.168.1.200/255.255.255.0/eth0:eth1
GSD exists.
ONS daemon exists.
Listener exists.

Display the configuration for the ASM instance(s)

$ srvctl config asm -n linux1
+ASM1 /u01/app/oracle/product/10.2.0/db_1

All running instances in the cluster

SELECT
 inst_id
 , instance_number inst_no
 , instance_name inst_name
 , parallel
 , status
 , database_status db_status
 , active_state state
 , host_name host
FROM gv$instance
ORDER BY inst_id;
INST_ID INST_NO INST_NAME PAR STATUS DB_STATUS STATE HOST
-------- -------- ---------- --- ------- ------------ --------- -------
1 1 orcl1 YES OPEN ACTIVE NORMAL linux1
2 2 orcl2 YES OPEN ACTIVE NORMAL linux2

All data files which are in the disk group

select name from v$datafile
union
select member from v$logfile
union
select name from v$controlfile
union
select name from v$tempfile;
NAME
-------------------------------------------
+FLASH_RECOVERY_AREA/orcl/controlfile/current.258.570913191
+FLASH_RECOVERY_AREA/orcl/onlinelog/group_1.257.570913201
+FLASH_RECOVERY_AREA/orcl/onlinelog/group_2.256.570913211
+FLASH_RECOVERY_AREA/orcl/onlinelog/group_3.259.570918285
+FLASH_RECOVERY_AREA/orcl/onlinelog/group_4.260.570918295
+ORCL_DATA1/orcl/controlfile/current.259.570913189
+ORCL_DATA1/orcl/datafile/example.257.570913311
+ORCL_DATA1/orcl/datafile/indx.270.570920045
+ORCL_DATA1/orcl/datafile/sysaux.260.570913287
+ORCL_DATA1/orcl/datafile/system.262.570913215
+ORCL_DATA1/orcl/datafile/undotbs1.261.570913263
+ORCL_DATA1/orcl/datafile/undotbs1.271.570920865
+ORCL_DATA1/orcl/datafile/undotbs2.265.570913331
+ORCL_DATA1/orcl/datafile/undotbs2.272.570921065
+ORCL_DATA1/orcl/datafile/users.264.570913355
+ORCL_DATA1/orcl/datafile/users.269.570919829
+ORCL_DATA1/orcl/onlinelog/group_1.256.570913195
+ORCL_DATA1/orcl/onlinelog/group_2.263.570913205
+ORCL_DATA1/orcl/onlinelog/group_3.266.570918279
+ORCL_DATA1/orcl/onlinelog/group_4.267.570918289
+ORCL_DATA1/orcl/tempfile/temp.258.570913303
21 rows selected.

All ASM disk that belong to the 'ORCL_DATA1' disk group

SELECT path
FROM v$asm_disk
WHERE group_number IN (select group_number
 from v$asm_diskgroup
 where name = 'ORCL_DATA1');
PATH
----------------------------------
ORCL:VOL1
ORCL:VOL2

At this point, we've installed and configured Oracle RAC 10g entirely and have a fully functional clustered database.

After all the work done up to this point, you may well ask, "OK, so how do I start and stop services?" If you have followed the instructions in this guide, all services—including Oracle Clusterware, all Oracle instances, Enterprise Manager Database Console, and so on—should start automatically on each reboot of the Linux nodes.

There are times, however, when you might want to shut down a node and manually start it back up. Or you may find that Enterprise Manager is not running and need to start it. This section provides the commands (using SRVCTL) responsible for starting and stopping the cluster environment.

Ensure that you are logged in as the oracle UNIX user. We will run all commands in this section from linux1:

# su - oracle
$ hostname
linux1

Stopping the Oracle RAC 10g Environment

The first step is to stop the Oracle instance. When the instance (and related services) is down, then bring down the ASM instance. Finally, shut down the node applications (Virtual IP, GSD, TNS Listener, and ONS).

$ export ORACLE_SID=orcl1
$ emctl stop dbconsole
$ srvctl stop instance -d orcl -i orcl1
$ srvctl stop asm -n linux1
$ srvctl stop nodeapps -n linux1
      

Starting the Oracle RAC 10g Environment

The first step is to start the node applications (Virtual IP, GSD, TNS Listener, and ONS). When the node applications are successfully started, then bring up the ASM instance. Finally, bring up the Oracle instance (and related services) and the Enterprise Manager Database console.

$ export ORACLE_SID=orcl1
$ srvctl start nodeapps -n linux1
$ srvctl start asm -n linux1
$ srvctl start instance -d orcl -i orcl1
$ emctl start dbconsole
      

Start/Stop All Instances with SRVCTL

Start/stop all the instances and their enabled services. I have included this step just for fun as a way to bring down all instances!

$ srvctl start database -d orcl
$ srvctl stop database -d orcl
      

It is not uncommon for businesses to demand 99.99% (or even 99.999%) availability for their enterprise applications. Think about what it would take to ensure a downtime of no more than .5 hours or even no downtime during the year. To answer many of these high-availability requirements, businesses are investing in mechanisms that provide for automatic failover when one participating system fails. When considering the availability of the Oracle database, Oracle RAC 10g provides a superior solution with its advanced failover mechanisms. Oracle RAC 10g includes the required components that all work within a clustered configuration responsible for providing continuous availability; when one of the participating systems fail within the cluster, the users are automatically migrated to the other available systems.

A major component of Oracle RAC 10g that is responsible for failover processing is the Transparent Application Failover (TAF) option. All database connections (and processes) that lose connections are reconnected to another node within the cluster. The failover is completely transparent to the user.

This final section provides a short demonstration on how TAF works in Oracle RAC 10g. Please note that a complete discussion of failover in Oracle RAC 10g would require an article in itself; my intention here is to present only a brief overview.

One important note is that TAF happens automatically within the OCI libraries. Thus your application (client) code does not need to change in order to take advantage of TAF. Certain configuration steps, however, will need to be done on the Oracle TNS file tnsnames.ora. (Keep in mind that as of this writing, the Java thin client will not be able to participate in TAF because it never reads tnsnames.ora.)

Setup the tnsnames.ora File

Before demonstrating TAF, we need to verify that a valid entry exists in the tnsnames.ora file on a non-RAC client machine (if you have a Windows machine lying around). Ensure that you have the Oracle RDBMS software installed. (Actually, you only need a client install of the Oracle software.)

During the creation of the clustered database in this guide, we created a new service that will be used for testing TAF named ORCLTEST. It provides all the necessary configuration parameters for load balancing and failover. You can copy the contents of this entry to the %ORACLE_HOME%\network\admin\tnsnames.ora file on the client machine (my Windows laptop is being used in this example) in order to connect to the new Oracle clustered database:

...
ORCLTEST =
 (DESCRIPTION =
 (ADDRESS = (PROTOCOL = TCP)(HOST = linux1-vip)(PORT = 1521))
 (ADDRESS = (PROTOCOL = TCP)(HOST = linux2-vip)(PORT = 1521))
 (LOAD_BALANCE = yes)
 (CONNECT_DATA =
 (SERVER = DEDICATED)
 (SERVICE_NAME = orcltest.idevelopment.info)
 (FAILOVER_MODE =
 (TYPE = SELECT)
 (METHOD = BASIC)
 (RETRIES = 180)
 (DELAY = 5)
 )
 )
 )
...

The following SQL query can be used to check a session's failover type, failover method, and if a failover has occurred. We will be using this query throughout this example.

COLUMN instance_name FORMAT a13
COLUMN host_name FORMAT a9
COLUMN failover_method FORMAT a15
COLUMN failed_over FORMAT a11

SELECT
 instance_name
 , host_name
 , NULL AS failover_type
 , NULL AS failover_method
 , NULL AS failed_over
FROM v$instance
UNION
SELECT
 NULL
 , NULL
 , failover_type
 , failover_method
 , failed_over
FROM v$session
WHERE username = 'SYSTEM';

TAF Demo

From a Windows machine (or other non-RAC client machine), login to the clustered database using the orcltest service as the SYSTEM user:

C:\> sqlplus system/manager@orcltest

COLUMN instance_name FORMAT a13
COLUMN host_name FORMAT a9
COLUMN failover_method FORMAT a15
COLUMN failed_over FORMAT a11

SELECT
 instance_name
 , host_name
 , NULL AS failover_type
 , NULL AS failover_method
 , NULL AS failed_over
FROM v$instance
UNION
SELECT
 NULL
 , NULL
 , failover_type
 , failover_method
 , failed_over
FROM v$session
WHERE username = 'SYSTEM';


INSTANCE_NAME HOST_NAME FAILOVER_TYPE FAILOVER_METHOD FAILED_OVER
------------- --------- ------------- --------------- -----------
orcl1 linux1
 SELECT BASIC NO

DO NOT log out of the above SQL*Plus session!

Now that we have run the query (above), we should now shutdown the instance orcl1 on linux1 using the abort option. To perform this operation, you can use the srvctl command-line utility as follows:

# su - oracle
$ srvctl status database -d orcl
Instance orcl1 is running on node linux1
Instance orcl2 is running on node linux2

$ srvctl stop instance -d orcl -i orcl1 -o abort

$ srvctl status database -d orcl
Instance orcl1 is not running on node linux1
Instance orcl2 is running on node linux2

COLUMN instance_name FORMAT a13
COLUMN host_name FORMAT a9
COLUMN failover_method FORMAT a15
COLUMN failed_over FORMAT a11

SELECT
 instance_name
 , host_name
 , NULL AS failover_type
 , NULL AS failover_method
 , NULL AS failed_over
FROM v$instance
UNION
SELECT
 NULL
 , NULL
 , failover_type
 , failover_method
 , failed_over
FROM v$session
WHERE username = 'SYSTEM';

INSTANCE_NAME HOST_NAME FAILOVER_TYPE FAILOVER_METHOD FAILED_OVER
------------- --------- ------------- --------------- -----------
orcl2 linux2
 SELECT BASIC YES

SQL> exit

From the above demonstration, we can see that the above session has now been failed over to instance orcl2 on linux2.

Ideally this guide has provided an economical solution to setting up and configuring an inexpensive Oracle RAC 10g Release 2 cluster using CentOS 4.2 Enterprise Linux (or RHEL4) and FireWire technology. The RAC solution presented here can be put together for around US$1,800 and will provide the DBA with a fully functional development Oracle RAC cluster.

Remember, although this solution should be stable enough for testing and development, it should never be considered for a production environment.

An article of this magnitude and complexity is generally not the work of one person alone. Although I was able to author and successfully demonstrate the validity of the components that make up this configuration, there are several other individuals that deserve credit in making this article a success.

First, I would like to thank Werner Puschitz for his outstanding work on "Installing Oracle Database 10g with Real Application Cluster (RAC) on Red Hat Enterprise Linux Advanced Server 3". This article, along with several others of his, provided information on Oracle RAC10g that could not be found in any other Oracle documentation. Without his hard work and research into issues like configuring and installing the hangcheck-timer kernel module, properly configuring UNIX shared memory, and configuring ASMLib, this article may have never come to fruition. If you are interested in examining technical articles on Linux internals and in-depth Oracle configurations written by Werner Puschitz, please visit his excellent website at www.puschitz.com.

I would next like to thank Wim Coekaerts, Joel Becker, Manish Singh and the entire team at Oracle's Linux Projects Development Group. The professionals in this group made the job of upgrading the Linux kernel to support IEEE1394 devices with multiple logins (and several other significant modifications) a seamless task. The group provides the pre-compiled FireWire modules for Red Hat Enterprise Linux 4.2 (which also works with CentOS Linux) along with many other useful tools and documentation at oss.oracle.com.

Send us your comments

Page 1  Page 2  Page 3