| DBA:
Linux
Upgrade Guide: Oracle9i Release 2 RAC to Oracle RAC 10g Release 2 on Linux
by Murali Vallath
Take advantage of the extensive list of new features and functionality in Oracle Database 10g and Oracle RAC 10g configured on Oracle Automatic Storage Management.
Published February 2006
If you've been waiting to take advantage of the extensive list of new features and functionality in Oracle Database 10g and Oracle Real Application Clusters (RAC) 10g, there's no longer any need for delay. In September 2005, Oracle Database 10g Release 2 and Oracle RAC 10g Release 2 became available on all major platforms.
In this guide, I will describe the various steps involved in upgrading an Oracle9i RAC Release 2 environment configured on Oracle Cluster File System (OCFS) to Oracle RAC 10g Release 2 configured on Oracle Automatic Storage Management (ASM)—which is best practice for shared storage on RAC systems. Part 1 will cover the basic upgrade process; Part 2 will cover OCFS-to-ASM migration.
Please note that this guide is intended for evaluation purposes only; it is not an amendment of, or replacement for, official Oracle documentation.
Part 1: Upgrade Process
The environment selected for the upgrade in this scenario is a four-node Red Hat Linux AS 2.1 cluster :
| Database Name: |
SSKYDB |
| Number of Nodes: |
Four: oradb1, oradb2, oradb3 and oradb4 |
| Database Version: |
9.2.0.4 |
| Number of Instances: |
Four: SSKY1, SSKY2, SSKY3, and SSKY4 |
| O/S Kernel Version: |
Red Hat Advanced Server 2.1
Kernel - 2.4.9-e.41smp |
| Clustered File System: |
OCFS 1.0 |
| Cluster Manager: |
Oracle Cluster Manager |
The upgrade process can be broken down into the following steps:
- Verify system requirements and path levels
- Configure network components
- Install Oracle Clusterware
- Install Oracle software
- Upgrade Your Database
- Housekeeping
Verify System Requirements and Patch Levels
The first step in any software installation is to verify the system requirements and certification matrix for the operating system and the hardware platform; in the case of an upgrade, you will also need to verify all application patch levels.
For example, one primary requirement for Oracle Database 10g Release 2 is that it requires Red Hat Enterprise Linux (RHEL) 3 update 4 or RHEL 4. Apart from the basic operating system version, verify the following additional packages. But first, ensure a valid complete backup of the system is available before making any changes to the existing configuration—including OS and Oracle data files.
First, upgrade your OS from Red Hat AS 2.1 to RHEL3 update 4 (from Kernel version 2.4.9-e.41smp to 2.4.21-27). You will need to install and verify the required additional O/S packages.
[root@oradb3 oracle]# rpm -qa | grep -i gcc
compat-gcc-c++-7.3-2.96.128
compat-gcc-7.3-2.96.128
libgcc-3.2.3-42
gcc-3.2.3-42
[root@oradb3 oracle]# rpm -qa | grep -i openmotif
openmotif-2.2.3-3.RHEL3
openmotif21-2.1.30-8
[root@oradb3 oracle]# rpm -qa | grep -i glibc
glibc-2.3.3-74
glibc-utils-2.3.3-74
glibc-kernheaders-2.4-8.34.1
glibc-common-2.3.3-74
glibc-headers-2.3.3-74
glibc-devel-2.3.3-74
[root@oradb3 oracle]# rpm -qa | grep -i compat
compat-libstdc++-7.3-2.96.128
compat-gcc-c++-7.3-2.96.128
compat-gcc-7.3-2.96.128
compat-db-4.0.14-5
compat-libstdc++-devel-7.3-2.96.128
[root@oradb3 oracle]#
Next, update the kernel parameters with the following values.
kernel.core_uses_pid = 1
kernel.hostname = oradb3.sumsky.net
kernel.domainname = sumsky.net
kernel.shmall = 2097152
#kernel.shmmax = 536870912
kernel.shmmax = 2147483648
kernel.shmmni = 4096
kernel.shmseg = 4096
kernel.sem = 250 32000 100 150
kernel.msgmnl = 2878
kernel.msgmnb = 65535
fs.file-max = 65536
net.ipv4.ip_local_port_range = 1024 65000
net.core.rmem_default = 262144
net.core.wmem_default = 262144
net.core.rmem_max = 262144
net.core.wmem_max = 262144
Add the following parameters to /etc/security/limits.conf.
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
Next, you'll need to establish user equivalence with SSH. Depending on the node from which the Oracle installation will be performed using the Oracle universal installer (OUI), Oracle copies files from the node where the installation is performed to all the other remaining nodes in the cluster. Such a copy process is performed either by using ssh protocol where available or by using the remote copy (rcp). In order for the copy operation to be successful, the oracle user on all the RAC nodes must be able to login to other RAC nodes without having to provide a password or passphrase.
For security reasons, most organizations prefer ssh (secured shell)-based operations over rcp. Take the following steps to configure the oracle account to use ssh logins without using any passwords:
- Create the authentication key for user oracle. To create this key, change the current directory to the default login directory of the oracle user and perform the following operation:
[oracle@oradb4 oracle]$ ssh-keygen -t dsa -b 1024
Generating public/private dsa key pair.
Enter file in which to save the key (/home/oracle/.ssh/id_dsa):
Created directory '/home/oracle/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/oracle/.ssh/id_dsa.
Your public key has been saved in /home/oracle/.ssh/id_dsa.pub.
The key fingerprint is:
b6:07:42:ae:47:56:0a:a3:a5:bf:75:3e:21:85:8d:30 oracle@oradb4.sumsky.net
[oracle@oradb4 oracle]$
Perform this step on all nodes in the cluster. Keys generated from each of the nodes in the cluster should be appended to the authorized_keys file on all nodes—meaning each node should contain the keys from all other nodes in the cluster.
[oracle@oradb4 oracle]$ cd .ssh
[oracle@oradb3 .ssh]$ cat id_dsa.pub > authorized_keys
- After you've created the keys and copied them to all nodes, oracle user accounts can connect from one node to another oracle account on another node without using a password. This approach allows the OUI to copy files from the installing node to the other nodes in the cluster. The output below is the verification showing the ssh command from node oradb3 to other nodes in the cluster worked.
[oracle@oradb3 oracle]$ ssh oradb3 hostname
oradb3.sumsky.net
[oracle@oradb3 oracle]$ ssh oradb4 hostname
oradb4.sumsky.net
[oracle@oradb3 oracle]$ ssh oradb3-priv hostname
oradb3.sumsky.net
[oracle@oradb3 oracle]$ ssh oradb4-priv hostname
oradb4.sumsky.net
Note: When performing these tests for the first time, the operating system will display a key and prompt the user to accept or decline. Enter "Yes" to accept and register the key. Tests should be performed on all other nodes across all interfaces (with the exception of the VIP) in the cluster.
Configure Network Components
In addition to the pre-configured public and private network, Oracle Database 10g requires additional IP addresses that will be mapped to the public address as virtual IPs (VIPs). You should add the VIP to the /etc/hosts file on all nodes in the cluster as well as all nodes accessing the database. Oracle recommends that the VIP be added to DNS.
root@oradb3 oracle]# more /etc/hosts
127.0.0.1 localhost.localdomain localhost
192.168.2.10 oradb1.sumsky.net oradb1
192.168.2.20 oradb2.sumsky.net oradb2
192.168.2.30 oradb3.sumsky.net oradb3
192.168.2.40 oradb4.sumsky.net oradb4
#Private Network/interconnect
10.168.2.110 oradb1-priv.sumsky.net oradb1-priv
10.168.2.120 oradb2-priv.sumsky.net oradb2-priv
10.168.2.130 oradb3-priv.sumsky.net oradb3-priv
10.168.2.140 oradb4-priv.sumsky.net oradb4-priv
# VIP
192.168.2.15 oradb1-vip.sumsky.net oradb1-vip
192.168.2.25 oradb2-vip.sumsky.net oradb2-vip
192.168.2.35 oradb3-vip.sumsky.net oradb3-vip
192.168.2.45 oradb4-vip.sumsky.net oradb4-vip
Install Oracle Clusterware
Oracle Database 10g introduced a new version of the cluster manager software called Oracle Clusterware (initially called Cluster Ready Services in Oracle Database 10g Release 1). Oracle Clusterware differs from the Oracle Cluster Manager that exists in the Oracle9i environment configuration; it will have to be installed in a separate home.
Oracle Clusterware creates two files, Oracle Cluster Repository (OCR) and the Oracle Cluster Synchronization Service (CSS) voting disk. The Oracle9i server manager file identified in /var/opt/oracle/srvConfig.loc that exists on the shared storage will become the new OCR file required in Oracle Database 10g. The voting disk (similar to the quorum disk in Oracle9i) requires three isolated locations on the shared storage and should each be 20MB in size. Create three identical locations of 20MB on the shared storage.
Note: In Oracle Database 10g Release 2, Oracle also provides options to create and maintain redundant OCR files. Creation of redundant OCR files is covered in the "Housekeeping" section below.
Three locations are on mount point /u02, /u03, and /u04. As they will be stored on Oracle Cluster File System (OCFS), the devices need to be formatted and mounted on all nodes in the cluster.
[root@oradb3 root]# mkfs.ocfs -F -b 128 -L ora02 -m /u02 -u 200 -g 200 -p 0775 /dev/sdb6
Cleared volume header sectors
Cleared node config sectors
Cleared publish sectors
Cleared vote sectors
Cleared bitmap sectors
Cleared data block
Wrote volume header
Similarly, format the other two partitions using OCFS.
[root@oradb3 root]# mkfs.ocfs -F -b 128 -L ora03 -m /u03 -u 200 -g 200 -p 0775 /dev/sdb7
[root@oradb3 root]# mkfs.ocfs -F -b 128 -L ora04 -m /u04 -u 200 -g 200 -p 0775 /dev/sdb8
Now mount these devices on all nodes using the following command.
mount –t ocfs /dev/sdb6 /u02
mount –t ocfs /dev/sdb7 /u03
mount –t ocfs /dev/sdb8 /u04
To allow the operating system to mount these devices every time the node is restarted, add the following devices to the /etc/fstab file.
[oracle@oradb3 oracle]$ more /etc/fstab
LABEL=/ / ext3 defaults 1 1
none /dev/pts devpts gid=5,mode=620 0 0
none /proc proc defaults 0 0
none /dev/shm tmpfs defaults 0 0
/dev/sda2 swap swap defaults 0 0
/dev/cdrom /mnt/cdrom udf,iso9660 noauto,owner,kudzu,r
o 0 0
/dev/fd0 /mnt/floppy auto noauto,owner,kudzu 0 0
/dev/sdb5 /u01 ocfs _netdev 0 0
/dev/sdb6 /u02 ocfs _netdev 0 0
/dev/sdb7 /u03 ocfs _netdev 0 0
/dev/sdb8 /u04 ocfs _netdev 0 0
/dev/sdb9 /u05 ocfs _netdev 0 0
/dev/sdb10 /u06 ocfs _netdev 0 0
/dev/sdb14 /u14 ocfs _netdev 0 0
Cluster Verification Utility. Oracle has introduced a new utility called Oracle Cluster Verification Utility (CVU). CVU is part of the clusterware software and can be located on the DVD media under the "Clusterware" directory. Via various parameters, you can use this utility to determine the status of the cluster during the various stages of installation and configuration as well as to verify individual componenets of the cluster.
The OUI automatically runs CVU at the end of the installation of Oracle Clusterware to verify its status. At this stage before beginning installation of the Oracle Clusterware, the following two verifications should be performed. (Note: Read the instructions in the clusterware/cluvfy directory on the DVD about how to install the CVU before installing Oracle Clusterware.)
If the hardware and operating system configuration is complete:
[oracle@oradb3 bin]$ cluvfy stage -post hwos -n oradb3,oradb4
Performing post-checks for hardware and operating system setup
Checking node reachability...
Node reachability check passed from node "oradb3".
Checking user equivalence...
User equivalence check passed for user "oracle".
Checking node connectivity...
Node connectivity check passed for subnet "192.168.2.0" with node(s) oradb4,oradb3.
Node connectivity check passed for subnet "10.168.2.0" with node(s) oradb4,oradb3.
Suitable interfaces for the private interconnect on subnet "192.168.2.0":
Suitable interfaces for the private interconnect on subnet "192.168.2.0":
oradb4 eth0:192.168.2.40 eth0:192.168.2.45
oradb3 eth0:192.168.2.30 eth0:192.168.2.35
Suitable interfaces for the private interconnect on subnet "10.168.2.0":
oradb4 eth1:10.168.2.140
oradb3 eth1:10.168.2.130
Checking shared storage accessibility...
Shared storage check failed on nodes "oradb4,oradb3".
Post-check for hardware and operating system setup was unsuccessful on all the nodes.
Perform appropriate checks on all the nodes in the node list before setting up OCFS.
[oracle@oradb3 oracle]$ cluvfy stage -pre cfs -n oradb3,oradb4 -s /dev/sdb
Perform appropriate checks on all the nodes in the node list before setting up Oracle Clusterware.
[oracle@oradb3 cluvfy]$ cluvfy stage -pre crsinst -n oradb3,oradb4
Performing pre-checks for cluster services setup
Checking node reachability...
Node reachability check passed from node "oradb3".
Checking user equivalence...
User equivalence check passed for user "oracle".
Checking administrative privileges...
User existence check passed for "oracle".
Group existence check passed for "oinstall".
Membership check for user "oracle" in group "oinstall" [as Primary] failed.
Check failed on nodes:
oradb4,oradb3
Administrative privileges check passed.
Checking node connectivity...
Node connectivity check passed for subnet "192.168.2.0" with node(s) oradb4,oradb3.
Node connectivity check passed for subnet "10.168.2.0" with node(s) oradb4,oradb3.
Suitable interfaces for the private interconnect on subnet "192.168.2.0":
oradb4 eth0:192.168.2.40 eth0:192.168.2.45
oradb3 eth0:192.168.2.30 eth0:192.168.2.35
Suitable interfaces for the private interconnect on subnet "10.168.2.0":
oradb4 eth1:10.168.2.140
oradb3 eth1:10.168.2.130
Checking system requirements for 'crs'...
Total memory check passed.
Check failed on nodes:
oradb4,oradb3
Free disk space check passed.
Swap space check passed.
System architecture check passed.
Kernel version check passed.
Package existence check passed for "make-3.79".
Package existence check passed for "binutils-2.14".
Package existence check passed for "gcc-3.2".
Package existence check passed for "glibc-2.3.2-95.27".
Package existence check passed for "compat-db-4.0.14-5".
Package existence check passed for "compat-gcc-7.3-2.96.128".
Package existence check passed for "compat-gcc-c++-7.3-2.96.128".
Package existence check passed for "compat-libstdc++-7.3-2.96.128".
Package existence check passed for "compat-libstdc++-devel-7.3-2.96.128".
Package existence check passed for "openmotif-2.2.3".
Package existence check passed for "setarch-1.3-1".
Group existence check passed for "dba".
Group existence check passed for "oinstall".
User existence check passed for "nobody".
System requirement failed for 'crs'
Pre-check for cluster services setup was successful on all the nodes.
Install Oracle Clusterware
Before installing the clusterware, shut down the Oracle database and the Oracle9i listener. Ensure that the Oracle9i Cluster Manager (oracm) and GSD are currently up and running on all nodes in the cluster.
Running the OUI requires the use of an X-windows-compatible terminal. Otherwise, an appropriate X-windows emulator should be installed and the emulator invoked using the DISPLAY command using the following syntax.
export DISPLAY=<client IP address>:0.0
For example:
[oracle@oradb3 oracle]$export DISPLAY=192.168.2.101:0.0
You can run the OUI directly or you can execute the installer after copying the contents from the DVD on to the hard disk. If executing from DVD, change the directory to cd /media/dvd/clusterware. Execute the runInstaller file and note the following prompts.
- Unset ORACLE_HOME.
- Welcome; click on the Next button.
- Specify the CRS file locations, following the OFA standards and providing a separate home for CRS an appropriate file location would /usr/app/oracle/product/10.2.0/crs.
Name: OraCr10g_home1
Path: /usr/app/oracle/product/10.2.0/crs
Note: CRS needs to be installed in a separate home isolated from the Oracle home directory to allow upgrades of either product independently. (For example, if the Oracle RDBMS needs to be patched, the binaries of the Clusterware will not be affected. ) Refer to the environment variable definitions below.
- Specify Cluster Configuration:
Cluster Name: SskyClst (Default 'crs')
Cluster Nodes:
| Public Node Name |
Private Node Name |
Virtual Host Name |
| oradb1 |
oradb1-priv |
oradb1-vip |
| oradb2 |
oradb2-priv |
oradb2-vip |
| oradb3 |
oradb3-priv |
oradb3-vip |
| oradb4 |
oradb4-priv |
oradb4-vip |
- Specify Network Interface Usage:
| Interface Name |
Subnet |
Interface Type |
| eth0 |
192.168.2.0 |
Public |
| eth1 |
10.168.2.0 |
Private |
- Specify Voting Disk Location:
Select "Normal Redundancy." (If Oracle Clusterware is to maintain redundant copies of voting disk or if external mirroring is available, select "External Redundancy.")
| Voting Disk Location
|
/u02/oradata/CSSVoteDisk.dbf |
| Additional Voting Disk 1 Location |
/u03/oradata/CSSVoteDisk.dbf |
| Additional Voting Disk 2 Location |
/u04/oradata/CSSVoteDisk.dbf |
- Summary—Click on Install.
- Once installation is complete, OUI prompts to execute /usr/app/oracle/product/10.2.0/crs/root.sh script on all nodes in the cluster starting with the node where OUI is running. Root.sh file will create the CSS vote disk with the cluster information and start the required cluster service daemon processes. When the script is executed on the last node, apart from the cluster service daemon processes, the script will also configure the VIP and other cluster application services.
[root@oradb4 crs]# ./root.sh
WARNING: directory '/usr/app/oracle/product/10.2.0' is not owned by root
WARNING: directory '/usr/app/oracle/product' is not owned by root
WARNING: directory '/usr/app/oracle' is not owned by root
Checking to see if Oracle CRS stack is already configured
/etc/oracle does not exist. Creating it now.
Setting the permissions on OCR backup directory
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/usr/app/oracle/product/10.2.0' is not owned by root
WARNING: directory '/usr/app/oracle/product' is not owned by root
WARNING: directory '/usr/app/oracle' is not owned by root
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
assigning default hostname oradb3 for node 1.
assigning default hostname oradb2 for node 2.
assigning default hostname oradb1 for node 3.
assigning default hostname oradb4 for node 4.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 1: oradb3 oradb3-priv oradb3
node 2: oradb2 oradb2-priv oradb2
node 3: oradb1 oradb1-priv oradb1
node 4: oradb4 oradb4-priv oradb4
clscfg: Arguments check out successfully.
NO KEYS WERE WRITTEN. Supply -force parameter to override.
-force is destructive and will destroy any previous cluster
configuration.
Oracle Cluster Registry for cluster has already been initialized
Startup will be queued to init within 30+60 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
CSS is active on these nodes.
oradb3
oradb2
oradb1
oradb4
CSS is active on all nodes.
Waiting for the Oracle CRSD and EVMD to start
Waiting for the Oracle CRSD and EVMD to start
Waiting for the Oracle CRSD and EVMD to start
Waiting for the Oracle CRSD and EVMD to start
Oracle CRS stack installed and running under init(1M)
Running vipca(silent) for configuring nodeapps
Creating VIP application resource on (4) nodes...
Creating GSD application resource on (4) nodes...
Creating ONS application resource on (4) nodes...
Starting VIP application resource on (4) nodes...
Starting GSD application resource on (4) nodes...
Starting ONS application resource on (4) nodes...
Done
On completion of the Oracle Clusterware installation, the following files are created in their respective directories.
- Clusterware files
[root@oradb3 root]# ls -ltr /etc/init.d/init.*
-r-xr-xr-x 1 root root 3197 Aug 13 23:32 /etc/init.d/init.evmd
-r-xr-xr-x 1 root root 35401 Aug 13 23:32 /etc/init.d/init.cssd
-r-xr-xr-x 1 root root 4721 Aug 13 23:32 /etc/init.d/init.crsd
-r-xr-xr-x 1 root root 1951 Aug 13 23:32 /etc/init.d/init.crs
[root@oradb3 root]#
- The O/S provided inittab file is updated with the following entries.
[root@oradb3 root]# tail -5 /etc/inittab
# Run xdm in runlevel 5
x:5:respawn:/etc/X11/prefdm -nodaemon
h1:35:respawn:/etc/init.d/init.evmd run >/dev/null 2>&1 </dev/null
h2:35:respawn:/etc/init.d/init.cssd fatal >/dev/null 2>&1 </dev/null
h3:35:respawn:/etc/init.d/init.crsd run >/dev/null 2>&1 </dev/null
- The server configuration file locater present in /var/opt/oracle directory is nulled with the following value.
[root@oradb3 root]# more /var/opt/oracle/srvConfig.loc
srvconfig_loc=/dev/null
- Installer updates a new locater file, identified by ocr.loc present in the /etc/oracle directory with the repository location information.
[root@oradb3 root]# more /etc/oracle/ocr.loc
ocrconfig_loc=/u01/oradata/SrvConfig.dbf
local_only=FALSE
- Click on OK after the root.sh script has run on all nodes.
- Ensure completion all configuration assistants. (Oracle Notification Server Configuration Assistant, Oracle Private Interconnect Configuration Assistant, and Oracle Cluster Verification Utility).
- End of Installation—Click on Exit.
- Verify if the Clusterware has all the nodes registered using the olsnodes command.
[oracle@oradb1 oracle]$ olsnodes
oradb1
oradb2
oradb3
oradb4
[oracle@oradb1 oracle]$
- Verify if the cluster services is started, using the crs_stat command.
[oracle@oradb1 oracle]$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.oradb1.gsd application ONLINE ONLINE oradb1
ora.oradb1.ons application ONLINE ONLINE oradb1
ora.oradb1.vip application ONLINE ONLINE oradb1
ora.oradb2.gsd application ONLINE ONLINE oradb2
ora.oradb2.ons application ONLINE ONLINE oradb2
ora.oradb2.vip application ONLINE ONLINE oradb2
ora.oradb3.gsd application ONLINE ONLINE oradb3
ora.oradb3.ons application ONLINE ONLINE oradb3
ora.oradb3.vip application ONLINE ONLINE oradb3
ora.oradb4.gsd application ONLINE ONLINE oradb4
ora.oradb4.ons application ONLINE ONLINE oradb4
ora.oradb4.vip application ONLINE ONLINE oradb4
- Verify if the VIP services are configured at the OS level using the ifconfig command.
[oracle@oradb3 oracle]$ ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:D0:B7:6A:39:85
inet addr:192.168.2.30 Bcast:192.168.2.255 Mask:255.255.255.0
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:123 errors:0 dropped:0 overruns:0 frame:0
TX packets:67 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:11935 (11.6 Kb) TX bytes:5051 (4.9 Kb)
eth0:1 Link encap:Ethernet HWaddr 00:D0:B7:6A:39:85
inet addr:192.168.2.35 Bcast:192.168.3.255 Mask:255.255.252.0
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:14631 errors:0 dropped:0 overruns:0 frame:0
TX packets:21377 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:6950046 (6.6 Mb) TX bytes:19706526 (18.7 Mb)
Note:Tthe ":1" in "eth0:1" indicates the VIP address for the basic host eth0. When the node fails, eth0:1 will be moved to one of the surviving nodes in the cluster. The new identifier for the VIP on the failed-over server will be indicated by eth0:2 or higher, depending on to what other nodes the VIP has migrated.
Install Oracle Software
- Welcome; click on Next on this screen.
- Select installation type: Select “Enterprise Edition.”
- Specify Home Details:
Name: OraDb10g_home1
Path: /usr/app/oracle/product/10.2.0/db_1
- Specify Hardware Cluster Installation Mode:
Select “Cluster Installation”
Click on Select All
- Product-Specific Prerequisite Checks: Verify that all checks are successful before proceeding.
- Physical memory requirements (expected 922MB)
- Checking swap space (824MB expected)
- Checking network configuration requirements
- Oracle supports installation on systems with DHCP-assigned public IP. However the primary network interface should be configured with a static IP address.
- Validating ORACLE_BASE location
- Checking ORACLE_HOME path for space
- Checking for system clean-up
- Checking for Oracle home incompatibilities
- Checking Oracle Clusterware version
- Select Configuration Option; select “Install Database Software only.”
- Summary—Click on Install.
- When installation is complete, OUI will prompt you to execute /usr/app/oracle/product/10.2.0/db_1/root.sh script from another window as the root user on all nodes in the cluster.
- Click on OK after the root.sh has run on all nodes.
- End of Installation—Click on Exit.
Upgrade the Database
Oracle provides three methods for upgrading from Oracle 9i Release 2 to Oracle Database 10g Release 2: Using the database upgrade assistant (DBUA) through the OUI, using the database upgrade assistant (DBUA) in silent mode, or manually using Oracle provided upgrade scripts. The first option, using the DBUA through the OUI, is the most convenient and a more completely automated method for the upgrade. DBUA when invoked will perform all the tasks required from conversion to migration to adding all additional features present in Oracle Database 10g Release 2.
Requirements:
- Make a full cold backup of the database before commencing the upgrade process.
- The /etc/oratab file should have the Oracle9i database listed.
- oracm should be up and running on all nodes.
- Oracle9
- i
Listener should be running on all nodes and the database services should be registered.
- Oracle9
- i
Database and all instances should be up and running on all respective nodes in the cluster.
Procedure:
- Welcome; read instructions and click on Next.
- Step 1 of 9: Databases—A list of all available databases (“SSKYDB”) on the node is displayed. Select the database to upgrade, enter the password for username (sys), and click on “Next.”
- Step 2 of 9: SYSAUX Tablespace—Oracle Database 10g requires the SYSAUX tablespace to be created. If the definition and characteristics of the tablespace are satisfactory, click on “Next.”
- Step 3 of 9: Recompile Invalid Objects—Select “Recompile invalid objects at the end of upgrade”, click on “Next.”
- Step 4 of 9: Server Parameter File—Select “Use Server Parameter File (SPFILE),” click “Next.”
- Step 5 of 9: Backup—Select “I have already backed up my database”; if a valid backup of the database is available otherwise, select “I would like this tool to back up the database” and specify the backup directory. Click on “Next.”
- Step 6 of 9: Management Options—Select the appropriate management option.
- Step 7 of 9: Database Credentials—Select “Use Different Passwords” and enter the passwords for DBSNMP and SYSMAN users.
- Step 8 of 9: Database Services—Add all the desired services and its details.
- Step 9 of 9: Summary—After verifying the “Database Upgrade Summary”, click on “Finish” to begin the upgrade operation.
- Progress—DBUA performs the various steps to complete the upgrade process.
- Summary—Displays the upgrade summary. Click on “Exit”.
At this stage the following is the status of the upgrade:
- The database has been upgraded from Oracle9i Release 2 to Oracle Database 10g Release 2.
- The datafiles reside on OCFS 1.0.
- Like all other datafiles the spfile resides on a shared filesystem.
- A new OEM db console has been configured on port 1158.
- Verify if the database upgrade and all required components have been upgraded.
Verify if all the required modules have been upgrade using the following Oracle-provided - script:
SQL> @$ORACLE_HOME/rdbms/admin/utlu102s.sql
- Verify if all instances in the cluster are started.
- The DBUA has configured the OEM db console at the following address location:
http://oradb3.sumsky.com:1158/em
- Verify if the db console is up and running using the enterprise manager control utility (emctl):
[oracle@oradb3 oracle]$ emctl status dbconsole
TZ set to US/Eastern
Oracle Enterprise Manager 10g Database Control Release 10.2.0.1.0
Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
http://oradb3.sumsky.com:1158/em/console/aboutApplication
Oracle Enterprise Manager 10g is running.
------------------------------------------------------------------
Logs are generated in directory /usr/app/oracle/product/10.2.0/db_1/oradb3_SSKY1/sysman/log
[oracle@oradb3 oracle]$
Housekeeping
At this point you should perform the following housekeeping tasks:
- Setup redundancy for OCR files. The importance of OCR has increased dramatically in Oracle Database 10g. OCR is used by the Oracle Clusterware to determine what services and applications need to be started during startup of the instance, what services have been configured for HA solution and have been defined for failover when the current holding instance fails, and so on.
With the information contained in the OCR, Oracle Database 10g Release 2 supports duplexing of the OCR file. While the DBCA prompts the DBA with the OCR duplexing screen during a fresh install of Oracle Database 10g Release 2, it does not provide the option during an upgrade. The upgrade simply converts the OCR repository form Oracle9i to Oracle 10g, but does not duplex.
If external hardware mirroring is not available for the disk containing the OCR file, a redundant copy of the OCR file will have to be performed manually using the Oracle provided OCR configuration utility (ocrconfig).
Syntax:
ocrconfig –replace ocrmirror <new location>
[root@oradb4 root]# ocrconfig -replace ocr /u03/oradata/SrvConfig.dbf
Note: The OCR configuration utility generates a log file for all operations in the directory $ORACLE_HOME/log/<hostname>/client/ocrconfig_<pid>.log. - Back up the OCR files. During the Clusterware installation, an automatic backup feature is also configured to backup the OCR files from the Clusterware master node that has been up continuously for duration of four hours. The backup operation and file destinations could be verified using the OCR configuration utility.
[oracle@oradb3 oracle]$ ocrconfig -showbackup
oradb4 2005/08/18 18:14:59 /usr/app/oracle/product/10.2.0/crs/cdata/SskyClst
oradb4 2005/08/18 14:14:59 /usr/app/oracle/product/10.2.0/crs/cdata/SskyClst
oradb4 2005/08/18 10:14:58 /usr/app/oracle/product/10.2.0/crs/cdata/SskyClst
oradb4 2005/08/18 02:14:57 /usr/app/oracle/product/10.2.0/crs/cdata/SskyClst
The above command indicates that the backup for the OCR file has been running from node oradb4 on its local storage; however, it does not indicate the actual backups. The backup files could be listed using the following:
[oracle@oradb4 oracle]$ cd /usr/app/oracle/product/10.2.0/crs/cdata/SskyClst/
[oracle@oradb4 SskyClst]$ ls -ltr
total 19620
-rw-r--r-- 1 root root 4014080 Aug 18 02:14 week.ocr
-rw-r--r-- 1 root root 4014080 Aug 18 02:14 day.ocr
-rw-r--r-- 1 root root 4014080 Aug 18 10:14 backup02.ocr
-rw-r--r-- 1 root root 4014080 Aug 18 14:14 backup01.ocr
-rw-r--r-- 1 root root 4014080 Aug 18 18:14 backup00.ocr
[oracle@oradb4 SskyClst]$
The above output indicates three outputs—backup00.ocr, backup01.ocr, and backup02.ocr—that are four hours apart. Oracle also maintains one backup file for the day (day.ocr) and another for the week (week.ocr).
Storing OCR backups on the local storage of any single node limits access to the file when that node fails, thereby creating a single point of failure. For example, if node oradb4 has failed, the OCR backup files cannot be accessed from other files in the cluster. Thus, the backup destination has to be moved to a shared storage providing access to all nodes in the cluster using the backuploc parameter.
Syntax:
ocrconfig –backuploc < new location of backup>
The following command will change the OCR backup destination to shared storage located at /u13/oradata/Sskyclst.
[root@oradb4 root]# ocrconfig -backuploc /u13/oradata/Sskyclst
- Set up environment variables. For easy administration and navigation between the various disk locations, you should define several different environment variables in the login profile. For example:
[oracle@oradb3 oracle]$ more .bash_profile
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
export ORACLE_BASE=/usr/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/10.2.0/db_1
export ORA_CRS_HOME=$ORACLE_BASE/product/10.2.0/crs
export AGENT_HOME=/usr/app/oracle/product/10.2.0/EMAgent
export PATH=.:${PATH}:$HOME/bin:$ORACLE_HOME/bin
export PATH=${PATH}:$ORA_CRS_HOME/bin
export PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin:/sbin
export ORACLE_ADMIN=$ORACLE_BASE/admin
export TNS_ADMIN=$ORACLE_HOME/network/admin
export LD_ASSUME_KERNEL=2.4.19
export LD_LIBRARY=$ORACLE_HOME/lib
export LD_LIBRARY=${LD_LIBRARY}:/lib:/usr/lib:/usr/local/bin
export LD_LIBRARY=${LD_LIBRARY}:$ORA_CRS_HOME/lib
export CLASSPATH=$ORACLE_HOME/JRE
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib
export THREADS_FLAG=native
export ORACLE_SID=SSKY1
- Remove Oracle9i Release 2 home. If this is the only database that was present in the cluster and there are no more Oracle9i Release 2 databases, you can uninstall the Oracle9i Release 2 home and other related components and remove from the servers.
You have now migrated an Oracle9i Release 2 database to one running on Oracle Database 10g Release 2. Next, you'll learn how to migrate the database from OCFS to Oracle ASM.
Part 2: Migrating from OCFS to ASM
OCFS version 1.0 was introduced in Oracle9i Database to make management of database files in a clustered environment such as RAC easier on environments such as Linux and Windows. Previously, management of files dependent on creating raw partitions to enable sharing of files between multiple instances in the cluster. OCFS version 1.0, although written for clustered environment, had several limitations. For example, it did not permit storing of non-Oracle files and nor follow any true standards.
OCFS version 2.0 (OCFS2), which is released under the GPL, removes these barriers present in its predecessor by allowing non-Oracle files to be stored, supporting the use of a single Oracle home for the entire cluster. While several of the limitations that existed in version 1.0 have been removed, OCFS2 continues to carry limitations present in any traditional volume management software—functions such as adding or removing disks from an existing volume are quite difficult. Such limitations, however, are totally eliminated when using Oracle ASM.
Oracle ASM, introduced in Oracle Database 10g Release 1, alleviates the need for clustered file systems and third-party volume managers to store Oracle database files. ASM is also limited, however, to storing Oracle database files, and isn't suitable for storing Oracle Clusterware specific files or a shared Oracle home. (A shared Oracle home is supported with OCFS2.)
Aside from being built into the Oracle kernel, this new technology provides great benefits to a DBA in the areas of volume management, dynamic re-balancing of file extents, and performance benefits. The full description of Oracle ASM is beyond our scope here, but technical information is available at OTN's Oracle ASM Product Center.
Disclaimers: Please note that you are advised to read the ASM Overview and Technical Best Practices White Paper before proceeding with physical migration. Also note that this guide is intended for evaluation purposes only; it is not an amendment of, or replacement for, official Oracle documentation.
The upgrade process can be broken down into the following steps:
- Verify system requirements and install ASM
- Create ASM instances
- Migrate datafiles to ASM
Verify System Requirements and Install ASM
Disks for ASM can be setup as pure raw disks, raw logical unit (LUN), or using Oracle provided library files. In this section, you will learn how to set up disks for ASM using the library files.
- Download and install the following (optional) ASM library packages for your Linux distribution from OTN.
- Install the ASM library packages. Connect to the node as the root user to install the packages as follows:
# su root
# rpm -Uvh oracleasm-support-2.0.1-1.i386.rpm \
oracleasmlib-2.0.1-1.i386.rpm\
oracleasm-2.4.21-27.EL-1.0.4-2.i686.rpm
Preparing... ########################################## [100%]
The above output shows the installation of the ASM library packages on Red Hat Enterprise Linux (RHEL) 3 update 4 or RHEL 4. Install these packages on all nodes in the cluster.
- Set up disks using ASM Library (ASMlib). The installation of the ASM library packages places a utility in the /etc/init.d/ directory called oracleasm. This utility is used to configure and initialize the various disks for ASM. The next step is to configure ASM.
# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.
This will configure the on-boot properties of the Oracle ASM library driver. The following questions will determine whether the driver is loaded on boot and what permissions it will have. The current values will be shown in brackets ('[]'). Hitting without typing an answer will keep that current value. Ctrl-C will abort.
Default user to own the driver interface []: oracle
Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]: y
Fix permissions of Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration [ OK ]
Creating /dev/oracleasm mount point [ OK ]
Loading module "oracleasm" [ OK ]
Mounting ASMlib driver filesystem [ OK ]
Scanning system for ASM disks [ OK ]
ASM should be configured similarly on all nodes participating in the cluster.
Once the ASMlib configuration is complete, the ASMlib modules for Linux are loaded and scripts are configured to load them again at system boot time. The loading of the modules after system reboot may be verified using the lsmod command:
# lsmod
Module Size Used by Tainted: GF
parport_pc 18724 2 (autoclean)
lp 8932 0 (autoclean)
parport 36800 2 (autoclean) [parport_pc lp]
autofs 13204 0 (autoclean) (unused)
oracleasm 14224 1
ocfs 297856 7
audit 89592 1
- Create ASM disk volumes. This step is performed to initialize the disk volumes using the ASM library. Performing this step will make the disks available to ASM to configure into disk groups. This step is also performed using the oracleasm script:
/etc/init.d/oracleasm createdisk <volume name> <physical disk name>
# /etc/init.d/oracleasm createdisk AVOL11 /dev/sdg
Creating Oracle ASM disk "AVOL11" [ OK ]
# /etc/init.d/oracleasm createdisk AVOL12 /dev/sdg
Creating Oracle ASM disk "AVOL12" [ OK ]
# /etc/init.d/oracleasm createdisk AVOL13 /dev/sdg
Creating Oracle ASM disk "AVOL13" [ OK ]
- Verify disk creation. You can use the listdisks parameter with the oracleasm utility to verify that all the configured disks are visible to ASM.
$ /etc/init.d/oracleasm listdisks
AVOL11
AVOL12
AVOL13
[oracle@oradb3 oracle]$
Create ASM Instances
Once the disk volumes have been created using the Oracle-provided ASM utilities, the next step is to create an ASM instance.
In Oracle Database 10g Release 2, Oracle provides an option to install ASM in a separate home. Installing ASM in a separate home provides advantages in configuration management. In particular, it allows for upgrading the version of ASM while maintaining multiple versions of RDBMS.
If ASM is to be configured in a separate home, the previously covered installation steps will have to be repeated to a new home (ASM_HOME) location (/usr/app/oracle/product/10.2.0/asm, for example). However, if no separate home is configured, ASM instances can be created using the Database Configuration Assistant (DBCA) available under the original ORACLE_HOME as described below.
- Start DBA.
$ dbca
- Welcome—Select "Oracle Real Application Clusters database" and click on Next.
- Step 1 of 17: Options—Select "Configure Automatic Storage Management."
- Step 2 of 4: Node Selection—Click on Select All to select all nodes in the list of nodes.
- Step 3 of 4: Create ASM Instance—
- Add password for user SYS.
- Select "Create server parameter file (SPFILE)". Specify a destination on the shared storage. After these parameters are entered, DBCA will start creating the ASM instances on all nodes in the cluster.
- Click on OK to confirm creation of ASM instances.
- Step 3 of 3: ASM Disk Groups—Lists any available ASM diskgroups. If a new group is required, click on Create New.
- Create Disk Group—A list of ASM ready volumes is listed. Enter a valid ASM disk group name (ASMGRP1) and select the volumes to be part of this group (ORCL:AVOL11 and ORCL:AVOL12). Click on OK when selection is complete. This will create the disk group ASMGRP1 and mount the disk group on all ASM instances. (Note: Depending on the number of datafiles to be migrated, multiple ASM disk groups may be required.)
- Click on Finish when all required diskgroups are created.
- Verify if the ASM instances is up and running using the following OS command.
$ ps -ef | grep asm_
oracle 7450 1 0 22:45 ? 00:00:00 asm_pmon_+ASM1
oracle 7452 1 0 22:45 ? 00:00:00 asm_diag_+ASM1
oracle 7455 1 0 22:45 ? 00:00:00 asm_psp0_+ASM1
oracle 7464 1 0 22:45 ? 00:00:01 asm_lmon_+ASM1
oracle 7475 1 0 22:45 ? 00:00:00 asm_lmd0_+ASM1
oracle 7479 1 0 22:45 ? 00:00:00 asm_lms0_+ASM1
oracle 7491 1 0 22:45 ? 00:00:00 asm_mman_+ASM1
oracle 7493 1 0 22:45 ? 00:00:00 asm_dbw0_+ASM1
oracle 7495 1 0 22:45 ? 00:00:00 asm_lgwr_+ASM1
oracle 7497 1 0 22:45 ? 00:00:00 asm_ckpt_+ASM1
oracle 7499 1 0 22:45 ? 00:00:00 asm_smon_+ASM1
oracle 7501 1 0 22:45 ? 00:00:00 asm_rbal_+ASM1
oracle 7503 1 0 22:45 ? 00:00:00 asm_gmon_+ASM1
oracle 7554 1 0 22:45 ? 00:00:00 asm_lck0_+ASM1
oracle 8504 1 0 22:46 ? 00:00:00 asm_pz99_+ASM1
oracle 12959 1 0 22:49 ? 00:00:00 asm_pz98_+ASM1
- Verify if the ASM disk group is created and mounted successfully.
$ export ORACLE_SID=+ASM1
$ sqlplus '/as sysdba'
SQL*Plus: Release 10.2.0.1.0 - Production on Wed Aug 17 22:52:09 2005
Copyright (c) 1982, 2005, Oracle. All rights reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
With the Partitioning, Real Application Clusters, OLAP and Data Mining options
SQL> SELECT INST_ID,NAME,STATE,TYPE FROM GV$ASM_DISKGROUP;
INST_ID NAME STATE TYPE
---------- ------------------------------ ----------- ------
1 ASMGRP1 MOUNTED NORMAL
2 ASMGRP1 MOUNTED NORMAL
3 ASMGRP1 MOUNTED NORMAL
4 ASMGRP1 MOUNTED NORMAL
Migrate Datafiles to ASM
Oracle supports conversion of the entire database or a subset of datafiles to ASM via two methods: Recovery Manager (RMAN) and Oracle Enterprise Manager (OEM) 10g. In this section, I will describe this process using the latter.
- First, verify the current datafiles in the database using SQL*Plus.
SQL> select name from v$datafile;
NAME
---------------------------------------------------------------
/u14/oradata/SSKYDB/system01.dbf
/u14/oradata/SSKYDB/undotbs01.dbf
/u14/oradata/SSKYDB/tools01.dbf
/u14/oradata/SSKYDB/users01.dbf
/u14/oradata/SSKYDB/cwmlite01.dbf
/u14/oradata/SSKYDB/drysys01.dbf
/u14/oradata/SSKYDB/indx01.dbf
/u14/oradata/SSKYDB/xdb01.dbf
/u14/oradata/SSKYDB/undotbs02.dbf
.............
.............
- Verify if the OEM console is started using the emctl utility:
$ emctl status dbconsole
TZ set to US/Eastern
Oracle Enterprise Manager 10g Database Control Release 10.2.0.1.0
Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
http://oradb3.sumsky.com:1158/em/console/aboutApplication
Oracle Enterprise Manager 10g is running.
------------------------------------------------------------------
Logs are generated in directory /usr/app/oracle/product/10.2.0/db_1/oradb3_SSKY1/sysman/log
If the dbconsole is not started, you can start it using the following command:
$ emctl start dbconsole
TZ set to US/Eastern
Oracle Enterprise Manager 10g Database Control Release 10.2.0.1.0
Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
http://oradb3.sumsky.com:1158/em/console/aboutApplication
Starting Oracle Enterprise Manager 10g Database Control........ started.
------------------------------------------------------------------
Logs are generated in directory /usr/app/oracle/product/10.2.0/db_1/oradb3_SSKY1/sysman/log
- Invoke the OEM dbconsole by connecting a browser to http://DATABASE NAME:1158/em.
- Select the administrator tab on this main page.
- Under the "Change Database" section, select the Migrate to ASM option.
- Enter SYS password and the "Host Credentials" where ASM instance will be created.
- Map the existing datafiles to the appropriate ASM diskgroups. When this step is complete, OEM will create a job to perform the task. OEM uses DBMS_SCHEDULER to create and schedule the job.
- Next is the review screen, highlighting the files that will be migrated to ASM. After verifying if the datafiles have been mapped to the appropriate disk groups, click on Submit to submit the job and start the migration process. Once the job has been submitted successfully, a verification screen is displayed.
- At this stage, the job will perform the following actions automatically.
- Shuts down all instances in the cluster
- Mounts the database on one instance, normally the instance from where the dbconsole is invoked
- Copies the files to the ASM diskgroup and renames them in the control file
- Drops the tempfiles
- Opens the database
- Adds the tempfiles in the ASM diskgroup
- Drops the logfiles and creates them in the ASM diskgroup with two members
- The alert log of the instance where the dbconsole is executed provides a step-by-step set of actions performed by Oracle to complete the migration. Verify against this log file to confirm status of the migration process.
- Once complete, verify the confirmation using the following query:
SQL> SELECT NAME FROM V$DATAFILE;
NAME
------------------------------------------------
+ASMGRP1/sskydb/datafile/system.259.566609701
+ASMGRP1/sskydb/datafile/undotbs1.262.566609713
+ASMGRP1/sskydb/datafile/sysaux.260.566609707
+ASMGRP1/sskydb/datafile/users.265.566609923
+ASMGRP1/sskydb/datafile/example.261.566609707
+ASMGRP1/sskydb/datafile/undotbs2.263.566609817
.............
.............
SQL> SHOW PARAMETER PFILE
NAME TYPE VALUE
-------------------- ----------- ------------------------------
spfile string +ASMGRP1/sskydb/spfilessky2.ora_22
SQL> SHOW PARAMETER RECOVERY_FILE
NAME TYPE VALUE
------------------------------------ ----------- ----------------
db_recovery_file_dest string +ASMGRP1
db_recovery_file_dest_size big integer 2280M
Congratulations; you have successfully migrated your datafiles from OCFS to Oracle ASM.
Murali Vallath (murali.vallath@summersky.biz)
has over 18 years of IT experience with over 13 years using Oracle products. He is an Oracle Certified Database Administrator, serves as the president of the RAC SIG and the Charlotte Oracle Users Group, and provides independent Oracle consulting services focusing on designing and performance tuning of Oracle databases through Summersky Enterprises in Charlotte, N.C. |
Bookmark
