| DBA: Linux
Installing Oracle RAC 10g Release 1 on
Linux x86
by John Smiley
Learn the basics of installing Oracle RAC 10g
Release 1 on Red Hat Enterprise Linux or Novell SUSE Enterprise Linux,
from the bare metal up (for evaluation purposes only)
Contents
Overview
Background
Part I: Install Linux
Part II: Configure Linux for Oracle
Part III: Prepare the Shared Disks
Part IV: Install Oracle RAC Software
Conclusion
Overview
This is the second in a series of guides that provide
all the steps for installing the major components of Oracle Database 10g
software on Linux. All five of the certified English-language distributions
of Linux are covered in detail (Asianux is not covered), and the guides
assume that inexpensive Intel x86 hardware is being used. The guides walk
you through the process of installation and configuration on commodity
hardware for the purpose of evaluating the major Oracle 10g products.
The ultimate goal of this series is to help you install and configure
all the components of an Oracle 10g Grid.
This guide will take the approach of offering the
easiest paths, with the fewest number of steps, for accomplishing a task.
This approach often means making configuration choices that would be inappropriate
for anything other than an evaluation. For that reason, this guide is
not appropriate for building production-quality environments, nor does
it reflect best practices.
The five Linux distributions certified for Oracle
Database 10g covered are:
- Red Hat Enterprise Linux 4 (RHEL4)
- Red Hat Enterprise Linux 3 (RHEL3)
- Red Hat Enterprise Linux 2.1 (RHEL2.1)
- Novell SUSE Linux Enterprise Server 9
- Novell SUSE Linux Enterprise Server 8
If you are new to Linux and/or Oracle, this guide is
for you. It starts with the basics and walks you through an installation
of Oracle Database 10g from the bare metal up.
This guide is divided into four parts: Part I covers
the installation of the Linux operating system, Part II covers configuring
Linux for Oracle, Part III discusses the essentials of partitioning shared
disk, and Part IV covers installation of the Oracle software.
Click
here to see the Release 2 version of this guide.
Background
The illustration below shows the major components of an Oracle RAC 10g
configuration. Nodes in the cluster are typically separate servers (hosts).
Hardware
At the hardware level, each node in a RAC cluster shares three things:
- Access to shared disk storage
- Connection to a private network
- Access to a public network.
Shared Disk Storage
Oracle RAC relies on a shared disk architecture. The database files, online
redo logs, and control files for the database must be accessible to each
node in the cluster. The shared disks also store the Oracle Cluster Registry
and Voting Disk (discussed later). There are a variety of ways to configure
shared storage including direct attached disks (typically SCSI over copper
or fiber), Storage Area Networks (SAN), and Network Attached Storage (NAS).
Private Network
Each cluster node is connected to all other nodes via a private high-speed
network, also known as the cluster interconnect or high-speed interconnect
(HSI). This network is used by Oracle's Cache Fusion technology to effectively
combine the physical memory (RAM) in each host into a single cache. Oracle
Cache Fusion allows data stored in the cache of one Oracle instance to
be accessed by any other instance by transferring it across the private
network. It also preserves data integrity and cache coherency by transmitting
locking and other synchronization information across cluster nodes.
The private network is typically built with Gigabit Ethernet, but for
high-volume environments, many vendors offer proprietary low-latency,
high-bandwidth solutions specifically designed for Oracle RAC. Linux also
offers a means of bonding multiple physical NICs into a single virtual
NIC (not covered here) to provide increased bandwidth and availability.
Public Network
To maintain high availability, each cluster node is assigned a virtual
IP address (VIP). In the event of host failure, the failed node's IP address
can be reassigned to a surviving node to allow applications to continue
accessing the database through the same IP address.
Configuring the Cluster Hardware
There are many, many different ways to configure the hardware for an Oracle
RAC cluster. Our configuration here uses two servers with two CPUs, 1GB
RAM, two Gigabit Ethernet NICs, a dual channel SCSI host bus adapter (HBA),
and eight SCSI disks connected via copper to each host (four disks per
channel). The disks were configured as Just a Bunch Of Disks (JBOD)—that
is, with no hardware RAID controller.
Software
At the software level, each node in a RAC cluster needs:
- An operating system
- Oracle Cluster Ready Services
- Oracle RAC software, and optionally
- An Oracle Automated Storage Management instance.
Operating System
Oracle RAC is supported on many different operating systems. This guide
focuses on Linux. The operating system must be properly configured for
the OS--including installing the necessary software packages, setting
kernel parameters, configuring the network, establishing an account with
the proper security, configuring disk devices, and creating directory
structures. All these tasks are described in this guide.
Oracle Cluster Ready Services
Oracle RAC 10g introduced Oracle Cluster Ready Services (CRS),
a platform-independent set of system services for cluster environments.
In prior releases of RAC and Oracle Parallel Server, Oracle relied on
vendor-supplied cluster management software to provide these services.
Although CRS works in concert with vendor-supplied clusterware, the only
required component for Oracle RAC 10g is CRS. Indeed, CRS must
be installed prior to installing RAC.
CRS maintains two files: the Oracle Cluster Registry (OCR) and the Voting
Disk. The OCR and the Voting Disk must reside on shared disks as either
raw partitions or files in a cluster filesystem. This guide describes
creating the OCR and Voting Disks using both methods and walks through
the CRS installation.
Oracle RAC Software
Oracle RAC 10g software is the heart of the RAC database and
must be installed on each cluster node. Fortunately, the Oracle Universal
Installer (OUI) does most of the work of installing the RAC software on
each node. You only have to install RAC on one node—OUI does the
rest.
Oracle Automated Storage Management (ASM)
ASM is a new feature in Oracle Database 10g that provides the
services of a filesystem, logical volume manager, and software RAID in
a platform-independent manner. Oracle ASM can stripe and mirror your disks,
allow disks to be added or removed while the database is under load, and
automatically balance I/O to remove "hot spots." It also supports direct
and asynchronous I/O and implements the Oracle Data Manager API (simplified
I/O system call interface) introduced in Oracle9i.
Oracle ASM is not a general-purpose filesystem and can be used only
for Oracle data files, redo logs, control files, and the RMAN Flash Recovery
Area. Files in ASM can be created and named automatically by the database
(by use of the Oracle Managed Files feature) or manually by the DBA. Because
the files stored in ASM are not accessible to the operating system, the
only way to perform backup and recovery operations on databases that use
ASM files is through Recovery Manager (RMAN).
ASM is implemented as a separate Oracle instance that must be up if
other databases are to be able to access it. Memory requirements for ASM
are light: only 64MB for most systems. In Oracle RAC environments, an
ASM instance must be running on each cluster node.
Part I: Installing Linux
Install and Configure Linux as described in the first
guide in this series. You will need three IP addresses for each server:
one for the private network, one for the public network, and one for the
virtual IP address. Use the operating system's network configuration tools
to assign the private and public network addresses. Do not assign the
virtual IP address using the operating system's network configuration
tools; this will be done by the Oracle Virtual IP Configuration Assistant
(VIPCA) during Oracle RAC software installation.
(A note about orarun.rpm for Novell SUSE environments: Novell provides
a packaged called orarun.rpm that is designed to simplify the installation
and administration of Oracle products on SLES. While it is an excellent
tool, its use requires a different set of installation steps. This guide
forgoes the use of orarun.rpm in favor of a uniform set of installation
instructions that apply to both SUSE and Red Hat.)
Red Hat Enterprise Linux 4 (RHEL4)
Required Kernel:
2.6.9-5.EL or higher
Verify kernel version:
# uname -r
2.6.9-5.ELsmp
Other required package versions (or higher):
make-3.80-5
gcc-3.4.3-9.EL4
compat-db-4.1.25-9
Verify installed packages:
# rpm -q make gcc compat-db
make-3.80-5
gcc-3.4.3-9.EL4
compat-db-4.1.25-9
Red Hat Enterprise Linux 3 (RHEL3)
Required Kernel:
2.4.21-4.EL or higher
Verify kernel version:
# uname -r
2.4.21-4.0.1.ELsmp
Other required package versions (or higher):
make-3.79
binutils-2.11.90.0.8-12
gcc-3.2.3-2
compat-db-4.0.14.5
compat-gcc-7.3-2.96.122
compat-gcc-c++-7.3-2.96.122
compat-libstdc++-7.3-2.96.122
compat-libstdc++-devel-7.3-2.96.122
openmotif21-2.1.30-8
setarch-1.3-1
Verify installed packages:
# rpm -q make binutils gcc compat-db compat-gcc compat-gcc-c++ compat-libstdc++
compat-libstdc++-devel openmotif setarch
make-3.79.1-17
binutils-2.14.90.0.4-26
gcc-3.2.3-20
compat-db-4.0.14-5
compat-gcc-7.3-2.96.122
compat-gcc-c++-7.3-2.96.122
compat-libstdc++-7.3-2.96.122
compat-libstdc++-devel-7.3-2.96.122
openmotif-2.2.2-16
setarch-1.3-1
Red Hat Enterprise Linux 2.1
Required Kernel:
2.4.9-e.25 or higher
Verify kernel version:
# uname -r
2.4.9-e.27smp
Other required package versions (or higher):
gcc-2.96.108.1
make-3.79
binutils-2.11.90.0.8-12
openmotif-2.1.30-11
glibc-2.2.4-31
Verify installed packages:
# rpm -q gcc make binutils openmotif glibc
gcc-2.96-118.7.2
make-3.79.1-8
binutils-2.11.90.0.8-12
openmotif-2.1.30-11
glibc-2.2.4-32.8
SUSE Linux Enterprise Server 9 (SLES9)
Required Package Sets:
Basis Runtime System
YaST
Graphical Base System
Linux Tools
KDE Desktop Environment
C/C++ Compiler and Tools (not selected by default)
Do not install:
Authentication Server (NIS, LDAP, Kerberos)
Required Kernel:
2.6.5-7.5 or higher
Verify kernel version:
# uname -r
2.6.5-7.97-smp
Other required package versions (or higher):
make-3.8
gcc-3.3.3-43
gcc-c++=3.3.3-43
libaio-0.3.98-18
libaio-devel-0.3.98-18
openmotif-libs-2.2.2-519
Verify installed packages:
# rpm -q make gcc gcc-c++ libaio libaio-devel openmotif-libs
make-3.80-184.1
gcc-3.3.3-43.24
gcc-c++-3.3.3-43.24
libaio-0.3.98-18.3
libaio-devel-0.3.98-18.3
openmotif-libs-2.2.2-519.1
SUSE Linux Enterprise Server 8 (SLES8)
The minimum required kernel version depends upon which shared storage
option you choose:
| Storage Option |
Kernel Version(s) |
| Raw |
2.4.21-138 or higher |
| ASM with Raw Devices |
2.4.21-138 or higher |
| ASM with ASMLib |
2.4.21-251 or higher |
| OCFS v1 (1.0.14-1) |
2.4.21-266 or higher |
Verify kernel version:
# uname -r
k_smp-2.4.21-215
Other required package versions (or higher):
make-3.79
binutils-2.11.90.0.8-12
gcc-3.2.2-38
openmotif-2.2.2-124
In order to install Cluster Ready Services, you will also need the following
package available from rpmfind.net ncompress-4.2.4-24.i386.rpm (or later)
The ncompress package conflicts with the gzip package, so use the --force
command line option to rpm.
Example:
# rpm -ivh --force ncompress-4.2.4-36.i386.rpm
Verify installed packages:
# rpm rpm -q gcc make binutils openmotif ncompress
gcc-3.2.2-38
make-3.79.1-407
binutils-2.12.90.0.15-50
openmotif-2.2.2-124
ncompress-4.2.4-36
Part II: Configure Linux for
Oracle
Create the Oracle Groups and User Account
Next we'll create the Linux groups and user account that will be used
to install and maintain the Oracle 10g software. The user account
will be called 'oracle' and the groups will be 'oinstall' and 'dba.' Execute
the following commands as root on one cluster host only:
/usr/sbin/groupadd oinstall
/usr/sbin/groupadd dba
/usr/sbin/useradd -m -g oinstall -G dba oracle
id oracle
Ex:
# /usr/sbin/groupadd oinstall
# /usr/sbin/groupadd dba
# /usr/sbin/useradd -m -g oinstall -G dba oracle
# id oracle
uid=501(oracle) gid=501(oinstall) groups=501(oinstall),502(dba)
The User ID and Group IDs must be the same on all cluster hosts. Using the
information from the id oracle command, create the Oracle Groups
and User Account on the remaining cluster hosts:
/usr/sbin/groupadd -g 501 oinstall
/usr/sbin/groupadd -g 502 dba
/usr/sbin/useradd -m -u 501 -g oinstall -G dba oracle
Ex:
# /usr/sbin/groupadd -g 501 oinstall
# /usr/sbin/groupadd -g 502 dba
# /usr/sbin/useradd -m -u 501 -g oinstall -G dba oracle
# id oracle
uid=501(oracle) gid=501(oinstall) groups=501(oinstall),502(dba)
Set the password on the oracle account:
# passwd oracle
Changing password for user oracle.
New password:
Retype new password:
passwd: all authentication tokens updated successfully.
Create Mount Points
Now create mount points to store the Oracle 10g software. This
guide will adhere to the Optimal Flexible Architecture (OFA) for the naming
conventions used in creating the directory structure. For more information
on OFA standards, see Appendix D of the Oracle Database 10g Installation
Guide.
Issue the following commands as root:
mkdir -p /u01/app/oracle
chown -R oracle:oinstall /u01/app/oracle
chmod -R 775 /u01/app/oracle
Ex:
# mkdir -p /u01/app/oracle
# chown -R oracle:oinstall /u01/app/oracle
# chmod -R 775 /u01/app/oracle
Configure Kernel Parameters
Login as root and configure the Linux kernel parameters on each host.
cat >> /etc/sysctl.conf >>EOF
kernel.shmall = 2097152
kernel.shmmax = 2147483648
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 65536
net.ipv4.ip_local_port_range = 1024 65000
net.core.rmem_default=262144
net.core.wmem_default=262144
net.core.rmem_max=262144
net.core.wmem_max=262144
EOF
/sbin/sysctl -p
For Novell SUSE releases, run the following command after completing the
step above:
/sbin/chkconfig boot.sysctl on
Setting Shell Limits for the oracle User
Oracle recommends setting the limits to the number of processes and number
of open files each Linux account may use. To make these changes, cut and
past the following commands as root.
cat >> /etc/security/limits.conf >>EOF
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
EOF
cat >> /etc/pam.d/login >>EOF
session required /lib/security/pam_limits.so
EOF
For Red Hat Enterprise Linux releases, use the following:
cat >> /etc/profile >>EOF
if [ \$USER = "oracle" ]; then
if [ \$SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
umask 022
fi
EOF
cat >> /etc/csh.login >>EOF
if ( \$USER == "oracle" ) then
limit maxproc 16384
limit descriptors 65536
umask 022
endif
EOF
For Novell SUSE releases, use the following:
cat >> /etc/profile.local >>EOF
if [ \$USER = "oracle" ]; then
if [ \$SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
umask 022
fi
EOF
cat >> /etc/csh.login.local >>EOF
if ( \$USER == "oracle" ) then
limit maxproc 16384
limit descriptors 65536
umask 022
endif
EOF
SLES8 and SLES9: Avoid the Bug!
A bug in the installation of Oracle Enterprise Manager 10g
on SLES8 and SLES9 will cause it to fail due to unavailable network ports.
The OEM DBConsole needs port 1830 and in SLES environments this port is
already reserved in /etc/services. This bug is documented on MetaLink
as bug# 3513603.
To avoid problems during the installation, log in as root and comment out
the lines for ports 1830 through 1849 in the /etc/services file prior to
installing the Oracle 10g software.
Configure the Hangcheck Timer
All RHEL releases:
modprobe hangcheck-timer hangcheck_tick=30 hangcheck_margin=180
cat >> /etc/rc.d/rc.local >>EOF
modprobe hangcheck-timer hangcheck_tick=30 hangcheck_margin=180
EOF
All SLES releases:
modprobe hangcheck-timer hangcheck_tick=30 hangcheck_margin=180
cat >> /etc/init.d/boot.local >>EOF
modprobe hangcheck-timer hangcheck_tick=30 hangcheck_margin=180
EOF
Configure /etc/hosts
Some Linux distributions associate the host name with the loopback address
(127.0.0.1). If this occurs, remove the host name from the loopback address.
/etc/hosts file used for this walkthrough:
127.0.0.1 localhost.localdomain localhost
192.168.100.51 ds1-priv.orademo.org ds1-priv # ds1 private
192.168.100.52 ds2-priv.orademo.org ds2-priv # ds1 private
192.168.200.51 ds1.orademo.org ds1 # ds1 public
192.168.200.52 ds2.orademo.org ds2 # ds2 public
192.168.200.61 ds1-vip.orademo.org ds1-vip # ds1 virtual
192.168.200.62 ds2-vip.orademo.org ds2-vip # ds2 virtual
Configure SSH for User Equivalence
During the installation of Oracle RAC 10g, OUI needs to copy files
to and execute programs on the other hosts in the cluster. In order to allow
OUI to do that, you must configure SSH to allow user equivalence. Establishing
user equivalence with SSH provides a secure means of copying files and executing
programs on other hosts in the cluster without requiring password prompts.
The first step is to generate public and private keys for SSH. There are
two versions of the SSH protocol; version 1 uses RSA and version 2 uses
DSA, so we will create both types of keys to ensure that SSH can use either
version. The ssh-keygen program will generate public and private keys of
either type depending upon the parameters passed to it.
When you run ssh-keygen, you will be prompted for a location to save the
keys. Just press Enter when prompted to accept the default. You will then
be prompted for a passphrase. Enter a password that you will remember and
then enter it again to confirm. When you have completed the steps below,
you will have four files in the ~/.ssh directory: id_rsa, id_rsa.pub, id_dsa,
and id_dsa.pub. The id_rsa and id_dsa files are your private keys and must
not be shared with anyone. The id_rsa.pub and id_dsa.pub files are your
public keys and must be copied to each of the other hosts in the cluster.
From each host, logged in as oracle:
mkdir ~/.ssh
chmod 755 ~/.ssh
/usr/bin/ssh-keygen -t rsa
Cut and paste the following line separately:
/usr/bin/ssh-keygen -t dsa
Ex:
$ mkdir ~/.ssh
$ chmod 755 ~/.ssh
$ /usr/bin/ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/oracle/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/oracle/.ssh/id_rsa.
Your public key has been saved in /home/oracle/.ssh/id_rsa.pub.
The key fingerprint is:
4b:df:76:77:72:ba:31:cd:c4:e2:0c:e6:ef:30:fc:37 oracle@ds1.orademo.org
$ /usr/bin/ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/oracle/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/oracle/.ssh/id_dsa.
Your public key has been saved in /home/oracle/.ssh/id_dsa.pub.
The key fingerprint is:
af:37:ca:69:3c:a0:08:97:cb:9c:0b:b0:20:70:e3:4a oracle@ds1.orademo.org
Now the contents of the public key files id_rsa.pub and id_dsa.pub on each
host must be copied to the ~/.ssh/authorized_keys file on every other host.
Use ssh to copy the contents of each file to the ~/.ssh/authorized_keys
file. Note that the first time you access a remote host with ssh its RSA
key will be unknown and you will be prompted to confirm that you wish to
connect to the host. SSH will record the RSA key for the remote hosts and
will not prompt for this on subsequent connections to that host.
From the first host ONLY, logged in as oracle (copy the local account's
keys so that ssh to the local host will work):
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
ssh oracle@ds2 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
(If you are cut and pasting these commands, run each of them separately.
SSH will prompt for the oracle password each time and if the commands are
pasted at the same time, the other commands will be lost when the first
one flushes the input buffer prior to prompting for the password.)
ssh oracle@ds2 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
chmod 644 ~/.ssh/authorized_keys
Ex:
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
$ ssh oracle@ds2 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
The authenticity of host 'ds2 (192.168.200.52)' can't be established.
RSA key fingerprint is d1:23:a7:df:c5:fc:4e:10:d2:83:60:49:25:e8:eb:11.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ds2,192.168.200.52' (RSA) to the list of known hosts.
oracle@ds2's password:
$ ssh oracle@ds2 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
oracle@ds2's password:
$ chmod 644 ~/.ssh/authorized_keys
Now do the same for the second host. Notice that this time SSH will prompt
for the passphrase you used when creating the keys rather than the oracle
password. This is because the first host (ds1) now knows the public keys
for the second host and SSH is now using a different authentication protocol.
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
ssh oracle@ds1 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh oracle@ds1 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
chmod 644 ~/.ssh/authorized_keys
Ex:
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
$ ssh oracle@ds1 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
The authenticity of host 'ds1 (192.168.200.51)' can't be established.
RSA key fingerprint is bd:0e:39:2a:23:2d:ca:f9:ea:71:f5:3d:d3:dd:3b:65.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ds1,192.168.200.51' (RSA) to the list of known hosts.
Enter passphrase for key '/home/oracle/.ssh/id_rsa':
$ ssh oracle@ds1 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
Enter passphrase for key '/home/oracle/.ssh/id_rsa':
$ chmod 644 ~/.ssh/authorized_keys
Establish User Equivalence
Finally, after all of the generating of keys, copying of files, and repeatedly
entering passwords and passphrases (isn't security fun?), you're ready to
establish user equivalence. When user equivalence is established, you won't
be prompted for a password again.
As oracle on the node where the Oracle 10g software will be installed
(ds1):
exec /usr/bin/ssh-agent $SHELL
/usr/bin/ssh-add
Ex:
$ exec /usr/bin/ssh-agent $SHELL
$ /usr/bin/ssh-add
Enter passphrase for /home/oracle/.ssh/id_rsa:
Identity added: /home/oracle/.ssh/id_rsa (/home/oracle/.ssh/id_rsa)
Identity added: /home/oracle/.ssh/id_dsa (/home/oracle/.ssh/id_dsa)
(Note that user equivalence is established for the current session only.
If you switch to a different session or log out and back in, you will have
to run ssh-agent and ssh-add again to re-establish user equivalence.)
Test Connectivity
If everything is set up correctly, you can now use ssh to log in, execute
programs, and copy files on the other cluster hosts without having to enter
a password. Verify user equivalence by running a simple command like date
on a remote cluster host:
$ ssh ds2 date
Sun Jun 27 19:07:19 CDT 2004
It is crucial that you to test connectivity in each direction from all servers.
That will ensure that messages like the one below do not occur when the
OUI attempts to copy files during CRS and database software installation.
This message will only occur the first time an operation on a remote node
is performed, so by testing the connectivity, you not only ensure that remote
operations work properly, you also complete the initial security key exchange.
The authenticity of host 'ds2 (192.168.200.52)' can't be established.
RSA key fingerprint is 8f:a3:19:76:ca:4f:71:85:42:c2:7a:da:eb:53:76:85.
Are you sure you want to continue connecting (yes/no)? yes
Part III: Prepare the Shared Disks
This section describes three methods of preparing the shared disks for
use with RAC:
1. Oracle Cluster File System (OCFS)
2. Automatic Storage Manager (ASM)
3. Raw Devices
Oracle Cluster File System (OCFS) version 1
OCFS version 1 is a great alternative to raw devices. Not only is it easier
to administer and maintain, it overcomes the limit of 255 raw devices. However,
it is not a general-purpose cluster filesystem. It may only be used to store
the following types of files:
- Oracle data files
- Online redo logs
- Archived redo logs
- Control files
- Spfiles
- CRS shared files (Oracle Cluster Registry and CRS
voting disk).
For more background on OCFS, read the OTN Technical Article "Introduction
to Linux Cluster Filesystems."
Obtain OCFS
Go to http://oss.oracle.com/projects/ocfs and download the OCFS RPMs for
your version of Linux. You will need three RPMs: ocfs-support, ocfs-tools,
and ocfs-kernel-specific module. To determine the kernel-specific module
that you need, use uname -r.
# uname -r
2.4.21-15.0.2.ELsmp
For this example I downloaded:
ocfs-support-1.0.10-1.i386.rpm
ocfs-tools-1.0.10-1.i386.rpm
ocfs-2.4.21-EL-smp-1.0.12-1.i686.rpm
Install OCFS
# rpm -Uvh ocfs-support-1.0.10-1.i386.rpm \
ocfs-tools-1.0.10-1.i386.rpm \
ocfs-2.4.21-EL-smp-1.0.12-1.i686.rpm
Preparing... ########################################### [100%]
1:ocfs-support ########################################### [ 33%]
2:ocfs-tools ########################################### [ 67%]
3:ocfs-2.4.21-EL-smp ########################################### [100%]
Linking OCFS module into the module path [ OK ]
Configure OCFS
Run ocfstool as root:
# ocfstool
Select Tasks → Generate Config
Select the Interface corresponding to the private network for RAC
Click OK to generate the /etc/ocfs.conf file and then exit ocfstool
Repeat for each node in the cluster.
Load OCFS on each node:
# /sbin/load_ocfs
/sbin/insmod ocfs node_name=ds1.orademo.org ip_address=192.168.100.51
cs=1795 guid=2FB60EDD8B872FC4216C00010324C023 comm_voting=1 ip_port=7000
Using /lib/modules/2.4.21-EL-smp-ABI/ocfs/ocfs.o
Warning: kernel-module version mismatch
/lib/modules/2.4.21-EL-smp-ABI/ocfs/ocfs.o was compiled for kernel version 2.4.21-4.ELsmp
while this kernel is version 2.4.21-15.0.2.ELsmp
Warning: loading /lib/modules/2.4.21-EL-smp-ABI/ocfs/ocfs.o will taint the kernel: forced load
See http://www.tux.org/lkml/#export-tainted for information about tainted modules
Module ocfs loaded, with warnings
Don't worry about warnings regarding different kernel versions and tainted
modules. This is just modutils letting us know that the ocfs module was
compiled for an older release of the Linux kernel. As long as our kernel
is at the same or later release than the ocfs module, ocfs will work just
fine.
Create a mount point for the OCFS filesystem:
mkdir /u02
Create an OCFS filesystem:
mkfs.ocfs -b 128 -L /u02 -m /u02 -p 0775 \
-u root -g root /dev/sdb1 -F
The Magic Mount
mkfs.ocfs only does a portion of the work of initializing an OCFS filesystem
The rest of the initialization occurs when the filesystem is mounted for
the first time. For this reason, you must mount the newly created file system
on the same host that created it.
mount -t ocfs -L /u02 /u02
Notice that the mount command uses the filesystem label (-L /u02) used during
the creation of the filesystem. This is a handy way to refer to the file
system without having to remember the device name.
To verify that the OCFS filesystem is mounted, issue the mount command or
run df:
mount -t ocfs
/dev/sdb1 on /u02 type ocfs (rw)
df /u02
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sdb1 35557856 36064 35521792 1% /u02
Before mounting the OCFS filesystem on the other nodes, it's best to "cycle"
the mount by unmounting and then remounting it on the node that created
the file system:
umount /u02
mount -t ocfs -L /u02 /u02
The OCFS filesystem can now be mounted on the other cluster nodes.
To automatically mount the OCFS filesystem at system boot, add a line similar
to the one below to /etc/fstab:
LABEL=/u02 /u02 ocfs _netdev 0 0
Once again I have chosen to use the device label (LABEL=/u02) rather than
the device name (/dev/sdb1). This is especially useful if the device name
is different on each node. In my case the OCFS device is /dev/sdb1 on node
ds1 and /dev/sdf1 on node ds2. By using the device label instead of the
device name, I can use the same /etc/fstab entry on each node.
Create the directories for shared files:
CRS files
mkdir /u02/oracrs
chown oracle:oinstall /u02/oracrs
chmod 775 /u02/oracrs
Database files
mkdir /u02/oradata
chown oracle:oinstall /u02/oradata
chmod 775 /u02/oradata
Repeat this procedure for each disk partition that will be used for ocfs.
Oracle Automatic Storage Manager (ASM)
Install ASMLib
On Linux platforms, ASM can use raw devices or devices managed via the ASMLib
interface. Oracle recommends ASMLib over raw devices for ease of use and
performance reasons. ASMLib is available free of charge from OTN. This section
will walk through the process of configuring a simple ASM instance using
ASMLib and building a database that uses ASM for disk storage.
(Note: As of March 2005, ASMLib was not available for RHES4 or SLES9. It
is anticipated that AMLib will be included in future patchsets starting
with 10.1.0.4.)
Determine which version of ASMLib you need; ASMLib is delivered as a set
of three Linux packages:
oracleasmlib—the ASM libraries
oracleasm-support—Utilities needed to administer ASMLib
oracleasm—A kernel module for the ASM library
Each Linux distribution has its own set of ASMLib packages and within each
distribution, each kernel version has a corresponding oracleasm package.
The following paragraphs describe how to determine which set of packages
you need. Use the table below to assist in identifying the correct oracleasmlib
module for your environment.
| Linux Distro |
Supported Kernels |
| RHES 2.1 |
2.4.9-e.25 or higher |
| RHES 3 |
2.4.21-15 or higher |
| RHES 4 |
N/A |
| SLES 8 |
2.4.21-138
2.4.21-190
2.4.21-198
2.4-21-215 |
| SLES 9 |
N/A |
Determine which kernel you are using by logging in as root and running the
following command:
# uname -rm
2.4.9-e.27smp i686
The example shows that this is a 2.4.9 kernel for an SMP (multiprocessor)
box using Intel i686 CPUs.
Use this information to find the correct ASMLib packages on OTN:
- Point your Web browser to http://otn.oracle.com/tech/linux/asmlib/index.html.
- Select the link for your version of Linux.
- Download the oracleasmlib and oracleasm-support packages
for your version of Linux.
- Download the oracleasm package corresponding to your
kernel.
For the example above, I downloaded the oracleasm-2.4.9-e-smp-1.0.0-1.i686.rpm
package.
Next, install the packages on each cluster host by executing the following
command as root:
rpm -Uvh oracleasm-kernel_version-asmlib_version.cpu_type.rpm \
oracleasmlib-asmlib_version.cpu_type.rpm \
oracleasm-support-asmlib_version.cpu_type.rpm
Ex:
# rpm -Uvh \
> oracleasm-2.4.9-e-smp-1.0.0-1.i686.rpm \
> oracleasmlib-1.0.0-1.i386.rpm \
> oracleasm-support-1.0.0-1.i386.rpm
Preparing... ########################################### [100%]
1:oracleasm-support ########################################### [ 33%]
2:oracleasm-2.4.9-e-smp ########################################### [ 66%]
Linking module oracleasm.o into the module path [ OK ]
3:oracleasmlib ########################################### [100%]
Configuring ASMLib
Before using ASMLib, you must run a configuration script to prepare the
driver. Run the following command as root on each cluster host and answer
the prompts as shown in the example below.
# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.
This will configure the on-boot properties of the Oracle ASM library driver.
The following questions will determine whether the driver is loaded on boot
and what permissions it will have. The current values will be shown in brackets
('[]'). Hitting <ENTER> without typing an answer will keep that current
value. Ctrl-C will abort.
Default user to own the driver interface []: oracle
Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]: y
Fix permissions of Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration [ OK ]
Creating /dev/oracleasm mount point [ OK ]
Loading module "oracleasm" [ OK ]
Mounting ASMlib driver filesystem [ OK ]
Scanning system for ASM disks [ OK ]
Now enable the ASMLib driver as shown below.
# /etc/init.d/oracleasm enable
Writing Oracle ASM library driver configuration [ OK ]
Scanning system for ASM disks [ OK ]
Configure Disks for ASM
Next tell the ASMLib driver which disks you want it to use. Note that these
are bare disks with nothing on them (not even a partition). It is possible
to use partitions of disks with ASM, but not recommended.
Disks are marked for use by ASMLib by running the following command as root.
Do this from one cluster host only.
/etc/init.d/oracleasm createdisk DISK_NAME device_name
Tip: Enter the DISK_NAME in UPPER CASE. There is a bug in the current release
that prevents the disks from being visible to the ASM instance if lower
case is used.
# /etc/init.d/oracleasm createdisk VOL1 /dev/sdb
Marking disk "/dev/sdb" as an ASM disk [ OK ]
# /etc/init.d/oracleasm createdisk VOL2 /dev/sdc
Marking disk "/dev/sdc" as an ASM disk [ OK ]
.
.
.
The following example shows how to list all of the disks marked for use
by ASMLib.
# /etc/init.d/oracleasm listdisks
VOL1
VOL2
.
.
.
On all other cluster hosts, run the following command as root to scan for
configured ASMLib disks:
/etc/init.d/oracleasm scandisks
Raw Partitions
Until cluster file systems and ASM became available, raw partitions were
the only choice available for configuring shared disk storage for Oracle
RAC. As such, they make a very solid platform on which to build a RAC cluster
because the technology has been around for a long time and is well proven.
However, there are a number of disadvantages to raw partitions. Primarily
these have to do with the limit of 15 usable partitions per disk, the limit
of 255 raw devices imposed by Linux, and the additional administrative tasks
and planning required. For single-instance environments, Logical Volume
Manager (LVM) can reduce the administrative effort and provide for more
flexibility in storage layout. Unfortunately, LVM is not cluster aware and
is therefore not supported for use in RAC environments.
When using raw partitions, it is usually best to choose a set of standard
partition sizes that are all multiples of some base size. This allows for
greater flexibility in the storage layout and makes it easy to relocate
a partition to another disk when we need to re-balance I/O. I have chosen
standard partition sizes of 50, 200, 600, and 1200MB for this example.
To further simplify administration, each shared disk will have the same
partition layout containing some of each standardized partition size. The
maximum number of usable partitions that can be created on a single disk
is 15, so plan your partition layout carefully. The table below shows the
partition layout used for the raw partitions example in this guide.
| Partition |
Type |
Size |
| 1 |
Primary |
50 |
| 2 |
Primary |
50 |
| 3 |
Primary |
200 |
| 4 |
Extended |
- |
| 5 |
Logical |
200 |
| 6 |
Logical |
200 |
| 7 |
Logical |
200 |
| 8 |
Logical |
200 |
| 9 |
Logical |
200 |
| 10 |
Logical |
600 |
| 11 |
Logical |
600 |
| 12 |
Logical |
600 |
| 13 |
Logical |
1200 |
| 14 |
Logical |
1200 |
| 15 |
Logical |
Free |
| 16 |
Logical |
Free |
Use fdisk (or favorite disk partitioning tool) to create the partitions
for each of your shared disks. The example below shows a portion of the
procedure used for one of the disks used for this guide:
# fdisk /dev/sdb
The number of cylinders for this disk is set to 4427.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Command (m for help): p
Disk /dev/sdb: 36.4 GB, 36420075008 bytes
255 heads, 63 sectors/track, 4427 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-4427, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-4427, default 4427): +50m
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (8-4427, default 8):
Using default value 8
Last cylinder or +size or +sizeM or +sizeK (8-4427, default 4427): +50m
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 3
First cylinder (15-4427, default 15):
Using default value 15
Last cylinder or +size or +sizeM or +sizeK (15-4427, default 4427): +200m
Command (m for help): n
Command action
e extended
p primary partition (1-4)
e
Selected partition 4
First cylinder (40-4427, default 40):
Using default value 40
Last cylinder or +size or +sizeM or +sizeK (40-4427, default 4427):
Using default value 4427
Command (m for help): n
First cylinder (40-4427, default 40):
Using default value 40
Last cylinder or +size or +sizeM or +sizeK (40-4427, default 4427): +200m
.
.
.
Command (m for help): p
Disk /dev/sdb: 36.4 GB, 36420075008 bytes
255 heads, 63 sectors/track, 4427 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 1 7 56196 83 Linux
/dev/sdb2 8 14 56227+ 83 Linux
/dev/sdb3 15 39 200812+ 83 Linux
/dev/sdb4 40 4427 35246610 5 Extended
/dev/sdb5 40 64 200781 83 Linux
/dev/sdb6 65 89 200781 83 Linux
/dev/sdb7 90 114 200781 83 Linux
/dev/sdb8 115 139 200781 83 Linux
/dev/sdb9 140 164 200781 83 Linux
/dev/sdb10 165 238 594373+ 83 Linux
/dev/sdb11 239 312 594373+ 83 Linux
/dev/sdb12 313 386 594373+ 83 Linux
/dev/sdb13 387 533 1180746 83 Linux
/dev/sdb14 534 680 1180746 83 Linux
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
Run partprobe on the other hosts to sync up the kernel with the
changes you've made:
# partprobe
The table below lists each of the partitions that are required for a minimal
RAC database. The minimum partition sizes have been mapped to the standard
partition sizes chosen for this example. In this example, two shared disks
were used (/dev/sdb and /dev/sdc). (Note that SLES8 uses a different naming
standard for raw devices. /dev/raw/raw1 in Red Hat is /dev/raw1 in SLES8.
)
| Purpose |
Minimum Size (in MB) |
Standardized Size (in MB) |
Disk Device |
Raw Device |
| Oracle Cluster Registry |
100 |
200 |
/dev/sdb3 |
/dev/raw/raw1 |
| Oracle CRS Voting |
20 |
50 |
/dev/sdb1 |
/dev/raw/raw2 |
| SYSTEM Tablespace |
500 |
600 |
/dev/sdb11 |
/dev/raw/raw3 |
| SYSAUX Tablespace |
800 |
1200 |
/dev/sdc13 |
/dev/raw/raw4 |
| UNDOTBS1 Tablespace |
500 |
600 |
/dev/sdb10 |
/dev/raw/raw5 |
| UNDOTBS2 Tablespace |
500 |
600 |
/dev/sdc10 |
/dev/raw/raw6 |
| EXAMPLE Tablespace |
160 |
200 |
/dev/sdb5 |
/dev/raw/raw7 |
| USERS Tablespace |
120 |
200 |
/dev/sdb6 |
/dev/raw/raw8 |
| TEMP Tablespace |
250 |
600 |
/dev/sdc11 |
/dev/raw/raw9 |
| SPFILE |
5 |
50 |
/dev/sdb2 |
/dev/raw/raw10 |
| Password File |
5 |
50 |
/dev/sdc1 |
/dev/raw/raw11 |
| Control File 1 |
110 |
200 |
/dev/sdb7 |
/dev/raw/raw12 |
| Control File 2 |
110 |
200 |
/dev/sdb7 |
/dev/raw/raw13 |
| Redo Log 1_1 |
120 |
200 |
/dev/sdb8 |
/dev/raw/raw14 |
| Redo Log 1_2 |
120 |
200 |
/dev/sdb9 |
/dev/raw/raw15 |
| Redo Log 2_1 |
120 |
200 |
/dev/sdb8 |
/dev/raw/raw16 |
| Redo Log 2_2 |
120 |
200 |
/dev/sdb9 |
/dev/raw/raw17 |
| Total Disk Space: |
3,660 |
5,550 |
|
|
Map the raw devices to each of the partitions in the table:
RHEL Releases
Add the following lines to /etc/sysconfig/rawdevices on each cluster node:
/dev/raw/raw1 /dev/sdb3
/dev/raw/raw2 /dev/sdb1
/dev/raw/raw3 /dev/sdb11
/dev/raw/raw4 /dev/sdc13
/dev/raw/raw5 /dev/sdb10
/dev/raw/raw6 /dev/sdc10
/dev/raw/raw7 /dev/sdb5
/dev/raw/raw8 /dev/sdb6
/dev/raw/raw9 /dev/sdc11
/dev/raw/raw10 /dev/sdb2
/dev/raw/raw11 /dev/sdc1
/dev/raw/raw12 /dev/sdb7
/dev/raw/raw13 /dev/sdc7
/dev/raw/raw14 /dev/sdb8
/dev/raw/raw15 /dev/sdb9
/dev/raw/raw16 /dev/sdc8
/dev/raw/raw17 /dev/sdc9
SLES 8 and SLES9
Add the following lines to /etc/raw on each cluster node:
raw1:sdb3
raw2:sdb1
raw3:sdb11
raw4:sdc13
raw5:sdb10
raw6:sdc10
raw7:sdb5
raw8:sdb6
raw9:sdc11
raw10:sdb2
raw11:sdc1
raw12:sdb7
raw13:sdc7
raw14:sdb8
raw15:sdb9
raw16:sdc8
raw17:sdc9
Set ownership and permissions for the CRS raw devices on each node. The
ownership and permissions for these files must be set precisely as shown
(both the Installation Guide and the Release Notes are incorrect). Failure
to do so will result in a failed CRS install. If you need to clean up after
a failed CRS install, see MetaLink Note:239998.1.
chown root:oinstall /dev/raw/raw[12]
chmod 660 /dev/raw/raw[12]
Set ownership and permissions for the database raw devices on each node:
chown oracle:oinstall /dev/raw/raw[3-9]
chown oracle:oinstall /dev/raw/raw1[0-7]
chmod 660 /dev/raw/raw[3-9]
chmod 660 /dev/raw/raw1[0-7]
(Note: RHEL4 implements raw devices differently than in previous versions.
Each time the raw device service is started, the /dev/raw/raw* device files
are re-created, removing any existing non-default ownership or permissions.
There are several ways to address this issue when using raw devices with
Oracle. The simplest approach is to add the oracle account to the disk
group (ex: usermod –G dba,disk oracle). Alternatively, you
can create a startup script to alter the ownership and permissions of the
raw devices on system boot.)
Restart the raw device service.
RHEL2/3
/sbin/service rawdevices restart
SLES8/9
/etc/init.d/raw start
chkconfig raw on
Now, you could let it go at that and move on with the rest of the install,
but if you're like me, having to remember what each raw device does is a
bit much to ask. It would be nice if we could name each raw device according
to what they are used for. That's where symbolic links come in.
The table below shows each raw device and the filename with which we'd like
to associate it. (Actually, I'd like to associate the spfile and password
file with locations in $ORACLE_HOME/dbs, but as that directory doesn't exist
at this point in the installation, I chose locations in /u01/oradata/dbname
for the time being. The symbolic links for these raw devices can be relocated
to the $ORACLE_HOME/dbs directory after the installation is complete, if
you wish.)
| Purpose |
Raw Device |
Filename |
| Oracle Cluster Registry |
/dev/raw/raw1 |
/u02/oracrs/ocr.crs |
| Oracle CRS Voting |
/dev/raw/raw2 |
/u02/oracrs/vote.crs |
| SYSTEM Tablespace |
/dev/raw/raw3 |
/u02/oradata/gemni/system_01.dbf |
| SYSAUX Tablespace |
/dev/raw/raw4 |
/u02/oradata/gemni/sysaux_01.dbf |
| UNDOTBS1 Tablespace |
/dev/raw/raw5 |
/u02/oradata/gemni/undo1_01.dbf |
| UNDOTBS2 Tablespace |
/dev/raw/raw6 |
/u02/oradata/gemni/undo2_01.dbf |
| EXAMPLE Tablespace |
/dev/raw/raw7 |
/u02/oradata/gemni/example_01.dbf |
| USERS Tablespace |
/dev/raw/raw8 |
/u02/oradata/gemni/users_01.dbf |
| TEMP Tablespace |
/dev/raw/raw9 |
/u02/oradata/gemni/temp_01.dbf |
| SPFILE |
/dev/raw/raw10 |
u01/oradata/gemni/spfilegemni.ora |
| Password File |
/dev/raw/raw11 |
/u01/oradata/gemni/orapwgemni |
| Control File 1 |
/dev/raw/raw12 |
/u01/oradata/gemni/control.ctl |
| Control File 2 |
/dev/raw/raw13 |
/u02/oradata/gemni/control.ctl |
| Redo Log 1_1 |
/dev/raw/raw14 |
/u01/oradata/gemni/redo1_1.log |
| Redo Log 1_2 |
/dev/raw/raw15 |
/u01/oradata/gemni/redo1_2.log |
| Redo Log 2_1 |
/dev/raw/raw16 |
/u01/oradata/gemni/redo2_1.log |
| Redo Log 2_2 |
/dev/raw/raw17 |
/u01/oradata/gemni/redo2_2.log |
Now it's just a matter of creating the directories and the links:
As root, create the directories for symbolic links. In this example, the
database name is gemni.
CRS files
mkdir -p /u02/oracrs
chown -R oracle:oinstall /u02/oracrs
chmod -R 775 /u02/oracrs
Database files
mkdir -p /u01/oradata/gemni /u02/oradata/gemni
chown -R oracle:oinstall /u0[12]/oradata
chmod -R 775 /u0[12]/oradata
As oracle, create the symbolic links.
ln -s /dev/raw/raw1 /u02/oracrs/ocr.crs
ln -s /dev/raw/raw2 /u02/oracrs/vote.crs
ln -s /dev/raw/raw3 /u02/oradata/gemni/system_01.dbf
ln -s /dev/raw/raw4 /u02/oradata/gemni/sysaux_01.dbf
ln -s /dev/raw/raw5 /u02/oradata/gemni/undo1_01.dbf
ln -s /dev/raw/raw6 /u02/oradata/gemni/undo2_01.dbf
ln -s /dev/raw/raw7 /u02/oradata/gemni/example_01.dbf
ln -s /dev/raw/raw8 /u02/oradata/gemni/users_01.dbf
ln -s /dev/raw/raw9 /u02/oradata/gemni/temp_01.dbf
ln -s /dev/raw/raw10 /u01/oradata/gemni/spfilegemni.ora
ln -s /dev/raw/raw11 /u01/oradata/gemni/orapwgemni
ln -s /dev/raw/raw12 /u01/oradata/gemni/control.ctl
ln -s /dev/raw/raw13 /u02/oradata/gemni/control.ctl
ln -s /dev/raw/raw14 /u01/oradata/gemni/redo1_1.log
ln -s /dev/raw/raw15 /u01/oradata/gemni/redo1_2.log
ln -s /dev/raw/raw16 /u02/oradata/gemni/redo2_1.log
ln -s /dev/raw/raw17 /u02/oradata/gemni/redo2_2.log
Last, we need to create the raw device mapping file for the Database Configuration
Assistant (DBCA). This file associates the raw devices you've created with
names DBCA will use to create the database. Cut and paste the following
lines to create the mapping file for this example:
cat > $HOME/gemni_raw.conf << EOF
system=/u02/oradata/gemni/system_01.dbf
sysaux=/u02/oradata/gemni/sysaux_01.dbf
example=/u02/oradata/gemni/example_01.dbf
users=/u02/oradata/gemni/users_01.dbf
temp=/u02/oradata/gemni/temp_01.dbf
undotbs1=/u02/oradata/gemni/undo1_01.dbf
undotbs2=/u02/oradata/gemni/undo2_01.dbf
redo1_1=/u01/oradata/gemni/redo1_1.log
redo1_2=/u01/oradata/gemni/redo1_2.log
redo2_1=/u02/oradata/gemni/redo2_1.log
redo2_2=/u02/oradata/gemni/redo2_2.log
control1=/u01/oradata/gemni/control.ctl
control2=/u02/oradata/gemni/control.ctl
spfile=/u01/oradata/gemni/spfilegemni.ora
pwdfile=/u01/oradata/gemni/orapwgemni
EOF
Then export the DBCA_RAW_CONFIG environment variable to point to this file:
export DBCA_RAW_CONFIG=$HOME/gemni_raw.conf
Now you are ready to install Oracle CRS, install the Oracle Database
software, and create the Oracle RAC database.
Part IV: Install Oracle Software
Important Note (for RHEL4 only): The Oracle 10g OUI will check
the operating system release to verify that it is a supported release.
As of Oracle Database 10.1.0.3, the installer does not recognize RHEL4
as a supported release. As a workaround, follow the steps below prior
to running runInstaller.
cp /etc/redhat-release /etc/redhat-release.orig
cat > /etc/redhat-release << EOF
Red Hat Enterprise Linux AS release 3 (Taroon)
EOF
Establish User Equivalency
If you have not already done so, login as oracle and establish user equivalency
between nodes:
exec /usr/bin/ssh-agent $SHELL
/usr/bin/ssh-add
Enter passphrase for /home/oracle/.ssh/id_rsa:
Identity added: /home/oracle/.ssh/id_rsa (/home/oracle/.ssh/id_rsa)
Identity added: /home/oracle/.ssh/id_dsa (/home/oracle/.ssh/id_dsa)
Set the ORACLE_BASE and ORACLE_HOME environment variable:
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/10.1.0/crs_1
RHEL4 and SLES9 Only
export LD_ASSUME_KERNEL=2.4.21
Install Oracle CRS
Before installing the Oracle RAC 10g database software, you must
first install CRS. CRS requires two files to be shared among all of the
hosts in the cluster: the Oracle Cluster Registry (100MB) and the CRS Voting
Disk (20MB). These files may be stored on raw devices or on a cluster filesystem.
(NFS is also supported for certified NAS systems, but that is beyond the
scope of this guide.) Oracle ASM may not be used for these files because
ASM is dependent upon services provided by CRS.
In single-instance environments, OUI automatically installs a simplified
version of CRS specifically for Oracle ASM's use. For RAC installations,
a separate installation is required in order to configure the full version
of CRS. The full version of CRS is found on a separate installation CD,
referred to as the CRS CD.
You must create the CRS files using raw devices or OCFS before installing
the CRS software.
Creating Oracle CRS Files on Raw Devices
Refer to the section on raw devices in Part
III for an example of creating Oracle CRS files on raw devices.
Creating CRS Files on a Cluster Filesystem
Refer to the section on OCFS in Part
III for instructions on installing and configuring a cluster filesystem.
Note that OCFS and Oracle ASM can coexist on the same system, allowing OCFS
to support the Oracle CRS files and Oracle ASM to support the database.
Installing Oracle CRS
Mount the CD or filesystem containing the CRS media and execute runInstaller.
Although it is no longer necessary to change CDs during an installation
with Oracle 10g, it is recommended that you do so from a directory
other than the CD mount point when using CDs.
- Welcome; Click on Next
- Specify Inventory Directory and Credentials - The defaults should
be correct; make sure that the inventory directory is located in your
ORACLE_BASE directory (ex: /u01/app/oracle/oraInventory) and that the
operating system group is "oinstall" run orainstRoot.sh on the installation
node (ds1) (make sure that your host is configured in /etc/hosts and
not just in DNS)
- Specify File Locations - Verify the defaults and continue
- Language Selection - Verify the default and continue
- Cluster Configuration - Enter the cluster name (or accept the default
of "crs");
- Private Interconnect Enforcement - Specify the Interface Type (public,
private, or "do no use") for each interface
- Oracle Cluster Registry - enter the public and private node names
for each node in the clusterSpecify OCR Location (ex: /u02/oracrs/ocr.crs)
- Voting Disk - Enter voting disk name (ex: /u02/oracrs/vote.crs)
- Run orainstRoot.sh on remaining nodes in the cluster
- Summary - Click Install When prompted, run root.sh in the Oracle CRS
home directory (ex: /u01/app/oracle/product/10.1.0/crs_1/root.sh) on
each node one at a time, starting with the installation node. Do not
run the scripts simultaneously. Wait for one to finish before starting
another.
End of CRS Installation
Verify that the installation succeeded by running olsnodes from the
$ORACLE_BASE/product/10.1.0/crs_1/bin directory; for example:
$ cd $ORACLE_BASE/product/10.1.0/crs_1/bin
$ olsnodes
ds1
ds2
Once CRS is installed and operating, it's time to install the rest of the
Oracle RAC software.
Install Oracle Database Software
Set the ORACLE_BASE and ORACLE_HOME environment variable:
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/10.1.0/db_1
RHES4 and SLES9 Only
export LD_ASSUME_KERNEL=2.4.21
- runInstaller from db CD
- Welcome - Click on Next
- Specify File Locations - Verify the defaults and continue
- Specify Hardware Cluster Installation Mode - Select Cluster Installation
and select other nodes in the cluster
- Select Installation Type - Enterprise Edition
- Product-specific Prerequisite Checks - All OK; SLES9 OK to ignore
openmotif-2.1.30-11 warning
- Select Database Configuration - Choose "Do not create a starter database."
(We will create the database in a separate step using the Database Configuration
Assistant (DBCA).)
Summary
While logged in as root, run root.sh in the Oracle database home directory
(ex: /u01/app/oracle/product/10.1.0/db_1/root.sh) on each host one at
a time, starting with the installation node. Do not run the scripts simultaneously.
Wait for one to finish before starting another.
During execution of the root.sh script on the installation node, the Virtual
IP Configuration Assistant will appear.
- Welcome - Click on Next
- Network Interfaces - Select the interface for the public network only
(eth0 in this example). The interface must be identical on all hosts
in the cluster. (If it is eth0 on the installation host, it must be
eth0 on all other hosts in the cluster.)
- Virtual IPs for cluster nodes - Enter the virtual IP alias (host name)
and virtual IP address for each host that was configured in your DNS.
- Summary - Click on Finish. VIP Configuration assistant creates and
starts VIP, GSD, and ONS application resources
- Configuration Results - Review the results and click on Exit. Run
root.sh on other nodes, one at a time.
- End of Installation
Create the Oracle RAC Database
DBCA—Cluster Filesystem
While logged in as oracle, set the environment and then run dbca.
Ex:
$ . oraenv
ORACLE_SID = [oracle] ? *
$ dbca
- Welcome - Select "Oracle Real Application Clusters database"
- Operations - Create a database
- Node Selection - Click on Select All (ds1 and ds2)
- Database Templates - General Purpose
- Database Identification - Global Database Name: gemni.orademo.org
- Management Options - Configure the Database with Enterprise Manager;
Use Database Control for Database Management
- Database Credentials - Use the Same Password for All Accounts; enter
password and again to confirm
- Storage Options - Cluster File system
- Database File Locations - Use Common Location for All Database Files
(/u02/oradata)
- Recovery Configuration - Click on Next
- Database Content - Sample Schemas
- Initialization Parameters - Memory, Typical
- Database Storage - Click on Next
- Create Options - Select "Create Database"
- Summary - Review summary and click on OK
DBCA—Oracle ASM
While logged in as oracle, set the environment and then run dbca.
Ex:
$ . oraenv
ORACLE_SID = [oracle] ? *
$ dbca
- Welcome - Select "Oracle Real Application Clusters database"
- Operations - Create a database
- Node Selection - Click on Select All (ds1 and ds2)
- Database Templates - General Purpose
- Database Identification - Global Database Name: gemni.orademo.org
- Management Options - Configure the Database with Enterprise Manager;
Use Database Control for Database Management
- Database Credentials - Use the Same Password for All Accounts; enter
password and again to confirm
- Storage Options - Automatic Storage Management (ASM)
- Create ASM Instance - SYS Password and confirm; create initialization
parameter file (IFILE)
- ASM Disk Groups - Create New>Create Disk Group>Disk Group Name (DATA);
Redundancy (Normal); Change Disk Discovery Path (ORCL:* - Still have
to enter this even if the disks show up as provisioned, get error otherwise);
Select Disks; Enter Failure Group Names; Select newly created disk group
- Database File Locations - Use Oracle-Managed Files
- Recovery Configuration - Click on Next
- Database Content - Sample Schemas
- Initialization Parameters - Memory, Typical
- Database Storage - Click on Next
- Create Options - Select "Create Database"
- Summary - Review summary and click on OK
DBCA – Raw Devices
While logged in as oracle, set the environment and then run dbca.
Ex:
$ . oraenv
ORACLE_SID = [oracle] ? *
$ dbca
- Welcome - Select "Oracle Real Application Clusters database"
- Operations - Create a database
- Node Selection - Click on Select All (ds1 and ds2)
- Database Templates - General Purpose
- Database Identification - Global Database Name: gemni.orademo.org
- Management Options - Configure the Database with Enterprise Manager;
Use Database Control for Database Management
- Database Credentials - Use the Same Password for All Accounts; enter
password and again to confirm
- Storage Options - Raw Devices; Specify Raw Devices Mapping File (/home/oracle/gemni_raw.conf)
- Recovery Configuration - Click on Next
- Database Content - Sample Schemas
- Initialization Parameters - Memory, Typical
- Database Storage - Click on Next
- Create Options - Select "Create Database"
- Summary - Review summary and click on OK
Conclusion
Now that your database is up and running, you can begin exploring the
many new features offered in Oracle Database 10g. A great place
to start is Oracle Enterprise Manager, which has been completely re-written
with a crisp new Web-based interface. If you're unsure where to begin,
the Oracle
Database 10g Concepts Guide and the 2-Day
DBA Guide will help familiarize you with your new database. OTN
also has a number of guides designed to help you get the most out of 10g.
One of my favorites is the series by Arup Nanda, Oracle
Database 10g: The Top 20 Features for DBAs.
John Smiley (smileyj@tusc.com) is a Technical
Management Consultant at TUSC Inc. He is an Oracle Certified Master DBA
with 18 years of experience with Oracle-based systems as a DBA, developer,
and project manager. He specializes in advanced performance tuning, Oracle
Real Application Clusters, PL/SQL programming, capacity planning, logical
and physical database design, and UNIX. |
Bookmark
