Communities
|
Social Applications
Networks
Support
|
|
C-Level Executives
Other Roles
|
|
Support
Education
Partner
Other Tasks
|
Articles
DBA: Linux
Adding New Nodes to Your Oracle RAC 10g Cluster on Linuxby Murali Vallath A step-by-step guide to adding a node to an existing Oracle RAC 10g Release 2 clusterPublished September 2006 In most businesses, a primary business requirement for an Oracle Real Application Clusters (RAC) configuration is scalability of the database tier across the entire system—so that when the number of users increases, additional instances can be added to the cluster to distribute the load. In Oracle RAC 10g, this specific feature has become much easier. Oracle incorporates the plug-and-play feature with a few minimum steps of setup, after the node/instance is brought to a usable state. In this article, I will discuss the steps required to add a node to an existing Oracle RAC 10g Release 2 cluster. Current EnvironmentFor demonstration purposes, our environment here is a four-node Red Hat Linux cluster. The task is to add an additional node, making it a five-node cluster.
The process will occur in the following seven steps:
Step 1: Account for Dependencies and PrerequisitesThe first primary step in any software installation or upgrade is to ensure that a complete backup of the system is available, including OS and data files. The next step is to verify system requirements, operating system versions, and all application patch levels. The new node should have the same version of the operating system as the existing nodes, including all patches required for Oracle. In this scenario, as the operating system residing on nodes 1 through 4 is Red Hat Enterprise Linux 3, the new node should have that version as well. And to maintain the current naming convention, you should call the new node oradb5. Apart from the basic operating system, the following packages required by Oracle should also be installed: [root@oradb5 root]# rpm -qa | grep -i gcc compat-gcc-c++-7.3-2.96.128 compat-gcc-7.3-2.96.128 libgcc-3.2.3-42 gcc-3.2.3-42 [root@oradb5 root]# rpm -qa | grep -i openmotif openmotif-2.2.3-3.RHEL3 openmotif21-2.1.30-8 [root@oradb5 root]# rpm -qa | grep -i glibc glibc-2.3.3-74 glibc-utils-2.3.3-74 glibc-kernheaders-2.4-8.34.1 glibc-common-2.3.3-74 glibc-headers-2.3.3-74 glibc-devel-2.3.3-74 [root@oradb5 root]# rpm -qa | grep -i compat compat-libstdc++-7.3-2.96.128 compat-gcc-c++-7.3-2.96.128 compat-gcc-7.3-2.96.128 compat-db-4.0.14-5 compat-libstdc++-devel-7.3-2.96.128 [root@oradb5 root]# Update the kernel parameters with the following values. kernel.core_uses_pid = 1 kernel.hostname = oradb5.sumsky.net kernel.domainname = sumsky.net kernel.shmall = 2097152 #kernel.shmmax = 536870912 kernel.shmmax = 2147483648 kernel.shmmni = 4096 kernel.shmseg = 4096 kernel.sem = 250 32000 100 150 kernel.msgmnl = 2878 kernel.msgmnb = 65535 fs.file-max = 65536 net.ipv4.ip_local_port_range = 1024 65000 net.core.rmem_default = 262144 net.core.wmem_default = 262144 net.core.rmem_max = 262144 net.core.wmem_max = 262144 Add the following parameters to /etc/security/limits.conf. oracle soft nproc 2047 oracle hard nproc 16384 oracle soft nofile 1024 oracle hard nofile 65536 Add devices to /etc/fstab--copy the devices definitions from one of the existing nodes to oradb5. [root@oradb5 root]$ more /etc/fstab LABEL=/ / ext3 defaults 1 1 none /dev/pts devpts gid=5,mode=620 0 0 none /proc proc defaults 0 0 none /dev/shm tmpfs defaults 0 0 /dev/sda2 swap swap defaults 0 0 /dev/cdrom /mnt/cdrom udf,iso9660 noauto,owner,kudzu,ro 0 0 /dev/fd0 /mnt/floppy auto noauto,owner,kudzu 0 0 /dev/sdb5 /u01 ocfs _netdev 0 0 /dev/sdb6 /u02 ocfs _netdev 0 0 /dev/sdb7 /u03 ocfs _netdev 0 0 /dev/sdb8 /u04 ocfs _netdev 0 0 /dev/sdb9 /u05 ocfs _netdev 0 0 /dev/sdb10 /u06 ocfs _netdev 0 0 /dev/sdb14 /u14 ocfs _netdev 0 0 Next, create the administrative user. Every installation of Oracle requires an administrative user account on each node. In all existing nodes the administrative owner is oracle, so the next step is to create an administrative user account on node oradb5. While creating this user account, it is important that the UID and the GID of user oracle are identical to that of the other RAC nodes. This information can be obtained using the following command: [oracle@oradb1 oracle]$ id oracle uid=500(oracle) gid=500(oinstall) groups=501(dba), 502(oper) Connect to oradb5 (Linux or Unix based environment) as root and create the following operating system groups. groupadd -g 500 oinstall groupadd -g 501 dba groupadd -g 502 oper Once the groups have been created, create the oracle user account as a member of the dba group using the following command, and subsequently reset the user password using the passwd (password) command. useradd -u 500 -g oinstall -G dba, oper oracle passwd oracle Changing password for user oracle. New password: Retype new password: passwd: all authentication tokens updated successfully. Once the groups and the user have been created, they should be verified to ensure that the output from the following command is identical across all nodes in the cluster. [root@oradb5 root]$ id oracle uid=500(oracle) gid=500(oinstall) groups=501(dba), 502(oper) Step 2: Configure Network ComponentsAdd all network address to the /etc/hosts file on node oradb5. Also cross-register node oradb5 information on other four nodes in the cluster.
root@oradb5 root]# more /etc/hosts
127.0.0.1 localhost.localdomain localhost
192.168.2.10 oradb1.sumsky.net oradb1
192.168.2.20 oradb2.sumsky.net oradb2
192.168.2.30 oradb3.sumsky.net oradb3
192.168.2.40 oradb4.sumsky.net oradb4
192.168.2.50 oradb5.sumsky.net oradb5
#Private Network/interconnect
10.168.2.110 oradb1-priv.sumsky.net oradb1-priv
10.168.2.120 oradb2-priv.sumsky.net oradb2-priv
10.168.2.130 oradb3-priv.sumsky.net oradb3-priv
10.168.2.140 oradb4-priv.sumsky.net oradb4-priv
10.168.2.150 oradb5-priv.sumsky.net oradb5-priv
# VIP
192.168.2.15 oradb1-vip.sumsky.net oradb1-vip
192.168.2.25 oradb2-vip.sumsky.net oradb2-vip
192.168.2.35 oradb3-vip.sumsky.net oradb3-vip
192.168.2.45 oradb4-vip.sumsky.net oradb4-vip
192.168.2.55 oradb5-vip.sumsky.net oradb5-vip
Establishing user equivalence with SSH. When adding nodes to the cluster, Oracle copies files from the node where the installation was originally performed to the new node in the cluster. Such a copy process is performed either by using ssh protocol where available or by using the remote copy ( rcp). In order for the copy operation to be successful, the oracle user on the RAC node must be able to login to the new RAC node without having to provide a password or passphrase. Currently the existing four nodes are configured to use ssh. To configure the oracle account on the new node to use ssh without using any passwords, perform the following tasks:
Step 3: Install Oracle ClusterwareOracle Clusterware is already installed on the cluster; the task here is to add the new node to the clustered configuration. This task is performed by executing the Oracle provided utility called addnode located in the Clusterware’s home oui/bin directory. Oracle Clusterware has two files (Oracle cluster repository, OCR; and Oracle Cluster Synchronization service, CSS, voting disk) that contain information concerning the cluster and the applications managed by the Oracle Clusterware. These files need to be updated with the information concerning the new node. The first step in the clusterware installation process is to verify if the new node is ready for the install. Cluster Verification. In Oracle Database 10g Release 2, Oracle introduced a new utility called Cluster Verification Utility (CVU)as part of the clusterware software. Executing the utility using the appropriate parameters determines the status of the cluster. At this stage, before beginning installation of the Oracle Clusterware, you should perform two verifications:
Step 4: Configure Oracle ClusterwareUsing the OUI requires that the terminal from where this installer is ran be X-windows compatible. If not, an appropriate X-windows emulator should be installed and the emulator invoked using the DISPLAY command using the following syntax. export DISPLAY=<client IP address>:0.0 For example: [oracle@oradb1 oracle]$export DISPLAY=192.168.2.101:0.0 The next step is to configure the Clusterware on the new node oradb5. For this, as mentioned earlier, Oracle has provided a new executable called addNode.sh located in the <Clusterware Home>/oui/bin directory.
Step 5: Install Oracle SoftwareThe next step is to install the Oracle software on the new node. As mentioned earlier, Oracle has provided a new executable called addNode.sh located in the $ORACLE_HOME/oui/bin directory.
Once the RDBMS software is installed, it is a good practice to run netca before moving to the next step. Netca will configure all required network files and parameters such as the listener, sql*net, and tnsnames.ora files. Step 6: Add New Instance(s)DBCA has all the required options to add additional instances to the cluster. Requirements:
Step 7: Do Housekeeping ChoresFor easy administration and navigation, you should define several different environment variables in the login profile. For example:
[oracle@oradb5 oracle]$ more .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
export ORACLE_BASE=/usr/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/10.2.0/db_1
export ORA_CRS_HOME=$ORACLE_BASE/product/10.2.0/crs
export PATH=.:${PATH}:$HOME/bin:$ORACLE_HOME/bin
export PATH=${PATH}:$ORA_CRS_HOME/bin
export PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin:/sbin
export ORACLE_ADMIN=$ORACLE_BASE/admin
export TNS_ADMIN=$ORACLE_HOME/network/admin
export LD_ASSUME_KERNEL=2.4.19
export LD_LIBRARY=$ORACLE_HOME/lib
export LD_LIBRARY=${LD_LIBRARY}:/lib:/usr/lib:/usr/local/bin
export LD_LIBRARY=${LD_LIBRARY}:$ORA_CRS_HOME/lib
export CLASSPATH=$ORACLE_HOME/JRE
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib
export THREADS_FLAG=native
export ORACLE_SID=SSKY5
Add network addresses to DNS for lookup. In order for the applications and clients connecting to the database using VIP to translate the alias to an appropriate IP address, it is important that the VIP addresses are added to the DNS. Also add the new network address to the clinet tnsnames.ora file to the appropriate connect descriptors.
CRAC =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = oradb1-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = oradb2-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = oradb3-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = oradb4-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = oradb5-vip)(PORT = 1521))
(LOAD_BALANCE = yes)
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = CRM)
)
)
If the servers are configured to use FAN features, add the new server address to the onsctl files on all database servers. The ons.config file is located in
[oracle@oradb4 oracle]$ more $ORACLE_HOME/opmn/conf/ons.config
localport=6101
remoteport=6201
loglevel=3
useocr=on
nodes=oradb4.sumsky.net:6101,oradb2.sumsky.net:6201,
oradb1.sumsky.net:6201,oradb3.sumsky.net:6201,
oradb5.sumsky.net:6201
onsclient1.sumsky.net:6200,onsclient2.sumsky.net:6200
ConclusionCongratulations, you have just added a new node to an existing configuration of four nodes. The following table shows the new configuration.
Murali Vallath has over 17 years of experience designing and developing databases. He provides independent Oracle consulting services focusing on designing and performance tuning of Oracle databases through Summersky Enterprises (www.summersky.biz). Vallath has successfully completed over 60 successful small, medium and terabyte sized RAC implementations (Oracle9i & Oracle 10g) for reputed corporate firms. Vallath is the author of the Elsevier books Oracle Real Application Clusters and Oracle 10g RAC, Grid, Services & Clustering. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Printer View