Articles
- Application Development Framework
- Application Express
- Big Data
- Business Intelligence
- Cloud Computing
- Communications
- Database Performance & Availability
- Data Warehousing
- .NET
- Dynamic Scripting Languages
- Embedded
- Social Enterprise
- Enterprise Architecture
- Enterprise Management
- Identity & Security
- Java
- Linux
- Service-Oriented Architecture
- Solaris
- SQL & PL/SQL
- Systems - All Articles
- Virtualization
| Page 1 Page 2 Page 3
Build Your Own Oracle RAC Cluster on Oracle Enterprise Linux and iSCSI (Continued) 21. Install Oracle 10g Clusterware Software Perform the following installation procedures from only one of the Oracle RAC nodes in the cluster (linux1)! The Oracle Clusterware software will be installed to both of the Oracle RAC nodes in the cluster by the Oracle Universal Installer. You are now ready to install the "cluster" part of the environment: the Oracle Clusterware. In the previous section, you downloaded and extracted the install files for Oracle Clusterware to linux1 in the directory /home/oracle/orainstall/clusterware. This is the only node from which you need to perform the install from. During the installation of Oracle Clusterware, you will be asked for the nodes involved and to configure in the RAC cluster. Once the actual installation starts, it will copy the required software to all nodes using the remote access we configured in the section Section 15 ("Configure RAC Nodes for Remote Access using SSH"). So, what exactly is the Oracle Clusterware responsible for? It contains all of the cluster and database configuration metadata along with several system management features for RAC. It allows the DBA to register and invite an Oracle instance (or instances) to the cluster. During normal operation, Oracle Clusterware will send messages (via a special ping operation) to all nodes configured in the cluster, often called the "heartbeat." If the heartbeat fails for any of the nodes, it checks with the Oracle Clusterware configuration files (on the shared disk) to distinguish between a real node failure and a network failure. After installing Oracle Clusterware, the Oracle Universal Installer (OUI) used to install the Oracle Database software (next section) will automatically recognize these nodes. Like the Oracle Clusterware install you will be performing in this section, the Oracle Database software only needs to be run from one node. The OUI will copy the software packages to all nodes configured in the RAC cluster. Oracle Clusterware Shared Files The two shared files (actually file groups) used by Oracle Clusterware will be stored on the Oracle Cluster File System, Release 2 (OFCS2) we created earlier. The two shared Oracle Clusterware file groups are:
Note: It is not possible to use Automatic Storage Management (ASM) for the two shared Oracle Clusterware files: Oracle Cluster Registry (OCR) or the CRS Voting Disk files. The problem is that these files need to be in place and accessible BEFORE any Oracle instances can be started. For ASM to be available, the ASM instance would need to be run first. Also note that the two shared files could be stored on the OCFS2, shared RAW devices, or another vendor's clustered file system. Verifying Terminal Shell Environment Before starting the Oracle Universal Installer, you should first verify you are logged onto the server you will be running the installer from (i.e. linux1) then run the xhost command as root from the console to allow X Server connections. Next, login as the oracle user account. If you are using a remote client to connect to the node performing the installation (SSH or Telnet to linux1 from a workstation configured with an X Server), you will need to set the DISPLAY variable to point to your local workstation. Finally, verify remote access / user equivalence to all nodes in the cluster:
Installing Cluster Ready Services Perform the following tasks to install the Oracle Clusterware:
$
cd ~oracle
$
/home/oracle/orainstall/clusterware/runInstaller -ignoreSysPrereqs
Verify Oracle Clusterware Installation After the installation of Oracle Clusterware, we can run through several tests to verify the install was successful. Run the following commands on both nodes in the RAC cluster. Check cluster nodes
Confirm Oracle Clusterware Function
Check CRS Status
Check Oracle Clusterware Auto-Start Scripts
22. Install Oracle Database 10g Software Perform the following installation procedures from only one of the Oracle RAC nodes in the cluster (linux1)! The Oracle Database software will be installed to both of Oracle RAC nodes in the cluster by the Oracle Universal Installer. After successfully installing the Oracle Clusterware software, the next step is to install Oracle Database 10g Release 2 (10.2.0.1.0) with RAC. For the purpose of this example, you will forgo the "Create Database" option when installing the software. You will, instead, create the database using the Database Configuration Assistant (DBCA) after the install. Like the Oracle Clusterware install (previous section), the Oracle 10g database software only needs to be run from one node. The OUI will copy the software packages to all nodes configured in the RAC cluster. Verifying Terminal Shell Environment As discussed in the previous section, (Install Oracle Clusterware 10g Software), the terminal shell environment needs to be configured for remote access and user equivalence to all nodes in the cluster before running the Oracle Universal Installer. Note that you can utilize the same terminal shell session used in the previous section which in this case, you do not have to perform any of the actions described below with regards to setting up remote access and the DISPLAY variable:
Run the Oracle Cluster Verification Utility Before installing the Oracle Database Software, we should run the following database pre-installation check using the Cluster Verification Utility (CVU). Note: Instructions for configuring CVU can be found in the section " Prerequisites for Using Cluster Verification Utility discussed earlier in this article.
$
cd /home/oracle/orainstall/clusterware/cluvfy
$
CV_HOME=/home/oracle/orainstall/clusterware/cluvfy; export CV_HOME
$
CV_JDKHOME=/home/oracle/orainstall/clusterware/cluvfy/jdk14; export CV_JDKHOME
$
./runcluvfy.sh stage -pre dbinst -n linux1,linux2 -r 10gR2 -verbose
Review the CVU report. Note that this report will contain the same errors we received when checking pre-installation tasks for CRS failure to find a suitable set of interfaces for VIPs and the failure to find specific RPM packages that are not required with Oracle Enterprise Linux 5. These two errors can be safely ignored. Install Oracle Database 10g Release 2 Software Install the Oracle Database 10g Release 2 software with the following:
$
cd ~oracle
$
/home/oracle/orainstall/database/runInstaller -ignoreSysPrereqs
Unset LD_ASSUME_KERNEL in SRVCTL During the Oracle Clusterware installation in the previous section, we needed to modify both vipca and srvctl in the CRS bin directory. Those same modifications need to be performed with the new srvctl in the Database bin directory:
This error occurs because these releases of the Linux kernel fix an old bug in the Linux threading that Oracle worked around using LD_ASSUME_KERNEL settings in both vipca and srvctl, this workaround is no longer valid on OEL5 or RHEL5 or SLES10 hence the failures. To workaround this issue, edit srvctl (in the Database bin directory on all nodes) to undo the setting of LD_ASSUME_KERNEL by adding one line, around line 168:
Note: Remember to re-edit these files on all nodes: <ORA_CRS_HOME>/bin/vipca <ORA_CRS_HOME>/bin/srvctl <RDBMS_HOME>/bin/srvctl <ASM_HOME>/bin/srvctl # (If exists) after applying the 10.2.0.2 or 10.2.0.3 patch sets, as these patchset will still include those settings unnecessary for OEL5 or RHEL5 or SLES10. This issue was raised with development and is fixed in the 10.2.0.4 patchsets. Also note that we are explicitly unsetting LD_ASSUME_KERNEL and not merely commenting out its setting to handle a case where the user has it set in their environment (login shell).
23. Install Oracle 10g Companion CD Software Perform the following installation procedures from only one of the Oracle RAC nodes in the cluster (linux1)! The Oracle10g Companion CD software will be installed to both of Oracle RAC nodes in the cluster by the Oracle Universal Installer. After successfully installing the Oracle Database software, the next step is to install the Oracle Database 10g Companion CD Release 2 software (10.2.0.1.0). Please keep in mind that this is an optional step. For the purpose of this article, my testing database will often make use of the Java Virtual Machine (Java VM) and Oracle interMedia and therefore will require the installation of the Oracle Database 10g Companion CD. The type of installation to perform will be the Oracle Database 10g Products installation type. This installation type includes the Natively Compiled Java Libraries (NCOMP) files to improve Java performance. If you do not install the NCOMP files, the ORA-29558:JAccelerator (NCOMP) not installed error occurs when a database that uses Java VM is upgraded to the patch release. Verifying Terminal Shell Environment As discussed in the previous section, (Install Oracle Database 10g Software), the terminal shell environment needs to be configured for remote access and user equivalence to all nodes in the cluster before running the Oracle Universal Installer. Note that you can utilize the same terminal shell session used in the previous section which in this case, you do not have to perform any of the actions described below with regards to setting up remote access and the DISPLAY variable:
Install Oracle Database 10g Companion CD Software Install the Oracle Database 10g Companion CD software with the following:
$
cd ~oracle
$
/home/oracle/orainstall/companion/runInstaller -ignoreSysPrereqs
24. Create TNS Listener Process Perform the following configuration procedures from only one of the Oracle RAC nodes in the cluster (linux1)! The Network Configuration Assistant (NETCA) will setup the TNS listener in a clustered configuration on both of Oracle RAC nodes in the cluster. The DBCA requires the Oracle TNS Listener process to be configured and running on all nodes in the RAC cluster before it can create the clustered database. The process of creating the TNS listener only needs to be performed on one node in the cluster. All changes will be made and replicated to all nodes in the cluster. On one of the nodes (I will be using linux1) bring up the NETCA and run through the process of creating a new TNS listener process and also configure the node for local access. Verifying Terminal Shell Environment As discussed in the previous section, (Install Oracle Database 10g Companion CD Software), the terminal shell environment needs to be configured for remote access and user equivalence to all nodes in the cluster before running the Oracle Universal Installer. Note that you can utilize the same terminal shell session used in the previous section which in this case, you do not have to perform any of the actions described below with regards to setting up remote access and the DISPLAY variable:
Run the Network Configuration Assistant To start the NETCA, run the following:
$
netca &
The following table walks you through the process of creating a new Oracle listener for our RAC environment.
The Oracle TNS listener process should now be running on all nodes in the RAC cluster:
$
hostname
linux1
$
ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}'
LISTENER_LINUX1
=====================
$
hostname
linux2
$
ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}'
LISTENER_LINUX2
25. Create the Oracle Cluster Database The database creation process should only be performed from one of the Oracle RAC nodes in the cluster (linux1)! We will use the DBCA to create the clustered database. Before executing the DBCA, make sure that $ORACLE_HOME and $PATH are set appropriately for the $ORACLE_BASE/product/10.2.0/db_1 environment. You should also verify that all services we have installed up to this point (Oracle TNS listener, Oracle Clusterware processes, etc.) are running before attempting to start the clustered database creation process. Verifying Terminal Shell Environment As discussed in the previous section, (Create TNS Listener Process), the terminal shell environment needs to be configured for remote access and user equivalence to all nodes in the cluster before running the Oracle Universal Installer. Note that you can utilize the same terminal shell session used in the previous section which in this case, you do not have to perform any of the actions described below with regards to setting up remote access and the DISPLAY variable:
Run the Oracle Cluster Verification Utility Before creating the Oracle clustered database, we should run the following database configuration check using the Cluster Verification Utility (CVU). Note: Instructions for configuring CVU can be found in the section " Prerequisites for Using Cluster Verification Utility discussed earlier in this article.
$
cd /home/oracle/orainstall/clusterware/cluvfy
$
CV_HOME=/home/oracle/orainstall/clusterware/cluvfy; export CV_HOME
$
CV_JDKHOME=/home/oracle/orainstall/clusterware/cluvfy/jdk14; export CV_JDKHOME
$
./runcluvfy.sh stage -pre dbcfg -n linux1,linux2 -d ${ORACLE_HOME} -verbose
Review the CVU report. Note that this report will contain the same error we received when checking pre-installation tasks for CRS failure to find a suitable set of interfaces for VIPs. This error can be safely ignored. Create the Clustered Database To start the database creation process, run the following:
$
dbca &
When the DBCA has completed, you will have a fully functional Oracle RAC cluster running!
$
$ORA_CRS_HOME/bin/crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....SM1.asm application ONLINE ONLINE linux1
ora....X1.lsnr application ONLINE ONLINE linux1
ora.linux1.gsd application ONLINE ONLINE linux1
ora.linux1.ons application ONLINE ONLINE linux1
ora.linux1.vip application ONLINE ONLINE linux1
ora....SM2.asm application ONLINE ONLINE linux2
ora....X2.lsnr application ONLINE ONLINE linux2
ora.linux2.gsd application ONLINE ONLINE linux2
ora.linux2.ons application ONLINE ONLINE linux2
ora.linux2.vip application ONLINE ONLINE linux2
ora.racdb.db application ONLINE ONLINE linux1
ora....b1.inst application ONLINE ONLINE linux1
ora....b2.inst application ONLINE ONLINE linux2
ora...._taf.cs application ONLINE ONLINE linux1
ora....db1.srv application ONLINE ONLINE linux1
ora....db2.srv application ONLINE ONLINE linux2
Verify the racdb_taf Service During the creation of the Oracle clustered database, you added a service named racdb_taf that will be used to connect to the database with TAF enabled. Use the following to verify the racdb_taf service was successfully added:
SQL>
show parameter service
NAME TYPE VALUE
-------------------- ----------- --------------------------------
service_names string racdb.idevelopment.info, racdb_taf
If the only service defined was for racdb.idevelopment.info, then you will need to manually add the service to both instances:
SQL>
show parameter service
NAME TYPE VALUE
-------------------- ----------- --------------------------
service_names string racdb.idevelopment.info
SQL>
alter system set service_names =
2
'racdb.idevelopment.info, racdb_taf' scope=both;
26. Post-Installation Tasks - (Optional) This chapter describes several optional tasks that can be applied to your new Oracle 10g environment in order to enhance availability as well as database management. Re-compile Invalid Objects Run the utlrp.sql script to recompile all invalid PL/SQL packages now instead of when the packages are accessed for the first time. This step is optional but recommended.
$
sqlplus / as sysdba
SQL>
@?/rdbms/admin/utlrp.sql
Enabling Archive Logs in a RAC Environment Whether a single instance or clustered database, Oracle tracks and logs all changes to database blocks in online redolog files. In an Oracle RAC environment, each instance will have its own set of online redolog files known as a thread. Each Oracle instance will use its group of online redologs in a circular manner. Once an online redolog fills, Oracle moves to the next one. If the database is in "Archive Log Mode", Oracle will make a copy of the online redo log before it gets reused. A thread must contain at least two online redologs (or online redolog groups). The same holds true for a single instance configuration. The single instance must contain at least two online redologs (or online redolog groups). The size of an online redolog file is completely independent of another intances' redolog size. Although in most configurations the size is the same, it may be different depending on the workload and backup / recovery considerations for each node. It is also worth mentioning that each instance has exclusive write access to its own online redolog files. In a correctly configured RAC environment, however, each instance can read another instance's current online redolog file to perform instance recovery if that instance was terminated abnormally. It is therefore a requirement that online redo logs be located on a shared storage device (just like the database files). As already mentioned, Oracle writes to its online redolog files in a circular manner. When the current online redolog fills, Oracle will switch to the next one. To facilitate media recovery, Oracle allows the DBA to put the database into "Archive Log Mode" which makes a copy of the online redolog after it fills (and before it gets reused). This is a process known as archiving. The Database Configuration Assistant (DBCA) allows users to configure a new database to be in archive log mode, however most DBA's opt to bypass this option during initial database creation. In cases like this where the database is in no archive log mode, it is a simple task to put the database into archive log mode. Note however that this will require a short database outage. From one of the nodes in the Oracle RAC configuration, use the following tasks to put a RAC enabled database into archive log mode. For the purpose of this article, I will use the node linux1 which runs the racdb1 instance:
After enabling Archive Log Mode, each instance in the RAC configuration can automatically archive redologs! Download and Install Custom Oracle Database Scripts DBA's rely on Oracle's data dictionary views and dynamic performance views in order to support and better manage their databases. Although these views provide a simple and easy mechanism to query critical information regarding the database, it helps to have a collection of accurate and readily available SQL scripts to query these views. In this section you will download and install a collection of Oracle DBA scripts that can be used to manage many aspects of your database including space management, performance, backups, security, and session management. The Oracle DBA scripts archive can be downloaded using the following link http://www.idevelopment.info/data/Oracle/DBA_scripts/dba_scripts_archive_Oracle.zip. As the oracle user account, download the common.zip archive to the $ORACLE_BASE directory of each node in the cluster. For the purpose of this example, the common.zip archive will be copied to /u01/app/oracle. Next, unzip the archive file to the $ORACLE_BASE directory. For example, perform the following on both nodes in the Oracle RAC cluster as the oracle user account:
$
mv common.zip /u01/app/oracle
$
cd /u01/app/oracle
$
unzip common.zip
The final step is to verify (or set) the appropriate environment variable for the current UNIX shell to ensure the Oracle SQL scripts can be run from SQL*Plus while in any directory. For UNIX verify the following environment variable is set and included in your login shell script:
ORACLE_PATH= $ORACLE_BASE/common/oracle/sql:.:$ORACLE_HOME/rdbms/admin
export ORACLE_PATH
Note: The ORACLE_PATH environment variable should already be set in the .bash_profile login script that was created in the section Create Login Script for oracle User Account. Now that the Oracle DBA scripts have been unzipped and the UNIX environment variable ( $ORACLE_PATH) has been set to the appropriate directory, you should now be able to run any of the SQL scripts in your $ORACLE_BASE/common/oracle/sql while logged into SQL*Plus. For example, to query tablespace information while logged into the Oracle database as a DBA user:
SQL>
@dba_tablespaces
Status Tablespace Name TS Type Ext. Mgt. Seg. Mgt. Tablespace Size Used (in bytes) Pct. Used
--------- --------------- ------------ ---------- --------- ------------------ ------------------ ---------
ONLINE UNDOTBS1 UNDO LOCAL MANUAL 1,283,457,024 9,043,968 1
ONLINE SYSAUX PERMANENT LOCAL AUTO 524,288,000 378,732,544 72
ONLINE USERS PERMANENT LOCAL AUTO 2,147,483,648 321,257,472 15
ONLINE SYSTEM PERMANENT LOCAL MANUAL 838,860,800 505,544,704 60
ONLINE INDX PERMANENT LOCAL AUTO 1,073,741,824 65,536 0
ONLINE UNDOTBS2 UNDO LOCAL MANUAL 1,283,457,024 22,282,240 2
ONLINE TEMP TEMPORARY LOCAL MANUAL 1,073,741,824 92,274,688 9
------------------ ------------------ ---------
avg 23
sum 8,225,030,144 1,329,201,152
7 rows selected.
To obtain a list of all available Oracle DBA scripts while logged into SQL*Plus, run the help.sql script:
SQL>
@help.sql
========================================
Automatic Shared Memory Management
========================================
asmm_components.sql
========================================
Automatic Storage Management
========================================
asm_alias.sql
asm_clients.sql
asm_diskgroups.sql
asm_disks.sql
asm_disks_perf.sql
asm_drop_files.sql
asm_files.sql
asm_files2.sql
asm_templates.sql
< --- SNIP --- >
perf_top_sql_by_buffer_gets.sql
perf_top_sql_by_disk_reads.sql
========================================
Workspace Manager
========================================
wm_create_workspace.sql
wm_disable_versioning.sql
wm_enable_versioning.sql
wm_freeze_workspace.sql
wm_get_workspace.sql
wm_goto_workspace.sql
wm_merge_workspace.sql
wm_refresh_workspace.sql
wm_remove_workspace.sql
wm_unfreeze_workspace.sql
wm_workspaces.sql
Create Shared Oracle Password Files In this section, I present the steps required to configure a shared Oracle password file between all instances in the Oracle clustered database. The password file for the database in UNIX is located at $ORACLE_HOME/dbs/orapw<ORACLE_SID> for each instance and contains a list of all database users that have SYSDBA privileges. When a database user is granted the SYSDBA role, the instance records this in the database password file for the instance you are logged into. But what about the other instances in the cluster? The database password file on other instances do not get updated and will not contain the user who was just granted the SYSDBA role. Therefore a program (like RMAN) that tries to login as this new user with SYSDBA privileges will fail if it tries to use an instance with a password file that does not contain his or her name. To resolve this problem, a common solution is to place a single database password file on a shared / clustered file system and then create symbolic links from each of the instances to this single version of the database password file. Since the environment described in this article makes use of the Oracle Clustered File System (OCFS2), we will use it to store the single version of the database password file. In this section, we will also be including the Oracle password file for the ASM instance.
Now, when a user is granted the SYSDBA role, all instances will have access to the same password file:
SQL>
GRANT sysdba TO scott;
27. Verify TNS Networking Files Ensure that the TNS networking files are configured on both Oracle RAC nodes in the cluster! listener.ora We already covered how to create a TNS listener configuration file ( listener.ora) for a clustered environment in Section 24. The listener.ora file should be properly configured and no modifications should be needed. For clarity, I have included a copy of the listener.ora file from my node linux1 in this guide support files. I've also included a copy of my tnsnames.ora file that was configured by Oracle and can be used for testing the Transparent Application Failover (TAF). This file should already be configured on both Oracle RAC nodes in the cluster. You can include any of these entries on other client machines that need access to the clustered database. Connecting to Clustered Database From an External Client This is an optional step, but I like to perform it in order to verify my TNS files are configured correctly. Use another machine (i.e. a Windows machine connected to the network) that has Oracle installed and add the TNS entries (in the tnsnames.ora) from either of the nodes in the cluster that were created for the clustered database. Note: Verify that the machine you are connecting from can resolve all host names exactly how they appear in the listener.ora and tnsnames.ora files. For the purpose of this document, the machine you are connecting from should be able to resolve the following host names in the local hosts file or through DNS: 192.168.1.100 linux1 192.168.1.101 linux2 192.168.1.200 linux1-vip 192.168.1.201 linux2-vip Try to connect to the clustered database using all available service names defined in the tnsnames.ora file:
C:\>
sqlplus system/manager@racdb2
C:\>
sqlplus system/manager@racdb1
C:\>
sqlplus system/manager@racdb_taf
C:\>
sqlplus system/manager@racdb
28. Create / Alter Tablespaces When creating the clustered database, we left all tablespaces set to their default size. If you are using a large drive for the shared storage, you may want to make a sizable testing database. Below are several optional SQL commands for modifying and creating all tablespaces for the test database. Please keep in mind that the database file names (OMF files) used in this example may differ from what the Oracle Database Configuration Assistant (DBCA) creates for your environment. When working through this section, substitute the data file names that were created in your environment where appropriate. The following query can be used to determine the file names for your environment:
SQL>
select tablespace_name, file_name
2
from dba_data_files
3
union
4
select tablespace_name, file_name
5
from dba_temp_files;
TABLESPACE_NAME FILE_NAME
--------------- --------------------------------------------------
EXAMPLE +RACDB_DATA1/racdb/datafile/example.257.570913311
INDX +RACDB_DATA1/racdb/datafile/indx.270.570920045
SYSAUX +RACDB_DATA1/racdb/datafile/sysaux.260.570913287
SYSTEM +RACDB_DATA1/racdb/datafile/system.262.570913215
TEMP +RACDB_DATA1/racdb/tempfile/temp.258.570913303
UNDOTBS1 +RACDB_DATA1/racdb/datafile/undotbs1.261.570913263
UNDOTBS2 +RACDB_DATA1/racdb/datafile/undotbs2.265.570913331
USERS +RACDB_DATA1/racdb/datafile/users.264.570913355
$
sqlplus / as sysdba
SQL>
create user scott identified by tiger default tablespace users;
SQL>
grant dba, resource, connect to scott;
SQL>
alter database datafile '+RACDB_DATA1/racdb/datafile/users.264.570913355' resize 1024m;
SQL>
alter tablespace users add datafile '+RACDB_DATA1' size 1024m autoextend off;
SQL>
create tablespace indx datafile '+RACDB_DATA1' size 1024m
2
autoextend on next 50m maxsize unlimited
3
extent management local autoallocate
4
segment space management auto;
SQL>
alter database datafile '+RACDB_DATA1/racdb/datafile/system.262.570913215' resize 800m;
SQL>
alter database datafile '+RACDB_DATA1/racdb/datafile/sysaux.260.570913287' resize 500m;
SQL>
alter tablespace undotbs1 add datafile '+RACDB_DATA1' size 1024m
2
autoextend on next 50m maxsize 2048m;
SQL>
alter tablespace undotbs2 add datafile '+RACDB_DATA1' size 1024m
2
autoextend on next 50m maxsize 2048m;
SQL>
alter database tempfile '+RACDB_DATA1/racdb/tempfile/temp.258.570913303' resize 1024m;
Here is a snapshot of the tablespaces I have defined for my test database environment:
Status Tablespace Name TS Type Ext. Mgt. Seg. Mgt. Tablespace Size Used (in bytes) Pct. Used
--------- --------------- ------------ ---------- --------- ------------------ ------------------ ---------
ONLINE UNDOTBS1 UNDO LOCAL MANUAL 1,283,457,024 85,065,728 7
ONLINE SYSAUX PERMANENT LOCAL AUTO 524,288,000 275,906,560 53
ONLINE USERS PERMANENT LOCAL AUTO 2,147,483,648 131,072 0
ONLINE SYSTEM PERMANENT LOCAL MANUAL 838,860,800 500,301,824 60
ONLINE EXAMPLE PERMANENT LOCAL AUTO 157,286,400 83,820,544 53
ONLINE INDX PERMANENT LOCAL AUTO 1,073,741,824 65,536 0
ONLINE UNDOTBS2 UNDO LOCAL MANUAL 1,283,457,024 3,801,088 0
ONLINE TEMP TEMPORARY LOCAL MANUAL 1,073,741,824 27,262,976 3
------------------ ------------------ ---------
avg 22
sum 8,382,316,544 976,355,328
8 rows selected.
29. Verify the RAC Cluster & Database Configuration The following RAC verification checks should be performed on both Oracle RAC nodes in the cluster! For this article, however, I will only be performing checks from linux1. This section provides several srvctl commands and SQL queries you can use to validate your Oracle RAC 10g configuration. There are five node-level tasks defined for SRVCTL:
Status of all instances and services
$
srvctl status database -d racdb
Instance racdb1 is running on node linux1
Instance racdb2 is running on node linux2
Status of a single instance
$
srvctl status instance -d racdb -i racdb2
Instance racdb2 is running on node linux2
Status of a named service globally across the database
$
srvctl status service -d racdb -s racdb_taf
Service racdb_taf is running on instance(s) racdb1, racdb2
Status of node applications on a particular node
$
srvctl status nodeapps -n linux1
VIP is running on node: linux1
GSD is running on node: linux1
Listener is running on node: linux1
ONS daemon is running on node: linux1
Status of an ASM instance
$
srvctl status asm -n linux1
ASM instance +ASM1 is running on node linux1.
List all configured databases
$
srvctl config database
racdb
Display configuration for our RAC database
$
srvctl config database -d racdb
linux1 racdb1 /u01/app/oracle/product/10.2.0/db_1
linux2 racdb2 /u01/app/oracle/product/10.2.0/db_1
Display all services for the specified cluster database
$
srvctl config service -d racdb
racdb_taf PREF: racdb1 racdb2 AVAIL:
Display the configuration for node applications - (VIP, GSD, ONS, Listener)
$
srvctl config nodeapps -n linux1 -a -g -s -l
VIP exists.: /linux1-vip/192.168.1.200/255.255.255.0/eth0
GSD exists.
ONS daemon exists.
Listener exists.
Display the configuration for the ASM instance(s)
$
srvctl config asm -n linux1
+ASM1 /u01/app/oracle/product/10.2.0/db_1
All running instances in the cluster
SELECT inst_id , instance_number inst_no , instance_name inst_name , parallel , status , database_status db_status , active_state state , host_name host FROM gv$instance ORDER BY inst_id;
INST_ID INST_NO INST_NAME PAR STATUS DB_STATUS STATE HOST
-------- -------- ---------- --- ------- ------------ --------- -------
1 1 racdb1 YES OPEN ACTIVE NORMAL linux1
2 2 racdb2 YES OPEN ACTIVE NORMAL linux2
All data files which are in the disk group
select name from v$datafile union select member from v$logfile union select name from v$controlfile union select name from v$tempfile;
NAME
-------------------------------------------
+FLASH_RECOVERY_AREA/racdb/controlfile/current.258.570913191
+FLASH_RECOVERY_AREA/racdb/onlinelog/group_1.257.570913201
+FLASH_RECOVERY_AREA/racdb/onlinelog/group_2.256.570913211
+FLASH_RECOVERY_AREA/racdb/onlinelog/group_3.259.570918285
+FLASH_RECOVERY_AREA/racdb/onlinelog/group_4.260.570918295
+RACDB_DATA1/racdb/controlfile/current.259.570913189
+RACDB_DATA1/racdb/datafile/example.257.570913311
+RACDB_DATA1/racdb/datafile/indx.270.570920045
+RACDB_DATA1/racdb/datafile/sysaux.260.570913287
+RACDB_DATA1/racdb/datafile/system.262.570913215
+RACDB_DATA1/racdb/datafile/undotbs1.261.570913263
+RACDB_DATA1/racdb/datafile/undotbs1.271.570920865
+RACDB_DATA1/racdb/datafile/undotbs2.265.570913331
+RACDB_DATA1/racdb/datafile/undotbs2.272.570921065
+RACDB_DATA1/racdb/datafile/users.264.570913355
+RACDB_DATA1/racdb/datafile/users.269.570919829
+RACDB_DATA1/racdb/onlinelog/group_1.256.570913195
+RACDB_DATA1/racdb/onlinelog/group_2.263.570913205
+RACDB_DATA1/racdb/onlinelog/group_3.266.570918279
+RACDB_DATA1/racdb/onlinelog/group_4.267.570918289
+RACDB_DATA1/racdb/tempfile/temp.258.570913303
21 rows selected.
All ASM disk that belong to the 'RACDB_DATA1' disk group
SELECT path FROM v$asm_disk WHERE group_number IN (select group_number from v$asm_diskgroup where name = 'RACDB_DATA1');
PATH
----------------------------------
ORCL:VOL1
ORCL:VOL2
30. Starting / Stopping the Cluster At this point, we've installed and configured Oracle RAC 10g entirely and have a fully functional clustered database. After all the work done up to this point, you may well ask, "OK, so how do I start and stop services?". If you have followed the instructions in this guide, all services—including Oracle Clusterware, all Oracle instances, Enterprise Manager Database Console, and so on—should start automatically on each reboot of the Linux nodes. There are times, however, when you might want to shut down a node and manually start it back up. Or you may find that Enterprise Manager is not running and need to start it. This section provides the commands responsible for starting and stopping the cluster environment. Ensure that you are logged in as the oracle UNIX user. We will run all commands in this section from linux1:
#
su - oracle
$
hostname
linux1
Stopping the Oracle RAC 10g Environment The first step is to stop the Oracle instance. When the instance (and related services) is down, then bring down the ASM instance. Finally, shut down the node applications (Virtual IP, GSD, TNS Listener, and ONS).
$
export ORACLE_SID=racdb1
$
emctl stop dbconsole
$
srvctl stop instance -d racdb -i racdb1
$
srvctl stop asm -n linux1
$
srvctl stop nodeapps -n linux1
Starting the Oracle RAC 10g Environment The first step is to start the node applications (Virtual IP, GSD, TNS Listener, and ONS). When the node applications are successfully started, then bring up the ASM instance. Finally, bring up the Oracle instance (and related services) and the Enterprise Manager Database console.
$
export ORACLE_SID=racdb1
$
srvctl start nodeapps -n linux1
$
srvctl start asm -n linux1
$
srvctl start instance -d racdb -i racdb1
$
emctl start dbconsole
Start/Stop All Instances with SRVCTL Start/stop all the instances and their enabled services. I have included this step just for fun as a way to bring down all instances!
$
srvctl start database -d racdb
$
srvctl stop database -d racdb
31. Transparent Application Failover (TAF) It is not uncommon for businesses to demand 99.99% (or even 99.999%) availability for their enterprise applications. Think about what it would take to ensure a downtime of no more than .5 hours or even no downtime during the year. To answer many of these high-availability requirements, businesses are investing in mechanisms that provide for automatic failover when one participating system fails. When considering the availability of the Oracle database, Oracle RAC 10g provides a superior solution with its advanced failover mechanisms. Oracle RAC 10g includes the required components that all work within a clustered configuration responsible for providing continuous availability; when one of the participating systems fail within the cluster, the users are automatically migrated to the other available systems. A major component of Oracle RAC 10g that is responsible for failover processing is the Transparent Application Failover (TAF) option. All database connections (and processes) that lose connections are reconnected to another node within the cluster. The failover is completely transparent to the user. This final section provides a short demonstration on how TAF works in Oracle RAC 10g. Please note that a complete discussion of failover in Oracle RAC 10g would require an article in itself; my intention here is to present only a brief overview. One important note is that TAF happens automatically within the OCI libraries. Thus your application (client) code does not need to change in order to take advantage of TAF. Certain configuration steps, however, will need to be done on the Oracle TNS file tnsnames.ora. (Keep in mind that as of this writing, the Java thin client will not be able to participate in TAF because it never reads tnsnames.ora.) Setup the tnsnames.ora File Before demonstrating TAF, we need to verify that a valid entry exists in the tnsnames.ora file on a non-RAC client machine (if you have a Windows machine lying around). Ensure that you have the Oracle RDBMS software installed. (Actually, you only need a client install of the Oracle software.) During the creation of the clustered database in this guide, we created a new service that will be used for testing TAF named RACDB_TAF. It provides all the necessary configuration parameters for load balancing and failover. You can copy the contents of this entry to the %ORACLE_HOME%\network\admin\tnsnames.ora file on the client machine (my Windows laptop is being used in this example) in order to connect to the new Oracle clustered database:
...
RACDB_TAF =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = linux1-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = linux2-vip)(PORT = 1521))
(LOAD_BALANCE = yes)
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = racdb_taf.idevelopment.info)
(FAILOVER_MODE =
(TYPE = SELECT)
(METHOD = BASIC)
(RETRIES = 180)
(DELAY = 5)
)
)
)
...
SQL Query to Check the Session's Failover Information
The following SQL query can be used to check a session's failover type, failover method, and if a failover has occurred. We will be using this query throughout this example.
COLUMN instance_name FORMAT a13
COLUMN host_name FORMAT a9
COLUMN failover_method FORMAT a15
COLUMN failed_over FORMAT a11
SELECT
instance_name
, host_name
, NULL AS failover_type
, NULL AS failover_method
, NULL AS failed_over
FROM v$instance
UNION
SELECT
NULL
, NULL
, failover_type
, failover_method
, failed_over
FROM v$session
WHERE username = 'SYSTEM';
TAF Demo From a Windows machine (or other non-RAC client machine), login to the clustered database using the racdb_taf service as the SYSTEM user:
C:\> sqlplus system/manager@racdb_taf
COLUMN instance_name FORMAT a13
COLUMN host_name FORMAT a9
COLUMN failover_method FORMAT a15
COLUMN failed_over FORMAT a11
SELECT
instance_name
, host_name
, NULL AS failover_type
, NULL AS failover_method
, NULL AS failed_over
FROM v$instance
UNION
SELECT
NULL
, NULL
, failover_type
, failover_method
, failed_over
FROM v$session
WHERE username = 'SYSTEM';
INSTANCE_NAME HOST_NAME FAILOVER_TYPE FAILOVER_METHOD FAILED_OVER
------------- --------- ------------- --------------- -----------
racdb1 linux1
SELECT BASIC NO
DO NOT log out of the above SQL*Plus session! Now that we have run the query (above), we should now shutdown the instance racdb1 on linux1 using the abort option. To perform this operation, you can use the srvctl command-line utility as follows: # su - oracle $ srvctl status database -d racdb Instance racdb1 is running on node linux1 Instance racdb2 is running on node linux2 $ srvctl stop instance -d racdb -i racdb1 -o abort $ srvctl status database -d racdb Instance racdb1 is not running on node linux1 Instance racdb2 is running on node linux2Now let's go back to our SQL session and rerun the SQL statement in the buffer:
COLUMN instance_name FORMAT a13
COLUMN host_name FORMAT a9
COLUMN failover_method FORMAT a15
COLUMN failed_over FORMAT a11
SELECT
instance_name
, host_name
, NULL AS failover_type
, NULL AS failover_method
, NULL AS failed_over
FROM v$instance
UNION
SELECT
NULL
, NULL
, failover_type
, failover_method
, failed_over
FROM v$session
WHERE username = 'SYSTEM';
INSTANCE_NAME HOST_NAME FAILOVER_TYPE FAILOVER_METHOD FAILED_OVER
------------- --------- ------------- --------------- -----------
racdb2 linux2
SELECT BASIC YES
SQL> exit
From the above demonstration, we can see that the above session has now been failed over to instance racdb2 on linux2.
32. Troubleshooting Confirm the RAC Node Name is Not Listed in Loopback Address
Confirm localhost is defined in the /etc/hosts file for the loopback address
Setting the Correct Date and Time on All Cluster Nodes
Openfiler - Logical Volumes Not Active on Boot One issue that I have run into several times occurs when using a USB drive connected to the Openfiler server. When the Openfiler server is rebooted, the system is able to recognize the USB drive however, it is not able to load the logical volumes and writes the following message to /var/log/messages - (also available through dmesg):iSCSI Enterprise Target Software - version 0.4.14 iotype_init(91) register fileio iotype_init(91) register blockio iotype_init(91) register nullio open_path(120) Can't open /dev/rac1/crs -2 fileio_attach(268) -2 open_path(120) Can't open /dev/rac1/asm1 -2 fileio_attach(268) -2 open_path(120) Can't open /dev/rac1/asm2 -2 fileio_attach(268) -2 open_path(120) Can't open /dev/rac1/asm3 -2 fileio_attach(268) -2 open_path(120) Can't open /dev/rac1/asm4 -2 fileio_attach(268) -2 OCFS2 - o2cb_ctl: Unable to access cluster service while creating node
OCFS2 - Adjusting the O2CB Heartbeat Threshold With previous versions of this article, (using FireWire as opposed to iSCSI for the shared storage), I was able to install and configure OCFS2, format the new volume, and finally install Oracle Clusterware (with its two required shared files; the voting disk and OCR file), located on the new OCFS2 volume. While I was able to install Oracle Clusterware and see the shared drive using FireWire, however, I was receiving many lock-ups and hanging after about 15 minutes when the Clusterware software was running on both nodes. It always varied on which node would hang (either linux1 or linux2 in my example). It also didn't matter whether there was a high I/O load or none at all for it to crash (hang). Oracle Clusterware Installation: Running root.sh Fails on the Last Node
33. Conclusion Oracle10g RAC allows the DBA to configure a database solution with superior fault tolerance and load balancing. For those DBAs, however, that want to become more familiar with the features and benefits of Oracle10g RAC will find the costs of configuring even a small RAC cluster costing in the range of US$15,000 to US$20,000. This article has hopefully given you an economical solution to setting up and configuring an inexpensive Oracle10g Release 2 RAC Cluster using Oracle Enterprise Linux and iSCSI technology. The RAC solution presented in this article can be put together for around US$2,700 and will provide the DBA with a fully functional Oracle10g Release 2 RAC cluster. While the hardware used for this article should be stable enough for educational purposes, it should never be considered for a production environment.
34. Acknowledgements An article of this magnitude and complexity is generally not the work of one person alone. Although I was able to author and successfully demonstrate the validity of the components that make up this configuration, there are several other individuals that deserve credit in making this article a success. First, I would like to thank Bane Radulovic from the Server BDE Team at Oracle. Bane not only introduced me to Openfiler, but shared with me his experience and knowledge of the product and how to best utilize it for Oracle RAC. His research and hard work made the task of configuring Openfiler seamless. Bane was also involved with hardware recommendations and testing. A special thanks to K Gopalakrishnan for his assistance in delivering the Oracle RAC 10g Overview section of this article. In this section, much of the content regarding the history of Oracle RAC can be found in his very popular book Oracle Database 10g Real Application Clusters Handbook . This book comes highly recommended for both DBAs and Developers wanting to successfully implement Oracle RAC and fully understand how many of the advanced services like Cache Fusion and Global Resource Directory operate. I would next like to thank Oracle ACE Werner Puschitz for his outstanding work on "Installing Oracle Database 10g with Real Application Cluster (RAC) on Red Hat Enterprise Linux Advanced Server 3". This article, along with several others of his, provided information on Oracle RAC that could not be found in any other Oracle documentation. Without his hard work and research into issues like configuring OCFS2 and ASMLib, this article may have never come to fruition. If you are interested in examining technical articles on Linux internals and in-depth Oracle configurations written by Werner Puschitz, please visit his website at www.puschitz.com. Also, thanks to Tzvika Lemel for his comments and suggestions on using Oracle's Cluster Verification Utility (CVU). Lastly, I would like to express my appreciation to the following vendors for generously supplying the hardware for this article; Stallard Technologies, Inc., Seagate, Avocent Corporation, Intel, D-Link, SIIG, and LaCie.
Jeffrey Hunter [ www.idevelopment.info] graduated from Stanislaus State University in Turlock, California, with a Bachelor's degree in Computer Science and has been a senior DBA and software engineer for over 16 years. He is an Oracle Certified Professional, Java Development Certified Professional, an Oracle ACE, and author, and currently works as a Senior Database Administrator for The DBA Zone, Inc. Jeff's work includes advanced performance tuning, Java and PL/SQL programming, capacity planning, database security, and physical / logical database design in a UNIX, Linux, and Windows NT environment. Jeff's other interests include mathematical encryption theory, programming language processors (compilers and interpreters) in Java and C, LDAP, writing web-based database administration tools, and of course Linux.
|
Printer View