What You See Is What You Get Element

Creating the Standby Database

Part VI of How to Upgrade to Oracle Solaris Cluster 4.0

by Tim Read

Published May 2012

Part I - Overview of the Example Configuration
Part II - Configuring the Oracle Database for Clustering
Part III - Installing the Target Cluster
Part IV - Creating the Zone Cluster
Part V - Installing the New Application Software Stack
Part VI - Creating the Standby Database
Part VII - Creating the Oracle Solaris Cluster Geographic Edition Configuration
Part VIII - How Oracle Solaris Cluster Geographic Edition Simplifies the Upgrade Process

Once the Oracle Database software has been installed on the target cluster and a logical host has been configured, set up an Oracle Database listener and begin the process of creating a standby database. The tnsnames.ora and sqlnet.ora files are identical to their equivalents on the original cluster. The listener.ora file differs only in the name of the host on which it listens (vzpyrus1a) and the ORACLE_SID it uses (salesdr).

Caution: If you choose to use this article as a guide for performing a similar process, you need to pay close attention to the nodes on which the individual commands are run. For that reason, the system prompts shown in the example steps include both the node name and the user name to indicate both where, and as whom, a command must be run.

First, create the listener.ora file if it does not already exist. Listing 1 shows an example listener.ora file.

vzpyrus3a (oracle) $ cd /u02/app/oracle/product/11.2.0/dbhome_1/network/admin
vzpyrus3a (oracle) $ cat listener.ora
# listener.ora Network Configuration File: 
/u02/app/oracle/product/11.2.0/dbhome_1/network/admin/listener.ora
# Generated by Oracle configuration tools.

SID_LIST_LISTENER =
  (SID_LIST =
    (SID_DESC =
      (GLOBAL_DBNAME = salesdr_DGMGRL)
      (ORACLE_HOME = /u02/app/oracle/product/11.2.0/dbhome_1)
      (SID_NAME = salesdr)
    )
  )

LISTENER =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = vzpyrus1a)(PORT = 1521))
    )
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))
    )
  )

ADR_BASE_LISTENER = /u02/app/oracle

Listing 1. Example listener.ora File

After you create the listener.ora file, start the listener process manually, as shown in Listing 2.

vzpyrus3a (oracle) $ export ORACLE_HOME=/u02/app/oracle/product/11.2.0/dbhome_1
vzpyrus3a (oracle) $ export PATH=$PATH:${ORACLE_HOME}/bin 
vzpyrus3a (oracle) $ lsnrctl start

LSNRCTL for Solaris: Version 11.2.0.3.0 - Production on 18-JAN-2012 03:06:19

Copyright (c) 1991, 2011, Oracle.  All rights reserved.

Starting /u02/app/oracle/product/11.2.0/dbhome_1//bin/tnslsnr: please wait...

TNSLSNR for Solaris: Version 11.2.0.3.0 - Production
System parameter file is /u02/app/oracle/product/11.2.0/dbhome_1/network/admin/listener.ora
Log messages written to /u02/app/oracle/diag/tnslsnr/vzpyrus3a/listener/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.134.108.111)(PORT=1521)))
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=vzpyrus1a)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Solaris: Version 11.2.0.3.0 - Production
Start Date                18-JAN-2012 03:06:20
Uptime                    0 days 0 hr. 0 min. 0 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u02/app/oracle/product/11.2.0/dbhome_1/network/admin/listener.ora
Listener Log File         /u02/app/oracle/diag/tnslsnr/vzpyrus3a/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.134.108.111)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
Services Summary...
Service "salesdr_DGMGRL" has 1 instance(s).
  Instance "salesdr", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully

Listing 2. Starting the Listener Process

You also need to create an Oracle wallet on the new cluster, as shown in Listing 3. Use credentials that use a different service name (salesdr) for the Oracle Solaris Cluster database probe that will check the health of the standby database when we promote it to the primary database.

vzpyrus3a (oracle) $ export ORACLE_HOME=/u02/app/oracle/product/11.2.0/dbhome_1
vzpyrus3a (oracle) $ export PATH=$PATH:${ORACLE_HOME}/bin
vzpyrus3a (oracle) $ mkstore -wrl /oradata/wallet -create Oracle Secret Store Tool : Version 11.2.0.3.0 - Production Copyright (c) 2004, 2011, Oracle and/or its affiliates. All rights reserved. Enter password: Enter password again: vzpyrus3a (oracle) $ mkstore -wrl /oradata/wallet -createCredential salesdr hamon Oracle Secret Store Tool : Version 11.2.0.3.0 - Production Copyright (c) 2004, 2011, Oracle and/or its affiliates. All rights reserved. Your secret/Password is missing in the command line Enter your secret/Password: Re-enter your secret/Password: Enter wallet password: Create credential oracle.security.client.connect_string1

Listing 3. Creating a Wallet

Next, use Oracle rman to take a backup of the database on the original cluster, as shown in Listing 4.

pgyruss1 (oracle) $ export ORACLE_HOME=/u02/app/oracle/product/11.2.0/dbhome_1
pgyruss1 (oracle) $ export PATH=$PATH:${ORACLE_HOME}/bin
pgyruss1 (oracle) $ export ORACLE_SID=sales
pgyruss1 (oracle) $ rman 

Recovery Manager: Release 11.2.0.3.0 - Production on Wed Jan 18 03:14:55 2012

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

RMAN> connect target sys/oracle@sales;

connected to target database: SALES (DBID=645630077)

RMAN> connect auxiliary /;

connected to auxiliary database: SALES (DBID=645630077)

RMAN> backup device type disk tag 'mybkup' database include current controlfile for standby;

Starting backup at 18-JAN-12
using channel ORA_DISK_1
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00001 name=/oradata/SALES/datafile/o1_mf_system_7k0sfhtw_.dbf
input datafile file number=00002 name=/oradata/SALES/datafile/o1_mf_sysaux_7k0sfhxg_.dbf
input datafile file number=00003 name=/oradata/SALES/datafile/o1_mf_undotbs1_7k0sfhxp_.dbf
input datafile file number=00004 name=/oradata/SALES/datafile/o1_mf_users_7k0sfj0f_.dbf
channel ORA_DISK_1: starting piece 1 at 18-JAN-12
channel ORA_DISK_1: finished piece 1 at 18-JAN-12
piece handle=/oradata/fast_recovery_area/SALES/backupset/2012_01_18/o1_mf_nnndf_MYBKUP_7kfbho4k_.bkp 
tag=MYBKUP comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:35
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including standby control file in backup set
including current SPFILE in backup set
channel ORA_DISK_1: starting piece 1 at 18-JAN-12
channel ORA_DISK_1: finished piece 1 at 18-JAN-12
piece handle=/oradata/fast_recovery_area/SALES/backupset/2012_01_18/o1_mf_ncsnf_MYBKUP_7kfbjshm_.bkp 
tag=MYBKUP comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 18-JAN-12

RMAN> backup device type disk tag 'mybkup' archivelog all not backed up;

Starting backup at 18-JAN-12
current log archived
using channel ORA_DISK_1
channel ORA_DISK_1: starting archived log backup set
channel ORA_DISK_1: specifying archived log(s) in backup set
input archived log thread=1 sequence=5 RECID=1 STAMP=772495231
input archived log thread=1 sequence=6 RECID=2 STAMP=772527653
input archived log thread=1 sequence=7 RECID=3 STAMP=772538549
input archived log thread=1 sequence=8 RECID=4 STAMP=772570848
input archived log thread=1 sequence=9 RECID=5 STAMP=772614042
input archived log thread=1 sequence=10 RECID=6 STAMP=772653634
input archived log thread=1 sequence=11 RECID=7 STAMP=772732808
input archived log thread=1 sequence=12 RECID=8 STAMP=772783210
input archived log thread=1 sequence=13 RECID=9 STAMP=772795868
input archived log thread=1 sequence=14 RECID=10 STAMP=772840877
input archived log thread=1 sequence=15 RECID=11 STAMP=772855395
input archived log thread=1 sequence=16 RECID=12 STAMP=772856892
input archived log thread=1 sequence=17 RECID=13 STAMP=772859064
input archived log thread=1 sequence=18 RECID=14 STAMP=772859071
input archived log thread=1 sequence=19 RECID=18 STAMP=772859451
input archived log thread=1 sequence=20 RECID=20 STAMP=772859819
channel ORA_DISK_1: starting piece 1 at 18-JAN-12
channel ORA_DISK_1: finished piece 1 at 18-JAN-12
piece handle=/oradata/fast_recovery_area/SALES/backupset/2012_01_18/o1_mf_annnn_MYBKUP_7kfbkdcy_.bkp 
tag=MYBKUP comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:15
Finished backup at 18-JAN-12

RMAN> quit
Recovery Manager complete.

Listing 4. Making a Backup

When this process is complete, make the equivalent backup directory on the target cluster and copy the backup files over in preparation for the restore phase, as shown in Listing 5. Although several methods are available, this approach is shown primarily because of its simplicity.

vzpyrus3a (oracle) $ mkdir -p /oradata/fast_recovery_area/SALES/backupset/2012_01_18
vzpyrus3a (oracle) $ cd /oradata/fast_recovery_area/SALES/backupset/2012_01_18
vzpyrus3a (oracle) $ scp oracle@vzgyruss1b:/oradata/fast_recovery_area/SALES/backupset/2012_01_18/\* .
Password:
o1_mf_annnn_MYBKUP_7 100% |********************************************|   380 MB    00:18    
o1_mf_ncsnf_MYBKUP_7 100% |********************************************|  9600 KB    00:00    
o1_mf_nnndf_MYBKUP_7 100% |********************************************|  1071 MB    00:52

Listing 5. Copying the Backup Files

Before you take a copy of the initialization parameters for the original database, set the parameters necessary to enable the log transfer to take place, as shown in Listing 6.

pgyruss1 (oracle) $ sqlplus /NOLOG

SQL*Plus: Release 11.2.0.3.0 Production on Tue Jan 17 09:34:57 2012

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

SQL> connect sys/oracle as sysdba
Connected.

SQL> alter system set log_archive_dest_1='location=use_db_recovery_file_dest \
valid_for=(all_logfiles,all_roles) db_unique_name=sales' scope=both;
System altered. SQL> alter system set log_archive_dest_2='service=salesdr \
valid_for= (online_logfiles,primary_role) db_unique_name=salesdr' scope=both;
System altered. SQL> alter system set standby_file_management='AUTO' scope=both; System altered. SQL> alter system set fal_server='salesdr' scope=both; System altered. SQL> alter system set fal_client='sales' scope=both; System altered. SQL> create pfile='/tmp/inittmp.ora' from spfile; File created. SQL> Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options

Listing 6. Setting Parameters

Copy the temporary initialization file to the new cluster:

vzpyrus3a (oracle) $ scp oracle@vzgyruss1b:/tmp/inittmp.ora /tmp/inittmp.ora
Password: 
inittmp.ora          100% |********************************************|  1246 00:00  
vzpyrus3a (oracle) $

Now, edit the temporary initialization file to make it suitable for the new database, as shown in Listing 7.

vzpyrus3a (oracle) $ cat /tmp/inittmp.ora
salesdr.__db_cache_size=855638016
salesdr.__java_pool_size=16777216
salesdr.__large_pool_size=16777216
salesdr.__oracle_base='/u02/app/oracle'#ORACLE_BASE set from environment
salesdr.__pga_aggregate_target=872415232
salesdr.__sga_target=1275068416
salesdr.__shared_io_pool_size=0
salesdr.__shared_pool_size=369098752
salesdr.__streams_pool_size=0
*.audit_file_dest='/u02/app/oracle/admin/salesdr/adump'
*.audit_trail='db'
*.compatible='11.2.0.0.0'
*.db_block_size=8192
*.db_create_file_dest='/oradata'
*.db_domain=''
*.db_name='sales'
*.db_unique_name='salesdr'
*.db_recovery_file_dest='/oradata/fast_recovery_area'
*.db_recovery_file_dest_size=4322230272
*.db_unique_name='salesdr'
*.diagnostic_dest='/u02/app/oracle'
*.dispatchers='(PROTOCOL=TCP) (SERVICE=salesdrXDB)'
*.fal_client='salesdr'
*.fal_server='sales'
*.local_listener='LISTENER_SALESDR'
*.log_archive_dest_1='location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles) 
db_unique_name=salesdr'
*.log_archive_dest_2='service=sales valid_for=(online_logfiles,primary_role) 
db_unique_name=sales'
*.log_archive_format='%t_%s_%r.dbf'
*.memory_target=2147483648
*.open_cursors=300
*.processes=150
*.remote_login_passwordfile='EXCLUSIVE'
*.standby_file_management='AUTO'
*.undo_tablespace='UNDOTBS1'

Listing 7. Editing Initialization Files

Create the audit directory on all the zone cluster nodes, and then start up the database in nomount mode, as shown in Listing 8.

vzpyrus3a (oracle) $ mkdir -p /u02/app/oracle/admin/salesdr/adump

vzpyrus3b (oracle) $ mkdir -p /u02/app/oracle/admin/salesdr/adump

vzpyrus3a (oracle) $ export ORACLE_HOME=/u02/app/oracle/product/11.2.0/dbhome_1
vzpyrus3a (oracle) $ cd /u02/app/oracle/product/11.2.0/dbhome_1/dbs
vzpyrus3a (oracle) $ scp oracle@vzgyruss1b:/u02/app/oracle/product/11.2.0/dbhome_1/dbs/orapwsales \
orapwsalesdr
Password: orapwsales 100% |********************************************| 1536 00:00 vzpyrus3a (oracle) $ vzpyrus3a (oracle) $ export PATH=$PATH:${ORACLE_HOME}/bin vzpyrus3a (oracle) $ export ORACLE_SID=salesdr vzpyrus3a (oracle) $ sqlplus /NOLOG SQL*Plus: Release 11.2.0.3.0 Production on Wed Jan 18 02:05:33 2012 Copyright (c) 1982, 2011, Oracle. All rights reserved. SQL> connect sys/oracle as sysdba Connected to an idle instance. SQL> create spfile from pfile='/tmp/inittmp.ora'; File created. SQL> ! vzpyrus3a (oracle) $ ls -l $ORACLE_HOME/dbs total 15 -rw-r--r-- 1 oracle oinstall 2851 May 15 2009 init.ora -rw-r----- 1 oracle oinstall 1536 Jan 18 02:46 orapwsalesdr -rw-r----- 1 oracle oinstall 3584 Jan 18 05:08 spfilesalesdr.ora $ exit SQL> startup nomount ORACLE instance started. Total System Global Area 2138521600 bytes Fixed Size 2161024 bytes Variable Size 1275070080 bytes Database Buffers 855638016 bytes Redo Buffers 5652480 bytes SQL> Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options

Listing 8. Starting the Database

Next, run rman to restore the database, as shown in Listing 9.

vzpyrus3a (oracle) $ rman

Recovery Manager: Release 11.2.0.3.0 - Production on Wed Jan 18 05:09:25 2012

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

RMAN> connect target sys/oracle@sales;

connected to target database: SALES (DBID=645630077)

RMAN> connect auxiliary /;

connected to auxiliary database: SALES (not mounted)

RMAN> duplicate target database for standby nofilenamecheck;

Starting Duplicate Db at 18-JAN-12
using target database control file instead of recovery catalog
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: SID=134 device type=DISK

contents of Memory Script:
{
   sql clone "alter system set  control_files = 
  ''/oradata/SALESDR/controlfile/o1_mf_7kfk4wd8_.ctl'', 
  ''/oradata/fast_recovery_area/SALESDR/controlfile/o1_mf_7kfk4wdd_.ctl'' comment=
 ''Set by RMAN'' scope=spfile";
   restore clone standby controlfile;
}
executing Memory Script

sql statement: alter system set  control_files =   
''/oradata/SALESDR/controlfile/o1_mf_7kfk4wd8_.ctl'', 
''/oradata/fast_recovery_area/SALESDR/controlfile/o1_mf_7kfk4wdd_.ctl'' 
comment= ''Set by RMAN'' scope=spfile

Starting restore at 18-JAN-12
using channel ORA_AUX_DISK_1

channel ORA_AUX_DISK_1: starting datafile backup set restore
channel ORA_AUX_DISK_1: restoring control file
channel ORA_AUX_DISK_1: reading from backup piece 
/oradata/fast_recovery_area/SALES/backupset/2012_01_18/o1_mf_ncsnf_MYBKUP_7kfbjshm_.bkp
channel ORA_AUX_DISK_1: piece 
handle=/oradata/fast_recovery_area/SALES/backupset/2012_01_18/o1_mf_ncsnf_MYBKUP_7kfbjshm_.bkp tag=MYBKUP
channel ORA_AUX_DISK_1: restored backup piece 1
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:01
output file name=/oradata/SALESDR/controlfile/o1_mf_7kfk4wrx_.ctl
output file name=/oradata/fast_recovery_area/SALESDR/controlfile/o1_mf_7kfk4wx3_.ctl
Finished restore at 18-JAN-12

contents of Memory Script:
{
   sql clone 'alter database mount standby database';
}
executing Memory Script

sql statement: alter database mount standby database

contents of Memory Script:
{
   set newname for clone tempfile  1 to new;
   switch clone tempfile all;
   set newname for clone datafile  1 to new;
   set newname for clone datafile  2 to new;
   set newname for clone datafile  3 to new;
   set newname for clone datafile  4 to new;
   restore
   clone database
   ;
}
executing Memory Script

executing command: SET NEWNAME

renamed tempfile 1 to /oradata/SALESDR/datafile/o1_mf_temp_%u_.tmp in control file

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

Starting restore at 18-JAN-12
using channel ORA_AUX_DISK_1

channel ORA_AUX_DISK_1: starting datafile backup set restore
channel ORA_AUX_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_AUX_DISK_1: restoring datafile 00001 to /oradata/SALESDR/datafile/o1_mf_system_%u_.dbf
channel ORA_AUX_DISK_1: restoring datafile 00002 to /oradata/SALESDR/datafile/o1_mf_sysaux_%u_.dbf
channel ORA_AUX_DISK_1: restoring datafile 00003 to /oradata/SALESDR/datafile/o1_mf_undotbs1_%u_.dbf
channel ORA_AUX_DISK_1: restoring datafile 00004 to /oradata/SALESDR/datafile/o1_mf_users_%u_.dbf
channel ORA_AUX_DISK_1: reading from backup piece 
/oradata/fast_recovery_area/SALES/backupset/2012_01_18/o1_mf_nnndf_MYBKUP_7kfbho4k_.bkp
channel ORA_AUX_DISK_1: piece 
handle=/oradata/fast_recovery_area/SALES/backupset/2012_01_18/o1_mf_nnndf_MYBKUP_7kfbho4k_.bkp tag=MYBKUP
channel ORA_AUX_DISK_1: restored backup piece 1
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:25
Finished restore at 18-JAN-12

contents of Memory Script:
{
   switch clone datafile all;
}
executing Memory Script

datafile 1 switched to datafile copy
input datafile copy RECID=5 STAMP=772866621 file 
name=/oradata/SALESDR/datafile/o1_mf_system_7kfk546q_.dbf
datafile 2 switched to datafile copy
input datafile copy RECID=6 STAMP=772866621 file 
name=/oradata/SALESDR/datafile/o1_mf_sysaux_7kfk546y_.dbf
datafile 3 switched to datafile copy
input datafile copy RECID=7 STAMP=772866621 file 
name=/oradata/SALESDR/datafile/o1_mf_undotbs1_7kfk5476_.dbf
datafile 4 switched to datafile copy
input datafile copy RECID=8 STAMP=772866621 file 
name=/oradata/SALESDR/datafile/o1_mf_users_7kfk547d_.dbf
Finished Duplicate Db at 18-JAN-12

RMAN> quit
Recovery Manager complete.

Listing 9. Restoring the Database

When the restore process is complete, add standby log files to the new database, check that the recovery process is working, and enable flashback. Then, start the database recovery process, as shown in Listing 10.

vzpyrus3a (oracle) $ sqlplus /NOLOG

SQL*Plus: Release 11.2.0.3.0 Production on Wed Jan 18 05:13:18 2012

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

SQL> connect sys/oracle as sysdba
Connected.
SQL> alter database add standby logfile size 50m;

Database altered.

SQL> alter database add standby logfile size 50m;

Database altered.

SQL> alter database add standby logfile size 50m;

Database altered.

SQL> alter database recover managed standby database using current logfile disconnect ;

Database altered.

SQL> shutdown immediate ;
ORA-01109: database not open


Database dismounted.
ORACLE instance shut down.
SQL> startup mount;
ORACLE instance started.

Total System Global Area 2138521600 bytes
Fixed Size		    2161024 bytes
Variable Size		 1275070080 bytes
Database Buffers	  855638016 bytes
Redo Buffers		    5652480 bytes
Database mounted.
SQL> alter database flashback on;

Database altered.

SQL> alter database recover managed standby database using current logfile disconnect;

Database altered.

SQL> quit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

Listing 10. Starting the Database Recovery

Determine whether the recovery process is working correctly by performing a log switch on the original database and then checking whether the log files are recovered on the new instance by looking in the alert log, as shown in Listing 11.

pgyruss1 (oracle) $ sqlplus /NOLOG

SQL*Plus: Release 11.2.0.3.0 Production on Wed Jan 18 05:22:57 2012

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

SQL> connect sys/oracle as sysdba
Connected.
SQL> alter system switch logfile ;

System altered.

SQL> quit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

vzpyrus3a (oracle) $ tail -f alert_salesdr.log 
Primary database is in MAXIMUM PERFORMANCE mode
RFS[2]: Assigned to RFS process 11124
RFS[2]: Selected log 4 for thread 1 sequence 24 dbid 645630077 branch 772449215
Wed Jan 18 05:26:50 2012
Archived Log entry 4 added for thread 1 sequence 23 ID 0x267ae67d dest 1:
Wed Jan 18 05:26:50 2012
Media Recovery Waiting for thread 1 sequence 24 (in transit)
Recovery of Online Redo Log: Thread 1 Group 4 Seq 24 Reading mem 0
  Mem# 0: /oradata/SALESDR/onlinelog/o1_mf_4_7kfk61h2_.log
  Mem# 1: /oradata/fast_recovery_area/SALESDR/onlinelog/o1_mf_4_7kfk61os_.log
^C
vzpyrus3a (oracle) $

Listing 11. Determining Whether the Recovery Process Is Working

Now that the database is successfully recovering, propagate a copy of the server parameter file and the database password file to the other zone cluster node:

vzpyrus3a (oracle) $ cd /u02/app/oracle/product/11.2.0/dbhome_1/dbs
vzpyrus3a (oracle) $  scp orapwsalesdr spfilesalesdr.ora oracle@vzpyrus3b:`pwd`
Password: 
orapwsalesdr         100% |********************************************|  1536       00:00    
spfilesalesdr.ora    100% |********************************************|  3584 00:00 

Next, create the Oracle Solaris Cluster resources for the listener process and database instance, and then test the switchover of the Oracle resource group to ensure that it is completed successfully, as shown in Listing 12.

vzpyrus3a (root) # clrt register SUNW.oracle_listener
vzpyrus3a (root) # clrs create -g oracle-rg -t SUNW.oracle_listener \
-p resource_dependencies=oracle-lh-rs \
-p ORACLE_HOME=/u02/app/oracle/product/11.2.0/dbhome_1 oracle-lsnr-rs
vzpyrus3a (root) # vzpyrus3a (root) # clrt register SUNW.oracle_server vzpyrus3a (root) # clrs create -g oracle-rg -t SUNW.oracle_server \
-p ORACLE_HOME=/u02/app/oracle/product/11.2.0/dbhome_1 \
-p connect_string=/@salesdr \
-p ORACLE_SID=salesdr \
-p Alert_log_file=/u02/app/oracle/diag/rdbms/salesdr/salesdr/trace/alert_salesdr.log \
-p Dataguard_role=STANDBY \
-p Standby_mode=PHYSICAL \
-p resource_dependencies_offline_restart=oracle-hasp-rs,oracle-lsnr-rs \
oracle-svr-rs
ppyrus2:oracle-zc - ALERT_LOG_FILE /u02/app/oracle/diag/rdbms/salesdr/salesdr/trace/alert_salesdr.log doesn't exist ppyrus2:oracle-zc - This resource depends on a HAStoragePlus resource that is not online on this node. Ignoring validation errors. vzpyrus3a (root) # vzpyrus3a (root) # clrs status === Cluster Resources === Resource Name Node Name State Status Message ------------- --------- ----- -------------- oracle-svr-rs vzpyrus3a Online Online vzpyrus3b Offline Offline oracle-lsnr-rs vzpyrus3a Online Online vzpyrus3b Offline Offline oracle-hasp-rs vzpyrus3a Online Online vzpyrus3b Offline Offline oracle-lh-rs vzpyrus3a Online Online - LogicalHostname online. vzpyrus3b Offline Offline vzpyrus3a (root) # clrg switch -n vzpyrus3b oracle-rg vzpyrus3a (root) # clrg switch -n vzpyrus3a oracle-rg vzpyrus3a (root) #

Listing 12. Create the Resources and Test the Switchover

You might notice that the Solaris Cluster Oracle agent only starts up the database in mount mode and does not restart the recovery process. This is normal and expected behavior for the agent. We will address the database recovery process when we create the Data Guard broker configuration in the next steps.

Before we do that, update the Oracle server resource on the original cluster to ensure that its properties now match the fact that it is an Oracle Data Guard primary database:

pgyruss1 (root) # clrs set -p Dataguard_role=PRIMARY \
-p Standby_mode=PHYSICAL oracle-svr-rs

Complete the replication set up by creating the Oracle Data Guard configuration. First, alter the databases settings for the broker configuration files so they are located on the shared storage and then enable Data Guard broker, as shown in Listing 13.

vzpyrus3a (oracle) $ sqlplus /NOLOG 

SQL*Plus: Release 11.2.0.3.0 Production on Thu Jan 19 04:10:34 2012

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

SQL> connect sys/oracle as sysdba
Connected.
SQL> alter system set dg_broker_config_file1='/oradata/dr1salesdr.dat' scope=both;

System altered.

SQL> alter system set dg_broker_config_file2='/oradata/dr2salesdr.dat' scope=both;

System altered.

SQL> alter system set dg_broker_start=true scope=both;

System altered.

SQL> connect sys/oracle@sales as sysdba
Connected.
SQL> alter system set dg_broker_config_file1='/oradata/dr1sales.dat' scope=both;

System altered.

SQL> alter system set dg_broker_config_file2='/oradata/dr2sales.dat' scope=both;

System altered.

SQL> alter system set dg_broker_start=true scope=both;

System altered.

SQL> Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
oracle@vzpyrus3a:~$ 

Listing 13. Creating the Oracle Data Guard Configuration

Now, start Data Guard broker and begin to define the configuration, as shown in Listing 14. You can create the Data Guard broker configuration from either cluster by specifying the appropriate connection string in the broker session.

vzpyrus3a (oracle) $ dgmgrl 
DGMGRL for Solaris: Version 11.2.0.3.0 - 64bit Production

Copyright (c) 2000, 2009, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
DGMGRL> connect sys/oracle@sales
Connected.
DGMGRL> create configuration sales as primary database is sales
> connect identifier is sales;
Configuration "sales" created with primary database "sales"
DGMGRL> add database salesdr as connect identifier is salesdr
> maintained as physical;
Database "salesdr" added
DGMGRL> enable configuration;
Enabled.
DGMGRL> edit configuration set property BystandersFollowRoleChange='NONE';
Property "bystandersfollowrolechange" updated
DGMGRL> show configuration verbose;

Configuration - sales

  Protection Mode: MaxPerformance
  Databases:
    sales   - Primary database
    salesdr - Physical standby database

  Properties:
    FastStartFailoverThreshold      = '30'
    OperationTimeout                = '30'
    FastStartFailoverLagLimit       = '30'
    CommunicationTimeout            = '180'
    FastStartFailoverAutoReinstate  = 'TRUE'
    FastStartFailoverPmyShutdown    = 'TRUE'
    BystandersFollowRoleChange      = 'NONE'

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS

DGMGRL> quit

Listing 14. Starting Data Guard Broker

If you specified everything correctly, the configuration status shows SUCCESS. Before placing the configuration under Oracle Solaris Cluster Geographic Edition control, be sure to test the switchover process from within Data Guard broker. Clearly, in a production environment, you must choose an appropriate time to perform this operation, because it will have an impact on your database clients.

To check that your switchover works correctly, first disable the Oracle Solaris Cluster Oracle agent monitoring on both clusters, because the health probes react to both your database being shut down and to the fact that your Oracle Data Guard roles do not match the Oracle resource property setting you specified. Once you verify that the switchover works correctly, re-enable Oracle resource monitoring. See Listing 15.

When a Data Guard broker configuration is under Oracle Solaris Cluster Geographic Edition control, the value for the Dataguard_role property in the Oracle resources is managed by the Oracle Solaris Cluster Geographic Edition software's Oracle Data Guard module software.

pgyruss1 (root) # clrs status

=== Cluster Resources ===

Resource Name       Node Name      State        Status Message
-------------       ---------      -----        --------------
oracle-svr-rs       pgyruss1       Offline      Offline
                    pgyruss2       Online       Online

oracle-lsnr-rs      pgyruss1       Offline      Offline
                    pgyruss2       Online       Online

oracle-hasp-rs      pgyruss1       Offline      Offline
                    pgyruss2       Online       Online

oracle-lh-rs        pgyruss1       Offline      Offline - LogicalHostname offline.
                    pgyruss2       Online       Online - LogicalHostname online.

pgyruss1 (root) # clrs unmonitor oracle-svr-rs

vzpyrus3a (root) # clrs status

=== Cluster Resources ===

Resource Name       Node Name      State        Status Message
-------------       ---------      -----        --------------
oracle-svr-rs       vzpyrus3a      Online       Online
                    vzpyrus3b      Offline      Offline

oracle-lsnr-rs      vzpyrus3a      Online       Online
                    vzpyrus3b      Offline      Offline

oracle-hasp-rs      vzpyrus3a      Online       Online
                    vzpyrus3b      Offline      Offline

oracle-lh-rs        vzpyrus3a      Online       Online - LogicalHostname online.
                    vzpyrus3b      Offline      Offline - LogicalHostname offline.

vzpyrus3a (root) # clrs unmonitor oracle-svr-rs
vzpyrus3a (root) # su - oracle
vzpyrus3a (oracle) $ dgmgrl
DGMGRL for Solaris: Version 11.2.0.3.0 - 64bit Production

Copyright (c) 2000, 2009, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
DGMGRL> connect sys/oracle@sales
Connected.
DGMGRL> show configuration

Configuration - sales

  Protection Mode: MaxPerformance
  Databases:
    sales   - Primary database
    salesdr - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS

DGMGRL> switchover to salesdr;
Performing switchover NOW, please wait...
New primary database "salesdr" is opening...
Operation requires shutdown of instance "sales" on database "sales"
Shutting down instance "sales"...
ORACLE instance shut down.
Operation requires startup of instance "sales" on database "sales"
Starting instance "sales"...
ORACLE instance started.
Database mounted.
Switchover succeeded, new primary is "salesdr"
DGMGRL> show configuration;

Configuration - sales

  Protection Mode: MaxPerformance
  Databases:
    salesdr - Primary database
    sales   - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS

DGMGRL> switchover to sales;
Performing switchover NOW, please wait...
New primary database "sales" is opening...
Operation requires shutdown of instance "salesdr" on database "salesdr"
Shutting down instance "salesdr"...
ORACLE instance shut down.
Operation requires startup of instance "salesdr" on database "salesdr"
Starting instance "salesdr"...
ORACLE instance started.
Database mounted.
Switchover succeeded, new primary is "sales"
DGMGRL> show configuration;

Configuration - sales

  Protection Mode: MaxPerformance
  Databases:
    sales   - Primary database
    salesdr - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS

DGMGRL> quit

pgyruss1 (root) # clrs status

=== Cluster Resources ===

Resource Name    Node Name   State                  Status Message
-------------    ---------   -----                  --------------
oracle-svr-rs    pgyruss1    Offline                Offline
                 pgyruss2    Online_not_monitored   Online_not_monitored

oracle-lsnr-rs   pgyruss1    Offline                Offline
                 pgyruss2    Online                 Online

oracle-hasp-rs   pgyruss1    Offline                Offline
                 pgyruss2    Online                 Online

oracle-lh-rs     pgyruss1    Offline                Offline - LogicalHostname offline.
                 pgyruss2    Online                 Online - LogicalHostname online.

pgyruss1 (root) # clrs monitor oracle-svr-rs
pgyruss1 (root) # clrs status

=== Cluster Resources ===

Resource Name       Node Name      State        Status Message
-------------       ---------      -----        --------------
oracle-svr-rs       pgyruss1       Offline      Offline
                    pgyruss2       Online       Online

oracle-lsnr-rs      pgyruss1       Offline      Offline
                    pgyruss2       Online       Online

oracle-hasp-rs      pgyruss1       Offline      Offline
                    pgyruss2       Online       Online

oracle-lh-rs        pgyruss1       Offline      Offline - LogicalHostname offline.
                    pgyruss2       Online       Online - LogicalHostname online.

vzpyrus3a (root) # clrs monitor oracle-svr-rs

Listing 15. Checking Whether Switchover Works

Revision 1.0, 05/01/2012

Follow us on Facebook, Twitter, or Oracle Blogs.