How to Deploy Oracle RAC 11.2.0.2 On Oracle Solaris Zone Clusters

Using Oracle RAC 11.2.0.2, Oracle Solaris 10, and Oracle Solaris Cluster 3.3 5/11

by Vinh Tran

How to create an Oracle Solaris Zone cluster, install and configure Oracle Grid Infrastructure 11.2.0.2 and Oracle Real Application Clusters 11.2.0.2 in the zone cluster, and create an Oracle Solaris Cluster resource for Oracle RAC.


OTN is all about helping you become familiar enough with Oracle technologies to make an informed decision. Articles, software downloads, documentation, and more. Join up and get the technical resources you need to do your job.

Published May 2012

Download the Oracle Software Patches
Prerequisites
Create the Zone Cluster Using the cfg File
Create the Oracle RAC Framework for the Zone Cluster
Set Up the Root Environment in Local Zone Cluster z11gr2A
Create a User and Group for the Oracle Software
Install Oracle Grid Infrastructure 11.2.0.2 in the Oracle Solaris Zone Cluster Nodes
Replace OPatch Version 11.2.0.1.1 with Version 11.2.0.1.6
Apply Oracle Grid Infrastructure 11.2.0.2.3 GI PSU patch to GRID_HOME Using OPatch
Apply Patch 10055663 to GRID_HOME Using OPatch
Configure Oracle Grid Infrastructure
Install and Patch Oracle Database
Apply Oracle Grid Infrastructure 11.2.0.2.3 GI PSU patch  to Oracle Database Using OPatch
Create the Database
Create the Oracle Solaris Cluster Resource
See Also
About the Author

This article describes how to use Oracle Solaris Cluster 3.3 5/11 to provide a virtual, multitiered architecture for Oracle Real Application Cluster (Oracle RAC) 11.2.0.2.  This is one of three similar articles that explain how to do the same thing but with different versions of the software.  This table summarizes them:

Article RAC Version Solaris Version Cluster Version
How to Deploy Oracle RAC 11.2.0.2 on Oracle Solaris Zone Clusters 11.2.0.2 10 3.3
How to Deploy Oracle RAC 11.2.0.3 on Oracle Solaris Zone Clusters 11.2.0.3 10 3.3
How to Deploy Oracle RAC 11.2.0.3 on Oracle Solaris 11 Zone Clusters 11.2.0.3 11 4.0

 

Oracle Solaris Cluster 3.3 provides the capability to create high-availability zone clusters. A zone cluster consists of several Oracle Solaris Zones, each of which resides on its own separate server; the zones that comprise the cluster are linked together into a single virtual cluster. Because zone clusters are isolated from each other, they provide increased security. Because the zones are clustered, they provide high availability for the applications they host.

By installing Oracle RAC inside a zone cluster, you can run multiple instances of an Oracle database at the same time. This allows you to have separate database versions or separate deployments of the same database (for example, one for production and one for development). Using this architecture, you can also deploy different parts of your multitiered solution into different virtual zone clusters. For example, you could deploy Oracle RAC and an application server in different zones of the same cluster. This approach allows you to isolate tiers and administrative domains from each other, while taking advantage of the simplified administration provided by Oracle Solaris Cluster.

Note: This document is not a performance best practices guide and it does not cover the following topics:

  • Oracle Solaris OS installation
  • Storage configuration
  • Network configuration
  • Oracle Solaris Cluster installation

Download the Oracle Software Patches

To download Oracle patches, do the following after you have logged in to My Oracle Support:

  1. Click the Patches & Updates tab.
  2. In the Patch Name or Number field, type the <patch ID>.
  3. In the Platform field, select the relevant platform.
  4. Click the Search button.
  5. Select the patch that corresponds to the Oracle release that is installed on your system.

These are the Oracle software patches you need to download:

  • 11.2.0.2 Patch Set for Oracle Database Server, patch ID 10098816
  • 11.2.0.2.3 GI PSU patch, patch ID 12419353
  • Patch ID 6880880 for OPatch
  • Patch ID 10055663 for the bug CONFIGURATION WIZARD FAILS TO PROCEED AFTER NODES ARE SPECIFIED

Prerequisites

Ensure that the following prerequisites have been met.

  • Oracle Solaris 10 9/10 and Oracle Solaris Cluster 3.3 5/11 are already installed and configured.
     
  • Oracle Solaris 10 kernel parameters are configured in the /etc/system file in the global zone. Examples of the recommended values are as follows:

    noexec_user_stack 1 
    semsys:seminfo_semmni 100 
    semsys:seminfo_semmns 1024 
    semsys:seminfo_semmsl  256 
    semsys:seminfo_semvmx 32767 
    shmsys:shminfo_shmmax 4294967295 
    shmsys:shminfo_shmmni 100
    
  • Shared disks (also known as /dev/did/rdsk devices) are known. Listing 1 is an example of how to identify a shared disk from the global zone of any cluster node.

    phyhost1# cldev status
    
    === Cluster DID Devices ===
    
    Device Instance	Node		Status
    
    /dev/did/rdsk/d1	phyhost1	Ok
    
    /dev/did/rdsk/d10	phyhost1	Ok
    			phyhost2	Ok
    			phyhost3	Ok
    			phyhost4	Ok
    
    /dev/did/rdsk/d14	phyhost		Ok
    			phyhost2	Ok
    			phyhost3	Ok
    			phyhost4	Ok
    
    /dev/did/rdsk/d15	phyhost1	Ok
    			phyhost2	Ok
    			phyhost3	Ok
    			phyhost4	Ok
    
    /dev/did/rdsk/d16	phyhost1	Ok
    			phyhost2	Ok
    			phyhost3	Ok
    			phyhost4	Ok
    
    /dev/did/rdsk/d17	phyhost1	Ok
    			phyhost2	Ok
    			phyhost3	Ok
    			phyhost3	Ok
    
    /dev/did/rdsk/d18	phyhost1	Ok
    			phyhost2	Ok
    			phyhost3	Ok
    			phyhost4	Ok
    
    /dev/did/rdsk/d19	phyhost2	Ok
    
    /dev/did/rdsk/d2	phyhost1	Ok
    
    /dev/did/rdsk/d20	phyhost2	Ok
    
    /dev/did/rdsk/d21	phyhost3	Ok
    
    /dev/did/rdsk/d22	phyhost3	Ok
    
    /dev/did/rdsk/d23	phyhost4	Ok
    
    /dev/did/rdsk/d24	phyhost4	Ok
    
    /dev/did/rdsk/d6	phyhost1	Ok
    			phyhost2	Ok
    			phyhost3	Ok
    			phyhost4	Ok
    
    /dev/did/rdsk/d7	phyhost1	Ok
    			phyhost2	Ok
    			phyhost3	Ok
    			phyhost4	Ok
    
    /dev/did/rdsk/d8	phyhost1	Ok
    			phyhost2	Ok
    			phyhost3	Ok
    			phyhost4	Ok
    
    /dev/did/rdsk/d9	phyhost1	Ok
    			phyhost2	Ok
    			phyhost3	Ok
    			phyhost4	Ok
    
    Listing 1. Identifying a Shared Disk

    The output shows that the disk is shared by phyhost1, phyhost2, phyhost3, and phyhost4 as device d10. Devices d14, d15, d16, d17, d18, d6, d7, d8, and d9 are also shared.

  • The following shared disks will be used for the Oracle Automatic Storage Management disk group to store Oracle Cluster Registry and voting files:

    /dev/did/rdsk/d6s6
    /dev/did/rdsk/d7s6
    ev/did/rdsk/d8s6
    

    In this example, slice 6 is 6 GB. Please check the Oracle Grid Infrastructure Installation Guide for disk size requirements.

  • The following shared disks will be used for the Oracle Automatic Storage Management disk group to store data files:

    /dev/did/rdsk/d9s6
    /dev/did/rdsk/d10s6
    /dev/did/rdsk/d14s6
    /dev/did/rdsk/d15s6
    

    In this example, slice 6 is 102 GB.

  • Oracle virtual IP (VIP) and Single Client Access Name (SCAN) IP requirements have been set up, for example:

    • vzhost1d, IP address 10.134.35.99, is used for SCAN IP.
    • vzhost1e, IP address 10.134.35.100, is used as VIP for vzhost1c.
    • vzhost2e, IP address 10.134.35.101, is used as VIP for vzhost2c.
    • vzhost3e, IP address 10.134.35.102, is used as VIP for vzhost3c.
    • vzhost4e, IP address 10.134.35.103, is used as VIP for vzhost4c.
       
  • There is an IPMP group for the public network with one active and one standby interface. Here is an example of the /etc/hostname.e1000g0 and /etc/hostname.e1000g1 settings for an IPMP group called SC_ipmp0 from the global zone:

    cat /etc/hostname.e1000g0
    phyhost1 netmask + broadcast + group sc_ipmp0 up
    cat /etc/hostname.e1000g1
    group sc_ipmp0 standby up
    

Create the Zone Cluster Using the cfg File

Perform the following steps to create an Oracle Solaris Zone cluster:

  1. Create a cfg file with the content shown in Listing 2.

    cat /var/tmp/zone.cfg
    
    create 
    set zonepath=/export/zones/z11gR2A
    add node
    set physical-host=phyhost1
    set hostname=vzhost1c
    add net
    set address=10.134.35.56
    set physical=e1000g0
    end
    end
    add sysid
    set name_service="NIS{domain_name=solaris.us.oracle.com}"
    set root_password=passwd
    end
    add node
    set physical-host=phyhost2
    set hostname=vzhost2c
    add net
    set address=10.134.35.59
    set physical=e1000g0
    end
    end
    add node
    set physical-host=phyhost3
    set hostname=vzhost3c
    add net
    set address=10.134.35.62
    set physical=e1000g0
    end
    end
    add node
    set physical-host=phyhost4
    set hostname=vzhost4c
    add net
    set address=10.134.35.65
    set physical=e1000g0
    end
    end
    add net
    set address=vzhost1d
    end
    add net
    set address=vzhost2d
    end
    add net
    set address=vzhost3d
    end
    add net
    set address=vzhost4d
    end
    add net
    set address=vzhost1e
    end
    add net
    set address=vzhost2e
    end
    add net
    set address=vzhost3e
    end
    add net
    set address=vzhost4e
    end
    set limitpriv ="default,proc_priocntl,proc_clock_highres,sys_time"
    add dedicated-cpu
    set ncpus=16
    end
    add capped-memory
    set physical=12g
    set swap=12g
    set locked=12g
    end
    add device
    set match="/dev/did/rdsk/d6s6"
    end
    add device
    set match="/dev/did/rdsk/d7s6"
    end
    add device
    set match="/dev/did/rdsk/d8s6"
    end
    add device
    set match="/dev/did/rdsk/d9s6"
    end
    add device
    set match="/dev/did/rdsk/d10s6"
    end
    add device
    set match="/dev/did/rdsk/d14s6"
    end
    add device
    set match="/dev/did/rdsk/d15s6"
    end
    
    Listing 2. Creating the cfg File
  2. Run the commands shown in Listing 3 from one node:

    # clzonecluster configure -f /var/tmp/zone.cfg z11gr2A 
    # clzonecluster install z11gr2A
    
    # clzonecluster status
    
    === Zone Clusters ===
    
    --- Zone Cluster Status ---
    
    Name      Node Name    Zone HostName   Status   Zone Status
    ----      ---------    -------------   ------   -----------
    z11gr2A   phyhost1     vzhost1c       Offline   Installed
              phyhost2     vzhost2c       Offline   Installed
              phyhost3     vzhost3c       Offline   Installed
              phyhost4     vzhost4c       Offline   Installed
    
    # clzc boot z11gr2A
    
    # clzc status
    
    === Zone Clusters ===
    
    --- Zone Cluster Status ---
    
    Name      Node Name     Zone HostName   Status   Zone Status
    ----      ---------     -------------   ------   -----------
    z11gr2A   phyhost1      vzhost1c        Online   Running
              phyhost2      vzhost2c        Online   Running
              phyhost3      vzhost3c        Online   Running
              phyhost4      vzhost4c        Online   Running
    
    Listing 3. Creating the Oracle Solaris Zone Cluster
  3. Make sure NTP is disabled but has dummy ntp.conf file so that Grid Infrastructure time synchronization service will run in Observer mode:

    # svcs ntp
    disabled Mar_28 svc:/network/ntp:default
    
    # cat >  /etc/inet/ntp.conf
    slewalways yes
    disable pll
    ^D
    
    
  4. Create the Oracle RAC Framework for the Zone Cluster

    Perform the following steps to create the Oracle RAC framework:

    1. As root, execute clsetup from one global zone cluster node, as shown in Listing 4.

      # /usr/cluster/bin/clsetup
        *** Main Menu ***
      
          Please select from one of the following options:
      
              1) Quorum
              2) Resource groups
              3) Data Services
              4) Cluster interconnect
              5) Device groups and volumes
              6) Private hostnames
              7) New nodes
              8) Other cluster tasks
      
              ?) Help with menu options
              q) Quit
      
          Option:  3
      
        *** Data Services Menu ***
      
          Please select from one of the following options:
      
            * 1) Apache Web Server
            * 2) Oracle
            * 3) NFS
            * 4) Oracle Real Application Clusters
            * 5) SAP Web Application Server
            * 6) Highly Available Storage
            * 7) Logical Hostname
            * 8) Shared Address
      
            * ?) Help
            * q) Return to the Main Menu
      
          Option:  4
      
        *** Oracle Solaris Cluster Support for Oracle RAC ***
      
          Oracle Solaris Cluster provides a support layer for running Oracle 
          Real Application Clusters (RAC) database instances. This option allows
          you to create the Oracle RAC framework resource group, storage resources, 
          database resources and administer them, for managing the Oracle 
          Solaris Cluster support for Oracle RAC.
      
          After the Oracle RAC framework resource group has been created, you can use 
          the Oracle Solaris Cluster system administration tools to administer a
          Oracle RAC framework resource group that is configured on a global cluster. 
          To administer a Oracle RAC framework resource group that is configured on a 
          zone cluster, instead use the appropriate Oracle Solaris Cluster command.
      
      Listing 4. Executing clsetup
    2. Enter yes (the default) to continue and then answer subsequent questions, as shown in Listing 5.

          Is it okay to continue (yes/no) [yes]?  
          Please select from one of the following options:
      
              1)  Oracle RAC Create Configuration 
              2)  Oracle RAC Ongoing Administration 
      
              q)  Return to the Data Services Menu 
      
          Option:  1
      
         >>> Select Oracle Real Application Clusters Location <<<
      
          Oracle Real Application Clusters Location:
      
              1) Global Cluster
              2) Zone Cluster
      
          Option [2]:  2
      
         >>> Select Zone Cluster <<<
      
          From the list of zone clusters, select the zone cluster where you would like to
          configure Oracle Real Application Clusters.
      
              1) z11gr2A
      
              ?) Help
              d) Done
      
          Selected:  [z11gr2A]
      
         >>> Select Oracle Real Application Clusters Components to Configure <<<
      
          Select the component of Oracle Real Application Clusters that you are configuring:
      
              1) RAC Framework Resource Group
              2) Storage Resources for Oracle Files
              3) Oracle Clusterware Framework Resource
              4) Oracle Automatic Storage Management (ASM)
              5) Resources for Oracle Real Application Clusters Database Instances
      
          Option [1]: 1 
      
         >>> Verify Prerequisites <<<
      
      
          This wizard guides you through the creation and configuration of the Real Application
          Clusters (RAC) framework resource group.
          
          Before you use this wizard, ensure that the following prerequisites are met:
      
      
          * All pre-installation tasks for Oracle Real Application Clusters are completed.
      
          * The Oracle Solaris Cluster nodes are prepared.
      
          * The data services packages are installed.
      
          * All storage management software that you intend to use is installed and configured
          on all nodes where Oracle Real Application Clusters is to run.
      
      
      
          Press RETURN to continue 
      
         >>> Select Nodes <<<
      
          Specify, in order of preference, a list of names of nodes where Oracle Real
          Application Clusters is to run. If you do not explicitly specify a list, the list
          defaults to all nodes in an arbitrary order.
      
      
          The following nodes are available on the zone cluster z11skgxn:
      
              1) vzhost1c
              2) vzhost2c
              3) vzhost3c
              4) vzhost4c
      
              r) Refresh and Clear All
              a) All
              ?) Help
              d) Done
      
          Selected:  [vzhost1c, vzhost2c, vzhost3c, vzhost4c]
          Options:  d
      
         >>> Select Clusterware Support <<<
      
      
          Select the vendor clusterware support that you would like to use.
      
              1) Native
              2) UDLM based
      
          Option [1]: 1 
      
         >>> Review Oracle Solaris Cluster Objects <<<
      
      
          The following Oracle Solaris Cluster objects will be created.
      
          Select the value you are changing:
      
                Property Name               Current Setting
                =============               ===============
              1) Resource Group Name        rac-framework-rg
              2) RAC Framework Resource N...rac-framework-rs
      
              d) Done
              ?) Help
      
          Option:  d
      
         >>> Review Configuration of RAC Framework Resource Group <<<
      
      
          The following Oracle Solaris Cluster configuration will be created.
      
          To view the details for an option, select the option.
      
                Name                         Value
                ====                         =====
              1) Resource Group Name        rac-framework-rg
              2) RAC Framework Resource N...rac-framework-rs
      
              c) Create Configuration
              ?) Help
      
          Option:  c
      
      Listing 5. Answering Subsequent Questions
    3. From one global zone cluster node, verify the Oracle RAC framework resources, as shown in Listing 6.

      # clrs status -Z z11gr2A 
      
      === Cluster Resources ===
      
      Resource Name         Node Name      State      Status Message
      -------------         ---------      -----      --------------
      rac-framework-rs      vzhost1c       Online     Online
                            vzhost2c       Online     Online
                            vzhost3c       Online     Online
                            vzhost4c       Online     Online
      
      Listing 6. Verifying the Resources

    Set Up the Root Environment in Local Zone Cluster z11gr2A

    From each global zone cluster node (phyhost1, phyhost2, phyhost3, and phyhost4), do the following.

    1. Log in to the local zone node and execute the following command as root:

      # /usr/sbin/zlogin z11gr2A
      [Connected to zone 'z11gr2A' pts/2]
      Last login: Thu Aug 25 17:30:14 on pts/2
      Oracle Corporation      SunOS 5.10      Generic Patch   January 2005
      
    2. (Optional) Change the root shell to bash:

      # passwd -e
      Old shell: /sbin/sh
      New shell: bash
      passwd: password information changed for root
      
    3. Include these paths in .bash_profile:

      /u01/grid/product/11.2.0.2/bin
      /usr/cluster/bin
      

    Create a User and Group for the Oracle Software

    To create a user and group for the Oracle software, do the following from each node.

    1. As root, execute the following commands:

      # groupadd -g 300 oinstall   
      # groupadd -g 301 dba
      # useradd -g 300 -G 301 -u 302 -d /u01/ora_home -s /usr/bin/bash ouser  
      # mkdir -p /u01/ora_home
      # chown ouser:oinstall /u01/ora_home
      # mkdir /u01/oracle
      # chown ouser:oinstall /u01/oracle
      # mkdir /u01/grid
      # chown ouser:oinstall /u01/grid
      # mkdir /u01/oraInventory
      # chown ouser:oinstall /u01/oraInventory
      
    2. Create a password for the software owner, ouser:

      # passwd ouser
      New Password:
      Re-enter new Password:
      passwd: password successfully changed for ouser
      bash-3.00#
      
    3. For the Oracle software owner environment, as the software owner ouser, set up SSH from each node:

      $ mkdir .ssh
      $ chmod 700 .ssh
      $ cd .ssh
      $ ssh-keygen -t rsa
      Generating public/private rsa key pair.
      Enter file in which to save the key (/u01/ora_home/.ssh/id_rsa):
      Enter passphrase (empty for no passphrase): 
      Enter same passphrase again:
      Your identification has been saved in /u01/ora_home/.ssh/id_rsa.
      Your public key has been saved in /u01/ora_home/.ssh/id_rsa.pub.
      The key fingerprint is:
      e6:63:c9:71:fe:d1:8f:71:77:70:97:25:2a:ee:a9:33 local1@vzhost1c
      $
      $ pwd
      /u01/ora_home/.ssh
      
    4. From the first node, vzhost1c, do the following:

      $ cat id_rsa.pub >> authorized_keys
      $ chmod 600 authorized_keys
      $ scp authorized_keys vzhost2c:/u01/ora_home/.ssh
      
    5. From the second node, vzhost2c, do the following:

      $ cd /u01/ora_home/.ssh
      $ cat id_rsa.pub >> authorized_keys
      $ scp authorized_keys vzhost3c:/u01/ora_home/.ssh
      
    6. From the third node, vzhost3c,do the following:

      $ cd /u01/ora_home/.ssh
      $ cat id_rsa.pub >> authorized_keys
      $ scp authorized_keys vzhost4c:/u01/ora_home/.ssh
      
    7. From the fourth node, vzhost4c, do the following:

      $ cd /u01/ora_home/.ssh
      $ cat id_rsa.pub >> authorized_keys
      $ scp authorized_keys vzhost1c:/u01/ora_home/.ssh
      
    8. From the first node, vzhost1c, do the following:

      $ cd /u01/ora_home/.ssh
      $ scp authorized_keys vzhost2c:/u01/ora_home/.ssh
      $ scp authorized_keys vzhost3c:/u01/ora_home/.ssh
      
    9. From each node, test the ssh setup:

      $ ssh vzhost1c date
      $ ssh vzhost2c date
      $ ssh vzhost3c date
      $ ssh vzhost4c date
      
    10. In each local zone cluster node, as root, set up the Oracle Automatic Storage Management candidate disk:

      # for i in 6 7 8 9 10 14 15 
      > do
      > chown ouser:oinstall /dev/did/rdsk/d${i}s6
      > chmod 660 /dev/did/rdsk/d${i}s6
      > done
      
    11. In the local zone cluster, as the software owner, do the following from one node:

      $ for i in 6 7 8 9 10 14 15 
      > do
      > dd if=/dev/zero of=/dev/did/rdsk/d${i}s6 bs=1024k count=200
      > done
      

    Install Oracle Grid Infrastructure 11.2.0.2 in the Oracle Solaris Zone Cluster Nodes

    On each node, perform the following procedure.

    1. As the software owner, execute the following:

      $ bash
      $ export DISPLAY=<hostname>:<n>
      $ cd <PATH to 11.2.0.2 based software image>/grid/
      $ ./runInstaller
      
    2. Provide the following input to Oracle Universal Installer:

      1. On the Select Installation Option page, select Install Oracle Grid Infrastructure Software Only.
      2. On the Select Product Languages page, select the appropriate language.
      3. On the Privileged Operating System Groups page, select the following, as shown in Figure 1:

        • For Oracle ASM DBA (OSDBA for ASM) Group, select oinstall.
        • For Oracle ASM Operator (OSOPER for ASM) Group (Optional), select a group.
        • For Oracle ASM Administrator (OSASM) Group, select oinstall.

          Figure 1

          Figure 1. Privileged Operating System Groups Page

      4. On the Specify Installation Location page, do the following:
        • For Oracle Base, select /u01/oracle.
        • For Software Location, select /u01/grid/product/11.2.0.2.
      5. On the Create Inventory page, select /u01/oraInventory for Inventory Directory. The oraInventory Group Name should be oinstall.
      6. On the Perform Prerequisite Checks page, select Ignore All.
      7. On the Summary page, check all the settings you selected to ensure they are appropriate, and then click Install to install the product.

        The Execute Configuration Scripts dialog box asks you to execute the /u01/oraInventory/orainstRoot.sh and /u01/grid/product/11.2.0.2/root.sh scripts as root, as shown in Figure 2.

        Figure 2

        Figure 2. Execute Configuration Scripts Dialog Box

    3. Open a terminal window and execute the scripts, as shown in Listing 7.

      # /u01/oraInventory/orainstRoot.sh
      Changing permissions of /u01/oraInventory.
      Adding read,write permissions for group.
      Removing read,write,execute permissions for world.
      
      Changing groupname of /u01/oraInventory to oinstall.
      The execution of the script is complete.
      
      # /u01/grid/product/11.2.0.2/root.sh
      Running Oracle 11g root script...
      
      The following environment variables are set as:
          ORACLE_OWNER= ouser
          ORACLE_HOME=  /u01/grid/product/11.2.0.2
      
      Enter the full pathname of the local bin directory: [/usr/local/bin]: /opt/local/bin
         Copying dbhome to /opt/local/bin ...
         Copying oraenv to /opt/local/bin ...
         Copying coraenv to /opt/local/bin ...
      
      
      Creating /var/opt/oracle/oratab file...
      Entries will be added to the /var/opt/oracle/oratab file as needed by
      Database Configuration Assistant when a database is created
      Finished running generic part of root script.
      Now product-specific root actions will be performed.
      
      To configure Grid Infrastructure for a Stand-Alone Server run the following command as the root user:
      /u01/grid/product/11.2.0.2/perl/bin/perl 
      -I/u01/grid/product/11.2.0.2/perl/lib 
      -I/u01/grid/product/11.2.0.2/crs/install 
      /u01/grid/product/11.2.0.2/crs/install/roothas.pl
      
      
      To configure Grid Infrastructure for a Cluster execute the following command:
      /u01/grid/product/11.2.0.2/crs/config/config.sh
      This command launches the Grid Infrastructure Configuration Wizard. The wizard also supports silent 
      operation, and the parameters can be passed through the response file that is 
      available in the installation media.
      
      Listing 7. Executing the Scripts

    Replace OPatch Version 11.2.0.1.1 with Version 11.2.0.1.6

    On each node, perform the following procedure to update OPatch to the latest version.

    1. As the software owner, back up the OPatch version in GRID_Home, for example:

      $ cd /u01/grid/product/11.2.0.2/ 
      $ mv OPatch OPatch.bak
      
    2. As the software owner, copy p6880880_112000_SOLARIS64.zip to GRID_HOME:

      $ cd /u01/grid/product/11.2.0.2
      $ unzip p6880880_112000_SOLARIS64.zip
      
    3. Verify the OPatch version:

      $ export ORACLE_HOME=/u01/grid/product/11.2.0.2
      $ /u01/grid/product/11.2.0.2/OPatch/opatch -version
      
      Invoking OPatch 11.2.0.1.6
      
      Oracle Interim Patch Installer version 11.2.0.1.6
      Copyright (c) 2011, Oracle Corporation.  All rights reserved.
      

    Apply Oracle Grid Infrastructure 11.2.0.2.3 GI PSU patch to GRID_HOME Using OPatch

    Apply patch 11.2.0.2 GIB3 to each node by performing the following steps.

    1. As the software owner, copy the patch to the node, for example:

      $ cd /var/tmp
      $ mkdir gib3
      $ cp p12419353_112020_SOLARIS64.zip /var/tmp/gib3
      $ cd /var/tmp/gib3
      $ unzip p12419353_112020_SOLARIS64.zip
      
    2. As the software owner, apply the patch, as shown in Listing 8:

      $ /u01/grid/product/11.2.0.2/OPatch/opatch napply -oh /u01/grid/product/11.2.0.2 -local /var/tmp/gib3
      
      Invoking OPatch 11.2.0.1.6
      
      Oracle Interim Patch Installer version 11.2.0.1.6
      Copyright (c) 2011, Oracle Corporation.  All rights reserved.
      
      UTIL session
      
      Oracle Home       : /u01/grid/product/11.2.0.2
      Central Inventory : /u01/oraInventory
         from           : /var/opt/oracle/oraInst.loc
      OPatch version    : 11.2.0.1.6
      OUI version       : 11.2.0.2.0
      Log file location : /u01/grid/product/11.2.0.2/cfgtoollogs/opatch/opatch2011-08-
      12_17-50-22PM.log
      
      Verifying environment and performing prerequisite checks...
      
      Patch continues with these patches:   12419331  12419353  
      
      Do you want to proceed? [y|n]
      
      Listing 8. Applying the Patch
    3. Enter y to continue:

      Provide your email address to be informed of security issues, install and
      initiate Oracle Configuration Manager. Easier for you if you use your My
      Oracle Support Email address/User Name.
      Visit http://www.oracle.com/support/policies.html for details.
      Email address/User Name: 
      
    4. Enter your user e-mail address.
      Provide your My Oracle Support password to receive security updates via your My 
      Oracle Support account.
      Password (optional):  
      
    5. No password is required, so press Enter.

      Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
      (Oracle Home = '/u01/grid/product/11.2.0.2')
      
      Is the local system ready for patching? [y|n]
      
    6. Enter y to continue, as shown in Listing 9.

      Backing up files...
      Applying interim patch '12419331' to OH '/u01/grid/product/11.2.0.2'
      ApplySession: Optional component(s) [ oracle.sysman.console.db, 11.2.0.2.0 ] , 
      [ oracle.sysman.oms.core, 10.2.0.4.3 ] , [ oracle.rdbms.dv, 11.2.0.2.0 ] , 
      [ oracle.sysman.plugin.db.main.repository, 11.2.0.2.0 ]  not present in the Oracle Home 
      or a higher version is found.
      
      Patching component oracle.rdbms.rsf, 11.2.0.2.0...
      
      Patching component oracle.rdbms, 11.2.0.2.0...
      Copying file to "/u01/grid/product/11.2.0.2/psu/11.2.0.2.3/catpsu.sql"
      Copying file to "/u01/grid/product/11.2.0.2/psu/11.2.0.2.3/catpsu_rollback.sql"
      Copying file to "/u01/grid/product/11.2.0.2/cpu/scripts/patch_8837510.sql"
      Copying file to "/u01/grid/product/11.2.0.2/cpu/scripts/emdb_recomp_invalids.sql"
      
      Patching component oracle.ldap.rsf, 11.2.0.2.0...
      
      Patching component oracle.rdbms.dbscripts, 11.2.0.2.0...
      
      Patching component oracle.rdbms.rman, 11.2.0.2.0...
      Applying interim patch '12419353' to OH '/u01/grid/product/11.2.0.2'
      
      Patching component oracle.crs, 11.2.0.2.0...
      Copying file to "/u01/grid/product/11.2.0.2/crs/install/crsconfig_lib.pm"
      Copying file to "/u01/grid/product/11.2.0.2/crs/install/crspatch.pm"
      Copying file to "/u01/grid/product/11.2.0.2/crs/install/s_crsconfig_lib.pm"
      
      Patching component oracle.usm, 11.2.0.2.0...
      Patches 12419331,12419353 successfully applied.
      Log file location: /u01/grid/product/11.2.0.2/cfgtoollogs/opatch/opatch2011-08-1
      2_17-50-22PM.log
      
      OPatch succeeded.
      
      Listing 9. Finishing Applying the Patch
    7. As the software owner, verify that the patch was installed correctly, as shown in Listing 10.

      $ /u01/grid/product/11.2.0.2/OPatch/opatch lsinventory -oh /u01/grid/product/11.2.0.2 
      
      Invoking OPatch 11.2.0.1.6
      
      Oracle Interim Patch Installer version 11.2.0.1.6
      Copyright (c) 2011, Oracle Corporation.  All rights reserved.
      
      
      Oracle Home       : /u01/grid/product/11.2.0.2
      Central Inventory : /u01/oraInventory
         from           : /var/opt/oracle/oraInst.loc
      OPatch version    : 11.2.0.1.6
      OUI version       : 11.2.0.2.0
      Log file location : /u01/grid/product/11.2.0.2/cfgtoollogs/opatch/opatch2011-08-12_21-32-53PM.log
      
      Lsinventory Output file location : 
      /u01/grid/product/11.2.0.2/cfgtoollogs/opatch/lsinv/lsinventory2011-08-12_21-32-53PM.txt
      
      --------------------------------------------------------------------------------
      Installed Top-level Products (1): 
      
      Oracle Grid Infrastructure                                           11.2.0.2.0
      There are 1 products installed in this Oracle Home.
      
      
      Interim patches (2) :
      
      Patch  12419353     : applied on Fri Aug 12 18:11:35 PDT 2011
      Unique Patch ID:  13850636
         Created on 18 Jul 2011, 05:19:12 hrs UTC
         Bugs fixed:
           12419353, 10157506, 10178670, 10425672, 12311357, 9959110, 10272615
      	     10314123, 10014392, 10089120, 10057296, 9864003, 11775080, 9916145
           10044622, 12399977, 12421404, 12340700, 10056713, 10637741, 9939306
      	     9902536, 10007185, 10376847, 10038791, 11741224, 11655840, 10048487
           10322157, 10260251, 10052721, 10028235, 10027079, 10357258, 10045436
      	     10231906, 10622973, 9891341, 10072474, 10036834, 10029900, 9974223
           10016083, 9918485, 11781515, 10040647, 10069541, 10029119, 10233159
      	     12332919, 9812956, 10036193, 10015210, 12340501, 10621175, 11877079
           8906163, 10111010, 10115514, 10104377, 10057680, 10280665, 10078086
      	     9944948, 10146768, 10052529, 10011084, 10012319, 10073075, 10233811
           10299006, 10248739, 10236074, 10128191, 11071429, 10019726, 9975837
      	     10253630, 9949676, 11936945, 10637483, 10157622, 11698552, 10385838
           10053985, 10425674, 9812970, 11828633, 11899801, 10083789, 9876201
      	     10073372, 9963327, 11077756, 10375649, 9336825, 11682409, 10062301
           10018215, 10105195, 10419987, 10071992, 10634513, 9926027, 10103954
      	     10028343, 11866171, 10065216, 9907089, 9897335, 10190153, 11744313
           10175855, 10284828, 10028637, 10361177, 9979706, 10324594, 10015460
      	     9971646, 11782423, 11654726, 9978765, 10398810, 11904778, 10397652
           9915329, 10107380, 10110969, 10305361, 10331452, 10083009, 10631693
      	     10008467, 10048027, 10040109, 9944978, 10033106, 9978195, 11840629
           10042143, 10284693, 10638381, 9679401, 11663339, 10075643, 10205290
      	     10124517, 11069614, 9593552, 10168006, 12677816, 11807012, 11846686
           9867867, 10228079, 10015603, 10241696, 9942881, 10252497, 10283058
      	     10157625, 10283167, 9906432, 10216878, 10045316, 10425675, 10061534
           11789566, 10283549, 10311856, 10150020, 12421420, 12378675, 10113899
      	     10069698, 9861790, 10087118, 10056808, 10146744, 10326548, 10019796
           9975343, 9936659, 10244210, 10029794, 10266447, 10193581, 12318560
      	     11804097, 10070563, 10268642, 10283596
      
      Patch  12419331     : applied on Fri Aug 12 18:06:44 PDT 2011
      Unique Patch ID:  13850636
         Created on 12 Jul 2011, 07:01:28 hrs PST8PDT
         Bugs fixed:
           10151017, 10158965, 11724916, 10190642, 12586486, 12586487, 10129643
      	     12586488, 12586489, 10018789, 9744252, 10248523, 9956713, 10356513
           9715581, 9770451, 10378005, 10170431, 10425676, 10222719, 10126094
      	     9591812, 10127360, 10132870, 10094201, 9443361, 10193846, 11664046
           11069199, 10324294, 10245086, 12586490, 10205230, 12586491, 10052141
      	     12586492, 12586493, 12586494, 10142788, 11818335, 11830776, 12586495
           9905049, 11830777, 12586496, 11830778, 6892311, 10040921, 10077191
      	     10358019, 12431716, 10219576, 10258337, 11707699, 10264680, 10209232
           11651810, 10102506, 11067567, 9881076, 10278372, 10040531, 10621169
      	     10155605, 10082277, 10356782, 10218814, 9078442, 9788588, 10157249
           9735237, 10317487, 12326246, 11707302, 10310299, 10636231, 10230571
      	     11065646, 12419321, 10368698, 10079168, 10013431, 10228151, 10233732
           10324526, 8223165, 10238786, 10217802, 10061015, 9953542, 9572787
      	     10052956, 10080579, 11699057, 12620422, 10332111, 10227288, 10329146
           10332589, 10110863, 10073683, 9869401, 10019218, 10229719, 11664719
      	     9539440, 10373381, 9735282, 9748749, 11724984, 10022980, 10411618
           11800854, 12419331, 11674485, 10187168, 6523037, 10648873, 9724970
      	     10053725, 10084145, 10367188, 11800170, 11695285, 10157402, 9651350, 10299224
      
      
      --------------------------------------------------------------------------------
      
      OPatch succeeded.
      
      Listing 10. Verifying the Patch Installation

    Apply Patch 10055663 to GRID_HOME Using OPatch

    Apply patch 10055663 to each node by performing the following steps.

    1. Download the patch and place it in /var/tmp.
       
    2. As the software owner, uncompress the patch, for example:

      $ cd /var/tmp
      $ unzip p10055663_112020_Generic.zip
      
    3. Apply the patch using OPatch, as shown in Listing 11.

      $ export ORACLE_HOME=/u01/grid/product/11.2.0.2 
      $ cd /var/tmp/10055663
      $ /u01/grid/product/11.2.0.2/OPatch/opatch apply
      
      Invoking OPatch 11.2.0.1.6
      
      Oracle Interim Patch Installer version 11.2.0.1.6
      Copyright (c) 2011, Oracle Corporation.  All rights reserved.
      
      
      Oracle Home       : /u01/grid/product/11.2.0.2
      Central Inventory : /u01/oraInventory
         from           : /var/opt/oracle/oraInst.loc
      OPatch version    : 11.2.0.1.6
      OUI version       : 11.2.0.2.0
      Log file location : /u01/grid/product/11.2.0.2/cfgtoollogs/opatch/opatch2011-08-10_10-46-04AM.log
      
      Applying interim patch '10055663' to OH '/u01/grid/product/11.2.0.2'
      Verifying environment and performing prerequisite checks...
      Do you want to proceed? [y|n]
      
      Listing 11. Applying the Patch
    4. Enter Y to continue.

      User Responded with: Y
      .....
      Patching component oracle.rdbms.install.plugins, 11.2.0.2.0...
      Patch 10055663 successfully applied
      Log file location: /u01/grid/product/11.2.0.2/cfgtoollogs/opatch/opatch2011-08-12_15-24-31PM.log
      
      OPatch succeeded.
      
    5. As the software owner, verify that the patch was installed correctly, as shown in Listing 12.

      $ /u01/grid/product/11.2.0.2/OPatch/opatch lsinventory -oh /u01/grid/product/11.2.0.2
      
      Invoking OPatch 11.2.0.1.6
      
      Oracle Interim Patch Installer version 11.2.0.1.6
      Copyright (c) 2011, Oracle Corporation.  All rights reserved.
      
      
      Oracle Home       : /u01/grid/product/11.2.0.2
      Central Inventory : /u01/oraInventory
         from           : /var/opt/oracle/oraInst.loc
      OPatch version    : 11.2.0.1.6
      OUI version       : 11.2.0.2.0
      Log file location : /u01/grid/product/11.2.0.2/cfgtoollogs/opatch/opatch2011-08-10_10-48-42AM.log
      
      Lsinventory Output file location : 
      /u01/grid/product/11.2.0.2/cfgtoollogs/opatch/lsinv/lsinventory2011-08-10_10-48-42AM.txt
      
      --------------------------------------------------------------------------------
      Installed Top-level Products (1): 
      
      Oracle Grid Infrastructure                                           11.2.0.2.0
      There are 1 products installed in this Oracle Home.
      
      
      Interim patches (3) :
      
      Patch  10055663     : applied on Wed Aug 10 10:47:16 PDT 2011
      Unique Patch ID:  13606606
         Created on 22 Mar 2011, 01:03:34 hrs PST8PDT
         Bugs fixed:
           10055663
      
      Patch  12419353     : applied on Wed Aug 10 10:32:47 PDT 2011
      Unique Patch ID:  13850636
         Created on 18 Jul 2011, 05:19:12 hrs UTC
         Bugs fixed:
           12419353, 10157506, 10178670, 10425672, 12311357, 9959110, 10272615
      	     10314123, 10014392, 10089120, 10057296, 9864003, 11775080, 9916145
           10044622, 12399977, 12421404, 12340700, 10056713, 10637741, 9939306
      	     9902536, 10007185, 10376847, 10038791, 11741224, 11655840, 10048487
           10322157, 10260251, 10052721, 10028235, 10027079, 10357258, 10045436
      	     10231906, 10622973, 9891341, 10072474, 10036834, 10029900, 9974223
           10016083, 9918485, 11781515, 10040647, 10069541, 10029119, 10233159
      	     12332919, 9812956, 10036193, 10015210, 12340501, 10621175, 11877079
           8906163, 10111010, 10115514, 10104377, 10057680, 10280665, 10078086
      	     9944948, 10146768, 10052529, 10011084, 10012319, 10073075, 10233811
           10299006, 10248739, 10236074, 10128191, 11071429, 10019726, 9975837
      	     10253630, 9949676, 11936945, 10637483, 10157622, 11698552, 10385838
           10053985, 10425674, 9812970, 11828633, 11899801, 10083789, 9876201
      	     10073372, 9963327, 11077756, 10375649, 9336825, 11682409, 10062301
           10018215, 10105195, 10419987, 10071992, 10634513, 9926027, 10103954
      	     10028343, 11866171, 10065216, 9907089, 9897335, 10190153, 11744313
           10175855, 10284828, 10028637, 10361177, 9979706, 10324594, 10015460
      	     9971646, 11782423, 11654726, 9978765, 10398810, 11904778, 10397652
           9915329, 10107380, 10110969, 10305361, 10331452, 10083009, 10631693
      	     10008467, 10048027, 10040109, 9944978, 10033106, 9978195, 11840629
           10042143, 10284693, 10638381, 9679401, 11663339, 10075643, 10205290
      	     10124517, 11069614, 9593552, 10168006, 12677816, 11807012, 11846686
           9867867, 10228079, 10015603, 10241696, 9942881, 10252497, 10283058
      	     10157625, 10283167, 9906432, 10216878, 10045316, 10425675, 10061534
           11789566, 10283549, 10311856, 10150020, 12421420, 12378675, 10113899
      	     10069698, 9861790, 10087118, 10056808, 10146744, 10326548, 10019796
           9975343, 9936659, 10244210, 10029794, 10266447, 10193581, 12318560
      	     11804097, 10070563, 10268642, 10283596
      
      Patch  12419331     : applied on Wed Aug 10 10:27:15 PDT 2011
      Unique Patch ID:  13850636
         Created on 12 Jul 2011, 07:01:28 hrs PST8PDT
         Bugs fixed:
           10151017, 10158965, 11724916, 10190642, 12586486, 12586487, 10129643
      	     12586488, 12586489, 10018789, 9744252, 10248523, 9956713, 10356513
           9715581, 9770451, 10378005, 10170431, 10425676, 10222719, 10126094
      	     9591812, 10127360, 10132870, 10094201, 9443361, 10193846, 11664046
           11069199, 10324294, 10245086, 12586490, 10205230, 12586491, 10052141
      	     12586492, 12586493, 12586494, 10142788, 11818335, 11830776, 12586495
           9905049, 11830777, 12586496, 11830778, 6892311, 10040921, 10077191
      	     10358019, 12431716, 10219576, 10258337, 11707699, 10264680, 10209232
           11651810, 10102506, 11067567, 9881076, 10278372, 10040531, 10621169
      	     10155605, 10082277, 10356782, 10218814, 9078442, 9788588, 10157249
           9735237, 10317487, 12326246, 11707302, 10310299, 10636231, 10230571
      	     11065646, 12419321, 10368698, 10079168, 10013431, 10228151, 10233732
           10324526, 8223165, 10238786, 10217802, 10061015, 9953542, 9572787
      	     10052956, 10080579, 11699057, 12620422, 10332111, 10227288, 10329146
           10332589, 10110863, 10073683, 9869401, 10019218, 10229719, 11664719
      	     9539440, 10373381, 9735282, 9748749, 11724984, 10022980, 10411618
           11800854, 12419331, 11674485, 10187168, 6523037, 10648873, 9724970
      	     10053725, 10084145, 10367188, 11800170, 11695285, 10157402, 9651350, 10299224
      
      --------------------------------------------------------------------------------
      
      
      OPatch succeeded.
      
      Listing 12. Verifying the Patch Installation

    Configure Oracle Grid Infrastructure

    To configure Oracle Grid Infrastructure, do the following from one node.

    1. Run the following command, providing values appropriate for your environment:

      $ export DISPLAY=<hostname>:<n>
      $ /u01/grid/product/11.2.0.2/crs/config/config.sh
      
    2. Provide the following input to Oracle Universal Installer:

      1. On the Select Configuration Option page, select the Configure Oracle Grid Infrastructure for a Cluster option.
      2. On the Grid Plug and Play Information page, do the following, as shown in Figure 3:
        • For Cluster Name, select vzhost-cluster.
        • For SCAN Name, select vzhost1d.
        • For SCAN Port, select 1521.

          Figure 3

          Figure 3. Grid Plug and Play Information Page

      3. On the Cluster Node Information page, specify the following, as shown in Figure 4:

        Hostname       Virtual IP Name
        vzhost1c       vzhost1e
        vzhost2c       vzhost2e
        vzhost3c       vzhost3e
        vzhost4c       vzhost4e
        
        Figure 4

        Figure 4. Cluster Node Information Page

      4. On the Specify Network Interface Usage page, specify the following, as shown in Figure 5:

        Interface Name   Subnet         Interface Type
        e1000g0          10.134.35.0    Public
        clprivnet0       172.16.7.0     Private
        
        Figure 5

        Figure 5. Specify Network Interface Usage Page

      5. On the Storage Option Information page, select the Oracle Automatic Storage Management (Oracle ASM) option.
         
      6. On the Create ASM Disk Group page, create a disk group by doing the following, as shown in Figure 6 through Figure 8:

        • For Disk Group Name, specify crsdg.
        • Click Change Discovery Path.
        • In the Change Discovery Path dialog box, specify the discovery path as /dev/did/rdsk/d*s6, click OK, and ensure the user has the correct permissions.
        • Select /dev/did/rdsk/d6s6, /dev/did/rdsk/d7s6, and /dev/did/rdsk/d8s6 for the crsdg disk group.

          Figure 6

          Figure 6. Specifying the Disk Group Name

          Figure 7

          Figure 7. Specifying the Discovery Path

          Figure 8

          Figure 8. Created Disk Group

      7. On the Specify ASM Password page, choose whether to use the same password for the SYS and ASMSNMP accounts, and then specify the password, as shown in Figure 9. If desired, you can choose to use different password for the accounts.

        Figure 9

        Figure 9. Specify ASM Password Page

      8. On the Perform Prerequisite Checks page, select Ignore All.
         
      9. On the Summary page, check all the settings you selected to ensure they are appropriate and click Install.

        The Execute Configuration Scripts dialog box asks you to execute the GRID_Home/root.sh script on each node as root, as shown in Figure 10.

        Figure 10

        Figure 10. Execute Configuration Scripts Dialog Box

    3. Open a terminal window and execute the script, as shown in Listing 13.

      # /u01/grid/product/11.2.0.2/root.sh
      
      Running Oracle 11g root script...
      
      The following environment variables are set as:
          ORACLE_OWNER= oracle
          ORACLE_HOME=  /u01/grid/product/11.2.0.2
      
      Enter the full pathname of the local bin directory: [/usr/local/bin]: /opt/local/bin
      The contents of "dbhome" have not changed. No need to overwrite.
      The contents of "oraenv" have not changed. No need to overwrite.
      The contents of "coraenv" have not changed. No need to overwrite.
      
      Entries will be added to the /var/opt/oracle/oratab file as needed by
      Database Configuration Assistant when a database is created
      Finished running generic part of root script.
      Now product-specific root actions will be performed.
      Relinking oracle with rac_on option
      Using configuration parameter file: /u01/grid/product/11.2.0.2/crs/install/crsconfig_params
      Creating trace directory
      LOCAL ADD MODE 
      Creating OCR keys for user 'root', privgrp 'root'..
      Operation successful.
      OLR initialization - successful
        root wallet
        root wallet cert
        root cert export
        peer wallet
        profile reader wallet
        pa wallet
        peer wallet keys
        pa wallet keys
        peer cert request
        pa cert request
        peer cert
        pa cert
        peer root cert TP
        profile reader root cert TP
        pa root cert TP
        peer pa cert TP
        pa peer cert TP
        profile reader pa cert TP
        profile reader peer cert TP
        peer user cert
        pa user cert
      Adding daemon to inittab
      ACFS-9459: ADVM/ACFS is not supported on this OS version: 'Solaris Container: Solaris 10 update 9'
      ACFS-9201: Not Supported
      ACFS-9459: ADVM/ACFS is not supported on this OS version: 'Solaris Container: Solaris 10 update 9'
      CRS-2672: Attempting to start 'ora.mdnsd' on 'vzhost1c'
      CRS-2676: Start of 'ora.mdnsd' on 'vzhost1c' succeeded
      CRS-2672: Attempting to start 'ora.gpnpd' on 'vzhost1c'
      CRS-2676: Start of 'ora.gpnpd' on 'vzhost1c' succeeded
      CRS-2672: Attempting to start 'ora.cssdmonitor' on 'vzhost1c'
      CRS-2672: Attempting to start 'ora.gipcd' on 'vzhost1c'
      CRS-2676: Start of 'ora.cssdmonitor' on 'vzhost1c' succeeded
      CRS-2676: Start of 'ora.gipcd' on 'vzhost1c' succeeded
      CRS-2672: Attempting to start 'ora.cssd' on 'vzhost1c'
      CRS-2672: Attempting to start 'ora.diskmon' on 'vzhost1c'
      CRS-2676: Start of 'ora.diskmon' on 'vzhost1c' succeeded
      CRS-2676: Start of 'ora.cssd' on 'vzhost1c' succeeded
      
      ASM created and started successfully.
      
      Disk Group crsdg created successfully.
      
      clscfg: -install mode specified
      Successfully accumulated necessary OCR keys.
      Creating OCR keys for user 'root', privgrp 'root'..
      Operation successful.
      CRS-4256: Updating the profile
      Successful addition of voting disk 3e08bbbbc8e94fa4bf78bfe416d5f4c9.
      Successful addition of voting disk 2a0642961e9c4f84bf0d8f87c47fc201.
      Successful addition of voting disk 994f063bcf5b4fb9bf328f34eecc95d9.
      Successfully replaced voting disk group with +crsdg.
      CRS-4256: Updating the profile
      CRS-4266: Voting file(s) successfully replaced
      ##  STATE    File Universal Id                File Name Disk group
      --  -----    -----------------                --------- ---------
       1. ONLINE   3e08bbbbc8e94fa4bf78bfe416d5f4c9 (/dev/did/rdsk/d6s6) [CRSDG]
       2. ONLINE   2a0642961e9c4f84bf0d8f87c47fc201 (/dev/did/rdsk/d7s6) [CRSDG]
       3. ONLINE   994f063bcf5b4fb9bf328f34eecc95d9 (/dev/did/rdsk/d8s6) [CRSDG]
      Located 3 voting disk(s).
      CRS-2672: Attempting to start 'ora.asm' on 'vzhost1c'
      CRS-2676: Start of 'ora.asm' on 'vzhost1c' succeeded
      CRS-2672: Attempting to start 'ora.CRSDG.dg' on 'vzhost1c'
      CRS-2676: Start of 'ora.CRSDG.dg' on 'vzhost1c' succeeded
      ACFS-9459: ADVM/ACFS is not supported on this OS version: 'Solaris Container: Solaris 10 update 9'
      ACFS-9201: Not Supported
      ACFS-9459: ADVM/ACFS is not supported on this OS version: 'Solaris Container: Solaris 10 update 9'
      ACFS-9201: Not Supported
      Configure Oracle Grid Infrastructure for a Cluster ... succeeded
      
      Listing 13. Executing the Script
    4. Run root.sh from the remaining nodes, as shown in Listing 14.

      # /u01/grid/product/11.2.0.2/root.sh
      Running Oracle 11g root script...
      
      The following environment variables are set as:
          ORACLE_OWNER= oracle
          ORACLE_HOME=  /u01/grid/product/11.2.0.2
      
      Enter the full pathname of the local bin directory: [/usr/local/bin]: /opt/local/bin
      The contents of "dbhome" have not changed. No need to overwrite.
      The contents of "oraenv" have not changed. No need to overwrite.
      The contents of "coraenv" have not changed. No need to overwrite.
      
      Entries will be added to the /var/opt/oracle/oratab file as needed by
      Database Configuration Assistant when a database is created
      Finished running generic part of root script.
      Now product-specific root actions will be performed.
      Relinking oracle with rac_on option
      Using configuration parameter file: /u01/grid/product/11.2.0.2/crs/install/crsconfig_params
      Creating trace directory
      LOCAL ADD MODE 
      Creating OCR keys for user 'root', privgrp 'root'..
      Operation successful.
      OLR initialization - successful
      Adding daemon to inittab
      ACFS-9459: ADVM/ACFS is not supported on this OS version: 'Solaris Container: Solaris 10 update 9'
      ACFS-9201: Not Supported
      ACFS-9459: ADVM/ACFS is not supported on this OS version: 'Solaris Container: Solaris 10 update 9'
      CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node unknown, 
      number unknown, and is terminating
      An active cluster was found during exclusive startup, restarting to join the cluster
      Configure Oracle Grid Infrastructure for a Cluster ... succeeded
      
      Listing 14. Running root.sh
    5. After the root.sh script has been completed on the last node, proceed to the next step of the Oracle Universal Installer.

      Note: There will be an error: [INS-20802] Oracle Verification Utility failed. You can ignore this error by selecting OK, selecting Skip, selecting Next, and selecting Yes to the continue in spite of the [INS-32091] warning messages, as shown in Figure 11 through Figure 14.

      Figure 11

      Figure 11. Error Message

      Figure 12

      Figure 12. Click Skip Button

      Figure 13

      Figure 13. Click Yes to Continue

      Figure 14

      Figure 14. Successful Cluster Configuration

    6. From any node, check the Cluster Ready Services, as shown in Listing 15.

      # /u01/grid/product/11.2.0.2/bin/crsctl check cluster -all
      **************************************************************
      vzhost1c:
      CRS-4537: Cluster Ready Services is online
      CRS-4529: Cluster Synchronization Services is online
      CRS-4533: Event Manager is online
      **************************************************************
      vzhost2c:
      CRS-4537: Cluster Ready Services is online
      CRS-4529: Cluster Synchronization Services is online
      CRS-4533: Event Manager is online
      **************************************************************
      vzhost3c:
      CRS-4537: Cluster Ready Services is online
      CRS-4529: Cluster Synchronization Services is online
      CRS-4533: Event Manager is online
      **************************************************************
      vzhost4c:
      CRS-4537: Cluster Ready Services is online
      CRS-4529: Cluster Synchronization Services is online
      CRS-4533: Event Manager is online
      **************************************************************
      
      Listing 15. Checking the Cluster Ready Services
    7. Check the status of the resources, as shown in Listing 16.

      # /u01/grid/product/11.2.0.2/bin/crsctl status res -t
      --------------------------------------------------------------------------------
      NAME           TARGET  STATE        SERVER                 STATE_DETAILS       
      --------------------------------------------------------------------------------
      Local Resources
      --------------------------------------------------------------------------------
      ora.CRSDG.dg
                     ONLINE  ONLINE       vzhost1c                                   
                     ONLINE  ONLINE       vzhost2c                                   
                     ONLINE  ONLINE       vzhost3c                                   
                     ONLINE  ONLINE       vzhost4c                                   
      ora.LISTENER.lsnr
                     ONLINE  ONLINE       vzhost1c                                   
                     ONLINE  ONLINE       vzhost2c                                   
                     ONLINE  ONLINE       vzhost3c                                   
                     ONLINE  ONLINE       vzhost4c                                   
      ora.asm
                     ONLINE  ONLINE       vzhost1c               Started             
                     ONLINE  ONLINE       vzhost2c                                   
                     ONLINE  ONLINE       vzhost3c                                   
                     ONLINE  ONLINE       vzhost4c                                   
      ora.gsd
                     OFFLINE OFFLINE      vzhost1c                                   
                     OFFLINE OFFLINE      vzhost2c                                   
                     OFFLINE OFFLINE      vzhost3c                                   
                     OFFLINE OFFLINE      vzhost4c                                   
      ora.net1.network
                     ONLINE  ONLINE       vzhost1c                                   
                     ONLINE  ONLINE       vzhost2c                                   
                     ONLINE  ONLINE       vzhost3c                                   
                     ONLINE  ONLINE       vzhost4c                                   
      ora.ons
                     ONLINE  ONLINE       vzhost1c                                   
                     ONLINE  ONLINE       vzhost2c                                   
                     ONLINE  ONLINE       vzhost3c                                   
                     ONLINE  ONLINE       vzhost4c                                   
      --------------------------------------------------------------------------------
      Cluster Resources
      --------------------------------------------------------------------------------
      ora.LISTENER_SCAN1.lsnr
            1        ONLINE  ONLINE       vzhost1c                                   
      ora.cvu
            1        ONLINE  ONLINE       vzhost1c                                   
      ora.oc4j
            1        ONLINE  ONLINE       vzhost1c                                   
      ora.scan1.vip
            1        ONLINE  ONLINE       vzhost1c                                   
      ora.vzhost1e.vip
            1        ONLINE  ONLINE       vzhost1c                                   
      ora.vzhost2e.vip
            1        ONLINE  ONLINE       vzhost2c                                   
      ora.vzhost3e.vip
            1        ONLINE  ONLINE       vzhost3c                                   
      ora.vzhost4e.vip
            1        ONLINE  ONLINE       vzhost4c           
      
      Listing 16. Checking the Status of the Resources

    Install and Patch Oracle Database

    Note: Installing and creating the database in one step is optional. Create an Oracle Automatic Storage Management disk group for the database file if you are planning to create the database at this point.

    1. From one node, install Oracle Database 11.2.0.2, for example:

      $ export DISPLAY=<hostname>:<n>
      $ cd 10098816_112020_SOLARIS64/database
      $ ./runInstaller
      
    2. Provide input to Oracle Universal Installer:

      1. Provide the requested information on the Configure Security Updates page and the Download Software Updates page.
      2. On the Select Installation Option page, select the Install database software only option.
      3. On the Grid Installation Options page, select the Oracle Real Application Clusters database installation option.
      4. On the Select Product Languages page, select the appropriate language.
      5. On the Select Database Edition page, select the Enterprise Edition option.
      6. On the Specify Installation Location page, specify the following, as shown in Figure 15:
        • For Oracle Base, select /u01/oracle.
        • For Software Location, select /u01/oracle/product/11.2.0.2.

          Figure 15

          Figure 15. Specify Installation Location Page

      7. On the Privileged Operating System Groups page, specify the following, as shown in Figure 16:

        • For Database Administrator (OSDBA) Group, select dba.
        • For Database Operator (OSOPER) Group (Optional), select oinstall.

          Figure 16

          Figure 16. Privileged Operating System Groups Page

      8. On the Perform Prerequisite Checks page, select Ignore All.
      9. On the Summary page, check all the settings you selected to ensure they are appropriate and then click Install.

        The Execute Configuration Scripts dialog box asks you to execute root.sh, as shown in Figure 17.

        Figure 17

        Figure 17. Execute Configuration Scripts Dialog Box

    3. To complete the installation, open a terminal window and execute the script on all zone cluster nodes, as instructed, as shown in Listing 17.

      vzhost1c# /u01/oracle/product/11.2.0.2/root.sh
      Running Oracle 11g root script...
      
      The following environment variables are set as:
          ORACLE_OWNER= oracle
          ORACLE_HOME=  /u01/oracle/product/11.2.0.2
      
      Enter the full pathname of the local bin directory: [/usr/local/bin]: /opt/local/bin
      The contents of "dbhome" have not changed. No need to overwrite.
      The contents of "oraenv" have not changed. No need to overwrite.
      The contents of "coraenv" have not changed. No need to overwrite.
      
      Entries will be added to the /var/opt/oracle/oratab file as needed by
      Database Configuration Assistant when a database is created
      Finished running generic part of root script.
      Now product-specific root actions will be performed.
      chmod: WARNING: can't change /u01/oracle/product/11.2.0.2/lib/libskgxn2.so
      Finished product-specific root actions
      
      Listing 17. Executing the Script

    Apply Oracle Grid Infrastructure 11.2.0.2.3 GI PSU patch to Oracle Database Using OPatch

    Apply patch 11.2.0.2 GIB3 to each zone cluster node by performing the following steps.

    1. Replace OPatch with the latest version, 11.2.0.1.6.
    2. Back up the OPatch version in DB_HOME, for example:

      $ cd /u01/oracle/product/11.2.0.2/ 
      $ mv OPatch OPatch.bak
      
    3. As the software owner, copy p6880880_112000_SOLARIS64.zip to DB_HOME:

      $ cp p6880880_112000_SOLARIS64.zip /u01/oracle/product/11.2.0.2
      $ cd /u01/oracle/product/11.2.0.2
      $ unzip p6880880_112000_SOLARIS64.zip
      
    4. As the software owner, create an Oracle Configuration Manager file on each node, for example:

      $ cd /var/tmp
      $ /u01/oracle/product/11.2.0.2/OPatch/ocm/bin/emocmrsp 
      
      OCM Installation Response Generator 10.3.4.0.0 - Production
      Copyright (c) 2005, 2010, Oracle and/or its affiliates.  All rights reserved.
      
      Provide your email address to be informed of security issues, install and
      initiate Oracle Configuration Manager. Easier for you if you use your My
      Oracle Support Email address/User Name.
      Visit http://www.oracle.com/support/policies.html for details.
      
    5. Provide your e-mail address and then press Enter to bypass providing a password, for example:

      Email address/User Name: <your email address>
      Provide your My Oracle Support password to receive security updates via your 
      My Oracle Support account.
      Password (optional):            
      The OCM configuration response file (ocm.rsp) was successfully created.
      
    6. Execute the following as root:

      # /u01/oracle/product/11.2.0.2/OPatch/opatch auto /var/tmp/gib3 -oh /u01/oracle/product/11.2.0.2
      
      Executing /usr/bin/perl /u01/oracle/product/11.2.0.2/OPatch/crs/patch112.pl 
      -patchdir /var/tmp -patchn gib3 -oh /u01/oracle/product/11.2.0.2 -paramfile 
      /u01/grid/product/11.2.0.2/crs/install/crsconfig_params
      opatch auto log file location is 
      /u01/oracle/product/11.2.0.2/OPatch/crs/../../cfgtoollogs/opatchauto2011-08-10_20-30-35.log
      Detected Oracle Clusterware install
      Using configuration parameter file: /u01/grid/product/11.2.0.2/crs/install/crsconfig_params
      OPatch  is bundled with OCM, Enter the absolute OCM response file path:
      
    7. Provide the full path to the ocm.rsp file created earlier, for example:

      /var/tmp/ocm.rsp
      
    8. As the software owner, verify the patch installation, as shown in Listing 18.

      $ export ORACLE_HOME=/u01/oracle/product/11.2.0.2  
      $ /u01/oracle/product/11.2.0.2/OPatch/opatch lsinventory -oh $ORACLE_HOME
      
      Invoking OPatch 11.2.0.1.6
      
      Oracle Interim Patch Installer version 11.2.0.1.6
      Copyright (c) 2011, Oracle Corporation.  All rights reserved.
      
      
      Oracle Home       : /u01/oracle/product/11.2.0.2
      Central Inventory : /u01/oraInventory
         from           : /var/opt/oracle/oraInst.loc
      OPatch version    : 11.2.0.1.6
      OUI version       : 11.2.0.2.0
      Log file location : /u01/oracle/product/11.2.0.2/cfgtoollogs/opatch/opatch2011-08-11_10-38-51AM.log
      
      Lsinventory Output file location : 
      /u01/oracle/product/11.2.0.2/cfgtoollogs/opatch/lsinv/lsinventory2011-08-11_10-38-51AM.txt
      
      --------------------------------------------------------------------------------
      Installed Top-level Products (1): 
      
      Oracle Database 11g                                                  11.2.0.2.0
      There are 1 products installed in this Oracle Home.
      
      
      Interim patches (2) :
      
      Patch  12419331     : applied on Wed Aug 10 20:53:24 PDT 2011
      Unique Patch ID:  13850636
         Created on 12 Jul 2011, 07:01:28 hrs PST8PDT
         Bugs fixed:
           10151017, 10158965, 11724916, 10190642, 12586486, 12586487, 10129643
           12586488, 12586489, 10018789, 9744252, 10248523, 9956713, 10356513
           9715581, 9770451, 10378005, 10170431, 10425676, 10222719, 10126094
           9591812, 10127360, 10132870, 10094201, 9443361, 10193846, 11664046
           11069199, 10324294, 10245086, 12586490, 10205230, 12586491, 10052141
           12586492, 12586493, 12586494, 10142788, 11818335, 11830776, 12586495
           9905049, 11830777, 12586496, 11830778, 6892311, 10040921, 10077191
           10358019, 12431716, 10219576, 10258337, 11707699, 10264680, 10209232
           11651810, 10102506, 11067567, 9881076, 10278372, 10040531, 10621169
           10155605, 10082277, 10356782, 10218814, 9078442, 9788588, 10157249
           9735237, 10317487, 12326246, 11707302, 10310299, 10636231, 10230571
           11065646, 12419321, 10368698, 10079168, 10013431, 10228151, 10233732
           10324526, 8223165, 10238786, 10217802, 10061015, 9953542, 9572787
           10052956, 10080579, 11699057, 12620422, 10332111, 10227288, 10329146
           10332589, 10110863, 10073683, 9869401, 10019218, 10229719, 11664719
           9539440, 10373381, 9735282, 9748749, 11724984, 10022980, 10411618
           11800854, 12419331, 11674485, 10187168, 6523037, 10648873, 9724970
           10053725, 10084145, 10367188, 11800170, 11695285, 10157402, 9651350, 10299224
      
      Patch  12419353     : applied on Wed Aug 10 20:37:44 PDT 2011
       Patch ID:  13850636
         Created on 18 Jul 2011, 05:19:12 hrs UTC
         Bugs fixed:
           12419353, 10157506, 10178670, 10425672, 12311357, 9959110, 10272615
           10314123, 10014392, 10089120, 10057296, 9864003, 11775080, 9916145
           10044622, 12399977, 12421404, 12340700, 10056713, 10637741, 9939306
           9902536, 10007185, 10376847, 10038791, 11741224, 11655840, 10048487
           10322157, 10260251, 10052721, 10028235, 10027079, 10357258, 10045436
           10231906, 10622973, 9891341, 10072474, 10036834, 10029900, 9974223
           10016083, 9918485, 11781515, 10040647, 10069541, 10029119, 10233159
           12332919, 9812956, 10036193, 10015210, 12340501, 10621175, 11877079
           8906163, 10111010, 10115514, 10104377, 10057680, 10280665, 10078086
           9944948, 10146768, 10052529, 10011084, 10012319, 10073075, 10233811
           10299006, 10248739, 10236074, 10128191, 11071429, 10019726, 9975837
           10253630, 9949676, 11936945, 10637483, 10157622, 11698552, 10385838
           10053985, 10425674, 9812970, 11828633, 11899801, 10083789, 9876201
           10073372, 9963327, 11077756, 10375649, 9336825, 11682409, 10062301
           10018215, 10105195, 10419987, 10071992, 10634513, 9926027, 10103954
           10028343, 11866171, 10065216, 9907089, 9897335, 10190153, 11744313
           10175855, 10284828, 10028637, 10361177, 9979706, 10324594, 10015460
           9971646, 11782423, 11654726, 9978765, 10398810, 11904778, 10397652
           9915329, 10107380, 10110969, 10305361, 10331452, 10083009, 10631693
           10008467, 10048027, 10040109, 9944978, 10033106, 9978195, 11840629
           10042143, 10284693, 10638381, 9679401, 11663339, 10075643, 10205290
           10124517, 11069614, 9593552, 10168006, 12677816, 11807012
      
      
      
      Rac system comprising of multiple nodes
        Local node = vzhost1c
      
        Remote node = vzhost2c
        Remote node = vzhost3c
        Remote node = vzhost4c
      
      Listing 18. Verifying the Patch Installation

    Create the Database

    From one zone cluster node, as the Oracle software owner, create an Oracle ASM disk group for a test database called swb, as follows.

    1. Execute the following commands to launch the ASM Configuration Assistant:

      $ export DISPLAY=<hostname>:<n>
      $ export ORACLE_HOME=/u01/oracle/product/11.2.0.2
      $ /u01/grid/product/11.2.0.2/bin/asmca
      
    2. In the ASM Configuration Assistant, do the following:

      1. In the Disk Groups tab, click Create.
         
      2. Do the following in the Create Disk Group page, as shown in Figure 18.

        • Specify swbdg for Disk Group Name.
        • Select /dev/did/rdsk/d9s6, /dev/did/rdsk/d10s6, /dev/did/rdsk/d14s6, and /dev/did/rdsk/d15s6.
        • For Failure Group, specify fgr1 for /dev/did/rdsk/d9s6 and /dev/did/rdsk/d10s6.
        • For Failure Group, specify fgr2 for /dev/did/rdsk/d14s6 and /dev/did/rdsk/d15s6.
        • Click OK to create the disk group.
        • After the swbdg disk group creation is complete, click Exit to close the ASM Configuration Assistant.

          Figure 18

          Figure 18. Create Disk Group Page

    3. Start the Database Configuration Assistant, for example:

      $ u01/oracle/product/11.2.0.2/bin/dbca
      
    4. In the Database Configuration Assistant, do the following:

      1. Select the Oracle Real Application Clusters (RAC) database option.
      2. On the Operations page, select the Create a Database option.
      3. On the Database Templates page, select the Custom Database option.
      4. On the Database Identification page, click Select All and make the following selections, as shown in Figure 19:

        • For Configuration Type, select Admin-Managed.
        • For Global Database Name, specify swb.
        • For SID Prefix, specify swb.

          Figure 19

          Figure 19. Database Identification Page

      5. On the Management Options page, make sure Configure Enterprise Manager and Enable automatic maintenance tasks are not selected, as shown in Figure 20 and Figure 21.

        Figure 20

        Figure 20. Enterprise Manager Tab

        Figure 21

        Figure 21. Automatic Maintenance Tasks Tab

      6. On the Database Credentials page, select the Use the Same Administrative Password for All Accounts option.
      7. On the Database File Locations page, select the Use Oracle-Managed Files option and specify +SWBDG for Database Area.
      8. On the Recovery Configuration page, make the following selections, as shown in Figure 22:

        • Select the Specify Fast Recovery Area checkbox.
        • For Fast Recovery Area, select +SWBDG.
        • For Fast Recovery Area Size, select 7527 M Bytes.

          Figure 22

          Figure 22. Recovery Configuration Page

      9. On the Database Content page, use the default selections, as shown in Figure 23.

        Figure 23

        Figure 23. Database Content Page

      10. On the Initialization Parameters page, use the default values, as shown in Figure 24, and then click Finish to start the database creation.

        Figure 24

        Figure 24. Initialization Parameters page

    5. Verify that database was created and is running:

      $ /u01/grid/product/11.2.0.2/bin/srvctl status database -d swb
      Instance swb1 is running on node vzhost1c
      Instance swb2 is running on node vzhost2c
      Instance swb3 is running on node vzhost3c
      Instance swb4 is running on node vzhost4c
      
    6. To add the second standby NIC of the IPMP group to Cluster Ready Services, as root from one zone cluster node, run the following command:

      $ GRID_HOME/bin/srvctl modify nodeapps -n <hostname> -A <VIP address>/<subnet>/NIC0\|NIC1
      

      For example:

      $ srvctl modify nodeapps -n vzhost1c -A  10.134.35.100/255.255.255.0/e1000g0\|e1000g1
      
    7. Use srvctl config to show that e1000g1 was added:

      $ /u01/grid/product/11.2.0.3/bin/srvctl config nodeapps
      Network exists: 1/10.134.35.0/255.255.255.0/e1000g0:e1000g1, type static
      VIP exists: /vzhost1e/10.134.35.100/10.134.35.0/255.255.255.0/e1000g0:e1000g1, hosting node vzhost1c
      VIP exists: /vzhost2e/10.134.35.101/10.134.35.0/255.255.255.0/e1000g0:e1000g1, hosting node vzhost2c
      VIP exists: /vzhost3e/10.134.35.102/10.134.35.0/255.255.255.0/e1000g0:e1000g1, hosting node vzhost3c
      VIP exists: /vzhost4e/10.134.35.103/10.134.35.0/255.255.255.0/e1000g0:e1000g1, hosting node vzhost4c
      

    Create the Oracle Solaris Cluster Resource

    Use the following procedure to create the Oracle Solaris Cluster Resource. Alternatively, you can use clsetup.

    1. From one zone cluster node, execute the following command to register the SUNW.crs_framework resource type in the zone cluster:

      # clrt register SUNW.crs_framework
      
    2. Add an instance of the SUNW.crs_framework resource type to the Oracle RAC framework resource group:

      # clresource create -t SUNW.crs_framework \
        -g rac-framework-rg \
        -p resource_dependencies=rac-framework-rs \
        crs-framework-rs
      
    3. Register the scalable Oracle Automatic Storage Management instance proxy resource type:

      # clresourcetype register SUNW.scalable_asm_instance_proxy
      
    4. Register the Oracle Automatic Storage Management disk group resource type:

      # clresourcetype register SUNW.scalable_asm_diskgroup_proxy
      
    5. Create resource groups asm-inst-rg and asm-dg-rg:

      # clresourcegroup create -S asm-inst-rg asm-dg-rg
      
    6. Set a strong positive affinity on rac-fmwk-rg by asm-inst-rg:

      # clresourcegroup set -p rg_affinities=++rac-framework-rg asm-inst-rg
      
    7. Set a strong positive affinity on asm-inst-rg by asm-dg-rg:

      # clresourcegroup set -p rg_affinities=++asm-inst-rg asm-dg-rg
      
    8. Create a SUNW.scalable_asm_instance_proxy resource and set the resource dependencies:

      # clresource create asm-inst-rg \
        -t SUNW.scalable_asm_instance_proxy \
        -p ORACLE_HOME=/u01/grid/product/11.2.0.2 \
        -p CRS_HOME=/u01/grid/product/11.2.0.2 \
        -p "ORACLE_SID{vzhost1c}"=+ASM1 \
        -p "ORACLE_SID{vzhost2c}"=+ASM2 \
        -p "ORACLE_SID{vzhost3c}"=+ASM3 \
        -p "ORACLE_SID{vzhost4c}"=+ASM4 \
        -p resource_dependencies_offline_restart=crs-framework-rs \
        -d asm-inst-rs
      
    9. Add an Oracle Automatic Storage Management disk group resource type to the asm-dg-rg resource group:

      # clresource create -g asm-dg-rg -t SUNW.scalable_asm_diskgroup_proxy \
        -p asm_diskgroups=CRSDG,SWBDG \
        -p resource_dependencies_offline_restart=asm-inst-rs \
        -d asm-dg-rs
      
    10. On a cluster node, bring the asm-inst-rg resource group online in a managed state:

      # clresourcegroup online -eM asm-inst-rg
      
    11. On a cluster node, bring the asm-dg-rg resource group online in a managed state:

      # clresourcegroup online -eM asm-dg-rg
      
    12. Create a scalable resource group to contain the proxy resource for the Oracle RAC database server.

      # clresourcegroup create -S \ 
        -p rg_affinities=++rac-framework-rg,++asm-dg-rg \ 
        rac-swbdb-rg
      
    13. Register the SUNW.scalable_rac_server_proxy resource type:

      # clresourcetype register SUNW.scalable_rac_server_proxy
      
    14. Add a database resource to the resource group:

      # clresource create -g rac-swbdb-rg \
        -t SUNW.scalable_rac_server_proxy \
        -p resource_dependencies=rac-framework-rs \
        -p resource_dependencies_offline_restart=crs-framework-rs,asm-dg-rs \
        -p oracle_home=/u01/oracle/product/11.2.0.2 \
        -p crs_home=/u01/grid/product/11.2.0.2 \
        -p db_name=swb \
        -p "oracle_sid{vzhost1c}"=swb1 \
        -p "oracle_sid{vzhost2c}"=swb2 \
        -p "oracle_sid{vzhost3c}"=swb3 \
        -p "oracle_sid{vzhost4c}"=swb4 \
        -d rac-swb-srvr-proxy-rs
      
    15. Bring the resource group online:

      # clresourcegroup online -emM rac-swbdb-rg
      
    16. Check the status of the cluster resource, as shown in Listing 19:

      # clrs status
      
      === Cluster Resources ===
      
      Resource Name             Node Name         State      Status Message	
      -------------            ---------         -----       -------------------
      crs_framework-rs          vzhost1c          Online     Online
                                vzhost2c          Online     Online
                                vzhost3c          Online     Online
                                vzhost4c          Online     Online
      
      rac-framework-rs          vzhost1c          Online     Online
                                vzhost2c          Online     Online
                                vzhost3c          Online     Online
                                vzhost4c          Online     Online
      
      asm-inst-rs               vzhost1c          Online     Online - +ASM1 is UP and ENABLED
                                vzhost2c          Online     Online - +ASM2 is UP and ENABLED
                                vzhost3c          Online     Online - +ASM3 is UP and ENABLED
                                vzhost4c          Online     Online - +ASM4 is UP and ENABLED
      
      asm-dg-rs                 vzhost1c          Online     Online - Mounted: SWBDG
                                vzhost2c          Online     Online - Mounted: SWBDG
                                vzhost3c          Online     Online - Mounted: SWBDG
                                vzhost4c          Online     Online - Mounted: SWBDG
      
      rac-swb-srvr-proxy-rs     vzhost1c          Online     Online - Oracle instance UP
                                vzhost2c          Online     Online - Oracle instance UP
                                vzhost3c          Online     Online - Oracle instance UP
                                vzhost4c          Online     Online - Oracle instance UP
      
      Listing 19. Checking the Status of the Cluster Resource

    See Also

    Here are some additional resources:

    About the Author

    Vinh Tran is a Quality Engineer in the Oracle Solaris Cluster Group. His responsibilities include, but are not limited to, certification and qualification of Oracle RAC on Oracle Solaris Cluster.

    Revision 2.0, 06/06/2014
    Revision 1.0, 05/09/2012

    See sysadmin-related content for all Oracle technologies by following OTN Systems on Facebook and Twitter.