by Porus Homi Havewala
Learn how to seamlessly provision multi-node Oracle RAC systems in a totally automated manner - including provisioning of RAC databases
Published January 2011
Everyone in IT knows about Oracle Real Application Clusters (RAC), Oracle’s leading-edge active-active database cluster system – one fantastically scalable database with multiple instances on many servers, accessing a shared storage system, preferably using Oracle Automatic Storage Management (ASM). In Oracle’s visionary idea of the Grid, clusters of Oracle RAC database servers and Oracle ASM storage, along with Oracle WebLogic Server clusters, are used to create a dynamic grid of shared resources that can be re-distributed on demand to any corporate-wide application service.
Put it all in a box, and you have Oracle Exadata for the database and storage grid, and the Oracle Exalogic Elastic Cloud for the application grid. And in this vision, the Grid is managed centrally by a singular management interface: Oracle Enterprise Manager Grid Control.
Historically, the word “easy” has seldom been used in the context of active-active database clusters. Setting up clusters and Oracle RAC has always been a challenging and tedious procedure when OS clusters such as the Sun Cluster (or other OS-specific clusters) are used as the clustering foundation. The set up of such clusters necessitated involvement by different teams: such as the storage team (for configuring shared storage), the UNIX team (for installing the OS and the OS-specific Cluster), and the database team (for installing the actual database software and creating the RAC database). Getting these teams to work together to install a single multi-node RAC cluster was a major project.
Over the last several releases, however, Oracle RAC installation has steadily simplified. Third-party clustering software is no longer required (it was replaced by Oracle’s own clusterware system), and storage setup is vastly easier thanks to Oracle ASM. Using that technology, the DBA team can now do many activities that used to be done by the storage team -- and some things, such as automatic rebalancing of storage hot spots, are even fully automated. In the current version of the RAC cluster as of this writing (11.2), installation and configuration is even easier. And with the “Grid-in-a-box” and “Cloud-in-a-box” features of Oracle Exadata and Oracle Exalogic respectively, things are even more streamlined because everything, including storage, is self-contained.
Even with these enhancements, until recently, setting up a RAC system manually using Oracle Universal Installer was a multi-step, intensive, repetitive process that included:
Installing and configuring Oracle Clusterware in a separate home
Virtual IP setup
Installing ASM software in a separate home (from Oracle RAC 11g onward, ASM software is installed along with the clusterware in the new Grid Infrastructure home)
ASM setup of disk groups for the database and flash recovery area
Database software install in a separate home
Database and services setup.
Today, Oracle Enterprise Manager 11g Grid Control further simplifies and automates this process by letting you create a Gold Copy from an existing RAC installation and then using it to provision new target RAC systems of any number of nodes, with even the repetitive inputs to the Enterprise Manager Provisioning interview process being captured and saved in RAC provisioning profiles (in Oracle RAC 11g Release 2).
Oracle Enterprise Manager is Oracle’s Enterprise management system, used for managing the entire Oracle stack – from applications to disk. The Oracle Enterprise Manager Provisioning and Patch Automation Management Pack, once licenced, allows the provisioning of the entire software stack including the operating system, middleware, Oracle Database Homes, Oracle ASM Homes as well as Oracle Clusterware Homes [Grid Infrastructure Homes in the case of 11g Release 2]. You can also clone single-instance databases, or multi-instance [RAC] databases.
The Oracle Clusterware, Oracle ASM and Oracle Database software can be installed and configured in a series of automated steps known as Oracle Enterprise Manager deployment procedures that can be used out-of-the-box, or customised if need be. In this technical article, you will learn how to use these deployment procedures for provisioning a complex Oracle RAC database with the utmost ease and minimum of effort. Provisioning of single-instance database servers is described in detail in my article “Manage Mass Provisioning Using Oracle Enterprise Manager Grid Control”, which you should read to gain an understanding of the Oracle Enterprise Manager Software Library and Deployment Procedure concepts before proceeding further.
In the current (Oracle Enterprise Manager 11g Grid Control Release 1, at the time of writing) and previous release (10g Release 5), the steps you follow are generally the same for provisioning an Oracle RAC 10g or RAC 11g Release 1 (11.1) database. However, for provisioning a RAC 11g Release 2 (11.2) database, you must use Oracle Enterprise Manager 11g Grid Control Release 1; the 10g release of Grid Control is not certified to provision RAC 11.2 because of the new Grid Infrastructure Home in this RAC version. There are also other enhancements, such as RAC provisioning profiles, in the 11g version of Grid Control that we will examine in detail later.
Provisioning Oracle RAC databases using the Provisioning/Deployment Procedure functionality of Oracle Enterprise Manager involves the following high-level steps:
Selecting the appropriate Provisioning Procedure
Answering simple interview questions in the procedure:
Select the source
Select the target hosts
Specify the credentials for the hosts
Schedule the actual provisioning
Configure the cluster: name, location of homes
Specify if also creating starter database
Configure the shared storage
Specify advanced configuration for the cluster such as a bonding interface
We will now go through these steps one by one.
Creating the Gold Image
Login to the Enterprise Manager Grid Control console, move to the Deployments tab, and click on Provisioning. This brings up the components to be included in the Software Library (Figure 1).
Figure 1 Selecting components
(Note that the Provisioning tab will be accessible only if your login to Grid Control has super-admin rights, such as the SYSMAN user. This is to prevent non-superadmin users from changing the Gold Images in the software library.)
You must have initially configured your software library before you can use it. The steps to do so are outlined in the Oracle Enterprise Manager Administrator's Guide for Software and Server Provisioning and Patching (11g Release 1) under the section "Setting Up Oracle Software Library".
These are simple steps and involve setting up a shared location for the software library if there are multiple Oracle Enterprise Manager management services (OMS), and then entering the software library details under the Administration tab visible in Figure 1.
Notice the RAC provisioning profiles prominently displayed on this page; we will examine these later on.
Expanding “Oracle Components” and then “RAC Provisioning” in the tree list of items above, we can see the RAC Provisioning components (Figure 2).
Figure 2 RAC Provisioning components
Installation Media for 10g can be uploaded to under the 10.2.0.1.0 sub-component, that for 11g Release 1 can be uploaded to under 184.108.40.206.0 and the 11g Release 2 media to under 220.127.116.11.0.
Expanding the 18.104.22.168.0 component now displays all the platforms available for the software (Figure 3). The linux32 component is also expanded to show the Oracle Clusterware shiphome, Oracle Database shiphome and the Oracle Database Replay Client Install media.
Figure 3 Available platforms
If you expand the 22.214.171.124.0 component and the linux32 component underneath it, the components shown are slightly different: there is a Oracle Grid Infrastructure Shiphome instead of the Oracle Clusterware shiphome. Both the sets of components are seen in Figure 3.
As discussed in the single-instance database provisioning article, the Oracle Replay Client is a program that is available as part of the standard Oracle Client or the Oracle Instant Client. It is a multithreaded executable “wrc” – each thread responsible for submitting a captured session's workload. This is used for the Database Replay feature of the Oracle Real Application Testing Option (licensable) for Oracle Database Enterprise Edition.
Currently the status of the components can be seen as “incomplete”, which indicates that these components cannot be used for provisioning at this stage since the binaries have not been uploaded yet. So the next step is to upload the installation media for the Oracle Clusterware as well as the Oracle database for this version.
For the sake of this exercise, we will provision an 11.1 RAC database, so the steps and screens will be similar in Enterprise Manager Grid Control 10g as well as Grid Control 11g.
Under 126.96.36.199.0 and linux32, select the radio button next to the Oracle Clusterware shiphome component, and click on Edit in the list of buttons at the top of the table. This brings up the Edit Component Page (Figure 4).
Figure 4 Edit Component Page
Pre-filled values are visible on this page since this is a default component that has been supplied out of the box; however it is possible to edit and change the values.
To upload the sofware binaries, move to the Upload File tab (Figure 5). This shows us that there is no currently associated file. You can now decide where to upload the binaries from, use “Upload from Agent Machine” to specify another host on which the binaries are loaded, such as one of the target servers.
Figure 5 Upload File tab
You have selected one of the target hosts and also specified the actual directory on the host where the installation zip file exists. When you click on the Finish button on this page, a job is created and scheduled to upload the actual file into this component of the Software Library (Figure 6).
Figure 6 Uploading the installation zip file to Software Library
You can check on the execution of the job by clicking on the Job Execution ID link. The job needs to complete successfully before you can proceed with the provisioning, and it may take some time to complete since it is uploading a large file to the shared location of the Software library.
You can now repeat the same process for the Oracle Database Shiphome. Select this component on the Components page (Figure 3), and click on Edit. Move to the Upload File tab, and specify a host and file name for the 11.1 database zip file (Figure 7). This too will create and schedule a job to upload the database zip file to the Software Library. Wait for both jobs to finish.
Figure 7 Specifying a host and file name for the 11.1 database zip file
Once the jobs have completed, you can verify if the Gold images uploaded are ready for provisioning.
To do this, move to the Deployments Provisioning tab and check the component status for 188.8.131.52.0 and linux32. The status can be seen as “Ready” with a revision version of 0.2 (Figure 8). The revision version increases in this way on any component change or upload.
Figure 8 Component status: Ready
Scheduling the Deployment Procedure
You are now ready to provision a RAC database. From the Deployments, select RAC Provisioning Procedures. This brings up a collection of Deployment Procedures supplied out-of-the-box by Enterprise Manager, along with customized procedures if any have been created.
From this list, select the “Oracle Clusterware/RAC Provisioning for Unix” procedure and click on the Schedule Deployment button (Figure 9). (We should note at this point that we are not selecting the “Oracle Grid Infrastructure/RAC Provisioning” procedure, which is only for provisioning 11.2 RAC systems.)
Figure 9 Schedule Deployment
This now commences the interview process for the Deployment Procedure (Figure 10).
Figure 10 Deployment Procedure interview process
In a series of steps shown in a linear form across the top of the page, you need to answer questions by the deployment procedure such as selecting the source for the provisoning, selecting the target hosts, specifying credentials and schedule, configuring the cluster and storage, and so on.
In the first step of the interview, you need to select the Source for the provisioning process. The source can either be from an existing Oracle installation or from the Software Library. Thus you can set up a reference RAC installation that is fully patched, and easily clone it to new target RACs. All you have to do is select the reference host on this screen, and specify the home names and locations of the clusterware, RAC, and ASM installations on that host. Ticking “Save to Software Library” will also simultaneously copy the reference installation to the software library while cloning it to the target servers. You can then reuse this fully patched Gold Copy stored in the software library, when provisioning more RAC systems in the future.
In this case we have already uploaded the installation software, so we will select the Software Library and also specify the Clusterware and Database shiphome sources under the Software Library (Figure 11).
Figure 11 Specifying Clusterware and Database shiphome sources
On this page, you must select the clusterware and database shiphome components for exactly the same version and the same platform from the software library or you will get an error. In our case we have selected both clusterware and database shiphome components as 184.108.40.206.0 and for linux32.
Notice the section “Source for ASM”. The default setting is not to provision ASM. If the second option “Use the above selected RAC component as the source” is selected, the Database component selected under “Source for RAC” will also be used for ASM provisioning. The third option is “Choose a component”, which allows you to select the ASM component directly from the Software Library.
Click Next to go to the next step of the Interview process. You are now on the Select Hosts step (Figure 12), which allows you to select the target hosts for the RAC provisioning.
Figure 12 Selecting hosts
Use the Add button to add suitable hosts to your list of targets for provisioning. The Public and Virtual Host Names of any such host added can be seen in the list. For demonstration purposes, we have selected only one host on this page.
You can expand the “Show Options” in the host list, this will show further details about each host such as private host name, private host IP, and virtual host IP (Figure 13). The private host name and virtual host name are pre-filled by default with the appropriate suffix to the hostnames, but it is possible to change these details as required.
Figure 13 Expanded “Show Options” in the host list
We should note at this point that only hosts that are managed by management agent 10.2.0.4 onward can be used with the RAC Provisioning deployment procedure. Although the screens shown in the images indicate the earlier agent version of 10.2.0.2, the requirement is actually agent version 10.2.0.4 and not 10.2.0.2. So if you have an installation where you have not upgraded the agents to the latest version of your OMS, the agents will have to be upgraded before you can use this type of provisioning. It is always a good idea to upgrade your agent to the same version as your OMS anyway due to increased functionality and fixes in each version. (An updated list of OMS and Agent Patches required for setting up Provisioning, Patching and Cloning in 10.2 to 11.1 Grid Control can be found in My Oracle Support Note 427577.1. Refer to this My Oracle Support Note at all times, regardless of the versions you are using.)
You can use a Cluster configuration file by clicking on the Import from File button--this configuration file contains information about the cluster name and node names and would be normally used to supply information to the Oracle Universal Installer in file format rather than having to type in everything. This makes perfect sense in a large cluster, say with 4 or more nodes.
Click on the Select Interfaces button on this page to configure the public and private network interfaces. This brings up a popup screen (Figure 14) in which you can select the appropriate interface from a list of existing network interfaces at the host level.
Figure 14 Selecting public and private network interfaces
This information is what you would normally enter when installing the Oracle Clusterware software using Oracle Universal Installer. So, this is where Enterprise Manager collects such information – in the interview process – and then supplies it to the installer in a silent install.
After you select the correct public and private interfaces in this popup screen, the information about the interfaces is now populated and visible under the section “Network Interface Configuration” on the Select Hosts page.
Proceed to the next step, where you need to supply the Credentials to log in to the hosts (Figure 15).
Figure 15 Supplying credentials
It is possible to supply different host credentials for different Oracle Homes, such as a different UNIX users for Clusterware, RAC (database), or the Agent Home. On the same page, you can also specify the schedule of implemention of this procedure.
Move to the next step of the interview process: the “Configure Cluster” step (Figure 16).
Figure 16 Configure Cluster
On this page, you can specify the Cluster Name and also the location details about the Cluster Home and Database Home, information that will be used during the creation of these Homes. (It is recommended to use a different Oracle Base for the clusterware and database homes.)
Moving down on the same page, under the Database Details section, you can specify whether a starter database is to be created. If you decide to create a starter database, you need to specify the global database name, the password to be used in the database, and database parameters to be set as required.
You can also select the template file for the database, by default the General Purpose template is used (General_Purpose.dbc) – however you can specify any other template that contains different database parameters, such as the Transaction Processing template (Transaction_Processing.dbc).
If you do not select the option to create a starter database, a software-only install will be performed. By including the database creation, you can provision a full working RAC system using the Deployment procedure – including the clusterware, ASM, database homes as well as the actual database itself with the corresponding instances on each node. In this case there is no need to run the Oracle Database Creation Assistant (dbca) separately after the software installation and configuration is complete.
Under the Backup and Recovery details on the same page, it is also possible to enable automated backups so that a default Oracle-suggested RMAN backup strategy for the RAC database is set up during the process of provisioning the RAC system.
The next page (Figure 17) displays the Shared Storage Configuration.
Figure 17 Shared Storage Configuration
In this step, you need to specify three mandatory locations – for the Oracle Cluster Registry (OCR), the Voting Disk, and the Oracle database files.
You can select the appropriate partition name, the mount location, and the mount format – whether nfs, ocfs, ocfs2 or raw.
This page automatically displays the partitions available on the target box, but it is possible to add your own partitions instead of the existing partitions listed in this table.
If you wanted to use raw devices, then you would have pre-configured the raw devices for the OCR and its mirror, the three voting disks (VDSK, VDSK1 and VDSK2 as seen in Figure 17), and the Oracle Automatic Storage Management (ASM) disk and ASM spfile before coming to this point in the procedure.
After this step, you move to the Advanced Configuration page (Figure 18).
Figure 18 Advanced Configuration page
On this page you can configure a Bonding Interface in your RAC system – this allows you to bond two network cards and use them as a single interface, with the capabilities of failover from one network card to the other. This is important for improving availability in your RAC system.
The other possible benefit is balancing the network traffic between the two network cards in your bonding interface, this is seen on the page where you can specify the bonding mode required such as balance or active backup.
Clicking on Next, you now move to the Configure Oracle Home page. The Custom Step parameters page is skipped since you have no custom steps in your Deployment Procedure – but it is always possible to make a copy of an out-of-the-box procedure and insert custom steps.
On the Configure Oracle Home page (Figure 19), which is the final step in the interview process, you simply specify the security update details and proxy server information if any, for your connection to the internet if you are using the security update feature. This is highly recommended.
Figure 19 Configure Oracle Home page
When you click Next, the review page appears (Figure 20). On top of the review page, certain informational items are also shown which you should carefully go through.
Figure 20 Review page
For example, the user id and group id should be the same on all nodes of the cluster for the operating system users such as oracle or crsuser that will be used to perform the installation on the nodes. The other instructions can be seen on the page displayed in Figure 20.
Underneath these instructions, you can review your actual input in the interview process, such as the sources for your clusterware and database software, the hosts you have selected, the schedule for the provisioning and the configuration details for the cluster and database.
When you have finished reviewing your input, and have also made sure that the requirements in the information points have been satisfied, click on the Finish button at the top of the page.
This immediately submits the RAC provisioning deployment procedure as an Enterprise Manager Job, and the job will then execute at the scheduled time – either immediately or in the future, per what you have specified (Figure 21). The job can be seen initially as “Scheduled” under the Status Detail section, which will then change to “Running”.
Figure 21 Job schedule
Once the job is running, you can move to the Job Details tab that shows all the detailed tasks that actually perform the provisioning (Figure 22).
Figure 22 Job details
Moving back to the Steps tab, it is possible to drill down on any of the job steps as seen in Figure 23, and see the success or failure of each step.
Figure 23 Job step results
For example, drilling down on the “RAC User Permission Checks” step, we can see the details and even the success log for the indiviual steps until you come to the final job output for this step (Figure 24).
Figure 24 Detailed job output
This helps you to understand what the job is doing, and any reason in case it has failed.
In the same manner, you could simultaneously submit a RAC provisioning job for Solaris (provided you have uploaded the binaries) and therefore you could have multiple provisioning jobs running in Enterprise Manager at any one point of time, as seen in Figure 25.
Figure 25 Multiple provisioning jobs
When the RAC Provisioning procedures complete, you have two fully provisioned RAC clusters – with all the appropriate clusterware, ASM, database software and even a RAC database provisioned and ready for your data consolidation.
Provisioning Profiles for 11g Release 2 RAC systems
In the case of Oracle Enterprise Manager Grid Control 11g Release 1, when you are provisioning an 11g Release 2 (11.2) RAC system, at the every end of your interview process, you can save all the inputs in the interview as a Provisioning Profile.
The Provisioning Profile you have saved is then visible under Deployments..Provisioning..Components, under RAC Provisioning Profiles. The type is seen to be a configuration template (Figure 26).
Figure 26 RAC Provisioning Profiles
When you schedule the deployment procedure “Oracle Grid Infrastructure/RAC Provisioning” for provisioning 11g Release 2 RAC systems, the very first step in the interview process is seen to be “Select Provisioning Profile” (Figure 27). This lets you select an existing profile for all your inputs.
Figure 27 Interview process for Oracle Grid Infrastructure/RAC Provisioning
The advantage is that you do not have to enter the same variables again and again in all your provisioning, saving time and avoiding typing mistakes as well.
As an example, selecting the very first provisioning profile in the list “Admin Managed RAC database using ASM storage” prefills the first step of the interview process, as shown in Figure 28. The destination host has been manually selected and not prefilled.
Figure 28 Interview process for Admin Managed RAC database using ASM storage
The list of Provisioning Profiles displayed in Figure 27 has in fact also introduced to us a new concept in 11g Release 2 RAC databases: Policy Managed RAC databases as opposed to Admin Managed RAC databases (the older and prevalent technology).
The Policy Managed RAC databases use the concept of server pools. You can define and use policies for the clusterware to automatically move nodes from the free pool or even the pools whose importance is low, to the pools whose importance is high. This new feature of Policy Management and Server Pools advances the technology of Oracle RAC one step closer to the goal of the self-balancing and auto-adjusting all-powerful Grid of database servers.
We have seen how easy it is to provision RAC systems using Oracle Enterprise Manager Grid Control.
Using the convenience of a centralized Software Library in which the Gold Copies of Oracle RAC software for different platforms can be stored, it is possible to seamlessly provision RAC systems with multiple nodes in a totally automated manner, with all the clusterware, ASM, and database software being installed and configured, and even a RAC database being provisioned.
For 11g Release 2 RAC databases the process is made even simpler via the new concept of Provisioning Profiles.
Oracle Enterprise Manager Grid Control is a powerful enterprise management product from Oracle. You can read the following articles from the author to gain a further understanding of its many capabilities.
|To learn how to:|| |
Automate the patching of Oracle RAC or non-RAC databases, Oracle Automatic Storage Management (ASM) instances, and Oracle Clusterware in your environment
Provision literally hundreds or even thousands of systems using mass provisioning techniques
Easily set up Oracle Recovery Manager (RMAN) backups for your corporate databases using Oracle Enterprise Manager Grid Control
Set up Data Integration using Oracle GoldenGate
Understand how Oracle Enterprise Manager Grid Control can save a lot of time and resources when used for Oracle Data Guard setup, management (including switchover or failover), and monitoring
See how easily the DataMasking Pack of Oracle Enterprise Manager Grid Control can obsfucate your confidential production data as an integral part of the database cloning process
For a broad introduction to all the advanced concepts of using Oracle Enterprise Manager Grid Control
For an overview of the Oracle Enterprise Manager Management Packs available for Oracle targets including hardware/systems, operating systems, virtualization, database and middleware
Enjoy the world of Oracle Enterprise Manager.
Porus Homi Havewala works as a Senior Manager (Enterprise Technology) at Oracle Corporation Singapore and is an Oracle Certified Master. He was awarded the Oracle ACE Director title by Oracle HQ in 2008 and is now an Oracle Employee ACE. He has extensive experience in Oracle technology since 1994, including as a Senior Production DBA, Principal Database Consultant, Database Architect, E-Business Technical DBA, Development DBA, and Database Designer Modeller (using Oracle Designer). He has published numerous articles on Grid Control on OTN, and created the world's first blog dedicated to Grid Control. Porus is also the author of the popular new book Oracle Enterprise Manager Grid Control: Advanced Techniques for the Real World.