by Porus Homi Havewala
This article is excerpted from Oracle Enterprise Manager Cloud Control 12c: Managing Data Center Chaos, by Porus Homi Havewala, published by Packt Publishing.
Studies show that many corporations world-wide expect their IT footprint to grow in the coming years. They expect more servers, more databases, more data, and more of everything.
They require more floor space in their data centers, and also the corresponding greater power footprint. Have you heard of a data center where no more servers can be added because the power supply has reached its limit, or the UPS (Uninterruptible Power Supply) can no longer cope? This story is not new.
The growth seems to be endless, and this is fuelled by today's information age, where larger and larger volumes of data need to be stored and distributed to satisfy an ever-growing demand. More applications are coming to use those databases, on more and more application servers.
So for the IT Manager, this will mean more of everything in his/her data center. There may be different hardware platforms, there may be different operating systems (e.g. Solaris, Linux, IBM-AIX, Microsoft Windows), and in each case there may be different versions – such as the different flavors of Linux supplied by different vendors including Oracle Enterprise Linux, Red Hat, SUSE Linux, and so on.
To add to the complexity, nothing placed in production should be treated as static. Software changes in development cycles, enhancements take place, or security/functional issues are found. For almost anything in the IT world, new patches are bound to be released. These will also need to be applied to production, test, reporting, staging, and development environments in the Data Center on an ongoing basis.
For the Database side of things, Oracle releases quarterly a combination of security fixes known as the Critical Patch Update (CPU). Other patches are bundled together and released every quarter in the form of a Patch Set Update (PSU), and this also includes the CPU for that quarter.
Oracle strongly recommends applying either the PSU or the CPU every calendar quarter. If you prefer to apply the CPU, continue doing so. If you wish to move to the PSU, you can do so, but in that case continue only with the PSU.
The quarterly patching requirement, as a direct recommendation from Oracle, is followed by many companies which prefer to have their Databases secured with the latest security fixes. This underscores the importance of patching.
However, if there are hundreds of development, test, staging and production Databases in the Data Center to be patched, the situation quickly turns into a major manual exercise every three months. DBAs and their Managers start planning for the patch exercise in advance, and a lot of resources are allocated to make it happen – with the Administrators working on each Database serially, at times overnight and at times over the weekend.
There are a number of steps involved in patching each Database, such as locating the appropriate patch in My Oracle Support (MOS), downloading the patch, transferring it to each of the target servers, upgrading the OPATCH facility in each Oracle Home, shutting down the Databases and Listeners running from that Home, applying the patch, starting each of the Databases in restricted mode, applying any supplied SQL scripts, restarting the Databases in normal mode, and checking the patch inventory.
These steps have to be manually repeated on every Database Home on every server, and on every Database in that Home. Dull repetition of these steps in patching the hundreds of servers in a data center is a very monotonous task, and it can lead to an increase in human errors.
To avoid these issues inherent in manual patching, some companies decide not to apply the quarterly patches on their Databases. They wait for a year, or a couple of years, before they consider patching, and some even prefer to apply year-old patches instead of the latest patches. This is counter-productive and leads to their Databases being insecure and vulnerable to attacks, since the latest recommended CPUs from Oracle have not been applied.
What then is the solution, to convince these companies to apply patches regularly? If the patching process can be mostly automated (but still under the control of the DBAs), it would reduce the quarterly patching effort to a great extent. Companies would then have the confidence that their existing team of DBAs would be able to manage the patching of hundreds of Databases in a controlled and automated manner, keeping human error to a minimum.
The Database Lifecycle Management pack for Oracle Enterprise Manager Cloud Control 12c is able to achieve this via its Patch Automation capability. We will now look into Patch Automation and the close integration of Enterprise Manager with My Oracle Support.
We assume that Enterprise Manager Cloud Control 12c has been installed with the appropriate architecture on a central site, and Enterprise Manager Agents have been pushed to all production, test and development targets. (We have covered installation or upgrade details in an earlier chapter of the book.) You are now logged in to Enterprise Manager Cloud Control 12c.
On the Enterprise | Summary screen, there is a Patch Recommendations section on the lower left hand corner, as shown in Figure 1 below.
The graph displays either the Classification of the recommended patches, or by Target Type. Currently for this system, more than five security patches are recommended as can be seen in this graph. This recommendation has been derived via a connection to My Oracle Support (the OMS can be connected either directly to the Internet, or via a Proxy Server). Target configuration information is collected by the Enterprise Manager agent and stored in the Configuration Management Database (CMDB) within the repository. This configuration information is collated regularly by the Enterprise Manager Harvester process and pushed to My Oracle Support.
Thus, configuration information about your targets is known to My Oracle Support, which can then recommend appropriate patches as they are released.
It is also possible to get patch recommendations from My Oracle Support in an offline mode, but more manual steps are involved in this case, so Internet connectivity is recommended to get the full benefits of Enterprise Manager's integration with My Oracle Support.
To view the details about the patches, click on All Recommendations or on the graph itself. This connects to My Oracle Support (you may be asked to login to your company-specific MOS account) and brings up the list of Patch Recommendations (Figure 2, below).
The Database (and other types of) targets managed by the Enterprise Manager system are displayed on the screen, along with the recommended CPU (or other) patches. We select the CPU July patch for our saiprod Database. This displays the details about the patch in the section at the lower part of the screen.
We can see the list of Bugs resolved by this patch, the Last Updated date and Size of the patch, and also the Read Me – which has important information about the patch.
The number of total downloads for this patch is visible, as is the Community Discussion on this patch in the Oracle Forums. You can add your own comment for this patch, if required, by selecting Reply to the Discussion.
Thus, at a glance you can find out how popular the patch is (number of downloads) and any experience of other Oracle DBAs regarding this patch – whether positive or negative.
You can view the information about the patch by clicking on the Full Screen button. You can download the patch either to the Software Library in Enterprise Manager or to your desktop. Finally, you can directly add this patch to a new or existing Patch Plan, which we will do next.
Select Add to Plan | Add to New, and enter the Plan Name as Sainath_patchplan. Then click on Create Plan. If you want to add multiple patches to the Plan, select the patches first and then add to them the Plan. (You can also add patches to the plan later.)
After the plan is created, click on View Plan. This brings up the screen shown in Figure 3, below.
A Patch Plan is nothing but a collection of patches that can be applied as a group to one or more targets. On the Create Plan page that appears, there are five steps that can be seen in the left-hand pane. By default, the second step appears first. In this step, you can see all the patches that have been added to the plan.
It is possible to include more patches by clicking on the Add Patch button. Besides the ability to manually add a patch to this list, the Analysis process may also result in additional patches being added to the plan.
Click Step 1: Plan Information to enter a description for this plan. You can also change the Plan permissions, Full or View, for various Enterprise Manager roles. Note that the Full permission allows the role to validate the plan; the View permission does not allow validation.
Move to Step 3: Deployment Options. The following screen (Figure 4) appears.
A new mechanism for patching has been provided in Oracle Enterprise Manager Cloud Control 12c, known as Out of Place Patching. This is now the recommended method and creates a new Oracle Home, which is then patched while the previous Home is still operational. All this is done using an out-of-box Deployment Procedure in Enterprise Manager.
Using this mechanism means that the only downtime will take place when the Databases from the previous Home are switched to run from the new Home. If there is any issue with the Database patch, you can switch back to the previous unpatched Home since it is still available. So, patch rollback is a lot faster.
Also, if there are multiple Databases running in the previous Home, you can decide which ones to switch to the new patched Home. This is obviously an advantage, otherwise you would be forced to simultaneously patch all the Databases in a Home. A disadvantage of this method would be the space requirements for a duplicate Home. Also, if proper housekeeping is not done later on, it can lead to a proliferation of Oracle Homes on a server where patches are being applied regularly using this mechanism.
This kind of selective patching and minimal downtime is not possible if you use the previously available method of In Place Patching, which uses a separate Deployment Procedure to shut down all Databases running from an Oracle Home before applying the patches on the same Home. The Databases can only be restarted normally after the patching process is over, and this obviously takes more downtime and affects all Databases in a Home.
Depending on the method you choose, the appropriate Deployment Procedure will be automatically selected and used.
We will now use the Out of Place method in this Patch Plan. On the Step 3 page, make sure the Out of Place option is selected. Then click on Create New Location. The window shown in Figure 5 appears.
Type in the name and location of the new Oracle Home, and click on Validate. This checks the Oracle Home path on the target server. After this is done, click on Create.
The Deployment Options of the patch plan are successfully updated, and the new Home appears on the Step 3 page.Click on the Credentials tab. Here you need to select or enter the normal and privileged credentials for the Oracle Home.
Click on the Next button. This moves to Step 4: Validation (Figure 6).
Click on the Analyze button. A job to perform pre-patching analysis is started in the background. This will compare the installed software and patches on the targets with the new patches you have selected in your plan, and attempt to validate them.
This validation may take a few minutes to complete, since it also checks the Oracle Home for readiness, computes the space requirements for the home, and conducts other checks such as cluster node connectivity (if you are patching a RAC Database).
If you drill down to the analysis job itself by clicking on Show Detailed Progress, you can see that it does a number of checks to validate if the targets are supported for patching, verifies the normal and super user credentials of the Oracle Home, verifies the target tools, commands, and permissions, upgrades OPATCH to the latest version, stages the selected patches to Oracle Homes, and then runs the prerequisite checks, including those for cloning an Oracle Home.
If the prerequisite checks succeed, the analysis job skips the remaining steps and stops at this point with a successful status. The patch is seen as Ready for Deployment (Figure 7).
If there are any issues, they will show up at this point. For example, if there is a conflict with any of the patches, a Replacement patch or a Merge patch may be suggested. If there is no Replacement or Merge patch and you want to request such a patch, you can do so directly from the screen.
If you are applying a PSU and the CPU for that same release is already applied to the Oracle Home, for example the July 2011 CPU, then because the PSU is a superset of the CPU, the MOS Analysis will stop and mention that the existing patch fixes the issues. Such a message can be seen in the Informational Messages section of the Validation page.
Our patch is Ready for Deployment. At this point you can move directly to Step 5: Review & Deploy, by clicking on it in the left pane.
On the Review & Deploy page (Figure 8), the Patch Plan is described in detail along with the Impacted Targets. Along with the Database which is in the Patch Plan, a new impacted target has been found by the analysis process and added to the list of impacted targets. This is the Listener, which is running from the Home, which is to be cloned and patched.
The patches that are to be applied are also listed on this review page. In this case the CPUJUL2011 patch is shown with the status Conflict Free.
The Deployment Procedure that will be used is Clone and Patch Oracle Database, since Out of Place patching is being used, and all Instances and Listeners running in the previous Oracle Home are being switched to the new Home.
Click on the Prepare button. The status on the screen changes to Preparation in Progress. A job for preparation of the Out of Place Patching starts, including Cloning of the original Oracle Home and applying the patches to the cloned home. No downtime is required while this job is running, it can happen in the background.
This prepare phase is like a Pre-Deploy and is only possible in the case of Out of Place Patching. In contrast, with In Place Patching there is no Prepare button and you deploy straight away.
Clicking on Show Detailed Progress here opens a new window showing the job details.
When the preparation job has successfully completed (after about 2 hours in our Virtual Machine), we can see that it performs the cloning of the Oracle Home, applies the patches on the new Home, validates the patches, runs the post patch scripts, and then skips all the remaining steps. It also collects target properties for the Oracle Home in order to refresh the configurations in Enterprise Manager.
The Review & Deploy page now shows Preparation Successful (Figure 9). The plan is now ready to be deployed.
Click on the Deploy button. The status on the screen changes to Deployment in Progress. A job for deployment of the Out of Place Patching starts.
At this time, downtime will be required since the Database instances using the previous Oracle Home will be shut down and switched across.
The Deploy job successfully completes (after about 21 minutes in our Virtual Machine), we can see that it works iteratively over the list of Hosts and Oracle Homes in the Patch Plan. It starts a Blackout for the Database instances in the Oracle Home (so that no alerts are raised), stops the instances, migrates them to the cloned Oracle Home, starts them in Upgrade Mode, applies SQL scripts to patch the instance, applies Post-SQL scripts, and then restarts the Database in normal mode.
The Deploy job applies other SQL scripts and recompiles invalid objects (except in the case of Patch Sets). It then migrates the Listener from the previous Oracle Home using the Network Configuration Assistant (netca), updates the Target properties, stops the Blackout, and detaches the previous Oracle Home. Finally, the configuration information of the cloned Oracle Home is refreshed.The Review & Deploy page of the Patch Plan now shows the status of Deployment Successful, as shown in Figure 10.
On the Deployment Successful page, it is possible to click on Save as Template at the bottom of the screen in order to save a Patch Plan as a Plan Template. The Patch Plan should be successfully analyzed and deployable, or successfully deployed, before it can be saved as a Template.
The Plan Template when thus created will not include any targets. Such a template can then be used to apply the successful Patch Plan to multiple other targets. Inside the Plan Template, the Create Plan button is used to create a new plan based on this template and this can be done repeatedly for multiple targets.
Select Enterprise | Provisioning and Patching | Patches & Updates. This screen displays a list of all the Patch Plans and Plan Templates that have been created. The successfully deployed Sainath Patch Plan and the new Patch Plan Template also shows up here (Figure 11).
To see a list of the saved patches in the Software Library, select Enterprise | Provisioning and Patching | Saved Patches. This brings up the screen shown in Figure 12.
This page also allows you to manually upload patches to the Software Library. This scenario is mostly used when there is no connection to the Internet (either direct or via a Proxy Server) from the Enterprise Manager OMS servers, and consequently you need to download the patches manually.
For more details on setting up the Offline Mode and downloading the Patch Recommendations and latest Patch information in the form of XML files from My Oracle Support, please refer to the Enterprise Manager Cloud Control 12c Lifecycle Management Administrator's Guide.
The new version of Oracle Enterprise Manager Cloud Control 12c supplies out-of-box Administrator roles specifically for patching. These roles are EM_PATCH_ADMINISTRATOR, EM_PATCH_DESIGNER, and EM_PATCH_OPERATOR. You need to grant these roles to the appropriate Administrators.
Move to Setup | Security | Roles. On this page, search for the Roles specifically meant for patching. The three Roles appear as shown in Figure 13.
The Patch Administrator can create, edit, deploy or delete any patch plan and can also grant privileges to other Administrators after creating them. This role has full privileges on any Patch Plan or Patch Template in the Enterprise Manager system and maintains the Patching infrastructure.
The Patch Designer normally identifies patches to be used in the patching cycle across development, testing and production. This role would be the Senior DBA in real life. The Patch Designer creates Patch Plans, then Plan Templates, and grants privileges on these Plan Templates to the Patch Operator.
As an example, the Patch Designer will select a set of recommended and other manually selected patches for an Oracle 11g Database and create a Patch Plan. This role will then test the patching process in a development environment, and save the successfully analyzed or deployed Patch Plan as a Plan Template.
The Patch Designer will then publish the Oracle 11g Database Patching Plan Template to the Patch Operator –probably the Junior DBA or Application DBA in a real world scenario.
Next, the Patch Operator creates new Patch Plans out of the Template (but cannot create a Template), and adds a different list of targets, such as other Oracle 11g Databases in the test, staging, or production environment. This role then schedules the deployment of the patches to all these environments – using the same Template again and again.
An out-of-box Enterprise Manager job runs everyday and gets the latest patch information from My Oracle Support (MOS). This job is important for you to receive timely notification of Critical Patch Advisories and other patches from Oracle.
You can check to see if this job is scheduled and running by visiting Enterprise | Job | Activity and selecting Advanced Search. Then select the job type as Refresh From My Oracle Support and Scheduled Start of Last 7 days.
This displays all the out-of-box refreshed jobs that have run and are scheduled for the future. If there is a requirement to set up your own refresh job (for example, if you would like the MOS Patch Recommendations for your Databases to be refreshed immediately instead of waiting for the next scheduled running of the job), then perform the following steps.
Select Enterprise | Job | Library. On this page, select Refresh from My Oracle Support from the dropdown box of Create Library Job, and click on Go.
Give a new name to the job, such as "Sainath Refresh from MOS". On the Schedule tab, set it to run One Time (Immediately). Click on Save to Library.
Next, search for the Job in the Job Library, and click on Submit. The Refresh job executes and completes in a few minutes. You now have up to date My Oracle Support Patch Recommendations for your Enterprise Manager targets.
My Oracle Support obviously needs to know about the Configuration information of all your targets, in order to make the right recommendations. For example, it needs to know if your Database is already patched up to a certain CPU and PSU. If your Database is not, Oracle Support can recommend the appropriate CPU/PSU.
There is another recurring refresh cycle that occurs overnight and performs the Configuration inventory collection for all the targets managed and monitored by Enterprise Manager. If you wish, this collection can be forced manually in two ways:
Move to the appropriate Oracle Home Target Home page (after finding it in Targets | All Targets by sorting on Target Type). Select Oracle Home | Configuration | Last Collected. On that page, select Actions | Refresh. This submits a request to the agent to re-collect the Configuration inventory for that Oracle Home.
The other method to force the collection is by using the emctl command in the Agent Home on the actual Target Host. First find the Oracle Home Target name given by Enterprise Manager to our Oracle Homes, then use the Target Name in the runCollection command as shown below:
cd /u01/oracle/middleware/agent/agent_inst/bin [oracle@havipori bin]$ ./emctl config agent listtargets Oracle Enterprise Manager 12c Cloud Control 22.214.171.124.0 Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved. [/EMGC_GCDomain/GCDomain/EMGC_OMS1/OCMRepeater, j2ee_application] [/EMGC_GCDomain/GCDomain/EMGC_OMS1/emgc, j2ee_application] [/EMGC_GCDomain/GCDomain/EMGC_OMS1/empbs, j2ee_application] [/EMGC_GCDomain/GCDomain/EMGC_ADMINSERVER/mds-owsm, metadata_repository] [/EMGC_GCDomain/GCDomain/EMGC_ADMINSERVER/mds-sysman_mds, metadata_repository] [/EMGC_GCDomain/instance1/ohs1, oracle_apache] [/EMGC_GCDomain/GCDomain/EMGC_OMS1/oracle.security.apm(126.96.36.199.0), oracle_apm] [EM Management Beacon, oracle_beacon] [emrepos.sainath.com, oracle_database] [orcl, oracle_database] [havipori.sainath.com:3872, oracle_emd] [Management Services and Repository, oracle_emrep] [OraDb11g_home1_1_havipori, oracle_home] [WebLogicServer10_3_5_0_0_havipori, oracle_home] [agent12g1_13_havipori, oracle_home] [common12g1_24_havipori, oracle_home] [jdk1_2_havipori, oracle_home] [oms12g1_3_havipori, oracle_home] [sbin12g1_14_havipori, oracle_home] [webtier12g1_25_havipori, oracle_home] [EMGC_GCDomain, oracle_ias_farm] [LISTENER_havipori.sainath.com, oracle_listener] [havipori.sainath.com:4889_Management_Service, oracle_oms] [havipori.sainath.com:4889_Management_Service_CONSOLE, oracle_oms_console] [havipori.sainath.com:4889_Management_Service_PBS, oracle_oms_pbs] [/EMGC_GCDomain/GCDomain, weblogic_domain] [/EMGC_GCDomain/GCDomain/EMGC_ADMINSERVER, weblogic_j2eeserver] [/EMGC_GCDomain/GCDomain/EMGC_OMS1, weblogic_j2eeserver] [havipori.sainath.com_csa, oracle_csa_collector] [havipori.sainath.com, host] [SAINATH_LISTENER_havipori.sainath.com_1522, oracle_listener] [saiprod.sainath.com, oracle_database] [Orasidb11g_home1_2012_02_05_05_15_29_havipori, oracle_home] [oracle@havipori bin]$ ./emctl control agent runCollection OraDb11g_home1_1_havipori:oracle_home oracle_home_config Oracle Enterprise Manager 12c Cloud Control 188.8.131.52.0 Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved. --------------------------------------------------------------- EMD runCollection completed successfully [oracle@havipori bin]$ ./emctl control agent runCollection Orasidb11g_home1_2012_02_05_05_15_29_havipori:oracle_home oracle_home_config Oracle Enterprise Manager 12c Cloud Control 184.108.40.206.0 Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved. --------------------------------------------------------------- EMD runCollection completed successfully
There are a number of other Deployment Procedures available for patching. Select Enterprise | Provisioning and Patching | Procedure Library, and search for patch in the text fields. This brings up the list of Procedures as shown in Figure 14, below.
In the case of Out of Place patching, the Clone and Patch Oracle Database Procedure will be used. Other procedures are available to perform actions including:
The RAC Rolling Patching procedures perform shutdown, patching and startup across all the nodes in the cluster in a rolling manner. The prerequisite checks include detailed checks for cluster health, so as to avoid unexpected failures. The SQL scripts, such as catbundle.sql (the patching SQL) or utlrp.sql (the recompile invalid objects SQL) are executed on one node at the end of the patching procedure.
This Rolling Patching procedure is very useful for automating the patching of multiple RAC nodes, especially when there are more than two nodes. In the case of Oracle Exadata Database Machines, the use of such procedures is highly recommended to ease the administration burden. CPUs can be applied in the rolling manner, also as Clusterware patches or ASM patches.
Oracle WebLogic Server patching is now also possible from Oracle Enterprise Manager Cloud Control 12c. This requires a different license: the Oracle WebLogic Server Management Pack Enterprise Edition (EE). The WebLogic Administrator needs to search MOS for patches to apply to a WebLogic Domain, depending on the WLS version and the Platform. The Patching Wizard in Enterprise Manager checks for conflicts, and allows you to apply the patches in parallel or in a rolling manner across the Domain. This can greatly automate the patching of WebLogic Domains. However, WLS Patch Recommendations are not possible at the time of this writing. You need to search for the WLS patches yourself.
A rich set of Reports is available. including an Applied Patches Report and a Target Patchability Report. These are accessed via Enterprise | Reports | Information Publisher Reports.
Oracle BI Publisher Enterprise Reports can also be used, but BI Publisher needs to be configured for use with Enterprise Manager. For more information on the setup required, refer to the Cloud Control Advanced Installation and Configuration Guide.
Oracle Enterprise Manager Cloud Control 12c allows automation of the tedious patching procedure used in many organizations today to patch their Oracle Databases and Servers. This is achieved via the Oracle Database Lifecycle Management Pack, which is one of the main licensable packs for Oracle Enterprise Manager.
Sophisticated Deployment Procedures are provided out-of-the-box to fulfill many different types of patching tasks. This helps you to achieve mass patching of multiple targets with multiple patches in a fully automated manner, for substantial savings in administrative time and effort. Some companies have estimated savings up to 98% in patching tasks in their data centers. Different types of patches can be applied in this manner, including CPUs, PSUs, Patch sets, and other one-off patches. Different versions of databases are supported, such as 9i, 10g and 11g. For the first time, upgrade of single-instance databases is also possible via Oracle Enterprise Manager Cloud Control 12c.
There is full integration of Oracle Enterprise Manager's patching capabilities with My Oracle Support (MOS). The support site retains the configuration of all the components managed by Oracle Enterprise Manager inside your company. Since the current version and patch information of the components is known, My Oracle Support is able to provide appropriate patch recommendations for many targets, including the latest security fixes. This ensures that your company is up to date with regards to security protection.
A full Division of Roles is available, such as Patch Administrator, Designer, and Operator. It is possible to take the My Oracle Support recommendations, select patches for targets, put them into a patch plan, deploy the patch plan, and then create a plan template from it. The template can then be published to any operator who can then create patch plans for other targets. In this way patching can be tested, verified, and then pushed to production.
In all, Oracle Enterprise Manager Cloud Control 12c offers valuable automation methods for Mass Patching, allowing Administrators to ensure that their systems have the latest security patches, and enabling them to control the application of patches on development, test and production servers from the centralized location of the Software Library.