Best Way to Update Software in Zones

Part III of Software Management Best Practices for Oracle Solaris 11 Express

By Ginny Henningsen, August 2011

Part I - Best Way to Update Software with IPS
Part II - Best Way to Automate ZFS Snapshots and Track Software Updates


Introduction

This is the third article in a series highlighting best practices for software updates in Oracle Solaris 11 Express. The first article introduced the IPS software packaging model and highlighted best practices for creating a new Boot Environment (BE) before performing an update. The second article discussed the Time Slider and auto-snapshot services, describing how to initialize and use these services to periodically snapshot BEs and other ZFS volumes.

This third article dives more deeply into the topic of software updates, exploring the process of updating an Oracle Solaris 11 Express system configured with zones. This topic is especially pertinent since zones in this release differ somewhat from those in Oracle Solaris 10, as does the software upgrade process for zoned systems.

Please note that when Oracle Solaris 11 is released, it will change and simplify the process for creating and upgrading zones. This article focuses strictly on how to perform zone upgrades currently under Oracle Solaris 11 Express, and will be updated when the process changes. For reference, refer to the full documentation set for Oracle Solaris 11 Express.

For the Novice: Some Background on Zones

First introduced in Oracle Solaris 10, zones are built-in, lightweight virtual machines that isolate workloads (see the System Administration Guide: Oracle Solaris Zones, Oracle Solaris 10 Containers, and Resource Management). Processes within a zone are restricted to accessing resources in that zone, and they can't interfere with processes or resources in other zones. The global zone contains the core operating system (OS), and administrators can define multiple non-global zones to isolate user-level workloads.

How Do Zones Differ in Oracle Solaris 11 Express?

From a functional standpoint, zones in Oracle Solaris 10 and Oracle Solaris 11 Express are similar, but there are a few noteworthy differences, summarized in Table 1.

Table 1: Zone Differences Between Oracle Solaris 10 and Oracle Solaris 11 Express
Feature Oracle Solaris 10 Oracle Solaris 11 Express
Global zone brand Branded as "native" Branded as ipkg and based on the new IPS software packaging model
Non-global zone brands Branded as "native" zones or as Linux, Solaris 8, or Solaris 9 brand zones Branded as ipkg zones or as solaris10 zones; see the solaris10(5) man page in man pages section 5: Standards, Environments, and Macros
Non-global zone roots Whole or sparse root (sparse root zones share text segments from executables and shared libraries from the global zone) Whole root only and reside on own ZFS dataset
Non-global zone contents Packages must be the same as in global zone Packages in non-global zone can differ from that in global zone
Patch application? Yes, can be applied to multiple zones in parallel No patching (pkg updates instead)
Upgrading global zone also updates non-global zones? Yes No

As Table 1 shows, ipkg zones in Oracle Solaris 11 Express are "whole root" only and reside on their own ZFS dataset. As Jeff Savit's blog ("Ours Goes to 11--Features of Oracle Solaris 11 Express") describes, creating non-global zones in Oracle Solaris 11 Express takes advantage of ZFS cloning, which inherently conserves space. (Jeff's blog goes on to explain how to install a solaris10 branded zone on Oracle Solaris 11 Express.)

Upgrading zones in Oracle Solaris 11 Express differs from upgrading zones in Oracle Solaris 10. Currently, ipkg brand zones in Oracle Solaris 11 Express are not updated when the global zone is updated. Work is underway to allow zones to be updated in parallel, but until the release of Oracle Solaris 11, non-global zones in Oracle Solaris 11 Express must be updated manually.

Remember the best practice in Oracle Solaris 11 Express:
Update non-global zones manually to keep them in sync with the global zone.

At this time, updating a non-global zone in Oracle Solaris 11 Express is similar to migrating a non-global zone to another server; in both cases, system software for non-global zones must be updated to the same version level as the global zone. Global zone contents can differ from non-global zones in Oracle Solaris 11 Express, but specific release levels must be in sync.

This article steps through a simple example of creating zones on Oracle Solaris 11 Express and current best practices for updating both global and non-global zones. Note that installing non-global zones currently requires a network connection and access to an Oracle Solaris 11 Express package repository, unless the zone is cloned from an existing non-global zone.

Creating Zones in Oracle Solaris 11 Express

To set the stage, let's start by creating a non-global zone in Oracle Solaris 11 Express. The process for creating a non-global zone in Oracle Solaris 11 Express is similar to defining one in Oracle Solaris 10. First, configure the non-global zone, install it, and then boot it. Oracle Solaris 11 Express offers some new configuration options (such as those to construct virtual networks; see Jeff Victor's blog articles on this topic), but for the most part, zone configuration is much the same. One significant difference is that an Oracle Solaris 11 Express zone must reside on its own ZFS dataset, which can be explicitly created before the zone is configured:

# zfs create rpool/zones

(All command examples in this article presume a privileged user. See "User Accounts, Roles, and Rights Profiles" in Getting Started With Oracle Solaris 11 Express.)

The following command defines a mount point for the ZFS dataset rpool/zones:

# zfs set mountpoint=/export/zfs rpool/zones

How Do I Configure a Non-Global Zone?

If you already know how to configure and install a zone, skip ahead to How Do I Upgrade the Global Zone? If you are new to zones, the next few paragraphs step through the configuration and installation process.

The following commands configure a new non-global zone called my-zone on the ZFS dataset created previously:

# zonecfg -z my-zone
my-zone: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:my-zone> create
zonecfg:my-zone> set zonepath=/export/zfs/my-zone
zonecfg:my-zone> add net
zonecfg:my-zone:net> set address=192.168.1.99
zonecfg:my-zone:net> set physical=e1000g0
zonecfg:my-zone:net> end
zonecfg:my-zone> verify
zonecfg:my-zone> commit
zonecfg:my-zone> exit

How Do I Install a Non-Global Zone?

For Oracle Solaris 11 Express, zone installation accesses IPS package repositories, pulling packages from referenced or default repositories. By default, the zone installation uses packages from the release repository at http://pkg.oracle.com/solaris/release:

# zoneadm -z my-zone install
A ZFS file system has been created for this zone.
   Publisher: Using solaris (http://pkg.oracle.com/solaris/release/ ).
       Image: Preparing at /zones/my-zone/root.
       Cache: Using /var/pkg/download.
Sanity Check: Looking for 'entire' incorporation.
  Installing: Core System (output follows)
------------------------------------------------------------
Package:
pkg://solaris/consolidation/osnet/osnet-incorporation@0.5.11,5.11-0.151.0.1:20101104T230646Z
License: usr/src/pkg/license_files/lic_OTN
 	.
 	.
 	.
 Done: Installation completed in 371.635 seconds.

  Next Steps: Boot the zone, then log into the zone console (zlogin -C)
              to complete the configuration process.

How Do I Finalize Zone Installation?

Boot the zone and log into its console to complete the configuration:

# zoneadm -z my-zone boot
# zlogin -C my-zone
[Connected to zone 'my-zone' console]

At this point, specify final installation parameters (host name, name service, language, locale, time zone, root password, and so forth). When the install concludes, this message appears and zone login is enabled:

System identification is completed.
.
.
.
my-zone console login:

In the global zone, the following command shows the status for all zones:

# zoneadm list -iv
  ID NAME             STATUS     PATH                           BRAND    IP
   0 global           running    /                              ipkg     shared
   1 my-zone          running    /export/zfs/my-zone            ipkg     shared

How Do I Clone a Zone?

As a precaution or to speed provisioning, you can optionally clone a zone while it's inactive. First, halt the non-global zone and then export its configuration:

# zoneadm -z my-zone halt
# zonecfg -z my-zone export -f /export/zfs/master 

Edit the zone configuration, changing the zonepath, the network definition, and other parameters as needed:

# vi /export/zfs/master

Configure and clone the zone, and then boot the non-global zone and its clone:

# zonecfg -z my-zone2 -f /export/zfs/master
# zoneadm -z my-zone2 clone my-zone
# zoneadm -z my-zone boot
# zoneadm -z my-zone2 boot

List the zones:

# zoneadm list -iv
  ID NAME             STATUS     PATH                           BRAND    IP
   0 global           running    /                              ipkg     shared
   - my-zone          running    /export/zfs/my-zone            ipkg     shared
   - my-zone2         running    /export/zfs/my-zone2           ipkg     shared

How Do I Install Packages on a Zone?

First, let's make a distinction between a software upgrade and a software install. If you use the pkg install command in the global zone to add a package, the package is installed there and not propagated to non-global zones. To install a package in a non-global zone, an authorized zone administrator can log in to the non-global zone and execute the pkg install command there.

As an example, let's install Apache HTTP Server version 2.2 to build a Web server on the non-global zone my-zone (for brevity, command output is not shown):

root@my-zone:~# pkg install apache-22

Executing the pkg history command in my-zone shows the Apache installation. (Compare this output to the results of the pkg history command in the global zone.)

How Do I Upgrade the Global Zone?

Best practice in Oracle Solaris 11 Express is to generate a new Boot Environment (BE) prior to a software change (see the first article in this series). In some cases, as in a full update, a new BE is automatically created and activated on reboot. In other cases you must explicitly create one. There are several ways to initiate a system software update:

  • Via an "Update All" in the Package Manager or Update Manager GUI
  • Via the pkg(1) command, as in pkg update

Oracle plans three different types of updates for Oracle Solaris 11:

  • Support Repository Updates (SRUs). Customers with an active Oracle Solaris 11 Express support contract will be able to access the support repository containing periodically released software package updates. These updates include bug fixes and security updates.
  • Periodic Update Releases. Similar to update releases for Oracle Solaris 10, Oracle will issue periodic updates for Oracle Solaris 11. About every 6 to 12 months there will be an update release that contains all the SRUs to the previous release plus the potential for some new features (just as is the case with Oracle Solaris 10 updates today).
  • Full Upgrades. A full upgrade, like that of updating from Oracle Solaris 11 Express to Oracle Solaris 11 (when it's available) requires access to the release repository at pkg.oracle.com or to a mirror of the release repository.

How Do I Access the Support Repository?

To access SRUs and periodic update releases, you must have an Oracle Solaris 11 Express support contract and a CSI-registered account on My Oracle Support (see the article Support Repositories Explained [ID 1021281.1]). Log in to My Oracle Support to download the certificate and key files that enable support repository access. Before updating the global zone, define a directory for the certificate and key files:

# mkdir -m 0755 -p /var/pkg/ssl
# cp -i ./Oracle_Solaris_11_Express_Support.certificate.pem /var/pkg/ssl
# cp -i ./Oracle_Solaris_11_Express_Support.key.pem /var/pkg/ssl

Then, define the support repository location and publisher for pkg, specifying the certificate and key:

# pkg set-publisher -k /var/pkg/ssl/Oracle_Solaris_11_Express_Support.key.pem -c /var/pkg/ssl/Oracle_Solaris_11_Express_Support.certificate.pem -O https://pkg.oracle.com/solaris/support solaris

If you are using the packagemanager GUI, the updated package list will be visible after you restart the GUI. The last entry in the pkg history -l command reflects the change in publisher:

Operation: update-publisher
        Outcome: Succeeded
         Client: pkg
        Version: 052adf36c3f4
           User: ghenning (101)
     Start Time: 2011-04-21T10:16:40
       End Time: 2011-04-21T10:16:43
        Command: /usr/bin/pkg set-publisher -k
/var/pkg/ssl/Oracle_Solaris_11_Express_Support.key.pem -c
/var/pkg/ssl/Oracle_Solaris_11_Express_Support.certificate.pem -O
https://pkg.oracle.com/solaris/support/ solaris
    Start State:
None
      End State:
None

Upgrading the Global Zone

Running the pkg update -nv command shows what will happen during an update, without actually changing anything. The first time, you might get a warning indicating that pkg is out of date:

# pkg update -nv
WARNING: pkg(5) appears to be out of date, and should be updated before
running update.  Please update pkg(5) using 'pfexec pkg install
pkg:/package/pkg' and then retry the update.

After installing the new version of pkg, run the update command again:

# pkg install pkg:/package/pkg
Packages to update:     1
           Create boot environment:    No
DOWNLOAD                                  PKGS       FILES    XFER (MB)
Completed                                  1/1     126/126      0.7/0.7

PHASE                                        ACTIONS
Install Phase                                    1/1
Update Phase                                 242/242

PHASE                                          ITEMS
Package State Update Phase                       2/2
Package Cache Update Phase                       1/1
Image State Update Phase                         2/2

# pkg update -nv  

                Packages to update:    45
           Create boot environment:   Yes 
              Rebuild boot archive:   Yes
Changed fmris:
  pkg://solaris/entire@0.5.11,5.11-0.151.0.1:20101105T054056Z ->
pkg://solaris/entire@0.5.11,5.11-0.151.0.1.6:20110328T230730Z
.
.
.

As highlighted in the output above, the global zone's OS version (5.11-0.151.0.1) lags the version in the support repository (5.11-0.151.0.1.6). The update will also automatically create a new BE. Remember, if the update will not automatically create a new BE, best practice is to explicitly create one.

Without the -nv option, the pkg update command updates the global zone, creating a new BE with the default name of solaris-1. Best practice is to specify a BE name on the update command line explicitly, so that the BE is named something meaningful, for example:

# pkg update --require-new-be --be-name "S11E_SRU6"
Packages to update:    45
           Create boot environment:   Yes
DOWNLOAD                                  PKGS       FILES    XFER (MB)
Completed                                45/45   1235/1235    70.2/70.2

PHASE                                        ACTIONS
Removal Phase                                184/184
Install Phase                                350/350
Update Phase                               3349/3349

PHASE                                          ITEMS
Package State Update Phase                     90/90
Package Cache Update Phase                     45/45
Image State Update Phase                         2/2



A clone of solaris exists and has been updated and activated.
On the next boot the Boot Environment S11E_SRU6 will be mounted on '/'.
Reboot when ready to switch to this updated BE.

---------------------------------------------------------------------------
NOTE: Please review release notes posted at:
http://docs.sun.com/doc/821-1479
---------------------------------------------------------------------------

After updating the global zone, reboot the system to run the updated BE. Note that the update affects only currently installed packages. In a minimized system (such as one installed with the server_install package bundle), the upgrade won't install packages that aren't present.

Upgrading a Non-Global Zone

At this time, you must manually update Oracle Solaris 11 Express non-global zones to keep them in sync with the global zone. After updating the global zone, reboot the system, and halt the non-global zone:

# zoneadm -z my-zone halt

To upgrade the non-global zone my-zone, first detach it as if you were migrating it to another server:

# zoneadm -z my-zone detach

# zoneadm list -iv
  ID NAME             STATUS     PATH                           BRAND    IP
   0 global           running    /                              ipkg     shared
   - my-zone2         installed  /export/zfs/my-zone2           ipkg     shared

Next, issue a zoneadm attach command with the -u option. The -u option upgrades the zone during the reattachment:

# zoneadm -z my-zone attach -u
Log File: /var/tmp/my-zone.attach_log.meay8c
Attaching...

preferred global publisher: solaris
       Global zone version: entire@0.5.11,5.11-0.151.0.1.6:20110504T002250Z
   Non-Global zone version: entire@0.5.11,5.11-0.151.0.1:20101105T054056Z

                     Cache: Using /var/pkg/download.
  Updating non-global zone: Output follows
                Packages to update:    17
           Create boot environment:    No
DOWNLOAD                                  PKGS       FILES    XFER (MB)
Completed                                17/17     447/447    14.7/14.7

PHASE                                        ACTIONS
Removal Phase                                106/106
Install Phase                                115/115
Update Phase                               1734/1734

PHASE                                          ITEMS
Package State Update Phase                     34/34
Package Cache Update Phase                     17/17
Image State Update Phase                         2/2
  Updating non-global zone: Zone updated.
                    Result: Attach Succeeded.

The command compares the global zone's version (5.11-0.151.0.1.6) with the non-global zone's version (5.11-0.151.0.1) and performs the update accordingly. Once the ipkg non-global zone is attached and updated, it can be booted:

#  zoneadm -z my-zone boot
#  zoneadm list -iv
 ID NAME             STATUS     PATH                           BRAND    IP
   0 global           running    /                              ipkg     shared
   1 my-zone          running    /export/zfs/my-zone            ipkg     shared
   - my-zone2         installed  /export/zfs/my-zone2           ipkg     shared

Each non-global zone on the system must be detached, attached/upgraded, and booted in this manner to be in sync with the global zone. Future developments are planned to simplify zone updates, but for now, the process is manual. When Oracle Enterprise Manager Ops Center supports Oracle Solaris 11, it will greatly simplify system management, including tasks for managing operating systems, firmware updates, virtual machines, storage, and network fabrics.

What If the Upgrade Causes a Problem?

How to recover, of course, depends on the nature of the problem. If the global zone upgrade is successful but the non-global zone upgrade exhibits a problem, check the log file produced during the attach -u operation. The log file is labeled with the zone name (for example, /var/tmp/my-zone.attach.log.meay8c). Based on the log file, try to troubleshoot the problem. If necessary, it is possible to get back to the software state that existed prior to the updates, since the non-global zone's clone and the initial BE still exist. Restoring the previous software state is also the approach to take if the global zone is problematic.

To revert to the software state that existed before the upgrades, first halt and detach all non-global zones:

# zoneadm -z my-zone halt
# zoneadm -z my-zone2 halt
# zoneadm list -iv
  ID NAME             STATUS     PATH                           BRAND    IP
   0 global           running    /                              ipkg     shared
   - my-zone          installed  /export/zfs/my-zone            ipkg     shared
   - my-zone2         installed  /export/zfs/my-zone2           ipkg     shared
# zoneadm -z my-zone detach
# zoneadm -z my-zone2 detach

The zoneadm list command then shows only the global zone as running:

# zoneadm list -iv
  ID NAME             STATUS     PATH                           BRAND    IP
   0 global           running    /                              ipkg     shared

Next, activate and boot the original BE, which was called solaris:

# beadm activate solaris
# beadm list
BE        Active Mountpoint Space   Policy Created
--        ------ ---------- -----   ------ -------
S11E_SRU6 N      /          336.37M static 2011-06-02 11:28
solaris   R      -          2.35G   static 2011-05-26 11:09
# reboot

The clone of the non-global zone (my-zone2, which hasn't yet been updated, unlike the non-global zone my-zone) can be attached and booted until the problem is resolved:

# zoneadm -z my-zone2 attach -u 
Log File: /var/tmp/my-zone2.attach_log.mPaq6g
Attaching...

preferred global publisher: solaris
       Global zone version: entire@0.5.11,5.11-0.151.0.1:20101105T054056Z
   Non-Global zone version: entire@0.5.11,5.11-0.151.0.1:20101105T054056Z
                     Cache: Using /var/pkg/download.
  Updating non-global zone: Output follows
No updates necessary for this image.
  Updating non-global zone: Zone updated.
                    Result: Attach Succeeded.
# zoneadm -z my-zone2 boot

As shown in the output above, the global zone and the non-global zone my-zone2 are at the same version level, specifically, the version that existed prior to any updates.

Final Thoughts

BEs in Oracle Solaris 11 Express act as a safety net for upgrades, similar to Live Upgrade environments in Oracle Solaris 10. When updating an Oracle Solaris 11 Express global zone, always create a new BE so you can backtrack. Until Oracle Solaris 11 is released and the zone upgrade process changes, manually update all native non-global zones using the zoneadm -z zonename attach -u command to keep non-global zones in sync with the global zone.

Resources

Here are resources that were referenced earlier in this document:

And here is an additional resource:

Oracle Solaris 11 Express Image Packaging System: http://download.oracle.com/docs/cd/E19963-01/index.html

Revision 1, 07/11/2011