by Glynn Foster
Published August 2014
In Oracle Solaris 11.2, several new improvements were added to the Image Packaging System (IPS) to help administrators install software and manage system updates, and also to help developers create and publish software to IPS repositories. You can read about this list of improvements at Tim Foster's blog.
In this article, we will look at the new repository mirroring service that has been included in Oracle Solaris 11.2 to help administrators keep their local repositories in sync with Oracle's hosted package repositories. We will also look at some of the new capabilities that have been designed around compliance to ensure that your systems are installed with only the software that you expect them to have.
Oracle hosts a number of package repositories publicly that allow administrators to install and update software directly to systems in their data center. However, in many data center environments there are network restrictions that might require administrators to set up local repositories. By creating a local package repositories, administrators can ensure fast, reliable access to software and provide additional mechanisms to constrain the software versions installed on systems (either through package incorporations or freezing) that have gone through internal testing and validation processes to comply with a corporate standard.
In this article, we'll start by first creating a local package repository from the ISO images that are available on the Oracle Technology Network. Let's assume that we have downloaded these images to
/mnt. These images contain all the software associated with Oracle Solaris 11.2—both the software that's installed as part of a standard media installation and optional software that's available in the package repository.
root@solaris:# ls -1 /mnt install-repo.ksh README-zipped-repo.txt sol-11_2-repo-1of4.zip sol-11_2-repo-2of4.zip sol-11_2-repo-3of4.zip sol-11_2-repo-4of4.zip sol-11_2-repo-md5sums.txt
Let's construct a local package repository. We'll first create a ZFS dataset at
tank/repositories (assuming there is an additional ZFS pool called
tank) in which to host the repository. This means that the repository will exist independently outside the scope of boot environments. We will then run our script (as indicated by the instructions in the README file):
root@solaris:# zfs create -o mountpoint=/repositories tank/repositories root@solaris:# cd /mnt root@solaris:/mnt# ./install-repo.ksh -d /repositories Uncompressing sol-11_2-repo-1of4.zip...done. Uncompressing sol-11_2-repo-2of4.zip...done. Uncompressing sol-11_2-repo-3of4.zip...done. Uncompressing sol-11_2-repo-4of4.zip...done. Repository can be found in /repositories.
Let's take a look at the resulting
/repositories ZFS dataset:
root@solaris:/mnt# cd root@solaris:# ls -1 /repositories COPYRIGHT NOTICES pkg5.repository publisher README-repo-iso.txt
Now that we have copied over the data, let's change our default IPS publisher to point to this new repository. Note, that we have not enabled the IPS repository depot service that allows clients to connect to this repository over HTTP or HTTPS; for now, we'll just access this repository over the standard file system.
root@solaris:~# pkg set-publisher -G http://pkg.oracle.com/solaris/release -g /repositories/ solaris root@solaris:~# pkg publisher PUBLISHER TYPE STATUS P LOCATION solaris origin online F file:///repositories/
To show that the repository is working correctly, let's install the
cloud/openstack group package as an example. We will use the presence or absence of OpenStack core components for verifying that our repository syncing is working. By installing the group package, we will also install all the software needed for the OpenStack cloud controller software.
root@solaris:~# pkg install cloud/openstack Packages to install: 177 Services to change: 3 Create boot environment: No Create backup boot environment: Yes DOWNLOAD PKGS FILES XFER (MB) SPEED Completed 177/177 17109/17109 58.6/58.6 0B/s PHASE ITEMS Installing new actions 20396/20396 Updating package state database Done Updating package cache 0/0 Updating image state Done Creating fast lookup database Done Updating package cache 1/1 root@solaris:~# pkg list cloud/openstack* NAME (PUBLISHER) VERSION IFO cloud/openstack 0.2013.2.3-0.175.2.0.0.42.1 i-- cloud/openstack/cinder 0.2013.2.3-0.175.2.0.0.42.1 i-- cloud/openstack/glance 0.2013.2.3-0.175.2.0.0.42.1 i-- cloud/openstack/horizon 0.2013.2.3-0.175.2.0.0.42.1 i-- cloud/openstack/keystone 0.2013.2.3-0.175.2.0.0.42.1 i-- cloud/openstack/neutron 0.2013.2.3-0.175.2.0.0.42.1 i-- cloud/openstack/nova 0.2013.2.3-0.175.2.0.0.42.1 i-- cloud/openstack/swift 1.10.0-0.175.2.0.0.42.1 i--
Now that we have an IPS repository up and running, let's take a ZFS snapshot so we can have a state to revert back to easily.
root@solaris:~# zfs snapshot tank/repositories@original
Administrators can use the
pkgrecv command to retrieve packages from one repository to another. In Listing 1 below, we will use the
-m latest option to retrieve the latest versions of all packages—as opposed to fetching every package regardless of the versions—as denoted by the
* match for the package Fault Management Resource Identifier (FMRI). We will use the Oracle Solaris release repository as the source repository. We will also turn on verbose mode using the
root@solaris:~# pkgrecv -s http://pkg.oracle.com/solaris/release -d /repositories -m latest -v * Processing packages for publisher solaris ... Retrieving and evaluating 4870 package(s)... Retrieving packages ... Packages to add: 24 Files to retrieve: 10479 Estimated transfer size: 446.99 MB Packages to transfer: email@example.com,5.11-0.175.2.1.0.2.0:20140723T183814Z firstname.lastname@example.org,5.11-0.175.2.1.0.2.0:20140723T183837Z email@example.com,5.11-0.175.2.1.0.2.0:20140723T183851Z firstname.lastname@example.org,5.11-0.175.2.1.0.2.0:20140723T183928Z email@example.com,5.11-0.175.2.1.0.2.0:20140723T183943Z firstname.lastname@example.org,5.11-0.175.2.1.0.2.0:20140723T184006Z email@example.com,5.11-0.175.2.1.0.2.0:20140723T184039Z firstname.lastname@example.org,5.11-0:20140625T004018Z email@example.com,5.11-0:20140625T004323Z firstname.lastname@example.org,5.11-0.175.2.1.0.2.0:20140723T184045Z email@example.com,5.11:20140625T004034Z firstname.lastname@example.org,5.11:20140625T004019Z email@example.com,5.11:20140625T004324Z firstname.lastname@example.org,5.11-0.175.2.1.0.2.0:20140723T184046Z email@example.com,5.11:20140625T004046Z firstname.lastname@example.org,5.11:20140625T004035Z email@example.com,5.11:20140625T004340Z firstname.lastname@example.org,5.11-0.175.2.1.0.2.0:20140723T184048Z email@example.com,5.11-0.175.2.1.0.2.0:20140723T184047Z firstname.lastname@example.org,5.11-0.175.2.1.0.2.0:20140723T184048Z email@example.com,5.11:20140625T004254Z firstname.lastname@example.org,5.11:20140625T004046Z email@example.com,5.11:20140625T004347Z firstname.lastname@example.org,5.11-0.175.2.1.0.2.0:20140723T184049Z PROCESS ITEMS GET (MB) SEND (MB) Completed 24/24 447.0/447.0 1164/1164
In Listing 1, we can see 24 packages have been refreshed in the repository, which affect Java and OpenStack. This was an update that was provided soon after the availability of Oracle Solaris 11.2.
Let's double-check that we have those versions by looking at the
cloud/openstack/nova package (a cloud service of the OpenStack infrastructure that handles compute capabilities) on our system, and then check the versions that are in our repository, as shown in Listing 2.
root@solaris:~# pkg list cloud/openstack/nova NAME (PUBLISHER) VERSION IFO cloud/openstack/nova 0.2013.2.3-0.175.2.0.0.42.1 i-- root@solaris:~# pkgrepo -s /repositories/ list cloud/openstack/nova PUBLISHER NAME O VERSION solaris cloud/openstack/nova 0.2013.2.3,5.11-0.175.2.1.0.2.0:20140723T184006Z solaris cloud/openstack/nova 0.2013.2.3,5.11-0.175.2.0.0.42.1:20140623T010318Z
In Listing 2, we can see that version
0.2013.2.3-0.175.2.0.0.42.1 is installed, but we have a new version—
0.2013.2.3,5.11-0.175.2.1.0.2.0—in the repository as well. We can now use
pkg update to update to this new version, as shown in Listing 3.
root@solaris:~# pkg update Packages to install: 2 Packages to update: 12 Create boot environment: No Create backup boot environment: Yes DOWNLOAD PKGS FILES XFER (MB) SPEED Completed 14/14 154/154 0.8/0.8 0B/s PHASE ITEMS Removing old actions 83/83 Installing new actions 124/124 Updating modified actions 188/188 Updating package state database Done Updating package cache 12/12 Updating image state Done Creating fast lookup database Done Updating package cache 1/1 --------------------------------------------------------------------------- NOTE: Please review release notes posted at: http://www.oracle.com/pls/topic/lookup?ctx=solaris11&id=SERNS --------------------------------------------------------------------------- root@solaris:~# pkg list cloud/openstack/nova NAME (PUBLISHER) VERSION IFO cloud/openstack/nova 0.2013.2.3-0.175.2.1.0.2.0 i--
In Listing 3, we can see that we've installed the latest OpenStack packages, including the Nova compute service.
pkgrecv works great to refresh our local package repository.
pkgrecv also takes another option
--clone to produce an exact clone of a repository (including all time stamps). If you are copying a repository from scratch, it is recommended to use the
--clone option because it result in a faster operation.
pkgrecv is a pretty manual process. We could easily create a
cron job to do this for us, but instead we will use the new mirroring service feature included in Oracle Solaris 11.2. Using this feature provides a few more benefits that just using
cron alone, for example, integration into the Oracle Solaris Service Management Facility (SMF), logging, publisher configuration management, and more.
Before we look at the new IPS package mirroring service, let's roll back our local package repository to the original state and revert the changes we made when we installed the OpenStack packages, as shown in Listing 4. We'll use a new command—
pkg exact-install—to install and update our system as if we were installing onto a bare system. This command is especially useful for administrators trying to enforce compliance, for example, to ensure that a validated configuration is used and that no additional unnecessary packages are installed on a system.
This particular system was installed using the Oracle Solaris 11.2 Interactive Text Installer, which uses the
solaris-large-server group package as the boundary in which to install a basic server profile. We will use this as the boundary that we want to revert to. Performing this exact installation will cause a new boot environment to be created that contains our changes, and it will be active on our next system reboot.
root@solaris:~# zfs rollback tank/repositories@original root@solaris:~# pkg exact-install solaris-large-server Packages to remove: 198 Create boot environment: Yes Create backup boot environment: No PHASE ITEMS Removing old actions 23784/23784 Updating package state database Done Updating package cache 198/198 Updating image state Done Creating fast lookup database Done Updating package cache 1/1 A clone of solaris exists and has been updated and activated. On the next boot the Boot Environment solaris-1 will be mounted on '/'. Reboot when ready to switch to this updated BE. Updating package cache 1/1 # reboot
In Listing 4, we can see that 198 packages were removed from the system and a new boot environment called
solaris-1 was created. Once the system has rebooted, let's check the status of our system and repository for our OpenStack packages.
root@solaris:~# pkg list cloud/openstack/nova pkg list: No packages matching 'cloud/openstack/nova' installed root@solaris:~# pkgrepo -s /repositories/ list cloud/openstack/nova PUBLISHER NAME O VERSION solaris cloud/openstack/nova 0.2013.2.3,5.11-0.175.2.0.0.42.1:20140623T010318Z
The IPS package mirroring service,
application/pkg/mirror, helps to automate the process of updating package repositories with content from another package repository. As before, we'll update our package repository located at
/repositories with the updated content in the Oracle Solaris 11.2 release repository. This service is managed using SMF and is initially disabled, as shown below:
root@solaris:~# svcs pkg/mirror STATE STIME FMRI disabled 3:29:10 svc:/application/pkg/mirror:default
We will use the standard SMF administrative commands to make the changes that we need to make to the configuration. Let's first check the default configuration of this service instance using the
svccfg command, as shown in Listing 5:
root@solaris:~# svccfg -s pkg/mirror:default svc:/application/pkg/mirror:default> listprop config config application config/cache_dir astring /var/cache/pkg/mirror config/crontab_period astring "30 2 random * *" config/debug boolean false config/publishers astring solaris config/ref_image astring / config/repository astring /var/share/pkg/repositories/solaris config/stability astring Evolving config/value_authorization astring solaris.smf.manage.pkg-mirror
Right away, we can see in Listing 5 that we'll want to change the
config/repository property to point to
/repositories instead, and depending on when we want to have this service pull updates, we might want to change the
config/crontab_period property as well.
For this article, we will pull updates every five minutes so we can see that it is working, but the recommendation would be to pull updates during periods of relative inactivity within the data center, for example, outside of business hours.
svc:/application/pkg/mirror:default> setprop config/repository = astring: "/repositories" svc:/application/pkg/mirror:default> setprop config/crontab_period = astring: "0,5,10,15,20,25,30,35,40,45,50,55 * * * *" svc:/application/pkg/mirror:default> listprop config config application config/cache_dir astring /var/cache/pkg/mirror config/crontab_period astring "0,5,10,15,20,25,30,35,40,45,50,55 * * * *" config/debug boolean false config/publishers astring solaris config/ref_image astring / config/repository astring /repositories config/stability astring Evolving config/value_authorization astring solaris.smf.manage.pkg-mirror svc:/application/pkg/mirror:default> refresh svc:/application/pkg/mirror:default> exit
By default, the service uses the publisher configuration on the system as the source repositories to mirror. This is configured using the
config/ref_image SMF repository (as indicated by the value of
/). This means that it will try to pull from
file:///repositories/, since that is what is configured as our default publisher.
We have two options. We can either change our publisher back to the Oracle-hosted publisher, or we can create a new image using
pkg image-create to provide our correct publisher information and then supply this as the reference image for the mirror service. Let's choose the first option for now (the other example is covered in the product documentation).
We will also need to change the user and group ownership of
/repositories, since the mirror service runs as the
pkg5srv user; this is a problem only when mirroring to existing repositories.
root@solaris:~# pkg set-publisher -G /repositories -g http://pkg.oracle.com/solaris/release solaris root@solaris:~# pkg publisher PUBLISHER TYPE STATUS P LOCATION solaris origin online F http://pkg.oracle.com/solaris/release/ root@solaris:~# chmod -R pkg5srv:pkg5srv /repositories
Now that we have completed the configuration, let's enable this service instance using the
svcadm enable command and then check that it's online using
root@solaris:~# svcadm enable pkg/mirror:default root@solaris:~# svcs pkg/mirror:default STATE STIME FMRI online 4:40:58 svc:/application/pkg/mirror:default
The service logs activity to the
/var/log/pkg/mirror/ location. We can check to see that this service is working correctly by checking a few different logs:
root@solaris:# tail -f /var/log/pkg/mirror/mirror.default.log 20140806T044502Z: svc:/application/pkg/mirror:default updates to /repositories from http://pkg.oracle.com/solaris/release/ : ^C root@solaris:/var/log/pkg/mirror# tail -f /var/log/pkg/mirror/mirror.default.log.tmp Fetching manifests: 2411/10563 22% complete Fetching manifests: 2450/10563 23% complete Fetching manifests: 2478/10563 23% complete Fetching manifests: 2501/10563 23% complete Fetching manifests: 2529/10563 23% complete Fetching manifests: 2544/10563 24% complete Fetching manifests: 2577/10563 24% complete Fetching manifests: 2596/10563 24% complete Fetching manifests: 2607/10563 24% complete Fetching manifests: 2624/10563 24% complete ^C
After several hours (the Oracle Solaris 11 release repository is huge!), the mirroring service will have been completed, and we can check the status of our repository for packages that we'd expect to be in our refreshed repository.
root@solaris:~# pkgrepo -s /repositories/ list cloud/openstack/nova PUBLISHER NAME O VERSION solaris cloud/openstack/nova 0.2013.2.3,5.11-0.175.2.1.0.2.0:20140723T184006Z solaris cloud/openstack/nova 0.2013.2.3,5.11-0.175.2.0.0.42.1:20140623T010318Z
We can even look at what versions of
entire are included in our repository (
entire is a package incorporation that represents the release boundaries of Oracle Solaris 11):
root@solaris:~# pkgrepo -s /repositories/ list entire PUBLISHER NAME O VERSION solaris entire 0.5.11,5.11-0.175.2.1.0.2.0:20140723T184046Z solaris entire 0.5.11,5.11-0.175.2.0.0.42.0:20140624T193832Z solaris entire 0.5.11,5.11-0.175.1.0.0.24.2:20120919T190135Z solaris entire 0.5.11,5.11-0.175.0.10.1.0.0:20120918T160900Z solaris entire 0.5.11,5.11-0.175.0.0.0.2.0:20111020T143822Z solaris entire 0.5.11,5.11-0.151.0.1:20101105T054056Z
The IPS mirroring service is a very useful way of mirroring repositories, both Oracle-hosted repositories and local repositories. Using multiple SMF instances of the
application/pkg/mirror service, administrators can quickly sync different repositories to allow better geographical distribution of repositories serving different clients across a global data center environment.
Also see these additional resources:
Glynn Foster is a principal product manager for Oracle Solaris. He is responsible for a number of technology areas including OpenStack, the Oracle Solaris Image Packaging System, installation, and configuration management.
|Revision 1.0, 08/13/2014|