How to Set Up a Repository Mirroring Service with the Oracle Solaris 11 Image Packaging Service

by Glynn Foster

Ensure your systems are up to date and installed with the correct software.


Published August 2014


Want to comment on this article? Post the link on Facebook's OTN Garage page.  Have a similar article to share? Bring it up on Facebook or Twitter and let's discuss.

In Oracle Solaris 11.2, several new improvements were added to the Image Packaging System (IPS) to help administrators install software and manage system updates, and also to help developers create and publish software to IPS repositories. You can read about this list of improvements at Tim Foster's blog.

In this article, we will look at the new repository mirroring service that has been included in Oracle Solaris 11.2 to help administrators keep their local repositories in sync with Oracle's hosted package repositories. We will also look at some of the new capabilities that have been designed around compliance to ensure that your systems are installed with only the software that you expect them to have.

Creating a Local Package Repository

Oracle hosts a number of package repositories publicly that allow administrators to install and update software directly to systems in their data center. However, in many data center environments there are network restrictions that might require administrators to set up local repositories. By creating a local package repositories, administrators can ensure fast, reliable access to software and provide additional mechanisms to constrain the software versions installed on systems (either through package incorporations or freezing) that have gone through internal testing and validation processes to comply with a corporate standard.

In this article, we'll start by first creating a local package repository from the ISO images that are available on the Oracle Technology Network. Let's assume that we have downloaded these images to /mnt. These images contain all the software associated with Oracle Solaris 11.2—both the software that's installed as part of a standard media installation and optional software that's available in the package repository.

root@solaris:# ls -1 /mnt
install-repo.ksh
README-zipped-repo.txt
sol-11_2-repo-1of4.zip
sol-11_2-repo-2of4.zip
sol-11_2-repo-3of4.zip
sol-11_2-repo-4of4.zip
sol-11_2-repo-md5sums.txt

Let's construct a local package repository. We'll first create a ZFS dataset at tank/repositories (assuming there is an additional ZFS pool called tank) in which to host the repository. This means that the repository will exist independently outside the scope of boot environments. We will then run our script (as indicated by the instructions in the README file):

root@solaris:# zfs create -o mountpoint=/repositories tank/repositories
root@solaris:# cd /mnt
root@solaris:/mnt# ./install-repo.ksh -d /repositories
Uncompressing sol-11_2-repo-1of4.zip...done.
Uncompressing sol-11_2-repo-2of4.zip...done.
Uncompressing sol-11_2-repo-3of4.zip...done.
Uncompressing sol-11_2-repo-4of4.zip...done.
Repository can be found in /repositories.

Let's take a look at the resulting /repositories ZFS dataset:

root@solaris:/mnt# cd
root@solaris:# ls -1 /repositories
COPYRIGHT
NOTICES
pkg5.repository
publisher
README-repo-iso.txt

Now that we have copied over the data, let's change our default IPS publisher to point to this new repository. Note, that we have not enabled the IPS repository depot service that allows clients to connect to this repository over HTTP or HTTPS; for now, we'll just access this repository over the standard file system.

root@solaris:~# pkg set-publisher -G http://pkg.oracle.com/solaris/release -g /repositories/ solaris
root@solaris:~# pkg publisher
PUBLISHER                   TYPE     STATUS P LOCATION
solaris                     origin   online F file:///repositories/

To show that the repository is working correctly, let's install the cloud/openstack group package as an example. We will use the presence or absence of OpenStack core components for verifying that our repository syncing is working. By installing the group package, we will also install all the software needed for the OpenStack cloud controller software.

root@solaris:~# pkg install cloud/openstack
           Packages to install: 177
            Services to change:   3
       Create boot environment:  No
Create backup boot environment: Yes
DOWNLOAD                                PKGS         FILES    XFER (MB)   SPEED
Completed                            177/177   17109/17109    58.6/58.6    0B/s

PHASE                                          ITEMS
Installing new actions                   20396/20396
Updating package state database                 Done 
Updating package cache                           0/0 
Updating image state                            Done 
Creating fast lookup database                   Done 
Updating package cache                           1/1 
root@solaris:~# pkg list cloud/openstack*
NAME (PUBLISHER)                                  VERSION                    IFO
cloud/openstack                                   0.2013.2.3-0.175.2.0.0.42.1 i--
cloud/openstack/cinder                            0.2013.2.3-0.175.2.0.0.42.1 i--
cloud/openstack/glance                            0.2013.2.3-0.175.2.0.0.42.1 i--
cloud/openstack/horizon                           0.2013.2.3-0.175.2.0.0.42.1 i--
cloud/openstack/keystone                          0.2013.2.3-0.175.2.0.0.42.1 i--
cloud/openstack/neutron                           0.2013.2.3-0.175.2.0.0.42.1 i--
cloud/openstack/nova                              0.2013.2.3-0.175.2.0.0.42.1 i--
cloud/openstack/swift                             1.10.0-0.175.2.0.0.42.1    i--

Now that we have an IPS repository up and running, let's take a ZFS snapshot so we can have a state to revert back to easily.

root@solaris:~# zfs snapshot tank/repositories@original

Manually Mirroring a Package Repository

Administrators can use the pkgrecv command to retrieve packages from one repository to another. In Listing 1 below, we will use the -m latest option to retrieve the latest versions of all packages—as opposed to fetching every package regardless of the versions—as denoted by the * match for the package Fault Management Resource Identifier (FMRI). We will use the Oracle Solaris release repository as the source repository. We will also turn on verbose mode using the -v option.

root@solaris:~# pkgrecv -s http://pkg.oracle.com/solaris/release -d /repositories -m latest -v *
Processing packages for publisher solaris ...
Retrieving and evaluating 4870 package(s)...

Retrieving packages ...
        Packages to add:        24
      Files to retrieve:     10479
Estimated transfer size: 446.99 MB

Packages to transfer:
cloud/openstack/cinder@0.2013.2.3,5.11-0.175.2.1.0.2.0:20140723T183814Z
cloud/openstack/glance@0.2013.2.3,5.11-0.175.2.1.0.2.0:20140723T183837Z
cloud/openstack/horizon@0.2013.2.3,5.11-0.175.2.1.0.2.0:20140723T183851Z
cloud/openstack/keystone@0.2013.2.3,5.11-0.175.2.1.0.2.0:20140723T183928Z
cloud/openstack/neutron@0.2013.2.3,5.11-0.175.2.1.0.2.0:20140723T183943Z
cloud/openstack/nova@0.2013.2.3,5.11-0.175.2.1.0.2.0:20140723T184006Z
cloud/openstack/swift@1.10.0,5.11-0.175.2.1.0.2.0:20140723T184039Z
consolidation/java-7/java-7-incorporation@1.7.0.65.17,5.11-0:20140625T004018Z
consolidation/java-8/java-8-incorporation@1.8.0.11.12,5.11-0:20140625T004323Z
consolidation/userland/userland-incorporation@0.5.11,5.11-0.175.2.1.0.2.0:20140723T184045Z
developer/java/jdk@1.7.0.65.17,5.11:20140625T004034Z
developer/java/jdk-7@1.7.0.65.17,5.11:20140625T004019Z
developer/java/jdk-8@1.8.0.11.12,5.11:20140625T004324Z
entire@0.5.11,5.11-0.175.2.1.0.2.0:20140723T184046Z
library/java/java-demo@1.7.0.65.17,5.11:20140625T004046Z
library/java/java-demo-7@1.7.0.65.17,5.11:20140625T004035Z
library/java/java-demo-8@1.8.0.11.12,5.11:20140625T004340Z
library/python-2/jsonpatch@1.1,5.11-0.175.2.1.0.2.0:20140723T184048Z
library/python-2/jsonpatch-26@1.1,5.11-0.175.2.1.0.2.0:20140723T184047Z
library/python-2/jsonpatch-27@1.1,5.11-0.175.2.1.0.2.0:20140723T184048Z
runtime/java/jre@1.7.0.65.17,5.11:20140625T004254Z
runtime/java/jre-7@1.7.0.65.17,5.11:20140625T004046Z
runtime/java/jre-8@1.8.0.11.12,5.11:20140625T004347Z
service/network/dnsmasq@2.68,5.11-0.175.2.1.0.2.0:20140723T184049Z

PROCESS                                         ITEMS    GET (MB)   SEND (MB)
Completed                                       24/24 447.0/447.0   1164/1164

Listing 1

In Listing 1, we can see 24 packages have been refreshed in the repository, which affect Java and OpenStack. This was an update that was provided soon after the availability of Oracle Solaris 11.2.

Let's double-check that we have those versions by looking at the cloud/openstack/nova package (a cloud service of the OpenStack infrastructure that handles compute capabilities) on our system, and then check the versions that are in our repository, as shown in Listing 2.

root@solaris:~# pkg list cloud/openstack/nova
NAME (PUBLISHER)                                  VERSION                    IFO
cloud/openstack/nova                              0.2013.2.3-0.175.2.0.0.42.1 i--
root@solaris:~# pkgrepo -s /repositories/ list cloud/openstack/nova
PUBLISHER NAME                                          O VERSION
solaris   cloud/openstack/nova                            0.2013.2.3,5.11-0.175.2.1.0.2.0:20140723T184006Z
solaris   cloud/openstack/nova                            0.2013.2.3,5.11-0.175.2.0.0.42.1:20140623T010318Z

Listing 2

In Listing 2, we can see that version 0.2013.2.3-0.175.2.0.0.42.1 is installed, but we have a new version—0.2013.2.3,5.11-0.175.2.1.0.2.0—in the repository as well. We can now use pkg update to update to this new version, as shown in Listing 3.

root@solaris:~# pkg update
           Packages to install:   2
            Packages to update:  12
       Create boot environment:  No
Create backup boot environment: Yes
DOWNLOAD                                PKGS         FILES    XFER (MB)   SPEED
Completed                              14/14       154/154      0.8/0.8    0B/s

PHASE                                          ITEMS
Removing old actions                           83/83
Installing new actions                       124/124
Updating modified actions                    188/188
Updating package state database                 Done 
Updating package cache                         12/12 
Updating image state                            Done 
Creating fast lookup database                   Done 
Updating package cache                           1/1 

---------------------------------------------------------------------------
NOTE: Please review release notes posted at:

http://www.oracle.com/pls/topic/lookup?ctx=solaris11&id=SERNS
---------------------------------------------------------------------------

root@solaris:~# pkg list cloud/openstack/nova
NAME (PUBLISHER)                                  VERSION                    IFO
cloud/openstack/nova                              0.2013.2.3-0.175.2.1.0.2.0 i--

Listing 3

In Listing 3, we can see that we've installed the latest OpenStack packages, including the Nova compute service. pkgrecv works great to refresh our local package repository.

pkgrecv also takes another option --clone to produce an exact clone of a repository (including all time stamps). If you are copying a repository from scratch, it is recommended to use the --clone option because it result in a faster operation.

Using pkgrecv is a pretty manual process. We could easily create a cron job to do this for us, but instead we will use the new mirroring service feature included in Oracle Solaris 11.2. Using this feature provides a few more benefits that just using cron alone, for example, integration into the Oracle Solaris Service Management Facility (SMF), logging, publisher configuration management, and more.

Before we look at the new IPS package mirroring service, let's roll back our local package repository to the original state and revert the changes we made when we installed the OpenStack packages, as shown in Listing 4. We'll use a new command—pkg exact-install—to install and update our system as if we were installing onto a bare system. This command is especially useful for administrators trying to enforce compliance, for example, to ensure that a validated configuration is used and that no additional unnecessary packages are installed on a system.

This particular system was installed using the Oracle Solaris 11.2 Interactive Text Installer, which uses the solaris-large-server group package as the boundary in which to install a basic server profile. We will use this as the boundary that we want to revert to. Performing this exact installation will cause a new boot environment to be created that contains our changes, and it will be active on our next system reboot.

root@solaris:~# zfs rollback tank/repositories@original
root@solaris:~# pkg exact-install solaris-large-server
            Packages to remove: 198
       Create boot environment: Yes
Create backup boot environment:  No
PHASE                                          ITEMS
Removing old actions                     23784/23784
Updating package state database                 Done 
Updating package cache                       198/198 
Updating image state                            Done 
Creating fast lookup database                   Done 
Updating package cache                           1/1 

A clone of solaris exists and has been updated and activated.
On the next boot the Boot Environment solaris-1 will be
mounted on '/'.  Reboot when ready to switch to this updated BE.

Updating package cache                           1/1 
# reboot

Listing 4

In Listing 4, we can see that 198 packages were removed from the system and a new boot environment called solaris-1 was created. Once the system has rebooted, let's check the status of our system and repository for our OpenStack packages.

root@solaris:~# pkg list cloud/openstack/nova
pkg list: No packages matching 'cloud/openstack/nova' installed
root@solaris:~# pkgrepo -s /repositories/ list cloud/openstack/nova
PUBLISHER NAME                                          O VERSION
solaris   cloud/openstack/nova                            0.2013.2.3,5.11-0.175.2.0.0.42.1:20140623T010318Z

Automatic Mirroring with the IPS Repository Mirroring Service

The IPS package mirroring service, application/pkg/mirror, helps to automate the process of updating package repositories with content from another package repository. As before, we'll update our package repository located at /repositories with the updated content in the Oracle Solaris 11.2 release repository. This service is managed using SMF and is initially disabled, as shown below:

root@solaris:~# svcs pkg/mirror
STATE          STIME    FMRI
disabled        3:29:10 svc:/application/pkg/mirror:default

We will use the standard SMF administrative commands to make the changes that we need to make to the configuration. Let's first check the default configuration of this service instance using the svccfg command, as shown in Listing 5:

root@solaris:~# svccfg -s pkg/mirror:default
svc:/application/pkg/mirror:default> listprop config
config                      application        
config/cache_dir           astring     /var/cache/pkg/mirror
config/crontab_period      astring     "30 2 random * *"
config/debug               boolean     false
config/publishers          astring     solaris
config/ref_image           astring     /
config/repository          astring     /var/share/pkg/repositories/solaris
config/stability           astring     Evolving
config/value_authorization astring     solaris.smf.manage.pkg-mirror

Listing 5

Right away, we can see in Listing 5 that we'll want to change the config/repository property to point to /repositories instead, and depending on when we want to have this service pull updates, we might want to change the config/crontab_period property as well.

For this article, we will pull updates every five minutes so we can see that it is working, but the recommendation would be to pull updates during periods of relative inactivity within the data center, for example, outside of business hours.

svc:/application/pkg/mirror:default> setprop config/repository = astring: "/repositories"
svc:/application/pkg/mirror:default> setprop config/crontab_period = astring: "0,5,10,15,20,25,30,35,40,45,50,55 * * * *"
svc:/application/pkg/mirror:default> listprop config
config                      application        
config/cache_dir           astring     /var/cache/pkg/mirror
config/crontab_period      astring     "0,5,10,15,20,25,30,35,40,45,50,55 * * * *"
config/debug               boolean     false
config/publishers          astring     solaris
config/ref_image           astring     /
config/repository          astring     /repositories
config/stability           astring     Evolving
config/value_authorization astring     solaris.smf.manage.pkg-mirror
svc:/application/pkg/mirror:default> refresh
svc:/application/pkg/mirror:default> exit

By default, the service uses the publisher configuration on the system as the source repositories to mirror. This is configured using the config/ref_image SMF repository (as indicated by the value of /). This means that it will try to pull from file:///repositories/, since that is what is configured as our default publisher.

We have two options. We can either change our publisher back to the Oracle-hosted publisher, or we can create a new image using pkg image-create to provide our correct publisher information and then supply this as the reference image for the mirror service. Let's choose the first option for now (the other example is covered in the product documentation).

We will also need to change the user and group ownership of /repositories, since the mirror service runs as the pkg5srv user; this is a problem only when mirroring to existing repositories.

root@solaris:~# pkg set-publisher -G /repositories -g http://pkg.oracle.com/solaris/release solaris
root@solaris:~# pkg publisher
PUBLISHER                   TYPE     STATUS P LOCATION
solaris                     origin   online F http://pkg.oracle.com/solaris/release/
root@solaris:~# chmod -R pkg5srv:pkg5srv /repositories

Now that we have completed the configuration, let's enable this service instance using the svcadm enable command and then check that it's online using svcs.

root@solaris:~# svcadm enable pkg/mirror:default
root@solaris:~# svcs pkg/mirror:default
STATE          STIME    FMRI
online          4:40:58 svc:/application/pkg/mirror:default

The service logs activity to the /var/log/pkg/mirror/ location. We can check to see that this service is working correctly by checking a few different logs:

root@solaris:# tail -f /var/log/pkg/mirror/mirror.default.log
20140806T044502Z: svc:/application/pkg/mirror:default updates to /repositories from 
http://pkg.oracle.com/solaris/release/ :

^C
root@solaris:/var/log/pkg/mirror# tail -f /var/log/pkg/mirror/mirror.default.log.tmp
Fetching manifests:  2411/10563  22% complete
Fetching manifests:  2450/10563  23% complete
Fetching manifests:  2478/10563  23% complete
Fetching manifests:  2501/10563  23% complete
Fetching manifests:  2529/10563  23% complete
Fetching manifests:  2544/10563  24% complete
Fetching manifests:  2577/10563  24% complete
Fetching manifests:  2596/10563  24% complete
Fetching manifests:  2607/10563  24% complete
Fetching manifests:  2624/10563  24% complete
^C

After several hours (the Oracle Solaris 11 release repository is huge!), the mirroring service will have been completed, and we can check the status of our repository for packages that we'd expect to be in our refreshed repository.

root@solaris:~# pkgrepo -s /repositories/ list cloud/openstack/nova
PUBLISHER NAME                                          O VERSION
solaris   cloud/openstack/nova                            0.2013.2.3,5.11-0.175.2.1.0.2.0:20140723T184006Z
solaris   cloud/openstack/nova                            0.2013.2.3,5.11-0.175.2.0.0.42.1:20140623T010318Z

We can even look at what versions of entire are included in our repository (entire is a package incorporation that represents the release boundaries of Oracle Solaris 11):

root@solaris:~# pkgrepo -s /repositories/ list entire
PUBLISHER NAME                                          O VERSION
solaris   entire                                          0.5.11,5.11-0.175.2.1.0.2.0:20140723T184046Z
solaris   entire                                          0.5.11,5.11-0.175.2.0.0.42.0:20140624T193832Z
solaris   entire                                          0.5.11,5.11-0.175.1.0.0.24.2:20120919T190135Z
solaris   entire                                          0.5.11,5.11-0.175.0.10.1.0.0:20120918T160900Z
solaris   entire                                          0.5.11,5.11-0.175.0.0.0.2.0:20111020T143822Z
solaris   entire                                          0.5.11,5.11-0.151.0.1:20101105T054056Z

Conclusion

The IPS mirroring service is a very useful way of mirroring repositories, both Oracle-hosted repositories and local repositories. Using multiple SMF instances of the application/pkg/mirror service, administrators can quickly sync different repositories to allow better geographical distribution of repositories serving different clients across a global data center environment.

See Also

Also see these additional resources:

About the Author

Glynn Foster is a principal product manager for Oracle Solaris. He is responsible for a number of technology areas including OpenStack, the Oracle Solaris Image Packaging System, installation, and configuration management.

Revision 1.0, 08/13/2014

Follow us:
Blog | Facebook | Twitter | YouTube