How to Set Up a Repository Mirroring Service with the Oracle Solaris 11 Image Packaging Service
By Glynn Foster
Published August 2014, updated June 2018
Ensure your systems are up to date and installed with the correct software.
In Oracle Solaris 11.2, several new improvements were added to the Image Packaging System (IPS) to help administrators install software and manage system updates, and also to help developers create and publish software to IPS repositories. You can read about this list of improvements at Tim Foster’s blog.
In this article, we will look at the repository mirroring service that was included in Oracle Solaris 11.2 to help administrators keep their local repositories in sync with Oracle’ss hosted package repositories. We will also look at some of the capabilities that were designed around compliance to ensure that your systems are installed with only the software that you expect them to have.
Creating a Local Package Repository
Oracle hosts a number of package repositories publicly that allow administrators to install and update software directly to systems in their data center. However, in many data center environments there are network restrictions that might require administrators to set up local repositories. By creating a local package repositories, administrators can ensure fast, reliable access to software updates and provide additional mechanisms to constrain the software versions installed on systems (either through package incorporations or freezing) that have gone through internal testing and validation processes to comply with a corporate standard.
In this article, we’ll start by first creating a local package repository from the ISO images that are available on the Oracle Technology Network. Let’s assume that we have downloaded these images to /mnt. These images contain all the software associated with Oracle Solaris 11.3—both the software that’s installed as part of a standard media installation and optional software that’s available in the package repository.
root@solaris:~# ls -1 /mnt
README-zipped-repo.txt
install-repo.ksh
sol-11_3-repo_1of5.zip
sol-11_3-repo_2of5.zip
sol-11_3-repo_3of5.zip
sol-11_3-repo_4of5.zip
sol-11_3-repo_5of5.zip
sol-11_3-repo_md5sums.txt
Let’s construct a local package repository. We’ll first create a ZFS dataset at tank/repositories (assuming there is an additional ZFS pool called tank) in which to host the repository. This means that the repository will exist independently outside the scope of boot environments. We will then run our script (as indicated by the instructions in the README file):
root@solaris:~# zfs create -o mountpoint=/repositories tank/repositories
root@solaris:~# zfs set atime=off tank/repositories
root@solaris:~# zfs set compression=on tank/repositories
root@solaris:~# zfs get atime,compression tank/repositories
NAME PROPERTY VALUE SOURCE
tank/repositories atime off local
tank/repositories compression on local
root@solaris:~# cd /mnt
root@solaris:/mnt# ./install-repo.ksh -d /repositories
Using sol-11_3-repo download.
Uncompressing sol-11_3-repo_1of5.zip...done.
Uncompressing sol-11_3-repo_2of5.zip...done.
Uncompressing sol-11_3-repo_3of5.zip...done.
Uncompressing sol-11_3-repo_4of5.zip...done.
Uncompressing sol-11_3-repo_5of5.zip...done.
Repository can be found in /repositories.
Let’s take a look at the resulting /repositories ZFS dataset:
root@solaris:/mnt# cd
root@solaris:# ls -1 /repositories
COPYRIGHT
NOTICES
README-repo-iso.txt
pkg5.repository
publisher
readme.txt
Now that we have copied over the data, let’s change our default IPS publisher to point to this new repository. Note, that we have not enabled the IPS repository depot service that allows clients to connect to this repository over HTTP or HTTPS; for now, we’ll just access this repository over the standard file system.
root@solaris:~# pkg set-publisher -G http://pkg.oracle.com/solaris/release -g /repositories/ solaris
root@solaris:~# pkg publisher
PUBLISHER TYPE STATUS P LOCATION
solaris origin online F file:///repositories/
To show that the repository is working correctly, let’s install the developer-gnu group package as an example. We will use the presence or absence of these GNU components for verifying that our repository syncing is working. By installing the group package, we will also install all the software needed for a typical GNU developer toolchain.
root@solaris:~# pkg install developer-gnu
Packages to install: 84
Mediators to change: 2
Services to change: 6
Create boot environment: No
Create backup boot environment: No
DOWNLOAD PKGS FILES XFER (MB) SPEED
Completed 84/84 12666/12666 217.9/217.9 0B/s
PHASE ITEMS
Installing new actions 16680/16680
Updating package state database Done
Updating package cache 0/0
Updating image state Done
Creating fast lookup database Done
Updating package cache 1/1
root@solaris:~# pkg list developer*
NAME (PUBLISHER) VERSION IFO
developer/assembler 0.5.11-0.175.3.0.0.25.0 i--
developer/base-developer-utilities 0.5.11-0.175.3.0.0.30.0 i--
developer/build/autoconf 2.68-0.175.3.0.0.30.0 i--
developer/build/automake 1.11.2-0.175.3.0.0.30.0 i--
developer/build/automake-111 1.11.2-0.175.3.0.0.30.0 i--
developer/build/gnu-make 3.82-0.175.3.0.0.30.0 i--
developer/build/libtool 2.4.2-0.175.3.0.0.30.0 i--
developer/build/make 0.5.11-0.175.2.0.0.34.0 i--
developer/build/pkg-config 0.23-0.175.3.0.0.30.0 i--
developer/debug/gdb 7.6-0.175.3.0.0.30.0 i--
developer/debug/mdb 0.5.11-0.175.3.1.0.5.0 i--
developer/documentation-tool/openjade 1.3.2-0.175.2.0.0.24.0 i--
developer/documentation-tool/opensp 1.5.1-0.175.2.0.0.24.0 i--
developer/gcc 4.8.2-0.175.3.0.0.30.0 i--
developer/gcc-4/gcc-c++-48 4.8.2-0.175.3.0.0.30.0 i--
developer/gcc-4/gcc-c-48 4.8.2-0.175.3.0.0.30.0 i--
developer/gcc-4/gcc-common-48 4.8.2-0.175.3.0.0.30.0 i--
developer/gcc-4/gcc-gfortran-48 4.8.2-0.175.3.0.0.30.0 i--
developer/gcc-4/gcc-gobjc-48 4.8.2-0.175.3.0.0.30.0 i--
developer/gcc-48 4.8.2-0.175.3.0.0.30.0 i--
developer/gnome/gettext 2.30.0-0.175.3.0.0.10.0 i--
developer/gnu-binutils 2.23.1-0.175.3.0.0.30.0 i--
developer/lexer/flex 2.5.35-0.175.3.0.0.30.0 i--
developer/macro/cpp 0.5.11-0.175.2.0.0.6.0 i--
developer/macro/gnu-m4 1.4.12-0.175.3.0.0.30.0 i--
developer/parser/bison 2.3-0.175.3.0.0.30.0 i--
developer/versioning/cvs 1.12.13-0.175.3.0.0.30.0 i--
developer/versioning/git 1.7.9.2-0.175.3.0.0.30.0 i--
developer/versioning/mercurial 3.4-0.175.3.0.0.30.0 i--
developer/versioning/mercurial-26 3.4-0.175.3.0.0.30.0 i--
developer/versioning/mercurial-27 3.4-0.175.3.0.0.30.0 i--
developer/versioning/sccs 0.5.11-0.175.3.0.0.10.0 i--
developer/versioning/subversion 1.7.20-0.175.3.0.0.30.0 i--
Now that we have an IPS repository up and running, let’s take a ZFS snapshot so we can have a state to revert back to easily.
root@solaris:~# zfs snapshot tank/repositories@original
Manually Mirroring a Package Repository
Administrators can use the pkgrecv command to retrieve packages from one repository to another. In Listing 1 below, we will use the -m latest option to retrieve the latest versions of all packages—as opposed to fetching every package regardless of the versions—as denoted by the * match for the package Fault Management Resource Identifier (FMRI). We will use the Oracle Solaris support repository as the source repository. To do this we’re using our My Oracle Support certificates that give us access to this repository. For more details refer to the documentation. Also, we will turn on verbose mode using the -v option.
root@solaris:~# ls
pkg.oracle.com.certificate.pem pkg.oracle.com.key.pem
root@solaris:~# pkgrecv -s https://pkg.oracle.com/solaris/support -d /repositories -m latest \
--key ./pkg.oracle.com.key.pem --cert ./pkg.oracle.com.certificate.pem -v '*'
Processing packages for publisher solaris ...
Retrieving and evaluating 7537 package(s)...
Retrieving packages ...
Packages to add: 3751
Files to retrieve: 362564
Estimated transfer size: 10.74 GB
Packages to transfer:
SUNWIPython@0.5.11,5.11-0.175.3.20.0.1.0:20170412T185710Z
SUNWPython26@2.6.4,5.11-0.175.3.20.0.1.0:20170412T185712Z
SUNWPython26-extra@0.5.11,5.11-0.175.3.20.0.1.0:20170412T185711Z
...
x11/server/xvnc@1.7.1,5.11-0.175.3.31.0.4.1547:20180330T234729Z
x11/session/sessreg@1.1.0,5.11-0.175.3.2.0.2.1493:20151020T015534Z
x11/xfs@1.1.4,5.11-0.175.3.7.0.3.1501:20160328T221125Z
PROCESS ITEMS GET (MB) SEND (MB)
Completed 3751/3751 10995/10995 29301/29301
Listing 1
In Listing 1, we can see 3751 packages have been refreshed in the repository. This is SRU 32 which has updates across the board also including newer versions of the various GNU tools as part of our commitment keep FOSS updated.
Let’s double-check that we have those versions by looking at the developer/gcc package on our system, and then check the versions that are in our repository, as shown in Listing 2.
root@solaris:~# pkg list developer/gcc
NAME (PUBLISHER) VERSION IFO
developer/gcc 4.8.2-0.175.3.0.0.30.0 i--
root@solaris:~# pkgrepo -s /repositories/ list developer/gcc
PUBLISHER NAME O VERSION
solaris developer/gcc 5.4.0,5.11-0.175.3.22.0.1.0:20170620T160441Z
solaris developer/gcc 4.8.2,5.11-0.175.3.0.0.30.0:20150821T164527Z
Listing 2
In Listing 2, we can see that version 4.8.2-0.175.3.0.0.30.0 is installed, but we have a new version—5.4.0,5.11-0.175.3.22.0.1.0—in the repository as well. We can now use pkg update to update to this new version, as shown in Listing 3. Note we use the -- accept flag to accept the licenses of the newer software.
root@solaris:~# pkg update --accept
Package: pkg://solaris/consolidation/osnet/osnet-incorporation@0.5.11,5.11-0.175.3.32.0.4.0:20180426T184953Z
License: lic_OTN
You acknowledge that your use of this Oracle Solaris software product
is subject to, and may not exceed the use for which you are authorized,
(i) the license or cloud services terms that you accepted when you
obtained the right to use Oracle Solaris software; or (ii) the license
terms that you agreed to when you placed your Oracle Solaris software
order with Oracle; or (iii) the Oracle Solaris software license terms
included with the hardware that you acquired from Oracle; or, if (i),
(ii) or (iii) are not applicable, then, (iv) the OTN License Agreement
for Oracle Solaris (which you acknowledge you have read and agree to)
available at
http://www.oracle.com/technetwork/licenses/solaris-cluster-express-license-167852.html.
Note: Software downloaded for trial use or downloaded as replacement
media may not be used to update any unsupported software.
Packages to remove: 28
Packages to install: 44
Packages to update: 483
Packages to change: 1
Mediators to change: 3
Create boot environment: Yes
Create backup boot environment: No
DOWNLOAD PKGS FILES XFER (MB) SPEED
Completed 556/556 38685/38685 1188/1188 0B/s
PHASE ITEMS
Removing old actions 13786/13786
Installing new actions 24188/24188
Updating modified actions 24545/24545
Updating package state database Done
Updating package cache 511/511
Updating image state Done
Creating fast lookup database Done
Updating package cache 1/1
A clone of solaris exists and has been updated and activated.
On the next boot the Boot Environment solaris-1 will be
mounted on ‘/’. Reboot when ready to switch to this updated BE.
Updating package cache 1/1
---------------------------------------------------------------------------
NOTE: Please review release notes posted at:
http://www.oracle.com/pls/topic/lookup?ctx=solaris11&id=SERNS
---------------------------------------------------------------------------
root@solaris:~# reboot
root@solaris:~# pkg list developer/gcc
NAME (PUBLISHER) VERSION IFO
developer/gcc 5.4.0-0.175.3.22.0.1.0 i--
Listing 3
In Listing 3, we can see that we’ve installed the latest GNU packages, including the gcc compiler. pkgrecv works great to refresh our local package repository. A new boot environment called solaris-1 was created. And the system was rebooted to move to this new boot environment.
pkgrecv also takes another option --clone to produce an exact clone of a repository (including all time stamps). If you are copying a repository from scratch, it is recommended to use the --clone option because it result in a faster operation.
Using pkgrecv is a pretty manual process. We could easily create a cron job to do this for us, but instead we will use the new mirroring service feature included since Oracle Solaris 11.2. Using this feature provides a few more benefits that just using cron alone, for example, integration into the Oracle Solaris Service Management Facility (SMF), logging, publisher configuration management, and more.
Before we look at the new IPS package mirroring service, let’s roll back our local package repository to the original state and revert the changes we made when we installed the GNU developer packages, as shown in Listing 4. First we roll back to the original boot environment by activating it and then rebooting, then we roll back the zfs snapshot on the other zpool. And finally we’ll use the command—pkg exact-install—to install and update our system as if we were installing onto a bare system. This command is especially useful for administrators trying to enforce compliance, for example, to ensure that a validated configuration is used and that no additional unnecessary packages are installed on a system.
This particular system was installed using the Oracle Solaris 11.3 Interactive Text Installer, which uses the solaris-large-server group package as the boundary in which to install a basic server profile. We will use this as the boundary that we want to revert to. Because in this case we’ve already moved back and rebooted to the older boot environment, performing this exact installation not will cause a new boot environment, and only packages will be removed. No additional reboot is necessary.
root@solaris:~# beadm activate solaris
root@solaris:~# beadm list
BE Flags Mountpoint Space Policy Created
-- ----- ---------- ----- ------ -------
solaris R - 12.51G static 2018-06-01 17:04
solaris-1 N / 3.66G static 2018-06-03 23:15
root@solaris:~# reboot
...
root@solaris:~# zfs rollback tank/repositories@original
root@solaris:~# pkg exact-install solaris-large-server
Packages to remove: 100
Mediators to change: 2
Services to change: 6
Create boot environment: No
Create backup boot environment: No
PHASE ITEMS
Removing old actions 19015/19015
Updating package state database Done
Updating package cache 100/100
Updating image state Done
Creating fast lookup database Done
Updating package cache 1/1
The following unexpected or editable files and directories were
salvaged while executing the requested package operation; they
have been moved to the displayed location in the image:
tc/gconf/gconf.xml.defaults -> /var/pkg/lost+found/etc/gconf/gconf.xml.defaults-20180603T232530Z
Listing 4
In Listing 4, we can see that 100 packages were removed from the system, let’s check the status of our system and repository for our GNU Developer packages.
root@solaris:~# pkg list developer/gcc
pkg list: No packages matching 'developer/gcc' installed
root@solaris:~# pkgrepo -s /repositories/ list developer/gcc
PUBLISHER NAME O VERSION
solaris developer/gcc 4.8.2,5.11-0.175.3.0.0.30.0:20150821T164527Z
Automatic Mirroring with the IPS Repository Mirroring Service
The IPS package mirroring service, application/pkg/mirror, helps to automate the process of updating package repositories with content from another package repository. As before, we’ll update our package repository located at /repositories with the updated content in the Oracle Solaris 11.3 support repository. Note, this will take a very long time and quite some space as the support repository is pretty large by now. This service is managed using SMF and is initially disabled, as shown below:
root@solaris:~# svcs pkg/mirror
STATE STIME FMRI
disabled 23:23:17 svc:/application/pkg/mirror:default
We will use the standard SMF administrative commands to make the changes that we need to make to the configuration. Let’s first check the default configuration of this service instance using the svccfg command, as shown in Listing 5:
root@solaris:~# svccfg -s pkg/mirror:default
svc:/application/pkg/mirror:default> listprop config
config application
config/cache_dir astring /var/cache/pkg/mirror
config/crontab_period astring "30 2 random * *"
config/debug boolean false
config/publishers astring solaris
config/ref_image astring /
config/repository astring /var/share/pkg/repositories/solaris
config/stability astring Evolving
config/value_authorization astring solaris.smf.manage.pkg-mirror
Listing 5
Right away, we can see in Listing 5 that we’ll want to change the config/repository property to point to /repositories instead, and depending on when we want to have this service pull updates, we might want to change the config/crontab_period property as well.
For this article, we will pull updates every five minutes so we can see that it is working, but the recommendation would be to pull updates during periods of relative inactivity within the data center, for example, outside of business hours.
svc:/application/pkg/mirror:default> setprop config/repository = astring: "/repositories"
svc:/application/pkg/mirror:default> setprop config/crontab_period = astring: "0,5,10,15,20,25,30,35,40,45,50,55 * * * *"
svc:/application/pkg/mirror:default> listprop config
config application
config/cache_dir astring /var/cache/pkg/mirror
config/debug boolean false
config/publishers astring solaris
config/ref_image astring /
config/stability astring Evolving
config/value_authorization astring solaris.smf.manage.pkg-mirror
config/repository astring /repositories
config/crontab_period astring "0,5,10,15,20,25,30,35,40,45,50,55 * * * *"
svc:/application/pkg/mirror:default> refresh
svc:/application/pkg/mirror:default> exit
By default, the service uses the publisher configuration on the system as the source repositories to mirror. This is configured using the config/ref_image SMF repository (as indicated by the value of /). This means that it will try to pull from file:///repositories/, since that is what is configured as our default publisher.
We have two options. We can either change our publisher back to the Oracle-hosted publisher, or we can create a new image using pkg image-create to provide our correct publisher information and then supply this as the reference image for the mirror service. Let’s choose the first option for now (the other example is covered in the product documentation).
We will also need to change the user and group ownership of /repositories, since the mirror service runs as the pkg5srv user; this is a problem only when mirroring to existing repositories.
root@solaris:~# ls
pkg.oracle.com.certificate.pem pkg.oracle.com.key.pem
root@solaris:~# pkg set-publisher -k pkg.oracle.com.key.pem -c pkg.oracle.com.certificate.pem \
-G /repositories -g https://pkg.oracle.com/solaris/support solaris
root@solaris:~# pkg publisher
PUBLISHER TYPE STATUS P LOCATION
solaris origin online F https://pkg.oracle.com/solaris/support/
root@solaris:~# chown -R pkg5srv:pkg5srv /repositories
Now that we have completed the configuration, let’s enable this service instance using the svcadm enable command and then check that it’s online using svcs.
root@solaris:~# svcadm enable pkg/mirror:default
root@solaris:~# svcs pkg/mirror:default
STATE STIME FMRI
online 23:38:00 svc:/application/pkg/mirror:default
The service logs activity to the /var/log/pkg/mirror/ location. We can check to see that this service is working correctly by checking a few different logs:
root@solaris:# tail -f /var/log/pkg/mirror/mirror.default.log
20180603T234001Z: svc:/application/pkg/mirror:default updates to /repositories from
https://pkg.oracle.com/solaris/support/ :
^C
root@solaris:/var/log/pkg/mirror# tail -f /var/log/pkg/mirror/mirror.default.log.tmp
Processing packages for publisher solaris ...
Retrieving target catalog 'solaris' ... Done
Retrieving catalog 'solaris' ... Done
Retrieving and evaluating 26229 package(s)...
Fetching manifests: 1/26229 0% complete
Fetching manifests: 52/26229 0% complete
Fetching manifests: 56/26229 0% complete
Fetching manifests: 96/26229 0% complete
Fetching manifests: 162/26229 0% complete
Fetching manifests: 211/26229 0% complete
Fetching manifests: 268/26229 1% complete
Fetching manifests: 299/26229 1% complete
Fetching manifests: 359/26229 1% complete
^C
After many hours (the Oracle Solaris 11 support repository is huge!), the mirroring service will have been completed, and we can check the status of our repository for packages that we’d expect to be in our refreshed repository.
root@solaris:~# pkgrepo -s /repositories/ list developer/gcc
PUBLISHER NAME O VERSION
solaris developer/gcc 5.4.0,5.11-0.175.3.22.0.1.0:20170620T160441Z
solaris developer/gcc 4.8.2,5.11-0.175.3.0.0.30.0:20150821T164527Z
solaris developer/gcc 4.8.2,5.11-0.175.2.6.0.3.0:20141222T164952Z
solaris developer/gcc 4.8.2,5.11-0.175.2.0.0.42.1:20140623T012838Z
We can even look at what versions of entire are included in our repository (entire is a package incorporation that represents the release boundaries of Oracle Solaris 11):
root@solaris:~# pkgrepo -s /repositories/ list entire
PUBLISHER NAME O VERSION
solaris entire 0.5.11,5.11-0.175.3.32.0.4.0:20180427T232405Z
solaris entire 0.5.11,5.11-0.175.3.31.0.6.0:20180414T162528Z
solaris entire 0.5.11,5.11-0.175.3.30.0.5.0:20180322T171255Z
solaris entire 0.5.11,5.11-0.175.3.30.0.4.0:20180306T161112Z
...
solaris entire 0.5.11,5.11-0.151.0.1.2:20110127T225841Z
solaris entire 0.5.11,5.11-0.151.0.1.2:20110111T213419Z
solaris entire 0.5.11,5.11-0.151.0.1.1:20101222T214417Z
solaris entire 0.5.11,5.11-0.151.0.1:20101105T054056Z
It works!
A final remark is on storage usage for this and the need to plan for this. There are two locations that potentially will require a lot of space. The first is expected, the location of the repository, which in this case is /repositories. The second is /var/cache/pkg/mirror/default, this will fill while the mirror service is being used to fill the repository. In this case because we used mirror to fill the full repository it is also very large. to work around this in this case a symbolic link was created to a directory to the /tank pool which has plenty of space.
root@solaris:~# ls -la /var/cache/pkg/mirror
total 7
drwxr-xr-x 2 pkg5srv pkg5srv 3 Jun 6 23:42 .
drwxr-xr-x 5 pkg5srv bin 5 Oct 6 2015 ..
lrwxrwxrwx 1 root root 13 Jun 6 23:42 default -> /tank/default
root@solaris:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 19.7G 19.4G 73.5K /rpool
rpool/ROOT 16.6G 19.4G 31K legacy
rpool/ROOT/initial 98.0M 19.4G 11.4G /
rpool/ROOT/initial/var 1.95M 19.4G 238M /var
rpool/ROOT/solaris 12.9G 19.4G 10.8G /
rpool/ROOT/solaris-1 3.58G 19.4G 12.8G /
rpool/ROOT/solaris-1/var 65.8M 19.4G 183M /var
rpool/ROOT/solaris/var 1.38G 19.4G 1.15G /var
rpool/VARSHARE 2.65M 19.4G 2.56M /var/share
rpool/VARSHARE/pkg 63K 19.4G 32K /var/share/pkg
rpool/VARSHARE/pkg/repositories 31K 19.4G 31K /var/share/pkg/repositories
rpool/VARSHARE/zones 31K 19.4G 31K /system/zones
rpool/dump 2.06G 19.5G 2.00G -
rpool/export 99.5K 19.4G 32K /export
rpool/export/home 67.5K 19.4G 32K /export/home
rpool/export/home/demo 35.5K 19.4G 35.5K /export/home/demo
rpool/swap 1.03G 19.5G 1.00G -
tank 278G 211G 139G /tank
tank/repositories 138G 211G 138G /repositories
The full repository now is 138GB, and also note that the total on /tank is 278GB. This is in part because of the symbolic link that was created. So plan to have enough storage space.
Conclusion
The IPS mirroring service is a very useful way of mirroring repositories, both Oracle-hosted repositories and local repositories. Using multiple SMF instances of the application/pkg/mirror service, administrators can quickly sync different repositories to allow better geographical distribution of repositories serving different clients across a global data center environment.
See Also
- "How to Automatically Copy a Repository From the Internet" in the product documentation
- Creating Package Repositories in Oracle Solaris 11.4 in the product documentation
- Oracle Solaris 11 Lifecycle Management technology spotlight page
Also see these additional resources:
- Download Oracle Solaris 11
- Access Oracle Solaris 11 product documentation
- Access all Oracle Solaris 11 how-to articles
- Learn more with Oracle Solaris 11 training and support
- See the official Oracle Solaris blog
- Follow Oracle Solaris on Facebook and Twitter
About the Author
Glynn Foster is a principal product manager for Oracle Solaris. He is responsible for a number of technology areas including OpenStack, the Oracle Solaris Image Packaging System, installation, and configuration management.
| Revision 1.2, 06/14/2018 |
| Revision 1.1, 01/13/2015 |
| Revision 1.0, 08/13/2014 |