by Glynn Foster
Published November 2011 (updated June 2013)
Oracle Solaris 11 takes a new approach to lifecycle and package management to greatly simplify the process of managing system software helping reduce the risk of operating system maintenance, including reducing unplanned and planned downtime. With Image Packaging System (IPS), administrators can install and update software from locally connected or remote software package repositories using a much-improved and modernized process.
In this article, we look at what is involved in creating and publishing packages to an IPS repository that can then be installed on Oracle Solaris 11 systems. We will use the popular open source version control system, Bazaar, as an example.
To learn more about IPS, check out a variety of content at the IPS Technology Spotlight page on Oracle Technology Network. Also see the Oracle Solaris 11 Cheat Sheet for Image Packaging System.
Before we dive into the specific details of creating and publishing a package to an IPS repository, here's a summary of the steps we will be taking.
Depending on what software you are planning to package, this step might involve setting up a build environment and running a script to collect some basic information about the state of your system to ensure that dependent packages are installed, that a compiler is available, and how best to install files related to the software onto the file system. We will not go into too much detail for this step because it can vary significantly depending on the software that is to be packaged. We will build our software in our local workspace,
The main part of the process is creating an IPS manifest that describes how the package is put together, including basic information about the package name and version, what files or other content are installed as part of the software package, and what other software dependencies must be met prior to installing the package.
All software packages are hosted by a package repository that systems connect to in order to install software. Repositories can either be locally available on a file system or remote over the network through
https:// connections. This article uses a repository over
http:// and uses the Service Management Facility (SMF) to manage the repository.
After a package manifest has been created and a repository is provisioned to host the package, the final step of the process is to publish the package to the repository so the package can be installed by systems that connect to the repository.
The source code for Bazaar, a Python-based application, can be downloaded from http://launchpad.net/bzr/2.4/2.4.1/+download/bzr-2.4.1.tar.gz.
We're going to unpack the source tarball, run the configuration script and install the software into a temporary location in our workspace that is separate from our final destination file system. This method allows us to determine which files and directories should be part of our software package. This temporary location is sometimes known as a proto area.
# cd $HOME # mkdir proto_install # wget http://launchpad.net/bzr/2.4/2.4.1/+download/bzr-2.4.1.tar.gz # tar -zxf bzr-2.4.1.tar.gz # ls bzr-2.4.1/ proto_install/
Let's now go into the Bazaar source directory.
# cd bzr-2.4.1 # ls apport/ contrib./ man1/ README BRANCH.TODO COPYING.txt MANIFEST.in README_BDIST_RPM bzr/ doc/ NEWS setup.py bzr.ico INSTALL po/ TODO bzrlib Makefile profile_imports.po tools/
Bazaar uses the standard Python mechanism
setup.py to configure, build, and install itself. We can start the build using the
setup.py install target, providing it with the location of our proto area and the "prefix" location of the final destination where we would like to install on our file system, in this case
Building Bazaar requires a C compiler (for convenience, we chosen the GNU GCC compiler in this case) and necessary development headers, which we can quickly install using the Package Manager.
# pkg install developer/gcc-3 system/header # python setup.py install --root=$HOME/proto_install --prefix=/usr
After these commands have completed and a successful build has been achieved, a quick browse around the contents of
proto_install shows the following layout:
# cd $HOME # cd proto_install # ls bin/ lib/ man/ # cd bin # ls bzr
As we can see, within the
bin directory, we now have the recently built
bzr binary, which we can run directly by setting an appropriate
PYTHONPATH environmental variable, as shown in Listing 1.
# export PYTHONPATH=$HOME/proto_install/usr/lib/python2.6/site-packages/ # ./bzr Bazaar 2.4.1 - a free distributed version-control tool http://bazaar-canonical.com/ Basic commands: bzr init makes this directory a versioned branch bzr branch makes a copy of another branch bzr add makes files or directories versioned bzr ignore ignore a file or pattern bzr mv move or rename a versioned file bzr status summarize changes in working copy bzr diff show detailed diffs bzr merge pull in changes from another branch bzr commit save some or all changes bzr send send changes via email bzr log show history of changes bzr check validate storage bzr help init more help on e.g. init command bzr help commands list all commands bzr help topics list all help topics
Listing 1: Running the
We are now ready to take the next step to create an IPS package from the files that were just installed in the
Before we create a manifest, let's explore an existing manifest.
Behind every IPS package is a manifest that provides some basic metadata about the package (name, description, version, category, and so on), what files and directories are included, and what dependencies need to be installed for the package. Packages may specify what services to restart in order to refresh some configuration on the system. They may also specify what aliases to update for a given hardware driver or what users and groups to create as part of the package installation process.
To get familiar with a package manifest, we can use the
pkg contents command to list a manifest that is already associated with the
gzip package and output it into a file
p5m is a standard file name extension convention we use for package manifests), as shown in Listing 2.
# cd $HOME # pkg contents -m gzip > gzip.p5m # cat gzip.p5m set name=pkg.fmri value=pkg://email@example.com,5.11-0.174.0.0.0.0.504:20110920T223615Z set name=info.source-url value=http://alpha.gnu.org/gnu/gzip/gzip-1.3.5.tar.gz set name=pkg.summary value="GNU Zip (gzip)" set name=pkg.description value="The GNU Zip (gzip) compression utility" set name=info.classification value="org.opensolaris.category.2008:Applications/System Utilities" set name=org.opensolaris.arc-caseid value=PSARC/2000/488 set name=info.upstream-url value=http://directory.fsf.org/GNU/gzip.html set name=org.opensolaris.consolidation value=userland license 91533ecafdca90d38563499351a030e497bc9dae chash=33000d74d113ed003308b6b2c6f6565597517ab8 license=GPLv2 pkg.csize=657 pkg.size=1278 set name=variant.arch value=i386 value=sparc depend fmri=pkg:/firstname.lastname@example.org type=require depend fmri=pkg:/email@example.com type=require dir group=sys mode=0755 owner=root path=usr dir group=bin mode=0755 owner=root path=usr/bin dir group=sys mode=0755 owner=root path=usr/share dir facet.doc.info=true group=bin mode=0755 owner=root path=usr/share/info dir facet.doc.man=true group=bin mode=0755 owner=root path=usr/share/man dir facet.doc.man=true group=bin mode=0755 owner=root path=usr/share/man/man1 ...
Listing 2: Listing a Package Manifest
As you can see from Listing 2, each line specifies a part of the package, using a series of actions. Every action has a simple representation:
<action_name> <attribute1=value1> <attribute2=value2> ...
Table 1 lists what actions are available during package manifest creation.Table 1. Package Manifest Actions
| ||Specifies some basic metadata for the package such as a name, description, categorization, and so on.|
| ||Specifies what license is associated with the package and whether, for example, the license must be acknowledged prior to package installation.|
| ||Specifies a single file that is installed by the package. Attributes to the file action specify some basic permissions and where the file is installed.|
| ||Specifies a single directory that is installed by the package. Attributes to the |
| ||Specifies a symbolic link. The |
| ||Specifies a hard link.|
| ||Specifies any dependencies a package has on other software. Different types of dependencies can be specified, from simple required dependencies on other packages, to those that are optional or those that indicate that a package cannot be installed if other software is already installed. For a full list of dependency types, see the Packaging and Delivering Software with the Image Packaging System guide.|
| ||Specifies a device driver that should be installed, along with any aliases that should be set up. The actual driver binary should be installed using a |
| ||Specifies that a user account should be created. Attributes to the |
| ||Specifies that a group should be created. Attributes to the |
| ||Specifies that information should be installed in the legacy package database for SVR4 compatibility to ensure that older packages can be installed that may have to satisfy particular dependencies.|
The package manifest can generally be divided into three different parts:
As we start to generate a manifest for our package, we will tackle these different parts in turn.
To set some of the package metadata, we use the
set action. For this example, we'll pick a few important items, shown in Table 2, and fill in others at a later stage if we need them.
| ||Package name and version|
| ||Package description|
| ||Package summary|
| ||The architectures that the package supports|
| ||A classification scheme that is used to order packages in the Package Manager graphical client|
So in our example, let's provide the following detail in a file called
bzr.mog. We will use the
mog file extension to indicate that this content needs to go through another processing step before we include it in the manifest.
# cat > bzr.mog << EOF set name=pkg.fmri firstname.lastname@example.org,0.1 set name=pkg.description value="Bazaar (bzr) Source Version Control System" set name=pkg.summary value="Bazaar is a source version control system that helps to track project changes over time" set name=variant.arch value=$(ARCH) set name=info.classification value="org.opensolaris.category.2008:Development/Source Code Management" EOF # ls bzr-2.4.1/ gzip.p5m bzr.mog proto_install/
Listing 3: Providing Metadata
Let's quickly discuss the contents of the last two lines. Variants are a feature of IPS that allow multiple architectures to be included in a single package within the repository, reducing the amount of space consumed by multiple packages that might have mostly the same content but are built for different architectures. In this case, we could manually provide either
sparc, but a useful way is to use a macro for which we will substitute later in the process.
Classification determines how the package will be shown within the Package Manager graphical interface. There is a list of classifications detailed in the appendix of the Packaging and Delivering Software with the Image Packaging System guide.
The next step of our manifest creation is specifying what files, directories, links and hard links should be installed as part of the package. In this step, we use the command
pkgsend generate to parse through our proto area, recursively list what it is there with the appropriate manifest action, and then pass this output through the
pkgfmt utility to give us a nicer formatting, as shown in Listing 4.
# pkgsend generate proto_install | pkgfmt > bzr.p5m.1 # cat bzr.p5m.1 dir path=usr owner=root group=bin mode=0755 dir path=usr/bin owner=root group=bin mode=0755 file usr/bin/bzr path=usr/bin/bzr owner=root group=bin mode=0755 dir path=usr/lib owner=root group=bin mode=0755 dir path=usr/lib/python2.6 owner=root group=bin mode=0755 dir path=usr/lib/python2.6/site-packages owner=root group=bin mode=0755 file usr/lib/python2.6/site-packages/bzr-2.4.1-py2.6.egg-info \ path=usr/lib/python2.6/site-packages/bzr-2.4.1-py2.6.egg-info owner=root \ group=bin mode=0644 dir path=usr/lib/python2.6/site-packages/bzrlib owner=root group=bin mode=0755 file usr/lib/python2.6/site-packages/bzrlib/__init__.py \ path=usr/lib/python2.6/site-packages/bzrlib/__init__.py owner=root \ group=bin mode=0644 file usr/lib/python2.6/site-packages/bzrlib/__init__.pyc \ path=usr/lib/python2.6/site-packages/bzrlib/__init__.pyc owner=root \ group=bin mode=0644 .... file usr/lib/python2.6/site-packages/bzrlib/xml_serializer.pyc \ path=usr/lib/python2.6/site-packages/bzrlib/xml_serializer.pyc owner=root \ group=bin mode=0644 dir path=usr/man owner=root group=bin mode=0755 dir path=usr/man/man1 owner=root group=bin mode=0755 file usr/man/man1/bzr.1 path=usr/man/man1/bzr.1 owner=root group=bin mode=0644
Listing 4: Parsing Through the Proto Area
Notice in Listing 4 that you can see a number of
file actions. In particular for the
file action, you can see two paths being specified: one to the final location on the file system where we want to install the package and the other to the location within the
proto_install area, as the following snippet shows:
file usr/lib/python2.6/site-packages/bzrlib/__init__.py \ path=usr/lib/python2.6/site-packages/bzrlib/__init__.py owner=root \ group=bin mode=0644
This is useful because we can directly modify where we want files and directories to be installed without requiring a similar set of modifications in the
It's now a good time to review what contents are proposed to be included in this package. Prior to publishing this package, we can choose to drop certain content that we don't want installed, or we can correct any permissions that are inconsistent or incorrect when compared to the final install location on the file system.
Rather than modifying the manifest directly for these changes, we can simply use
pkgmogrify to provide a set of transforms that we want to make. These transforms can be specified in our earlier
bzr.mog file. We can then merge the contents of these two files, as shown in Listing 5.
# pkgmogrify -DARCH=`uname -p` bzr.p5m.1 bzr.mog | pkgfmt > bzr.p5m.2 # cat bzr.p5m.2 set name=pkg.fmri email@example.com,0.1 set name=pkg.description value="Bazaar (bzr) Source Version Control System" set name=pkg.summary value="Bazaar is a source version control system that helps to track project changes over time" set name=variant.arch value=i386 set name=info.classification value="org.opensolaris.category.2008:Development/Source Code Management" dir path=usr owner=root group=bin mode=0755 dir path=usr/bin owner=root group=bin mode=0755 file usr/bin/bzr path=usr/bin/bzr owner=root group=bin mode=0755 dir path=usr/lib owner=root group=bin mode=0755 dir path=usr/lib/python2.6 owner=root group=bin mode=0755 dir path=usr/lib/python2.6/site-packages owner=root group=bin mode=0755 file usr/lib/python2.6/site-packages/bzr-2.4.1-py2.6.egg-info \ path=usr/lib/python2.6/site-packages/bzr-2.4.1-py2.6.egg-info owner=root \ group=bin mode=0644 .... file usr/lib/python2.6/site-packages/bzrlib/xml_serializer.pyc \ path=usr/lib/python2.6/site-packages/bzrlib/xml_serializer.pyc owner=root \ group=bin mode=0644 dir path=usr/man owner=root group=bin mode=0755 dir path=usr/man/man1 owner=root group=bin mode=0755 file usr/man/man1/bzr.1 path=usr/man/man1/bzr.1 owner=root group=bin mode=0644 # ls bzr-2.4.1/ bzr.p5m.2 bzr.mog gzip.p5m bzr.p5m.1 proto_install/
Listing 5: Using
pkgmogrify to Specify Transforms
In this step we also take the opportunity to define our architecture macro
Having successfully identified the package contents, let's move to the next step and identify any package dependencies we should include.
As mentioned previously, there are a number of package dependencies that can be specified when creating a new package. In this example, we will just focus on
require dependencies, that is, those software packages that must be installed in order for Bazaar to function correctly. Other dependency types are described in the Packaging and Delivering Software with the Image Packaging System guide.
Fortunately, we can automate this step using the
pkgdepend generate command:
# pkgdepend generate -md proto_install bzr.p5m.2 | pkgfmt > bzr.p5m.3 # ls bzr-2.4.1/ bzr.p5m.3 bzr.mog gzip.p5m bzr.p5m.1 proto_install/ bzr.p5m.2
In this step, we have taken the package manifest contents we generated earlier, and for each file listed in the manifest, we do some more analysis on the actual file located in the
proto_install area. This step automatically identifies what file dependencies are required by taking a number of different analytical approaches, for example, ELF headers for compiled binaries,
#! references in shell scripts, and
import statements for Python files.
Looking through the resulting
bzr.p5m.3 file, you can see some additional lines similar to those shown in Listing 6.
depend type=require fmri=__TBD \ pkg.debug.depend.reason=usr/lib/python2.6/site-packages/bzrlib/tests/ssl_certs/create_ssls.py \ pkg.debug.depend.type=python \ pkg.debug.depend.file=cStringIO.py \ pkg.debug.depend.file=cStringIO.pyc \ pkg.debug.depend.file=cStringIO.pyo \ pkg.debug.depend.file=cStringIO.so \ pkg.debug.depend.file=cStringIO/__init__.py \ pkg.debug.depend.file=cStringIOmodule.so \ pkg.debug.depend.path=usr/lib/python2.6 \ pkg.debug.depend.path=usr/lib/python2.6/lib-dynload \ pkg.debug.depend.path=usr/lib/python2.6/lib-old \ pkg.debug.depend.path=usr/lib/python2.6/lib-tk \ pkg.debug.depend.path=usr/lib/python2.6/plat-sunos5 \ pkg.debug.depend.path=usr/lib/python2.6/site-packages \ pkg.debug.depend.path=usr/lib/python2.6/site-packages/bzrlib/tests/ssl_certs \ pkg.debug.depend.path=usr/lib/python2.6/vendor-packages \ pkg.debug.depend.path=usr/lib/python2.6/vendor-packages/gst-0.10 \ pkg.debug.depend.path=usr/lib/python2.6/vendor-packages/gtk-2.0 \ pkg.debug.depend.path=usr/lib/python26.zip
Listing 6: Additional Dependencies
In the output shown in Listing 6, we can see that the
create_ssls.py file depends on a number of other Python files:
cStringIO.pyo, and so on. The next step is resolving these file dependencies for package dependencies that are already installed by using the
pkgdepend resolve command:
# pkgdepend resolve -m bzr.p5m.3 # ls bzr-2.4.1/ bzr.p5m.3 bzr.mog bzr.p5m.3.res bzr.p5m.1 gzip.p5m bzr.p5m.2 proto_install/
After a little while, the
bzr.p5m.3.res file is created, which now has successfully resolved the file dependencies into packages, as shown in Listing 7.
# tail bzr.p5m.3 .... file usr/lib/python2.6/site-packages/bzrlib/xml_serializer.pyc \ path=usr/lib/python2.6/site-packages/bzrlib/xml_serializer.pyc owner=root \ group=bin mode=0644 dir path=usr/man owner=root group=bin mode=0755 dir path=usr/man/man1 owner=root group=bin mode=0755 file usr/man/man1/bzr.1 path=usr/man/man1/bzr.1 owner=root group=bin mode=0644 depend fmri=pkg:/firstname.lastname@example.org type=require depend fmri=pkg:/email@example.com type=require
Listing 7: Resolved File Dependencies
Now that we have completed this step, we have all the basic elements of a package manifest in place: metadata, file and directory contents, and package dependencies.
Now that we have our final manifest, it's a good idea to check it prior to publishing it and the package contents to the repository. This can be done manually or by using the
pkglint tool, which helps check for consistency against other packages already published in the package repository by creating a local cache of content and checking for validity against it.
We won't get into the specifics of linting a package, but a simple lint check can be done using the following command:
# pkglint -c ./lint-cache -r http://pkg.oracle.com/solaris/release bzr.p5m.3.res # ls bzr-2.4.1/ bzr.p5m.3.res bzr.mog gzip.p5m bzr.p5m.1 lint-cache/ bzr.p5m.2 proto_install/ bzr.p5m.3
Quite often file permissions are different on the system to which you are installing and the permissions need to be corrected. Also, default file locations for content such as man pages or system configuration can be different. If you need to correct these types of issues, you can apply additional transforms to the package manifest prior to publication.
In this example, it was necessary to add the following transforms to our original
bzr.mog file to account for man pages in
/usr/share/man and corrected group permissions for
<transform dir path=usr/man -> edit path usr/man usr/share/man> <transform dir path=usr/man/man1 -> edit path usr/man/man1 usr/share/man/man1> <transform file -> edit path usr/share/man1 usr/share/man/man1> <transform dir path=usr$ -> edit group bin sys>
We can start the process again at any stage until we are happy with our final file output. Let's review the steps again.
We start creating the initial manifest by using the
pkgsend generate command to search through our proto area looking for files and directories we want to include in the package.
# pkgsend generate proto_install | pkgfmt > bzr.p5m.1
We then use the
pkgmogrify command to add package metadata along with any transformations we want to make to the package contents.
# pkgmogrify -DARCH=`uname -p` bzr.p5m.1 bzr.mog | pkgfmt > bzr.p5m.2
We then use the
pkgdepend generate command to start to identify any file dependencies we might have by looking at the list of files included in the proto area and identifying what files on the file system are required.
# pkgdepend generate -md proto_install bzr.p5m.2 | pkgfmt > bzr.p5m.3
Finally, we use the
pkgdepend resolve command to take these file dependencies and resolve them into package dependencies.
# pkgdepend resolve -m bzr.p5m.3
The next step in the process is to create and configure an IPS repository that will host our software package. There are a number of different approaches we can take to create a repository (for example, whether we want a file-based repository or something that is available over
https://). However, in this example, we will use SMF to create a new IPS repository service over
http:// that will automatically restart itself in case of a failure, due to the advantages of SMF service restart.
The first thing to do is to create a new ZFS data set that can host our repository. Using a ZFS data set provides more flexibility in terms of storage as the IPS package repository grows, including the ability to snapshot, clone, or send backups to an alternate server. We use the
zfs create command to achieve this by creating a new data set called
repository on top of the existing root zpool called
# zfs create rpool/export/repository
Let's now create an IPS repository within that ZFS data set by using the
pkgrepo create command:
# pkgrepo create /export/repository # ls /export/repository pkg5.repository
As you can see from the output shown above, we now have a new empty repository. We could stop at this step if we only cared about having a file system-based local repository available, but in this example, we would like a repository that is available through
http://. The important thing to note here is that although we have created a place to store our IPS repository, the IPS repository service itself has not been started.
After we have created the initial repository, we need to set a few different properties to reflect our environment. We use the
pkgrepo set command to set the prefix of the repository:
# pkgrepo set -s /export/repository publisher/prefix=bazaar
We then set a few different properties on the SMF service: the location of the repository itself and what port it should reside on, and we also change the state to make sure we can write to it during the publication process.
# svccfg -s application/pkg/server setprop \ pkg/inst_root=/export/repository # svccfg -s application/pkg/server setprop pkg/port=9001 # svccfg -s application/pkg/server setprop pkg/readonly=false
We now need to start the SMF service using the
# svcadm enable application/pkg/server # svcs application/pkg/server STATE STIME FMRI online 2:16:21 svc:/application/pkg/server:default
Now that we have a running service, we can use the
pkgrepo info command to get some information about the repository:
# pkgrepo info -s http://localhost:9001 PUBLISHER PACKAGES STATUS UPDATED bazaar 0 online 2011-10-10T13:16:21.193453Z
We can see the repository contains no packages currently. We can now move on to the next step, which is publishing our package to the repository.
We are now in the position where we can publish our package to the repository. For this, we use the
pkgsend publish command:
# pkgsend publish -s http://localhost:9001 -d proto_install bzr.p5m.3.res PUBLISHED pkg://bazaar/developer/versioning/bzr-2.4.1,0.1:10111010T144709Z
As we can see, the package has successfully been published. We can now check the status of our repository again:
# pkgrepo info -s http://localhost:9001 PUBLISHER PACKAGES STATUS UPDATED bazaar 1 online 2011-10-10T14:47:29.298624Z
We can see that we now have one package available at this repository. We can add this repository to our configuration and install a package from it, as shown in Listing 8.
# pkg set-publisher -p http://localhost:9001 Added publisher(s): bazaar # pkg info -r bzr Name: developer/versioning/bzr Summary: Bazaar is a source version control system that helps to track project changes over time Description: Bazaar (bzr) Source Version Control System Category: Development/Source Code Management State: Not installed Publisher: bazaar Version: 2.4.1 Build Release: 0.1 Branch: None Packaging Date: October 10, 2011 02:47:09 PM Size: 25.64 MB FMRI: pkg://firstname.lastname@example.org,0.1:20111010T144709Z # pkg install bzr Packages to install: 1 Create boot environment: No Create backup boot environment: No DOWNLOAD PKGS FILES XFER (MB) Completed 1/1 1617/1617 7.0/7.0 PHASE ACTIONS Install Phase 1693/1693 PHASE ITEMS Package State Update Phase 1/1 Image State Update Phase 2/2
Listing 8: Adding the Repository and Installing a Package from It
Now that we see the package can be successfully installed, we should ensure that the repository is set to read-only once again:
# svccfg -s application/pkg/server setprop pkg/readonly=true # svccfg -s application/pkg/server refresh # svcadm restart application/pkg/server
Image Packaging System is a major advancement in software management on Oracle Solaris 11, providing a much-improved and automated way of distributing and installing software over the network across many different clients. After a few basics are understood, creating packages for your software is an easy process and helps reduce errors during software updates, providing a much more reliable and consistent experience in the data center.
Glynn Foster is a Principal Product Manager for Oracle Solaris and works on technology areas that include the Image Packaging System and Service Management Facility. Glynn joined Oracle in 2010 as part of the Sun Microsystems acquisition.
|Revision 1.1, fixed typo in commands, 06/19/2013|