by Glynn Foster
Published August 2012
The Service Management Facility (SMF), first introduced in Oracle Solaris 10, is a feature of the operating system for managing system and application services, and it replaces the legacy
init scripting start-up mechanism common to prior releases of Oracle Solaris and other UNIX operating systems. SMF improves the availability of a system by ensuring that essential system and application services run continuously even in the event of any hardware or software failures. SMF is one of the components of the wider Oracle Solaris Predictive Self Healing capability.
Another article, "Introducing the Basics of Service Management Facility (SMF) on Oracle Solaris 11," provided a quick overview of SMF and some basic examples of how to administer systems with SMF, including how to get information about running services, how to start and stop instances of a service, and how to do basic service configuration. It also provided some troubleshooting tips in the event that things go wrong. It is recommended that you read that article first if you are unfamiliar with SMF.
This article covers some more-advanced administrative tasks with SMF, including an introduction to service manifests, understanding layering within the SMF configuration repository, and how best to apply configuration to a system. To learn more about SMF, check out a variety of content at the SMF technology page on Oracle Technology Network.
At the core of SMF is the concept of a service bundle, an XML-based file that is used to describe everything about a service or an instance of a service—including its configuration—prior to it being imported into the SMF configuration repository. Service bundles are used to deliver services into Oracle Solaris, but they also act as a useful way for administrators to deliver custom configuration across a variety of systems.
Two types of service bundles are used in SMF: manifests and profiles. Both are based on an XML DTD found in
/usr/share/lib/xml/dtd/service_bundle.dtd.1 and documented in the
service_bundle(4) manual page.
Manifests are used to describe services and instances of a service, including any property groups and properties they might have. Manifests are stored in
/lib/svc/manifest and automatically imported into the SMF repository. SMF automatically detects if there have been any changes to the manifest and ensures that these changes have been reflected prior to bringing the service online by the
svc:/system/early-manifest-import:default service. Administrators can manually restart the
svc:/system/manifest-import:default service at any time to manage the import of service manifests and profiles that have been put into known locations (as detailed in Table 1) without requiring a system reboot.
/lib/svc/manifest directory is subdivided into a number of different directories that represent some of the categorization that you can see when you list the services on a system. For example, the
svc:/system/zones:default service instance is a result of delivering the
/lib/svc/manifest/system/zones.xml manifest, while the
svc:/network/ssh:default service instance is a result of delivering the
/lib/svc/manifest/network/ssh.xml manifest. These manifests are stored in directory locations that match their service category for convenience only. Their category is specified within an attribute of the
<service> element within the manifest itself, which we will see below.
While direct manipulation of vendor manifests or other third-party manifests is possible, it is heavily discouraged, because customizations can be lost during a system update. Administrators wishing to create their own SMF manifests should deliver them to
Let's take a quick look at an example manifest to get an idea of what the key elements are. We'll examine the manifest for the SSH server, which is located in
<service name='network/ssh' type='service' version='1'> <create_default_instance enabled='false' /> <single_instance /> .... <template> <common_name> <loctext xml:lang='C'> SSH server </loctext> </common_name> <documentation> <manpage title='sshd' section='1M' manpath='/usr/share/man' /> </documentation> </template>
In this case, we're specifying that the service be called
network/ssh, and it has a single default instance that should not be enabled. We also provide a localizable description of the service and details about what documentation exists are—in this case, the
sshd (1M) manpage.
<dependency name='fs-local' grouping='require_all' restart_on='none' type='service'> <service_fmri value='svc:/system/filesystem/local' /> </dependency> <dependency name='fs-autofs' grouping='optional_all' restart_on='none' type='service'> <service_fmri value='svc:/system/filesystem/autofs' /> </dependency> ....
In the lines above, we specify service dependencies on
network/ssh. In this case, we have a required dependency on
system/filesystem/local and an optional dependency on
system/filesystem/autofs. SMF will use this information to start services in the correct order during system boot, optimizing the startup by starting services in parallel when possible. In total, we have 10 service dependencies for the SSH server. A list of all possible dependency types and restart options are listed in the
smf(5) manual page.
svc.startdor a delegated restarter uses:
<exec_method type='method' name='start' exec='/lib/svc/method/sshd start' timeout_seconds='60'/> <exec_method type='method' name='stop' exec=':kill' timeout_seconds='60' /> <exec_method type='method' name='refresh' exec='/lib/svc/method/sshd restart' timeout_seconds='60' />
For the SSH server, we specify four methods:
unconfigure. In this case, we're calling into another script,
/lib/svc/method/sshd, which does a few checks (such as making sure that host keys are created) prior to executing
<property_group name='general' type='framework'> <!-- to start stop sshd --> <propval name='action_authorization' type='astring' value='solaris.smf.manage.ssh' /> </property_group> <property_group name='firewall_context' type='com.sun,fw_definition'> <propval name='name' type='astring' value='ssh' /> <propval name='ipf_method' type='astring' value='/lib/svc/method/sshd ipfilter' /> </property_group> ....
Within each property group, we have a number of properties defined. For the SSH server, we add a new
action_authorization property to the
general property group, which indicates who can start and stop the service (
solaris.smf.manage.ssh), and we also have a
firewall_context property group that defines two properties that help generate firewall rules for IP packet filtering. In total, we have four property groups defined for the SSH server.
Profiles are, in many ways, similar to manifests in that they use the same XML DTD. However, instead of providing information about a service, its dependencies, and methods, a profile is used to provide customization of a service or an instance of a service. Customizations include whether an instance of a service should be enabled or disabled and any modifications to service configuration properties. While vendor-provided profiles are located in
/etc/svc/profiles, profiles containing local customizations should be stored in
/etc/svc/profiles/site. Profiles are applied during a system reboot or if the
svc:/system/manifest-import:default service instance is restarted.
As an example of a profile, let's look at a profile that can be used to create and start a new instance of the
svc:/application/pkg/server service, which allows us to host an IPS package repository over the Apache Web server:
# cat pkg-custom.xml <?xml version="1.0" ?> <!DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'> <service_bundle type="profile" name="application/pkg/server"> <service version="1" type="service" name="application/pkg/server"> <instance enabled="true" name="default"> <property_group type="application" name="pkg"> <propval type="astring" name="inst_root" value="/export/repository"/> <propval type="count" name="port" value="10000"/> <propval type="boolean" name="readonly" value="false"/> </property_group> </instance> </service> </service_bundle>
Listing 1. Example Profile
In Listing 1, we can see that a service instance called
default has been enabled with values set for the properties
pkg/readonly. All that is left to do is to copy this file into
/etc/svc/profile/site and restart
# cp pkg-custom.xml /etc/svc/profile/site # svcadm restart manifest-import # svcs pkg/server STATE STIME FMRI online 19:13:54 svc:/application/pkg/server:default
Now that we have looked at service manifests and profiles, it's time to take a look at how configuration is managed in SMF. The introduction of layers in Oracle Solaris 11 provides better tracking of vendor-supplied customizations and administrative customizations for services and instances of services in four discrete layers, as shown in Table 1.Table 1. Layers Used by SMF
||Administrative changes made directly through the command line or running applications|
||Administrative changes made through SMF profiles in
||Vendor-provided configuration through SMF profiles in
||Vendor-provided configuration through SMF manifests in
SMF uses these layers with a simple priority ordering scheme to ensure that customizations are not overwritten during a system update. The
admin layer takes the highest priority. If a property is not found in the
admin layer, the
site-profile layer is checked, then the
system-profile layer is checked, and finally the
manifest layer is checked. Layering also provides the ability for an administrator to view configuration at the lower layers that may be overridden by higher layers.
For a quick example, let's take a look at the
svc:/system/identity:node service instance, which is responsible for the
nodename system configuration setting, the local name for a system. Let's quickly check the value of this using the
# svcprop -p config/nodename svc:/system/identity:node solaris
As we can see above, the name of this system is
solaris. Now let's look at where this value has been set in the SMF repository. We can use the
svccfg listcust -L command, as follows, which will list all customizations across the
# svccfg -s svc:/system/identity:node listcust -L config/nodename config/nodename astring site-profile solaris
We can see that the value
solaris has come from the
site-profile layer (it's provided as part of the initial system configuration profile in
/etc/svc/profile/site/sc_profile.xml). Let's use the
svcprop command with the
-l all option to track what's happening at each of the layers:
# svcprop -p config/nodename -l all svc:/system/identity:node config/nodename astring manifest "" config/nodename astring site-profile solaris
As we track through the four different layers, we can see the definition of the property
config/nodename (being of type
astring) is provided at the
manifest layer, but the value
solaris is set at the
site-profile layer. Let's go a little further and set the
config/nodename property using the interactive
svccfg command, as follows:
# svccfg svc:> select identity:node svc:/system/identity:node> listprop config/nodename config/nodename astring solaris svc:/system/identity:node> setprop config/nodename=myserver svc:/system/identity:node> listprop config/nodename config/nodename astring myserver
Let's again look at the different layers of the repository:
# svcprop -p config/nodename -l all svc:/system/identity:node config/nodename astring manifest "" config/nodename astring site-profile solaris config/nodename astring admin myserver
As we can see, configuration is preserved at each layer of the repository. Even though we have changed the
config/nodename property to
myserver, we still have a different value that is stored at the
site-profile layer. If we are a little unsure about the customizations that we have made at any stage, we can use the
svc:/system/identity:node> listcust config/nodename config/nodename astring admin myserver
To delete a customization at the
admin layer, we can use the
svc:/system/identity:node> delcust config/nodename Deleting customizations for property: config/nodename svc:/system/identity:node> listcust config/nodename
An administrator cannot make changes to configuration using the SMF commands to any layer other than the
admin layer. If for example, we tried to delete a property using the
delprop command, the property would become "masked" across the layers. In essence, the property has been deleted from view, but the underlying values have been stored to facilitate an "undelete" capability.
svc:/system/identity:node> listcust config/nodename config/nodename astring admin myserver svc:/system/identity:node> delprop config/nodename svc:/system/identity:node> listprop config/nodename svc:/system/identity:node> listcust config/nodename config/nodename astring admin MASKED myserver svc:/system/identity:node> listcust -l site-profile config/nodename config/nodename astring site-profile MASKED solaris
As we can see, the
listprop command returns nothing, but we can still see the customizations that have been made in the repository at the
admin layer and also provided through the
site-profile layer. This masking ability means that administrators can make direct changes to services, instances of services, property groups, and properties without affecting the underlying vendor-provided service configuration.
We can also do the same with entire services if we wish. Let's take the example of an administrator wanting to remove the
svc:/network/telnet service from being started. One option could be to remove the telnet packages from the system, but another option would be deleting the service using the
svccfg delete command:
# svccfg delete network/telnet # svcs network/telnet svcs: Pattern 'network/telnet' doesn't match any instances STATE STIME FMRI # svccfg listcust -M | grep svc:/network/telnet svc:/network/telnet system-profile MASKED svc:/network/telnet:default system-profile MASKED
Even though the service looks like it has been removed on the system, the underlying configuration still exists at the
system-profile layer. In this case, since the service is masked, any instances of those services are also masked. We could remove these changes using the
svccfg delcust command. If we wanted to remove a service entirely from the system, including its configuration, we can either remove the package or manually delete the manifest and any references in either system or site profiles and restart the
While administering systems on a one-to-one basis will certainly work, a better approach for applying service configuration across many different machines is using site profiles. SMF includes the ability to extract profiles from the current configuration on a system. Once you have tailored one system to your needs, a profile can be extracted and applied to other systems in your data center.
Let's take the generic case of extracting the current state of a system rather than individual services (we can do services on a case-by-case basis by specifying them on the command line, as necessary). As shown in Listing 2, we can use the
svccfg extract command to get a profile that will list all the changes that have been made to the system in the
site-profile layers, including any data that is marked as protected. We need to output the result to a file that has the file extension
.xml so it can be applied on other systems.
# svccfg extract -a > my-custom-profile.xml # cat my-profile.xml <?xml version='1.0'?> <!DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'> <service_bundle type='profile' name='extract'> <service name='application/font/fc-cache' type='service' version='0'> <create_default_instance enabled='true'/> <single_instance/> </service> <service name='application/cups/scheduler' type='service' version='0'> <create_default_instance enabled='true'/> </service> <service name='application/cups/in-lpd' type='service' version='0'> <create_default_instance enabled='false'/> </service> <service name='application/pkg/zones-proxyd' type='service' version='0'> <create_default_instance enabled='false'/> <single_instance/> </service> ....
Listing 2. Extracting the State of a System
Much of the above profile is simply enabling or disabling services, but the profile also includes configuration changes to property groups and properties, as the following snippet shows:
<service name='network/dns/client' type='service' version='0'> <create_default_instance enabled='true'/> <single_instance/> <property_group name='config' type='application'> <propval name='nameserver' type='net_address' value='192.168.0.1'/> </property_group> </service>
Once we have checked this profile to make sure it's everything we expect, we can quickly modify the
<service> XML tag, as shown below:
<service_bundle type='profile' name='my-custom-profile'>
If we want to apply this profile to a system, we simply copy it into
/etc/svc/profile/site on the target system and restart the
svc:/system/manifest-import service. This profile could also be included in an IPS software package during installation time for deployment across many machines in an automated way.
# cp my-custom-profile.xml /etc/svc/profile/site/ # svcadm restart manifest-import
For administrators managing system services and applications on Oracle Solaris 11, SMF provides a number of benefits including automatic service restart, consolidated service configuration, and integration into the fault management framework. Unlike the legacy
init system, administrators manage services, rather than processes, with full service-dependency checking and parallel service startup, leading to a more consistent system state and more manageability. Through the use of the SMF configuration repository and configuration layers, administrators can easily manage a profile of customizations across multiple systems and deploy site-specific services in their data centers.
Glynn Foster is a Principal Product Manager for Oracle Solaris and works on technology areas that include the Image Packaging System and Service Management Facility. Glynn joined Oracle in 2010 as part of the Sun Microsystems acquisition.
|Revision 1.0, 08/06/2012|