Advanced Administration with the Service Management Facility (SMF) on Oracle Solaris 11

by Glynn Foster
Published August 2012

Advanced administrative tasks with SMF, including an introduction to service manifests, understanding layering within the SMF configuration repository, and how best to apply configuration to a system.

The Service Management Facility (SMF), first introduced in Oracle Solaris 10, is a feature of the operating system for managing system and application services, and it replaces the legacy init scripting start-up mechanism common to prior releases of Oracle Solaris and other UNIX operating systems. SMF improves the availability of a system by ensuring that essential system and application services run continuously even in the event of any hardware or software failures. SMF is one of the components of the wider Oracle Solaris Predictive Self Healing capability.

Another article, "Introducing the Basics of Service Management Facility (SMF) on Oracle Solaris 11," provided a quick overview of SMF and some basic examples of how to administer systems with SMF, including how to get information about running services, how to start and stop instances of a service, and how to do basic service configuration. It also provided some troubleshooting tips in the event that things go wrong. It is recommended that you read that article first if you are unfamiliar with SMF.

This article covers some more-advanced administrative tasks with SMF, including an introduction to service manifests, understanding layering within the SMF configuration repository, and how best to apply configuration to a system.

OTN is all about helping you become familiar enough with Oracle technologies to make an informed decision. Articles, software downloads, documentation, and more.

Exploring Service Bundles: Manifests and Profiles

At the core of SMF is the concept of a service bundle, an XML-based file that is used to describe everything about a service or an instance of a service—including its configuration—prior to it being imported into the SMF configuration repository. Service bundles are used to deliver services into Oracle Solaris, but they also act as a useful way for administrators to deliver custom configuration across a variety of systems.

Two types of service bundles are used in SMF: manifests and profiles. Both are based on an XML DTD found in /usr/share/lib/xml/dtd/service_bundle.dtd.1 and documented in the service_bundle(4) manual page.

Manifests

Manifests are used to describe services and instances of a service, including any property groups and properties they might have. Manifests are stored in /lib/svc/manifest and automatically imported into the SMF repository. SMF automatically detects if there have been any changes to the manifest and ensures that these changes have been reflected prior to bringing the service online by the svc:/system/early-manifest-import:default service. Administrators can manually restart the svc:/system/manifest-import:default service at any time to manage the import of service manifests and profiles that have been put into known locations (as detailed in Table 1) without requiring a system reboot.

The /lib/svc/manifest directory is subdivided into a number of different directories that represent some of the categorization that you can see when you list the services on a system. For example, the svc:/system/zones:default service instance is a result of delivering the /lib/svc/manifest/system/zones.xml manifest, while the svc:/network/ssh:default service instance is a result of delivering the /lib/svc/manifest/network/ssh.xml manifest. These manifests are stored in directory locations that match their service category for convenience only. Their category is specified within an attribute of the <service> element within the manifest itself, which we will see below.

While direct manipulation of vendor manifests or other third-party manifests is possible, it is heavily discouraged, because customizations can be lost during a system update. Administrators wishing to create their own SMF manifests should deliver them to /lib/svc/manifest/site.

Let's take a quick look at an example manifest to get an idea of what the key elements are. We'll examine the manifest for the SSH server, which is located in /lib/svc/manifest/network/ssh.xml.

Information about the service (name, description, links to documentation):



    <service
        name='network/ssh'
        type='service'
        version='1'>

        <create_default_instance enabled='false' />

        <single_instance />
    ....
        <template>
                <common_name>
                        <loctext xml:lang='C'>
                        SSH server
                        </loctext>
                </common_name>
                <documentation>
                        <manpage title='sshd' section='1M' manpath='/usr/share/man' />
                </documentation>
        </template>

In this case, we're specifying that the service be called network/ssh, and it has a single default instance that should not be enabled. We also provide a localizable description of the service and details about what documentation exists are—in this case, the sshd (1M) manpage.

Dependencies of the service:
```
<dependency name='fs-local'
          grouping='require_all'
          restart_on='none'
          type='service'>
          <service_fmri
                  value='svc:/system/filesystem/local' />
</dependency>

<dependency name='fs-autofs'
          grouping='optional_all'
          restart_on='none'
          type='service'>
          <service_fmri value='svc:/system/filesystem/autofs' />
</dependency>
....
```
In the lines above, we specify service dependencies on network/ssh. In this case, we have a required dependency on system/filesystem/local and an optional dependency on system/filesystem/autofs. SMF will use this information to start services in the correct order during system boot, optimizing the startup by starting services in parallel when possible. In total, we have 10 service dependencies for the SSH server. A list of all possible dependency types and restart options are listed in the smf(5) manual page.

A set of methods to start, stop, and refresh the service that either the default svc.startd or a delegated restarter uses:



<exec_method
          type='method'
          name='start'
          exec='/lib/svc/method/sshd start'
          timeout_seconds='60'/>

<exec_method
          type='method'
          name='stop'
          exec=':kill'
          timeout_seconds='60' />

<exec_method
          type='method'
          name='refresh'
          exec='/lib/svc/method/sshd restart'
          timeout_seconds='60' />

For the SSH server, we specify four methods: start, stop, refresh, and unconfigure. In this case, we're calling into another script, /lib/svc/method/sshd, which does a few checks (such as making sure that host keys are created) prior to executing /usr/lib/ssh/sshd.

A set of property groups and properties for the service:
```
<property_group name='general' type='framework'>
          
          <propval name='action_authorization' type='astring'
                  value='solaris.smf.manage.ssh' />
</property_group>

<property_group name='firewall_context' type='com.sun,fw_definition'>
          <propval name='name' type='astring' value='ssh' />
          <propval name='ipf_method' type='astring'
              value='/lib/svc/method/sshd ipfilter' />
</property_group>
....
```
Within each property group, we have a number of properties defined. For the SSH server, we add a new action_authorization property to the general property group, which indicates who can start and stop the service (solaris.smf.manage.ssh), and we also have a firewall_context property group that defines two properties that help generate firewall rules for IP packet filtering. In total, we have four property groups defined for the SSH server.

Profiles

Profiles are, in many ways, similar to manifests in that they use the same XML DTD. However, instead of providing information about a service, its dependencies, and methods, a profile is used to provide customization of a service or an instance of a service. Customizations include whether an instance of a service should be enabled or disabled and any modifications to service configuration properties. While vendor-provided profiles are located in /etc/svc/profile, profiles containing local customizations should be stored in /etc/svc/profile/site. Profiles are applied during a system reboot or if the svc:/system/manifest-import:default service instance is restarted.

As an example of a profile, let's look at a profile that can be used to create and start a new instance of the svc:/application/pkg/server service, which allows us to host an IPS package repository over the Apache Web server:



# cat pkg-custom.xml
< ?xml version="1.0" ? >
< !DOCTYPE service_bundle
  SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1' >
< service_bundle type="profile" name="application/pkg/server" >
    < service version="1" type="service" name="application/pkg/server" >
        < instance enabled="true" name="default" >
            < property_group type="application" name="pkg" >
                < propval type="astring" name="inst_root"
                    value="/export/repository"/ >
                < propval type="count" name="port" value="10000"/ >
                < propval type="boolean" name="readonly" value="false"/ >
            < /property_group >
        < /instance >
    < /service >
< /service_bundle >

Listing 1. Example Profile

In Listing 1, we can see that a service instance called default has been enabled with values set for the properties pkg/inst_root, pkg/proxy, and pkg/readonly. All that is left to do is to copy this file into /etc/svc/profile/site and restart svc:/system/manifest-import:default.



# cp pkg-custom.xml /etc/svc/profile/site
# svcadm restart manifest-import
# svcs pkg/server
STATE          STIME    FMRI
online         19:13:54 svc:/application/pkg/server:default

Layering in the SMF Configuration Repository

Now that we have looked at service manifests and profiles, it's time to take a look at how configuration is managed in SMF. The introduction of layers in Oracle Solaris 11 provides better tracking of vendor-supplied customizations and administrative customizations for services and instances of services in four discrete layers, as shown in Table 1.

Table 1. Layers Used by SMF

Layer	Description
`admin`	Administrative changes made directly through the command line or running applications
`site-profile`	Administrative changes made through SMF profiles in `/etc/svc/profile/site`
`system-profile`	Vendor-provided configuration through SMF profiles in `/etc/svc/profile/generic.xml` or `/etc/svc/profile/platform.xml`
`manifest`	Vendor-provided configuration through SMF manifests in `/lib/svc/manifests`

SMF uses these layers with a simple priority ordering scheme to ensure that customizations are not overwritten during a system update. The admin layer takes the highest priority. If a property is not found in the admin layer, the site-profile layer is checked, then the system-profile layer is checked, and finally the manifest layer is checked. Layering also provides the ability for an administrator to view configuration at the lower layers that may be overridden by higher layers.

For a quick example, let's take a look at the svc:/system/identity:node service instance, which is responsible for the nodename system configuration setting, the local name for a system. Let's quickly check the value of this using the svcprop command:



# svcprop -p config/nodename svc:/system/identity:node
solaris

As we can see above, the name of this system is solaris. Now let's look at where this value has been set in the SMF repository. We can use the svccfg listcust -L command, as follows, which will list all customizations across the admin and site-profile layers:



# svccfg -s svc:/system/identity:node listcust -L config/nodename
config/nodename astring     site-profile          solaris

We can see that the value solaris has come from the site-profile layer (it's provided as part of the initial system configuration profile in /etc/svc/profile/site/sc_profile.xml). Let's use the svcprop command with the -l all option to track what's happening at each of the layers:



# svcprop -p config/nodename -l all svc:/system/identity:node
config/nodename astring manifest ""
config/nodename astring site-profile solaris

As we track through the four different layers, we can see the definition of the property config/nodename (being of type astring) is provided at the manifest layer, but the value solaris is set at the site-profile layer. Let's go a little further and set the config/nodename property using the interactive svccfg command, as follows:



# svccfg
svc:> select identity:node
svc:/system/identity:node> listprop config/nodename
config/nodename astring     solaris
svc:/system/identity:node> setprop config/nodename=myserver
svc:/system/identity:node> listprop config/nodename
config/nodename astring     myserver

Let's again look at the different layers of the repository:



# svcprop -p config/nodename -l all svc:/system/identity:node
config/nodename astring manifest ""
config/nodename astring site-profile solaris
config/nodename astring admin myserver

As we can see, configuration is preserved at each layer of the repository. Even though we have changed the config/nodename property to myserver, we still have a different value that is stored at the site-profile layer. If we are a little unsure about the customizations that we have made at any stage, we can use the listcust command:



svc:/system/identity:node> listcust config/nodename
config/nodename astring     admin                 myserver

To delete a customization at the admin layer, we can use the delcust command:



svc:/system/identity:node> delcust config/nodename
 Deleting customizations for property: config/nodename
svc:/system/identity:node> listcust config/nodename

An administrator cannot make changes to configuration using the SMF commands to any layer other than the admin layer. If for example, we tried to delete a property using the delprop command, the property would become "masked" across the layers. In essence, the property has been deleted from view, but the underlying values have been stored to facilitate an "undelete" capability.



svc:/system/identity:node> listcust config/nodename
config/nodename astring     admin                 myserver
svc:/system/identity:node> delprop config/nodename
svc:/system/identity:node> listprop config/nodename
svc:/system/identity:node> listcust config/nodename
config/nodename astring     admin          MASKED myserver
svc:/system/identity:node> listcust -l site-profile config/nodename
config/nodename astring     site-profile   MASKED solaris

As we can see, the listprop command returns nothing, but we can still see the customizations that have been made in the repository at the admin layer and also provided through the site-profile layer. This masking ability means that administrators can make direct changes to services, instances of services, property groups, and properties without affecting the underlying vendor-provided service configuration.

We can also do the same with entire services if we wish. Let's take the example of an administrator wanting to remove the svc:/network/telnet service from being started. One option could be to remove the telnet packages from the system, but another option would be deleting the service using the svccfg delete command:



# svccfg delete network/telnet
# svcs network/telnet
svcs: Pattern 'network/telnet' doesn't match any instances
STATE          STIME    FMRI
# svccfg listcust -M | grep svc:/network/telnet
svc:/network/telnet system-profile MASKED
svc:/network/telnet:default system-profile MASKED

Even though the service looks like it has been removed on the system, the underlying configuration still exists at the system-profile layer. In this case, since the service is masked, any instances of those services are also masked. We could remove these changes using the svccfg delcust command. If we wanted to remove a service entirely from the system, including its configuration, we can either remove the package or manually delete the manifest and any references in either system or site profiles and restart the svc:/system/manifest-import service.

Applying Service Configuration

While administering systems on a one-to-one basis will certainly work, a better approach for applying service configuration across many different machines is using site profiles. SMF includes the ability to extract profiles from the current configuration on a system. Once you have tailored one system to your needs, a profile can be extracted and applied to other systems in your data center.

Let's take the generic case of extracting the current state of a system rather than individual services (we can do services on a case-by-case basis by specifying them on the command line, as necessary). As shown in Listing 2, we can use the svccfg extract command to get a profile that will list all the changes that have been made to the system in the admin and site-profile layers, including any data that is marked as protected. We need to output the result to a file that has the file extension .xml so it can be applied on other systems.



# svccfg extract -a > my-custom-profile.xml
# cat my-profile.xml
<?xml version='1.0'?>
<!DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
<service_bundle type='profile' name='extract'>
  <service name='application/font/fc-cache' type='service' version='0'>
    <create_default_instance enabled='true'/>
    <single_instance/>
  </service>
  <service name='application/cups/scheduler' type='service' version='0'>
    <create_default_instance enabled='true'/>
  </service>
  <service name='application/cups/in-lpd' type='service' version='0'>
    <create_default_instance enabled='false'/>
  </service>
  <service name='application/pkg/zones-proxyd' type='service' version='0'>
    <create_default_instance enabled='false'/>
    <single_instance/>
  </service>
  ....

Listing 2. Extracting the State of a System

Much of the above profile is simply enabling or disabling services, but the profile also includes configuration changes to property groups and properties, as the following snippet shows:



    <service name='network/dns/client' type='service' version='0'>
    <create_default_instance enabled='true'/>
    <single_instance/>
    <property_group name='config' type='application'>
      <propval name='nameserver' type='net_address' value='192.168.0.1'/>
    </property_group>
  </service>

Once we have checked this profile to make sure it's everything we expect, we can quickly modify the <service> XML tag, as shown below:

<service_bundle type='profile' name='my-custom-profile'>

If we want to apply this profile to a system, we simply copy it into /etc/svc/profile/site on the target system and restart the svc:/system/manifest-import service. This profile could also be included in an IPS software package during installation time for deployment across many machines in an automated way.



# cp my-custom-profile.xml /etc/svc/profile/site/
# svcadm restart manifest-import

Summary

For administrators managing system services and applications on Oracle Solaris 11, SMF provides a number of benefits including automatic service restart, consolidated service configuration, and integration into the fault management framework. Unlike the legacy init system, administrators manage services, rather than processes, with full service-dependency checking and parallel service startup, leading to a more consistent system state and more manageability. Through the use of the SMF configuration repository and configuration layers, administrators can easily manage a profile of customizations across multiple systems and deploy site-specific services in their data centers.

About the Author

Glynn Foster is a Principal Product Manager for Oracle Solaris and works on technology areas that include the Image Packaging System and Service Management Facility. Glynn joined Oracle in 2010 as part of the Sun Microsystems acquisition.

Revision 1.1, 03/10/2014

See sysadmin-related content for all Oracle technologies by following OTN Systems on Facebook and Twitter.