CPU Power Management Features for SPARC CMT Systems With the Solaris 10 10/08 OS

Nicholas O'Donnell, May 2009

This article covers the following CPU power management (PM) topics for SPARC chip multithreading (CMT) systems running the Solaris 10 10/08 Operating System:

CPU PM Feature Description

  • What is CPU PM?

    CPU PM turns power on and off to individual cores of chip multithreading (CMT) CPU chips to save power. If all the CPU threads of a chip core are not being used, the chip core's power is automatically turned off if PM is turned on. When a CPU thread of a chip core needs to be used again, power to the core is automatically restored.

  • What determines whether power is off or on to a core?

    Upper and lower thresholds of average domain utilization determine when power is turned on or off to cores based on CPU threads allocated to domains.

    When a domain's average utilization drops below the lower threshold, some CPU threads are powered off to get average domain utilization back between the lower and upper thresholds (inband). When enough CPU threads have been powered off that all CPU threads for a particular core are powered off, that core is disabled, and this is the point at which power is saved.

    When average domain utilization is above the upper threshold, CPU threads are powered on until average domain utilization is back inband. If not enough CPU threads are available on existing powered up cores, another core is powered on to bring more CPU threads online and bring average domain utilization inband.

  • How does PM work with the software?

    CPU PM works within the virtualization layer. It is implemented inside Logical Domains (LDoms), Sun's SPARC CMT virtualization software. By implementing PM within the virtualization layer, any software installed on LDoms can benefit from CMT PM.

  • How does PM work with the hardware?

    The hypervisor (the virtualization layer which manages the hardware) controls power management functions to individual CPU threads and cores.

    Most CMT chips have 8 CPU threads per core, and most CMT chips have 4,6, or 8 cores. This gives the hypervisor between 32 and 64 strands that can be power managed per CMT CPU chip. (However, one core per chip must always remain powered on.)

    When a CPU thread is not being used, the hypervisor "parks" it. When the hypervisor has parked all the CPU threads on a core, the core is then powered down.

Limitations

  • Power savings occur when CPU PM can power off a core because all the core's CPU threads are idle. This scenario occurs only when an LDom has more than one core, because each domain needs a minimum number of active CPU threads to always be available. If a system has many domain configurations that do not consist of more than one core, the power savings are small.

  • LDoms that have crypto units bound to them cannot be power managed.

  • CPU PM uses CPU Dynamic Reconfiguration (DR) to power manage domains. Therefore, any domain that uses processor sets or pools is not compatible with CPU PM. This limitation impacts functionality such as Zones in a guest if CPU pools are used. (Together with the Solaris Resource management framework, Solaris Zones form the basis of Solaris Containers.)

Software and Platform Requirements

  • CPU PM is available in systems that support the UltraSPARC T2 and UltraSPARC T2 Plus chips. The UltraSPARC T1 chip is not supported. The Sun Servers web page provides processor information for various servers.

    CPU PM is compatible with Fujitsu systems that use the UltraSPARC T2 and UltraSPARC T2 Plus chips and the software combinations described next.

  • CPU PM requires the following software stack:

    • ILOM System Firmware (SysFW) 7.2.2 (or later). Go to the Firmware Downloads and Release History for Sun Systems section of the BigAdmin Patches Center, locate your system in the table, and then click the SunSolve Patch-ID link associated with your system to download the firmware.
    • Logical Domains Manager 1.1. Download it from the LDoms web page.
    • Patch 140809-02 (or later). On the SunSolve web site, search for patch 140809-02 and download it.
  • Refer to the Logical Domains (LDoms) 1.1 Administration Guide, the Logical Domains (LDoms) 1.1 Release Notes, and System Firmware documentation for installing or upgrading LDoms and the System Firmware.

  • CPU PM requires the following Solaris OS software:

  • If you upgrade to Logical Domains Manager 1.1 (plus patch 140809-02), you must also upgrade the system firmware to at least System Firmware version 7.2.2, or you won't be able to use CPU PM.

User Interface

You can control PM with the Service Processor (SP) Integrated Lights Out Manager (ILOM) command line interface (CLI) or browser user interface (BUI). You cannot control it with the Advanced Lights Out Manager (ALOM) CLI. Instructions follow.

To View PM Status Using the ILOM CLI

1. Log on to the SP.

2. Use the ls command:

-> ls /SP/powermgmt

/SP/powermgmt
   Targets:

   Properties:
   actual_power = 337
   policy = elastic

   Commands:
   cd
   set
   show

The previous example shows that the policy property is set to elastic mode.

The policy property is set to:

  • elastic to enable CPU PM
  • performance to disable CPU PM

To Enable or Disable PM Using the ILOM CLI

1. Log on to the SP.

2. Use the set command:

To set the policy property to elastic mode (enable PM):

-> set /SP/powermgmt policy=elastic

To set the policy property to performance mode (disable PM):

-> set /SP/powermgmt policy=performance

To Show PM Status Using the ldm Command

1. Log on to the SP.

2. Use the ldm list command:

# /opt/SUNWldm/bin/ldm list -l -o cpu
NAME
primary

VCPU
    VID    PID    UTIL STRAND
    0      0      0.6%   100%
    1      1      0.4%   100%
    2      2      ----   100%
    3      3      ----   100%
    4      4      ----   100%
    5      5      ----   100%
    6      6      ----   100%
    7      7      ----   100%
    8      8      ----   100%
[...]

In previous example, CPUs 2 through 8 are power managed for the primary domain.

In the following example, CPUs 2 through 8 are power managed and the other CPUs, which are unallocated, are power managed, too.

Note : In elastic mode, CPUs unallocated to a domain are power managed.

# /opt/SUNWldm/bin/ldm list-devices -a cpu
VCPU
    PID     %FREE   PM
    0       0       no
    1       0       no
    2       0       yes
    3       0       yes
    4       0       yes
    5       0       yes
    6       0       yes
    7       0       yes
    8       0       yes
[...]
    16      100     ---
    17      100     ---
    18      100     ---
    19      100     ---
    20      100     ---
    21      100     ---
    22      100     ---
    23      100     ---
[...]

psrinfo Side Effects

CPU PM uses CPU DR to remove idle CPU threads from LDoms so the CPU threads can be powered off. This process has the side effect of modifying the number of CPU threads that the Solaris OS "sees" when running in a guest domain or the control domain. This side effect can be seen from the output of the psrinfo command.

In performance mode, the number of CPU threads seen from psrinfo is the same number of CPU threads allocated to the LDom. In elastic mode, the psrinfo command shows between two CPU threads and the number of CPU threads allocated to the LDom.

When a domain is idle, CPU threads are dynamically reconfigured. So an LDom that is idle for a long time shows a minimum of two CPU threads (CR 6729131). More CPU threads are brought back by DR as average domain load increases. The point at which CPU threads are dynamically reconfigured into a domain is reflected in the output of psrinfo.

# psrinfo
0       on-line   since 11/21/2008 12:54:42
1       on-line   since 11/21/2008 14:45:50
2       on-line   since 11/21/2008 14:45:50
3       on-line   since 11/21/2008 14:45:50
4       on-line   since 11/21/2008 14:45:50
5       on-line   since 11/21/2008 14:45:50
6       on-line   since 11/21/2008 14:45:50

CPU PM Messages in LDoms Log

  • Power Management policy == elastic/performance mode

    • The SP PM policy has been altered.
    • This is not an error.
  • Power Management is not supported on this platform.

    • The System Firmware does not support CPU PM.
    • Install a version of System Firmware that supports CPU PM, if available. Refer to the LDoms documentation for the appropriate System Firmware. (Also see the Software and Platform Requirements section earlier in this article.)
  • Unable to obtain the PRI. Exiting power management.

    • There was a communications error between the Solaris control domain and the SP.
    • Reboot the Solaris control domain to clear the error. A reboot of the SP might also be required.
  • Unable to shut down power management gracefully.

    • When ldmd was exiting, an attempt to power up a CPU resource failed, which might mean a resource is still in a PM state.
    • Restart ldmd to clear such an issue, or reboot the control domain as a last option.
  • Error while setting Power Management policy to elastic/performance mode.

    • A failure to power up or power down a resource occurred during a transition from one mode to another.
    • Retry the operation on the SP.
  • Unable to bring resources to initial power states.

    • CPU PM could not initialize all the resources when LDoms Manager was starting.
    • Try restarting LDoms Manager using the svcadm restart ldmd command.
  • Unable to get the initial PM Policy - timeout.

    • If CPU PM cannot retrieve the PM policy from the SP, it allows LDoms Manager to start up as expected but logs this error to the LDoms log and remains in performance mode.
    • Add forceload: drv/ds_snmp to /etc/system, and then reboot the control domain.

For More Information

Here are some additional resources:


Comments (latest comments first)

Discuss and comment on this resource in the BigAdmin Wiki
 

Unless otherwise licensed, code in all technical manuals herein (including articles, FAQs, samples) is provided under this License.

Left Curve
Popular Downloads
Right Curve
Untitled Document
Left Curve
More Systems Downloads
Right Curve