Nicholas O'Donnell, May 2009
This article covers the following CPU power management (PM) topics for SPARC chip multithreading (CMT) systems running the Solaris 10 10/08 Operating System:
What is CPU PM?
CPU PM turns power on and off to individual cores of chip multithreading (CMT) CPU chips to save power. If all the CPU threads of a chip core are not being used, the chip core's power is automatically turned off if PM is turned on. When a CPU thread of a chip core needs to be used again, power to the core is automatically restored.
What determines whether power is off or on to a core?
Upper and lower thresholds of average domain utilization determine when power is turned on or off to cores based on CPU threads allocated to domains.
When a domain's average utilization drops below the lower threshold, some CPU threads are powered off to get average domain utilization back between the lower and upper thresholds (inband). When enough CPU threads have been powered off that all CPU threads for a particular core are powered off, that core is disabled, and this is the point at which power is saved.
When average domain utilization is above the upper threshold, CPU threads are powered on until average domain utilization is back inband. If not enough CPU threads are available on existing powered up cores, another core is powered on to bring more CPU threads online and bring average domain utilization inband.
How does PM work with the software?
CPU PM works within the virtualization layer. It is implemented inside Logical Domains (LDoms), Sun's SPARC CMT virtualization software. By implementing PM within the virtualization layer, any software installed on LDoms can benefit from CMT PM.
How does PM work with the hardware?
The hypervisor (the virtualization layer which manages the hardware) controls power management functions to individual CPU threads and cores.
Most CMT chips have 8 CPU threads per core, and most CMT chips have 4,6, or 8 cores. This gives the hypervisor between 32 and 64 strands that can be power managed per CMT CPU chip. (However, one core per chip must always remain powered on.)
When a CPU thread is not being used, the hypervisor "parks" it. When the hypervisor has parked all the CPU threads on a core, the core is then powered down.
Power savings occur when CPU PM can power off a core because all the core's CPU threads are idle. This scenario occurs only when an LDom has more than one core, because each domain needs a minimum number of active CPU threads to always be available. If a system has many domain configurations that do not consist of more than one core, the power savings are small.
LDoms that have crypto units bound to them cannot be power managed.
CPU PM uses CPU Dynamic Reconfiguration (DR) to power manage domains. Therefore, any domain that uses processor sets or pools is not compatible with CPU PM. This limitation impacts functionality such as Zones in a guest if CPU pools are used. (Together with the Solaris Resource management framework, Solaris Zones form the basis of Solaris Containers.)
CPU PM is available in systems that support the UltraSPARC T2 and UltraSPARC T2 Plus chips. The UltraSPARC T1 chip is not supported. The Sun Servers web page provides processor information for various servers.
CPU PM is compatible with Fujitsu systems that use the UltraSPARC T2 and UltraSPARC T2 Plus chips and the software combinations described next.
CPU PM requires the following software stack:
Refer to the Logical Domains (LDoms) 1.1 Administration Guide, the Logical Domains (LDoms) 1.1 Release Notes, and System Firmware documentation for installing or upgrading LDoms and the System Firmware.
CPU PM requires the following Solaris OS software:
If you upgrade to Logical Domains Manager 1.1 (plus patch 140809-02), you must also upgrade the system firmware to at least System Firmware version 7.2.2, or you won't be able to use CPU PM.
You can control PM with the Service Processor (SP) Integrated Lights Out Manager (ILOM) command line interface (CLI) or browser user interface (BUI). You cannot control it with the Advanced Lights Out Manager (ALOM) CLI. Instructions follow.
1. Log on to the SP.
2. Use the
-> ls /SP/powermgmt /SP/powermgmt Targets: Properties: actual_power = 337 policy = elastic Commands: cd set show
The previous example shows that the
policy property is set to elastic mode.
policy property is set to:
elasticto enable CPU PM
performanceto disable CPU PM
1. Log on to the SP.
2. Use the
To set the
policy property to elastic mode (enable PM):
-> set /SP/powermgmt policy=elastic
To set the
policy property to performance mode (disable PM):
-> set /SP/powermgmt policy=performance
1. Log on to the SP.
2. Use the
ldm list command:
# /opt/SUNWldm/bin/ldm list -l -o cpu NAME primary VCPU VID PID UTIL STRAND 0 0 0.6% 100% 1 1 0.4% 100% 2 2 ---- 100% 3 3 ---- 100% 4 4 ---- 100% 5 5 ---- 100% 6 6 ---- 100% 7 7 ---- 100% 8 8 ---- 100% [...]
In previous example, CPUs 2 through 8 are power managed for the primary domain.
In the following example, CPUs 2 through 8 are power managed and the other CPUs, which are unallocated, are power managed, too.
Note : In elastic mode, CPUs unallocated to a domain are power managed.
# /opt/SUNWldm/bin/ldm list-devices -a cpu VCPU PID %FREE PM 0 0 no 1 0 no 2 0 yes 3 0 yes 4 0 yes 5 0 yes 6 0 yes 7 0 yes 8 0 yes [...] 16 100 --- 17 100 --- 18 100 --- 19 100 --- 20 100 --- 21 100 --- 22 100 --- 23 100 --- [...]
CPU PM uses CPU DR to remove idle CPU threads from LDoms so the CPU threads can be powered off. This process has the side effect of modifying the number of CPU threads that the Solaris OS "sees" when running in a guest domain or the control domain. This side effect can be seen from the output of the
In performance mode, the number of CPU threads seen from
psrinfo is the same number of CPU threads allocated to the LDom. In elastic mode, the
psrinfo command shows between two CPU threads and the number of CPU threads allocated to the LDom.
When a domain is idle, CPU threads are dynamically reconfigured. So an LDom that is idle for a long time shows a minimum of two CPU threads (CR 6729131). More CPU threads are brought back by DR as average domain load increases. The point at which CPU threads are dynamically reconfigured into a domain is reflected in the output of
# psrinfo 0 on-line since 11/21/2008 12:54:42 1 on-line since 11/21/2008 14:45:50 2 on-line since 11/21/2008 14:45:50 3 on-line since 11/21/2008 14:45:50 4 on-line since 11/21/2008 14:45:50 5 on-line since 11/21/2008 14:45:50 6 on-line since 11/21/2008 14:45:50
Power Management policy == elastic/performance mode
Power Management is not supported on this platform.
Unable to obtain the PRI. Exiting power management.
Unable to shut down power management gracefully.
ldmdwas exiting, an attempt to power up a CPU resource failed, which might mean a resource is still in a PM state.
ldmdto clear such an issue, or reboot the control domain as a last option.
Error while setting Power Management policy to elastic/performance mode.
Unable to bring resources to initial power states.
svcadm restart ldmdcommand.
Unable to get the initial PM Policy - timeout.
/etc/system, and then reboot the control domain.
Here are some additional resources:
Unless otherwise licensed, code in all technical manuals herein (including articles, FAQs, samples) is provided under this License.
More Systems Downloads