How to Migrate a Running Oracle Database to Another System Using Oracle VM Server for SPARC 2.1 (continued)

Live Migration Process

The Logical Domains Manager on the source machine accepts the request to migrate a domain and establishes a secure network connection with the Logical Domains Manager that runs on the target machine. After this connection has been established, the migration occurs. The migration itself can be broken down into five different phases:

If you'd like to download software, participate in forums, and get access to other technical how-to goodies in addition to content like this, become an OTN member. No spam!

Phase 1: After connecting with the Logical Domains Manager running on the target machine, information about the source machine and the domain to be migrated are transferred to the target machine. This information is used to perform a series of checks to determine whether a migration is possible.

Figure 3. Live Migration Phase 1
Phase 2: When all checks in Phase 1 have passed, the source and target machines prepare for the migration. An inactive domain is created on the target machine to receive the source domain.

The source machine begins to track which memory blocks of the active guest domain change during the migration operation. If the domain to be migrated is inactive or bound, the migration operation proceeds to Phase 5.
Phase 3: All the runtime state information for the domain is transferred to the target. This information includes the CPU states and the domain memory. The source machine sends all memory blocks to the target machine. When the guest domain memory is transferred to the target machine, the source machine begins to send blocks that are marked as being changed. This transfer operation occurs over several iterations as the guest domain continues to run.
Phase 4: The domain to be migrated is suspended. At this time, the rest of the modified source domain memory blocks and state information is transferred to the target machine.
Phase 5: After all state information is transferred, the handoff occurs when the target domain resumes execution and the source domain is destroyed. From this point on, the target domain is the sole version of the domain running.

Figure 4. Live Migration Phase 5

Live Migration Examples

The following examples use the ldm migrate-domain command to migrate a guest domain from one system to another system.

You can optionally specify an alternate user for authentication on the target machine. See "Assign an Authorization to a User" in the Oracle VM Server for SPARC 2.1 Administration Guide of the documentation.

The ldm migrate-domain -n command enables you to perform a dry-run migration, which performs migration checks but does not migrate the specified domain. Any requirement that is not satisfied is reported as an error, which enables you to correct any configuration errors before attempting an actual migration.

The following examples show various ways in which to perform domain migrations:

Perform a dry-run migration of the ldg1 domain to a target machine called t4-1.
```
# ldm migrate-domain -n ldg1 t4-1
Target Password: password
```
Perform a migration of the ldg1 domain to a target machine called t4-1.
```
# ldm migrate-domain ldg1 t4-1
Target Password: password
```
Perform a non-interactive migration of the ldg1 guest domain to a target machine called t4-1. The -p option specifies the name of a file, pfile, that contains the plain-text superuser password for the target machine, which enables you to access the t4-1 target machine.
```
# ldm migrate-domain -p pfile ldg1 t4-1
```
Note: Ensure that you protect the file that contains the superuser password so that only the root owner, or a privileged user, can read it.
```
# chmod 400 pfile 
```
Perform a migration of the ldg1 domain to a target machine called t4-1 and specify an alternate user, tracy, to be used for authentication on the t4-1 machine.
```
# ldm migrate-domain ldg1 tracy@t4-1
```
For information about assigning an authorization to a user, see "Managing User Profiles" in the Oracle VM Server for SPARC 2.1 Administration Guide.
Monitor the progress of the migration operation:
```
# ldm list -o status ldg1
```
Note: The migration time might vary depending on the amount of memory assigned and the network utilization.

Workload Description and Performance Results

The SwingBench kit contains a load generator and benchmarks that are designed to stress test an Oracle Database. The code that ships with SwingBench includes the OrderEntry, SalesHistory, CallingCircle, and StressTest benchmarks.

For this paper, the SwingBench OrderEntry workload was used to generate workload on the guest domain being migrated. OrderEntry is based on the oe schema that ships with Oracle Database 11g. The workload introduces heavy contention on a small number of tables, and it is designed to stress the system interconnects and memory.

The following describes the workload that is being generated on the guest domain:

Size of the database on disk: 30 GB
System global area (sga_target): 18 GB

The system global area (SGA) is a group of shared memory structures that contain data and control information for one Oracle Database instance.

Number of CPUs assigned to the guest domain: 24 CPUs
Amount of memory assigned to the guest domain: 24 GB
Number of workload users (users) for workload: 50 users
Time to think (time between actions taken by the user): 100 milliseconds (msec)
Duration of the workload: 30 minutes

Table 1 shows the results of live migration operations that ran between two SPARC T4-1 servers.

Table 1. SPARC T4-1 Server Live Migration Results

Number of CPUs on the Control Domain	Overall Migration Time	Suspension Time	Guest Domain CPU Usage
8 CPUs	8 min 12 sec	26 sec	70.00%
16 CPUs	4 min 2 sec	13 sec	80.00%
24 CPUs	2 min 3 sec	7 sec	85.00%

Before You Begin a Domain Migration

The following describes the live migration operation behavior that is typically seen when migrating a domain from one machine to another machine. Note that you might observe slightly different results because the migration time depends on the number of CPUs and the amount of memory assigned to the control domain, as well as the workload type.

Oracle VM Server for SPARC live migration is designed to work over any type of network, including shared networks and insecure networks. The migrations that were run on the main NIC for this paper did not show any performance degradation when compared to those migrations that were run on the dedicated NIC. For best performance, it is best to use unsaturated, high-speed network interfaces, such as 10 GbE.
Use the mpstat command to observe the effect of system memory tracking on the guest domain's CPU. See the USR column to view the usage for each CPU.
The guest domain CPU load rises during live migration, as the guest domain CPU threads perform memory change tracking in the hypervisor. This tracking is reported as CPU time for the application process.
It is best to configure sufficient CPU resources to both the guest domain that runs the database and the control domain that orchestrates the live migration. For a database workload similar to the one used in this white paper, assign 24 CPUs (3 cores) from a SPARC T4-generation system for the guest domain (database) and 8 CPUs (1 core) for the control domain. Further, it is best that you perform your own benchmarks with your unique workload to determine the appropriate CPU resource allocations. See Table 1. For information about how to add more CPUs to the control domain, see the Using CPU Dynamic Reconfiguration section.
Migration time also depends on the following factors: the amount of memory that is assigned to the guest domain, the size of the database SGA, the number of database block changes, and the amount of read-write I/O to the disk.
You can use the Automatic Workload Repository (AWR) tool to monitor the number of database block changes. Look for the db_block_changes information in the Instance Activity Stats section of the generated AWR report. For more information about the AWR and AWR reports, see Section 5.2 ("Overview of the Automatic Workload Repository") in the Oracle Database Performance Tuning Guide.
It is best to configure a guest domain with the Network Time Synchronization Protocol (NTP) time synchronization service so that the time on the guest domain is correctly reset after the migration operation completes. For an example of NTP setup on an Oracle Solaris system, see the Configuring NTP section.
Do not use Live Migration as an emergency tool. However, if you must do so, ensure that your environment is ready by setting the storage paths in advance. Also ensure that you perform live migration operations in an environment that is as close to the production environment as possible. By ensuring this readiness in advance, you will know how much time it takes to complete a migration.

Related Information

This section contains information that is relevant to the Oracle VM for SPARC Live Migration feature, including configuring NTP and using CPU dynamic reconfiguration.

Configuring NTP

Ensure that the system clock on the domain to be migrated is synchronized by using NTP.

In this example, the control domain on the source machine is used as a time source, and it configured as an NTP server. Note that it is best to select an NTP server that can be a dedicated time synchronization source, so that other services are not negatively affected if the machine is brought down for planned maintenance.

The example in Listing 4 shows how to configure an NTP server.

Listing 4. Configuring an NTP Server

# grep -v ^# /etc/inet/ntp.conf
server 127.127.1.0 prefer
broadcast 224.0.1.1 ttl 4
enable auth monitor
driftfile /var/ntp/ntp.drift
statsdir /var/ntp/ntpstats/
filegen peerstats file peerstats type day enable
filegen loopstats file loopstats type day enable
filegen clockstats file clockstats type day enable
keys /etc/inet/ntp.keys
trustedkey 0
requestkey 0
controlkey 0
# touch /var/ntp/ntp.drift
# svcadm enable ntp

The following example shows how to configure an NTP client.

# grep -v ^# /etc/inet/ntp.conf
server source prefer
slewalways yes
disable pll
# svcadm enable ntp

Using CPU Dynamic Reconfiguration

Oracle VM Server for SPARC supports the dynamic reconfiguration of CPUs. CPUs can be dynamically added or removed from any active domain, including the control domain, by using the ldm add-vcpu or ldm rm-vcpu commands.

These commands can be issued from the control domain to increase or decrease the number of allocated CPUs. By increasing the number of CPUs on the control domain, migration time can be decreased. For systems that have discrete cryptographic units, such as the UltraSPARC T2, UltraSPARC T2 Plus, and SPARC T3 platforms, you should add cryptographic units to the control domain as you dynamically add CPU resources.

The following example shows how to add eight CPUs to the control domain:

primary# ldm add-vcpu 8 primary

After the migration operation is completed, you can remove the virtual CPUs from the control domain and allocate them to a different domain. The following example removes eight CPUs from the control domain and allocates them to the ldg1 guest domain:

primary# ldm rm-vcpu 8 primary
primary# ldm add-vcpu 8 ldg1

Note: When reducing the number of CPUs, ensure that a sufficient number of CPUs are still available to efficiently handle the domain's workload. For instance, an Oracle Database single-instance guest domain should always have at least eight CPUs.

Conclusion

An IT organization can reach tactical and strategic goals by choosing the right hardware and software to best manage application availability and to optimize data center resources.

This paper demonstrated the benefits of using the Oracle VM Server for SPARC 2.1 Live Migration feature to manage the production lifecycle of an Oracle Database 11g Release 2 single-instance database. This paper showed the complete configuration process, including the creation and configuration of the domains, storage configuration, and software that were used to demonstrate these benefits. In addition to showing the benefits of the Live Migration feature, this paper described how to achieve better application availability during planned downtime and how to reduce hardware and software costs by means of dynamic hardware resource management in a production environment.

For more information about Oracle's virtualization solutions, visit Oracle's virtualization Web site.

About the Authors

Orgad Kimchi joined Sun/Oracle in September 2007. He currently works in the Independent Software Vendors (ISV) Engineering organization to assist software vendors in adopting Oracle technology that improves the performance of Oracle hardware and software. See Orgad's blog.

Roman Ivanov joined Sun/Oracle in January 2006. He currently works in the ISV Engineering organization to assist software vendors in adopting Oracle technology that improves the performance of Oracle hardware. See Roman's blog.