Implementing Fibre Channel SAN Boot with Oracle's Sun ZFS Storage Appliance

by Tom Hanvey; updated by Peter Brouwer

This article describes how to implement a Fibre Channel (FC) SAN boot solution using Oracle's Sun ZFS Storage Appliance on a high availability SAN. This solution has been validated with a variety of servers, operating systems, and hardware configurations.

Published November 2012

Introduction

Organizations are continually looking for ways to simplify their management infrastructure, enhance scalability, and reduce costs, while increasing reliability and performance in their data center. Booting from a storage area network (SAN) offers many benefits, leading to cost savings as well as higher levels of protection, ease of management, increased flexibility, and reduced down time.

Traditionally, operating systems have been installed on internal disks on individual servers or on direct attached storage (DAS). This approach presents a number of challenges to an IT organization. Dedicated internal boot devices cannot be shared with other servers, so are often underutilized. IT staff must perform management tasks on these systems locally rather than from a central management system, leading to increased administrative costs. For optimal redundancy and performance, additional RAID software or host bus adapters (HBAs) are required to manage these storage devices.

Local disks on individual servers present particular challenges for multi-site administration and disaster recovery site maintenance. Creating clones of disk content to off-site hosts or replicating server operating systems and to a disaster recovery backup site can be complex operations requiring specialized software.

The complex task of managing the servers of an entire enterprise can be simplified by enabling data center administrators to centrally manage all storage-related tasks, such as operating system maintenance, at the array level rather than at the individual server level. Locating server boot devices on a Sun ZFS Storage Appliance accessed by servers across a high-availability Fibre Channel (FC) SAN enables increased efficiency, and even automation, of many administrative tasks, significantly reducing operating expenses.

If a server goes down, a system administrator can boot up a standby server in a matter of minutes to resume business. Snapshots and clones of operating system images stored on the Sun ZFS Storage Appliance can be simultaneously deployed to servers into development and test environments or to secondary disaster recovery sites.

Booting from the SAN reduces the amount of time required for server upgrades. With minimal reconfiguration, you can replace an outdated or underpowered server with a new server, which you can then point to the original FC SAN boot device.

By locating server boot devices on a RAID-protected shared storage device such as the Sun ZFS Storage Appliance, you can eliminate the need for hardware or software RAID devices in each server, which helps reduce hardware costs.

Overview

A boot-from-SAN solution implemented with a Sun ZFS Storage Appliance located on a high availability FC SAN is shown in Figure 1. In this solution, servers are booted from a pool of centrally managed storage volumes located in the Sun ZFS Storage Appliance. Each storage volume in the pool serves as a boot LUN for a specific server. The figure shows two types of servers used in the validation testing (x86 and Oracle Solaris SPARC) and the operating systems validated on each server type.

When the Sun ZFS Storage Appliance is also used for data storage, best practices dictate that a dedicated pool and separate data paths be used for boot devices. See the section "Best Practices" for more details.

You can use any server and host bus adapter (HBA) combination that supports SAN boot to implement an FC boot solution using a Sun ZFS Storage Appliance.

Figure 1. Fibre Channel boot solution using a Sun ZFS Storage Appliance.

Benefits of Booting from a SAN

A boot from FC SAN solution provides significant benefits.

• Reduces data center footprint and facility costs. Booting from FC SAN enables you to use diskless servers and blade servers, which take up less space, consume less power, and require less cooling.
• Lowers administrative overhead. All operating system storage is provisioned and managed from the Sun ZFS Storage Appliance. A server can be easily replaced by re-mapping its corresponding boot LUN to a new server. If the new server has the same profile as the server being replaced, it will boot the operating system from the SAN without requiring reconfiguration. Snapshots and clones of operating system images can be created and mapped to new servers on the SAN with just a few clicks of a mouse, simplifying migration and scalability tasks.
• Facilitates disaster and server failure recovery. By installing operating systems on the Sun ZFS Storage Appliance rather on than individual servers, you can take advantage of the data protection and redundancy features of the appliance to help reduce downtime during maintenance and fault outages. Operating system images can be protected using snapshots and clones or backed up using Network Data Management Protocol (NDMP)

FC SAN Boot Configuration Requirements

Configuring a Fibre Channel boot solution using a Sun ZFS Storage Appliance requires the following:

• Zoning must be configured in the local SAN such that the server FC ports can see the Sun ZFS Storage Appliance FC target ports. For more details, refer to documentation listed in the "See Also" section.
• In the Sun ZFS Storage Appliance, at least one FC PCIe card must be installed with one port enabled in target mode. For details, see the section "Configuring the Sun ZFS Storage Appliance for Fibre Channel Boot."
• An HBA that supports SAN boots must be installed in each server to be provisioned from the SAN. The solution described in this article was tested with the following FC HBA driver and firmware versions:
  
  
  • QLogic QLE2562 (Firmware Version 4.03.02, BIOS Revision 2.02)
  • Emulex LPe12002 (BIOS Version 2.11a2)
• The FC HBA on each server must be configured as the primary boot device, a storage target LUN in the Sun ZFS Storage Appliance must be provisioned with the appropriate operating system, and the LUN mapped to the server’s initiator port. For details, see the section "Configuring the Client Server for Fibre Channel SAN Boot."

Configuring the Sun ZFS Storage Appliance for Fibre Channel Boot

To configure the Sun ZFS Storage Appliance for FC SAN boot, complete the following steps:

1. Check that at least one FC PCIe card is installed in the Sun ZFS Storage Appliance.
2. By default, all the FC ports on the Sun ZFS Storage Appliance are set to initiator mode. To enable a port in target mode, complete these steps:
   
   
   1. Log in to the appliance and navigate to Configuration > SAN > Fibre Channel Ports.
   2. Set the selected port to Target mode, as shown in Figure 2.
   3. Click the Apply button.
      
      

      NOTE: Changing this setting will require a reboot.

      

      Figure 2. Setting a PCIe port to target mode in the Sun ZFS Storage Appliance.

3. Provision each LUN that will serve as a server boot LUN with the appropriate initiator and target groups.

Configuring the Client Server for Fibre Channel SAN Boot

To configure each client server for an FC SAN boot, first confirm that a Fibre Channel HBA is installed on the client and that the HBA supports SAN boot. Then set the boot precedence in the system BIOS to make the FC HBA card the highest priority boot device and configure the HBA to boot from the LUN on which the operating system for that server has been installed in the Sun ZFS Storage Appliance. These procedures are described in the following sections.

Setting Boot Precedence in the System BIOS

Set the boot precedence in the system BIOS so that the FC HBA card is the highest priority boot device by completing these steps:

1. Reboot the server. While the server is initializing, press F2 to display the system BIOS menu.
2. If the server has an LSI PCI card installed, disable the PCI slot in the system BIOS. In some servers, such as Sun x86 servers, the LSI card takes higher boot precedence and will try to boot the local operating system. To prevent this from happening, complete the following steps:
   
   
   1. Select Advanced to display the PCI Configuration screen.
   2. Disable the PCI slot in which the LSI card is installed, as shown in Figure 3.
   3. Press F10 to save the setting, exit the screen, and reboot the server.

   

   Figure 3. System BIOS PCI Configuration screen showing PCI Slot1 disabled.

3. Set the FC HBA card to be the highest priority boot device.
   
   
   1. From the BIOS menu, select Boot to display the Boot Device Priority screen.
   2. Select the FC HBA as the 1st Boot Device, as shown in Figure 4.
   3. Press F10 to save the settings, exit the screen, and reboot the server.

   

   Figure 4. System BIOS PCI Configuration screen showing the FC HBA set as the primary boot device.

Configuring the Host Bus Adapter for Fibre Channel Boot

One or more ports on the FC HBA on the server must be configured to boot from the LUN on which the operating system for that server has been installed in the Sun ZFS Storage Appliance. The following procedure shows the steps for a QLogic FC HBA. The procedure is similar for an Emulex FC HBA.

1. Reboot the system. When the initialization screen shown in Figure 5 appears, log in to the HBA BIOS menu.
   
   

   

   Figure 5. QLogic FC HBA Initialization screen providing access to HBA BIOS settings.

2. Select one of the two HBA ports, as shown in Figure 6.
   
   

   

   Figure 6. Selecting an HBA port to configure.

3. In the menu displayed, select Configuration Settings, as shown in Figure 7.
   
   

   

   Figure 7. Selecting the HBA BIOS utility Configuration Settings option.

4. On the Configuration Settings menu, select Adapter Settings, as shown in Figure 8.
   
   

   

   Figure 8. Accessing the Adapter Settings for the HBA.

5. On the Adapter Settings screen, select Host Adapter BIOS and press Enter to enable the host adaptor BIOS, as shown in Figure 9 (the host adapter BIOS is disabled by default).
   
   

   

   Figure 9. Enabling the host adaptor BIOS.

6. To change the boot device priority level, press Esc to return to the Configuration Settings menu and select Selectable Boot Settings, as shown in Figure 10.
   
   

   

   Figure 10. Accessing the HBA boot settings.

   A list of available FC target ports is displayed, as shown in Figure 11.

7. Select the FC target port to be used by the HBA on the Sun ZFS Storage Appliance, as shown in Figure 11, and press Enter.
   
   

   

   Figure 11. Selecting the FC target port to be used by the HBA on the Sun ZFS Storage Appliance.

   The Selectable Boot Setting screen for the HBA port is displayed, as shown in Figure 12.

8. Select (Primary) Boot Port Name, as shown in Figure 12, and press Enter.
   
   

   

   Figure 12. Selecting the primary boot port.

   A list of all the available LUNs is displayed, as shown in Figure 13.

9. Select the number of the LUN from which the server operating system is to be booted, as shown in Figure 13.
   
   

   

   Figure 13. Selecting the boot LUN for the server.

   When configuring the Emulex BIOS, you have the option to boot the server using the World Wide Port Name (WWPN) or Device ID, as shown in Figure 14.

   

   Figure 14. Selecting the boot device identification method when configuring an Emulex HBA.

10. For a second HBA port, repeat Step 2 through Step 10. Use the same settings as for the first HBA port.
11. Press Esc and save the configuration settings, as shown in Figure 15.
    
    

    

    Figure 15. Saving the HBA configuration settings.

12. Reboot the server.
    
    

    When the server boots, it will now choose the FC HBA as the primary boot device. It will use the primary boot setting in the HBA BIOS to select the appropriate LUN in the Sun ZFS Storage Appliance from which to boot the operating system.

Installing the Operating System on the Server

To install the operating system on the server, follow the instructions for the specific operating system.

Installing Solaris

To install Solaris on a server, during the installation process, select the appropriate FC LUN from which to install the operating system.

Installing Oracle Linux

To install Oracle Linux (5 u4) on a server, during the installation process, select Advanced Configuration to install the GRUB boot loader and the operating system on the same FC LUN device. Otherwise, the GRUB master boot record (MBR) will be installed on the local disk and the operating system will not boot from the primary boot FC LUN.

Installing SUSE Linux Enterprise 11 (SP 1)

To install SUSE Linux Enterprise 11 (SP 1) on a server, during the installation process, select Advanced Configuration to install the GRUB boot loader and the operating system on the same FC LUN device. Otherwise, the GRUB master boot record (MBR) will be installed on the local disk and the operating system will not boot from the primary boot FC LUN.

Installing Microsoft Windows 2003

To install Microsoft Windows 2003 on a server, complete the following steps:

1. Create an image on a floppy disk of the driver for the QLogic or Emulex FC HBA installed on the server.
2. Reboot the server to start the installation process. During installation initialization, press F6 to provide the path to the HBA driver image on the floppy disk.
3. Continue the installation using the HBA driver image. The driver enables the HBA to see the FC LUN that is serving as the primary boot LUN for the operating system installation.
4. Install the operating system on the new FC bootable LUN.

Installing Microsoft Windows 2008

To install Microsoft Windows 2008 on a server, complete the following steps:

1. Set the boot LUN as the primary boot device on the FC HBA on the server.
2. Proceed with the installation.

FC SAN Boot Time Test Results

Tests show that booting from a Fibre Channel SAN takes about the same amount of time as booting from a DAS system. Table 1 shows examples of FC SAN server boot times for operating systems installed on a variety of host servers configured with several different FC HBAs. The boot time was measured from when the operating system began to load (start of disk load).

*Note: The SPARC T5440 Server is no longer being sold and has been replaced with more powerful servers. Please see http://www.oracle.com/us/products/servers-storage/servers/sparc-enterprise/t-series/overview/index.html for details.

Best Practices

Consider these practices for optimal configuration of the Sun ZFS Storage Appliance:

• Configure storage on the Sun ZFS Storage Appliance for the highest level of fault tolerance for FC SAN boot solutions.
• Storage pool(s) should be configured for optimum fault tolerance and performance by using mirroring.
• Storage LUNs should be mapped to multiple HBA ports accessing the SAN. Identify multiple ports on the client server in the systems BIOS. The BIOS will go through the list of targets upon boot until it can find the active path to the boot device. NOTE: Asynchronous Logical Unit Access (ALUA) is not supported for boot solutions because most HBAs do not support ALUA in the HBA firmware.
• When configuring an FC boot device in a Sun ZFS Storage Appliance that is also hosting LUNS for other applications, be sure to separate the boot paths from the application data paths. Sharing FC target ports and storage pools designated for booting servers with FC target ports and storage pools servicing applications can adversely affect performance and is not recommended.
• Configure dedicated boot storage pools (mirrored) and separate application pools (variable).
• Map targets across alternate target ports on the Sun ZFS Storage Appliance so they are not shared with boot ports.

Conclusion

A Fibre Channel SAN boot solution using Oracle's Sun ZFS Storage Appliance on a high availability SAN provides a high level of protection while lowering administrative overhead. Your operating system assets can be protected using a variety of methods including snapshots, clones, replication, or Network Data Management Protocol (NDMP) backup to tape. The ability to manage and maintain operating systems on servers throughout the organization from a central appliance lowers administrative costs.

See Also

For troubleshooting information related to setting up the FC driver and ALUA, see the Solaris Express SAN Configuration and Multipath Guide: http://docs.oracle.com/cd/E19082-01/820-3070/index.html.

Note that Sun ZFS Storage Appliance ALUA is not supported for boot devices from Sun ZFS Storage Appliance cluster configurations.

Other useful links:

• Sun ZFS Storage 7000 Administration Guide, Chapter 3: http://docs.oracle.com/cd/E26765_01/html/E26397/index.html
• SUSE 11 multipath configuration information: http://www.suse.com/documentation/sles11/stor_admin/?page=/documentation/sles11/stor_admin/data/be5rvii.html
• Microsoft Windows Server 2008 multipath configuration information: http://blogs.msdn.com/b/san/archive/2008/07/27/multipathing-support-in-windows-server-2008.aspx
• Red Hat Enterprise Linux multipath configuration information: http://sources.redhat.com/lvm2/wiki/MultipathUsageGuide