Implementing Fibre Channel SAN Boot with Oracle's Sun ZFS Storage Appliance

April 2011

By Tom Hanvey

This paper describes how to implement a FC SAN boot solution using a SUN ZFS Storage Appliance on a high availability SAN. This solution has been validated with a variety of servers, operating systems, and hardware configurations (see Appendix B. Components Validated for Fibre Channel SAN Boot Solution for details).

• Introduction
• Overview
• Benefits of Booting from a SAN
• FC SAN Boot Configuration Requirements
• Configuring the Sun ZFS Storage Appliance for Fibre Channel Boot
• Configuring the Client Server for Fibre Channel SAN Boot
• Installing the Operating System on the Server
• FC SAN Boot Time Test Results
• Best Practices
• Conclusion
• Appendix A. References
• Appendix B. Components Validated for Fibre Channel SAN Boot Solution

Introduction

Organizations are continually looking for ways to simplify their management infrastructure, enhance scalability, and reduce costs, while increasing reliability and performance in their data center. Booting from a storage area network (SAN) offers many benefits, leading to cost savings as well as higher levels of protection, ease of management, increased flexibility, and reduced down time.

Traditionally, operating systems have been installed on internal disks on individual servers or on direct attached storage (DAS). This approach presents a number of challenges to an IT organization. Dedicated internal boot devices cannot be shared with other servers, so are often underutilized. IT staff must perform management tasks on these systems locally rather than from a central management system, leading to increased administrative costs. For optimal redundancy and performance, additional RAID software or host bus adapters (HBAs) are required to manage these storage devices.

Local disks on individual servers present particular challenges for multi-site administration and disaster recovery site maintenance. Creating clones of disk content to off-site hosts or replicating server operating systems and to a disaster recovery backup site can be complex operations requiring specialized software.

The complex task of managing the servers of an entire enterprise can be simplified by enabling data center administrators to centrally manage all storage-related tasks, such as operating system maintenance, at the array level rather than at the individual server level. Locating server boot devices on a Sun ZFS Storage Appliance accessed by servers across a high-availability Fibre Channel (FC) SAN enables increased efficiency, and even automation, of many administrative tasks, significantly reducing operating expenses.

If a server goes down, a system administrator can boot up a standby server in a matter of minutes to resume business. Snapshots and clones of operating system images stored on the Sun ZFS Storage Appliance can be simultaneously deployed to servers into development and test environments or to secondary disaster recovery sites.

Booting from the SAN reduces the amount of time required for server upgrades. With minimal reconfiguration, an outdated or underpowered server can be replaced with a new server, which can then be pointed to the original FC SAN boot device.

When server boot devices are located on a RAID-protected shared storage device like the Sun ZFS Storage Appliance, the need for hardware or software RAID devices in each server is eliminated, helping reduce hardware costs as well.

Overview

A boot-from-SAN solution implemented with a Sun ZFS Storage Appliance located on a high availability FC SAN is shown in Figure 1. In this solution, servers are booted from a pool of centrally-managed storage volumes located in the Sun ZFS Storage Appliance. Each storage volume in the pool serves as a boot LUN for a specific server. The figure shows three types of servers used in the validation testing (x64, x86, and Solaris SPARC) and the operating systems validated on each server type.

When the Sun ZFS Storage Appliance is also used for data storage, best practices dictate that a dedicated pool and separate data paths be used for boot devices. See the section Best Practices below for more details.

Any server and host bus adapter (HBA) combination that supports SAN boot can be used to implement a FC boot solution using a Sun ZFS Storage Appliance. See Appendix B. Components Validated for Fibre Channel SAN Boot Solution for a list of server types, HBAs, operating systems and configurations that have been tested and are supported by Oracle for use in this solution.

Figure 1. Fibre Channel boot solution using a Sun ZFS Storage Appliance.

Benefits of Booting from a SAN

A boot from FC SAN solution provides significant benefits.

Reduces Data Center Footprint and Facility Costs

Booting from FC SAN enables diskless servers and blade servers to be used, which take up less space, consume less power, and require less cooling.

 Lowers Administrative Overhead

All operating system storage is provisioned and managed from the Sun ZFS Storage Appliance. A server can be easily replaced by re-mapping its corresponding boot LUN to a new server. If the new server has the same profile as the server being replaced, it will boot the operating system from the SAN without requiring reconfiguration. Snapshots and clones of operating system images can be created and mapped to new servers on the SAN with just a few clicks of a mouse, simplifying migration and scalability tasks.

Facilitates Disaster and Server Failure Recovery

When operating systems are installed on the Sun ZFS Storage Appliance rather than individual servers, advantage can be taken of the data protection and redundancy features of the appliance to help reduce downtime during maintenance and fault outages. Operating system images can be protected using snapshots and clones or backed up using Network Data Management Protocol (NDMP)

FC SAN Boot Configuration Requirements

Configuring a Fibre Channel boot solution using a Sun ZFS Storage Appliance requires the following:

• Zoning must be configured in the local SAN such that the server FC ports can see the Sun ZFS Storage Appliance FC target ports. For more details, refer to documentation in Appendix A. References.
• In the Sun ZFS Storage Appliance, at least one FC PCIe card must be installed with one port enabled in target mode. For details, see the section Configuring the Sun ZFS Storage Appliance for Fibre Channel Boot below.
• An HBA that supports SAN boots must be installed in each server to be provisioned from the SAN. The solution described in this paper was tested with the following FC HBA driver and firmware versions:
  • QLogic QLE2562 (Firmware Version 4.03.02, BIOS Revision 2.02)
  • Emulex LPe12002 (BIOS Version 2.11a2)
  • The FC HBA on each server must be configured as the primary boot device, a storage target LUN in the Sun ZFS Storage Appliance must be provisioned with the appropriate operating system, and the LUN mapped to the server’s initiator port. For details, see the section Configuring the Client Server for Fibre Channel SAN Boot below.

Configuring the Sun ZFS Storage Appliance for Fibre Channel Boot

To configure the Sun ZFS Storage Appliance for FC SAN boot, complete the steps below:

1. Check that at least one FC PCIe card is installed in the Sun ZFS Storage Appliance.
2. By default, all the FC ports on the Sun ZFS Storage Appliance are set to initiator mode. To enable a port in target mode, complete these steps:
   1. Login to the appliance and navigate to Configuration > SAN > Fibre Channel Ports.
   2. Set the selected port to Target mode as shown in Figure 2.
   3.  Click the Apply button.
      NOTE: Changing this setting will require a reboot

Figure 2. Setting a PCIe port to target mode in the Sun ZFS Storage Appliance.

3. Provision each LUN that will serve as a server boot LUN with the appropriate initiator and target groups.

Configuring the Client Server for Fibre Channel SAN Boot

To configure each client server for a FC SAN boot, first confirm that a Fibre Channel HBA is installed on the client and that the HBA supports SAN boot. For a list of FC HBAs that support SAN boot, see Appendix B. Components Validated for Fibre Channel SAN Boot Solution. Then set the boot precedence in the system BIOS to make the FC HBA card the highest priority boot device and configure the HBA to boot from the LUN on which the operating system for that server has been installed in the Sun ZFS Storage Appliance. These procedures are described in the following sections.

Setting Boot Precedence in the System BIOS

Set the boot precedence in the system BIOS so that the FC HBA card is the highest priority boot device by completing these steps:

1. Reboot the server. While the server is initializing, press F2 to display the system BIOS menu.
2. If the server has an LSI PCI card installed, disable the PCI slot in the system BIOS. In some servers, such as Sun x86 servers, the LSI card takes higher boot precedence and will try to boot the local operating system. To prevent this from happening, complete the steps below:
   1. Select Advanced, to display the PCI Configuration screen.
   2. Disable the PCI slot in which the LSI card is installed as shown in Figure 3.
   3. Press F10 to save the setting and exit the screen and reboot the server.

Figure 3. System BIOS PCI Configuration screen showing PCI Slot1 disabled.

3. Set the FC HBA card to be the highest priority boot device.
   1. From the BIOS menu, select Boot, to display the Boot Device Priority screen.
   2. Select the FC HBA as the 1st Boot Device as shown in Figure 4.
   3. Press F10 to save settings, exit, and reboot the server.

Figure 4. System BIOS PCI Configuration screen showing the FC HBA set as the primary boot device.

Configuring the Host Bus Adaptor for Fibre Channel Boot

One or more ports on the FC HBA on the server must be configured to boot from the LUN on which the operating system for that server has been installed in the Sun ZFS Storage Appliance. The procedure below shows the steps for a QLogic FC HBA. The procedure is similar for an Emulex FC HBA.

1. Reboot the system. When the initialization screen shown in Figure 5 appears, login to the HBA BIOS menu.

Figure 5. QLogic FC HBA Initialization screen providing access to HBA BIOS settings.

2. Select one of the two HBA ports as shown in Figure 6.

Figure 6. Selecting an HBA port to configure.

3. In the menu displayed, select Configuration Settings as shown in Figure 7.

Figure 7. Selecting the HBA BIOS utility Configuration Settings option.

4. On the Configuration Settings menu, select Adapter Settings as shown in Figure 8.

Figure 8. Accessing the Adapter Settings for the HBA.

5. On the Adapter Settings screen, select Host Adapter BIOS and press ENTER to enable the host adaptor BIOS as shown in Figure 9 (the host adapter BIOS is disabled by default).

Figure 9. Enabling the host adaptor BIOS.

6. To change the boot device priority level, press <Esc> to return to the Configuration Settings menu and select Selectable Boot Settings as shown in Figure 10. A list of available FC target ports is displayed as shown in Figure 11.

Figure 10. Accessing the HBA boot settings.

7. Select the FC target port to be used by the HBA on the Sun ZFS Storage Appliance as shown in Figure 11 and press <Enter>. The Selectable Boot Setting screen for the HBA port is displayed as shown in Figure 12.

    

    

Figure 11. Selecting the FC target port to be used by the HBA on the Sun ZFS Storage Appliance.

8. Select (Primary) Boot Port Name as shown in Figure 12 and press <Enter> to display a list of all the available LUNs as shown in Figure 13.

Figure 12. Selecting the primary boot port.

9. Select the number of the LUN from which the server operating system is to be booted as shown in Figure 13.

Figure 13. Selecting the boot LUN for the server.

When configuring the Emulex BIOS, an option is provided to boot the server via World Wide Port Name (WWPN) or Device ID as shown in Figure 14.

Figure 14. Selecting the boot device identification method when configuring an Emulex HBA.

10. For a second HBA port, repeat Steps 2 through 10. Use the same settings as for the first HBA port.
11. Press <Exc> and save the configuration settings as shown in Figure 15.

Figure 15. Saving the HBA configuration settings.

12. Reboot the server. 

When the server is booted, it will now choose the FC HBA as the primary boot device. It will use the primary boot setting in the HBA BIOS to select the appropriate LUN in the Sun ZFS Storage Appliance from which to boot the operating system.

Installing the Operating System on the Server

To install the operating system on the server, follow the instructions for the specific operating system below.

Installing Microsoft Windows 2003

To install Microsoft Windows 2003 on a server, complete the following steps:

1. Create an image on a floppy disk of the driver for the QLogic or Emulex FC HBA installed on the server.
2. Reboot the server to start the installation process. During installation initialization, press F6 to provide the path to the HBA driver image on the floppy disk.
3. Continue the installation using the HBA driver image. The driver enables the HBA to see the FC LUN that is serving as the primary boot LUN for the operating system installation.
4. Install the operating system on the new FC bootable LUN.

Installing Microsoft Windows 2008

To install Microsoft Windows 2008 on a server, complete the following steps:

1. Set the boot LUN as the primary boot device on the FC HBA on the server.
2. Proceed with the installation.

Installing Solaris

To install Solaris on a server, during the installation process, select the appropriate FC LUN from which to install the operating system.

Installing SUSE Linux Enterprise 11 (SP 1)

To install SUSE Linux Enterprise 11 (SP 1) on a server, during the installation process, select Advanced Configuration to install the GRUB boot loader and the operating system on the same FC LUN device. Otherwise, the GRUB master boot record (MBR) will be installed on the local disk and the operating system will not boot from the primary boot FC LUN.

Installing Oracle Enterprise Linux

To install Oracle Enterprise Linux (5 u4) on a server, during the installation process, select Advanced Configuration to install the GRUB boot loader and the operating system on the same FC LUN device. Otherwise, the GRUB master boot record (MBR) will be installed on the local disk and the operating system will not boot from the primary boot FC LUN.

FC SAN Boot Time Test Results

Tests show that booting from a Fibre Channel SAN takes about the same amount of time as booting from a DAS system. Table 1 shows examples of FC SAN server boot times for operating systems installed on a variety of host servers configured with several different FC HBAs. The boot time was measured from when the operating system began to load (start of disk load).

Best Practices

Configure storage on the Sun ZFS Storage Appliance for the highest level of fault tolerance for FC SAN boot solutions.

Storage pool(s) should be configured for optimum fault tolerance and performance by using mirroring. 

Storage LUNs should be mapped to multiple HBA ports accessing the SAN. Identify multiple ports on the client server in the systems BIOS. The BIOS will go through the list of targets upon boot until it can find the active path to the boot device. NOTE: Asynchronous Logical Unit Access (ALUA) is not supported for boot solutions because most HBAs do not support ALUA in the HBA firmware.

When configuring a FC boot device in a Sun ZFS Storage Appliance that is also hosting LUNS for other applications, care should be taken to separate the boot paths from the application data paths. Sharing FC target ports and storage pools designated for booting servers with FC target ports and storage pools servicing applications can adversely affect performance and is not recommended.

Configure dedicated boot storage pools (mirrored) and separate application pools (variable).

Map targets across alternate target ports on the Sun ZFS Storage Appliance so they are not shared with boot ports.

Conclusion

A FC SAN boot solution using a SUN ZFS Storage Appliance on a high availability SAN provides a high level of protection while lowering administrative overhead. Your operating system assets can be protected using a variety of methods including snapshots, clones, replication, or Network Data Management Protocol (NDMP) backup to tape. The ability to manage and maintain operating systems on servers throughout the organization from a central appliance lowers administrative costs.

Appendix A. References

For trouble shooting information related to setting up the FC driver and ALUA, see:

http://stohome.us.oracle.com/wiki/index.php/Troubleshooting_on_FC_Boot

Note that ALUA is not supported for boot devices, but can be configured for other non-bootable data paths once the system is booted.

Other useful links are listed below:

• Sun ZFS Storage 7000 Administration Guide, Chapter 3:
  http://wikis.sun.com/display/fishworks/documentation
• SUSE 11 multipath configuration:
  http://www.novell.com/documentation/sles11/stor_admin/?page=/documentation/sles11/stor_admin/data/be5rvii.html
• Windows 2008 multipath configuration:
  http://blogs.msdn.com/b/san/archive/2008/07/27/multipathing-support-in-windows-server-2008.aspx
• Oracle Enterprise Linux multipath configuration:
  http://sources.redhat.com/lvm2/wiki/MultipathUsageGuide

Appendix B. Components Validated for Fibre Channel SAN Boot Solution

The tables in this appendix list hardware components and operating systems that have been validated by Oracle for use in this Fibre Channel SAN boot solution.

Table 2 lists SUN ZFS Storage Appliances on which booting from a Fibre Channel SAN has been validated.

*This is an older product that is no longer sold by Oracle.

Table 3 lists Oracle Sun servers and HBA combinations that have been validated in the SUN ZFS Storage Appliance FC SAN boot solution.

Table 4 shows details about the HBAs used in validation testing.