How to Consolidate Zones Storage on an Oracle ZFS Storage Appliance

by Detlef Drewanz and Cindy Swearingen

This article describes how to configure Oracle Solaris Zones on iSCSI-based shared storage to reduce the management that is required for iSCSI devices and to consolidate a zones infrastructure on an Oracle ZFS Storage Appliance.


Published June 2014


Want to comment on this article? Post the link on Facebook's OTN Garage page.  Have a similar article to share? Bring it up on Facebook or Twitter and let's discuss.
Table of Contents
About Zones on Shared Storage
Overview of the Components, Process, and Architecture
Create an ISCSI Target and Export the LU
Create and Install the Zone on the First Oracle Solaris System
Detach the Zone and Migrate It to the Second Oracle Solaris System
Add Multipathing to the iSCSI Storage Configuration
Control iSCSI LU and Target Discovery
(Optional) Use ZFS-Encrypted File Systems
See Also
About the Authors

About Zones on Shared Storage

Introduced in Oracle Solaris 11, the Zones on Shared Storage (ZOSS) feature provides the following advantages:

  • Hosts zone installation and data on shared storage over flexible FC, iSCSI, or SAS protocols for easier storage management
  • Provides a simple ZFS configuration:

    • One rpool (<zonename>_rpool) per zone
    • Data pools (<zonename>_<zpool>)
  • Uses standard zone command-line interfaces and storage setup capabilities:

    • Perform basic zone configuration (zonecfg)
    • Identify storage components (suriadm)
    • Create, import, and export a pool (zoneadm)
  • Easily migrates a zone between systems using a standard command-line interface to detach and attach zones (zoneadm)
  • Uses iSCSI initiator groups and the Challenge-Handshake Authentication Protocol (CHAP) to control the discovery of iSCSI logical units and targets
  • Secures storage and data paths with ZFS encryption (optional)

Overview of the Components, Process, and Architecture

In addition to an Oracle Solaris Zone, the configuration described in this article consists of the following components:

  • The iSCSI protocol, a block-based storage protocol over an IP network, is used.
  • The logical unit (LU) is a numbered storage component identified by a logical unit ID.
  • The iSCSI initiator is a client (driver) that initiates iSCSI requests to the iSCSI target. In this article, two Oracle Solaris systems are initiators.
  • The iSCSI target, which is hosted from an Oracle ZFS Storage Appliance, is the storage component that receives the iSCSI requests.

The basic configuration steps are as follows:

  • Create an iSCSI target and export the LU on the Oracle ZFS Storage Appliance.
  • Create and install an Oracle Solaris Zone on the first Oracle Solaris system.
  • Detach the Oracle Solaris Zone and migrate it to the second Oracle Solaris system.
  • Enable iSCSI multipathing.
  • Control iSCSI LU and target discovery.
  • (Optional) Secure the zone's ZFS data by manually creating an encrypted zone rpool.

The next sections of this article demonstrate how to configure an Oracle Solaris Zone (zoss-zone) on an iSCSI device that is hosted from an Oracle ZFS Storage Appliance (zfssa) and then migrate the zone from one Oracle Solaris system (hostA) to a second Oracle Solaris system (hostB). Figure 1 illustrates the architecture:

Figure 1. Diagram of the architecture

Figure 1. Diagram of the architecture

Create an ISCSI Target and Export the LU

Perform the following steps on the Oracle ZFS Storage Appliance.

  1. Create the iSCSI target, as shown in Figure 2:

    1. Select Configuration->SAN->iSCSI.
    2. Click the + (plus) sign next to Targets.
    3. Create an iSCSI target by specifying the zone name (zoss-zone) in the Alias field.
    4. From the Network interfaces list, select the network interface.

      Figure 2. Creating the iSCSI target

      Figure 2. Creating the iSCSI target

  2. Create and export the iSCSI LUN, as shown in Figure 3:

    1. Select Configuration->Shares->LUNs, and then click the + (plus) sign next to LUNs.
    2. Create and export the LUN by completing the Create LUN screen components:

      • Project: default
      • Name: zoss-zone-rpool-primary
      • Volume size: 10 G
      • Volume block size: 8K
      • Target group: All targets
      • Initiator group(s): All initiators
      • LU number: Auto-assign
    3. Click Apply.

      Figure 3. Creating and exporting the iSCSI LU

      Figure 3. Creating and exporting the iSCSI LU

Create and Install the Zone on the First Oracle Solaris System

This section describes how to create and install the zone on the first Oracle Solaris system (hostA).

  1. Create an Oracle Solaris Zone called zoss-zone by running the following commands:

    Note: When you run the add storage command, specify the LU GUID shown in Figure 3 as the storage identifier.

    root@hostA:~# zonecfg -z zoss-zone
    zonecfg:zoss-zone> create
    zonecfg:zoss-zone> set zonepath=/zones/zoss-zone
    zonecfg:zoss-zone> add rootzpool
    zonecfg:zoss-zone:rootzpool> add storage \
    iscsi://192.168.202.10/luname.naa.600144F0949056290000529625DD001
    zonecfg:zoss-zone:rootzpool> end
    zonecfg:zoss-zone> commit
    zonecfg:zoss-zone> exit
    
  2. Install the zone:

    Installing the zone completes the following tasks:

    • Configures the iSCSI initiator
    • Discovers the iSCSI target and LU
    • Creates logical device links
    • Creates the zone rpool
    root@hostA:~# zoneadm -z zoss-zone install
    Configured zone storage resource(s) from:
    iscsi://192.168.202.10/luname.naa.600144F0949056290000529625DD0001
    Created zone zpool: zoss-zone_rpool
    

Detach the Zone and Migrate It to the Second Oracle Solaris System

This section describes how to detach the zone from hostA and migrate it to the second Oracle Solaris system (hostB).

  1. Shut down and detach the zone:

    root@hostA:~# zoneadm -z zoss-zone shutdown
    root@hostA:~# zoneadm -z zoss-zone detach
    Exported zone zpool: zoss-zone_rpool
    Unconfigured zone storage resource(s) from:
    iscsi://192.168.202.10/luname.naa.600144F0949056290000529625DD0001
    
  2. Transfer the zone configuration from hostA to hostB:

    admin@hostA:~$ zonecfg -z zoss-zone export | ssh hostB 'cat - > /home/admin/zonecfg.zfg' 
    root@hostB:~# zonecfg -z zoss-zone -f /home/admin/zonecfg.zfg
    
  3. Attach the zone to hostB:

    Attaching the zone completes the following tasks:

    • Configures the iSCSI initiator
    • Discovers the iSCSI target and LU
    • Creates the logical device links
    root@hostB:~# zoneadm -z zoss-zone attach
    

Add Multipathing to the iSCSI Storage Configuration

No additional changes are required to the basic configuration established in the previous sections. However, consider adding multipathing to achieve the following additional benefits:

  • Ability to use dedicated NICs and network links for iSCSI traffic
  • Ability to connect iSCSI targets to multiple networks
  • Ability to limit LU discovery to a dedicated iSCSI target group

Figure 4 shows the multipath configuration described in this section:

Figure 4. Multipath configuration

Figure 4. Multipath configuration

  1. Assign the iSCSI target to multiple NICs on the Oracle ZFS Storage Appliance:

    1. Select Configuration->SAN, and create a target group called zoss-zone.

      • Select the LU and drag it over to the Target Groups box.
      • Click the edit button.
      • Add the zoss-zone name.
    2. Assign multiple network interfaces to the target group:

      • Select the LU target.
      • Select the network interface.
  2. Limit LU discovery to the zoss-zone target group on the Oracle ZFS Storage Appliance, as shown in Figure 5:

    1. Select Configuration->Shares, and then select the zoss-zone LU.
    2. Click the edit button.
    3. Click the Protocols tab.
    4. Select the zoss-zone target group.

      Figure 5. Limiting LU discovery

      Figure 5. Limiting LU discovery

  3. Configure dedicated iSCSI network links on the first Oracle Solaris initiator (hostA).

    If multiple addresses are configured for the Oracle ZFS Storage Appliance, multiple discovery addresses are configured automatically.

    root@hostA:~# cat /etc/hosts | grep zfssa
    192.168.202.10	zfssa-iscsi
    192.168.203.10	zfssa-iscsi
    root@hostA:~# zonecfg -z zoss-zone info rootzpool
    rootzpool:
           storage: iscsi://zfssa-iscsi/luname.naa.600144F0949056290000529625DD0001
    root@hostA:~# ipadm create-ip net1
    root@hostA:~# ipadm create-ip net2
    root@hostA:~# ipadm create-addr -T static -a 192.168.202.7 net1/iscsi202
    root@hostA:~# ipadm create-addr -T static -a 192.168.203.7 net2/iscsi203
    root@cantaloup:~# iscsiadm modify discovery -t enable
    root@cantaloup:~# iscsiadm list discovery-address
    Discovery Address: 192.168.202.7:3260
    Discovery Address: 192.168.203.7:3260
    
  4. List the targets that have multipathing disabled (via stmsboot -d -D iscsi) on the Oracle Solaris initiator.
    root@hostA:~# iscsiadm list target -S
    
    Target: iqn.1986-03.com.sun:02:12c5dc60-1854-62c4-8f61-aa78d0815dc2
        Alias: zoss-zone
        TPGT: 3
        ISID: 4000002a0000
        Connections: 1
        LUN: 0
             Vendor:  SUN     
             Product: Sun Storage 7000
             OS Device Name: /dev/rdsk/c1t6d0s2
    
    Target: iqn.1986-03.com.sun:02:12c5dc60-1854-62c4-8f61-aa78d0815dc2
        Alias: zoss-zone
        TPGT: 2
        ISID: 4000002a0000
        Connections: 1
        LUN: 0
             Vendor:  SUN     
             Product: Sun Storage 7000
             OS Device Name: /dev/rdsk/c1t3d0s2
    
  5. Enabled iSCSI multipathing and list the iSCSI targets on the Oracle Solaris initiator:

    root@hostA:~# stmsboot -e -D iscsi
    root@hostA:~# iscsiadm list target -S
    
    Target: iqn.1986-03.com.sun:02:12c5dc60-1854-62c4-8f61-aa78d0815dc2
        Alias: zoss-zone
        TPGT: 3
        ISID: 4000002a0000
        Connections: 1
        LUN: 0
             Vendor:  SUN     
             Product: Sun Storage 7000
             OS Device Name: /dev/rdsk/c0t600144F0949056290000529625DD0001d0s2
    
    Target: iqn.1986-03.com.sun:02:12c5dc60-1854-62c4-8f61-aa78d0815dc2
        Alias: zoss-zone
        TPGT: 2
        ISID: 4000002a0000
        Connections: 1
        LUN: 0
             Vendor:  SUN     
             Product: Sun Storage 7000
             OS Device Name: /dev/rdsk/c0t600144F0949056290000529625DD0001d0s2
    

Control iSCSI LU and Target Discovery

You can control the discovery of an iSCSI target or an LU. If a target or LU is not discovered, it is not visible and it is not accessible.

To control LU discovery, you assign an LU to group of initiators and then specify which initiator group is permitted to discover the LU. To control target discovery, you enable unidirectional CHAP to control which initiators are allowed to discover a target.

  1. Identify the iSCSI qualified name (IQN) on the first Oracle Solaris initiator:
    root@hostA:~# iscsiadm list initiator-node
    Initiator node name: iqn.1986-03.com.sun:01:e00000000000.5295332b
    
  2. On the Oracle ZFS Storage Appliance, create an initiator group by dragging the allowed initiators into a new initiator group, and then assign the initiator group to the LU, as shown in Figure 6.

    1. Select the initiator and drag it to the new initiator group, zoss-zone-hosts.
    2. Select Configuration->Shares, and then select the zoss-zone target LU.
    3. Click the edit button.
    4. Click the Protocols tab.
    5. Click the edit button next to the Initiator group and select the zoss-zone-hosts initiator group.

      Figure 6. Creating an initiator group

      Figure 6. Creating an initiator group

  3. Enable unidirectional CHAP on the first Oracle Solaris initiator:

    root@hostA:~# iscsiadm modify initiator-node --authentication CHAP
    root@hostA:~# iscsiadm modify initiator-node --CHAP-secret
    Enter secret: ************
    Re-enter secret: ************
    
    root@hostA:~# iscsiadm list initiator-node
    Initiator node name: iqn.1986-03.com.sun:01:e00000000000.5295332b
    Initiator node alias: hostA
        Authentication Type: CHAP
        CHAP Name: iqn.1986-03.com.sun:01:e00000000000.5295332b
    
  4. On the Oracle ZFS Storage Appliance, enable CHAP for the initiator and the target, as shown in Figure 7:

    1. Select Configuration->SAN->Initiators, and then select the initiator and click the edit button.
    2. Click the Use CHAP button.
    3. Enter the CHAP name, which is the IQN.
    4. Enter the CHAP secret password.
    5. Select Configuration->SAN->Targets, and then select the target and click the edit button.
    6. For Initiator authentication node, select the CHAP option.

      Figure 7. Enabling CHAP

      Figure 7. Enabling CHAP

  5. Disable and then re-enable discovery on the first Oracle Solaris initiator:

    root@hostA:~# iscsiadm modify discovery -t disable
    root@hostA:~# iscsiadm modify discovery -t enable
    

(Optional) Use ZFS-Encrypted File Systems

You can secure storage and the data path by enabling ZFS encryption when you create a ZFS file system or volume. To do this, you select a wrapping key; the source can be the prompt, a file, HTTPS, or PKCS#1. The system selects the random encryption key.

In the following steps, the zoss-zone rpool is re-created because the ZOSS framework creates only unencrypted datasets. After the rpool is re-created, ZOSS finds and uses the re-created rpool.

  1. Identify the iSCSI device of the existing zoss-zone rootzpool device:

    root@hostA:~# zonecfg -z zoss-zone info rootzpool
    rootzpool:
        storage: iscsi://zfssa-iscsi/luname.naa.600144F0949056290000529625DD0001
    
  2. Identify the zone's iSCSI device to the mapped device name:

    root@hostA:~# suriadm map iscsi://zfssa-iscsi/luname.naa.600144F0949056290000529625DD0001
    PROPERTY     VALUE
    mapped-dev   /dev/dsk/c0t600144F0949056290000529625DD0001d0s0
    
  3. Create the encryption key for the zone's rpool (zoss-zone_rpool) by using the pktool command to generate an AES256 encryption key:

    root@hostA:~# pktool genkey keystore=file outkey=/etc/zones/zoss-zonekey.file keytype=aes keylen=256
    root@hostA:~# chmod 600 /etc/zones/zoss-zonekey.file  
    
  4. Create the zone's rpool (zoss-zone_rpool) and specify that the top-level pool file system be encrypted. Also specify the encrypted key file and device that was mapped in Step 2:

    root@hostA:~# zpool create -m /zones/zoss-zone -O encryption=on 
    -O keysource=raw,file:///etc/zones/zoss-zonekey.file zoss-zone_rpool  c0t600144F0949056290000529625DD0001d0
    
  5. Export the zoss-zone rpool and re-install the zoss-zone:

    root@hostA:~# zpool export zoss-zone_rpool
    root@hostA:~# zoneadm -z zoss-zone install 
    

See Also

About the Authors

Detlef Drewanz is a Master Principal Sales Consultant on the Oracle Systems Sales Consulting team in Germany. He is an Oracle Solaris and virtualization specialist.

Cindy Swearingen is an Oracle Solaris Product Manager who specializes in ZFS and storage features.

Revision 1.0, 06/18/2014

Follow us:
Blog | Facebook | Twitter | YouTube