Articles
Server and Storage Administration
By Andrew Ness
October 2011
If your server is running out of internal storage capacity, you can quickly allocate just the right amount of storage from resilient storage pools on the Sun ZFS Storage Appliance from Oracle. This article describes how to use Sun ZFS Storage Appliance iSCSI LUNs in an Oracle Linux environment. Oracle Linux comes with a software iSCSI initiator package that can be configured to integrate iSCSI LUNs presented by the Sun ZFS Storage Appliance into the Oracle Linux environment.
This article first describes how to install the Oracle Linux iSCSI initiator on an Oracle Linux server. Then it describes how to configure an iSCSI LUN on the Sun ZFS Storage Appliance so the LUN can be accessed by an Oracle Linux server using iSCSI protocol. Procedures are provided for configuring the LUN using either the appliance's browser user interface (BUI) or the appliance's command line interface (CLI) . Finally, the article shows how to prepare the LUN for use by the Oracle Linux server.
The procedures in this article assume the following:
root password for the Sun ZFS Storage Appliance is known.root password for the Oracle Linux server is known.The Oracle Linux iSCSI initiator package is not installed by default so it must be installed manually. The package can be installed using one of these options:
yum(8) command as root to execute a text-based installation, as shown in Listing 1.[root@oel ~]# yum install iscsi-initiator-utils Loaded plugins: refresh-packagekit Setting up Install Process Resolving Dependencies --> Running transaction check --> Package iscsi-initiator-utils.x86_64 0:6.2.0.872-21.0.1.el6 will be --> installed --> Finished Dependency Resolution Dependencies Resolved ======================================================================= Package Arch Version Repository Size ======================================================================= Installing: iscsi-initiator-utils x86_64 6.2.0.872-21.0.1.el6 ol6_u1_base 578 k Transaction Summary ======================================================================= Install 1 Package(s) Total download size: 578 k Installed size: 2.0 M Is this ok [y/N]: y Downloading Packages: iscsi-initiator-utils-6.2.0.872-21.0.1.el6.x86_64.rpm | 578 kB 00:03 Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Installing : iscsi-initiator-utils-6.2.0.872-21.0.1.el6.x86_64 1/1 Installed: iscsi-initiator-utils.x86_64 0:6.2.0.872-21.0.1.el6 Complete! [root@oel ~]#
Once the iSCSI Initiator package is installed, configure the package to start on reboot by completing the following steps:
[root@oel ~]# chkconfig --list iscsi iscsi 0:off 1:off 2:off 3:off 4:off 5:off 6:off [root@oel ~]# chkconfig --list iscsid iscsid 0:off 1:off 2:off 3:off 4:off 5:off 6:off [root@oel ~]#
This example shows that both the iscsi and iscsid services are set to off at all run levels, so the iSCSI Initiator will not start at system reboot.
on for either the iscsi or iscsid service, enable them, as shown below:[root@oel ~]# chkconfig iscsi on [root@oel ~]# chkconfig iscsid on [root@oel ~]# chkconfig --list iscsi iscsi 0:off 1:off 2:on 3:on 4:on 5:on 6:off [root@oel ~]# chkconfig --list iscsid iscsid 0:off 1:off 2:on 3:on 4:on 5:on 6:off [root@oel ~]# service iscsi start [root@oel ~]# service iscsid start
The iscsi and iscsid services are now set to start at system reboot.
The iSCSI Qualified Name (IQN) of the host is used to identify the host to the Sun ZFS Storage Appliance. The IQN is automatically generated when the iSCSI initiator package is installed and it is stored in a file in /etc/iscsi. To find the IQN for the host, enter the following:
[root@oel ~]# cat /etc/iscsi/initiatorname.iscsi InitiatorName=iqn.1988-12.com.oracle:5fa4af61c4a0
The IQN for the host used in this example is iqn.1988-12.com.oracle:5fa4af61c4a0.
In the following example, CHAP is used to authenticate communication between the host and the Sun ZFS Storage Appliance. The IQN is used as the CHAP name for the iSCSI initiator and a secure password is used for the CHAP secret.
To configure CHAP authentication, edit the file /etc/iscsi/iscsid.conf to make the following changes:
# character from the beginning of the following line:node.session.auth.authmethod = CHAP
# character from the beginning of the following lines:node.session.auth.username = username node.session.auth.password = password
username to the IQN for the host, for example:iqn.1988-12.com.oracle:5fa4af61c4a0
password to a reasonably secure password to be used for authentication. In this example, the password CHAPsecret11 is used.The lines now look like this:
node.session.auth.username = iqn.1988-12.com.oracle:5fa4af61c4a0 node.session.auth.password = CHAPsecret11
# character from the beginning of the following lines:node.session.auth.username_in = username node.session.auth.password_in = password
node.session.auth.username_in = chapuser node.session.auth.password_in = CHAPsecret22
Since this user name and password are used for all targets, which may include more than one Sun ZFS Storage Appliance or devices from other manufacturers, the user name should be generic rather than associated with a particular IQN. For this example, the storage administrator has provided the user name chapuser. The CHAP secret is the reasonably secure password, CHAPsecret22.
Next, the Sun ZFS Storage Appliance must be configured to recognize the Oracle Linux host and to present iSCSI LUNs to it.
As a unified storage platform, the Sun ZFS Storage Appliance supports access to block-protocol LUNs using iSCSI and Fibre Channel protocols. This section describes how to use the Sun ZFS Storage Appliance BUI to configure the appliance to recognize the Oracle Linux host and present iSCSI LUNs to it.
To open the Sun ZFS Storage Appliance BUI, do the following:
https://<appliance IP address or host name>:215
The login dialog box shown in Figure 1 is displayed.

Figure 1. Sun ZFS Storage Appliance Login
A target group is created on the Sun ZFS Storage Appliance to define the ports and the protocol by which the LUN to be presented to the Oracle Linux server is accessed. For this example, an iSCSI target group is created that contains the LUN as an iSCSI target that is identified by the default IQN for the Sun ZFS Storage Appliance and presented over default appliance interfaces. Since CHAP will be used for authentication between the storage and the host, CHAP parameters are also specified in this example.
To define an iSCSI target group on the Sun ZFS Storage Appliance, complete these steps:

Figure 2. iSCSI Target Configuration
chapuser and CHAPsecret22 are used.
Figure 3. New iSCSI Target Dialog Box

Figure 4. Selecting the iSCSI Target Entry

Figure 5. Creating an iSCSI Target Group
targets-n, where n is an integer, as shown in Figure 6.
Figure 6. Selecting the New Target Group for Editing
iscsi-TG is used.
Figure 7. Renaming the iSCSI Target Group

Figure 8. Viewing the New iSCSI Target Group
An iSCSI initiator is defined to restrict which servers have access to a particular volume. If more than one host can write to a given volume concurrently, inconsistency in file system caching between hosts can cause corruption in the on-disk image. Typically, a single initiator is given access to a volume, unless a specialized cluster file system is being used.
The iSCSI initiator defines the "host" from the point of view of the Sun ZFS Storage Appliance. To identify the Oracle Linux server to the Sun ZFS Storage Appliance, the Oracle Linux iSCSI initiator IQN must be registered with the appliance by completing the following steps:

Figure 9. Selecting the iSCSI Initiators Panel

Figure 10. Configuring a New iSCSI Initiator
Related iSCSI initiators can be combined into logical groups to allow single commands to be executed on multiple iSCSI initiators, for example, assigning LUN access to all iSCSI initiators in a group by using one command. For this example, the iSCSI initiator group contains one initiator, but in a cluster, where multiple servers are treated as a single logical entity, the initiator group may contain multiple initiators.
To create an iSCSI initiator group, complete these steps:

Figure 11. Displaying the Move Icon for a New iSCSI Initiator

Figure 12. Creating an iSCSI Initiator Group
initiators-n, where n is an integer, as shown in Figure 13.
Figure 13. Selecting the iSCSI Initiator Group
linux-servers is used.
Figure 14. Renaming the iSCSI Initiator Group

Figure 15. iSCSI Initiator Configuration Completed
A project can be defined in the Sun ZFS Storage Appliance to group related volumes. A project allows inheritance of properties for file systems and LUNs presented from the project and also allows quotas and reservations to be applied.
To create a project, complete the following steps:

Figure 16. Viewing a Project

Figure 17. Create Project Dialog Box

Figure 18. Displaying the New linux-project Project
A LUN must now be created from an existing pool of storage resources, which the Oracle Linux server will access. For this example, a thin-provisioned 64 GB LUN called DocArchive1 will be created. The LUN will be set up for an EXT3 file system.
The target group will be the iSCSI target group created in the Defining an iSCSI Target Group section to ensure that this LUN can be accessed using iSCSI protocol. The initiator group defined in the Defining an iSCSI Initiator Group section will be used to ensure that only the server(s) defined in the linux-servers group can access this LUN (in this example, this group contains only one server).
To create a target group, complete the following steps:

Figure 19. Viewing LUNs for the linux-project Project
DocArchive1 and the Volume size field to 64 GB, and select the Thin-provisioned check box. Set the Target Group to the iSCSI target group iscsi-TG and the Initiator Group to linux-servers. Set the Volume block size to 32K because the LUN will hold an EXT3 file system.
Figure 20. Create LUN Dialog Box
As a unified storage platform, the Sun ZFS Storage Appliance supports access to block-protocol LUNs using iSCSI and Fibre Channel protocols. This section describes how to use the Sun ZFS Storage Appliance CLI to configure the Sun ZFS Storage Appliance to recognize the Oracle Linux host and present iSCSI LUNs to it.
The CLI must be accessed using an SSH-enabled terminal session. On a host that has an SSH client available, enter the following:
user@host> ssh root@<appliance IP address or host name> login as: root Using keyboard-interactive authentication. Password: XXXXX Last login: Mon Jul XX XX:XX:XX XXXX from Linux zfssa:>
Navigation follows a similar pattern in the CLI as in the BUI with paths used in commands often corresponding to paths through tabs and screens in the BUI.
In the example in Listing 2, a target group is created on the Sun ZFS Storage Appliance to define the ports and the protocol by which the LUN to be presented to the Oracle Linux server is accessed. For this example, an iSCSI target group is created that contains the LUN as an iSCSI target that is identified by the default IQN for the Sun ZFS Storage Appliance and presented over default appliance interfaces.
Since CHAP will be used for authentication between the storage and the host, CHAP parameters are also specified in this example.
Listing 2: Defining an iSCSI Target Group
zfssa:>
configuration san targets
zfssa:configuration san targets>
iscsi
zfssa:configuration san targets iscsi>
create
zfssa:configuration san targets iscsi target (uncommitted)>
set alias=iscsi-Target alias = iscsi-Target (uncommitted)
zfssa:configuration san targets iscsi target (uncommitted)>
commit
zfssa:configuration san targets iscsi>
list
TARGET ALIAS
target-000 iscsi-Target
|
+-> IQN
iqn.1986-03.com.sun:02:e8589be5-144d-c9b2-89d4-f7fe4e887881
zfssa:configuration san targets iscsi>
select target-000
zfssa:configuration san targets iscsi target-000>
list
Properties:
alias = iscsi-Target
iqn = iqn.1986-03.com.sun:02:e8589be5-144d-c9b2-89d4-f7fe4e887881
auth = none
targetchapuser = (unset)
targetchapsecret = (unset)
interfaces = nge0
zfssa:configuration san targets iscsi target-000>
set targetchapuser=chapuser
targetchapuser =
iqn.1986-03.com.sun:02:e8589be5-144d-c9b2-89d4-f7fe4e887881
(uncommitted)
zfssa:configuration san targets iscsi target-000>
set targetchapsecret=CHAPsecret22
targetchapsecret = ************ (uncommitted)
zfssa:configuration san targets iscsi target-000>
commit
zfssa:configuration san targets iscsi target-000>
list
Properties:
alias =
iqn =
auth =
targetchapuser = \
iqn.1986-03.com.sun:02:e8589be5-144d-c9b2-89d4-f7fe4e887881
targetchapsecret = ************
interfaces =
zfssa:configuration san targets iscsi target-000>
cd ..
zfssa:configuration san targets iscsi>
list
TARGET ALIAS
target-000 iscsi-Target
|
+-> IQN
iqn.1986-03.com.sun:02:e8589be5-144d-c9b2-89d4-f7fe4e887881
zfssa:configuration san targets iscsi>
group
zfssa:configuration san targets iscsi groups>
create
zfssa:configuration san targets iscsi group (uncommitted)>
set name=iscsi-TG name = iscsi-TG (uncommitted)
zfssa:configuration san targets iscsi group (uncommitted)>
set targets=iqn.1986-03.com.sun:02:e8589be5-144d-c9b2-89d4-f7fe4e887881
targets = iqn.1986-03.com.sun:02:e8589be5-144d-c9b2-89d4-f7fe4e887881 (uncommitted)
zfssa:configuration san targets iscsi group (uncommitted)>
commit
zfssa:configuration san targets iscsi groups>
list
GROUP NAME
group-000 iscsi-TG
|
+-> TARGETS
iqn.1986-03.com.sun:02:e8589be5-144d-c9b2-89d4-f7fe4e887881
zfssa:configuration san targets iscsi groups>
cd /
zfssa:>
An iSCSI initiator is defined to restrict which servers have access to a particular volume. If more than one host can write to a given volume concurrently, inconsistency in file system caching between hosts can cause corruption in the on-disk image. Typically, a single initiator is given access to a volume, unless a specialized cluster file system is being used.
The iSCSI initiator defines the "host" from the point of view of the Sun ZFS Storage Appliance. To identify the Oracle Linux server to the Sun ZFS Storage Appliance, the Oracle Linux iSCSI IQN must be registered with the appliance, as shown in the example in Listing 3.
Listing 3: Defining an iSCSI Initiator
zfssa:>
configuration san initiators
zfssa:configuration san initiators>
iscsi
zfssa:configuration san initiators iscsi>
ls
Children:
groups => Manage groups
zfssa:configuration san initiators iscsi>
create
zfssa:configuration san initiators iscsi initiator (uncommitted)>
set alias=OEL alias = OEL (uncommitted)
zfssa:configuration san initiators iscsi initiator (uncommitted)>
set initiator= iqn.1988-12.com.oracle:5fa4af61c4a0 initiator = iqn.1988-12.com.oracle:5fa4af61c4a0 (uncommitted)
zfssa:configuration san initiators iscsi initiator (uncommitted)>
set chapuser= iqn.1988-12.com.oracle:5fa4af61c4a0 chapuser = iqn.1988-12.com.oracle:5fa4af61c4a0 (uncommitted)
zfssa:configuration san initiators iscsi initiator (uncommitted)>
set chapsecret=CHAPsecret11 chapsecret = ************ (uncommitted)
zfssa:configuration san initiators iscsi initiator (uncommitted)>
commit
zfssa:configuration san initiators iscsi> ls
Initiators:
NAME ALIAS
initiator-000 Linux
|
+-> INITIATOR
iqn.1988-12.com.oracle:5fa4af61c4a0
Children:
groups => Manage groups
zfssa:configuration san initiators iscsi>
Related iSCSI initiators can be combined into logical groups to allow single commands to be executed on multiple iSCSI initiators, for example, assigning LUN access to all iSCSI initiators in a group by using one command. In the example in Listing 4, the iSCSI initiator group contains one initiator, but in a cluster, where multiple servers are treated as a single logical entity, the initiator group may contain multiple initiators.
Listing 4: Defining an iSCSI Initiator Group
zfssa:configuration san initiators iscsi>
groups
zfssa:configuration san initiators iscsi groups>
create
zfssa:configuration san initiators iscsi group (uncommitted)>
set name=linux-servers name = linux-servers (uncommitted)
zfssa:configuration san initiators iscsi group (uncommitted)>
set initiators= iqn.1988-12.com.oracle:5fa4af61c4a0 initiators = iqn.1988-12.com.oracle:5fa4af61c4a0 (uncommitted)
zfssa:configuration san initiators iscsi group (uncommitted)>
commit
zfssa:configuration san initiators iscsi groups>
ls
Groups:
GROUP NAME
group-000 linux-servers
|
+-> INITIATORS
iqn.1988-12.com.oracle:5fa4af61c4a0
zfssa:configuration san initiators iscsi groups>
cd /
zfssa:>
A project can be defined in the Sun ZFS Storage Appliance to group related volumes. A project allows inheritance of properties for file systems and LUNs presented from the project and also allows quotas and reservations to be applied.
zfssa:> shares zfssa:shares> project linux-project zfssa:shares linux-project (uncommitted)> commit zfssa:shares> cd / zfssa:>
A LUN must now be created from an existing pool of storage resources, which the Oracle Linux server will access. In the following example, a thin-provisioned 64 GB LUN called DocArchive1 is created. The LUN is set up for an EXT3 file system.
The target group is the iSCSI target group created in the Defining an iSCSI Target Group section to ensure that this LUN can be accessed using iSCSI protocol. The initiator group defined in the Defining an iSCSI Initiator Group section is used to ensure that only the server(s) defined in the linux-servers group can access this LUN (in the example in Listing 5, this group contains only one server).
Listing 5: Defining a Sun ZFS Storage Appliance LUN
zfssa:shares>
select linux-project zfssa:shares linux-project>
lun DocArchive1 zfssa:shares linux-project/DocArchive1 (uncommitted)>
set volsize=64G volsize = 64G (uncommitted)
zfssa:shares linux-project/DocArchive1 (uncommitted)>
set targetgroup=iscsi-TG targetgroup = iscsi-TG (uncommitted)
zfssa:shares linux-project/DocArchive1 (uncommitted)>
set initiatorgroup=linux-servers initiatorgroup = linux-servers (uncommitted)
zfssa:shares linux-project/DocArchive1 (uncommitted)>
list
Properties:
checksum = fletcher4 (inherited)
compression = off (inherited)
dedup = false (inherited)
copies = 1 (inherited)
logbias = latency (inherited)
secondarycache = all (inherited)
volblocksize = 8K (default)
volsize = 64G (uncommitted)
sparse = false (default)
exported = true (inherited)
targetgroup = iscsi-TG (uncommitted)
initiatorgroup = linux-servers (uncommitted)
lunumber = (default)
assignednumber = (default)
status = (default)
fixednumber = (default)
lunguid = (default)
canonical_name = (default)
nodestroy = (default)
Children:
snapshots => Manage snapshots
replication => Manage remote replication
zfssa:shares linux-project/DocArchive1 (uncommitted)>
set volblocksize=32k volblocksize = 32K (uncommitted)
zfssa:shares linux-project/DocArchive1 (uncommitted)>
commit
zfssa:shares linux-project>
cd / zfssa:shares>
Now that the LUN is prepared and available to an authenticated iSCSI initiator, configure the LUN for use by the Oracle Linux server by completing the following steps:
[root@oel ~]# service iscsi start [root@oel ~]# iscsiadm -m discovery -t sendtargets -p zfssa 192.168.1.12:3260,2 iqn.1986-03.com.sun:02:4ec2b85a-9b5d-e02c-ee9c-bdd55452e7de [fe80::214:4fff:fe0f:92b4]:3260,2 iqn.1986-03.com.sun:02:4ec2b85a-9b5d-e02c-ee9c-bdd55452e7de [root@oel ~]# iscsiadm -m node -p zfssa -login [root@oel ~]# tail /var/log/messages Jul 13 14:35:37 oel kernel: scsi21 : iSCSI Initiator over TCP/IP Jul 13 14:35:37 oel iscsid: Could not set session19 priority. READ/WRITE throughout and latency could be affected. Jul 13 14:35:38 oel kernel: scsi 21:0:0:0: Direct-Access SUN Sun Storage NAS 1.0 PQ: 0 ANSI: 5 Jul 13 14:35:38 oel kernel: sd 21:0:0:0: Attached scsi generic sg2 type 0 Jul 13 14:35:38 oel kernel: sd 21:0:0:0: [sdb] 134217728 512-byte logical blocks: (68.7 GB/64.0 GiB) Jul 13 14:35:38 oel kernel: sd 21:0:0:0: [sdb] Write Protect is off Jul 13 14:35:38 oel kernel: sd 21:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Jul 13 14:35:38 oel kernel: sdb: unknown partition table Jul 13 14:35:39 oel kernel: sd 21:0:0:0: [sdb] Attached SCSI disk Jul 13 14:35:39 oel iscsid: Connection19:0 to [target: iqn.1986-03.com.sun:02:4ec2b85a-9b5d- e02c-ee9c-bdd55452e7de, portal: 192.168.1.12,3260] through [iface: default] is operational now [root@oel ~]# ls -l /dev/sd? brw-rw----. 1 root disk 8, 0 Jul 13 13:06 /dev/sda brw-rw----. 1 root disk 8, 16 Jul 13 14:35 /dev/sdb
fdisk(8) command, as shown in Listing 7.
root@oel:~# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xda0cb962.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to switch off the mode (command 'c') and change display units to sectors (command 'u').
Command (m for help): p
Disk /dev/sdb: 68.7 GB, 68719476736 bytes
255 heads, 63 sectors/track, 8354 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xda0cb962
Device Boot Start End Blocks Id System
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-8354, default 1): 1
Last cylinder, +cylinders or +size{K,M,G} (1-8354, default 8354): 8354
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
This device is now available for selection similar to an internal server disk. Examples of disk deployment include:
[root@oel ~]# mkfs.ext3 -L DocArchive1 /dev/sdb1
mke2fs 1.41.12 (17-May-2010)
Filesystem label=DocArchive1
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
4194304 inodes, 16775868 blocks
838793 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
512 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632,
2654208, 4096000, 7962624, 11239424
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
This filesystem will be automatically checked every 27 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
[root@oel ~]# vi /etc/fstab
#
# /etc/fstab
# Created by anaconda on Wed Jul 13 09:19:08 2011
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/vg_oel-lv_root / ext4 defaults 1 1
UUID=043800a8-30b9-4ba8-9ce2-0c114273d71b /boot ext4 defaults 1 2
/dev/mapper/vg_oel-lv_swap swap swap defaults 0 0
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
LABEL=DocArchive1 /docarchive1 ext3 defaults 2 2
[root@oel ~]# mkdir /docarchive1
[root@oel ~]# mount /docarchive1
[root@oel ~]# df -k
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/vg_oel-lv_root
29745220 3067908 25166300 11% /
tmpfs 704376 420 703956 1% /dev/shm
/dev/sda1 495844 70382 399862 15% /boot
/dev/sdb1 66050540 184268 62511100 1% /docarchive1
[root@oel ~]#
The final two lines of the output from the df(1) command show that approximately 64 GB of new space is now available.
This article described how to configure the software iSCSI initiator package supplied with Oracle Linux to enable an Oracle Linux server to access iSCSI LUNs presented by Oracle's Sun ZFS Storage Appliance.
| Revision 1.0, 10/12/2011 |