Building on StorageBy Arup Nanda
Create sophisticated, feature-rich clustered file systems on Oracle Automatic Storage Management disk groups.
John, the DBA at ACME Corporation, gets a unique visit from Maureen, the business lead of the bank’s customer service application, and her technical team. “We’re rolling out a new Website,” explains Maureen, “that displays images of cleared checks. We budgeted for a dedicated image server, but those funds evaporated with the economic downturn. So we’d like to use our Oracle Real Application Clusters [Oracle RAC] database servers to also store and retrieve images.”
Keith, the lead architect, chimes in: “But we’ve hit a roadblock. The database servers are part of our Oracle RAC cluster, but any images we put on one server won’t be visible to the instances running on the other servers. We could put them on Oracle Automatic Storage Management disk groups as large objects [LOBs], but our legacy systems require a cooked—not raw—file system. We thought of investing in a cluster file system, but it would be prohibitively expensive.”
“In addition,” continues the lead developer, Ben, “we want to make sure a copy of each image file is automatically saved before it gets deleted or modified. We need these copies to help us create a consistent set of backup files to move to tape.”
Craig, the system architect, adds, “We also don’t know how much space will be required up front, and we don’t have the money to buy a lot of disks, so we want to be able to add storage space dynamically without any application outages.”
John ponders these requirements. “So what you’re saying is that you want a regular [cooked] file system—not an Oracle Automatic Storage Management disk group—that’s visible across all the nodes of an Oracle RAC cluster and is also capable of advanced file system operations such as automatic backup and dynamic resizing. But you want all this without investing a single penny in a sophisticated clustered file system?”
“Precisely,” answers Maureen rather sheepishly—keeping her fingers crossed.
“I have good news,” replies a smiling John. “This is all possible with a new feature of Oracle Database 11g Release 2 called Oracle Automatic Storage Management Cluster File System. With it, we can create a regular file system on top of Oracle Automatic Storage Management disk groups that are already visible to all the nodes in the cluster. When we’ve created the regular file system, any application can access it to store and retrieve files.” The mood in the room brightens visibly, and Maureen presses John to walk through the details.
In this article, you will learn how John uses Oracle Automatic Storage Management Cluster File System to accomplish the various objectives Maureen’s team has put on the table.
Oracle Automatic Storage Management Dynamic VolumesJohn points out that creating an Oracle Automatic Storage Management Cluster File System means first installing an Oracle grid infrastructure. With that prerequisite in place, John starts by creating an Oracle Automatic Storage Management dynamic volume. Oracle Automatic Storage Management dynamic volumes are placed in an Oracle Automatic Storage Management disk group. John uses the following SQL to scan his available disk groups to find one that has sufficient empty space:
select name, total_mb, free_mb from v$asm_diskgroup; NAME TOTAL_MB FREE_MB ------- --------- --------- ACFSDG1 152523 149908
The output shows that the disk group ACFSDG1 has about 152 GB of total capacity and about 149 GB free. John considers this disk group an ideal candidate for the Oracle Automatic Storage Management volume.
Next he checks the compatibility level of the disk group. For creating an Oracle Automatic Storage Management volume, the compatibility level must be at least 11.2. To perform the check, he issues the following SQL:
SQL> select compatibility, database_compatibility 2 from v$asm_diskgroup 3 where name = 'ACFSDG1'; COMPATIBILITY DATABASE_COMPATIBILITY ------------ ----------------------- 220.127.116.11.0 18.104.22.168.0
If the result had shown something other than 11.2, he would have had to set the compatibility to 11.2 by issuing the following series of SQL statements:
SQL> alter diskgroup acfsdg1 set attribute 'compatible.asm' = '11.2'; Diskgroup altered. SQL> alter diskgroup acfsdg1 set attribute 'compatible.rdbms' = '11.2'; Diskgroup altered. SQL> alter diskgroup acfsdg1 set attribute 'compatible.advm' = '11.2'; Diskgroup altered.
Next John creates the Oracle Automatic Storage Management dynamic volume. He mentions the four different ways to accomplish this task: using regular SQL statements, using the ASMCMD command line, using the ASMCA utility, and using Oracle Enterprise Manager. To keep it simple, John shows everyone the approach that involves regular SQL.
To create a 10 MB volume called ACFSVOL1A, John issues the following SQL:
alter diskgroup acfsdg1 add volume acfsvol1a size 10M;
Afterward he confirms that the volume was correctly created, by checking a view called V$ASM_VOLUME, in which the volume will appear if it was created successfully. The following query confirms that the volume was successfully created:
SQL> select volume_name, volume_device 2 from v$asm_volume; VOLUME_NAME VOLUME_DEVICE ----------- --------------------- ACFSVOL1A /dev/asm/acfsvol1a-153 ACFSVOL1B /dev/asm/acfsvol1b-153
Creating an Oracle Automatic Storage Management Cluster File SystemThe preceding ALTER DISKGROUP statement initiated an action that created an Oracle Automatic Storage Management dynamic volume file as well as an associated Linux dynamic volume device that is directly addressable by the operating system. (The name of the device is shown in the VOLUME_DEVICE column of the V$ASM_VOLUME view.) This device enables Linux to identify the Oracle Automatic Storage Management volume as a device and makes it possible to create a file system on the volume by using the mkfs command. John demonstrates this by issuing the command shown in Listing 1 to create a file system on the Oracle Automatic Storage Management volume.
Code Listing 1: Using mkfs to create the file system
$ mkfs -t acfs -b 4k /dev/asm/acfsvol1a-153 mkfs.acfs: version = 22.214.171.124.0.0 mkfs.acfs: on-disk version = 39.0 mkfs.acfs: volume = /dev/asm/acfsvol1a-153 mkfs.acfs: volume size = 1073741824 mkfs.acfs: Format complete.
The -t flag in the mkfs statement indicates that the file system should be of type acfs. Note that although the mkfs statement prepared the volume to be mounted as a file system, it did not actually mount it. To mount the file system, John first creates a directory named /acfsdir1, as the root user, as shown below.
$ mkdir -p /acfsdir1
He then mounts the file system /acfsdir1, using the regular Linux mount command:
$ mount -t acfs /dev/asm/acfsvol1a-153 /acfsdir1
To verify that the file system was mounted properly, he then uses the mount command shown in Listing 2.
Code Listing 2: Verifying that the file system is mounted
$ mount /dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) /dev/sda1 on /boot type ext3 (rw) tmpfs on /dev/shm type tmpfs (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) /dev/asm/acfsvol1-153 on /acfsdir1 type acfs (rw)
The output clearly shows that Linux has mounted the file system /acfsdir1. Once the file system is mounted, users can access /acfsdir1 as a regular file system. John then changes the directory ownership to oracle by issuing the following Linux command:
$ chown oracle:dba /acfsdir1
With this command, the oracle user can now read from and write to this file system. Furthermore, because Oracle Automatic Storage Management Cluster File System is a regular file system, any application can now access files on this file system without any special access mechanisms.
And John has accomplished everything without any additional software. Everyone breathes a sigh of relief.
Oracle Automatic Storage Management Cluster File System ComponentsTo help the team understand how Oracle Automatic Storage Management Cluster File System works, John draws everyone’s attention to Figure 1, which shows various components of Oracle Automatic Storage Management Cluster File System. One or more disks make up an Oracle Automatic Storage Management disk group—shown in the figure as ACFSDG1—on which one or more Oracle Automatic Storage Management volumes can be created. John points to the two Oracle Automatic Storage Management volumes in the figure—acfsvol1 and acfsvol2—and then to the two file systems—/acfsdir1 and /acfsdir2—on those volumes. As long as free space exists inside the disk group, the volumes can be expanded and the file system will expand as well.
When users access the file system, a component called Oracle Automatic Storage Management Dynamic Volume Manager translates the I/O calls and fetches the data from the appropriate location on the disk. The presence of this Oracle Automatic Storage Management Dynamic Volume Manager component, John points out, is what makes the volumes dynamic (and not bound to a physical device). Because of this component, new disks can be added and old ones removed, but nothing happens to the data on the volumes and file systems—making applications resilient.
John also mentions that when the system reboots, the file system does not by default remount automatically. The Oracle grid infrastructure can mount the file system after a reboot, but the volume must be registered in its registry. John shows how to register the Oracle Automatic Storage Management volume in the registry by using a utility called acfsutil, installed in the /sbin directory as part of the Oracle grid infrastructure installation:
$ acfsutil registry -a /dev/asm /acfsvol1a-153 /acfsdir1 acfsutil registry: mount point /acfsdir1 successfully added to Oracle Registry
The grid infrastructure registry, John explains, acts as a clusterwide file system reference file, much like the /etc/fstab file on a Linux system. After the file system is mounted in the registry, it is automatically brought up after a node reboots or the grid restarts.
Oracle Automatic Storage Management Cluster File System SnapshotsBen then asks, “What about my requirement to automatically save files before modification?” John demonstrates how easy it would be to perform that task. He describes another feature of Oracle Automatic Storage Management Cluster File System—called snapshots—that literally takes snapshots of the changed blocks of a file at a different location. To conserve space, snapshots work at the block level instead of the file level. When a file is modified, only the changed blocks are copied over to the new location. To demonstrate, John shows a listing of the files in the directory:
$ ls -l total 72 -rw-r--r-- 1 oracle dba 218 May 25 17:57 file1.gif -rw-r--r-- 1 oracle dba 436 May 25 17:57 file2.gif drwx------ 2 root root 65536 May 25 17:11 lost+found
He then creates a snapshot called backup1:
$ /sbin/acfsutil snap create backup1 /acfsdir1 acfsutil snap create: Snapshot operation is complete.
This operation creates a directory called backup1 inside a directory called snaps under a hidden directory called .ACFS (note the leading period). This directory contains modified file blocks. To trigger an automatic backup, John deletes a file:
$ rm file2.gif
He then demonstrates how the file is still available in the .ACFS location:
$ ls -l .ACFS/snaps/*/file2.gif -rw-r--r-- 1 oracle dba 218 May 25 17:57 .ACFS/snaps/backup1/file1.gif
When the snapshots are first created on the file system, he stresses, nothing is actually stored, so there is no space consumption. When the file is deleted, the previous images of the file blocks are copied to the snapshot. So if Ben ever wants to go back to a previous version of the file, he can access the snapshot. Similarly, he can also copy the snapshots to tape to create a consistent backup set of the files. He couldn’t be happier.
Oracle Automatic Storage Management Cluster File System Administration“It sounds too good to be true,” opines a skeptical Craig. He isn’t sure how flexible the system is and wants to know how difficult it is to manage. John assures him that administration is a breeze with either regular SQL, the ASMCMD command-line tool, or Oracle Enterprise Manager. With some example operations, he demonstrates the fine level of detail enabled by Oracle Automatic Storage Management Cluster File System administration tools.
SQL> alter diskgroup ACFSDG1 drop volume ACFSVOL11;
To unregister a file system from the Oracle Automatic Storage Management volume registry, he uses
$ acfsutil registry -d /acfs1
To delete the snapshots, he issues
$ /sbin/acfsutil snap delete backup1 /acfsdir1
One of Craig’s requirements is to add space to the file system on the fly without any application outages. John shows how to add 2,000 MB to the file system, using a completely online operation:
$ /sbin/acfsutil size +2000M /acfsdir1
John explains that other commands are equally simple and that the graphical interface of Oracle Enterprise Manager offers an even easier-to-use administrative interface.
Finally, John reviews the team’s original problems and shows how his solution addresses them, as summarized in Table 1.
Table 1: Issues and their solutions
ConclusionOracle Automatic Storage Management Cluster File System is a very powerful addition to Oracle Database 11g Release 2 that enables the creation of sophisticated, feature-rich clustered file systems. Oracle Automatic Storage Management Cluster File System makes it possible for legacy applications to access files through a regular file system visible to all the nodes of the cluster without requiring any specialized software.
Arup Nanda (firstname.lastname@example.org) has been an Oracle DBA for more than 14 years, handling all aspects of database administration, from performance tuning to security and disaster recovery. He was Oracle Magazine’s DBA of the Year in 2003.