Oracle Exadata Commands Reference

Part 3: Storage Management

by Arup Nanda

A detailed explanation of the CellCLI and DCLI commands, all their parameters and options, and how they are used for managing storage cells

> Back to Series TOC

(The purpose of this guide is educational: to jump-start the transition from DBA to DMA and serve as a quick reference for practicing DMAs. It is not intended to replace official Oracle-provided manuals or other documentation. The information in this guide is not validated by Oracle, is not supported by Oracle, and should only be used at your own risk.)

In the previous two installments, you learned how the Oracle Exadata Database Machine works, about its various components, and the categories of commands applicable to each of those components. One such component is the storage cells, which manage the disks. For many DBAs transitioning into the role of DMAs, perhaps the biggest challenge is becoming comfortable with storage operations, something many DBAs never had to address.

The storage cells in Exadata Database Machine are managed via two tools called CellCLI and DCLI. You had a glimpse of the CellCLI commands in action in Part 2, when configuring Exadata for the first time. In this installment, you will learn the detailed commands of these tools. In doing so, hopefully you will appreciate that it is not as difficult as you thought it would be.

Basics

The cellcli command is invoked from the Linux command line in the storage cells, not in the compute nodes. Similar to SQL*Plus showing SQL> prompt, CellCLI show the CellCLI> prompt where you will enter the commands.

# cellcli
CellCLI: Release 11.2.2.2.0 - Production on Fri Apr 01 10:39:58 EDT 2011

Copyright (c) 2007, 2009, Oracle.  All rights reserved.
Cell Efficiency Ratio: 372M

This will display the CellCLI prompt where you are expected to enter the commands.

CellCLI> … enter your commands here

HELP

The first command you may want to enter is know about all the other commands available. Well, get some help:

CellCLI> help

 HELP [topic]
   Available Topics:
        ALTER
        ALTER ALERTHISTORY
        ALTER CELL
… output truncated …

If you want to get more context sensitive help on a specific command,

CellCLI> help list

By the way, you don’t need to be in the CellCLI command prompt to give a command. If you know the command you want to give, you call it directly from Linux command prompt with the -e option.

# cellcli -e help

Or, you can use the typical Linux type command terminators, as shown below:

[celladmin@prolcel14 ~]# cellcli << EOF
> list cell
> EOF
CellCLI: Release 11.2.2.2.2 - Production on Sat May 14 16:28:43 EDT 2011

Copyright (c) 2007, 2009, Oracle.  All rights reserved.
Cell Efficiency Ratio: 160M

CellCLI> list cell
         prolcel14       online

[celladmin@prolcel14 ~]#

To exit from CellCLI, use the commands exit or quit.

Recording the Output

Like SQL*Plus, CellCLI also has a spool command that records the output to a file. Here is an example where you want to record the output to a file named mycellcli.txt

CellCLI> spool mycellcli.txt

To append to a file you already have, use append

CellCLI> spool mycellcli.txt append

Or, if you want to overwrite the file, use replace

CellCLI> spool mycellcli.txt replace

To stop spooling, just issue SPOOL OFF.

What if you want to know what file the output is being written to? Use SPOOL without any argument. It will show the name of the file.

CellCLI> spool
currently spooling to mycellcli.txt

Continuation Character

You will learn later in the article that you can issue complex queries in CellCLI. Some of these queries might span multiple lines. Since there is no command terminator like “;” in SQL, ending a line makes CellCLI to start interpreting it immediately. To make sure CellCLI understands the line extends beyond the current line, you can use the continuation character “-“, just as you would have done in SQL*Plus.

CellCLI> spool -

> mycellcli.txt

Note the continuation prompt “>” above. However, unlike SQL*Plus, CellCLI does not show line numbers for the continued lines.

Scripting

When you want to execute a standard command in CellCLI, you can create a script file with the commands and execute it using START or @ commands. For instance, to execute a script called listdisks.cli, you would issue one of the following:

CellCLI> @listdisks.cli
CellCLI>
start listdisks.cli

Note that the there is no restriction on the extension. I used cli merely as a convenience, it could have been anything else.

You can also use the routine Linux redirection command. Here is an example where you put all the commands in the file mycellci.commands and you want to spool the results to mycellci.out. This is good for automatic monitoring systems.

# cellcli <mycellci.commands >mycellci.out

Comments

You can put comments in scripts as well. Comments start with REM, REMARK or "--", as shown below:

REM This is a comment
REMARK This is another comment
-- This is yet another comment
list physicaldisk

Environment

You can define two environmental settings

  • dateformat – to show the  format of the date when it is displayed.

    CellCLI> set dateformat local
    Date format was set: Apr 1, 2011 3:35:17 PM.

    CellCLI> set dateformat standard
    Date format was set: 2011-04-01T15:35:30-04:00.

  • echo – to toggle the display of commands within scripts.

    CellCLI> set echo on
    CellCLI>
    set echo off


I am sure you can find resemblance of the control commands with SQL*Plus, a tool you are undoubtedly familiar with. The learning curve should not be tat difficult. Now that you know about the basic commands, let’s dive down to more advanced ones.

General Structure

The CellCLI commands have the following general structure:

<Verb> <Object> <Modifier> <Filter>

  • A verb is what you want or the action, e.g. display something.
  • An object is what you want the action on, e.g. a disk.
  • A modifier (optional) shows how you want to operation to be modified, e.g. all disks, or a specific disk, etc.
  • A filter (optional) is similar to the WHERE predicate of a SQL statement, used with WHERE clause.

There are only a few primary verbs you will use mostly and need to remember. They are:

  • LIST – to show something, e.g. disks, statistics, Resource Manager Plans, etc.
  • CREATE – to create something, e.g. a cell disk, a threshold
  • ALTER – to change something that has been established, e.g. change the size of a disk
  • DROP – to delete something, e.g. a dropping a disk
  • DESCRIBE – to display the various attributes of an object


There are more verbs but these five are the most common ones among the CellCLI commands.

Let’s see how they are used in common operations. In the following sections you will learn how to perform the management operations – View, Creation, Deletion and Modification – of various components of the storage, e.g. Physical Disks, Grid Disks, LUN, etc.

Physical Disk Operations

Storage cell is all about disks. As you learned in first two installments, each storage cell has 12 physical disks. To display the physical disks in this particular cell, you execute the following command:

CellCLI> list physicaldisk
         34:0            E1K5JW                  normal
         34:1            E1L9NC                  normal
… output truncted …
         [4:0:2:0]       5080020000f3e16FMOD2    normal
         [4:0:3:0]       5080020000f3e16FMOD3    normal

You can notice that there is no heading above the output, making it difficult to understand what these values mean. Fortunately we can fix that as we will see later. For now let’s focus on something else: To know more detail about each disk, you can use the detail modifier. It will show details on each disk. Here is a partial output:

CellCLI> list physicaldisk detail
         name:                   34:0
         deviceId:               33
         diskType:               HardDisk
         enclosureDeviceId:      34
         errMediaCount:          0
         errOtherCount:          0
         foreignState:           false
         luns:                   0_0
         makeModel:              "SEAGATE ST360057SSUN600G"
         physicalFirmware:       0805
         physicalInsertTime:     2011-01-21T14:32:35-05:00
         physicalInterface:      sas
         physicalSerial:         XXXXXX
         physicalSize:           558.9109999993816G
         slotNumber:             0
         status:                 normal

         name:                   34:1
         deviceId:               32
         diskType:               HardDisk
         enclosureDeviceId:      34
         errMediaCount:          0
         errOtherCount:          0
         foreignState:           false
         luns:                   0_1
         makeModel:              "SEAGATE ST360057SSUN600G"
         physicalFirmware:       0805
         physicalInsertTime:     2011-01-21T14:32:40-05:00
         physicalInterface:      sas
         physicalSerial:         XXXXXX
         physicalSize:           558.9109999993816G
         slotNumber:             1
         status:                 normal
… output truncated …

The output shows the details of every disk, making it somewhat unreadable. Many times when you encounter issues with specific disks you may want to find the details of a particular disk. For that, use the name of the disk as another modifier. The name is shown in both types of commands, normal and detail.

CellCLI> list physicaldisk 34:0
CellCLI>
list physicaldisk 34:0 detail

Attributes

While the above command is useful for reading, it lacks a few things.

  • You still don’t get a lot of other details. For instance the RPM of the disk is not shown.
  • The results are not in a tabular format, if you want to see more than one disk.

To get these details you can specify named “attributes” in the listing. You can specify these attributes after the modifier attribute. In this example, we have type of the disk (hard disk, or flash disk), model, RPM, Port, and status:

CellCLI> list physicaldisk attributes name, disktype, makemodel, physicalrpm, physicalport, status
         34:0            HardDisk        "SEAGATE ST360057SSUN600G"      normal
         34:1            HardDisk        "SEAGATE ST360057SSUN600G"      normal
… output truncated …
         [4:0:2:0]       FlashDisk       "MARVELL SD88SA02"              normal
         [4:0:3:0]       FlashDisk       "MARVELL SD88SA02"              normal

This is a much better output that shows the details against each disk. This type of output is quite useful in scripts where you may want to pull the details and format them in certain pre-specified manner or parse them for further processing.

Describe

While you may appreciate the above output where using attributes specifically increased the readability of the output, you may also wonder what the valid names of these attributes are. The attribute names vary with the object as well. (For example, diskType is relevant in case of disks only, not in cells.)

Do you need to remember all of them and the context in which they are relevant? Not at all. That’s where a different verb DESCRIBE comes handy; it shows the attributes of an object, similar to what the describe command in SQL*Plus command does for a table to display the columns. Here is how you display the attributes of the physicaldisk object. Remember, unlike SQL*Plus, the command is describe; it cannot be shortened to desc.

CellCLI> describe physicaldisk
        name
        ctrlFirmware
        ctrlHwVersion
        deviceId
        diskType
        enclosureDeviceId
        errCmdTimeoutCount
        errHardReadCount
        errHardWriteCount
        errMediaCount
        errOtherCount
        errSeekCount
        errorCount
        foreignState
        hotPlugCount
        lastFailureReason
        luns
        makeModel
        notPresentSince
        physicalFirmware
        physicalInsertTime
        physicalInterface
        physicalPort
        physicalRPM
        physicalSerial
        physicalSize
        physicalUseType
        sectorRemapCount
        slotNumber
        status

What if you want to display all the attributes of the physical disks; not just a few? You can list all the attribute names explicitly; or – as an easier way – you can just use the option all, as shown below.

CellCLI> list physicaldisk attributes all
         34:0            33              HardDisk        34                      0       0                               false   0_0                     "SEAGATE ST360057SSUN600G"      0805                    2011-01-21T14:32:35-05:00       sas     E1K5JW  558.9109999993816G      0       normal
         34:1            32              HardDisk        34                      0       0                               false   0_1                     "SEAGATE ST360057SSUN600G"      0805                    2011-01-21T14:32:40-05:00       sas     E1L9NC  558.9109999993816G      1       normal
… output truncated …
         [1:0:0:0]       FlashDisk       4_0             "MARVELL SD88SA02"      D20Y    2011-01-21T14:33:32-05:00       sas     5080020000f21a2FMOD0    22.8880615234375G               "PCI Slot: 4; FDOM: 0"  normal
         [1:0:1:0]       FlashDisk       4_1             "MARVELL SD88SA02"      D20Y    2011-01-21T14:33:32-05:00       sas     5080020000f21a2FMOD1    22.8880615234375G     
… output truncated …

This output may not be very useful for reading but if you want to write a script to parse these details and take corrective action, it becomes extremely useful. However, in many cases you may want to see only a certain attributes; not all. In the previous subsection you saw how to select only a few of these attributes.

Checking for Errors

Let’s see a practical use of the named attributes clause. From time to time you will see errors popping up on disk drives, which may have to be replaced. To show the error related attributes, you may choose a script to select only a few attributes related to errors, as shown below:

CellCLI> list physicaldisk attributes name,disktype,errCmdTimeoutCount,errHardReadCount,errHardWriteCount
         34:0            HardDisk
… output truncated …
         34:11           HardDisk
         [1:0:0:0]       FlashDisk
… output truncated …
         [4:0:3:0]       FlashDisk

There is no error on any of these disks; so you see those fields remaining unpopulated.

Filtering

What if you are interested in only a certain type of disk, or filter on some attribute? You can use the SQL-esque predicate WHERE clause. Here you want to see some attributes for all hard disks.

CellCLI> list physicaldisk attributes name, physicalInterface, physicalInsertTime -
>
where disktype = 'HardDisk'

         34:0    sas     2011-01-21T14:32:35-05:00
         34:1    sas     2011-01-21T14:32:40-05:00
         34:2    sas     2011-01-21T14:32:45-05:00
         34:3    sas     2011-01-21T14:32:50-05:00
         34:4    sas     2011-01-21T14:32:55-05:00
         34:5    sas     2011-01-21T14:33:01-05:00
         34:6    sas     2011-01-21T14:33:06-05:00
         34:7    sas     2011-01-21T14:33:11-05:00
         34:8    sas     2011-01-21T14:33:16-05:00
         34:9    sas     2011-01-21T14:33:21-05:00
         34:10   sas     2011-01-21T14:33:26-05:00
         34:11   sas     2011-01-21T14:33:32-05:00


If you want to change the way the date and time are displayed, you can set the dateformat environmental variable.

CellCLI> set dateformat local
Date format was set: Apr 1, 2011 4:05:54 PM.

CellCLI> list physicaldisk attributes name, physicalInterface, physicalInsertTime -

>
where disktype = 'HardDisk'

         34:0    sas     Jan 21, 2011 2:32:35 PM
         34:1    sas     Jan 21, 2011 2:32:40 PM
… output truncated …

Filtering can also be specified with negation, i.e. !=.

CellCLI> list physicaldisk where diskType='Flashdisk'  
         [1:0:0:0]       5080020000f21a2FMOD0    normal
         [1:0:1:0]       5080020000f21a2FMOD1    normal
… output truncated …

CellCLI> list physicaldisk where diskType!='Flashdisk'
         34:0    E1K5JW  normal
         34:1    E1L9NC  normal
… output truncated …

Modifications

Now that you know how to display the objects, let’s see how you can modify the properties. The properties of the physical disks are not modifiable, except one: the display of service LED. You can turn on or off the service LED of disks 34:0 and 34:1 by issuing the following commands.

CellCLI> alter physicaldisk  34:0,34:1 serviceled on
CellCLI>
alter physicaldisk  34:0,34:1 serviceled off

To turn service LED on all the hard disks of that cell, use the following command:

CellCLI> alter physicaldisk harddisk serviceled on

On all disks (hard disks and flash disks):

CellCLI> alter physicaldisk all serviceled on

Creation

There is no creation operation of physical disks because they come with the Exadata machine.

Deletion

There is no deletion operation of physical disks either because they have to be removed by an engineer.

Managing LUN

From the earlier installments you learned that the physical disks are carved up into LUNs. Let’s see the common LUN management commands. To show the LUNs in this cell, you use the following command:

CellCLI> list lun
         0_0     0_0     normal
         0_1     0_1     normal
         0_2     0_2     normal
… output truncated …

As in the case of physical disks, you can also display the details of these LUNs by the following command:

CellCLI> list lun detail
         name:                   0_0
         cellDisk:               CD_00_cell01
         deviceName:             /dev/sda
         diskType:               HardDisk
         id:                     0_0
         isSystemLun:            TRUE
         lunAutoCreate:          FALSE
         lunSize:                557.861328125G
         lunUID:                 0_0
         physicalDrives:         34:0
         raidLevel:              0
         lunWriteCacheMode:      "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU"
         status:                 normal

         name:                   0_1
         cellDisk:               CD_01_cell01
… output truncated …

If you want the details for one specific LUN, use its name as the modifier.

CellCLI> list lun 0_0 detail

Like the previous cases you can show the LUNs in a tabular format by selecting all the attributes as well:

CellCLI> list lun attributes all
         0_0     CD_00_cell01    /dev/sda        HardDisk        0_0     TRUE    FALSE   557.861328125G          0_0     34:0            0       "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU"   normal
… output truncated …
         0_11    CD_11_cell01    /dev/sdl        HardDisk        0_11    FALSE   FALSE   557.861328125G          0_11    34:11           0       "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU"   normal
         1_0     FD_00_cell01    /dev/sdq        FlashDisk       1_0     FALSE   FALSE   22.8880615234375G       100.0   [2:0:0:0]       normal
… output truncated …
         5_3     FD_15_cell01    /dev/sdx        FlashDisk       5_3     FALSE   FALSE   22.8880615234375G       100.0   [3:0:3:0]       normal

If you want to see the available attributes of a LUN, use the describe command.

CellCLI> describe LUN
        name
        cellDisk
        deviceName
        diskType
        errorCount
        id
        isSystemLun
        lunAutoCreate
        lunSize
        lunUID
        overProvisioning
        physicalDrives
        raidLevel
        lunWriteCacheMode
        status

You can use any of these attributes instead of “all”.

CellCLI> list lun attributes name, cellDisk, raidLevel, status
         0_0     CD_00_prolcel14         0       normal
         0_1     CD_01_prolcel14         0       normal
         0_2     CD_02_prolcel14         0       normal
… output truncated …

Modification

The properties of a LUN are not modifiable, except one. The LUNs are re-enabled after a failure. If one is not automatically re-enabled after the failure is corrected, you can manually re-enable it by the following command:

CEllCLI> alter lun 0_0 reenable

Sometimes the LUN may be already enabled but may not be shown as such. To resync this reality with the system, you may need to enable it forceably. The force modifier allows that.

CEllCLI> alter lun 0_0 reenable force

Deletion

There is no delete operation for LUNs in the CellCLI.

Creation

Similarly you don’t create LUNs in CellCLI so there is no create lun command.

Managing Cells

From the previous installments you learned that storage cells, also called simply cells, are where the disks are located and they are presented to the database nodes. A full rack of Exadata contains fourteen such cells. To display the information on cells, you give the list cell command.

CellCLI> list cell
         Cell01       online

The output does not say much except the name of the cell and if it is online. To display more information, use the detail modifier.

CellCLI> list cell detail
         name:                   cell01
         bmcType:                IPMI
         cellVersion:            OSS_11.2.0.3.0_LINUX.X64_101206.2
         cpuCount:               24
         fanCount:               12/12
         fanStatus:              normal
         id:                     XXXXXXXXXX
         interconnectCount:      3
         interconnect1:          bondib0
         iormBoost:              0.0
         ipaddress1:             172.32.128.9/24
         kernelVersion:          2.6.18-194.3.1.0.3.el5
         locatorLEDStatus:       off
         makeModel:              SUN MICROSYSTEMS SUN FIRE X4270 M2 SERVER SAS
         metricHistoryDays:      7
         notificationMethod:     mail
         notificationPolicy:     critical,warning,clear
         offloadEfficiency:      372.1M
         powerCount:             2/2
         powerStatus:            normal
         smtpFrom:               "Proligence Test Database Machine"
         smtpFromAddr:           oncall.dba@proligence.com
         smtpPort:               25
         smtpPwd:                ******
         smtpServer:             standardrelay.proligence.com
         smtpToAddr:             oncall.dba@proligence.com,oncall.unix@proligence.com,oncall.storage@proligence.com
         smtpUser:              
         smtpUseSSL:             FALSE
         status:                 online
         temperatureReading:     24.0
         temperatureStatus:      normal
         upTime:                 32 days, 4:44
         cellsrvStatus:          running
         msStatus:               running
         rsStatus:               running

You will understand the meaning of all these attributes as you explore the cell modifications later in this section.To display the output in a tabular format in one line, as you saw in the case of the physical disks, use the attribute modifier:

CellCLI> list cell attributes all
         cell01  IPMI    OSS_11.2.0.3.0_LINUX.X64_101206.2       24      12/12   normal  XXXXXXXX      3       bondib0         0.0     172.32.128.9/24         2.6.18-194.3.1.0.3.el5  off     SUN MICROSYSTEMS SUN FIRE X4270 M2 SERVER SAS   7       mail    critical,warning,clear  372.1M  2/2     normal  "Proligence Test Database Machine"   oncall.dba@proligence.com  25      ******  standardrelay.proligence.com        oncall.dba@proligence.com,oncall.unix@proligence.com,oncall.storage@proligence.com                FALSE   online  25.0    normal  32 days, 21:52  running         running         running

The output comes as one line containing all the attributes. Of course, you can use specific attributes as you did in other cases. To see all the available attributes, use the previously explained describe command.

CellCLI> describe cell
        name                    modifiable
        bmcType
        cellNumber              modifiable
        cellVersion
        comment                 modifiable
        cpuCount
        emailFormat             modifiable
        events                  modifiable
        fanCount
        fanStatus
        id
        interconnectCount
        interconnect1           modifiable
        interconnect2           modifiable
        interconnect3           modifiable
        interconnect4           modifiable
        iormBoost
        ipBlock                 modifiable
        ipaddress1              modifiable
        ipaddress2              modifiable
        ipaddress3              modifiable
        ipaddress4              modifiable
        kernelVersion
        locatorLEDStatus
        location                modifiable
        makeModel
        metricCollection        modifiable
        metricHistoryDays       modifiable
        notificationMethod      modifiable
        notificationPolicy      modifiable
        offloadEfficiency
        powerCount
        powerStatus
        realmName               modifiable
        smtpFrom                modifiable
        smtpFromAddr            modifiable
        smtpPort                modifiable
        smtpPwd                 modifiable
        smtpServer              modifiable
        smtpToAddr              modifiable
        smtpUser                modifiable
        smtpUseSSL              modifiable
        snmpSubscriber          modifiable
        status
        temperatureReading
        temperatureStatus
        traceLevel              modifiable
        upTime
        cellsrvStatus
        msStatus
        rsStatus

Modification

The word modifiable above means that specific attribute can be modified by an ALTER verb. It’s not possible to give all the possible combinations in this short article and not all are used often anyway; so, let’s see the most common ones.

A cell runs muliple services, e.g. the Restart Server, the Management Server and Cell Services. To shut down a cell, you can shut down specific services by name or the entire service. To shutdown only one service, e.g. the Restart Server service, execute:

CellCLI> alter cell shutdown services rs

Stopping RS services...
The SHUTDOWN of RS services was successful
.

To restart that particular service, use the restart modifier, shown below:

CellCLI> alter cell restart services rs

Stopping RS services...
CELL-01509: Restart Server (RS) not responding.
Starting the RS services...
Getting the state of RS services...
 Running

At any point you can confirm it by checking the status:

CellCLI> list cell attributes rsStatus
         running

To shutdown the Management Server, execute:

CellCLI> alter cell shutdown services MS

To shutdown the Cell Services, execute:

CellCLI> alter cell shutdown services CELLSRV

What if you want to shutdown the entire cell? The “all” modifier is a shotcut to shutdown all the services:

CellCLI> alter cell shutdown services all

Stopping the RS, CELLSRV, and MS services...
The SHUTDOWN of services was successful.

To restart all the services, execute the following command:

CellCLI> alter cell restart services all

Starting the RS, CELLSRV, and MS services...
Getting the state of RS services...
 running
Starting CELLSRV services...
The STARTUP of CELLSRV services was successful.
Starting MS services...
The STARTUP of MS services was successful.

To turn on or off the LED on the chassis of the cell, execute:

CellCLI> alter cell led on
CellCLI>
alter cell led off

Mail Setup

The cell can communicate its status by sending emails. So configuration of email is vital for the proper monitoring of cells. In this section you will learn how to do that. First, verify the status of the email configuration in the cell.

CellCLI> alter cell validate mail

CELL-02578: An error was detected in the SMTP configuration: CELL-05503: An error was detected during notification. The text of the associated internal error is: Could not connect to SMTP host: standardrelay.proligence.com, port: 25, response: 421.
The notification recipient is oncall.dba@proligence.com.
CELL-05503: An error was detected during notification. The text of the associated internal error is: Could not connect to SMTP host: standardrelay.proligence.com, port: 25, response: 421.
The notification recipient is oncall.unix@proligence.com.
CELL-05503: An error was detected during notification. The text of the associated internal error is: Could not connect to SMTP host: standardrelay.proligence.com, port: 25, response: 421.
The notification recipient is oncall.storage@proligence.com.

Please verify your SMTP configuration.

The above output shows that the mail setup has not been done properly, meaning the cell will not be able to send accurate mails required for monitoring. Let’s configure the mail correctly.

First, let’s configue the address and name of the sender, i.e. the Cell 07 (this cell). We will give it an address of cell07@proligence.com. That address may or may not actually exist in the mail server; it’s not important. When the emails come, this address will be shown as the sender. We will also give a name to the sender, i.e. “Exadata Cell 07”, which helps us indetify the sender in the email.

CellCLI> alter cell smtpfromaddr='cell07@proligence.com'
Cell cell07 successfully altered

CellCLI> alter cell smtpfrom='Exadata Cell 07'
Cell cell07 successfully altered

Then, we configure the address the cell should the emails to. This is typically the on-call DBA or DMA. You can configure more than one address as well.

CellCLI> alter cell smtptoaddr='arup@proligence.com'
Cell cell07 successfully altered

The cell can send he mails as text or html, which can be configured as shown below.

CellCLI> alter cell emailFormat='text'
Cell cell07 successfully altered

CellCLI> alter cell emailFormat='html'
Cell cell07 successfully altered

With all these in place, let’s make sure the email works. You can validate the email setup by executing the following:

CellCLI> alter cell validate mail
Cell cell07 successfully altered


If the setup is correct, you will receive a mail from cell07@proligence.com (The email you configured as the sender). Here is the body of that email.

This test e-mail message from Oracle Cell cell07 indicates successful configuration of your e-mail address and mail server.

You may wonder what type of emails the cell sends out. Here is an email sent by the cell when there is a hardware failure:

 exadata-commands-p3f1

And here is the email on another failure of the cell disk as a result of flashdisk failure.

 exadata-commands-p3f2 

BMC Management

BMC, or Baseboard Management Controller, controls the compoments of the cell. The BMC should be running all the time. You can restart it, if needed by executing the command below. The comamnd also reboots the BMC if it is already running.

CellCLI> alter cell restart bmc

To make sure the BMC sends the alerts to the cell so that they show up as alerts, execute 

CellCLI> alter cell configurebmc
Cell Cell07 BMC configured successfully

The cells can also send SNMP traps. Any monitoring system based on SNMP traps can receive and process these SNMP traps to show the statistics on cells. To validate the SNMP configuration to be use dfor Automatic Service Requests (ASRs), use the following command:

CellCLI> alter cell validate snmp type=ASR
Cell cell01 successfully altered

To enable automatic service request generation by the SNMP, youneed to define a subscriber. Here is an example:

CellCLI> alter cell snmpsubscriber=((host='snmp01.proligence.com,type=ASR'))
Cell cell01 successfully altered

You may want to validate the firmware of the cell at any time by executing

CellCLI> alter cell validate configuration                   
Cell cell07 successfully altered

Finally, if you want to reset the cell to its factory settings, use:

CellCLI> drop cell

If you have defined griddisks in this cell, you have to drop them first. Otherwise the following forces them to be dropped.

CellCLI> drop cell force

Remember, the drop cell command removes the sell related properties of the server; it does not actually remove the physical server.

Creation

The cells are created in the beginning of the project, and is usually done via the automatic installation script. Later, you will not likely use this command but it’s shown here briefly for the sake of completeness. The command is create cell. Here is the help on the command:

CellCLI> help create cell

  Usage: CREATE CELL [<cellname>] [realmname=<realmvalue>,]
                [interconnect1=<ethvalue>,] [interconnect2=<ethvalue>,]
                [interconnect3=<ethvalue>,] [interconnect4=<ethvalue>,]
              ( ( [ipaddress1=<ipvalue>,] [ipaddress2=<ipvalue>,]
                  [ipaddress3=<ipvalue>,] [ipaddress4=<ipvalue>,] )
              | ( [ipblock=<ipblkvalue>, cellnumber=<numvalue>] ) )
  Purpose: Configures the cell network and starts services.

  Arguments:
    <cellname>: Name to be assigned to the cell.
                Uses hostname if nothing is specified
    <realmvalue>: Name of the realm this cell belongs to.
    <ethvalue>: Value of the eth interconnect assigned to this network interconnect.
    <ipvalue>: Value of the IP address to be assigned to this network interconnect.
    <ipblkvalue>: Value of the IP block to determine IP address for the network.
    <numvalue>: Number of the cell in this ip block.

Managing Cell Disks

Each physical disk in the cell is further exposed as cell disks. To show all the cell disks in this cell, use the list celldisk command as shown below.

CellCLI> list celldisk
         CD_00_prolcel14         normal
         CD_01_prolcel14         normal
         CD_02_prolcel14         normal
… output truncated …

This is a rather succinct output without much useful detail. The detail modifier allows display of more detailed information on the cell disks.

CellCLI> list celldisk detail
         name:                   CD_00_prolcel14
         comment:               
         creationTime:           2011-04-27T15:11:27-04:00
         deviceName:             /dev/sda
         devicePartition:        /dev/sda3
         diskType:               HardDisk
         errorCount:             0
         freeSpace:              0
         id:                     xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
         interleaving:           none
         lun:                    0_0
         raidLevel:              0
         size:                   1832.59375G
         status:                 normal

         name:                   CD_01_prolcel14
… output truncated …

This output shows all the cell disks one after the other, which makes up a rather long and unreadable list. If you would rather want to show the details in a tabular format with one line per cell disk, use the attribute all modifier shown below:

CellCLI> list celldisk attributes all
         CD_00_cell01            Mar 1, 2011 6:20:45 PM          /dev/sda        /dev/sda3       HardDisk        0       0       xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx    none   0_0      0               528.734375G     normal
… output truncated …
         FD_00_cell01            Jan 21, 2011 5:07:32 PM         /dev/sdq        /dev/sdq        FlashDisk       0       0       xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx    none   1_0      22.875G         normal

To know the columns, or rather the attributes shown here, use the describe command to display the attributes of the cell disk.

CellCLI> describe celldisk
        name                    modifiable
        comment                 modifiable
        creationTime
        deviceName
        devicePartition
        diskType
        errorCount
        freeSpace
        freeSpaceMap
        id
        interleaving
        lun
        physicalDisk
        raidLevel
        size
        status

The output columns are displayed in the same order. You can select only a few attributes from this list instead of all. For instance, let’s see only name of the cell disk and the corresponding Linux partition name.

CellCLI> list celldisk attributes name, devicePartition
         CD_00_prolcel14         /dev/sda3
         CD_01_prolcel14         /dev/sdb3
… output truncated …

Similarly you can use the attributes for filtering the output as well. The following shows an example where you wanted to see all cell disks larger than 23GB.

CellCLI> list celldisk attributes name, devicePartition where size>23G
         CD_00_prolcel14         /dev/sda3
         CD_01_prolcel14         /dev/sdb3
         CD_02_prolcel14         /dev/sdc
         CD_03_prolcel14         /dev/sdd
         CD_04_prolcel14         /dev/sde
         CD_05_prolcel14         /dev/sdf
         CD_06_prolcel14         /dev/sdg
         CD_07_prolcel14         /dev/sdh
         CD_08_prolcel14         /dev/sdi
         CD_09_prolcel14         /dev/sdj
         CD_10_prolcel14         /dev/sdk
         CD_11_prolcel14         /dev/sdl

Modification

Only two attributes of the cell disks are changeable: the comment and the name.

Suppose you want to add a comment “Flash Disk” to the cell disk FD_00_cell01. You need to execute:

CellCLI> alter celldisk FD_00_cell01 comment='Flash Disk'   
 
If you want to make the change to comments on all the hard disks, you would issue:

CellCLI> alter celldisk all harddisk comment=’Hard Disk’
CellDisk CD_00_cell01 successfully altered
CellDisk CD_01_cell01 successfully altered
… output truncated …

Similarly if you want to change the comment on all the flash disks:

CellCLI> alter celldisk all flashdisk comment='Flash Disk'

Deletion

This command is rarely used but it may be needed when cell disks fail and you want to drop the cell disks and create them fresh. Here is how to drop the cell disk named CD_00_cell01:

CellCLI> drop celldisk CD_00_cell01

If the cell disk contains grid disks, the drop command will fail. Either drop the grid disks (described in the next section) or use the force option.

CellCLI> drop celldisk CD_00_cell01 force

You can also drop all cell disks of certain types, e.g. hard disks or flash disks.

CellCLI> drop celldisk harddisk
CellCLI>
drop celldisk flashdisk

Or, drop them all:

CellCLI> drop celldisk all

Managing Grid Disks

In the previous installments you learned that grid disks are carved out of the cell disks and the grid disks are presented to the ASM instance as disks, which are eventually used to build ASM diskgroups.

CellCLI> list griddisk
         DBFS_DG_CD_02_prolcel14         active
         DBFS_DG_CD_03_prolcel14         active
         DBFS_DG_CD_04_prolcel14         active
… output truncated …

Or you can get the details of a specific grid disk:

CellCLI> list griddisk DBFS_DG_CD_02_cell01 detail
         name:                   DBFS_DG_CD_02_cell01
         availableTo:           
         cellDisk:               CD_02_cell01
         comment:               
         creationTime:           2011-03-01T18:21:41-05:00
         diskType:               HardDisk
         errorCount:             0
         id:                     xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
         offset:                 528.734375G
         size:                   29.125G
         status:                 active

As with the previous cases, you can also see:

CellCLI> describe griddisk
        name                    modifiable
        availableTo             modifiable
        cellDisk
        comment                 modifiable
        creationTime
        diskType
        errorCount
        id
        offset
        size                    modifiable
        status

You can use these keywords to display specific – not all – attributes of the grid disks. Here is a command to display the name, the cell disks, and the type of the disk:

CellCLI> list griddisk attributes name,cellDisk,diskType      
         DBFS_DG_CD_02_prolcel14         CD_02_prolcel14         HardDisk
         DBFS_DG_CD_03_prolcel14         CD_03_prolcel14         HardDisk
         DBFS_DG_CD_04_prolcel14         CD_04_prolcel14         HardDisk
… output truncated …

You could have also used the following to show all the attributes:

CellCLI> list griddisk attributes all

If you want to show details for a certain type of attribute only, you can use the filter before the attributes modifier. The following shows the command to show all the grid disks of size  476.546875GB:

CellCLI> list griddisk attributes name,cellDisk,status where size=476.546875G 
         PRORECO_CD_00_prolcel14         CD_00_prolcel14         active
         PRORECO_CD_01_prolcel14         CD_01_prolcel14         active
         PRORECO_CD_02_prolcel14         CD_02_prolcel14         active
         PRORECO_CD_03_prolcel14         CD_03_prolcel14         active
… output truncated …

ASM Status

The describe command does not show two important attributes:

  • ASMModeStatus – whether a current ASM diskgroup is using this griddisk. A value of ONLINE indicates this grid disk is being used.
  • ASMDeactivationOutcome – recall that grid disks can be deactivated, which is effectively taking them offline. Since ASM mirroring ensures that the data is located on another disk, making this disk offline does not lose data. However, if the mirror is offline, or is not present, then making this grid disk offline will result in loss of data. This attribute shows whether the grid disk can be deactivated without loss of data. A value of “Yes” indicates you can deactivate this grid disk without data loss.

    CellCLI> list griddisk attributes name, ASMDeactivationOutcome, ASMModeStatus
             DBFS_DG_CD_02_cell01    Yes     ONLINE
             DBFS_DG_CD_03_cell01    Yes     ONLINE
             DBFS_DG_CD_04_cell01    Yes     ONLINE
    … output truncated …


Modification

Only name, comment, Available To, status and size are modifiable. Let’s see how to change the comment for a specific grid disk:

CellCLI> alter griddisk PRORECO_CD_11_cell01 comment='Used for Reco';        
GridDisk PRORECO_CD_11_cell01 successfully altered

You can change the comment for all the grid disks of a cetain time, e.g. all hard disks:

CellCLI> alter griddisk all harddisk comment='Hard Disk';
GridDisk DBFS_DG_CD_02_cell01 successfully altered
GridDisk DBFS_DG_CD_03_cell01 successfully altered
… output truncated …

Making a grid disk inactive effectively offlines its associated ASM disk.

CellCLI> alter griddisk PRORECO_CD_11_cell01 inactive
GridDisk PRORECO_CD_11_cell01 successfully altered

In the process of doing so, you can use force, which forces the ASM disk to become offline even there is data on that disk.

CellCLI> alter griddisk PRORECO_CD_11_cell01 inactive force

The command will wait until the ASM disk becomes offline. If you want the prompt to come back immediately without waiting, you can use nowait clause.

CellCLI> alter griddisk PRORECO_CD_11_cell01 inactive nowait

You can make it active again:

CellCLI> alter griddisk PRORECO_CD_11_cell01 active  
GridDisk PRORECO_CD_11_cell01 successfully altered

Deletion

You rarely have to drop grid disks but some situations may warrant it (if you want to swap disks, for example.) You can do that by dropping the old one and creating a new one. Here is how you drop a specific named grid disk.

CellCLI> drop griddisk DBFS_DG_CD_02_prolcel14

Sometimes you may want to drop a number of grid disks. You can drop all grid disks with a specific prefix in their name, e.g. DBFS.

CellCLI> drop griddisk prefix=DBFS

Or, you can drop all the grid disks from this cell:

CellCLI> drop griddisk all

Sometimes you may want to drop all the grid disks of only a specific type – hard disks or flash disks. You will then execute one of the following:

CellCLI> drop griddisk flashdisk

CellCLI> drop griddisk harddisk

If the disk is active, it won’t be dropped. Here is an example:

CellCLI> drop griddisk PRORECO_CD_11_cell01;

CELL-02549: Grid disk is in use and FORCE is not specified for the operation.

In that case, you can force drop the disk using the force modifier.

CellCLI> drop griddisk PRORECO_CD_11_cell01 force
GridDisk PRORECO_CD_11_cell01 successfully dropped

Creation

Rarely will you need to create grid disks but you may need to for the same reason you drop them. The create griddisk command does it. Here is an example showing how to create a grid disk from a specific cell disk.

CellCLI> create griddisk PRORECO_CD_11_cell01 celldisk=CD_11_cell01
GridDisk PRORECO_CD_11_cell01 successfully created

If you want to create a grid disk in each cell disk, you can create them individually, or you can use a shortcut:

CellCLI> create griddisk all prefix PRORECO

This will create a grid disk in the naming convention <Prefix>_<Cell Disk Name>, which is usually the name of the database. But you can also use the type of the disk – hard or flash as a prefix instead of the database name.

CellCLI> create griddisk all flashdisk prefix FLASH

You can optionally specify a size, which can be less than or equal to that of the celldisk.

CellCLI> create griddisk PRORECO_CD_11_cell01 celldisk=CD_11_cell01 size=100M

You may want to specify a smaller size to keep less data on the disks.

Managing Flash Disks

As you recall, each storage cell comes with several flash disk cards and they are presented to the cells as cell disks. Each cell has only one flash cache, which is made up of the various flash disks as the cell disks. To see the flash cache, use:

CellCLI> list flashcache
         cell01_FLASHCACHE       normal

Of course the pithy output does not tell us much. The detail modifier takes care of that by displaying a lot more:

CellCLI> list flashcache detail
         name:                   cell01_FLASHCACHE
         cellDisk:               FD_13_cell01,FD_00_cell01,FD_10_cell01,FD_02_cell01,FD_06_cell01,FD_12_cell01,FD_05_cell01,FD_08_cell01,FD_15_cell01,FD_14_cell01,FD_07_cell01,FD_04_cell01,
FD_03_cell01,FD_11_cell01,FD_09_cell01,FD_01_cell01
         creationTime:           2011-01-21T17:07:43-05:00
         degradedCelldisks:     
         effectiveCacheSize:     365.25G
         id:                     xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
         size:                   365.25G
         status:                 normal

The output shows various important data on the flash cache configured for the cell:

  1. The flash disks configured as cell disks, e.g. FD_00_cell01, FD_01_cell01, etc. You can get information on these cell disks by list celldisk command, as shown previously. Here is another example:

    CellCLI> list celldisk FD_00_cell01 detail
             name:                   FD_00_cell01
             comment:                "Flash Disk"
             creationTime:           2011-01-21T17:07:32-05:00
             deviceName:             /dev/sdq
             devicePartition:        /dev/sdq
             diskType:               FlashDisk
             errorCount:             0
             freeSpace:              0
             id:                     xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
             interleaving:           none
             lun:                    1_0
             size:                   22.875G
             status:                 normal
  2. The total size is 365.25GB.
  3. It’s available.

You can also get the output in a single line:

CellCLI> list flashcache attributes all
         cell01_FLASHCACHE       FD_13_cell01,FD_00_cell01,FD_10_cell01,FD_02_cell01,FD_06_cell01,FD_12_cell01,FD_05_cell01,FD_08_cell01,FD_15_cell01,FD_14_cell01,FD_07_cell01,
FD_04_cell01,FD_03_cell01,FD_11_cell01,FD_09_cell01,FD_01_cell01         2011-01-21T17:07:43-05:00               365.25G         686510aa-1f06-4ee7-ab84-743db1ae01d8    365.25G normal

Like the previous cases, you can see what attributes are available for the flashcache.

CellCLI> describe flashcache
        name
        cellDisk
        creationTime
        degradedCelldisks
        effectiveCacheSize
        id
        size
        status

From these you can list only a subset of the attributes:

CellCLI> list flashcache attributes degradedCelldisks

Modification

There is no modification operation for flash cache.

Creation

You create the flash cache using the create flashcache command. You will at least need to specify the cell disks created on the flash disks.

CellCLI> create flashcache celldisk='FD_13_cell01,FD_00_cell01,FD_10_cell01,FD_02_cell01,FD_06_cell01,FD_12_cell01,FD_05_cell01,FD_08_cell01,FD_15_cell01,FD_14_cell01,FD_07_cell01,
FD_04_cell01,FD_03_cell01,FD_11_cell01,FD_09_cell01,FD_01_cell01'

If you want to use all the flash-based cell disks, you can use:

CellCLI> create flashcache all
Flash cache cell01_FLASHCACHE successfully created

To specify a certain size:

CellCLI> create flashcache all size=365.25G

If you want to use only a few cell disks as flash cache and the rest as a grid disks to be ultimately used by ASM diskgroups, you can use only a handful of the cell disks.

CellCLI> create flashcache celldisk='FD_00_cell01'
Flash cache cell01_FLASHCACHE successfully created

The rest of the flash disks can be used as grid disks and then used for ASM disk groups. Note how the flash cache size is reduced now.

CellCLI> list flashcache detail
         name:                   cell01_FLASHCACHE
         cellDisk:               FD_00_cell01
         creationTime:           2011-05-15T15:03:44-04:00
         degradedCelldisks:     
         effectiveCacheSize:     22.828125G
         id:                     xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
         size:                   22.828125G
         status:                 normal

Deletion

You generally don’t need to drop the flash cache, except under one circumstance: when you want to use part (or none) of the flash disks as storage and not as flash cache. To drop the flash cache from the cell, use the following command.

CellCLI> drop flashcache
Flash cache cell01_FLASHCACHE successfully dropped

After that you can either create the flash cache with fewer cell disks or not create it at all.

Flash Cache Content

Remember the purpose of the flash disks? To recap, their purpose is to provide a secondary buffer, known as flash cache. You've already learned how to manage the flash cache, but that brings up a very important point: How do you know if the flash cache has been effective?

A simple check is to look for the contents of the flash cache. To display the contents, use the following command. Beware, in a production system where a lot of data may be cached, this output may be very long. You can always press Control-C to break the output.

CellCLI> list flashcachecontent
         cachedKeepSize:         0
         cachedSize:             81920
         dbID:                   374480170
         dbUniqueName:           PROPRD
         hitCount:               5
         missCount:              0
         objectNumber:           131483
         tableSpaceNumber:       729

         cachedKeepSize:         0
         cachedSize:             65536
         dbID:                   374480170
         dbUniqueName:           PROPRD
         hitCount:               5
         missCount:              0
         objectNumber:           131484
         tableSpaceNumber:       729
… output truncated …

The output shows all the contents of the flash cache. In the previous example, it shows some important data:

  • dbID and dbUniqueName – DBID and name of the database of whose contents are in the flash cache. Remember, the DBM may contain more than one database so you should know how much is from which database.
  • objectNumber – the DATA_OBJECT_ID (not OBJECT_ID) value from DBA_OBJECTS view for the database object which is in the cache.
  • tableSpaceNumber – the tablespace where the object is stored in the database.


Other data points are self explanatory. If you want to see what all attributes of the flash cache contents are visible, you can use the attribute command shown below.

CellCLI> describe flashcachecontent
        cachedKeepSize
        cachedSize
        dbID
        dbUniqueName
        hitCount
        hoursToExpiration
        missCount
        objectNumber
        tableSpaceNumber

Let’s see with an example table called MONTEST1. First we need to know its data_object_id, which is the same as object_id in most cases, except where the segments may be different, e.g. in case of partitioned tables, materialized views and indexes.

SQL> select object_id, data_object_id from user_objects
  2 
where object_name = 'MONTEST1';

 OBJECT_ID DATA_OBJECT_ID
---------- --------------
    211409         211409

In this case the data_object_id and object_id are the same, because this is an non-partitioned table. Now, let’s see how many fragments of this table are in the flash cache:

CellCLI> list flashcachecontent where objectnumber=211409 detail
         cachedKeepSize:         0
         cachedSize:             1048576
         dbID:                   374480170
         dbUniqueName:           PROPRD
         hitCount:               48
         missCount:              2237
         objectNumber:           211409
         tableSpaceNumber:       846
… output truncated …

The output shows the chunks of the data from this segment in the flash cache. You could also use the same technique to find the objects of a single tablespace in the cache. You need to know the tablespace number, which is something you can get from the table TS$ in the SYS schema. In the output above, we see that the cache chunk is from the tablespace 846. If you want to check the tablespace name, use the following query:

SQL> select name
  2  from ts$
  3 
where ts# = 846;

NAME
------------------------------
USERS_DATA

Conversely, you can get the tablespace number from the name and use it to check the cache entries from CellCLI.

Creation

There is no creation operation for flash cache content. Data goes in there by the way of normal operations.

Modification

There is no modification operation; the flash cache is managed by the Exadata Storage Server software.

Deletion

There is no deletion operation since the entries are deleted from the cache by the Exadata Storage Server software.

Group Commands – DCLI

So far you have seen all the commands possible through CellCLI, which applies to the cell you are logged into at that time. What if you want to manage something on all the cells?

For instance, suppose you want to list the status of all 14 cells in the Exadata full-rack database machine. CellCLI’s list cell command gives that information right away, but for that cell only. You can log on to all the other 13 cells and issue the command to get the status – a process that is not only time consuming but which also may not work very well for scripting purposes, at least not easily. To address that, a new interface is available, named DCLI, which allows you to run commands on all other cells while logged in to single cell.

Let’s see the original problem – you are logged in to cell01 and you want to check the status of cells 1 through 14. You then execute the dcli operation as shown below:

[celladmin@prolcel01 ~]# dcli -c prolcel01,prolcel02,prolcel03,prolcel04,prolcel05,prolcel06,prolcel07,prolcel08, prolcel09,prolcel10,prolcel11,prolcel12,prolcel13,prolcel14 -l root "cellcli -e list cell"
prolcel01: cell01        online
prolcel02: cell02        online
prolcel03: cell03        online
prolcel04: cell04        online
… output truncated …

Let’s dissect the command.

  • The last part of the command cellcli –e list cell is what we want to run on all the other cells.
  • The -c option specifies the cells where this command cellcli –e list cell should be run
  • The -l option specifies the user the command should be run as. In this case it’s root. The default is celadmin.


The DCLI interface is not a command but rather a Python script that executes the command on the other cells. This remote execution is done by ssh command, therefore the cells should already have ssh equivalency. If you don’t have it, you can use dcli -k  to establish it.

Note the use of all the cell names with the -c option. What is you always select the cell names? The need to provide the cell names every time may not be convenient and is prone to mistakes. Another parameter, -g (group), allows definition of a group of cells that can be addressed as a whole. First create a file called all_cells with the hostnames of the cells as shown below:

[celladmin@prolcel01 ~]# cat all_cells
prolcel01
prolcel02
prolcel03
prolcel04
prolcel05
prolcel06
prolcel07
prolcel08
prolcel09
prolcel10
prolcel11
prolcel12
prolcel13
prolcel14

With this file in place, you can pass this file to the group (-g) parameter of DCLI instead of the cell names:

[celladmin@prolcel01 ~]# dcli -g all_cells -l root "cellcli -e list cell"

If you want to see which cells are used as targets of the execution, you can use the -t option, for “target”. The hostnames of the cells where the commands from DCLI will be executed will be displayed. Suppose you made a mistake by not placing all the cells in that all_cells file but rather only 8, this command will show it clearly.

[celladmin@prolcel01 ~]# dcli -g all_cells -t
Target cells: ['prolcel01', 'prolcel02', 'prolcel03', 'prolcel04', 'prolcel05', 'prolcel06', 'prolcel07', 'prolcel08']

Only cells up to #8 were displayed.

But DCLI is not just to execute CellCLI commands. It’s a remote execution tool. Any command that can be executed from the command line can be given to the DCLI interface to be executed remotely.

One such command is vmstat. Suppose you want to get a quick pulse on all the cells by executing vmstat 2 2, you can pass it to the DCLI interface, as shown below:

[celladmin@prolcel01 ~]# dcli -l root -g all_cells vmstat 2 2
prolcel01: procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
prolcel01: r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
prolcel01: 1  0      0 9077396 381232 1189992    0    0    13    12    0    0  1  0 99  0  0
prolcel01: 0  0      0 9121764 381232 1190032    0    0   260   564 1143 25691  3  1 96  0  0
prolcel02: procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
prolcel02: r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
prolcel02: 1  0      0 9418412 350612 970424    0    0     7    10    0    0  0  0 99  0  0
prolcel02: 1  0      0 9417852 350612 970424    0    0     0    28 1047 23568  3  1 96  0  0
prolcel03: procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
prolcel03: r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
prolcel03: 1  0      0 9312432 354948 1049780    0    0     7    12    0    0  0  0 99  0  0
prolcel03: 0  0      0 9313108 354948 1049892    0    0     0    60 1040 19046  0  0 100  0  0
prolcel04: procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
prolcel04: r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
prolcel04: 0  0      0 9275392 360644 1080236    0    0    21    13    0    0  0  0 99  0  0
prolcel04: 0  0      0 9275788 360644 1080240    0    0   280    68 1093 17136  0  0 100  0  0
prolcel05: procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
prolcel05: r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
prolcel05: 1  0      0 9200652 361064 1058544    0    0     6    12    0    0  0  0 99  0  0
prolcel05: 1  0      0 9200840 361064 1058544    0    0     0    52 1036 18000  0  4 96  0  0
prolcel06: procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
prolcel06: r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
prolcel06: 0  0      0 9280912 354832 1057432    0    0     8    11    0    0  0  0 99  0  0
prolcel06: 0  0      0 9281388 354832 1057440    0    0    32    24 1040 18619  0  0 100  0  0
prolcel07: procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
prolcel07: r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
prolcel07: 0  0      0 9795664 287140 877948    0    0     1     8    0    0  0  0 99  0  0
prolcel07: 0  0      0 9795136 287140 878004    0    0     0    36 1032 21026  0  1 99  0  0
prolcel08: procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
prolcel08: r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
prolcel08: 0  0      0 9295716 320564 1043408    0    0     6    10    0    0  0  0 99  0  0
prolcel08: 0  0      0 9295740 320564 1043412    0    0     0     4 1014 21830  0  0 100  0  0

The output is useful in many ways. Not only it saved you the need of typing the command on all the cells, it displayed the output on the same page allowing you to compare the results across the cells. For instance, you may notice that the values for blocks out (bo) and blocks in (bi) in the output are not similar across all the cells. They are more in case cell01 and cell04, indicating higher IO activity in those cells.

Here is another example. Suppose you want to make sure that OSWatcher is running on all the cells. You can use the following command to check that.

[celladmin@prolcel01 ~]# dcli -l root -g all_cells ps -aef|grep OSWatcher
prolcel01: root       557 19331  0 Feb28 ?        00:15:08 /bin/ksh ./OSWatcherFM.sh 168
prolcel01: root      5886 24280  0 21:14 pts/0    00:00:00 grep OSWatcher
prolcel01: root     19331     1  0 Feb28 ?        01:08:33 /bin/ksh ./OSWatcher.sh 15 168 bzip2
prolcel02: root       554 17501  0 Feb28 ?        00:14:47 /bin/ksh ./OSWatcherFM.sh 168

To complete the discussion on remote execution, please note that it can be executed on any of cells, where the current cell may be excluded. For instance, you are on cell01 and you want to check the status of a specific physical disk on cell08, you don’t need to logon to that cell to use the command. Right from cell0, you can use the dcli command with the -c option to execute on cell08, as shown below:

[celladmin@prolcel01 ~]# dcli -l root -c prolcel08 cellcli -e "list physicaldisk  20:8 detail"
prolcel08: name:                 20:8
prolcel08: deviceId:             11
prolcel08: diskType:             HardDisk
prolcel08: enclosureDeviceId:    20
prolcel08: errMediaCount:        0
prolcel08: errOtherCount:        1
prolcel08: foreignState:         false
prolcel08: luns:                 0_8
prolcel08: makeModel:            "SEAGATE ST360057SSUN600G"
prolcel08: physicalFirmware:     0805
prolcel08: physicalInsertTime:   2011-01-21T14:23:57-05:00
prolcel08: physicalInterface:    sas
prolcel08: physicalSerial:       XXXXXX
prolcel08: physicalSize:         558.9109999993816G
prolcel08: slotNumber:           8
prolcel08: status:               normal

Modification

Apart from checking values, DCLI is also useful to set values across a lot of cells at a time. For instance, suppose your mail SMTP server is changed and you need to set up a different value of smtServer for the cells. The alter cell smtpServer=’…’ command in CellCLI does that; but it needs to be executed on all the cells. Instead, we can use DCLI to execute them across all the cells specified as a group in the file all_cells, as shown below.

[celladmin@prolcel01 ~]# dcli -l celladmin -g all_cells cellcli -e "alter cell smtpServer=\'smtp.proligence.com\'"
prolcel01: Cell cell01 successfully altered
prolcel02: Cell cell02 successfully altered
prolcel03: Cell cell03 successfully altered
prolcel04: Cell cell04 successfully altered
prolcel05: Cell cell05 successfully altered
prolcel06: Cell cell06 successfully altered
prolcel07: Cell cell07 successfully altered
prolcel08: Cell cell08 successfully altered

Use the same technique to update the other SMTP parameters of the cell as well:

[celladmin@prolcel01 ~]# dcli -l root -g all_cells cellcli -e "alter cell smtpToAddr=\'arup@proligence.com\'"

Check up all settings at one place:

[celladmin@prolcel01 ~]# dcli -l root -g all_cells cellcli -e "list cell attributes name,smtpServer,smtpToAddr,smtpFrom,smtpFromAddr,smtpPort"
prolcel01: cell01        STCEXCPMB02.xxxx.xxxx   arup@proligence.com   "Proligence Test Database Machine"    oncall.dba@proligence.com         25
prolcel02: cell02        STCEXCPMB02.xxxx.xxxx   arup@proligence.com   "Proligence Test Database Machine"    oncall.dba@proligence.com         25
prolcel03: cell03        STCEXCPMB02.xxxx.xxxx   arup@proligence.com   "Proligence Test Database Machine"    oncall.dba@proligence.com         25
prolcel04: cell04        STCEXCPMB02.xxxx.xxxx   arup@proligence.com   "Proligence Test Database Machine"    oncall.dba@proligence.com         25
prolcel05: cell05        STCEXCPMB02.xxxx.xxxx   arup@proligence.com   "Proligence Test Database Machine"    oncall.dba@proligence.com         25
prolcel06: cell06        STCEXCPMB02.xxxx.xxxx   arup@proligence.com   "Proligence Test Database Machine"    oncall.dba@proligence.com         25
prolcel07: cell07        STCEXCPMB02.xxxx.xxxx   arup@proligence.com   "Exadata Cell 07"       cell07@proligence.com       25

After everything is set up properly, you may want to make sure the email set up is correct by running the alter cell validate command, across all the cells as shown below.

[celladmin@prolcel01 ~]# dcli -l root -g all_cells cellcli -e "alter cell validate mail"

If you get the email from these cells, then the emails have been configured properly.

Abbreviated Output

You might see that the output from DCLI is not rather large since it shows the output from all the cells. It might be difficult to read, especially for scripting purposes. To address that problem, DCLI has another parameter -n, which simply shows OK after a successful execution and shows the cells where it was successfully executed. For instance, you can execute the email validation shown earlier with the -n option:

[celladmin@prolcel01 ~]# dcli -l root -g all_cells -n cellcli -e "alter cell validate mail"
OK: ['prolcel01', 'prolcel02', 'prolcel03', 'prolcel04', 'prolcel05', 'prolcel06', 'prolcel07', 'prolcel08']

Regular Expressions

Sometimes you want to check something quickly but want to see only exceptions; not all output. For instance, suppose you want to find out which grid disks are inactive. Following command will show the status of the grid disks.

[celladmin@prolcel01 ~]# dcli -l root -g all_cells cellcli -e "list griddisk"

But it will list all the disks – active or inactive. With hundreds of grid disks, the output may be too overwhelming and less useful. Instead, you may want to see those ones that are inactive. The -r option allows you to enter a regular expression that will not be matched. In this case, we want to see the strings which are not “active”. We can write a command like the following.

[celladmin@prolcel01 ~]# dcli -r '.* active' -l root -g all_cells cellcli -e "list griddisk"
.* active: ['prolcel01', 'prolcel02', 'prolcel03', 'prolcel04', 'prolcel05', 'prolcel06', 'prolcel07', 'prolcel08']
prolcel01: PRORECO_CD_11_cell01  inactive

It clearly showed that the grid disk PRORECO_CD_11_cell01 in cell prolcel01 is inactive. It’s a quick check on abnormal conditions.

Command Scripts

You can place cellcli commands inside a script and call that script. Suppose you want to develop an error check script for CellCLI called err.dcl (the extension .dcl is not mandatory; it’s just convenient). Here is how the script looks like:

[celladmin@prolcel01 ~]# cat err.dcl
cellcli -e list physicaldisk attributes errCmdTimeoutCount,errHardReadCount,errHardWriteCount,errMediaCount,errOtherCount,errSeekCount,errorCount where disktype='HardDisk'

You saw earlier how to execute this script from CellCLI. You can execute this script from DCLI as well, using the -x option, as shown below:

[celladmin@prolcel01 ~]# dcli -l root -g all_cells -x err.dcl
prolcel01: 34:0          0       0
prolcel01: 34:1          0       0
prolcel01: 34:2          0       0
prolcel01: 34:3          0       0
… output truncated …
prolcel08: 20:6          0       0
prolcel08: 20:7          0       0
prolcel08: 20:8          0       1
prolcel08: 20:9          0       0
prolcel08: 20:10         0       0
prolcel08: 20:11         0       0

Note how the error is shown in prolcel08 but it is buried under tons of other output, making it less readable and prone to overlook. Here is where the -r option (described in the last subsection) comes handy, to remove the lines containing “active”, as shown below:

[celladmin@prolcel01 ~]# dcli -l root -g all_cells -r '.* 0' -x err.dcl
.* 0: ['prolcel01', 'prolcel02', 'prolcel03', 'prolcel04', 'prolcel05', 'prolcel06', 'prolcel07', 'prolcel08']
prolcel08: 0     1

This resultant output is short and simple.

Summary

In summary, the dcli command allows you to execute commands (including cellcli) on other cell servers. The general structure of the command is

# dcli <Option> <Command>

Where commands could be CellCLI commands, e.g. cellcli –e list cell, or Linux commands like vmstat 2 2.

Here are the most commonly used options and what they do.

Option

Value Expected after the Option

Example Usage

What It Does

-c

Names of the cell(s)

-c dwhcel01,dwhcel02

Executes the commands only on those cells.

-g

The file name where the cells are listed

-g all_cells

Executes the command on the cells mentioned in the file all_cells.

-l

The OS user that will execute the command

-l root

The default user is celladmin; but you can use any other user to use for remote ssh execution. Make sure the user has ssh equivalency across all the cells where you run this command

-n

Not Applicable

-n

This shows an abbreviated output instead of a long output from each command execution

-r

A regular expression

-r [*. active]

Suppresses the output that matches the regular expression

-t

Not Applicable

-t

Displays the cells where the command will run

-x

A script with executable commands

-x list.dcl

The script will be executed on the target cells

-k

Names of the cell(s)

-k dwhcel01,dwhcel02

Establishes the ssh user equivalency

-f

A filename

-f run.dcl

Copies the files to the other cells but does not execute them. It's useful for copying files and executing them later.


Conclusion

The commands CellCLI and DCLI are used to manage the Exadata Storage Server, and most likely are only commands that you are not familiar with as a DBA. In this installment you saw the most useful commands and their example usage. In the next and final installment, you will learn how to use some of these and Oracle Enterprise Manager Grid Control to check on the various components and get detailed reports.

> Back to Series TOC


Arup Nanda (arup@proligence.com) has been an Oracle DBA for more than 14 years, handling all aspects of database administration, from performance tuning to security and disaster recovery. He is an Oracle ACE Director and was Oracle Magazine's DBA of the Year in 2003.