Oracle Validated Configuration

Configuration Summary:	Oracle Database 10g Release 2 Single Instance and Oracle Real Application Clusters (RAC) on Red Hat Enterprise Linux 4 AS/ES x86-64 using Automatic Storage Management (ASM)
Publication Date:	July 21, 2006
Version:	1.2
Server Platform:	HP ProLiant DL585 Server series
Storage:	HP StorageWorks MSA 1000
Oracle Software:	Oracle Database 10g Release 2 (10.2.0.2) for Linux x86-64
Linux Distribution:	RHEL 4 AS/ES Update 3 x86-64

Server and Storage Platform Details

  2 x HP ProLiant DL585
    2 x Dual Core AMD Opteron 850 1.8 GHz (4 cores)
    8 GB RAM
    2 x 72.8 GB 15K HD (RAID 1)
    1 x StorageWorks 2Gb, Dual Port, 64-Bit/133 MHz PCI-X-to-Fibre Channel HBA
    2 x Embedded NC7782 Gigabit NIC 

 Settings in the server BIOS adjusted to allow memory/node interleaving 
 to work better with the 'numa=off' boot option

   1 x HP StorageWorks MSA 1000
   28 x 146.8 GB 15K HD (RAID 0+1)
   1 x HP StorageWorks 4/16 SAN Switch

Linux Distribution Details

  RHEL 4 AS/ES Update 3 x86-64(default install)
    kernel-smp-2.6.9-34.ELsmp.x86_64.rpm (from Update 3)


Additional packages needed from distribution

    binutils-2.15.92.0.2-18.x86_64.rpm
    compat-db-4.1.25-9.x86_64.rpm
    compat-libstdc++-33-3.2.3-47.3.x86_64.rpm
    control-center-2.8.0-12.rhel4.2.x86_64.rpm
    gcc-3.4.5-2.x86_64.rpm
    gcc-c++-3.4.5-2.x86_64.rpm
    glibc-2.3.4-2.19.i686.rpm
    glibc-2.3.4-2.19.x86_64.rpm
    glibc-common-2.3.4-2.19.x86_64.rpm
    glibc-devel-2.3.4-2.19.i386.rpm
    glibc-devel-2.3.4-2.19.x86_64.rpm
    glibc-headers-2.3.4-2.19.x86_64.rpm
    glibc-kernheaders-2.3.4-2.19.x86_64.rpm
    gnome-libs-1.4.1.2.90-44.1.x86_64.rpm
    libgcc-3.4.5-2.i386.rpm
    libstdc++-3.4.5-2.x86_64.rpm
    libstdc++-devel-3.4.5-2.x86_64.rpm
    libaio-0.3.105-2.x86_64.rpm
    make-3.80-5.x86_64.rpm
    pdksh-5.2.14-30.3.x86_64.rpm
    sysstat-5.0.5-7.rhel4.x86_64.rpm
    xorg-x11-deprecated-libs-6.8.2-1.EL.13.25.1.i386.rpm
    xscreensaver-4.18-5.rhel4.10.x86_64.rpm

Additional packages from Oracle 

    Required for DB on ASM
 
    oracleasm-2.6.9-34.ELsmp-2.0.2-1.x86_64.rpm
    oracleasmlib-2.0.2-1.x86_64.rpm
    oracleasm-support-2.0.2-1.x86_64.rpm

    Required for DB on OCFS2

    ocfs2-tools-1.2.1-1.x86_64.rpm
    ocfs2console-1.2.1-1.x86_64.rpm
    ocfs2-2.6.9-34.ELsmp-1.2.1-1.x86_64.rpm

Boot options

    elevator=deadline numa=off

/etc/modprobe.conf:

    options e1000 FlowControl=1 (if using e1000)
    options hangcheck_timer hangcheck_reboot=1

/etc/sysctl.conf settings

    fs.file-max=327679
    net.ipv4.ip_local_port_range=1024 65000
    kernel.msgmni=2878
    kernel.msgmax=8192
    kernel.msgmnb=65536
    kernel.sem=250 32000 100 142
    kernel.shmmni=4096
    kernel.shmall=3279547

# set to a value half the size of physical memory

    kernel.shmmax=3700000000
    kernel.sysrq=1

# rmem_max can be tuned based on workload to balance performance vs lowmem usage

    net.core.rmem_default=262144
    net.core.rmem_max=2097152
    net.core.wmem_default=262144
    net.core.wmem_max=262144
    fs.aio-max-nr=3145728
    vm.lower_zone_protection=100

/etc/security/limits.conf:

   # depending on size of db, these may need to be larger

    oracle  soft   nofile  131072
    oracle  hard   nofile  131072
    oracle  soft   nproc   131072
    oracle  hard   nproc   131072

# set memlock greater than or equal to the sga size to allow

# oracle to use hugepages if configured

  oracle  soft   memlock  50000000
  oracle  hard   memlock  50000000

/etc/sysconfig/o2cb

# Note: If the database is on OCFS2, timeouts need to be adjusted.

    Set O2CB_HEARTBEAT_THRESHOLD to a value which takes into consideration
    the underlying storage and the workload to avoid timeouts 
   ( E.g: O2CB_HEARTBEAT_THRESHOLD=28 )

Filesystems tested:

    ext3
    ASM using block devices and Automatic Storage Management Library Driver (asmlib)
    OCFS2

Mount options:

    For datafiles, ocr and voting disk files on OCFS2, use
    _netdev, datavoulume, nointr

Driver Modules:

  The HBA driver module used (qla2300) is the version supplied with RHEL 4 U3 x86-64
  The network driver module used (tg3) is the version supplied with RHEL 4 U3 x86-64
  The hangcheck-timer module used is the version supplied with RHEL 4 U3 x86-64

  The swap partition is the size of physical RAM

Miscellaneous

  2 gigabit networks configured: public and interconnect
 

Oracle Software Details

  Oracle Database 10g Release 2 (10.2.0.2) Single Instance and RAC forAMD64
      patch 5071492
      patch 4639236
      patch 4690794
      patch 5036588

Configuration Feedback
    
    Bug      Summary
    Number

    5041764  CFQ io scheduler can delay/hang io (bz 151368)
               affects:  heavy io access to raw/block devices?? 
               symptom:  io requests will start to hang or take a long to complete 
                         RAC node could be evicted 
               workaround:  boot w/ elevator=deadline

    5058952  e1000 flow control defaults to none in the 2.6 kernel
               affects:  e1000 network interfaces with heavy traffic
               symptom:  RAC interconnect may get lost blocks
               solution: load e1000 with FlowControl=1 (Rx)

    5093059  netdump doesn't load with bonding configured (it 85145)
               affects:  bringing up netdump over a bonded interface
               symptom:  /var/log/messages will have an error: 
                         netlog: bond0 doesn't support polling, aborting.
               workaround:  none at the moment

    4593892  repeated warnings in dmesg about lost ticks (it 80129/bz 139376)
               affects:  usually seen on x86-64, both Intel and Opteron based systems
               symptom:  dmesg will have errors like:
                         warning: many lost ticks.
                         Your time source seems to be instable or some driver is hogging interupts
                         rip __do_softirq+0x4d/0xd0 
               workaround:  none at the moment

    5215593  unable to start RAC with db_cache_size > 4gb
               affects:  RAC instances with > 4gb db_cache_size
               symptom:  nodes that started up correctly with 10.2.0.1
                         may fail to startup with ORA-4031 errors in 10.2.0.2
               workaround:  set _ksmg_granule_size=33554432 in the init.ora.
                            Larger sga sizes may need a higher value.

    5215622  libknlopt.a not relinked
               affects:  10.2.0.2 RAC (did not occur in 10.1)
               symptom:  may get ORA-7445 errors during startup
               workaround:  manually relink on each node

    5041394  relink errors after upgrading from 10.1 to 10.2 
               affects:  10.1 -> 10.2 upgrade process
               symptom:  during 10.2 upgrade, relink step will fail
                         with 'undefined reference' errors
               workaround:  after doing 10.2 prereq steps, rename
                            $ORACLE_HOME/lib/stubs to $ORACLE_HOME/lib/stubs.10.1
                            $ORACLE_HOME/lib32/stubs to $ORACLE_HOME/lib32/stubs.10.1
                            then start installing 10.2.0.1

    5071492  LMS crash with ORA-00600: internal error code, arguments: [kclastf_1], [2], [],
               affects:  10.2.0.2 RAC
               symptom:  the instance could terminate
               solution:  apply patch 5071492

    4639236  ORA-600 [kclcls_5] in RAC instance
               affects:  10.2.0.2 RAC
               symptom:  the instance could terminate 
               solution:  apply patch 4639236

    4690794  "ERROR IN KQLMBIVG SEE LCK TRACE FILE" [LT] [LB] KJUSERCLIENTLOCK 
               affects:  10.2.0.2 RAC
               symptom:  Error messages in the alert logs
               solution:  apply patch 4690794 
 
    5036588  LMD0 PROCESS RECEIVED OS SIGNAL #11
               affects:  10.2.0.2 RAC
               symptom:  the instance could terminate
               solution: apply patch 5036588

    5021707  CSS should open block device voting disk with o_direct
               affects:  10.2.0.2 RAC when voting or ocr disk is a block device
               symptom:  RAC reconfiguration may not occur properly
               solution: use raw devices for ocr and voting disks

    5219517  failing RAC interconnect causes ASM to hang
               affects:  10.2.0.2 RAC using ASM (2-node)
               symptom:  ifconfig down of RAC interconnect on node 1 causes
                         surviving node 2 ASM instance to hang. This may
                         occur during testing to simulate interconnect failure.
               solution: RAC properly handles this failure case when the
                         interconnect is physically disconnected vs. ifconfig down
                         which is simulating the failure

    5298421  asmlib ORA-15183 ASMLIB initialization error
               affects:  10.2 when using asmlib
               symptom:  ORA-15183 could occur while starting an asm instance
               workaround: a reboot is needed

    5333606  rpm throws errors if selinux is disabled at OS installation and is enabled at runtime
               affects:  error at rpm packages installs
               symptom:  during packages installation, errors indicate the scriplet failures
               workaround: do not change selinux mode from disable to enable or visa versa at runtime