Oracle Validated Configuration

Configuration Summary:	Oracle Database 10g Release 2 Single Instance and Oracle Real Application Clusters (RAC) on Red Hat Enterprise Linux 4 AS/ES x86-64 using Automatic Storage Management (ASM)
Publication Date:	August 14, 2006
Version:	1.0
Server Platform:	Dell Power Edge 2900
Storage:	Dell | EMC Clarion CX3-20
Oracle Software:	Oracle Database 10g Release 2 (10.2.0.2) for Linux x86-64
Linux Distribution:	RHEL 4 AS/ES Update 3 Intel EM64T

Server and Storage Platform Details

     2 x PE2900   Processors (in each node): 2 X Dual Core Intel Xeon 5100 
     Memory (in Each Node): 8 GB RAM
     Hard Drives (Each Node): 2 X 73 G 15K RPM U320 SCSI  (RAID 1)
     Interconnect: Dual Onboard Broadcom NICs
     HBAs (in each node): 2x Qlogic 2460 4G Fibre Channel HBA card 

Additional Server Model covered by this configuration : PE1950, PE2950

Storage: 1 X Dell | EMC CX3-80 running R22 Software
    MultiPathing: PowerPath version 4.5.1 for Linux 2.6 kernel (available at www.emc.com)
    FC HDDs: 8 X 73G 15K rpm Drives (RAID 1/0)
    FC Switch:  McData Sphereon 4700

Other storage covered by this document : CX3-20, CX3-40

Additional FC Switch covered by this document: McData Sphereon 4400

Linux Distribution Details

  RHEL 4 AS/ES Update 3 x86-64 (default install)
    kernel-smp-2.6.9-34.ELsmp.x86_64.rpm  (from Update 3)

  Additional packages needed from distribution
    binutils-2.15.92.0.2-18.x86_64.rpm
    compat-db-4.1.25-9.x86_64.rpm
    compat-libstdc++-33-3.2.3-47.3.x86_64.rpm
    control-center-2.8.0-12.rhel4.2.x86_64.rpm
    gcc-3.4.5-2.x86_64.rpm
    gcc-c++-3.4.5-2.x86_64.rpm
    glibc-2.3.4-2.19.i686.rpm
    glibc-2.3.4-2.19.x86_64.rpm
    glibc-common-2.3.4-2.19.x86_64.rpm
    glibc-devel-2.3.4-2.19.i386.rpm
    glibc-devel-2.3.4-2.19.x86_64.rpm
    glibc-headers-2.3.4-2.19.x86_64.rpm
    glibc-kernheaders-2.3.4-2.19.x86_64.rpm
    gnome-libs-1.4.1.2.90-44.1.x86_64.rpm
    libgcc-3.4.5-2.i386.rpm
    libstdc++-3.4.5-2.x86_64.rpm
    libstdc++-devel-3.4.5-2.x86_64.rpm
    libaio-0.3.105-2.x86_64.rpm
    make-3.80-5.x86_64.rpm
    pdksh-5.2.14-30.3.x86_64.rpm
    sysstat-5.0.5-7.rhel4.x86_64.rpm
    xorg-x11-deprecated-libs-6.8.2-1.EL.13.25.1.i386.rpm
    xscreensaver-4.18-5.rhel4.10.x86_64.rpm

  Additional packages from Oracle

  Required for DB on ASM
    oracleasm-2.6.9-34.ELsmp-2.0.2-1.x86_64.rpm
    oracleasmlib-2.0.2-1.x86_64.rpm
    oracleasm-support-2.0.2-1.x86_64.rpm

  Boot options
    elevator=deadline

  /etc/modprobe.conf:
    options e1000 FlowControl=1  (if using e1000)
    options hangcheck_timer hangcheck_reboot=1

  /etc/sysctl.conf settings
    fs.file-max=327679
    kernel.msgmni=2878
    kernel.msgmax=8192
    kernel.msgmnb=65536
    kernel.sem=250 32000 100 142
    kernel.shmmni=4096
    kernel.shmall=3279547
    # set to a value half the size of physical memory
    kernel.shmmax=3700000000
    kernel.sysrq=1
    #rmem_max can be tuned based on workload to balance performance vs lowmem usage
    net.core.rmem_default=262144
    net.core.rmem_max=2097152
    net.core.wmem_default=262144
    net.core.wmem_max=262144
    fs.aio-max-nr=3145728
    vm.lower_zone_protection=100
    net.ipv4.ip_local_port_range=1024 65000

   /etc/selinux/config:
    SELINUX=disabled

  /etc/security/limits.conf:  
    # depending on size of db, these may need to be larger
    oracle  soft    nofile  131072
    oracle  hard    nofile  131072
    oracle  soft    nproc   131072
    oracle  hard    nproc   131072
    oracle  soft    core    unlimited
    oracle  hard    core    unlimited
    # set memlock greater than or equal to the sga size to allow
    # oracle to use hugepages if configured
    oracle  soft    memlock 50000000
    oracle  hard    memlock 50000000

  Filesystems tested:

    ASM using block devices and ASMLib

  Driver Modules:

  The HBA driver module used (qla2400) is the version supplied with RHEL 4 U3 x86-64
  (version 8.01.02-d4)
  The network driver module used (bnx2) is the version supplied with RHEL 4 U3 x86-64
  updated to version 1.4.36b
  The hangcheck-timer module used is the version supplied with RHEL 4 U3 x86-64
   
  Miscellaneous

  2 gigabit networks configured:  public and interconnect
  The swap partition is the size of physical RAM
 

Oracle Software Details

  Oracle Database 10g Release 2 (10.2.0.2) Single Instance and Oracle Real Application Clusters (RAC) for Linux x86-64
      patch 5071492
      patch 4639236
      patch 4690794
      patch 5036588

Configuration Feedback

    Bug      Summary
    Number

    5041764  CFQ io scheduler can delay/hang io (bz 151368)
               affects:  heavy io access to raw/block devices   
               symptom:  io requests will start to hang or take a long to complete 
                         RAC node could be evicted 
               workaround:  boot w/ elevator=deadline
    5058952  e1000 flow control defaults to none in the 2.6 kernel
               affects:  e1000 network interfaces with heavy traffic
               symptom:  RAC interconnect may get lost blocks
               solution:  load e1000 with FlowControl=1 (Rx)
    5093059  netdump doesn't load with bonding configured (it 85145)
               affects:  bringing up netdump over a bonded interface
               symptom:  /var/log/messages will have an error: 
                           netlog: bond0 doesn't support polling, aborting.
               workaround:  none at the moment
    4593892  repeated warnings in dmesg about lost ticks (it 80129/bz 139376)
               affects:  usually seen on x86-64, both Intel and Opteron based systems
               symptom:  dmesg will have errors like:
                           warning: many lost ticks.
                           Your time source seems to be instable or some driver is hogging interupts
                           rip __do_softirq+0x4d/0xd0 
               workaround:  none at the moment
    5215593  unable to start RAC with db_cache_size > 4gb
               affects:  RAC instances with > 4gb db_cache_size
               symptom:  nodes that started up correctly with 10.2.0.1
                         may fail to startup with ORA-4031 errors in 10.2.0.2
               workaround:  set _ksmg_granule_size=33554432 in the init.ora.
                            Larger sga sizes may need a higher value.
    5215622  libknlopt.a not relinked
               affects:  10.2.0.2 RAC (did not occur in 10.1)
               symptom:  may get ORA-7445 errors during startup
               workaround:  manually relink on each node
    5041394  relink errors after upgrading from 10.1 to 10.2 
               affects:  10.1 -> 10.2 upgrade process
               symptom:  during 10.2 upgrade, relink step will fail
                         with 'undefined reference' errors
               workaround:  after doing 10.2 prereq steps, rename
                 $ORACLE_HOME/lib/stubs to $ORACLE_HOME/lib/stubs.10.1
                 $ORACLE_HOME/lib32/stubs to $ORACLE_HOME/lib32/stubs.10.1
                 then start installing 10.2.0.1         
    5071492  LMS crash with ORA-00600: internal error code, arguments: [kclastf_1], [2], [],
               affects:  10.2.0.2 RAC
               symptom:  the instance could terminate
               solution:  apply patch 5071492
    4639236  ORA-600 [kclcls_5] in RAC instance
               affects:  10.2.0.2 RAC
               symptom:  the instance could terminate 
               solution: apply patch 4639236
    4690794  "ERROR IN KQLMBIVG SEE LCK TRACE FILE" [LT] [LB]  KJUSERCLIENTLOCK 
               affects: 10.2.0.2 RAC
               symptom: Error messages in the alert logs
               solution: apply patch 4690794 
    5036588  LMD0 PROCESS RECEIVED OS SIGNAL #11
               affects: 10.2.0.2 RAC
               symptom: the instance could terminate
               solution: apply patch  5036588
    5021707  CSS should open block device voting disk with o_direct
               affects: 10.2.0.2 RAC when voting or ocr disk is a block device
               symptom: RAC reconfiguration may not occur properly
               solution: use raw devices for ocr and voting disks
    5219517  failing RAC interconnect causes ASM to hang
               affects: 10.2.0.2 RAC using ASM (2-node)
               symptom: ifconfig down of RAC interconnect on node 1 causes
                        surviving node 2 ASM instance to hang.  This may
                        occur during testing to simulate interconnect failure.
               solution: RAC properly handles this failure case when the
                         interconnect is physically disconnected  vs. ifconfig down
                         which is simulating the failure
    5298421  asmlib ORA-15183 ASMLIB initialization error
               affects: 10.2 when using asmlib
               symptom: ORA-15183 could occur while starting an asm instance
               workaround: a reboot is needed
    5017352  warning messages in trace files during certain workloads with async io turned on
               affects: performance
               symptom: Following warning in trace files
               WARNING: ORACLE PROCESS RUNNING OUT OF OS KERNEL I/O RESOURCES
               workaround: none so far
    5333606  rpm throws errors if selinux is disabled at OS installation and is enabled 
             at runtime
               affects:  error at rpm packages installs
               symptom:  during packages installation, errors indicate the scriplet failures
               workaround: do not change selinux mode from disable to enable or visa versa at runtime