Oracle Validated Configuration

Configuration Summary:	Oracle Database 10g Release 2 (Single Instance) and Oracle Real Application Clusters (RAC) on Red Hat Enterprise Linux 4 AS/ES x86-64 using Automatic Storage Management (ASM)
Updated:	August 14, 2006
Version:	1.1
Server Platform:	Dell Power Edge 6850
Storage:	Dell | EMC Clarion CX500
Oracle Software:	Oracle Database 10g Release 2 (10.2.0.2) for Linux x86-64
Linux Distribution:	RHEL 4 AS/ES Update 3 Intel EM64T

Server and Storage Platform Details

     2 X PE6850   Processors (in each node): 2 X Dual Core Intel Xeon  
     Memory (in Each Node): 16 GB RAM  
     Hard Drives (Each Node): 2 X 73 G 15K RPM U320 SCSI  (RAID 1)  
     Interconnect: Dual Onboard Broadcom NICs  
     HBAs (in each node): Dell 2362 dual port Fibre Channel daughter card, 
     powered by Qlogic

Storage: 1 X Dell | EMC CX500 running R19 Software  
    FC HDDs: 16 X 73G 15K rpm Drives (RAID 1/0)  
    FC Switch:  Brocade SilkWorm 4100 

Additional Server Model covered by this configuration : PE6800

Other storage covered by the same family : CX300 and CX700.


Linux Distribution Details

  RHEL 4 AS/ES Update 3 x86-64 (default install)
    kernel-smp-2.6.9-34.ELsmp.x86_64.rpm  (from Update 3)

  Additional packages needed from distribution
  #packages required from the distribution
    binutils-2.15.92.0.2-18.x86_64.rpm
    compat-db-4.1.25-9.x86_64.rpm
    compat-libstdc++-33-3.2.3-47.3.x86_64.rpm
    control-center-2.8.0-12.rhel4.2.x86_64.rpm
    gcc-3.4.5-2.x86_64.rpm
    gcc-c++-3.4.5-2.x86_64.rpm
    glibc-2.3.4-2.19.i686.rpm
    glibc-2.3.4-2.19.x86_64.rpm
    glibc-common-2.3.4-2.19.x86_64.rpm
    glibc-devel-2.3.4-2.19.i386.rpm
    glibc-devel-2.3.4-2.19.x86_64.rpm
    glibc-headers-2.3.4-2.19.x86_64.rpm
    glibc-kernheaders-2.3.4-2.19.x86_64.rpm
    gnome-libs-1.4.1.2.90-44.1.x86_64.rpm
    libgcc-3.4.5-2.i386.rpm
    libstdc++-3.4.5-2.x86_64.rpm
    libstdc++-devel-3.4.5-2.x86_64.rpm
    libaio-0.3.105-2.x86_64.rpm
    make-3.80-5.x86_64.rpm
    pdksh-5.2.14-30.3.x86_64.rpm
    sysstat-5.0.5-7.rhel4.x86_64.rpm
    xorg-x11-deprecated-libs-6.8.2-1.EL.13.25.1.i386.rpm
    xscreensaver-4.18-5.rhel4.10.x86_64.rpm

  Additional packages from Oracle
    oracleasm-2.6.9-34.ELsmp-2.0.2-1.x86_64.rpm
    oracleasmlib-2.0.2-1.x86_64.rpm
    oracleasm-support-2.0.2-1.x86_64.rpm

  Boot options
    elevator=deadline 

  /etc/modprobe.conf:
    options e1000 FlowControl=1  (if using e1000)
    options hangcheck_timer hangcheck_reboot=1

  /etc/sysctl.conf settings
    fs.file-max=327679
    kernel.msgmni=2878
    kernel.msgmax=8192
    kernel.msgmnb=65536
    kernel.sem=250 32000 100 142
    kernel.shmmni=4096
    kernel.shmall=3279547
#   set to a value half the size of physical memory
    kernel.shmmax=3700000000
    kernel.sysrq=1
#   rmem_max can be tuned based on workload to balance performance vs lowmem usage
    net.core.rmem_default=262144
    net.core.rmem_max=2097152
    net.core.wmem_default=262144
    net.core.wmem_max=262144
    fs.aio-max-nr=3145728
    vm.lower_zone_protection=100
    net.ipv4.ip_local_port_range=1024 65000

  /etc/selinux/config:
    SELINUX=disabled

  /etc/security/limits.conf:  
    # depending on size of db, these may need to be larger
    oracle  soft    nofile  131072
    oracle  hard    nofile  131072
    oracle  soft    nproc   131072
    oracle  hard    nproc   131072
    oracle  soft    core    unlimited
    oracle  hard    core    unlimited
    # set to allow oracle to use hugepages
    oracle  soft    memlock 50000000
    oracle  hard    memlock 50000000

  Filesystems tested:
    ext3
    ASM using block devices; asmlib is required
    ASM using ASMLib

  Driver Modules:

  The HBA driver module used (qla2322) is the version supplied with RHEL 4 U3 x86-64
  The network driver module used (tg3) is the version supplied with RHEL 4 U3 x86-64
  The hangcheck-timer module used is the version supplied with RHEL 4 U3 x86-64

  Miscellaneous
  2 gigabit networks configured:  public and interconnect
  The swap partition is the size of physical RAM

  Additional Software
   Tested multipathing solution is EMC PowerPath version 4.5.1

Oracle Software Details

  Oracle Database 10g Release 2 (10.2.0.2) Single Instance and Oracle Real Application Clusters (RAC) for Linux x86-64
      patch 5071492
      patch 4639236
      patch 4690794
      patch 5036588


Configuration Feedback

    Bug      Summary
    Number

    5041764  CFQ io scheduler can delay/hang io (bz 151368)
               affects:  heavy io access to raw/block devices   
               symptom:  io requests will start to hang or take a long to complete 
                         RAC node could be evicted 
               workaround:  boot w/ elevator=deadline
    5058952  e1000 flow control defaults to none in the 2.6 kernel
               affects:  e1000 network interfaces with heavy traffic
               symptom:  RAC interconnect may get lost blocks
               solution:  load e1000 with FlowControl=1 (Rx)
    5093059  netdump doesn't load with bonding configured (it 85145)
               affects:  bringing up netdump over a bonded interface
               symptom:  /var/log/messages will have an error: 
                           netlog: bond0 doesn't support polling, aborting.
               workaround:  none at the moment
    4593892  repeated warnings in dmesg about lost ticks (it 80129/bz 139376)
               affects:  usually seen on x86-64, both Intel and Opteron based systems
               symptom:  dmesg will have errors like:
                           warning: many lost ticks.
                           Your time source seems to be instable or some driver is hogging interupts
                           rip __do_softirq+0x4d/0xd0 
               workaround:  none at the moment
    5215593  unable to start RAC with db_cache_size > 4gb
               affects:  RAC instances with > 4gb db_cache_size
               symptom:  nodes that started up correctly with 10.2.0.1
                         may fail to startup with ORA-4031 errors in 10.2.0.2
               workaround:  set _ksmg_granule_size=33554432 in the init.ora.
                            Larger sga sizes may need a higher value.
    5215622  libknlopt.a not relinked
               affects:  10.2.0.2 RAC (did not occur in 10.1)
               symptom:  may get ORA-7445 errors during startup
               workaround:  manually relink on each node
    5041394  relink errors after upgrading from 10.1 to 10.2 
               affects:  10.1 -> 10.2 upgrade process
               symptom:  during 10.2 upgrade, relink step will fail
                         with 'undefined reference' errors
               workaround:  after doing 10.2 prereq steps, rename
                 $ORACLE_HOME/lib/stubs to $ORACLE_HOME/lib/stubs.10.1
                 $ORACLE_HOME/lib32/stubs to $ORACLE_HOME/lib32/stubs.10.1
                 then start installing 10.2.0.1         
    5071492  LMS crash with ORA-00600: internal error code, arguments: [kclastf_1], [2], [],
               affects:  10.2.0.2 RAC
               symptom:  the instance could terminate
               solution:  apply patch 5071492
    4639236  ORA-600 [kclcls_5] in RAC instance
               affects:  10.2.0.2 RAC
               symptom:  the instance could terminate 
               solution: apply patch 4639236
    4690794  "ERROR IN KQLMBIVG SEE LCK TRACE FILE" [LT] [LB]  KJUSERCLIENTLOCK 
               affects: 10.2.0.2 RAC
               symptom: Error messages in the alert logs
               solution: apply patch 4690794 
    5036588  LMD0 PROCESS RECEIVED OS SIGNAL #11
               affects: 10.2.0.2 RAC
               symptom: the instance could terminate
               solution: apply patch  5036588
    5021707  CSS should open block device voting disk with o_direct
               affects: 10.2.0.2 RAC when voting or ocr disk is a block device
               symptom: RAC reconfiguration may not occur properly
               solution: use raw devices for ocr and voting disks
    5219517  failing RAC interconnect causes ASM to hang
               affects: 10.2.0.2 RAC using ASM (2-node)
               symptom: ifconfig down of RAC interconnect on node 1 causes
                        surviving node 2 ASM instance to hang.  This may
                        occur during testing to simulate interconnect failure.
               solution: RAC properly handles this failure case when the
                         interconnect is physically disconnected  vs. ifconfig down
                         which is simulating the failure
    5298421  asmlib ORA-15183 ASMLIB initialization error
               affects: 10.2 when using asmlib
               symptom: ORA-15183 could occur while starting an asm instance
               workaround: a reboot is needed
    5333606  rpm throws errors if selinux is disabled at OS installation and is enabled at runtime
             affects:  error at rpm packages installs
             symptom:  during packages installation, errors indicate the scriplet failures
             workaround: do not change selinux mode from disable to enable or visa versa at runtime
    5017352  warning messages in trace files during certain workloads with async io turned on
             affects: performance
             symptom: Following warning in trace files
             WARNING: ORACLE PROCESS RUNNING OUT OF OS KERNEL I/O RESOURCES
             workaround: none so far