Oracle Validated Configuration

Configuration Summary:	Oracle Database 10g Release 2 (Single Instance) and Oracle Real Application Clusters (RAC) on SUSE Enterprise Server 9 x86-64 using Fibre Channel attached storage
Publication Date:	June 12, 2006
Version:	1.1
Server Platform:	IBM BladeCenter HS20
Storage:	IBM TotalStorage DS4500
Oracle Software:	Oracle Database 10g Release 2 (10.2.0.2) for Linux x86-64
Linux Distribution:	SLES 9 Service Pack 2 x86-64

Server and Storage Platform Details
    2 x IBM BladeCenter HS20
    2 x Intel(R) Xeon(TM) with Extended Memory 64 Technology (EM64T) 3.0 GHz
    8 GB RAM
    1 x 40 GB IDE HD
    2 x integrated Gigabit Ethernet ports (Broadcom)
    1 x addon BladeCenter Fibre Channel Expansion Card (2 ports, QLogic)

  1 x IBM TotalStorage DS4500
    28 x 73 GB 15K RPM HD

  Additional models covered by this configuration

    IBM System x:
      x3400 (x226)
      x3500 (x236)
      x3800 (x260)
      x3550 (x336) 
      x3650 (x346) 
      x3850 (x366)

    IBM System Storage DS4000 Family:

      DS4100
      DS4300 and DS4300 Turbo
      DS4700
      DS4800

Linux Distribution Details

  SLES 9 Service Pack 2 x86-64 (default install)
    kernel-smp-2.6.5-7.201.x86_64.rpm  (updated kernel)

  Additional packages needed from distribution
    binutils-2.15.90.0.1.1-32.10.x86_64.rpm
    gcc-3.3.3-43.34.x86_64.rpm
    gcc-c3.3.3-43.34.x86_64.rpm
    glibc-devel-2.3.3-98.47.x86_64.rpm
    glibc-devel-32bit-9-200506062332.x86_64.rpm
    glibc-2.3.3-98.47.x86_64.rpm
    glibc-32bit-9-200506071326.x86_64.rpm
    libaio-0.3.102-1.2.x86_64.rpm
    libstdc+++3.3.3-43.34.x86_64.rpm
    libstdc+++-devel-3.3.3-43.34.x86_64.rpm
    make-3.80-184.1.x86_64.rpm
    pdksh-5.2.14-780.7.x86_64.rpm
    sysstat-5.0.1-35.4.x86_64.rpm
    xscreensaver-4.16-2.6.x86_64.rpm

  Additional packages needed from IBM

    IBM FAStT FC-2 and DS4000 FC2-133 Host Bus Adapter non-failover device 
        driver for Linux Kernel 2.6 IA-32, IA-64, AMD-64 Version 8
    IBM TotalStorage DS4000 Linux RDAC for 2.6.x kernel Version 9

  Boot options
    selinux=0 elevator=deadline

  /etc/modprobe.conf.local:   
    options qla2xxx displayConfig=1 ql2xfailover=0
    remove qla2xxx /sbin/modprobe -r --first-time --ignore-remove qla2xxx && { /sbin/modprobe -r --ignore-remove qla2xxx_conf; }

 
  /etc/sysctl.conf settings

    fs.file-max=327679
    fs.aio-max-nr=3145728
    kernel.msgmni=2878
    kernel.msgmax=8192
    kernel.msgmnb=65536
    kernel.sem=250 32000 100 142
    kernel.shmmni=4096
    kernel.shmall=3279547
#   set to value half the size of physical memory
    kernel.shmmax=3700000000
    kernel.sysrq=1
    net.core.rmem_default=262144
#   rmem_max can be tuned based on workload to balance performance versus lowmem usage
    net.core.rmem_max=262144
    net.core.wmem_default=262144
    net.core.wmem_max=262144
#   needed to enable hugepage usage
    vm.disable_cap_mlock=1 

  /etc/security/limits.conf:
    # depending on size of db, these may need to be larger
    oracle  soft    nofile  131072
    oracle  hard    nofile  131072
    oracle  soft    nproc   131072
    oracle  hard    nproc   131072
    oracle  soft    memlock 50000000
    oracle  hard    memlock 50000000

  Driver Modules
  The network driver module use (tg3) is the version supplied with SLES 9 SP2 x86-64
  The hangcheck-timer module used is the version supplied with SLES 9 U2 x86-64

  Miscellaneous
  
  The swap partition is the size of physical RAM.
  2 gigabit networks configured:  public and interconnect


Oracle Software Details

  Oracle Database 10g Release 2 (10.2.0.2) Single Instance and 
  Oracle Real Application Clusters (RAC) for Linux x86-64 

      patch 5071492
      patch 4639236
      patch 4690794
      patch 5036588

  oracleasm-2.6.5-7.201-smp-2.0.1-1.x86_64.rpm
  oracleasm-support-2.0.1-1.x86_64.rpm
  oracleasmlib-2.0.1-1.x86_64.rpm

  init.ora settings:
    filesystemio_options=directio (required for RAC)
 
  Filesystems tested:
    ext2
    ASM using block devices
    ASM using ASMLib

  RAC ocr and voting disk files
    raw devices


Configuration Feedback

    Bug      Summary
    Number

    5041764  CFQ io scheduler can delay/hang io (bz 151368)
               affects:  heavy io access to raw/block devices   
               symptom:  io requests will start to hang or take a long to complete 
                         RAC node could be evicted 
               workaround:  boot w/ elevator=deadline
    5071492  LMS crash with ORA-00600: internal error code, arguments: [kclastf_1], [2], [],
               affects:  10.2.0.2 RAC
               symptom:  the instance could terminate
               solution:  apply patch 5071492
    4639236  ORA-600 [kclcls_5] in RAC instance
               affects:  10.2.0.2 RAC
               symptom:  the instance could terminate 
               solution: apply patch 4639236
    4690794  "ERROR IN KQLMBIVG SEE LCK TRACE FILE" [LT] [LB]  KJUSERCLIENTLOCK 
               affects: 10.2.0.2 RAC
               symptom: Error messages in the alert logs
               solution: apply patch 4690794 
    5215593  unable to start RAC with db_cache_size > 4gb
               affects:  RAC instances with > 4gb db_cache_size
               symptom:  nodes that started up correctly with 10.2.0.1
                         may fail to startup with ORA-4031 errors in 10.2.0.2
               workaround:  set _ksmg_granule_size=33554432 in the init.ora.
                            Larger sga sizes may need a higher value.
    5215622  libknlopt.a not relinked
               affects:  10.2.0.2 RAC (did not occur in 10.1)
               symptom:  may get ORA-7445 errors during startup
               workaround:  manually relink on each node
    5041394  relink errors after upgrading from 10.1 to 10.2 
               affects:  10.1 -> 10.2 upgrade process
               symptom:  during 10.2 upgrade, relink step will fail
                         with 'undefined reference' errors
               workaround:  after doing 10.2 prereq steps, rename
                 $ORACLE_HOME/lib/stubs to $ORACLE_HOME/lib/stubs.10.1
                 $ORACLE_HOME/lib32/stubs to $ORACLE_HOME/lib32/stubs.10.1
                 then start installing 10.2.0.1
    4639236  ORA-600 [kclcls_5] in RAC instance
               affects:  10.2.0.2 RAC
               symptom:  the instance could terminate 
               solution: apply patch 4639236
    4690794  "ERROR IN KQLMBIVG SEE LCK TRACE FILE" [LT] [LB]  KJUSERCLIENTLOCK 
               affects: 10.2.0.2 RAC
               symptom: Error messages in the alert logs
               solution: apply patch 4690794 
    5036588  LMD0 PROCESS RECEIVED OS SIGNAL #11
               affects: 10.2.0.2 RAC
               symptom: the instance could terminate
               solution: apply patch  5036588
    5021707  CSS should open block device voting disk with o_direct
               affects: 10.2.0.2 RAC when voting or ocr disk is a block device
               symptom: RAC reconfiguration may not occur properly
               solution: use raw devices for ocr and voting disks
    5219517  failing RAC interconnect causes ASM to hang
               affects: 10.2.0.2 RAC using ASM (2-node)
               symptom: ifconfig down of RAC interconnect on node 1 causes
                        surviving node 2 ASM instance to hang.  This may
                        occur during testing to simulate interconnect failure.
               solution: RAC properly handles this failure case when the
                         interconnect is physically disconnected  vs. ifconfig down
                         which is simulating the failure
    5058952  e1000 flow control defaults to none in the 2.6 kernel
               affects: e1000 network interfaces with heavy traffic
               symptom: RAC interconnect may get lost blocks
               solution: load e1000 with FlowControl1=1 (Rx)
    5292056  encountering "bad page state at prep_new_page" while shutting the database
               affects: RAC instances
               symprom: encountering "bad page state at prep_new_page" in /var/log/messages
               workaround: none at the moment