Oracle Validated Configuration

Configuration Summary:	Oracle Database 10g Release 2 Single Instance and Oracle Real Application Clusters (RAC) on SUSE Linux Enterprise Server 9 x86-64 using Automatic Storage Management (ASMLIB), OCFS2
Publication Date:	August 14, 2006
Version:	1.0
Server Platform:	HP ProLiant DL585 Server series
Storage:	HP StorageWorks 4000 Enterprise Virtual Array
Oracle Software:	Oracle Database 10g Release 2 (10.2.0.2) for Linux x86-64
Linux Distribution:	SLES 9 Service Pack 3 x86-64

Server Platform Details    
    2 x HP ProLiant DL585
    2 x Dual Core AMD Opteron 850 1.8 GHz (4 cores)
    8 GB RAM
    2 x 72.8 GB 15K HD (RAID 1)
    1 x StorageWorks 2Gb, Dual Port, 64-Bit/133 MHz PCI-X-to-Fibre Channel HBA
    2 x Embedded NC7782 Gigabit NIC

Settings in the server BIOS adjusted to allow memory/node interleaving
to work better with the 'numa=off' boot option

Storage Platform Details
    HP StorageWorks 4000 Enterprise Virtual Array
    2 x StorageWorks 2Gb, Dual Port, 64-Bit/133 MHz PCI-X-to-Fibre Channel HBA
    28 x 72.8 GB 15K HD (RAID 0+1)
    2 x HP StorageWorks 4/16 SAN Switch

    Other Storage Arrays covered by this configuration:
      HP StorageWorks 6000 Enterprise Virtual Array
      HP StorageWorks 8000 Enterprise Virtual Array

Linux Distribution Details
  SLES 9 Service Pack 3 86-64
   kernel-smp-2.6.5-7.267.x86_64.rpm (SP3 update kernel)

Additional packages needed from distribution
  binutils-2.15.90.0.1.1-32.10.x86_64.rpm
  gcc-3.3.3-43.41.x86_64.rpm
  gcc-c++-3.3.3-43.41.x86_64.rpm
  glibc-devel-2.3.3-98.61.x86_64.rpm
  glibc-devel-32bit-9-200512100801.x86_64.rpm
  glibc-2.3.3-98.61.x86_64.rpm
  glibc-32bit-9-200512100800.x86_64.rpm
  gnome-libs-1.4.1.7-671.1.x86_64.rpm
  libaio-0.3.102-1.2.x86_64.rpm
  libstdc++-3.3.3-43.41.x86_64.rpm
  libstdc++-devel-3.3.3-43.41.x86_64.rpm
  make-3.80-184.1.x86_64.rpm
  pdksh-5.2.14-780.7.x86_64.rpm
  sysstat-5.0.1-35.7.x86_64.rpm
  xscreensaver-4.16-2.6.x86_64.rpm

Additional packages from Oracle

  Required for DB on ASM
  
     oracleasm-support-2.0.3-1.x86_64.rpm
     oracleasmlib-2.0.2-1.x86_64.rpm
     oracleasm-2.6.5-7.267-smp-2.0.2-1.x86_64.rpm

  Required for DB on OCFS2

     ocfs2 modules which are distributed with   kernel-smp-2.6.5-7.267.x86_64.rpm
     ocfs2console-1.2.1-4.2.x86_64.rpm
     ocfs2-tools-1.2.1-4.2.x86_64.rpm

Additional packages needed for PSP from the distribution
     compat-sles8-1.3-93.3.x86_64.rpm
     db40-4.0.14-148.3.x86_64.rpm
     compat-libstdc++-lsb-4.0.2_20050901-0.4.x86_64.rpm

Boot options
   elevator=deadline numa=off

/etc/modprobe.conf:
   options e1000 FlowControl=1 (if using e1000)
   options hangcheck_timer hangcheck_reboot=1

/etc/sysctl.conf settings
    fs.file-max=327679
    net.ipv4.ip_local_port_range=1024 65000
    kernel.msgmni=2878
    kernel.msgmax=8192
    kernel.msgmnb=65536
    kernel.sem=250 32000 100 142
    kernel.shmmni=4096
    kernel.shmall=3279547
# set to a value half the size of physical memory
    kernel.shmmax=3700000000
    kernel.sysrq=1
# rmem_max can be tuned based on workload to balance performance vs lowmem usage
    net.core.rmem_default=262144
    net.core.rmem_max=2097152
    net.core.wmem_default=262144
    net.core.wmem_max=262144
    fs.aio-max-nr=3145728
    vm.disable_cap_mlock=1

/etc/security/limits.conf:

# depending on size of db, these may need to be larger
   oracle  soft   nofile  131072
   oracle  hard   nofile  131072
   oracle  soft   nproc   131072
   oracle  hard   nproc   131072
# set memlock greater than or equal to the sga size to allow
# oracle to use hugepages if configured
   oracle  soft   memlock  50000000
   oracle  hard   memlock  50000000

/etc/sysconfig/o2cb
# Note: If the database is on OCFS2, timeouts need to be adjusted.
 
  Set O2CB_HEARTBEAT_THRESHOLD to a value which takes into consideration
  the underlying storage and the workload to avoid timeouts
  ( E.g: O2CB_HEARTBEAT_THRESHOLD=14 )

Filesystems tested:

    ASM using Automatic Storage Management Library Driver (asmlib)
    OCFS2

Mount options:

    For datafiles, ocr and voting disk files on OCFS2, use
    _netdev, datavoulume, nointr

Driver Modules:

  The HBA driver module used (qla2300) is the version supplied with HP PSP 7.52   (Available from hp.com at the Software and Drivers download pages.)
  The network driver module used (bcm5700) is the version supplied with  HP PSP 7.52  (Available from hp.com at the Software and Drivers download pages.)
  The hangcheck-timer module used is the version supplied with SLES 9 SP3 x86-64

  The swap partition is the size of physical RAM

Miscellaneous
  2 gigabit networks configured: public and interconnect

Oracle Software Details

  Oracle Database 10g Release 2 (10.2.0.2) Single Instance and RAC forAMD64
    patch 5071492
    patch 4639236
    patch 4690794
    patch 5036588

Configuration Feedback

   Bug     Summary
   Number

   5041764  CFQ io scheduler can delay/hang io (bz 151368)
               affects:  heavy io access to raw/block devices??
               symptom:  io requests will start to hang or take a long to complete
                         RAC node could be evicted
               workaround:  boot w/ elevator=deadline

   5058952  e1000 flow control defaults to none in the 2.6 kernel
               affects:  e1000 network interfaces with heavy traffic
               symptom:  RAC interconnect may get lost blocks
               solution: load e1000 with FlowControl=1 (Rx)

   5215593  unable to start RAC with db_cache_size > 4gb
               affects:  RAC instances with > 4gb db_cache_size
               symptom:  nodes that started up correctly with 10.2.0.1
                         may fail to startup with ORA-4031 errors in 10.2.0.2
               workaround:  set _ksmg_granule_size=33554432 in the init.ora.
                            Larger sga sizes may need a higher value.

   5215622  libknlopt.a not relinked
               affects:  10.2.0.2 RAC (did not occur in 10.1)
               symptom:  may get ORA-7445 errors during startup
               workaround:  manually relink on each node

   5041394  relink errors after upgrading from 10.1 to 10.2
               affects:  10.1 -> 10.2 upgrade process
               symptom:  during 10.2 upgrade, relink step will fail
                          with 'undefined reference' errors
               workaround:  after doing 10.2 prereq steps, rename
                            $ORACLE_HOME/lib/stubs to $ORACLE_HOME/lib/stubs.10.1
                            $ORACLE_HOME/lib32/stubs to $ORACLE_HOME/lib32/stubs.10.1
                            then start installing 10.2.0.1

   5071492  LMS crash with ORA-00600: internal error code, arguments: [kclastf_1], [2], [],
               affects:  10.2.0.2 RAC
               symptom:  the instance could terminate
               solution:  apply patch 5071492

   4639236  ORA-600 [kclcls_5] in RAC instance
               affects:  10.2.0.2 RAC
               symptom:  the instance could terminate
               solution:  apply patch 4639236

   4690794  "ERROR IN KQLMBIVG SEE LCK TRACE FILE" [LT] [LB] KJUSERCLIENTLOCK
               affects:  10.2.0.2 RAC
               symptom:  Error messages in the alert logs
               solution:  apply patch 4690794

   5036588  LMD0 PROCESS RECEIVED OS SIGNAL #11
               affects:  10.2.0.2 RAC
               symptom:  the instance could terminate
               solution: apply patch 5036588

   5021707  CSS should open block device voting disk with o_direct
               affects:  10.2.0.2 RAC when voting or ocr disk is a block device
               symptom:  RAC reconfiguration may not occur properly
               solution: use raw devices for ocr and voting disks

   5219517  failing RAC interconnect causes ASM to hang
               affects:  10.2.0.2 RAC using ASM (2-node)
               symptom:  ifconfig down of RAC interconnect on node 1 causes
                         surviving node 2 ASM instance to hang. This may
                         occur during testing to simulate interconnect failure.
               solution: RAC properly handles this failure case when the
                         interconnect is physically disconnected vs. ifconfig down
                         which is simulating the failure

   5298421  asmlib ORA-15183 ASMLIB initialization error
               affects:  10.2 when using asmlib
               symptom:  ORA-15183 could occur while starting an asm instance
               workaround: a reboot is needed

   5453167 error messages in /var/log/messages after applying the PSP package
               affects: The error messages are seen after applying the PSP package
               symptom:  Error messages thrown in /var/log/messages
               workaround: None so far

   5017352 warning messages in trace files during certain workloads with async io turned on
               affects: performance
               symptom: Following warnings in trace files
                             WARNING:ORACLE PROCESS RUNNING OUT OF OS KERNEL I/O RESOURCES
               Workaround: None so far

   5172422  errors writing archive logs on reiserfs (bz 167592)
               affects:  writing archive logs on reiserfs
               symptom:  ORA-19502 write errors in alert log
                         file may briefly disappear, then reappear
             workaround:  mount reiserfs with 'notail' option to properly create/write files 
                          for directio access when accessing small files with directio