Oracle Validated Configuration

Configuration Summary:	Oracle Database 10g Release 2 Single Instance and Oracle Real Application Clusters (RAC) on SUSE Linux Enterprise Server 9 x86-64 using Automatic Storage Management (ASMLIB) on Fibre Channel attached storage
Publication Date:	September 22, 2006
Version:	1.0
Server Platform:	Huawei ATCA 4P BA40 server series
Storage:	LSI 3200 Array Storage
Oracle Software:	Oracle Database 10g Release 2 (10.2.0.2) for Linux x86-64
Linux Distribution:	SLES 9 Service Pack 3 x86-64

Server Platform Details
    4 X Huawei ATCA  4P BA40 servers
    4 X AMD Opteron 848(4 X 2.2G Sole Core)
    8 GB RAM 
    2 X 36 G (SAS)
    HBA Transfers speed: 2GB/SEC
    Dual Onboard Broadcom Gigabit NICs

Settings in the server BIOS adjusted to allow memory/node interleaving
to work better with the 'numa=off' boot option

Storage Platform Details
    1 X LSI 3200 Array Storage
    14 X 36.4GB 10K rpm Drives (2 X RAID 5)
    Huawei ATCA NX10 Blade Switch Transfers speed: FC:2GB/SEC GE:1GB/SEC 

Linux Distribution Details
  SLES 9 Service Pack 3 86-64
  kernel-smp-2.6.5-7.244 (SP3 update kernel)

Additional packages needed from distribution
  binutils-2.15.90.0.1.1-32.10.x86_64.rpm
  gcc-3.3.3-43.41.x86_64.rpm
  gcc-c++-3.3.3-43.41.x86_64.rpm
  glibc-devel-2.3.3-98.61.x86_64.rpm
  glibc-devel-32bit-9-200512100801.x86_64.rpm
  glibc-2.3.3-98.61.x86_64.rpm
  glibc-32bit-9-200512100800.x86_64.rpm
  gnome-libs-1.4.1.7-671.1.x86_64.rpm
  libaio-0.3.102-1.2.x86_64.rpm
  libstdc++-3.3.3-43.41.x86_64.rpm
  libstdc++-devel-3.3.3-43.41.x86_64.rpm
  make-3.80-184.1.x86_64.rpm
  pdksh-5.2.14-780.7.x86_64.rpm
  sysstat-5.0.1-35.7.x86_64.rpm
  xscreensaver-4.16-2.6.x86_64.rpm

Additional packages from Oracle

  Required for DB on ASM
     oracleasm-support-2.0.2-1.x86_64.rpm
     oracleasmlib-2.0.2-1.x86_64.rpm
     oracleasm-2.6.5-7.244-smp-2.0.2-1.x86_64.rpm

Boot options
   elevator=deadline numa=off

/etc/modprobe.conf:
   options e1000 FlowControl=1
   options hangcheck_timer hangcheck_reboot=1

/etc/sysctl.conf settings
   net.ipv4.icmp_echo_ignore_broadcasts = 1
   net.ipv4.conf.all.rp_filter = 1
   fs.file-max=327679
   kernel.msgmni=2878
   kernel.msgmax=8192
   kernel.msgmnb=65536
   kernel.sem=250 32000 100 142
   kernel.shmmni=4096
   kernel.shmall=3279547
   kernel.shmmax=3700000000
   kernel.sysrq=1
   net.core.rmem_default=262144
   net.core.rmem_max=2097152
   net.core.wmem_default=262144
   net.core.wmem_max=262144
   fs.aio-max-nr=3145728
   net.ipv4.ip_local_port_range=1024 65000
   vm.disable_cap_mlock=1
   vm.nr_hugepages=1000

/etc/security/limits.conf:

# depending on size of db, these may need to be larger
   oracle  soft    nofile  131072
   oracle  hard    nofile  131072
   oracle  soft    nproc   131072
   oracle  hard    nproc   131072
   oracle  soft    core    unlimited
   oracle  hard    core    unlimited
   oracle  soft    memlock 50000000
   oracle  hard    memlock 50000000

Filesystems tested:

    ASM using Automatic Storage Management Library Driver (asmlib)

Driver Modules:

  The HBA driver module used (qla2300) is the version supplied with SLES 9 SP3 x86-64
  The network driver module tg3 (Version 3.52e) is from Broadcom
  The network driver module e1000 (Version 6.3.9) is from Intel
  The hangcheck-timer module used is the version supplied with SLES 9 SP3 x86-64

  The swap partition is the size of physical RAM(8GB)

Miscellaneous
  2 gigabit networks configured: public and interconnect

Oracle Software Details

  Oracle Database 10g Release 2 (10.2.0.2) Single Instance and RAC forAMD64
  patch 5171356
  patch 5162682
  patch 5152261
  patch 5077520

Configuration Feedback

   Bug     Summary
   Number

   5041764  CFQ io scheduler can delay/hang io (bz 151368)
               affects:  heavy io access to raw/block devices??
               symptom:  io requests will start to hang or take a long to complete
                         RAC node could be evicted
               workaround:  boot w/ elevator=deadline

   5058952  e1000 flow control defaults to none in the 2.6 kernel
               affects:  e1000 network interfaces with heavy traffic
               symptom:  RAC interconnect may get lost blocks
               solution: load e1000 with FlowControl=1 (Rx)

   5172422  errors writing archive logs on reiserfs (bz 167592)
               affects:  writing archive logs on reiserfs
               symptom:  ORA-19502 write errors in alert log
                         file may briefly disappear, then reappear
               solution: mount reiserfs with 'notail' option to properly create/write 
			   files for directio access when accessing small files with directio 

   5215593  unable to start RAC with db_cache_size > 4gb
               affects:  RAC instances with > 4gb db_cache_size
               symptom:  nodes that started up correctly with 10.2.0.1
                         may fail to startup with ORA-4031 errors in 10.2.0.2
               workaround: set _ksmg_granule_size=33554432 in the init.ora.
                           Larger sga sizes may need a higher value.

   5215622  libknlopt.a not relinked
               affects:  10.2.0.2 RAC (did not occur in 10.1)
               symptom:  may get ORA-7445 errors during startup
            	workaround:  manually relink on each node

   5041394  relink errors after upgrading from 10.1 to 10.2
               affects:  10.1 -> 10.2 upgrade process
               symptom:  during 10.2 upgrade, relink step will fail
                         with 'undefined reference' errors
               workaround:  after doing 10.2 prereq steps, rename
                            $ORACLE_HOME/lib/stubs to $ORACLE_HOME/lib/stubs.10.1
                            $ORACLE_HOME/lib32/stubs to $ORACLE_HOME/lib32/stubs.10.1
                            then start installing 10.2.0.1

   5071492  LMS crash with ORA-00600: internal error code, arguments: 	[kclastf_1], [2], [],
               affects:  10.2.0.2 RAC
               symptom:  the instance could terminate
               solution:  apply patch 5077520

   4639236  ORA-600 [kclcls_5] in RAC instance
               affects:  10.2.0.2 RAC
               symptom:  the instance could terminate
               solution:  apply patch 5152261

   4690794  "ERROR IN KQLMBIVG SEE LCK TRACE FILE" [LT] [LB] KJUSERCLIENTLOCK
               affects:  10.2.0.2 RAC
               symptom:  Error messages in the alert logs
               solution: apply patch 5162682

   5036588  LMD0 PROCESS RECEIVED OS SIGNAL #11
               affects:  10.2.0.2 RAC
               symptom:  the instance could terminate
               solution: apply patch 5171356
		
   5330996  KSXPLOOKUP: warning cnh incarnation number may be stale
	    	symptom: message 'warning cnh incarnation number may be stale' in trace files.	
		affects: 10.2.0.2 RAC
	    	workaround: Not fixed yet

   5021707  CSS should open block device voting disk with o_direct
              affects:  10.2.0.2 RAC when voting or ocr disk is a block device
              symptom:  RAC reconfiguration may not occur properly
              solution:  use raw devices for ocr and voting disks

   5298421  asmlib ORA-15183 ASMLIB initialization error
              affects:  10.2 when using asmlib
              symptom:  ORA-15183 could occur while starting an asm instance
              workaround: a reboot is needed

   5017352  warning messages in trace files during certain workloads  with async io turned on
              affects: performance
              symptom: Following warnings in trace files
                        WARNING:ORACLE PROCESS RUNNING OUT OF OS KERNEL I/O RESOURCES
              Workaround: None so far

   5106311  FSCK on EXT3 partition invokes oom kill buring reboot
	      affects : System will not reboot
	      symptom: fsck got killed by OOM KILLER
	      Workaround: use kernel version 2.6.5-7.252 or above