Oracle Validated Configuration

Configuration Summary:	Oracle Database 10g Release 2 (Single Instance), Oracle Real Application Clusters (RAC) Red Hat Enterprise Linux 4 AS/ES Update 3 x86-64, NFS Vers 3
Publication Date:	December 12, 2006
Version:	1.2
Server Platform:	Sun Blade 8000 Modular System
Storage:	Sun StorageTek 5320 NAS Appliance
Oracle Software:	Oracle Database 10g Release 2 (10.2.0.2) for Linux x86-64
Linux Distribution:	RHEL4 Update 3 x86-64

Server and Storage Platform Details

   1 x Sun Blade 8000 Modular System 
   2 x Sun Blade x8400 Server Module with the following configuration:
   4 x Dual Core AMD Opteron 885 2.6 GHz (8 cores)
   32 GB RAM
   2 x 73 GB 10K HD
   4 x Integrated Gigabit NIC

Select "No" to disable the BIOS ACPI "MCFG" table in the Advanced ACPI Configuration.

Additional models covered by this configuration
	Sun Fire X4200 
	Sun Fire X4500
	Sun Fire X4600
	
Storage:
    Sun StorageTek 5320 NAS Appliance
    14 x 146 GB 10K HD

 Linux Distribution Details

	RHEL 4 AS Update 3 x86-64 (default install )
		kernel-smp-2.6.9-34.ELsmp.x86_64.rpm (from Update 3)

Additional packages needed from distribution
	binutils-2.15.92.0.2-18.x86_64.rpm
	compat-db-4.1.25-9.x86_64.rpm
	compat-gcc-32-3.2.3-47.3.x86_64.rpm
	compat-libstdc++-33-3.2.3-47.3.x86_64.rpm
	control-center-2.8.0-12.rhel4.2.x86_64.rpm
	gcc-3.4.5-2.x86_64.rpm
	gcc-c++-3.4.5-2.x86_64.rpm
	glibc-2.3.4-2.19.i686.rpm
	glibc-2.3.4-2.19.x86_64.rpm 
	glibc-common-2.3.4-2.19.x86_64.rpm
	glibc-devel-2.3.4-2.19.i386.rpm
	glibc-devel-2.3.4-2.19.x86_64.rpm
	glibc-headers-2.3.4-2.19.x86_64.rpm
	glibc-kernheaders-2.4-9.1.98.EL.x86_64.rpm
	gnome-libs-1.4.1.2.90-44.1.x86_64.rpm
	libaio-0.3.105-2.x86_64.rpm
	libaio-devel-0.3.105-2.x86_64.rpm
	libgcc-3.4.5-2.i386.rpm
	libstdc++-3.4.5-2.x86_64.rpm
	libstdc++-devel-3.4.5-2.x86_64.rpm
	make-3.80-5.x86_64.rpm 
	pdksh-5.2.14-30.3.x86_64.rpm 
	sysstat-5.0.5-7.rhel4.x86_64.rpm
	xorg-x11-deprecated-libs-6.8.2-1.EL.13.25.i386.rpm
	xscreensaver-4.18-5.rhel4.10.x86_64.rpm

Boot options
# Numa is enabled on default at server boot. Oracle and Sun recommend to set numa=off
	numa=off elevator=deadline

/etc/modprobe.conf:
	options e1000 FlowControl=1,1,1,1
	options hangcheck_timer hangcheck_reboot=1
 
/etc/sysctl.conf settings
	fs.file-max=327679
	kernel.msgmni=2878
	kernel.msgmax=8192
	kernel.sem=250 32000 100 142
	 kernel.shmmni=4096
	kernel.shmall=3279547
	#set to a value half the size of physical memory
	kernel.shmmax=3700000000
	 # rmem_max can be tuned based on workload to balance performance
	vs lowmem  usage
	net.core.rmem_default=262144
	net.core.rmem_max=2097152
	net.core.wmem_default=262144
	net.core.wmem_max=262144
	fs.aio-max-nr=3145728
	net.ipv4.ip_local_port_range=1024 65000
	vm.lower_zone_protection=100
	kernel.msgmnb=65536
	kernel.sysrq=1

/etc/security/limits.conf:
	oracle  soft  nofile  131072
	oracle  hard  nofile  131072
	oracle  soft  nproc   131072
	oracle  hard  nproc   131072
	oracle  soft  core    unlimited
	oracle  hard  core    unlimited
	# set memlock greater than or equal to the sga size to allow oracle 
                  use hugepages 
	oracle  soft  memlock 50000000
	oracle  hard  memlock 50000000

Filesystems tested:
	NFS vers 3.0

Mount options:
#Oracle Home Files
	rw,bg,hard,nointr,tcp,nfsvers=3,timeo=600,rsize=32768,wsize=32768 
#Single Instance Data Files
	rw,bg,hard,nointr,tcp,nfsvers=3,timeo=600,rsize=32768,wsize=32768
#RAC Instance Data Files
	rw,bg,hard,nointr,tcp,nfsvers3,timeo=600,rsize=32768,wsize=32768,actimeo=0# RAC ocr and voting disk files
	rw,bg,hard,nointr,tcp,nfsvers3,timeo=600,rsize=32768,wsize=32768,actimeo=0	

Driver Modules:
	The network driver module used (e1000) is the version supplied with 
	RHEL 4 U3 x86-64
	The hangcheck-timer module used is the version supplied with 
                RHEL 4 U3 x86-64

Miscellaneous
	2 gigabit networks configured: public and interconnect
	The swap partition's size is 32GB
	
Storage Server Configuration
	Operating System 
	OS Version 4.20 build with latest patches from Sunsolve

Oracle Software Details

Oracle Database 10g Release 2 (10.2.0.2) Single Instance and Oracle Real Application Clusters (RAC) for Linux x86-64
patch 5071492
patch 4639236
patch 4690794
patch 5036588

Configuration Feedback
Bug #       Summary

5041764 CFQ io scheduler can delay/hang io (bz 151368)
affects: heavy io access to raw/block devices  symptom: io requests will start to hang or take a long to complete 
RAC node could be evicted 
  workaround: Fixed in U4. Pre-U4, boot w/ elevator=deadline. 

5058952 e1000 flow control defaults to none in the 2.6 kernel
     affects: e1000 network interfaces with heavy traffic
     symptom: RAC interconnect may get lost blocks
     solution: load e1000 with FlowControl=1 (Rx)
	 
5215593 unable to start RAC with db_cache_size > 4gb
     affects: RAC instances with > 4gb db_cache_size
     symptom: nodes that started up correctly with 10.2.0.1 may fail to startup with ORA-4031 errors in 10.2.0.2
     workaround: set _ksmg_granule_size=33554432 in the init.ora. Larger sga sizes may need a higher value.

5215622 libknlopt.a not relinked
    affects: 10.2.0.2 RAC (did not occur in 10.1)
    symptom: may get ORA-7445 errors during startup
    workaround: manually relink on each node

5041394 relink errors after upgrading from 10.1 to 10.2
    affects: 10.1 -> 10.2 upgrade process
    symptom: during 10.2 upgrade, relink step will fail with 'undefined reference' errors
    workaround: after doing 10.2 prereq steps, rename
           $ORACLE_HOME/lib/stubs to $ORACLE_HOME/lib/stubs.10.1
		   $ORACLE_HOME/lib32/stubs to $ORACLE_HOME/lib32/stubs.10.1
		   
    then start installing 10.2.0.1
	
5071492 LMS crash with ORA-00600: internal error code, arguments: [kclastf_1], [2], [],
    affects: 10.2.0.2 RAC
    symptom: the instance could terminate
    solution: apply patch 5071492

4639236 ORA-600 [kclcls_5] in RAC instance
    affects: 10.2.0.2 RAC
    symptom: the instance could terminate 
	solution: apply patch 4639236

4690794 "ERROR IN KQLMBIVG SEE LCK TRACE FILE" [LT] [LB]  KJUSERCLIENTLOCK 
    affects: 10.2.0.2 RAC
    symptom: Error messages in the alert logs
    solution: apply patch 4690794 
   
5036588  LMD0 PROCESS RECEIVED OS SIGNAL #11
    affects: 10.2.0.2 RAC
    symptom: the instance could terminate
    solution: apply patch  5036588

5017352  warning messages in trace files during certain workloads with async io turned on
   affects: performance
   symptom: Following warning in trace files
            WARNING: ORACLE PROCESS RUNNING OUT OF OS KERNEL I/O RESOURCES
   workaround: none so far

5333606  rpm throws errors if selinux is disabled at OS installation and is enabled at runtime
    affects: error at rpm packages installs
    symptom: during packages installation, errors indicate the scriplet failures
    workaround: relabel the filesystems through 'touch /.autorelabel' or'fixfilesrelabel'

5093059  netdump doesn't load with bonding configured (it 85145)
    affects: bringing up netdump over a bonded interface
    symptom: /var/log/messages will have an error:
             netlog: bond0 doesn't support polling, aborting.
    workaround: none at the moment

4593892  repeated warnings in dmesg about lost ticks (it 80129/bz 139376)
    affects: usually seen on x86-64, both Intel and Opteron based systems
    symptom: dmesg will have errors like:
              warning: many lost ticks.
              Your time source seems to be instable or some driver is hogging
              Interupts rip __do_softirq+0x4d/0xd0
    workaround:  none at the moment

5593995 During compilation of proc programs with selinux on and their pcscfg.cfg configuration file, the process
consumes all available memory and swap 
    affects: Caused the system to hang.
    symptom: The system hangs
    workaround: Add valid pathnames to the pcscfg.cfg file

5021707 CSS should open block device voting disk with o_direct
    affects: 10.2.0.2 RAC when voting or ocr disk is a block device
    symptom: RAC reconfiguration may not occur properly
	solution: use raw devices for ocr and voting disks

5219517 failing RAC interconnect causes ASM to hang
    affects: 10.2.0.2 RAC using ASM (2-node)
    symptom: ifconfig down of RAC interconnect on node 1 causes surviving node
             2 ASM instance to hang. This may occur during testing to simulate
             interconnect failure.
	solution: RAC properly handles this failure case when the interconnect is
              physically disconnected vs. ifconfig down which is simulating the failure

5298421 asmlib ORA-15183 ASMLIB initialization error
    affects: 10.2 when using asmlib
    symptom: ORA-15183 could occur while starting an asm instance
    workaround: a reboot is needed


Sunsolve 6335612 The factory default BIOS settings do not support the installation or
running of Red Hat Enterprise Advance Server 4 Update 3 operationg system software.
    affects: Only RHEL AS 4-U3
    symptom: Installation of only RHEL4 U3 on SunBlade 8000 does not boot up.
    solution: Change the factory default BIOS settings before you install Linux software by following the following steps: 
              In the BIOS Setup utility, go to the Advanced -> ACPI Configuration -> Advanced ACPI Configuration menu;
              Select No to disable the BIOS ACPI MCFG table;
              Save settings and then exit.
This information is described in the Sun Blade 8000 Modular System Product Notes (819-5651-14)
 
Sunsolve 6474408 w.r.t aio+dio tests cannot access files from the 5320 NAS Appliance
    affects: RedHat NFS client is not properly handling short NFS read responses the way described in RFC 1813.
    symptom: ORA-00257: archiver error. 
    solution: Upgrade NAS OS from 4.12 build to 4.20 build and apply latest patch from Sunsolve.