Oracle Validated Configuration

Configuration Summary:	Oracle Database 10g Release 2 (Single Instance) and Oracle Real Application Clusters (RAC) on SUSE Linux Enterprise Server 9 x86-64 using NFS storage, OCFS2/iscsi
Publication Date:	August 14, 2006
Version:	1.1
Server Platform:	IBM eServer 326
Storage:	NetApp FAS900 Series
Oracle Software:	Oracle Database 10g Release 2 (10.2.0.2) for Linux x86-64
Linux Distribution:	SLES 9 Service Pack 3 x86-64

Server and Storage Platform Details

    2 x IBM eServer 326
    2 x Dual Core AMD Opteron 275 1.8 GHz (4 cores)
    8 GB RAM
    1 x 146.8 GB 10K HD
    2 x Integrated Gigabit NIC
    1 x NetXtreme 1000 Express Ethernet Adapter

    Settings in the server BIOS adjusted to allow memory/node interleaving
    to work better with the 'numa=off' boot option
 
  Additional models covered by this configuration
    eServer 326m
    BladeCenter LS20 blade

  1 x NetApp FAS960 running ONTAP v7.0.3
    14 x 72 GB 10K HD

  Additional models covered by this configuration
    IBM N Series storage (ONTAP v7.0.3) with NAS using ethernet attachments
    NetApp FAS900 Series storage running ONTAP v7.0.3


Linux Distribution Details

  SLES 9 Service Pack 3 86-64
    kernel-smp-2.6.5-7.252.x86_64.rpm (SP3 update kernel) for NFS
    kernel-smp-2.6.5-7.257.x86_64.rpm (SP3 update kernel) for ocfs2

  Additional packages needed from distribution
    binutils-2.15.90.0.1.1-32.10.x86_64.rpm
    gcc-3.3.3-43.41.x86_64.rpm
    gcc-c++-3.3.3-43.41.x86_64.rpm
    glibc-devel-2.3.3-98.61.x86_64.rpm
    glibc-devel-32bit-9-200512100801.x86_64.rpm
    glibc-2.3.3-98.61.x86_64.rpm
    glibc-32bit-9-200512100800.x86_64.rpm
    gnome-libs-1.4.1.7-671.1.x86_64.rpm
    libaio-0.3.102-1.2.x86_64.rpm
    libstdc++-3.3.3-43.41.x86_64.rpm
    libstdc++-devel-3.3.3-43.41.x86_64.rpm
    make-3.80-184.1.x86_64.rpm
    pdksh-5.2.14-780.7.x86_64.rpm
    sysstat-5.0.1-35.7.x86_64.rpm
    xscreensaver-4.16-2.6.x86_64.rpm

   Additional packages from Novell

    Required for DB on OCFS2
      ocfs2-tools-1.2.1-4.2.x86_64.rpm
      ocfs2console-1.2.1-4.2.x86_64.rpm

  Boot options
   numa=off elevator=deadline

  /etc/modprobe.conf:
    options e1000 FlowControl=1  (if using e1000)
    options hangcheck_timer hangcheck_reboot=1
 
  /etc/sysctl.conf settings
    fs.file-max=327679
    net.ipv4.ip_local_port_range=1024 65000
    kernel.msgmni=2878
    kernel.msgmax=8192
    kernel.msgmnb=65536
    kernel.sem=250 32000 100 142
    kernel.shmmni=4096
    kernel.shmall=3279547
    # set to a value half the size of physical memory
    kernel.shmmax=3700000000
    kernel.sysrq=1
#  rmem_max can be tuned based on workload to balance performance vs lowmem usage
    net.core.rmem_default=262144
    net.core.rmem_max=2097152
    net.core.wmem_default=262144
    net.core.wmem_max=262144
#  aio-max-nr not needed for nfs
#    fs.aio-max-nr=3145728
#  needed to enable hugepage usage
    vm.disable_cap_mlock=1 
  

  /etc/security/limits.conf:  
    # depending on size of db, these may need to be larger
    oracle  soft    nofile  131072
    oracle  hard    nofile  131072
    oracle  soft    nproc   131072
    oracle  hard    nproc   131072

 /etc/sysconfig/o2cb
  #Note: If the database is on OCFS2, timeouts need to be adjusted.
   Set O2CB_HEARTBEAT_THRESHOLD to a value which takes into consideration 
   the underlying storage and the workload to avoid timeouts
   ( E.g: O2CB_HEARTBEAT_THRESHOLD=31 )

  Filesystems tested:
    nfs  (see mount options)
    ext3
    reiserfs  (see mount option)
    ocfs2

  NFS mount options:
    Oracle Home files 
      rw,bg,hard,nointr,tcp,nfsvers=3,timeo=600,rsize=32768,wsize=32768
    Single Instance data files
      rw,bg,hard,nointr,tcp,nfsvers=3,timeo=600,rsize=32768,wsize=32768
    RAC instance data files
      rw,bg,hard,nointr,tcp,nfsvers=3,timeo=600,rsize=32768,wsize=32768,actimeo=0
    RAC ocr and voting disk files
      rw,bg,hard,nointr,tcp,nfsvers=3,timeo=600,rsize=32768,wsize=32768,actimeo=0


  reiserfs mount option:
    notail

  ocfs2 mount option:
   Data files, ocr and voting disk files
   _netdev, datavolume, nointr
    
  
  Miscellaneous

  The network driver module use (tg3) is the version supplied with SLES 9 SP3 x86-64
  The hangcheck-timer module used is the version supplied with SLES 9 SP3 x86-64

  The swap partition is the size of physical RAM

  3 gigabit networks configured:  public, nas, and interconnect


Oracle Software Details


  Oracle Database 10g Release 2 (10.2.0.2) Single Instance and 
  Oracle Real Application Clusters (RAC) for Linux x86-64
      patch 5071492
      patch 4639236
      patch 4690794
      patch 5036588

  init.ora settings:
    filesystemio_options=directio (required for RAC on NFS)

Configuration Feedback

    Bug      Summary
    Number

    5041764  CFQ io scheduler can delay/hang io (bz 151368)
               affects:  heavy io access to raw/block devices   
               symptom:  io requests will start to hang or take a long to complete 
                         RAC node could be evicted 
               workaround:  boot w/ elevator=deadline
    5058952  e1000 flow control defaults to none in the 2.6 kernel
               affects:  e1000 network interfaces with heavy traffic
               symptom:  RAC interconnect may get lost blocks
               solution:  load e1000 with FlowControl=1 (Rx)
    5172422  errors writing archive logs on reiserfs (bz 167592)
               affects:  writing archive logs on reiserfs
               symptom:  ORA-19502 write errors in alert log
                         file may briefly disappear, then reappear
               solution:  mount reiserfs with 'notail' option to properly create/write files 
                          for directio access
                    when accessing small files with directio 
    5215593  unable to start RAC with db_cache_size > 4gb
               affects:  RAC instances with > 4gb db_cache_size
               symptom:  nodes that started up correctly with 10.2.0.1
                         may fail to startup with ORA-4031 errors in 10.2.0.2
               workaround:  set _ksmg_granule_size=33554432 in the init.ora.
                            Larger sga sizes may need a higher value.
    5215622  libknlopt.a not relinked
               affects:  10.2.0.2 RAC (did not occur in 10.1)
               symptom:  may get ORA-7445 errors during startup
               workaround:  manually relink on each node
    5041394  relink errors after upgrading from 10.1 to 10.2 
               affects:  10.1 -> 10.2 upgrade process
               symptom:  during 10.2 upgrade, relink step will fail
                         with 'undefined reference' errors
               workaround:  after doing 10.2 prereq steps, rename
                 $ORACLE_HOME/lib/stubs to $ORACLE_HOME/lib/stubs.10.1
                 $ORACLE_HOME/lib32/stubs to $ORACLE_HOME/lib32/stubs.10.1
                 then start installing 10.2.0.1         
    5071492  LMS crash with ORA-00600: internal error code, arguments: [kclastf_1], [2], [],
               affects:  10.2.0.2 RAC
               symptom:  the instance could terminate
               solution:  apply patch 5071492
    4639236  ORA-600 [kclcls_5] in RAC instance
               affects:  10.2.0.2 RAC
               symptom:  the instance could terminate 
               solution: apply patch 4639236
    4690794  "ERROR IN KQLMBIVG SEE LCK TRACE FILE" [LT] [LB]  KJUSERCLIENTLOCK 
               affects: 10.2.0.2 RAC
               symptom: Error messages in the alert logs
               solution: apply patch 4690794 
    5036588  LMD0 PROCESS RECEIVED OS SIGNAL #11
               affects: 10.2.0.2 RAC
               symptom: the instance could terminate
               solution: apply patch  5036588
    5017352  warning messages in trace files during certain workloads with async io turned on
               affects: performance
               symptom: Following warnings in trace files
               WARNING: ORACLE PROCESS RUNNING OUT OF OSKER