Artigos
Grid Computing
Por Victor Armbrust
Postado em Janeiro 2014
Indice
1. Oracle Grid Infrastructure 12cR1 - Parte 1 - Visão Geral e Novos Recursos
2. Oracle Grid Infrastructure 12cR1 - Parte 2 - Configurando o Sistema Operacional
3. Oracle Grid Infrastructure 12cR1 - Parte 3 - Pré-Requisitos do Sistema Operacional
4. Oracle Grid Infrastructure 12cR1 - Parte 4 - Configurando DNS
5. Oracle Grid Infrastructure 12cR1 - Parte 5 - Pré-Requisitos do Grid Infrastructure
6. Oracle Grid Infrastructure 12cR1 - Parte 6 - Instalando o Grid Infrastructure
7. Oracle Grid Infrastructure 12cR1 - Parte 7 - Instalando o Database Software
8. Oracle Grid Infrastructure 12cR1 - Parte 8 - Criando Diskgroups no ASM
9. Oracle Grid Infrastructure 12cR1 - Parte 9 - Criando o Database
10.Oracle Grid Infrastructure 12cR1 - Parte 10 - Validações Finais
Executar os commandos abaixo com user “root”:
mkdir -p /u01/app/oraInventory chown -R grid:oinstall /u01/app/oraInventory chmod -R 775 /u01/app/oraInventory
Executar os commandos abaixo com user “root”:
mkdir -p /u01/app/12.1.0.1/grid chown -R grid:oinstall /u01/app/12.1.0.1/grid chmod -R 775 /u01/app/12.1.0.1/
Executar os commandos abaixo com user “root”:
mkdir -p /u01/app/oracle mkdir /u01/app/oracle/cfgtoollogs chown -R oracle:oinstall /u01/app/oracle chmod -R 775 /u01/app/oracle
Executar os commandos abaixo com user “root”:
mkdir -p /u01/app/oracle/product/12.1.0.1/dbhome_1 chown -R oracle:oinstall /u01/app/oracle/product/12.1.0.1/dbhome_1 chmod -R 775 /u01/app/oracle/product/12.1.0.1/dbhome_1
zlib-1.2.3-4.el5.i386.rpm unixODBC64-2.2.14-3.el5.x86_64.rpm sysstat-7.0.2-11.el5.x86_64.rpm pdksh-5.2.14-37.el5.x86_64.rpm numactl-0.9.8-12.0.1.el5_6.i386.rpm make-3.81-3.el5.x86_64.rpm libstdc++44-devel-4.4.6-3.el5.1.i386.rpm libobjc-4.1.2-52.el5.x86_64.rpm libgcj-devel-4.1.2-52.el5.i386.rpm libgcc-4.1.2-52.el5.x86_64.rpm libao-devel-0.8.6-7.x86_64.rpm kernel-xen-2.6.18-308.el5.x86_64.rpm kernel-headers-2.6.18-308.el5.x86_64.rpm glibc-common-2.5-81.x86_64.rpm gcc-4.1.2-52.el5.x86_64.rpm expat-devel-1.95.8-8.3.el5_5.3.x86_64.rpm elfutils-libs-0.137-3.el5.x86_64.rpm elfutils-0.137-3.el5.x86_64.rpm compat-libstdc++-296-2.96-138.i386.rpm elfutils-libelf-devel-static-0.137-3.el5.i386.rpm libao-0.8.6-7.i386.rpm expat-devel-1.95.8-8.3.el5_5.3.i386.rpm expat-1.95.8-8.3.el5_5.3.i386.rpm libaio-0.3.106-5.i386.rpm elfutils-devel-static-0.137-3.el5.i386.rpm elfutils-libelf-devel-0.137-3.el5.i386.rpm elfutils-libs-0.137-3.el5.i386.rpm libao-devel-0.8.6-7.i386.rpm elfutils-libelf-0.137-3.el5.i386.rpm libaio-devel-0.3.106-5.i386.rpm compat-libstdc++-33-3.2.3-61.x86_64.rpm elfutils-devel-static-0.137-3.el5.x86_64.rpm libao-0.8.6-7.x86_64.rpm libaio-0.3.106-5.x86_64.rpm numactl-devel-0.9.8-12.0.1.el5_6.i386.rpm libgcc-4.1.2-52.el5.i386.rpm libstdc++-devel-4.1.2-52.el5.i386.rpm libstdc++-4.1.2-52.el5.i386.rpm unixODBC64-libs-2.2.14-3.el5.i386.rpm unixODBC64-devel-2.2.14-3.el5.i386.rpm binutils-devel-2.17.50.0.6-20.el5.x86_64.rpm binutils-2.17.50.0.6-20.el5.x86_64.rpm gcc-objc++-4.1.2-52.el5.x86_64.rpm gcc-objc-4.1.2-52.el5.x86_64.rpm gcc-java-4.1.2-52.el5.x86_64.rpm gcc-c++-4.1.2-52.el5.x86_64.rpm libgcj-devel-4.1.2-52.el5.x86_64.rpm libstdc++-devel-4.1.2-52.el5.x86_64.rpm libstdc++44-devel-4.4.6-3.el5.1.x86_64.rpm glibc-devel-2.5-81.i386.rpm glibc-utils-2.5-81.x86_64.rpm glibc-headers-2.5-81.x86_64.rpm kernel-debug-2.6.18-308.el5.x86_64.rpm kernel-devel-2.6.18-308.el5.x86_64.rpm kernel-debug-devel-2.6.18-308.el5.x86_64.rpm kernel-xen-devel-2.6.18-308.el5.x86_64.rpm
Para fazer uma query dos pacotes acima, executar o shell abaixo: (32 ou 64 bits)
# rpm -q --qf '%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n' binutils \
compat-libstdc++-33 \
elfutils-libelf \
elfutils-libelf-devel \
gcc \
gcc-c++ \
glibc \
glibc-common \
glibc-devel \
glibc-headers \
ksh \
libaio \
libaio-devel \
libgcc \
libstdc++ \
libstdc++-devel \
make \
numactl \
pdksh \
sysstat \
unixODBC \
unixODBC-devel \
kernel-debug \
kernel-PAE \
kernel-devel \
kernel-xen \
kernel-xen-devel
Para instalação:
Inserir CD de instalação do OEL 5.8
# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda3 16G 3.5G 12G 24% / /dev/sdb1 20G 173M 19G 1% /u01 /dev/sda1 487M 42M 420M 9% /boot tmpfs 2.0G 0 2.0G 0% /dev/shm /dev/sr0 3.7G 3.7G 0 100% /media/OL5.8 x86_64 dvd 20120229
Criar repositório:
# cd /media # ln -s OL5.8\ x86_64\ dvd\ 20120229/ CDROM lrwxrwxrwx 1 root root 26 Jul 12 11:57 CDROM -> OL5.8 x86_64 dvd 20120229/ -rw-r--r-- 1 root root 73 Jul 12 11:56 .hal-mtab -rw------- 1 root root 0 Jul 12 11:56 .hal-mtab-lock drwxr-xr-x 8 root root 4096 Feb 29 2012 OL5.8 x86_64 dvd 20120229 cd /etc/yum.repos.d/ # cat repository.repo [oel5_x86_64_latest] name=Oracle OL 5 Linux $releasever - $basearch - latest baseurl=file:///media/CDROM/Server gpgcheck=0 enabled=0
Executar instalação:
yum --enablerepo=oel5_x86_64_latest install libao expat-devel expat libaio elfutils-devel-static elfutils-libelf-devel elfutils-libs libao-devel elfutils-libelf libaio-devel compat-libstdc elfutils-devel-static libao libaio numactl-devel libgcc libstdc++-devel libstdc++ unixODBC64-libs unixODBC64-devel binutils-devel binutils gcc-objc++ gcc-objc gcc-java gcc-c++ libgcj-devel libstdc++-devel libstdc++44-devel glibc-devel glibc-utils glibc-headers kernel-debug kernel-devel kernel-debug-devel kernel-xen-devel zlib unixODBC64 sysstat pdksh numactl make libstdc++ libobjc libgcj-devel libgcc libao-devel kernel-xen kernel-headers glibc-common gcc expat-devel elfutils-libs elfutils compat-libstdc++ elfutils-libelf-devel-static
su - oracle
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/bin
export PATH
# ORACLE RAC11g e DATABASE 12c
TMP=/tmp
TMPDIR=$TMP
TEMP=$TMP
export ORACLE_SID=oradb1
export ORACLE_BASE=/u01/app/oracle
export GRID_HOME=/u01/app/12.1.0.1/grid
export ORACLE_HOME=$ORACLE_BASE/product/12.1.0.1/dbhome_1
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export ORACLE_TERM=xterm
export PATH=$PATH:$ORACLE_HOME/bin:$PATH:$GRID_HOME/bin:/usr/bin:/sbin:/
usr/sbin:$ORACLE_HOME/OPatch
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib
export PS1="[${LOGNAME}@`hostname -s`:"'${PWD} [${ORACLE_SID}] ]$ '
export PERL_HOME=/usr/local/perl
export PERL5LIB=$PERL_HOME/lib
export SQLPATH=/home/oracle/scripts-ora
su - grid
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/bin
export PATH
# ORACLE RAC11g e DATABASE 12c
TMP=/tmp
TMPDIR=$TMP
TEMP=$TMP
export ORACLE_SID=+ASM1
export ORACLE_BASE=/u01/app/oracle
export GRID_HOME=/u01/app/12.1.0.1/grid
export ORACLE_HOME=$ORACLE_BASE/product/12.1.0.1/dbhome_1
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export ORACLE_TERM=xterm
export
PATH=$PATH:$ORACLE_HOME/bin:$PATH:$GRID_HOME/bin:/usr/bin:/sbin:/usr/
sbin:$ORACLE_HOME/OPatch
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib
export PS1="[${LOGNAME}@`hostname -s`:"'${PWD} [${ORACLE_SID}] ]$ '
export PERL_HOME=/usr/local/perl
export PERL5LIB=$PERL_HOME/lib
export SQLPATH=/home/oracle/scripts-ora
vi /etc/hosts #bond0 (eth0 + eth1)- PUBLIC 192.168.1.11racnode1.home.localracnode1 192.168.1.12racnode2.home.localracnode2 #VIP 192.168.1.21racnode1-vip.home.localracnode1-vip 192.168.1.22racnode2-vip.home.local racnode2-vip #eth2 - PRIVATE 10.1.0.1 racnode1-pvt 10.11.0.1racnode2-pvt #eth2 - PRIVATE #10.1.0.2 racnode1-pvt #10.11.0.2 racnode2-pvt #RAC SCAN #192.168.1.5rac-scan.home.local rac-scan #192.168.1.6 rac-scan.home.local rac-scan #192.168.1.7 rac-scan.home.local rac-scan #DNS - Gridserver 192.168.1.8 gridserver.home.local gridserver
Instalar o NTPD (/etc/init.d/ntpd)
rpm -ivh ntp*
manter o /etc/ntp.conf
editar o /etc/sysconfig/ntpd (acionar o "-x")
cat /etc/sysconfig/ntpd # Drop root to id 'ntp:ntp' by default. OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid" # Set to 'yes' to sync hw clock after successful ntpdate SYNC_HWCLOCK=no # Additional options for ntpdate NTPDATE_OPTIONS="" # service ntpd restart
O instalador do GRID / DATABASE possui este recurso, portanto neste momento é opcional.
#cd /home/oracle #mkdir ~/.ssh #chmod 700 ~/.ssh #ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/home/oracle/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Passphrases do not match. Try again. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/oracle/.ssh/id_rsa. Your public key has been saved in /home/oracle/.ssh/id_rsa.pub. The key fingerprint is: 30:f4:8f:b9:51:43:9b:65:0a:b6:36:cd:0e:a4:92:54 oracle@racnode1 #cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys #ssh racnode2 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys # cat .ssh/authorized_keys | ssh racnode2 "cat >> .ssh/authorized_keys"
Conectar em todos os host via ssh, através das interfaces públicas e privadas para popular o arquivo authorized_keys até que não sejam mais solicitadas confirmações pelo ssh.
Copiar todos os pacotes de instalação para o diretório /u01/install:
Efetuar download dos Binários do GRID + Database do link:
http://www.oracle.com/technetwork/database/enterprise-edition/downloads/database12c-linux-download-1959253.html
Nenhum Patch e Patchset Ainda disponível
Efetuar download do OPATCH 12.0.1.1 do note: 6880880
Efetuar download do htop (http://htop.sourceforge.net)
Efetuar download do YAST (https://oss.oracle.com/projects/yast/files/el5/x86_64/)
Efetuar download do INQ (ftp://ftp.emc.com/pub/elab/inquiry/v7.3.0.1)
O ASMLIB é uma biblioteca Oracle desenvolvida para identificar discos / luns ou partições para o ASM. Esta biblioteca facilita a administração dos discos para o Oracle Grid Infrastructure.
Local para Download dos drivers do ASMLIB:
http://www.oracle.com/technetwork/server-storage/linux/downloads/rhel5-084877.html
uname -r
** IMPORTANTE **
Devido a utilização do Kernel uek, deve-se apenas instalar o “Library and Tools” pois o driver do ASMLIB já está incluso no kernel.
oracleasm-support-2.1.8-1.el5.x86_64.rpm oracleasmlib-2.0.4-1.el5.x86_64.rpm
Copiar todos os arquivos de instalação para o Servidor Node 1
# mkdir –p /u01/install # chmod 777 /u01/install/ # chown oracle:oinstall /u01/install/
ASMLIBrpms
# mkdir –p /u01/install # rpm -Uvh oracleasm* warning: oracleasmlib-2.0.4-1.el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159 Preparing... ########################################### [100%] 1:oracleasm-support ########################################### [ 50%] 2:oracleasmlib ########################################### [100%]
Executar como “root”
# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.
This will configure the on-boot properties of the Oracle ASM library driver. The following questions will
determine whether the driver is loaded on boot and what permissions it will have. The current values will be
shown in brackets ('[]'). Hitting without typing an answer will keep that current value. Ctrl-C will abort. Default user to own the driver interface []: grid Default group to own the driver interface []: asmdba Start Oracle ASM library driver on boot (y/n) [n]: y Scan for Oracle ASM disks on boot (y/n) [y]: y Writing Oracle ASM library driver configuration: done Initializing the Oracle ASMLib driver: [ OK ] Scanning the system for Oracle ASMLib disks: [ OK ]
Particionando discos para o ASM
Executar como “root”
**IMPORTANTE**
Neste momento é importante observer que o exemplo abaixo somente mostra como é executada a criação de uma partição. Os tamanhos e devices são definidos conforme cada instalação.
# fdisk /dev/iscsi/openfiler-asm_ocr1/part
Command (m for help): p
Disk /dev/iscsi/openfiler-asm_ocr1/part: 2013 MB, 2013265920 bytes
62 heads, 62 sectors/track, 1022 cylinders
Units = cylinders of 3844 * 512 = 1968128 bytes
Device Boot Start End Blocks Id System
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1022, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-1022, default 1022):
Using default value 1022
Command (m for help): p
Disk /dev/iscsi/openfiler-asm_ocr1/part: 2013 MB, 2013265920 bytes
62 heads, 62 sectors/track, 1022 cylinders
Units = cylinders of 3844 * 512 = 1968128 bytes
Device Boot Start End Blocks Id System
/dev/iscsi/openfiler-asm_ocr1/part1 1 1022 1964253 83 Linux
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
Executar como “root”
#/usr/sbin/oracleasm createdisk disk_name device_partition_name
Exemplos:
#/usr/sbin/oracleasm createdisk ASMDATA01 /dev/iscsi/openfiler-asm_data1/part1
Writing disk header: done
Instantiating disk: done
#/usr/sbin/oracleasm createdisk ASMOCR01 /dev/iscsi/openfiler-asm_ocr1/part1
Writing disk header: done
Instantiating disk: done
Para excluir um disco:
/usr/sbin/oracleasm deletedisk disk_name
Após a Criação, fazer a listagem dos discos:
# /usr/sbin/oracleasm listdisks
ASMDATA01
ASMOCR01
Ir para o Node 2:
#/usr/sbin/oracleasm scandisks Reloading disk partitions: done Cleaning any stale ASM disks... Scanning system for ASM disks... Instantiating disk "ASMOCR01" Instantiating disk "ASMDATA01" # /usr/sbin/oracleasm listdisks ASMDATA01 ASMOCR01
# cd /u01/install # mkdir RAC_DB_12c # mv linuxamd64_12c_* RAC_DB_12c/ # cd RAC_DB_12c/ # chown oracle:oinstall -R RAC_DB_12c/ # unzip linuxamd64_12c_database_1of2.zip # unzip linuxamd64_12c_database_2of2.zip # su - grid # unzip linuxamd64_12c_grid_1of2.zip # unzip linuxamd64_12c_grid_2of2.zip
ValidaçãoPré-instalação
# su – grid # cd /u01/install/RAC_DB_12c/grid # ./runcluvfy.sh stage -pre crsinst -n racnode1,racnode2 # ./runcluvfy.sh stage -pre crsinst -n racnode1,racnode2 Performing pre-checks for cluster services setup Checking node reachability... Node reachability check passed from node "racnode1" Checking user equivalence... User equivalence check passed for user "grid" Checking node connectivity... Checking hosts config file... Verification of the hosts config file successful Node connectivity passed for subnet "10.11.0.0" with node(s) racnode1,racnode2 TCP connectivity check passed for subnet "10.11.0.0" Node connectivity passed for subnet "10.1.0.0" with node(s) racnode1,racnode2 TCP connectivity check passed for subnet "10.1.0.0" Node connectivity passed for subnet "192.168.1.0" with node(s) racnode1,racnode2 TCP connectivity check passed for subnet "192.168.1.0" Interfaces found on subnet "192.168.1.0" that are likely candidates for VIP are: racnode1 bond0:192.168.1.11 racnode2 bond0:192.168.1.12 Interfaces found on subnet "10.11.0.0" that are likely candidates for a private interconnect are: racnode1 eth2:10.11.0.1 racnode2 eth2:10.11.0.2 Interfaces found on subnet "10.1.0.0" that are likely candidates for a private interconnect are: racnode1 eth3:10.1.0.1 racnode2 eth3:10.1.0.2 Checking subnet mask consistency... Subnet mask consistency check passed for subnet "10.11.0.0". Subnet mask consistency check passed for subnet "10.1.0.0". Subnet mask consistency check passed for subnet "192.168.1.0". Subnet mask consistency check passed. Node connectivity check passed Checking multicast communication... Checking subnet "10.11.0.0" for multicast communication with multicast group "224.0.0.251"... Check of subnet "10.11.0.0" for multicast communication with multicast group "224.0.0.251" passed. Check of multicast communication passed. Checking ASMLib configuration. Check for ASMLib configuration passed. Total memory check passed Available memory check passed Swap space check passed Free disk space check passed for "racnode2:/usr,racnode2:/var,racnode2:/etc,racnode2:/sbin,racnode2:/tmp" Free disk space check passed for "racnode1:/usr,racnode1:/var,racnode1:/etc,racnode1:/sbin,racnode1:/tmp" Check for multiple users with UID value 501 passed User existence check passed for "grid" Group existence check passed for "oinstall" Group existence check passed for "dba" Membership check for user "grid" in group "oinstall" [as Primary] passed Membership check for user "grid" in group "dba" passed Run level check passed Hard limits check passed for "maximum open file descriptors" Soft limits check passed for "maximum open file descriptors" Hard limits check passed for "maximum user processes" Soft limits check passed for "maximum user processes" System architecture check passed Kernel version check passed Kernel parameter check passed for "semmsl" Kernel parameter check passed for "semmns" Kernel parameter check passed for "semopm" Kernel parameter check passed for "semmni" Kernel parameter check passed for "shmmax" Kernel parameter check passed for "shmmni" Kernel parameter check passed for "shmall" Kernel parameter check passed for "file-max" Kernel parameter check passed for "ip_local_port_range" Kernel parameter check passed for "rmem_default" Kernel parameter check passed for "rmem_max" Kernel parameter check passed for "wmem_default" Kernel parameter check passed for "wmem_max" Kernel parameter check passed for "aio-max-nr" Package existence check passed for "make" Package existence check passed for "binutils" Package existence check passed for "gcc(x86_64)" Package existence check passed for "libaio(x86_64)" Package existence check passed for "glibc(x86_64)" Package existence check passed for "compat-libstdc++-33(x86_64)" Package existence check passed for "glibc-devel(x86_64)" Package existence check passed for "gcc-c++(x86_64)" Package existence check passed for "libaio-devel(x86_64)" Package existence check passed for "libgcc(x86_64)" Package existence check passed for "libstdc++(x86_64)" Package existence check passed for "libstdc++-devel(x86_64)" Package existence check passed for "sysstat" Package existence check passed for "ksh" Package existence check passed for "nfs-utils" Checking availability of ports "6200,6100" required for component "Oracle Notification Service (ONS)" Port availability check passed for ports "6200,6100" Check for multiple users with UID value 0 passed Current group ID check passed Starting check for consistency of primary group of root user Check for consistency of root user's primary group passed Starting Clock synchronization checks using Network Time Protocol(NTP)... NTP Configuration file check started... NTP Configuration file check passed Checking daemon liveness... Liveness check passed for "ntpd" Check for NTP daemon or service alive passed on all nodes NTP common Time Server Check started... Check of common NTP Time Server passed Clock time offset check from NTP Time Server started... Clock time offset check passed Clock synchronization check using Network Time Protocol(NTP) passed Core file name pattern consistency check passed. User "grid" is not part of "root" group. Check passed Default user file creation mask check passed Checking integrity of file "/etc/resolv.conf" across nodes "domain" and "search" entries do not coexist in any "/etc/resolv.conf" file All nodes have same "search" order defined in file "/etc/resolv.conf" The DNS response time for an unreachable node is within acceptable limit on all nodes Check for integrity of file "/etc/resolv.conf" passed Time zone consistency check passed Checking integrity of name service switch configuration file "/etc/nsswitch.conf" ... All nodes have same "hosts" entry defined in file "/etc/nsswitch.conf" Check for integrity of name service switch configuration file "/etc/nsswitch.conf" passed Checking daemon "avahi-daemon" is not configured and running Daemon not configured check passed for process "avahi-daemon" Daemon not running check passed for process "avahi-daemon" Starting check for /dev/shm mounted as temporary file system ... Check for /dev/shm mounted as temporary file system passed Starting check for /boot mount ... Check for /boot mount passed Starting check for zeroconf check ... Check for zeroconf check passed Pre-check for cluster services setup was successful on all the nodes.
Victor Armbrust é DBA há 10 anos, especialista em Banco de Dados Oracle e Bacharel em Ciências da Computação. Com sólidos conhecimentos em Banco de Dados e Sistemas operacionais, possui certificações OCP 10g/11g. Eleito Oracle ACE Member em 2013. Consultor de Banco de Dados na Oracle ACS Brasil.