tauseefdbablogs: 02/08/18

Implementation and Configuration of RAC 11gR2 on Virtual Box

For virtual machine creation and linux installation please see my blog:

https://tauseefdba.blogspot.com/2018/02/linux-installation-on-virtual-box.html

After installation of OS please proceed with RAC implementation:

Making host file entry on both nodes!

127.0.0.1 localhost.localdomain localhost
# public IP - (eth:0)
192.168.1.101 racnode1 racnode1
192.168.1.102 racnode2 racnode2

# Private Interconnect - (eth1)
192.168.2.101 racnode1-priv racnode1-priv
192.168.2.102 racnode2-priv racnode2-priv

# Public Virtual IP (VIP) addresses - (eth0:1)
192.168.1.201 racnode1-vip racnode1-vip
192.168.1.202 racnode2-vip racnode2-vip

# Scan IP address
192.168.1.249 racnode-scan racnode-scan
192.168.1.250 racnode-scan racnode-scan
192.168.1.251 racnode-scan racnode-scan

~

/********************Setting Kernel Parameters**************************/

Setting kernal perameters on both nodes

[root@rac-node1 ~]# cat /etc/sysctl.conf

# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 4294967296
fs.suid_dumpable = 1
fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.shmall = 2097152
kernel.shmmax = 536870912
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048586
[root@rac-node1 ~]#

/**************To take effects these parameter********************/

Run below command on both nodes

[root@rac-node1 ~]# /sbin/sysctl -p

/********** Process Limits on both Oracle and Grid User (on both Nodes) **************/

oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 4096
oracle hard nofile 65536
oracle soft stack 10240
grid soft nproc 2047
grid hard nproc 16384
grid soft nofile 1024
grid hard nofile 65536

/*****************Now installing RPMS (on Both Nodes)**************************/
cd /media/Enterprise/Server

rpm -Uvh --force --nodeps binutils-2.*
rpm -Uvh --force --nodeps compat-libstdc++-33*
rpm -Uvh --force --nodeps compat-libstdc++-33*.i386.rpm
rpm -Uvh --force --nodeps elfutils-libelf*
rpm -Uvh --force --nodeps glibc-2.*
rpm -Uvh --force --nodeps glibc-common-2.*
rpm -Uvh --force --nodeps glibc-devel-2.*
rpm -Uvh --force --nodeps glibc-headers-2.*
rpm -Uvh --force --nodeps gcc-4.*
rpm -Uvh --force --nodeps gcc-c++-4.*
rpm -Uvh --force --nodeps ksh*
rpm -Uvh --force --nodeps libaio-0.*
rpm -Uvh --force --nodeps libaio-devel-0.*
rpm -Uvh --force --nodeps libgomp-4.*
rpm -Uvh --force --nodeps libgcc-4.*
rpm -Uvh --force --nodeps libstdc++-4.*
rpm -Uvh --force --nodeps libstdc++-devel-4.*
rpm -Uvh --force --nodeps make-3.*
rpm -Uvh --force --nodeps sysstat-7.*
rpm -Uvh --force --nodeps unixODBC-2.*
rpm -Uvh --force --nodeps unixODBC-devel-2.*
rpm -Uvh --force --nodeps numactl-devel-*

/*********************** Create users and groups: ******************************/

groupadd -g 501 oinstall
groupadd -g 502 dba
groupadd -g 503 oper
groupadd -g 504 asmadmin
groupadd -g 506 asmdba
groupadd -g 507 asmoper

/usr/sbin/useradd -g oinstall -G asmadmin,asmdba,asmoper,dba -d /home/grid -m grid

/usr/sbin/useradd -g oinstall -G dba,asmdba,oper -d /home/oracle -m oracle

/************************ Set new password for oracle user: **************************/

passwd grid
passwd oracle

/*********** Create the Oracle Inventory Directory (both nodes) *****************/

To create the Oracle Inventory directory, enter the following commands as the root user:

mkdir -p /u01/app/oraInventory
chown -R grid:oinstall /u01/app/oraInventory
chmod -R 775 /u01/app/oraInventory

/********** Create the Oracle Grid Infrastructure Home Directory ***************/

To create the Grid Infrastructure home and base directories, enter the following commands as the root user:

mkdir -p /u01/app/11.2.0/grid
chown -R grid:oinstall /u01/app/11.2.0/grid
chmod -R 775 /u01/app/11.2.0/grid

mkdir -p /u01/app/grid
chown -R grid:oinstall /u01/app/grid
chmod -R 775 /u01/app/grid

/******************* Create the Oracle Base Directory *******************/

To create the Oracle Base directory, enter the following commands as the root user:

mkdir -p /u01/app/oracle
mkdir /u01/app/oracle/cfgtoollogs
chown -R oracle:oinstall /u01/app/oracle
chmod -R 775 /u01/app/oracle

/******************* Create the Oracle RDBMS Home Directory *******************/

To create the Oracle RDBMS Home directory, enter the following commands as the root user:

mkdir -p /u01/app/oracle/product/11.2.0
chown -R oracle:oinstall /u01/app/oracle/product/11.2.0
chmod -R 775 /u01/app/oracle/product/11.2.0

Setting Home variables for Grid User

Edit the ‘/home/grid/.bash_profile’ file on first Oracle RAC Node for Oracle Grid Infrastructure Installation:

# Oracle 11gR2 Grid Infrastructure environment

export TMP=/tmp
export TMPDIR=$TMP
export ORACLE_BASE=/u01/app/grid
export ORACLE_HOME=/u01/app/11.2.0/grid
export PATH=$ORACLE_HOME/bin:$PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib

/***********Unzipping the Oracle Media and Grid media ************************/

cd /u01
unizip linux_11gr2_grid.zip

unzip linux_11gr2_database_1.zip
unzip linux_11gr2_database_2.zip

/******************Now partitioning the disks for ASM ********************/

fdisk -l

fdisk /dev/sdb

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-261, default 1): 1
Last cylinder, +cylinders or +size{K,M,G} (1-261, default 261):
Using default value 261

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

[root@rac-node1 Oracle 11g R2]#

-- We will create partition on 1 node and it will be done on 2nd automatically

run command

/sbin/partprobe

/*************** Installing RPM for ASMLIB *****************************/

a oracleasm-support-2.1.7-1.el5.i386.rpm
warning: oracleasm-support-2.1.7-1.el5.i386.rpm: Header V3 DSA/SHA1 Signature, key ID 1e5e0159: NOKEY
Preparing... ########################################### [100%]
1:oracleasm-support ########################################### [100%]
rpm -ivh kmod-oracleasm-2.0.6.rh1-2.el6.x86_64.rpm
warning: kmod-oracleasm-2.0.6.rh1-2.el6.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID fd431d51: NO
Preparing... ########################################### [100%]
1:kmod-oracleasm ########################################### [100%]
rpm -Uvh --force --nodeps oracleasmlib-2.0.4-1.el5.i386.rpm
warning: oracleasmlib-2.0.4-1.el5.i386.rpm: Header V3 DSA/SHA1 Signature, key ID 1e5e0159: NOKEY
Preparing... ########################################### [100%]
1:oracleasmlib ########################################### [100%]

/**************************** Configuring ASMLIB ********************************/

[root@rac-node1 u01]# /etc/init.d/oracleasm configure i
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting <ENTER> without typing an
answer will keep that current value. Ctrl-C will abort.

Default user to own the driver interface [grid]:
Default group to own the driver interface [asmadmin]:
Start Oracle ASM library driver on boot (y/n) [y]:
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver: [ OK ]
Scanning the system for Oracle ASMLib disks: [ OK ]

/********************************Creating ASM Disks **************************/

[root@rac-node1 ~]# /etc/init.d/oracleasm createdisk ocr1 /dev/sdb1
Marking disk "ocr1" as an ASM disk: [ OK ]
[root@rac-node1 ~]# /etc/init.d/oracleasm createdisk ocr2 /dev/sdc1
Marking disk "ocr2" as an ASM disk: [ OK ]
[root@rac-node1 ~]# /etc/init.d/oracleasm createdisk disk1 /dev/sdd1
Marking disk "disk1" as an ASM disk: [FAILED]
[root@rac-node1 ~]# /etc/init.d/oracleasm createdisk disk1 /dev/sde1
Marking disk "disk1" as an ASM disk: [ OK ]
[root@rac-node1 ~]# /etc/init.d/oracleasm createdisk disk2 /dev/sdf1
Marking disk "disk2" as an ASM disk: [ OK ]
[root@rac-node1 ~]# /etc/init.d/oracleasm createdisk disk3 /dev/sdg1
Marking disk "disk3" as an ASM disk: [ OK ]
[root@rac-node1 ~]# /etc/init.d/oracleasm createdisk disk4 /dev/sdh1
Marking disk "disk4" as an ASM disk: [ OK ]
[root@rac-node1 ~]# /etc/init.d/oracleasm createdisk disk5 /dev/sdi1
Marking disk "disk5" as an ASM disk: [ OK ]
[root@rac-node1 ~]#

-- Scanning the disks from 2nd node

[root@rac-node2 ~]# /etc/init.d/oracleasm listdisks
[root@rac-node2 ~]#
[root@rac-node2 ~]# /etc/init.d/oracleasm scandisks
Scanning the system for Oracle ASMLib disks: [ OK ]
[root@rac-node2 ~]#
[root@rac-node2 ~]# /etc/init.d/oracleasm listdisks
DISK1
DISK2
DISK4
DISK5
OCR1
OCR2
[root@rac-node2 ~]#

/*****************Now we are all set for running Cluster verification Utility**************/

[grid@racnode1 ~]$ cd /u01/grid/
[grid@racnode1 grid]$ ./runcluvfy.sh stage -pre crsinst -n racnode1,racnode2 -verbose

Performing pre-checks for cluster services setup

Checking node reachability...

Check: Node reachability from node "racnode1"
Destination Node Reachable?
------------------------------------ ------------------------
racnode1 yes
racnode2 yes
Result: Node reachability check passed from node "racnode1"

Checking user equivalence...

Check: User equivalence for user "grid"
Node Name Comment
------------------------------------ ------------------------
racnode2 passed
racnode1 passed
Result: User equivalence check passed for user "grid"

Checking node connectivity...

Checking hosts config file...
Node Name Status Comment
------------ ------------------------ ------------------------
racnode2 passed
racnode1 passed

Verification of the hosts config file successful

Interface information for node "racnode2"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
eth0 192.168.1.102 192.168.1.0 0.0.0.0 192.168.1.1 08:00:27:D1:4A:E1 1500
eth1 192.168.2.102 192.168.2.0 0.0.0.0 192.168.1.1 08:00:27:C6:90:E5 1500

Interface information for node "racnode1"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
eth0 192.168.1.101 192.168.1.0 0.0.0.0 192.168.1.1 08:00:27:D9:7F:5C 1500
eth1 192.168.2.101 192.168.2.0 0.0.0.0 192.168.1.1 08:00:27:E3:9E:EF 1500

Check: Node connectivity of subnet "192.168.1.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
racnode2:eth0 racnode1:eth0 yes
Result: Node connectivity passed for subnet "192.168.1.0" with node(s) racnode2,racnode1

Check: TCP connectivity of subnet "192.168.1.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
racnode1:192.168.1.101 racnode2:192.168.1.102 passed
Result: TCP connectivity check passed for subnet "192.168.1.0"

Check: Node connectivity of subnet "192.168.2.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
racnode2:eth1 racnode1:eth1 yes
Result: Node connectivity passed for subnet "192.168.2.0" with node(s) racnode2,racnode1

Check: TCP connectivity of subnet "192.168.2.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
racnode1:192.168.2.101 racnode2:192.168.2.102 passed
Result: TCP connectivity check passed for subnet "192.168.2.0"

Interfaces found on subnet "192.168.1.0" that are likely candidates for VIP are:
racnode2 eth0:192.168.1.102
racnode1 eth0:192.168.1.101

Interfaces found on subnet "192.168.2.0" that are likely candidates for a private interconnect are:
racnode2 eth1:192.168.2.102
racnode1 eth1:192.168.2.101

Result: Node connectivity check passed

Check: Total memory
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 3.96GB (4152312.0KB) 1.5GB (1572864.0KB) passed
racnode1 3.96GB (4152312.0KB) 1.5GB (1572864.0KB) passed
Result: Total memory check passed

Check: Available memory
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 3.86GB (4043380.0KB) 50MB (51200.0KB) passed
racnode1 3.77GB (3950288.0KB) 50MB (51200.0KB) passed
Result: Available memory check passed

Check: Swap space
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 7.9GB (8281496.0KB) 3.96GB (4152312.0KB) passed
racnode1 7.9GB (8281496.0KB) 3.96GB (4152312.0KB) passed
Result: Swap space check passed

Check: Free disk space for "racnode2:/tmp"
Path Node Name Mount point Available Required Comment
---------------- ------------ ------------ ------------ ------------ ------------
/tmp racnode2 /tmp 6.47GB 1GB passed
Result: Free disk space check passed for "racnode2:/tmp"

Check: Free disk space for "racnode1:/tmp"
Path Node Name Mount point Available Required Comment
---------------- ------------ ------------ ------------ ------------ ------------
/tmp racnode1 /tmp 6.46GB 1GB passed
Result: Free disk space check passed for "racnode1:/tmp"

Check: User existence for "grid"
Node Name Status Comment
------------ ------------------------ ------------------------
racnode2 exists passed
racnode1 exists passed
Result: User existence check passed for "grid"

Check: Group existence for "oinstall"
Node Name Status Comment
------------ ------------------------ ------------------------
racnode2 exists passed
racnode1 exists passed
Result: Group existence check passed for "oinstall"

Check: Group existence for "dba"
Node Name Status Comment
------------ ------------------------ ------------------------
racnode2 exists passed
racnode1 exists passed
Result: Group existence check passed for "dba"

Check: Membership of user "grid" in group "oinstall" [as Primary]
Node Name User Exists Group Exists User in Group Primary Comment
---------------- ------------ ------------ ------------ ------------ ------------
racnode2 yes yes yes yes passed
racnode1 yes yes yes yes passed
Result: Membership check for user "grid" in group "oinstall" [as Primary] passed

Check: Membership of user "grid" in group "dba"
Node Name User Exists Group Exists User in Group Comment
---------------- ------------ ------------ ------------ ----------------
racnode2 yes yes yes passed
racnode1 yes yes yes passed
Result: Membership check for user "grid" in group "dba" passed

Check: Run level
Node Name run level Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 5 3,5 passed
racnode1 5 3,5 passed
Result: Run level check passed

Check: Hard limits for "maximum open file descriptors"
Node Name Type Available Required Comment
---------------- ------------ ------------ ------------ ----------------
racnode2 hard 65536 65536 passed
racnode1 hard 65536 65536 passed
Result: Hard limits check passed for "maximum open file descriptors"

Check: Soft limits for "maximum open file descriptors"
Node Name Type Available Required Comment
---------------- ------------ ------------ ------------ ----------------
racnode2 soft 1024 1024 passed
racnode1 soft 1024 1024 passed
Result: Soft limits check passed for "maximum open file descriptors"

Check: Hard limits for "maximum user processes"
Node Name Type Available Required Comment
---------------- ------------ ------------ ------------ ----------------
racnode2 hard 16384 16384 passed
racnode1 hard 16384 16384 passed
Result: Hard limits check passed for "maximum user processes"

Check: Soft limits for "maximum user processes"
Node Name Type Available Required Comment
---------------- ------------ ------------ ------------ ----------------
racnode2 soft 2047 2047 passed
racnode1 soft 2047 2047 passed
Result: Soft limits check passed for "maximum user processes"

Check: System architecture
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 i686 x86 passed
racnode1 i686 x86 passed
Result: System architecture check passed

Check: Kernel version
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 2.6.18-92.el5PAE 2.6.18 passed
racnode1 2.6.18-92.el5PAE 2.6.18 passed
Result: Kernel version check passed

Check: Kernel parameter for "semmsl"
Node Name Configured Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 250 250 passed
racnode1 250 250 passed
Result: Kernel parameter check passed for "semmsl"

Check: Kernel parameter for "semmns"
Node Name Configured Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 32000 32000 passed
racnode1 32000 32000 passed
Result: Kernel parameter check passed for "semmns"

Check: Kernel parameter for "semopm"
Node Name Configured Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 100 100 passed
racnode1 100 100 passed
Result: Kernel parameter check passed for "semopm"

Check: Kernel parameter for "semmni"
Node Name Configured Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 128 128 passed
racnode1 128 128 passed
Result: Kernel parameter check passed for "semmni"

Check: Kernel parameter for "shmmax"
Node Name Configured Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 536870912 536870912 passed
racnode1 536870912 536870912 passed
Result: Kernel parameter check passed for "shmmax"

Check: Kernel parameter for "shmmni"
Node Name Configured Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 4096 4096 passed
racnode1 4096 4096 passed
Result: Kernel parameter check passed for "shmmni"

Check: Kernel parameter for "shmall"
Node Name Configured Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 2097152 2097152 passed
racnode1 2097152 2097152 passed
Result: Kernel parameter check passed for "shmall"

Check: Kernel parameter for "file-max"
Node Name Configured Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 6815744 6815744 passed
racnode1 6815744 6815744 passed
Result: Kernel parameter check passed for "file-max"

Check: Kernel parameter for "ip_local_port_range"
Node Name Configured Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 between 9000 & 65500 between 9000 & 65500 passed
racnode1 between 9000 & 65500 between 9000 & 65500 passed
Result: Kernel parameter check passed for "ip_local_port_range"

Check: Kernel parameter for "rmem_default"
Node Name Configured Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 262144 262144 passed
racnode1 262144 262144 passed
Result: Kernel parameter check passed for "rmem_default"

Check: Kernel parameter for "rmem_max"
Node Name Configured Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 4194304 4194304 passed
racnode1 4194304 4194304 passed
Result: Kernel parameter check passed for "rmem_max"

Check: Kernel parameter for "wmem_default"
Node Name Configured Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 262144 262144 passed
racnode1 262144 262144 passed
Result: Kernel parameter check passed for "wmem_default"

Check: Kernel parameter for "wmem_max"
Node Name Configured Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 1048586 1048576 passed
racnode1 1048586 1048576 passed
Result: Kernel parameter check passed for "wmem_max"

Check: Kernel parameter for "aio-max-nr"
Node Name Configured Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 1048576 1048576 passed
racnode1 1048576 1048576 passed
Result: Kernel parameter check passed for "aio-max-nr"

Check: Package existence for "make-3.81"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 make-3.81-3.el5 make-3.81 passed
racnode1 make-3.81-3.el5 make-3.81 passed
Result: Package existence check passed for "make-3.81"

Check: Package existence for "binutils-2.17.50.0.6"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 binutils-2.17.50.0.6-6.el5 binutils-2.17.50.0.6 passed
racnode1 binutils-2.17.50.0.6-6.el5 binutils-2.17.50.0.6 passed
Result: Package existence check passed for "binutils-2.17.50.0.6"

Check: Package existence for "gcc-4.1.2"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 gcc-4.1.2-42.el5 gcc-4.1.2 passed
racnode1 gcc-4.1.2-42.el5 gcc-4.1.2 passed
Result: Package existence check passed for "gcc-4.1.2"

Check: Package existence for "gcc-c++-4.1.2"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 gcc-c++-4.1.2-42.el5 gcc-c++-4.1.2 passed
racnode1 gcc-c++-4.1.2-42.el5 gcc-c++-4.1.2 passed
Result: Package existence check passed for "gcc-c++-4.1.2"

Check: Package existence for "libgomp-4.1.2"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 libgomp-4.1.2-42.el5 libgomp-4.1.2 passed
racnode1 libgomp-4.1.2-42.el5 libgomp-4.1.2 passed
Result: Package existence check passed for "libgomp-4.1.2"

Check: Package existence for "libaio-0.3.106"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 libaio-0.3.106-3.2 libaio-0.3.106 passed
racnode1 libaio-0.3.106-3.2 libaio-0.3.106 passed
Result: Package existence check passed for "libaio-0.3.106"

Check: Package existence for "glibc-2.5-24"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 glibc-2.5-24 glibc-2.5-24 passed
racnode1 glibc-2.5-24 glibc-2.5-24 passed
Result: Package existence check passed for "glibc-2.5-24"

Check: Package existence for "compat-libstdc++-33-3.2.3"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 compat-libstdc++-33-3.2.3-61 compat-libstdc++-33-3.2.3 passed
racnode1 compat-libstdc++-33-3.2.3-61 compat-libstdc++-33-3.2.3 passed
Result: Package existence check passed for "compat-libstdc++-33-3.2.3"

Check: Package existence for "elfutils-libelf-0.125"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 elfutils-libelf-0.125-3.el5 elfutils-libelf-0.125 passed
racnode1 elfutils-libelf-0.125-3.el5 elfutils-libelf-0.125 passed
Result: Package existence check passed for "elfutils-libelf-0.125"

Check: Package existence for "elfutils-libelf-devel-0.125"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 elfutils-libelf-devel-0.125-3.el5 elfutils-libelf-devel-0.125 passed
racnode1 elfutils-libelf-devel-0.125-3.el5 elfutils-libelf-devel-0.125 passed
Result: Package existence check passed for "elfutils-libelf-devel-0.125"

Check: Package existence for "glibc-common-2.5"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 glibc-common-2.5-24 glibc-common-2.5 passed
racnode1 glibc-common-2.5-24 glibc-common-2.5 passed
Result: Package existence check passed for "glibc-common-2.5"

Check: Package existence for "glibc-devel-2.5"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 glibc-devel-2.5-24 glibc-devel-2.5 passed
racnode1 glibc-devel-2.5-24 glibc-devel-2.5 passed
Result: Package existence check passed for "glibc-devel-2.5"

Check: Package existence for "glibc-headers-2.5"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 glibc-headers-2.5-24 glibc-headers-2.5 passed
racnode1 glibc-headers-2.5-24 glibc-headers-2.5 passed
Result: Package existence check passed for "glibc-headers-2.5"

Check: Package existence for "libaio-devel-0.3.106"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 libaio-devel-0.3.106-3.2 libaio-devel-0.3.106 passed
racnode1 libaio-devel-0.3.106-3.2 libaio-devel-0.3.106 passed
Result: Package existence check passed for "libaio-devel-0.3.106"

Check: Package existence for "libgcc-4.1.2"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 libgcc-4.1.2-42.el5 libgcc-4.1.2 passed
racnode1 libgcc-4.1.2-42.el5 libgcc-4.1.2 passed
Result: Package existence check passed for "libgcc-4.1.2"

Check: Package existence for "libstdc++-4.1.2"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 libstdc++-4.1.2-42.el5 libstdc++-4.1.2 passed
racnode1 libstdc++-4.1.2-42.el5 libstdc++-4.1.2 passed
Result: Package existence check passed for "libstdc++-4.1.2"

Check: Package existence for "libstdc++-devel-4.1.2"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 libstdc++-devel-4.1.2-42.el5 libstdc++-devel-4.1.2 passed
racnode1 libstdc++-devel-4.1.2-42.el5 libstdc++-devel-4.1.2 passed
Result: Package existence check passed for "libstdc++-devel-4.1.2"

Check: Package existence for "sysstat-7.0.2"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 sysstat-7.0.2-1.el5 sysstat-7.0.2 passed
racnode1 sysstat-7.0.2-1.el5 sysstat-7.0.2 passed
Result: Package existence check passed for "sysstat-7.0.2"

Check: Package existence for "unixODBC-2.2.11"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 unixODBC-2.2.11-7.1 unixODBC-2.2.11 passed
racnode1 unixODBC-2.2.11-7.1 unixODBC-2.2.11 passed
Result: Package existence check passed for "unixODBC-2.2.11"

Check: Package existence for "unixODBC-devel-2.2.11"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 unixODBC-devel-2.2.11-7.1 unixODBC-devel-2.2.11 passed
racnode1 unixODBC-devel-2.2.11-7.1 unixODBC-devel-2.2.11 passed
Result: Package existence check passed for "unixODBC-devel-2.2.11"

Check: Package existence for "ksh-20060214"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 ksh-20060214-1.7 ksh-20060214 passed
racnode1 ksh-20060214-1.7 ksh-20060214 passed
Result: Package existence check passed for "ksh-20060214"

Checking for multiple users with UID value 0
Result: Check for multiple users with UID value 0 passed

Check: Current group ID
Result: Current group ID check passed
Checking Core file name pattern consistency...
Core file name pattern consistency check passed.

Checking to make sure user "grid" is not in "root" group
Node Name Status Comment
------------ ------------------------ ------------------------
racnode2 does not exist passed
racnode1 does not exist passed
Result: User "grid" is not part of "root" group. Check passed

Check default user file creation mask
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
racnode2 0022 0022 passed
racnode1 0022 0022 passed
Result: Default user file creation mask check passed

Starting Clock synchronization checks using Network Time Protocol(NTP)...

NTP Configuration file check started...
The NTP configuration file "/etc/ntp.conf" is available on all nodes
NTP Configuration file check passed

Checking daemon liveness...

Check: Liveness for "ntpd"
Node Name Running?
------------------------------------ ------------------------
racnode2 no
racnode1 no
Result: Liveness check failed for "ntpd"
PRVF-5415 : Check to see if NTP daemon is running failed
Result: Clock synchronization check using Network Time Protocol(NTP) failed

Pre-check for cluster services setup was unsuccessful on all the nodes.
[grid@racnode1 grid]$

As we are set to go for Installation GI for cluster :

Specifying the cluster name and scan name, which should be same name as mentioned in scan IP.

Making the ssh password less connectivity:

Defining the second Node and it VIP:

Defining the public network IP and Private network IP

Clusterware to be deployed on ASM or file system:

Making a Diskgroup for OCR and Voting Disk, with normal redundancy it should be 3 disks atleast for OCR and voting disks:

Setting password for sysasm;

Checking the pre-requisites:

Installation started:

/****************running root scripts********************************/

[root@racnode1 ~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
[root@racnode1 ~]# /u01/app/11.2.0/grid/root.sh
Running Oracle 11g root.sh script...

The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...

Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2018-02-07 15:15:56: Parsing the host name
2018-02-07 15:15:56: Checking for super user privileges
2018-02-07 15:15:56: User has super user privileges
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
root wallet
root wallet cert
root cert export
peer wallet
profile reader wallet
pa wallet
peer wallet keys
pa wallet keys
peer cert request
pa cert request
peer cert
pa cert
peer root cert TP
profile reader root cert TP
pa root cert TP
peer pa cert TP
pa peer cert TP
profile reader pa cert TP
profile reader peer cert TP
peer user cert
pa user cert
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
CRS-2672: Attempting to start 'ora.gipcd' on 'racnode1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'racnode1'
CRS-2676: Start of 'ora.mdnsd' on 'racnode1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'racnode1'
CRS-2676: Start of 'ora.gpnpd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'racnode1'
CRS-2676: Start of 'ora.cssdmonitor' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'racnode1'
CRS-2672: Attempting to start 'ora.diskmon' on 'racnode1'
CRS-2676: Start of 'ora.diskmon' on 'racnode1' succeeded
CRS-2676: Start of 'ora.cssd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'racnode1'
CRS-2676: Start of 'ora.ctssd' on 'racnode1' succeeded

ASM created and started successfully.

DiskGroup CLUSTER_GROUP created successfully.

clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-2672: Attempting to start 'ora.crsd' on 'racnode1'
CRS-2676: Start of 'ora.crsd' on 'racnode1' succeeded
CRS-4256: Updating the profile
Successful addition of voting disk 36a9d100988d4f52bf586718f68b72eb.
Successful addition of voting disk 96c0e9da54c04fedbfaf137ea545bdf8.
Successful addition of voting disk 7b03c02ed0ed4ffdbf9013af4158097f.
Successfully replaced voting disk group with +CLUSTER_GROUP.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 36a9d100988d4f52bf586718f68b72eb (ORCL:OCR1) [CLUSTER_GROUP]
2. ONLINE 96c0e9da54c04fedbfaf137ea545bdf8 (ORCL:OCR2) [CLUSTER_GROUP]
3. ONLINE 7b03c02ed0ed4ffdbf9013af4158097f (ORCL:OCR3) [CLUSTER_GROUP]
Located 3 voting disk(s).
CRS-2673: Attempting to stop 'ora.crsd' on 'racnode1'
CRS-2677: Stop of 'ora.crsd' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'racnode1'
CRS-2677: Stop of 'ora.asm' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'racnode1'
CRS-2677: Stop of 'ora.ctssd' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'racnode1'
CRS-2677: Stop of 'ora.cssdmonitor' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'racnode1'
CRS-2677: Stop of 'ora.cssd' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'racnode1'
CRS-2677: Stop of 'ora.gpnpd' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'racnode1'
CRS-2677: Stop of 'ora.gipcd' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'racnode1'
CRS-2677: Stop of 'ora.mdnsd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.mdnsd' on 'racnode1'
CRS-2676: Start of 'ora.mdnsd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'racnode1'
CRS-2676: Start of 'ora.gipcd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'racnode1'
CRS-2676: Start of 'ora.gpnpd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'racnode1'
CRS-2676: Start of 'ora.cssdmonitor' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'racnode1'
CRS-2672: Attempting to start 'ora.diskmon' on 'racnode1'
CRS-2676: Start of 'ora.diskmon' on 'racnode1' succeeded
CRS-2676: Start of 'ora.cssd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'racnode1'
CRS-2676: Start of 'ora.ctssd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'racnode1'
CRS-2676: Start of 'ora.asm' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'racnode1'
CRS-2676: Start of 'ora.crsd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'racnode1'
CRS-2676: Start of 'ora.evmd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'racnode1'
CRS-2676: Start of 'ora.asm' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.CLUSTER_GROUP.dg' on 'racnode1'
CRS-2676: Start of 'ora.CLUSTER_GROUP.dg' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.registry.acfs' on 'racnode1'
CRS-2676: Start of 'ora.registry.acfs' on 'racnode1' succeeded

racnode1 2018/02/07 15:20:33 /u01/app/11.2.0/grid/cdata/racnode1/backup_20180207_152033.olr
Preparing packages for installation...
cvuqdisk-1.0.7-1
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Updating inventory properties for clusterware
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB. Actual 8087 MB Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oraInventory
'UpdateNodeList' was successful.
[root@racnode1 ~]#

Ignore the below error message:

checking the cluster ware services which are running:

/********************run script before DB installation*****************************/

/u01/app/11.2.0/grid/bin/cluvfy stage -pre dbinst -fixup -n racnode1,racnode2 -osdba dba -verbose

/****************After that run Installer for SW installation**********************/

Once all done, now its the time for Oracle SW installtion and DB creation, login through oracle user and set environment variables, aud executes runInstaller:

Install DB software only:

As we need DB software to be installed on all nodes:

Setting password less connectivity:

Selecting the advance features if requred:

Defining the oracle base and oracle home:

Checking the pre-requisites:

Starting the SW installation:

Run the root scripts:

/******************ASM Disk Group Creation for Datafiles and FRA********************/

Creating ASM DG for datafiles and FRA:

Specifying the Disks of which you want to create the disk group:

Disk groups created!!!

/******************************Database Creation**********************************/
Software has installed in oracle base, now is the time to create the DB according to parameter required, steps are as below:

Setting the environment for Database Variable:

Run database configuration assistant: