Monday 23 May 2016

11g RAC Step By Steps part 1



This Document shows the step by step of installing and setting up 2-Node E-Business Suites R 12.1.1 running on 11gR1 RAC database managed by 11gR2 Grid Infrastructure.
Below installation roadmap was followed (in sequence) to set up Multi-Node EBS R12 Application running on Multi-Node RAC Database.

Prepare the DB Tier for the EBS R12 Installation:

(a)    Install and Setup 11gR2 (11.2.0.1) Grid Infrastructure on 3-node cluster.
(b)   Create the ASM Diskgroups DATA and FRA with sufficient size.
(c)    Install the 11gR1 RAC Software for the Database Home (11.1.0.6) on this Cluster.
(d)   Install 11gR1 (11.1.0.6) Example Software to the 11gR1 DB Home.
(e)   Apply the 11gR1 (11.1.0.7) PatchSet on top of the 11.1.0.6 DB Home.

Prepare the Apps Tier for the EBS R12 Installation:

(f)     Prepare the Application Nodes (Pre-Install Tasks) for Installation.

Install the EBS R12.1.1:

(g)    Install EBS R12.1.1 using Rapid Install on Database Node 1 (Out of the BOX Non-RAC installation on RAC Node 1).
(h)   Install EBS R12.1.1 using Rapid Install on Apps Node 1.
(i)      Install EBS R12.1.1 using Rapid Install on Apps Node 2.

Apply the patch EBS Techstack 12.1.3:

(j)     Apply the EBS Techstack 12.1.3 Product Release Update Patch (9819489) on the Apps Nodes.

Convert non-RAC database to RAC database:

(k)    Convert the non-RAC 11gR1 database to RAC Database.

Enable AutoConfig on DB Tier and run autoconfig on DB as well as APPS Tier:

(l)      Enable and run the Autoconfig on the DB Tier for the 11gR1 RAC HOME.
(m) Run the AutoConfig on the APPS Tier to generate the new RAC enabled connect strings.
(n)   Take the Complete Backup of the Entire System.

Verify the Installation:

(o)   Check whether the enabled services on application nodes are running appropriately.
(p)   Login to the application Login Page to see if login to the application is successful.
(q)   Open Forms (like Concurrent Manager) to see if forms are opening up successfully.
(r)     Verify the Configuration in OAM.


Final Configuration Files of this Installation:

(s)    listnener.ora, tnsnames.ora, sqlnet.ora and their ifiles from all the RAC nodes.
(t)     listnener.ora and tnsnames.ora from both the application nodes.
(u)   CONTEXT_FILE from all the 5 nodes.
(v)    Pfile of the RAC database.



Multi-Node Apps Tier (2-Node) /Multi-Node DB Tier (3-Node RAC) Setup Information




Node Type
NAME
Version
Function
Application Node1
appsnode1.hingu.net
Oracle EBS 12.1.1
WEB Node
Application Node2
appsnode2.hingu.net
Oracle EBS 12.1.1
Concurrent Node (Batch Processing)
RAC DB Node1
node1.hingu.net
Oracle RDBMS 11.1.0.7
Clustered DB node1
RAC DB Node2
node2.hingu.net
Oracle RDBMS 11.1.0.7
Clustered DB node2
RAC DB Node3
node3.hingu.net
Oracle RDBMS 11.1.0.7
Clustered DB node3
DNS Server
lab-dns.hingu.net
OEL 5.5
DNS





               






Apps Tier
E-Business Suite
Oracle EBS R12.1.1
BASE Directory
/u01/app/oracle/ebsR12
ORACLE_HOME
/u01/app/oracle/ebsR12/apps/tech_st/10.1.2
IAS_ORACLE_HOME
/u01/app/oracle/ebsR12/apps/tech_st/10.1.3
APPL_TOP
/u01/app/oracle/ebsR12/apps/apps_st/appl
INST_TOP
/u01/app/oracle/ebsR12/inst/apps/<CONTEXT_NAME>
Port Pool
1
Application OS Owner
applmgr:(oinstall, dba)
OS Platform
Oracle Enterprise Linux 5.5 (32 bit)



DB Tier
RDBMS
Oracle 11gR1 RAC (11.1.0.7)
ORACLE_BASE
/u01/app/oracle
ORACLE_HOME
/u01/app/oracle/EBSR12
DB_HOME Owner
oracle:(oinstall, dba)
DB LISTENER
LABDB_LISTENER
DB Listener Host/port
node1-vip, node2-vip, node3-vip (TCP:1522)
DB Storage Type, File Management
ASM with OMFs
ASM diskgroups for DB and FRA
DATA (60GB), FRA (10GB)
OS Platform
Oracle Enterprise Linux 5.5 (32 bit)
Clusterware
Oracle 11gR2 Grid Infrastructure 11.2.0.1
ASM
Oracle 11gR2 Grid Infrastructure 11.2.0.1
Cluster Name
lab
Database Cluster Nodes
node1, node2, node3
Clusterware Home
/u01/app/grid11201 (GRID_HOME)
Clusterware Owner
oracle:(oinstall, dba)
VIPs
node1-vip, node2-vip, node3-vip
SCAN
lab-scan.hingu.net
SCAN VIPs
3 SCAN VIPs resolving lab-scan.hingu.net by DNS
SCAN_LISTENER Host/port
Scan VIPs Endpoint: (TCP:1525)
OCR and Voting Disks Storage Type
ASM
OCR Disks/Voting Disk
+GIS_FILES (Normal Redundancy)
ASM_HOME
/u01/app/grid11201
ASM_HOME Owner
oracle:(oinstall, dba)
ASMLib user:group
oracle:oinstall
ASM LISTENER
LISTENER (TCP:1521)


Software/Patches Required in this Setup:

Oracle Enterprise Linux 5.5 (2.6.18-194.el5PAE)
Oracle E-Business Suites 12.1.1
Oracle 11g R2 Grid Infrastructure (11.2.0.1)
Oracle Database 11g R1 (11.1.0.6)
Oracle Database example software 11g R1 (11.1.0.6)
Oracle Database PatchSet (11.1.0.7)
Oracle EBS 12.1 RUP 3 (R12.TXK.B.delta 3 Patch 8919489)

Documents/Notes Reviewed and Followed:

·         Oracle Applications Install Guide: Using Rapid Install (Part No. E12842-02)
·         Oracle E-Business Suite Installation and Upgrade Notes Release 12 (12.1.1) for Linux x86 [ID 761564.1]
·         Using Oracle 11g Release 1 (11.1.0.7) Real Application Clusters and Automatic Storage Management with Oracle E-Business Suite Release 12 [ID 466649.1]
·         Using AutoConfig to Manage System Configurations in Oracle E-Business Suite Release 12 [ID 387859.1]


Prepare the DB Tier for the EBS R12 Installation:


Install 11gR2 Grid Infrastructure:

In this phase, the 3-node 11gR2 Grid Infrastructure was installed with the below information by following my note on “Installing 11gR2 RAC on Linux”.
There, I had created ASM Diskgroups DATA and FRA with the size of 60G and 10G respectively.

Grid Infrastructure Software (Clusterware + ASM 11.2.0.1):
Server: All the RAC Nodes
ORACLE_BASE: /u01/app/oracle
ORACLE_HOME:  /u01/app/grid11201
Owner: oracle (Primary Group: oinstall, Secondary Group: dba)
Permissions: 755
OCR/Voting Disk Storage Type: ASM
Oracle Inventory Location: /u01/app/oraInventory
               
Install 11gR1 RAC HOME:

The EBS R 12.1.1 software comes with the default database version 11.1.0.7. So, I had to create the RAC HOME with the same patch level as the seeded Home. The below database software were downloaded from Oracle’s Website and installed them in the order listed by creating the new 11gR1 RAC (11.1.0.7) HOME /u01/app/oracle/EBSR12.
The Installation type was selected as “Software Only”.

Oracle Database 11g R1 (11.1.0.6)
Oracle Database example software 11g R1 (11.1.0.6)
Oracle Database PatchSet (11.1.0.7)

Here is the Step By Step Screenshots of the 11gR1 RAC installation.


Prepare the Application Tier for the EBS R12 Installation:

·         Linux Package Requirement
·         Apply Oracle Patch 6078836 to copy the Missing Library on Linux
·         Network Setup
·         Creating Application Software owners/Groups/Permissions
·         Setup E-Business Suites Software Owner’s (applmgr) Environment
·         Setting up SSH equivalency for OS user applmgr
·         Download and Stage the R12 Software.

Minimum Required RPMs for EBS R12.1.1 OEL 5.5 (All the Application Nodes):


compat-glibc-2.3.4-2.26
compat-glibc-headers-2.3.4-2.26
gcc-4.1.2-14.el5
gcc-c++-4.1.2-14.el5
glibc-2.5-123
glibc-common-2.5-123
glibc-devel-2.5-12
glibc-headers-2.5-12
kernel-headers-2.6.18-128.el5
libgcc-4.1.2-14.el53
libstdc++-devel-4.1.2-14.el5
libstdc++-4.1.2-14.el53
make-3.81-1.13
gdbm-1.8.0-26.2.13
libXp-1.0.0-8.1.el5
libaio-0.3.106-3.23
libgomp-4.1.2-14.el5
sysstat-7.0.0-3.el5
compat-libstdc++-296-2.96-138
compat-libstdc++-33-3.2.3-61
openmotif21-2.1.30-11.EL5.i386  ß available from http://oss.oracle.com/projects/compat-oracle/files/Enterprise_Linux
xorg-x11-libs-compat-6.8.2-1.EL.33.0.1.i386  ß available from above link

Below command verifies whether the specified rpms are installed or not. Any missing rpms can be installed from the OEL Media Pack

rpm -q compat-glibc compat-glibc-headers gcc gcc-c++ glibc glibc-common glibc-devel glibc-headers \
      kernel-headers libgcc libstdc++ libstdc++-devel make gdbm libXp libaio libgomp libaio-devel sysstat \
      compat-libstdc++-296 compat-libstdc++-33 openmotif21 xorg-x11-libs-compat

The following maintenance tools must be installed on each of the APPS nodes.

[root@appsnode2 ~]# which ar
/usr/bin/ar
[root@appsnode2 ~]# which gcc
/usr/bin/gcc
[root@appsnode2 ~]# which g++
/usr/bin/g++
[root@appsnode2 ~]# which ld
/usr/bin/ld
[root@appsnode2 ~]# which ksh
/bin/ksh
[root@appsnode2 ~]# which make
/usr/bin/make
[root@appsnode2 ~]# which X
/usr/bin/X


Apply patch 6078836 to copy the missing library (All the Application Nodes):

This patch comes with a file libdb.so.2 which was manually copied to /usr/lib on both the Application Nodes.


Network Configuration for the Application Nodes:


(a)    hostname:

For Apps Node node1:

       appsnode1.hingu.net: /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=yes
HOSTNAME=appsnode1.hingu.net

For Apps Node node2:

       appsnode2.hingu.net: /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=yes
HOSTNAME=appsnode2.hingu.net

(b)    Public Network:

appsnode1.hingu.net: /etc/sysconfig/network-scripts/ifcfg-eth2
# Broadcom Corporation NetXtreme BCM5751 Gigabit Ethernet PCI Express
DEVICE=eth2
BOOTPROTO=static
BROADCAST=192.168.2.255
HWADDR=00:13:72:A1:E9:1B
IPADDR=192.168.2.11
IPV6INIT=yes
IPV6_AUTOCONF=yes
NETMASK=255.255.255.0
NETWORK=192.168.2.0
ONBOOT=yes

appsnode2.hingu.net: /etc/sysconfig/network-scripts/ifcfg-eth2
# Broadcom Corporation NetXtreme BCM5751 Gigabit Ethernet PCI Express
DEVICE=eth2
BOOTPROTO=static
BROADCAST=192.168.2.255
HWADDR=00:14:22:5C:B0:A0
IPADDR=192.168.2.22
IPV6INIT=yes
IPV6_AUTOCONF=yes
NETMASK=255.255.255.0
NETWORK=192.168.2.0
ONBOOT=yes

(e)   /etc/hosts files (Same on both the Application Nodes):

/etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1            localhost.localdomain localhost
::1           localhost6.localdomain6 localhost6

##=======================================
#  Public Network
##=======================================

192.168.2.11            appsnode1.hingu.net appsnode1
192.168.2.22            appsnode2.hingu.net appsnode2
192.168.2.1             node1.hingu.net     node1
192.168.2.2             node2.hingu.net     node2
192.168.2.3             node3.hingu.net     node3

##=======================================
##=======================================

Enable Name Service Cache Daemon nscd: (On all the Apps Nodes)

chkconfig --level 35 nscd on
service ncsd start

Creating Oracle Users/Groups/Permissions and Installation Paths: (On all the Apps Nodes):


groupadd -g 1000 oinstall
groupadd –g 1031 dba
useradd -u 3100 -g oinstall –G dba applmgr
mkdir -p /u01
chown –R applmgr:oinstall /u01
chmod -R 775 /u01
passwd applmgr

Configure the applmgr User Environment:


(a)    Set the umask to 022 by putting below line into applmgr’s .bash_profile files:

umask 022

Then, executed the .bash_profile and verified that the correct value of umask is displayed.

[grid@node1 ~]$ . .bash_profile
[grid@node1 ~]$ umask

(b)   Setting up X11 forwarding:

Created the file ~/.ssh/config to disable the X11Forwadding by placing below line in it.

Host *
ForwardX11 no

(c)    Suppressed the Terminal output on STDOUT and STDERR to prevent Installation errors:
Modified the file ~/.bashrc (or .cshrc for C shell) with below entry.

Bourne, Bash, or Korn shell:

if [ -t 0 ]; then
stty intr ^C
fi

C shell:

test -t 0
if ($status == 0) then
stty intr ^C
endif

(d)   Increased the Shell Limits:
Recommended:

Resource
Soft Limit
Hard Limit
Processes
2047
16384
Open File Descriptors
1024
65536
Stack
10240
10240  - 32768

Set:

Resource
Soft Limit
Hard Limit
Processes
131072
131072
Open File Descriptors
131072
131072
Stack
32768
32768

Added the following lines to the /etc/security/limits.conf file:

applmgr   soft   nofile    131072
applmgr   hard   nofile    131072
applmgr   soft   nproc    131072
applmgr   hard   nproc    131072
applmgr   soft   core    unlimited
applmgr   hard   core    unlimited
applmgr   soft   memlock    3500000
applmgr   hard   memlock    3500000
# Recommended stack hard limit 32MB for applmgr installations
# applmgr   hard   stack    32768

Added the following line in the /etc/pam.d/login file, if it does not already exist:

session required /lib/security/pam_limits.so

For the Bourne, Bash, or Korn shell,  add the following lines to the /etc/profile:

        if [ $USER = "applmgr" ]; then
           if [ $SHELL = "/bin/ksh" ]; then
              ulimit -p 131072
              ulimit -n 131072
           else
              ulimit -u 131072 -n 131072
           fi
        fi


For the C shell (csh or tcsh), add the following lines to the /etc/csh.login.

       if ( $USER == "applmgr" ) then
          limit maxproc 131072
          limit descriptors 131072
       endif


(e)   Set the below Kernel Parameters with recommended range in /etc/sysctl.conf

/etc/sysctl.conf

# Controls the maximum shared segment size, in bytes
kernel.shmmax = 4294967295

# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 1073741824

# Controls the maximum size of a message, in bytes
kernel.msgmnb = 65536
# Controls the default maxmimum size of a mesage queue
kernel.msgmax = 8192
kernel.msgmni = 2878
# For 11g, recommended value for file-max is 6815744
fs.file-max = 6815744
kernel.sem = 250 32000 100 142
kernel.shmmni = 4096
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 262144
net.ipv4.ip_local_port_range = 10000 65500




(f)     Repeated this process for the second app node appsnode2.hingu.net.

SSH user Equivalency configuration (applmgr):


This step is not required but I would always set up the ssh user equivalency for ease of maintenance.

On All the Apps Nodes:

su - applmgr
mkdir ~/.ssh
chmod 700 ~/.ssh

Generate the RSA and DSA keys:

/usr/bin/ssh-keygen -t rsa
/usr/bin/ssh-keygen -t dsa

On appsnode1:

touch ~/.ssh/authorized_keys
cd ~/.ssh

(a)    Add these Keys to the Authorized_keys file.

cat id_rsa.pub >> authorized_keys
cat id_dsa.pub >> authorized_keys

(b)   Send this file to appsnode2.

scp authorized_keys appsnode2:.ssh/

On appsnode2:

(a)    Add these Keys to the Authorized_keys file.

cd ~/.ssh
cat id_rsa.pub >> authorized_keys
cat id_dsa.pub >> authorized_keys

(b)   Send this file to appsnode1.

scp authorized_keys appsnode1:.ssh/

On both the Apps Nodes:

chmod 600 ~/.ssh/authorized_keys

ssh appsnode1 date
ssh appsnode2 date
ssh appsnode1.hingu.net date
ssh appsnode2.hingu.net date

Entered 'yes' and continued when prompted

Download and Stage the R12 Software on one of APPS as well as DB Node:

(1)    Downloaded the E-Business Suite software Release 12.1.1 from otn.oracle.com and copied them to the /tmp directory as shown here.


(2)    Created the Staging Directory /u01/StageR12 to stage the R12 Software.
(3)    Created the script called /u01/StageR12/unzip.sh to unzip the downloaded software to the Stage Directory.

mkdir /u01/StageR12
cd /u01/StageR12
for i in `ls -1 /tmp/R12_for_Linux/*/*.zip`; do echo unzip $i; done > unzip.sh
for i in `ls -1 /tmp/R12_for_Linux/*/*/*.zip`; do echo unzip $i; done >> unzip.sh
chmod 755 unzip.sh



[applmgr@appsnode1 StageR12]$ cat unzip.sh
unzip /tmp/R12_for_Linux/Start/B53824-01_1of4.zip
unzip /tmp/R12_for_Linux/Start/B53824-01_2of4.zip
unzip /tmp/R12_for_Linux/Start/B53824-01_3of4.zip
unzip /tmp/R12_for_Linux/Start/B53824-01_4of4.zip
unzip /tmp/R12_for_Linux/APPL_TOP/DVD1/V15573-01_1of3.zip
unzip /tmp/R12_for_Linux/APPL_TOP/DVD1/V15573-01_2of3.zip
unzip /tmp/R12_for_Linux/APPL_TOP/DVD1/V15573-01_3of3.zip
unzip /tmp/R12_for_Linux/APPL_TOP/DVD2/V15574-01_1of3.zip
unzip /tmp/R12_for_Linux/APPL_TOP/DVD2/V15574-01_2of3.zip
unzip /tmp/R12_for_Linux/APPL_TOP/DVD2/V15574-01_3of3.zip
unzip /tmp/R12_for_Linux/Databases/DVD1/V15564-01_1of3.zip
unzip /tmp/R12_for_Linux/Databases/DVD1/V15564-01_2of3.zip
unzip /tmp/R12_for_Linux/Databases/DVD1/V15564-01_3of3.zip
unzip /tmp/R12_for_Linux/Databases/DVD2/V15565-01_1of2.zip
unzip /tmp/R12_for_Linux/Databases/DVD2/V15565-01_2of2.zip
unzip /tmp/R12_for_Linux/Databases/DVD3/V15566-01_1of3.zip
unzip /tmp/R12_for_Linux/Databases/DVD3/V15566-01_2of3.zip
unzip /tmp/R12_for_Linux/Databases/DVD3/V15566-01_3of3.zip
unzip /tmp/R12_for_Linux/Databases/DVD4/V15567-01_1of3.zip
unzip /tmp/R12_for_Linux/Databases/DVD4/V15567-01_2of3.zip
unzip /tmp/R12_for_Linux/Databases/DVD4/V15567-01_3of3.zip
unzip /tmp/R12_for_Linux/Databases/DVD5/V15568-01_1of3.zip
unzip /tmp/R12_for_Linux/Databases/DVD5/V15568-01_2of3.zip
unzip /tmp/R12_for_Linux/Databases/DVD5/V15568-01_3of3.zip
unzip /tmp/R12_for_Linux/Databases/DVD6/V15569-01_1of3.zip
unzip /tmp/R12_for_Linux/Databases/DVD6/V15569-01_2of3.zip
unzip /tmp/R12_for_Linux/Databases/DVD6/V15569-01_3of3.zip
unzip /tmp/R12_for_Linux/Databases/DVD7/V15570-01_1of3.zip
unzip /tmp/R12_for_Linux/Databases/DVD7/V15570-01_2of3.zip
unzip /tmp/R12_for_Linux/Databases/DVD7/V15570-01_3of3.zip
unzip /tmp/R12_for_Linux/Databases/DVD8/V15571-01_1of3.zip
unzip /tmp/R12_for_Linux/Databases/DVD8/V15571-01_2of3.zip
unzip /tmp/R12_for_Linux/Databases/DVD8/V15571-01_3of3.zip
unzip /tmp/R12_for_Linux/Databases/DVD9/V15572-01_1of3.zip
unzip /tmp/R12_for_Linux/Databases/DVD9/V15572-01_2of3.zip
unzip /tmp/R12_for_Linux/Databases/DVD9/V15572-01_3of3.zip
unzip /tmp/R12_for_Linux/RDBMS/DVD1/V15576-01_1of3.zip
unzip /tmp/R12_for_Linux/RDBMS/DVD1/V15576-01_2of3.zip
unzip /tmp/R12_for_Linux/RDBMS/DVD1/V15576-01_3of3.zip
unzip /tmp/R12_for_Linux/Tools/DVD1/V15575-01.zip
[applmgr@appsnode1 StageR12]$

(4)    Ran the unzip.sh script from the Stage directory (/u01/StageR12)

cd /u01/StageR12
./unzip.sh

During the unzipping process, when prompted, the dvd.label and component.label files were replaced. You can choose not to overwrite them as well because the dvd.label is same for all the DVDs belonging to the same category. For e.g, Databases has 9 DVDs and each of these DVDs dvd.label file is same.

cd /u01/StageR12
rm ora*/component.label
rm unzip.sh


After unzipping all the DVDs by running the unzip.sh script, the following directory structure was created.



With this, the pre-Installation steps are completed successfully and are ready to proceed with the Fresh Install of EBS R12 Next.

Fresh Install of EBS R12:


Install EBS R12 on Database Node using Rapid Install:

The Fresh Installation of R12 Database using Rapid Install cannot be directly installed on RAC HOME. It comes with non-RAC single instance binaries. So, I connected to one of the RAC Nodes (node1.hingu.net) and started the Rapid Install which installed the local non-RAC 11gR1 (11.1.0.7) Database. During the later stage, this database was then converted to RAC.

Using the X Terminal, Invoked the /u01/StageR12/startCD/Disk1/rapidwiz/rapidwiz.

Select Wizard operation:
 Install Oracle Application Release 12.1.1
Oracle Configuration Manager (OCM) Detail:
My Oracle Support Password: *********
Configuration Choice:
Create New Configuration
Global System Settings:
Port Pool: 1
                Database Node:
Database Type: Fresh Database
Database SID: LABDB
Host Name: node1
Domain Name: hingu.net
Database OS User: oracle
Database OS Group: dba
Base Directory: /u01/app/oracle/ebsR12
Licensing Selection:
Suite Licensing
License Additional Product:
Default.
Select Country-specific Functionalities:
No Selection (which means US as a default selection)
Internationalization Setting:
Selected Language: American_English – US
Default Territory: AMERICA
Database Character Set: AL32UTF8
APPL_TOP Character Set: AL32UTF8
Primary Application Node:
Host Name: appsnode1
Domain Name: hingu.net
Apps OS User: applmgr
Apps OS Group: dba
Base Directory: /u01/app/oracle/ebsR12
Instance Directory: /u01/app/oracle/ebsR12/inst
Application Services: All Enabled except “Batch Processing Services”
Node Information (Add Server):
Host Name: appsnode2
Domain Name: hingu.net
Apps OS User: applmgr
Apps OS Group: dba
Base Directory: /u01/app/oracle/ebsR12
Instance Directory: /u01/app/oracle/ebsR12/inst
Application Services: All Disabled except “Batch Processing Services” and “Other Sevices”
Node Information:
Shows the DB as well as Application nodes.
Pre-Install Checks:
Confirmed that all the Checks were passed
Component Installation Review:
Clicked ‘Next” to start the Rapid Installation Process.

25.JPG



No comments:

Post a Comment


No one has ever become poor by giving