11g RAC Step By Steps part 3

Part 3 continution ;

Start the RAC Instances on node2 and node2:

After setting the cluster_database parameter to true, restarted the instance LABDB1 and started the remaining instances LABDB2 and LABDB3.

On node1:

alter system set cluster_database=true scope=spfile sid='*';

shutdown immediate

startup

On node2:

startup

On node3:

startup

Register the Database and Instances to the OCR.

srvctl add database -d LABDB -o /u01/app/oracle/EBSR12 -p +data/labdb/spfileLABDB.ora -m hingu.net

srvctl add instance -d LABDB -i LABDB1 -n node1

srvctl add instance -d LABDB -i LABDB2 -n node2

srvctl add instance -d LABDB -i LABDB3 -n node3

srvctl modify instance -d LABDB -i LABDB1 -s +ASM1

srvctl modify instance -d LABDB -i LABDB2 -s +ASM2

srvctl modify instance -d LABDB -i LABDB3 -s +ASM3

Database was successfully converted and opened to RAC. At this stage, the application services on application nodes were still down.

At this stage, the application did not know about the RAC nodes until the autoconfig was enabled and run on the DB tier from the RAC HOME.

Enable AutoConfig on the DB Tier (11gR1 RAC HOME):

·         Setup appsutil directory under /u01/app/oracle/EBSR12 HOME.

·         Verify the TNS connection to each of the Individual RAC Instances

·         Create nls/data/9idata directory

·         Delete the FND Node Configuration from the Database.

·         Build XML on each RAC node of the DB Tier.

·         Run AutoConfig on each RAC node of the DB Tier.

Setup appsutil Directory on All the RAC Nodes.

(1)    On the application admin node, appsnode1, the appsutil.zip was generated and copied over to all the DB Tier nodes (node1, node2 and node3) under the /u01/app/oracle/EBSR12.

Generate the appsutil.zip

[applmgr@appsnode1 ~]$ $AD_TOP/bin/admkappsutil.pl

Starting the generation of appsutil.zip

Log file located at /u01/app/oracle/ebsR12/inst/apps/LABDB_appsnode1/admin/log/MakeAppsUtil_10291438.log

output located at /u01/app/oracle/ebsR12/inst/apps/LABDB_appsnode1/admin/out/appsutil.zip

MakeAppsUtil completed successfully.

copy to all the RAC Nodes FROM appsnode1:

scp /u01/app/oracle/ebsR12/inst/apps/LABDB_appsnode1/admin/out/appsutil.zip oracle@node1:/u01/app/oracle/EBSR12

scp /u01/app/oracle/ebsR12/inst/apps/LABDB_appsnode1/admin/out/appsutil.zip oracle@node2:/u01/app/oracle/EBSR12

scp /u01/app/oracle/ebsR12/inst/apps/LABDB_appsnode1/admin/out/appsutil.zip oracle@node3:/u01/app/oracle/EBSR12

unzip the appsutil.zip on all the RAC Nodes:

cd /u01/app/oracle/EBSR12

unzip –o appsutil.zip

(2)    The jre directory was copied from the OLD_11GR1_HOME (/u01/app/oracle/ebsR12/db/tech_st/11.1.0/appsutil/jre) to the NEW_11GR1_RAC_HOME (/u01/app/oracle/EBSR12/appsutil) on all the nodes.

cp –r /u01/app/oracle/ebsR12/db/tech_st/11.1.0/appsutil/jre oracle@node1:/u01/app/oracle/EBSR12

scp /u01/app/oracle/ebsR12/inst/apps/LABDB_appsnode1/admin/out/appsutil.zip oracle@node1:/u01/app/oracle/EBSR12

scp /u01/app/oracle/ebsR12/inst/apps/LABDB_appsnode1/admin/out/appsutil.zip oracle@node1:/u01/app/oracle/EBSR12

The Instance Connectivity from each of the RAC instances was verified using SQLPLUS from RAC nodes individually. I would not run adconfig.pl without having successful connection to each of the RAC instances.

RAC Instance Connections Verification:

The Instance Connectivity from each of the RAC instances was verified using SQLPLUS from RAC nodes individually. The connectivity was confirmed before running the adconfig.pl on each of the RAC instances.

node1:

export ORA_NLS10=/u01/app/oracle/EBSR12/nls/data/9idata

export ORACLE_HOME=/u01/app/oracle/EBSR12

export ORACLE_SID=LABDB1

export TNS_ADMIN=/u01/app/oracle/EBSR12/network/admin/LABDB1_node1

export LD_LIBRARY_PATH=/u01/app/oracle/EBSR12/lib:/u01/app/oracle/EBSR12/ctx/lib

export PATH=$ORACLE_HOME/bin:$PATH:.

sqlplus apps/apps@LABDB1

Repeated the above process for all the RAC Instances.

Create the nls/data/9idata Directory (on All the RAC Nodes):

On all the RAC Nodes, ran the /u01/app/oracle/EBSR12/nls/data/old/cr9idata.pl script to create the /u01/app/oracle/EBSR12/nls/data/9idata directory. Without this directory, the adconfig.sh on DB tier may fail on this.

[oracle@node1 old]$ perl cr9idata.pl

Creating directory /u01/app/oracle/EBSR12/nls/data/9idata ...

Copying files to /u01/app/oracle/EBSR12/nls/data/9idata...

Copy finished.

Please reset environment variable ORA_NLS10 to /u01/app/oracle/EBSR12/nls/data/9idata!

[oracle@node1 old]$

Clean the current Configuration in the Database

SQL> exec fnd_conc_clone.setup_clean;

Build XML (on all the RAC Nodes):

Ran the /u01/app/oracle/EBSR12/appsutil/bin/adbldxml.pl to build the XML on each of the RAC Nodes and provided the appropriate values of DB Host Name, Database Port, SID, and Service Name. This will generate the context file <SID>_hostname.xml under /u01/app/oracle/EBSR12/appsutil.

[oracle@node1 bin]$ ./adbldxml.pl

Starting context file generation for db tier..

Using JVM from /u01/app/oracle/EBSR12/appsutil/jre/bin/java to execute java programs..

APPS Password: apps

The log file for this adbldxml session is located at:

/u01/app/oracle/EBSR12/appsutil/log/adbldxml_10311041.log

UnsatisfiedLinkError exception loading native library: njni11

Could not Connect to the Database with the above parameters, Please answer the Questions below

Enter Hostname of Database server: node1

Enter Port of Database server: 1522

Enter SID of Database server: LABDB1

Enter Database Service Name: LABDB

Enter the value for Display Variable: 1

The context file has been created at:

/u01/app/oracle/EBSR12/appsutil/LABDB1_node1.xml

Once the context file got generated, the below variables were modified (in RED) to their appropriate values.

<!-- host information -->

   <oa_host>

      <host oa_var="s_hostname">node3</host>

      <host oa_var="s_virtual_hostname">node1-vip</host>

      <private_address oa_var="s_clusterInterConnects">node1-prv</private_address>

      <domain oa_var="s_domainname">hingu.net</domain>

Similar way, the XML file was generated on node2 and node3 by providing their node specific values.

Run autoconfig on all the RAC nodes one by one:

Ran the Autoconfig on all the RAC nodes separately using the script /u01/app/oracle/EBSR12/appsutil/bin/adconfig.sh.

export ORA_NLS10=/u01/app/oracle/EBSR12/nls/data/9idata

export ORACLE_HOME=/u01/app/oracle/EBSR12

export ORACLE_SID=LABDB2

export TNS_ADMIN=/u01/app/oracle/EBSR12/network/admin/LABDB2_node2

export LD_LIBRARY_PATH=/u01/app/oracle/EBSR12/lib:/u01/app/oracle/EBSR12/ctx/lib

export PATH=$ORACLE_HOME/bin:$PATH:.

cd /u01/app/oracle/EBSR12/appsutil/bin

./adconfig.sh

The AutoConfig was again ran on all the RAC Nodes using below command in the reverse order so that all the nodes’ tnsnames.ora file contains the same information.

/u01/app/oracle/EBSR12/appsutil/scripts/LABDB3_node3/adautocfg.sh  ß for node3

/u01/app/oracle/EBSR12/appsutil/scripts/LABDB2_node2/adautocfg.sh  ß for node2

/u01/app/oracle/EBSR12/appsutil/scripts/LABDB1_node1/adautocfg.sh  ß for node1

Modify the $ORACLE_HOME/dbs/<SID>_APPS_BASE.ora on all the RAC Nodes:

The <SID>_APPS_BASE.ora file under the /u01/app/oracle/EBSR12/dbs contains the wrong values for the control_files parameter. So, this file was modified on all the RAC nodes with the correct value of control_files parameter.

control_files                   = '+DATA/labdb/controlfile/current.267.765911037','+FRA/labdb/controlfile/current.263.765911059'

At this stage, the FND Nodes contained the RAC nodes information in the application database.

Run the AutoConfig on the Application nodes.

After enabling and running the AutoConfig successfully on the DB tier RAC nodes, it is required to run the autoconfig on all the Application nodes so that they can generate the new RAC specific TNS and jdbc_url entry. I followed the below steps to accomplish this requirement.

·         Modify $TNS_ADMIN/tnsnames.ora file and jdbc_url in the CONTEXT_FILE.

·         Check the connection to the database after the above modification.

·         Run the AutoConfig

·         Repeat the above 3 steps on remaining application node appsnode2.

·         At last, run the AutoConfig on the Admin Node (appsnode1).

Modify the tnanames.ora and jdbc_url:

The CONNECT_DATA entry in the $TNS_ADMIN/tnsnames.ora file was modified to have SERVICE_NAME and INSTANCE_NAME value for all the TNS Alias belonging to Database instance. (On both the Application Nodes appsnode1 and appsnode2)

LABDB=

        (DESCRIPTION=

                (ADDRESS=(PROTOCOL=tcp)(HOST=node1.hingu.net)(PORT=1522))

            (CONNECT_DATA=

                (SERVICE_NAME=LABDB)

                (INSTANCE_NAME=LABDB1)

            )

LABDB_BALANCE=

        (DESCRIPTION=

            (ADDRESS_LIST=

                (LOAD_BALANCE=YES)

                (FAILOVER=YES)

                (ADDRESS=(PROTOCOL=tcp)(HOST=node1.hingu.net)(PORT=1522))

            )

            (CONNECT_DATA=

                (SERVICE_NAME=LABDB)

            )

        )

Jdbc_url value in the below context file was modified with the same above information as shown on both the application nodes appsnode1 and appsnode2.

appsnode1:/u01/app/oracle/ebsR12/inst/apps/LABDB_appsnode1/appl/admin/LABDB_appsnode1.xml

appsnode2:/u01/app/oracle/ebsR12/inst/apps/LABDB_appsnode2/appl/admin/LABDB_appsnode2.xml

<jdbc_url oa_var="s_apps_jdbc_connect_descriptor">jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=YES)(FAILOVER=YES)(ADDRESS=(PROTOCOL=tcp)(HOST=node1.hingu.net)(PORT=1522)))(CONNECT_DATA=(SERVICE_NAME=LABDB)(INSTANCE_NAME=LABDB1)))</jdbc_url>

Run the AutoConfig on appsnode1:

$AD_TOP/bin/adconfig.sh contextfile=/u01/app/oracle/ebsR12/inst/apps/LABDB_appsnode1/appl/admin/LABDB_appsnode1.xml

After successful completion of autoconfig and re-sourcing the environment, the $TNS_ADMIN/tnsnames.ora file was containing the tns aliases for all the RAC instances individually and also had a BALANCE entry containing all the DB VIPs as shown below:

LABDB_BALANCE=

        (DESCRIPTION=

            (ADDRESS_LIST=

                (LOAD_BALANCE=YES)

                (FAILOVER=YES)

                (ADDRESS=(PROTOCOL=tcp)(HOST=node2-vip.hingu.net)(PORT=1522))

                (ADDRESS=(PROTOCOL=tcp)(HOST=node1-vip.hingu.net)(PORT=1522))

                (ADDRESS=(PROTOCOL=tcp)(HOST=node3-vip.hingu.net)(PORT=1522))

            )

            (CONNECT_DATA=

                (SERVICE_NAME=LABDB)

            )

        )

The new jdbc_url in the CONTEXT_FILE on appsnode1 was reflecting all the 3 VIPs of database RAC nodes at this stage.

Run the Autoconfig on appsnode2:

Same way, I ran the autoconfig on the appsnode2 using below command.

$AD_TOP/bin/adconfig.sh contextfile=/u01/app/oracle/ebsR12/inst/apps/LABDB_appsnode2/appl/admin/LABDB_appsnode2.xml

Re-Run the AutoConfig on appsnode2 and appsnode1:

I re-ran the autoconfig on the both the application nodes using below command before started the applications.

$INST_TOP/admin/scripts/adautocfg.sh

Tested the login page and verified the Nodes information in the OAM as well as database.

FND Nodes Configuration:

At this stage, the configuration has 5 nodes total, 3 are DB nodes and 2 are application nodes.

Start the Applications on both the Application Nodes

Started the Application on appsnode1.hingu.net and appsnode2.hingu.net

$INST_TOP/admin/scripts/adstrtal.sh

Here is the Screenshots of Running the AutoConfig on all RAC nodes as well as on Application Nodes after converting the Database to RAC.

Verify the Installation

Verified the OS processes on both the application nodes to confirm that the enabled services are started and running successfully.

Logged in to the Application using login page url.

http://appsnode1.hingu.net:8001/OA_HTML/AppsLogin

Verified that the Concurrent Managers are started and running on the Concurrent Node (appsnode2.hingu.net)

OAM Configuration:

Configuration Files of This Installations:

(a)    listnener.ora, tnsnames.ora, sqlnet.ora and their ifiles from all the RAC nodes.

(b)   listnener.ora and tnsnames.ora from both the application nodes.

(c)    CONTEXT_FILE from all the 5 nodes.

(d)   Pfile of the RAC database.

Here is the Final look of the above Configuration files at the end of the successful Installation of 2-node EBS R12 (12.1.1) on 3-node 11gR1 RAC database managed by 11gR2 Grid Infrastructure.