11g RAC Step By Steps part 2

Part 2 continuation ...

Install EBS R12 on Application Node (appsnode1.hingu.net) using Rapid Install:

After completing the installation on the DB node, the Rapid Install was started on the Application node appsnode1.hingu.net.

Using the X Terminal, Invoked the /u01/StageR12/startCD/Disk1/rapidwiz/rapidwiz.

Select Wizard operation:

 Install Oracle Application Release 12.1.1

Oracle Configuration Manager (OCM) Detail:

Email: bhavin@oracledba.org

My Oracle Support Password: *********

Configuration Choice:

Load the following saved configuration: node1.hingu.net:LABDB:1522

Pre-Install Checks:

Confirmed that all the Checks were passed

Component Installation Review:

Clicked “Next” to start the Rapid Installation process.

Verified the Installation by logging into the application using LocalLogin

Install EBS R12 on Application Node (appsnode2.hingu.net) using Rapid Install:

After completing the installation on the DB node, the Rapid Install was started on the Application node appsnode2.hingu.net.

Using the X Terminal, Invoked the /u01/StageR12/startCD/Disk1/rapidwiz/rapidwiz.

Select Wizard operation:

 Install Oracle Application Release 12.1.1

Oracle Configuration Manager (OCM) Detail:

Email: bhavin@oracledba.org

My Oracle Support Password: *********

Configuration Choice:

Load the following saved configuration: node1.hingu.net:LABDB:1522

Pre-Install Checks:

Confirmed that all the Checks were passed

Component Installation Review:

Clicked “Next” to start the Rapid Installation process.

Verified the Concurrent Processes running on appsnode2

Here is the complete Step By Step Screenshots of Installing EBS R12.1.1

Apply EBS R 12.1.3 Techstack Product Release Update Patch:

This patch needs to be applied to use the named listener on the DB tier. Without this patch, the listener name has to be LISTENER and I wanted to use the name LABDB_LISTENER for the application database listener so I decided to apply this patch.

a)      Downloaded the patch 8919489.

b)      Downloaded the Patch 9583541 (post requirement for the patch 8919489).

c)       Stopped the Application on both the nodes.

d)      Applied the patch 8919489 using adpatch by following its README on appsnode1

e)      Applied the patch 8919489 using adpatch by following its README on appsnode2

f)       Ran the AutoConfig again on Admin Node (appsnode1.hingu.net) after applying the patch on appsnode2.

g)      Tried to start the Application on appsnode1.hingu.net and it got warning message saying it could not deploy the latest EAR file.

The Forms were manually deployed by following the Metalink note “Deploying a New forms.ear File in Oracle Applications Release 12 [ID 397174.1]”. While running the script to deploy the forms, it got errored out with the below error message complaining it could not find the opmnctl executable.

cp $INST_TOP/ora/10.1.3/j2ee/forms/config/system-jazn-data.xml $INST_TOP/ora/10.1.3/j2ee/forms/config/system-jazn-data.xml.ori

*******FATAL ERROR*******

PROGRAM : /u01/app/oracle/ebsR12/apps/apps_st/appl/fnd/12.0.0/patch/115/bin/txkCfgOC4JApp.pl(/u01/app/oracle/ebsR12/apps/apps_st/appl/fnd/12.0.0/bin/txkrun.pl)

TIME    : Sat Oct 29 13:47:38 2011

FUNCTION: TXK::Process::run [ Level 3 ]

MESSAGES:

Command error: <rc> = 32512, <command> = /u01/app/oracle/ebsR12/apps/tech_st/10.1.3/opmn/bin/opmnctl start

STACK TRACE

       TXK::Error::abort('TXK::Error','HASH(0x923639c)') called at /u01/app/oracle/ebsR12/apps/apps_st/appl/au/12.0.0/perl/TXK/Common.pm line 299

       TXK::Common::doError('TXK::Process=HASH(0xb7cad114)','Command error: <rc> = 32512, <command> = /u01/app/oracle/ebsR...','undef') called at /u01/app/oracle/ebsR12/apps/apps_st/appl/au/12.0.0/perl/TXK/Common.pm line 314

       TXK::Common::setError('TXK::Process=HASH(0xb7cad114)','Command error: <rc> = 32512, <command> = /u01/app/oracle/ebsR...') called at /u01/app/oracle/ebsR12/apps/apps_st/appl/au/12.0.0/perl/TXK/Process.pm line 449

       TXK::Process::run('TXK::Process=HASH(0xb7cad114)','HASH(0x9756fe4)') called at /u01/app/oracle/ebsR12/apps/apps_st/appl/fnd/12.0.0/patch/115/bin/txkCfgOC4JApp.pl line 1587

       TXK::RunScript::execOPMNControl('HASH(0x9a3b224)') called at /u01/app/oracle/ebsR12/apps/apps_st/appl/fnd/12.0.0/patch/115/bin/txkCfgOC4JApp.pl line 599

       require /u01/app/oracle/ebsR12/apps/apps_st/appl/fnd/12.0.0/patch/115/bin/txkCfgOC4JApp.pl called at /u01/app/oracle/ebsR12/apps/apps_st/appl/au/12.0.0/perl/TXK/RunScript.pm line 105

       TXK::RunScript::require('TXK::RunScript','/u01/app/oracle/ebsR12/apps/apps_st/appl/fnd/12.0.0/patch/115...') called at /u01/app/oracle/ebsR12/apps/apps_st/appl/au/12.0.0/perl/TXK/Script.pm line 177

       eval {...} called at /u01/app/oracle/ebsR12/apps/apps_st/appl/au/12.0.0/perl/TXK/Script.pm line 177

       TXK::Script::run('TXK::Script=HASH(0x9b1c980)','/u01/app/oracle/ebsR12/inst/apps/LABDB_appsnode1/logs/appl/rg...','/u01/app/oracle/ebsR12/apps/apps_st/appl/fnd/12.0.0/patch/115...') called at /u01/app/oracle/ebsR12/apps/apps_st/appl/fnd/12.0.0/bin/txkrun.pl line 174

While further debugging, it was found out that the ORACLE_HOME and TNS_ADMIN variables in $IAS_ORACLE_HOME/opmn/bin/opmnctl were pointing to /nfs/bld/d26/PRDXBLD9/apps/tech_st/10.1.3 instead of /u01/app/oracle/ebsR12/apps/tech_st/10.1.3. These variables were correctly modified and then continued with the Note 397174.1 to deploy the forms.ear and after that, it got deployed successfully.

After that, the Application services were started on both the application nodes using $INST_TOP/admin/scripts/adstrtal.sh and tested logging in to the application and forms and the results were all successful.

Convert the Non-RAC EBS R 12.1.1 Database to the RAC Database:

OLD_11GR1_HOME: /u01/app/oracle/ebsR12/db/tech_st/11.1.0

NEW_11GR1_RAC_HOME: /u01/app/oracle/EBSR12

·         Shutdown the application services on both the application nodes.

·         Shutdown the listener LABDB on database node node1.

·         Shutdown the database LABDB on node1.

·         Performed the backup of Applications Tier on both the Application nodes.

·         Performed the backup of OLD_11GR1_HOME

·         Performed the backup of database.

·         Created the listener LABDB_LISTENER on the same port 1522 under the NEW_11GR1_RAC_HOME using netca.

·         Created the new tnsnames.ora and their ifiles on each node.

·         Created the RAC specific spfile and started the instance LABDB1 on node1

·         Added extra Redo Threads and UNDO TBS for the RAC Instances LABDB2 and LABDB3 on ASM diskgroup.

·         Move the Database to ASM Diskgroup DATA.

·         Enable the cluster mode in the database and started the LABDB database instances on all the RAC Nodes.

Perform the Application and Database Backup:

Appsnode1 and Appsnode2

$TNS_ADMIN/admin/scripts/adstpall.sh

On DB Node1:

Sqlplus / as sysdba

Shutdown immediate

Performed the Cold backup of database, Database Home and Application Home.

Start the Database after the backup

Sqlplus / as sysdba

startup

Created the RAC Listener LABDB_LISTENER:

In the NEW_11GR1_RAC_HOME which was created earlier during the preparation phase of DB Tier for the EBS R12 installation, created the new TNS_ADMIN directory on each RAC nodes.

[oracle@node1 admin]$ pwd

/u01/app/oracle/EBSR12/network/admin

[oracle@node1 admin]$ ssh node1 mkdir /u01/app/oracle/EBSR12/network/admin/LABDB1_node1

[oracle@node1 admin]$ ssh node2 mkdir /u01/app/oracle/EBSR12/network/admin/LABDB2_node2

[oracle@node1 admin]$ ssh node3 mkdir /u01/app/oracle/EBSR12/network/admin/LABDB3_node3

[oracle@node1 admin]$

Added the TNS_ADMIN entry in the /u01/app/oracle/EBSR12/bin/racgwrap (NEW_11GR1_RAC_HOME) file right under the ORACLE_HOME entry.

node1:/u01/app/oracle/EBSR12/bin/racgwrap

ORACLE_HOME=/u01/app/oracle/EBSR12

export ORACLE_HOME

export TNS_ADMIN=/u01/app/oracle/EBSR12/network/admin/LABDB1_node1

node2:/u01/app/oracle/EBSR12/bin/racgwrap

ORACLE_HOME=/u01/app/oracle/EBSR12

export ORACLE_HOME

export TNS_ADMIN=/u01/app/oracle/EBSR12/network/admin/LABDB2_node2

node3:/u01/app/oracle/EBSR12/bin/racgwrap

ORACLE_HOME=/u01/app/oracle/EBSR12

export ORACLE_HOME

export TNS_ADMIN=/u01/app/oracle/EBSR12/network/admin/LABDB3_node3

From RAC Node 1 (node1.hingu.net):

Stopped the listener LABDB created during the default installation on port 1522 from the OLD_11GR1_HOME.

Set the TNS_ADMIN for node1 as shown below and invoked the netca from NEW_11GR1_RAC_HOME to add the listener LABDB_LISTENER on RAC node node1

export TNS_ADMIN=/u01/app/oracle/EBSR12/network/admin/LABDB1_node1

/u01/app/oracle/EBSR12/bin/netca

Select the “Cluster Configuration”

Select node1

Select “Listener Configuration”

Select “Add”

Listener Name: LABDB_LISTENER

Selected Protocol: TCP

Port: 1522

Finish

Now, set the TNS_ADMIN for node 2 as shown below and invoked the same netca again add the listener LABDB_LISTENER on RAC node node2.

export TNS_ADMIN=/u01/app/oracle/EBSR12/network/admin/LABDB2_node2

/u01/app/oracle/EBSR12/bin/netca

Select the “Cluster Configuration”

Select node2

Select “Listener Configuration”

Select “Add”

Listener Name: LABDB_LISTENER

Selected Protocol: TCP

Port: 1522

Finish

Now, set the TNS_ADMIN for node 3 as shown below and invoked the same netca again to add the listener LABDB_LISTENER on RAC node node3.

export TNS_ADMIN=/u01/app/oracle/EBSR12/network/admin/LABDB3_node3

/u01/app/oracle/EBSR12/bin/netca

Select the “Cluster Configuration”

Select node3

Select “Listener Configuration”

Select “Add”

Listener Name: LABDB_LISTENER

Selected Protocol: TCP

Port: 1522

Finish

Created the tnsnames.ora under the NEW_11GR1_RAC_HOME:

node1:/u01/app/oracle/EBSR12/network/admin/LABDB1_node1/tnsnames.ora

node2:/u01/app/oracle/EBSR12/network/admin/LABDB2_node2/tnsnames.ora

node3:/u01/app/oracle/EBSR12/network/admin/LABDB3_node3/tnsnames.ora

The above mentioned tnsnames.ora files were created with the below contents on each of the database nodes. The IFILE entries were the only entry that was node specific and were adjusted appropriately on each node. The $TNS_ADMIN/<SID>_<nodename>_ifile.ora was created on each node with the same below information.

LABDB=

        (DESCRIPTION=

                (ADDRESS=(PROTOCOL=tcp)(HOST=node1-vip.hingu.net)(PORT=1522))

                (ADDRESS=(PROTOCOL=tcp)(HOST=node2-vip.hingu.net)(PORT=1522))

                (ADDRESS=(PROTOCOL=tcp)(HOST=node3-vip.hingu.net)(PORT=1522))

            (CONNECT_DATA=

                (SERVICE_NAME=LABDB)

            )

        )

LABDB1=

        (DESCRIPTION=

                (ADDRESS=(PROTOCOL=tcp)(HOST=node1-vip.hingu.net)(PORT=1522))

            (CONNECT_DATA=

                (SERVICE_NAME=LABDB)

                (INSTANCE_NAME=LABDB1)

            )

        )

LABDB2=

        (DESCRIPTION=

                (ADDRESS=(PROTOCOL=tcp)(HOST=node2-vip.hingu.net)(PORT=1522))

            (CONNECT_DATA=

                (SERVICE_NAME=LABDB)

                (INSTANCE_NAME=LABDB2)

            )

        )

LABDB3=

        (DESCRIPTION=

                (ADDRESS=(PROTOCOL=tcp)(HOST=node3-vip.hingu.net)(PORT=1522))

            (CONNECT_DATA=

                (SERVICE_NAME=LABDB)

                (INSTANCE_NAME=LABDB3)

            )

        )

extproc_connection_data =

    (DESCRIPTION=

         (ADDRESS_LIST =

             (ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROCLABDB))

        )

        (CONNECT_DATA=

            (SID=PLSExtProc)

            (PRESENTATION = RO)

        ) )

LISTENER_LABDB1 =

    (ADDRESS_LIST =

      (ADDRESS = (PROTOCOL = TCP)(HOST = node1.hingu.net)(PORT = 1522))

    )

LISTENER_LABDB2 =

    (ADDRESS_LIST =

      (ADDRESS = (PROTOCOL = TCP)(HOST = node2.hingu.net)(PORT = 1522))

    )

LISTENER_LABDB3 =

    (ADDRESS_LIST =

      (ADDRESS = (PROTOCOL = TCP)(HOST = node3.hingu.net)(PORT = 1522))

    )

LISTENERS_LABDB =

    (ADDRESS_LIST =

      (ADDRESS = (PROTOCOL = TCP)(HOST = node1-vip.hingu.net)(PORT = 1522))

      (ADDRESS = (PROTOCOL = TCP)(HOST = node2-vip.hingu.net)(PORT = 1522))

      (ADDRESS = (PROTOCOL = TCP)(HOST = node3-vip.hingu.net)(PORT = 1522))

    )

IFILE=/u01/app/oracle/EBSR12/network/admin/LABDB1_node1/LABDB1_node1_ifile.ora

Created the sqlnet.ora under the NEW_11GR1_RAC_HOME:

node1:/u01/app/oracle/EBSR12/network/admin/LABDB1_node1/sqlnet.ora

node2:/u01/app/oracle/EBSR12/network/admin/LABDB2_node2/sqlnet.ora

node3:/u01/app/oracle/EBSR12/network/admin/LABDB3_node3/sqlnet.ora

The above mentioned sqlnet.ora files were created with the below contents on each of the database nodes.

NAMES.DIRECTORY_PATH=(TNSNAMES, ONAMES, HOSTNAME)

SQLNET.EXPIRE_TIME= 10

SQLNET.INBOUND_CONNECT_TIMEOUT =60

DIAG_ADR_ENABLED=ON

Created RAC spfile and started the LABDB1 instance from NEW_11GR1_RAC_HOME:

Created the required Directories to support the Instance specific entries in the spfile:

ssh node1 mkdir -p /u01/app/oracle/EBSR12/admin/LABDB1_node1

ssh node2 mkdir -p /u01/app/oracle/EBSR12/admin/LABDB2_node2

ssh node3 mkdir -p /u01/app/oracle/EBSR12/admin/LABDB3_node3

ssh node1 mkdir –p /u01/app/oracle/EBSR12/plsql/nativelib

ssh node2 mkdir –p /u01/app/oracle/EBSR12/plsql/nativelib

ssh node3 mkdir –p /u01/app/oracle/EBSR12/plsql/nativelib

ssh node1 mkdir -p /u01/app/oracle/EBSR12/appsutil/outbound/LABDB1_node1

ssh node2 mkdir -p /u01/app/oracle/EBSR12/appsutil/outbound/LABDB2_node2

ssh node3 mkdir -p /u01/app/oracle/EBSR12/appsutil/outbound/LABDB3_node3

Created the /tmp/initLABDB.ora from OLD_11GR1_HOME/dbs/initLABDB.ora and modified as shown below.

/tmp/initLABDB.ora

db_name                         = LABDB

control_files=/u01/app/oracle/ebsR12/db/apps_st/data/cntrl01.dbf,/u01/app/oracle/ebsR12/db/apps_st/data/cntrl02.dbf,/u01/app/oracle/ebsR12/db/apps_st/data/cntrl03.dbf

db_block_size                   = 8192

compatible                      = 11.1.0

_system_trig_enabled            = true

nls_language               = american

nls_territory              = america

nls_date_format            = DD-MON-RR

nls_numeric_characters     = ".,"

nls_sort                   = binary  # Required 11i setting

nls_comp                   = binary  # Required 11i setting

nls_length_semantics            = BYTE    # Required 11i setting 

LABDB1.diagnostic_dest                  = /u01/app/oracle/EBSR12/admin/LABDB1_node1

LABDB2.diagnostic_dest                  = /u01/app/oracle/EBSR12/admin/LABDB2_node2

LABDB3.diagnostic_dest                  = /u01/app/oracle/EBSR12/admin/LABDB3_node3

max_dump_file_size              = 20480 # trace file size

timed_statistics                = true

_trace_files_public             = TRUE

processes                       = 200    # Max. no. of users x 2

sessions                        = 400     # 2 X processes 

db_files                        = 512         # Max. no. of database files

dml_locks                       = 10000          

cursor_sharing                  = EXACT # Required 11i settting

open_cursors                    = 600   # Consumes process memory, unless using MTS.

session_cached_cursors            = 500

sga_target               = 1G        

shared_pool_size = 400M

shared_pool_reserved_size = 40M

db_block_checking               = FALSE

db_block_checksum               = TRUE

log_checkpoint_timeout    = 1200 # Checkpoint at least every 20 mins.

log_checkpoint_interval   = 100000

log_checkpoints_to_alert  = TRUE

log_buffer                = 10485760

undo_management=AUTO                   # Required 11i setting

LABDB1.undo_tablespace=APPS_UNDOTS1     # Required 11i setting

LABDB2.undo_tablespace=APPS_UNDOTS2     # Required 11i setting

LABDB3.undo_tablespace=APPS_UNDOTS3     # Required 11i setting

workarea_size_policy=AUTO               # Required 11i setting

olap_page_pool_size     =          4194304

pga_aggregate_target=1G

plsql_optimize_level     = 2              # Required 11i setting

plsql_code_type          = INTERPRETED    # Default 11i setting

plsql_native_library_dir = /u01/app/oracle/EBSR12/plsql/nativelib

plsql_native_library_subdir_count = 149

LABDB1.utl_file_dir = /usr/tmp,/usr/tmp,/u01/app/oracle/EBSR12/appsutil/outbound/LABDB1_node1,/usr/tmp

LABDB2.utl_file_dir = /usr/tmp,/usr/tmp,/u01/app/oracle/EBSR12/appsutil/outbound/LABDB2_node2,/usr/tmp

LABDB3.utl_file_dir = /usr/tmp,/usr/tmp,/u01/app/oracle/EBSR12/appsutil/outbound/LABDB3_node3,/usr/tmp

aq_tm_processes                 = 1

job_queue_processes             = 2

parallel_max_servers = 8

parallel_min_servers = 0

_b_tree_bitmap_plans = FALSE      # Required 11i setting

_index_join_enabled = FALSE         # Required 11i setting

_like_with_bind_as_equality     = TRUE

_sort_elimination_cost_ratio=5

_fast_full_scan_enabled         = FALSE

query_rewrite_enabled  = true

_sqlexec_progression_cost=2147483647

optimizer_secure_view_merging=false

_optimizer_autostats_job=false # Turning off auto statistics

LABDB1.LOCAL_LISTENER=LISTENER_LABDB1   

LABDB2.LOCAL_LISTENER=LISTENER_LABDB2   

LABDB3.LOCAL_LISTENER=LISTENER_LABDB3   

REMOTE_LISTENER=LISTENERS_LABDB

LABDB1.thread=1

LABDB2.thread=2

LABDB3.thread=3

LABDB1.instance_number=1

LABDB2.instance_number=2

LABDB3.instance_number=3

db_recovery_file_dest_size=20971520000

db_recovery_file_dest='+FRA'

db_create_file_dest='+DATA'

db_create_online_log_dest_1='+DATA'

db_create_online_log_dest_2='+FRA'

Created the spfile +DATA/LABDB/spfileLABDB.ora from the above pfile.

export ORACLE_HOME=/u01/app/oracle/EBSR12

export ORACLE_SID=LABDB1

$ORACLE_HOME/bin/sqlplus / as sysdba  ß Database was still down at this point. Only connected to an idle instance to create spfile

create spfile='+DATA/LABDB/spfileLABDB.ora' from pfile='/tmp/initLABDB.ora';

Create the NEW_11GR1_RAC_HOME/dbs/init<SID>.ora on all the RAC Nodes with the below entry in it.

node1.hingu.net:/u01/app/oracle/EBSR12/dbs/initLABDB1.ora

spfile='+DATA/LABDB/spfileLABDB.ora'

node2.hingu.net:/u01/app/oracle/EBSR12/dbs/initLABDB2.ora

spfile='+DATA/LABDB/spfileLABDB.ora'

node3.hingu.net:/u01/app/oracle/EBSR12/dbs/initLABDB3.ora

spfile='+DATA/LABDB/spfileLABDB.ora'

Started the instance LABDB1 on node1 from NEW_11GR1_RAC_HOME:

The instance LABDB1 was started after setting the appropriate env variables on node1. The cluster_database parameter was still not enabled at this stage.

export ORACLE_HOME=/u01/app/oracle/EBSR12

export ORACLE_SID=LABDB1

$ORACLE_HOME/bin/sqlplus / as sysdba

startup

Created extra Redo Threads and Undo Tablespaces for the RAC instances LABDB2 and LABDB3 on ASM diskgroup DATA:

create undo tablespace APPS_UNDOTS2 datafile '+DATA' size 100M autoextend on next 1M maxsize 1G extent management local;

create undo tablespace APPS_UNDOTS3 datafile '+DATA' size 100M autoextend on next 1M maxsize 1G extent management local;

alter database add logfile thread 2 group 3 size 100M;

alter database add logfile thread 2 group 4 size 100M;

alter database add logfile thread 3 group 5 size 100M;

alter database add logfile thread 3 group 6 size 100M;

alter database enable public thread 2;

alter database enable public thread 3;

Move the database LABDB to ASM diskgroup DATA:

Move the Online Redo Logs belonging to LABDB1 instance from local Filesystem to ASM diskgroup DATA

sqlplus / as sysdba

alter database add logfile thread 1 group 50 size 10M;

alter database drop logfile group 1;

alter database add logfile thread 1 group 1 size 100M;

alter database drop logfile group 2;

alter database add logfile thread 1 group 2 size 100M;

alter database drop logfile group 50;

Move the TEMPORARY TABLESPACE from local Filesystem to ASM diskgroup DATA

create temporary tablespace temptbs tempfile '+DATA' size 100M;

alter database default temporary tablespace temptbs;

drop tablespace temp1 including contents and datafiles;

drop tablespace temp2 including contents and datafiles;

Move the control files from local Filesystem to ASM diskgroup DATA

SQL> alter system set control_files='+DATA','+FRA' scope=spfile sid='*';

shutdown immediate

startup nomount

[oracle@node1 ~]$ rman target /

Recovery Manager: Release 11.1.0.7.0 - Production on Sun Oct 30 17:03:53 2011

Copyright (c) 1982, 2007, Oracle.  All rights reserved.

connected to target database: LABDB (not mounted)

RMAN> restore controlfile from '/u01/app/oracle/ebsR12/db/apps_st/data/cntrl01.dbf';

Starting restore at 30-OCT-11

using target database control file instead of recovery catalog

allocated channel: ORA_DISK_1

channel ORA_DISK_1: SID=376 device type=DISK

channel ORA_DISK_1: copied control file copy

output file name=+DATA/labdb/controlfile/current.267.765911037

output file name=+FRA/labdb/controlfile/current.263.765911059

Finished restore at 30-OCT-11

SQL> alter system set control_files='+DATA/labdb/controlfile/current.267.765911037','+FRA/labdb/controlfile/current.263.765911059' scope=spfile sid='*';

Move the datafiles from local Filesystem to ASM diskgroup DATA

startup mount;

BACKUP AS COPY DATABASE FORMAT ‘+DATA’;

switch database to copy;

alter database open;