Sunday, 23 October 2022

Exadata Database Machine and Exadata Storage Server Supported Versions (Doc ID 888828.1)

 Dear's ..


Exadata Maser Notes :


Exadata Database Machine and Exadata Storage Server Supported Versions (Doc ID 888828.1)

PURPOSE

This document lists the software patches and releases for Oracle Exadata Database Machine. This document includes versions for both the database servers and storage servers of Oracle Exadata Database Machine with database servers running Intel x86-64 processors.


For an index and references to the most frequently used My Oracle Support Notes with respect to Oracle Exadata and Oracle Exadata Database Machine environments, refer to the Primary Note for Oracle Exadata Database Machine and Exadata Storage Server Note 1187674.1.


SCOPE

The information in this document applies only to Exadata software 11.2 and higher.  It does not apply to any previous version of Exadata software. Current releases for other Exadata software versions is maintained in a different note.


Note: The currently supported versions may change frequently, so it is important to review this document immediately prior to any Oracle Exadata Database Machine deployment.



Thanks,

Srini

Thursday, 29 September 2022

How to run Autoconfig on EBS r12.2 patch and run FS.

 

If we are changing any parameter in the Context file in R12.2 environment then the parameter should be change in both the RUN and PATCH file system context file and then we should run the Autoconfig.

Steps:


Stop all the Services


On RUN File System

1. Source the RUN file system environment.

2. Modify the Parameter in Context File

3. Run Autoconfig on RUN Filesystem

  cd $ADMIN_SCRIPTS_HOME

  ./adautocfg.sh

  

Important Steps

Disable the EBS LOGON TRIGGER

Connect to System User

  conn system/manager

  alter trigger ebs_logon disable;

  

On Patch File System  

1. Source the PATCH file system environment.

2. Modify the Parameter in Context File

3. Run Autoconfig on PATCH Filesystem

  cd $ADMIN_SCRIPTS_HOME

  ./adautocfg.sh

 Important Steps

Enable the EBS LOGON TRIGGER

Connect to System User

  conn system/manager

  alter trigger ebs_logon enable; 

 

Start all services  


Thanks,

Srini


Wednesday, 28 September 2022

Oracle R12.2 Patching bugs and fixups

 General

In this blog I have discussed fixes/workaround issues faced by our customers on ADOP.


Following is the list of occurred issues:


FS_CLONE/Prepare issues due to missed Patch Context File on DB

FS_CLONE/Prepare due to less /usr/tmp Space

FS_CLONE while wlst2apply apply

Node Abandon

Invalid Cross References

FS_CLONE/Prepare issue due to missed Patch Context File on DB

As part of cloning process, sometime, we ignore running autoconfig on Patch File system to gain time of delivering clone instance, and this leads to patch context files missing on the database. While running ADOP database context files get validated. However, due to unavailability of patch file system context files in database we get the following error.

Logfile Error details:


Validating configuration on node: [Appl_Host].


Log: /u01-DEV/app/appti/DEV/fs_ne/EBSapps/log/adop/98/20220131_021858/fs_clone/validate/Appl_Host

[UNEXPECTED]Error occurred running "perl /u01-DEV/app/appti/DEV/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPValidations.pl  -contextfile=/u01-DEV/app/appti/DEV/fs1/inst/apps/DEV_Appl_Host/appl/admin/DEV_Appl_Host.xml -patchctxfile=/u01-DEV/app/appti/DEV/fs2/inst/apps/DEV_Appl_Host/appl/admin/DEV_Appl_Host.xml -phase=fs_clone -logloc=/u01-DEV/app/appti/DEV/fs_ne/EBSapps/log/adop/98/20220131_021858/fs_clone/validate/Appl_Host -promptmsg=hide"

[UNEXPECTED]Error 1 occurred while Executing txkADOPValidation script on Appl_Host

Check Validation log:


NONPROD [appti@Appl_Host Appl_Host]$ cat txkADOPValidations.error Use of uninitialized value $result in split at /u01-DEV/app/appti/DEV/fs1/EBSapps/appl/au/12.0.0/perl/TXK/ADOPValidationUtils.pm line 1294. No such file or directory at /u01-DEV/app/appti/DEV/fs1/EBSapps/appl/au/12.0.0/perl/TXK/ADOPValidationUtils.pm line 230.


Issue: Issue occurred due to no Contextfile of Patch context file in fnd_oam_context_files . Use the following query to check Patch context file:


SQL> select * from FND_OAM_CONTEXT_FILES

where NAME not in (‘TEMPLATE’,‘METADATA’,‘config.txt’)

and CTX_TYPE=‘A’

and (status is null or upper(status) in (‘S’,‘F’))

and EXTRACTVALUE(XMLType(TEXT),'//file_edition_type') = ‘patch’; 2 3 4 5

no rows selected


Fix:


Upload Patch Context to DB using the following command:


Source Run file environment  $ADJVAPRG oracle.apps.ad.autoconfig.oam.CtxSynchronizer action=upload contextfile=<fulle patch of Patch Contex_file> logfile=<fulle path of logfile>  $ADJVAPRG oracle.apps.ad.autoconfig.oam.CtxSynchronizer action=upload contextfile=/u01-DEV/app/appti/DEV/fs2/inst/apps/DEV_Appl_Host/appl/admin/DEV_Appl_Host.xml logfile=/tmp/patch_context_upload.log


SQL> select path from FND_OAM_CONTEXT_FILES

where NAME not in (‘TEMPLATE’,‘METADATA’,‘config.txt’)

and CTX_TYPE=‘A’

and (status is null or upper(status) in (‘S’,‘F’))

and EXTRACTVALUE(XMLType(TEXT),'//file_edition_type') = ‘patch’; 2 3 4 5


PATH

/u01-DEV/app/appti/DEV/fs2/inst/apps/DEV_Appl_Host/appl/admin/DEV_Appl_Host.xml Query with Output :


Now FS_CLONE should complete fine


FS_CLONE/Prepare errors due to less /usr/tmp Space

Due to less /usr/tmp space on the host, you may encounter the following error while running adop fs_clone or in the prepare stage

Error details: START: Creating FMW archive.


Running /u01-DEV/app/appti/DEV/fs1/FMW_Home/oracle_common/bin/copyBinary.sh -javaHome /u01-DEV/app/appti/DEV/fs1/EBSapps/comn/adopclone_Appl_Host/FMW/t2pjdk -al /u01-DEV/app/appti/DEV/fs1/EBSapps/comn/adopclone_Appl_Host/FMW/FMW_Home.jar -smw /u01-DEV/app/appti/DEV/fs1/FMW_Home -ldl /u01-DEV/app/appti/DEV/fs1/inst/apps/DEV_Appl_Host/admin/log/clone/fmwT2PStage -invPtrLoc /etc/oraInst.loc -silent true -debug true Script Executed in 27558 milliseconds, returning status 255 ERROR: Script failed, exit code 255

Issue: Insufficient space in /usr/tmp . Require Min 15G free space .


Fix:

Create a directory with 16G available space

`mkdir -p /u02-DEV/app/appti/DEV/temp

T2P_JAVA_OPTIONS="-Djava.io.tmpdir=/u02-DEV/app/appti/DEV/temp"

TMP=/u02-DEV/app/appti/DEV/temp

TEMP=/u02-DEV/app/appti/DEV/temp

TMPDIR=/u02-DEV/app/appti/DEV/temp

export T2P_JAVA_OPTIONS

export TMP

export TEMP

export TMPDIR`

Now rerun FS_CLONE


FS_CLONE while wlst2apply apply

ADOP FS_CLONE/Prepare running creating wlst2apply errors out due to less JVM memory.


Error details: .jar -server_start_mode=prod -log=/u01-DEV/app/appti/DEV/fs1/inst/apps/DEV_Appl_Host/logs/appl/rgf/TXK/CLONINGCLIENT7100708042842072347/unpack.log -log_priority=debug « read template from “/u01-DEV/app/appti/DEV/fs1/inst/apps/DEV_Appl_Host/logs/appl/rgf/TXK/CLONINGCLIENT7100708042842072347/packed_template.jar”


succeed: read template from “/u01-DEV/app/appti/DEV/fs1/inst/apps/DEV_Appl_Host/logs/appl/rgf/TXK/CLONINGCLIENT7100708042842072347/packed_template.jar” « set config option ServerStartMode to “prod” succeed: set config option ServerStartMode to “prod” « write Domain to “/u01-DEV/app/appti/DEV/fs2/FMW_Home/user_projects/domains/EBS_domain” Exception in thread “Thread-1” java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOfRange(Arrays.java:2694) at java.lang.String.(String.java:203) at java.io.BufferedReader.readLine(BufferedReader.java:349) at java.io.BufferedReader.readLine(BufferedReader.java:382) at com.oracle.cie.common.util.CRLF.readData(CRLF.java:129) at com.oracle.cie.common.util.CRLF.processFile(CRLF.java:67) at com.oracle.cie.common.util.CRLF.process(CRLF.java:57) at com.oracle.cie.domain.util.stringsub.SubsScriptHelper.processCRLF(SubsScriptHelper.java:167) at com.oracle.cie.domain.DomainGenerator.generate(DomainGenerator.java:478) at com.oracle.cie.domain.script.ScriptExecutor$2.run(ScriptExecutor.java:2991)


Issue:

Insufficient JVM heap size to complete fs_clone

Fix:

export CONFIG_JVM_ARGS="-Xmx2048m  -Xms1024m -XX:MaxPermSize=512m -XX:-UseGCOverheadLimit"

Run FS_CLONE

Node Abandoned

When the node gets abandoned during adop in FS_CLONE/PREPARE, you can fix the abandoned node by:


Note: Try this workaround only with issues in the FS_CLONE, PREPARE phase. If the issue occurs in Cutover phase in multi node environment try Standard process of Deleting and Adding Abandoned Node

Error:

Checking for existing adop sessions.

No pending session exists.

Starting new adop session.

[ERROR] Nodes Appl_Host_Admin,Appl_Host_EXternal were abandoned in previous patching cycle

[ERROR] To recover these nodes, follow the instructions in My Oracle Support Knowledge Document 1677498.1

[UNEXPECTED]Unrecoverable error occurred. Exiting current adop session.


Fix:


a) Get current adop_session_id from ad_adop_sessions table . select * from ad_adop_sessions order by adop_session_id desc;


b) Take backup of ad_adop_sessions Create table applsys.ad_adop_sessions_bkp21jan22 as select * from ad_adop_sessions;


c)update ad_adop_sessions set abandon_flag=null where adop_session_id=156; Then run adop -validate* to check all nodes are validated successfully


Invalid Cross References

ValidateOHSConfigFilesForCrossRef …

    WARNING: File - /u01-DEV/app/appti/DEV/fs2/FMW_Home/webtier/instances/EBS_web_OHS1/bin/opmnctl contains cross reference in it.

    WARNING: File - /u01-DEV/app/appti/DEV/fs2/FMW_Home/webtier/instances/EBS_web_OHS1/bin/opmnctl contains cross reference in it.

    WARNING: File - /u01-DEV/app/appti/DEV/fs2/FMW_Home/webtier/instances/EBS_web_OHS1/bin/opmnctl contains cross reference in it.

   Corrective Action: Contact Oracle Support to identify the best course of action.

Check ADOP Validation Log in $ADOP_LOG_HOME/ 98/20220205_063351/validate/<Appl_Host> Fix:

Correct opmnctl file in $FMW_HOME/ webtier/instances/EBS_web_OHS1/bin with file system


Take backup before making changes


Thanks,

Srini

Patch script's in EBS 12.2

 How To Check if a Patch is Applied in 12.2.x using SQL*PLUS


In EBS 12.2 we all know that Online patching has been introduced. Thus during a patching cycle if a patch has been applied using adop phase=apply and later we aborted it before the cycle gets complete, the table ad_bugs and ad_applied_patches will not be updated. So we cannot query these two tables. As per metalink note:Doc ID 1963046.1, use the below query to find out if a patch has been applied or not.


**************************************************

select AD_PATCH.IS_PATCH_APPLIED(\'$release\',\'$appltop_id\',\'$patch_no\',\'$language\')

from dual;



Example sql using the APPL_TOP ID for a specific node in a multinode environment (1045 is the APPL_TOP ID):


SELECT adb.bug_number,ad_patch.is_patch_applied('R12', 1045, adb.bug_number)

FROM ad_bugs adb

WHERE adb.bug_number in (20034256);


or for single app tier installations:


select ad_patch.is_patch_applied('R12',-1,20034256) from dual;


expected results:

EXPLICIT = applied

NOT APPLIED = not applied / aborted



Also don't forget to complete the APPL_TOP snapshot using adadmin option.

***********************************************************************


ADOP Online patching concepts


adop phases:


prepare       : Prepare the instance for online patching.

apply         : Apply patch(es) to the Patch Edition.

finalize      : Ready the instance for cutover.

abort         : Abort the patching cycle.

cutover       : Promote the Patch Edition to Run Edition.

cleanup       : Drop obsolete objects and seed data from Old Editions.



actualize_all : Actualize all objects in the Patch Edition.

cleanup_full  : Cleanup and drop Old Editions.

abandon       : yes|no - Abandon failed patches.




adop patch log directory:


<INSTALL BASE>/fs_ne/EBSapps/log/adop



adop patch process cycle steps:


Download any required technology patches and unzip the contents. The patch contents may be unzipped into  $NE_BASE/EBSapps/patch.


1. Prepare the system for patching.


   source <EBS_ROOT>/EBSapps.env run


   $ adop phase=prepare


2. Apply technology patches to the Oracle Home under the Patch f/s using the information below.


   source <EBS_ROOT>/EBSapps.env patch


3. Apply any Oracle E-Business Suite patches planned for this patching cycle


   $ adop phase=apply patches=<patch_list>


4. After all patches have been successfully applied, complete the patching cycle.


   $ adop phase=finalize

   $ adop phase=cutover


Note: Below steps can be done when applications are up and running


5. source <EBS_ROOT>/EBSapps.env run

 

   $ adop phase=cleanup


6. To complete the process and synchronize the technology level between patch and run f/s.


   $ adop phase=fs_clone




adop hotpatch steps (we cannot abort hotpatch and abondon cannot be done):


Hotpatch which can apply directly on run fs


$ adop phase=apply patches=<patch_list> hotpatch=yes


After hotpatch please run phase=cleanup and phase=fs_clone to sync the run fs with patch fs to prepare for next patching cycle




adop re-apply patch forcefully:


If we try to re-apply the patch which is already applied/exists then adop patch terminates

with below message and by default it takes N when prompted. to overcome this we need to

re-apply patch with options=forceapply.

This Patch seems to have been applied already.

Would you like to continue anyway  [N] ? N *



$ adop phase=apply patches=<patch list> hotpatch=yes options=forceapply




 adop deal with "Continue As If It Were Successful" error:




$ adop phase=apply patches=<patch list> abandon=no restart=yes flags=autoskip




To define workers in adop:




$ adop phase=apply patches=<patch list> workers=5




To define patchtop in adop:




$ adop phase=apply patches=<patch list> patchtop=<patch location base>




adop merge patch:




$ adop phase=apply patches=<patch list> merge=yes




Restarting adop From A Failed Session:





 $ adop phase=abort

 $ adop phase=cleanup cleanup_mode=full

 $ adop phase=fs_clone


Then reapply the patch




adop apply for language patch:




$ adop phase=apply patches=18023722_ESA:u18023722.drv




adop non-interactive with patch top and define driver:






$ adop phase=apply options=nocopyportion patchtop=$XLA_TOP/patch/115 patches=driver:xla5584908.drv




adop Steps to follow to skip the failed workers:




1. Use adctrl and select option#8 (This will not be visible) to skip thefailed jobs

2. Restart adop using "restart=yes" parameter


** If the failed jobs are numerous, then better to re-start this patch once again with autoskip option.


ie.  adop restart=no abandon=yes flags=autoskip


This command will restart the patch once again from starting onwards and will skip all the failures if any comes. But make sure to review the log file at the end of the patch application that you have skipped the failures that you want to.





Weblogic Server Smart Update Patching :


Steps to apply weblogic server smart update patch:



Refer Note:

How to Apply WebLogic Server (WLS) Patches Using Smart Update [Video] (Doc ID 876004.1)


1.Download the patch

2.Copy the files (for example, E5W8.jar and WGQJ.jar) and the patch-catalog_xxx.xml from the zip file to the target machine. You do not need the readme file.

Path: $FMW_HOME/utils/bsu/cache_dir


3.cd $FMW_HOME/utils/bsu


To install patch:

./bsu.sh -prod_dir=$FMW_HOME/wlserver_10.3 -patchlist=7FC9 -verbose -install


To Verify:

./bsu.sh -prod_dir=$FMW_HOME/wlserver_10.3 -status=applied -verbose -view | grep 7FC9


To Rollback:

./bsu.sh -remove -patchlist=7FC9 -prod_dir=$FMW_HOME/wlserver_10.3 -verbose


We can also apply the Smart Update patch in graphical (GUI) mode.





Steps to apply opatch on FMW Web Tier HOME :



Set the Environment as below (replace <INSTALL_BASE> path as required):


$ export ORACLE_HOME=$FMW_HOME/webtier

$ export PATH=$ORACLE_HOME/OPatch:$PATH


$ opatch lsinventory



Apply opatch:


$ opatch apply





Steps to apply opatch on FMW oracle_common HOME :



Set the Environment as below:


$ export ORACLE_HOME=$FMW_HOME/oracle_common

$ export PATH=$ORACLE_HOME/OPatch:$PATH


$ opatch lsinventory


Apply opatch:


$ opatch apply





Queries :


 Query to check if a patch is applied in Oracle EBS R12.2.x:


In Oracle E Business Suite (ebs erp) R12.2.x you cannot query the AD_BUGS table to check if patches have been applied..



The AD_BUGS table may have entries for patches that were applied but later the patching cycle was aborted (not really applied).





The way to check whether a patch is really applied is to use the AD_PATCH.IS_PATCH_APPLIED PL/SQL function.


Usage:


select AD_PATCH.IS_PATCH_APPLIED(\'$release\',\'$appltop_id\',\'$patch_no\',\'$language\')

from dual;


Example sql:


SELECT adb.bug_number,ad_patch.is_patch_applied('11i', 1045, adb.bug_number)

FROM ad_bugs adb

WHERE adb.bug_number in (20034256);


or for single app tier installations:


select ad_patch.is_patch_applied('R12',-1,20034256) from dual;


Expected results:


EXPLICIT = applied

NOT APPLIED = not applied / aborted


Note: If you are sure patch is applied, but showing as not applied then do the following workaround.



1. Start adadmin after source the RUN FS env.

2. Select "2. Maintain Applications Files menu" in "AD Administration Main Menu".

3. In "Maintain Applications Files", select "4. Maintain snapshot information".

4. Select "2. Update current view snapshot" in the "Maintain Snapshot Information".

5. Select "1. Update Complete APPL_TOP" in the "Maintain Current View Snapshot Information".



Query to check current AD patchset:


SELECT a.application_short_name, b.patch_level

FROM fnd_application_vl a,fnd_product_installations b

WHERE a.application_id = b.application_id

  and application_short_name = 'AD';




Query to check patches applied correctly and in the expected sequence:




1.1.Run this sql statement:


   select * from ad_adop_session_patches order by end_date desc;


1.2. Run this piece of sql code:


   set pagesize 200;

   set linesize 160;

   column adop_session_id format 999999999999;

   column bug_number format a15;

   column status format a15;

   column applied_file_system_base format a23;

   column patch_file_system_base format a23;

   column adpatch_options format a15;

   column node_name format a15;

   column end_date format a15;

   column clone_status format a15;


   select ADOP_SESSION_ID, BUG_NUMBER, STATUS, APPLIED_FILE_SYSTEM_BASE, PATCH_FILE_SYS   TEM_BASE, ADPATCH_OPTIONS, NODE_NAME, END_DATE, CLONE_STATUS

   from ad_adop_session_patches

   order by end_date desc;



Below are possible values of STATUS column:

N - Not Applied In the current node but applied in other nodes

R - Patch Application is going on.

H - Patch failed in the middle. (Hard Failure)

F - Patch failed in the middle but user tried to skip some failures.

S - Patch Application succeeded after skipping the failed jobs.

Y - Patch Application succeeded.

C - Reserved for clone and config_clone. Indicates clone completed




Query to Check AD and TXK C Patch levels:




SELECT codelevel FROM AD_TRACKABLE_ENTITIES WHERE abbreviation in ('txk','ad');




Restarting adop


1 If you have shut down the workers, or if adop quits while performing processing actions, it saves all the actions completed up to that point in restart files.

Investigate and Patching Utilities resolve the problem that caused the failure,



then restart adop. After you restart adop, it will ask if you want to continue with the previous session (at the point where the processing stopped), or start a new session.

To restart from beginning,  use restart=no abandon=yes

To restart a patch you need to use restart=yes abandon=no .


eg. adop restart=no abandon=yes  phase=apply

If still can not restart,

2 Check the contents of the AD_ADOP_SESSIONS and AD_ADOP_SESSION_PATCHES tables

---------------------------------------------------------------------------------------

2.1

column id format 99

column nn format a10

column nt format a6

select adop_session_id id, prepare_status pr, apply_status ap,

finalize_status fi, cutover_status cu, cleanup_status cl, abort_status ab, status st, node_name nn, node_type nt

from ad_adop_sessions

order by adop_session_id desc;


2.2

column id format 99

column bn format a12

column pr format 99999999

column afs format a40

column pfs format a40

column cs format a9

column nn format a10

column ed format a28

column drv format a28

column pt format a35


select adop_session_id id, bug_number bn, patchrun_id pr, status st,

node_name nn, cast(end_date as timestamp) ed, driver_file_name drv, patch_top  pt

from ad_adop_session_patches

order by end_date desc;


2.3 explanation example of above output

=========================

 Examining the AD_ADOP_SESSIONS table,

 ID P A F C C A S NN         NT

 --- - - - - - - - ---------- ------

3 X P N X N X F test2    master

2 R N N N N Y C test2    master


It looks like the first ADOP cycle (prepare phase) was attempted at session ID 2 and then for some reason it didn't finish and then an abort was  initiated which did finish.  A new ADOP session (ID 3) was attempted but via hotpatch and it failed.  Since this table only reports what went on with ADOP there's no information about adpatch usage, i.e. if it was used before the

 first ADOP cycle or in between session 2 and 3, or after session 3.




2.4 To diagnostic further for the root cause why patch application failed, we need adpatch log for the failed patch xxxx.

cd  $APPL_TOP_NE/../log/adop/<session id>

eg.

 /d01/EBS/VIS/fs_ne/EBSapps/log/adop/<session id>/timestamp/context_name/xxxx/log


3 If you are just wants to start from a fresh page and considering the state of these tables

I would suggest making a manual modification to the  AD_ADOP_SESSIONS table just so it would not hold up ADOP execution due to an

incomplete hotpatch session.

 Do the following:

update ad_adop_sessions set status='C' where adop_session_id=3;


4 restart adop and retest.


How to Synchronize the AD tables FND_NODES, ADOP_VALID_NODES, and FND_OAM_CONTEXT_FILES in 12.2 when adop fails. (Doc ID 2064223.1)


thanks,
Srini