Weblogic stuck threads Overview in EBS R12.2.X

 What is stuck threads in weblogic server?
Stuck threads are  long running threads and It means thread has been running longer predefined timeout.
 
What stuck threads will do?
A thread might get stuck for various reasons.
 
Example:
 When an SQL query is running and the time it takes to complete is say 800 seconds, then, as the default stuck thread timing is 600 seconds in WebLogic Server, the thread allocated for that query waits for 600 seconds and goes to stuck state.
Then managed server status showing waning state
 
How to troubleshoot the stuck threads in EBS instance?
Stuck threads will reduce the ebs performance and it’s very important activity to DBA’s
Step 1: Go to the weblogic console and check for managed server status like oacore
Step 2: if It will show the oacore server in “warning state” then we need to check
 
Scenario 1:
Every alert we need to check the log file and investigate based on the log file what caused this.
 
Scenario 2:
Need to check execution context id (ECID) 
How to find ECID ?
There are two way to find the ECID
 one  is need to check weblogic console 
 And another is need to check log file
 
Go to the weblogic console
Click OA-Core server->Monitoring->Threads
OA-Core health showing “Warning”
Click àwarning
Scroll down click next->next page show the stuck threads
Find ECID
 
Example below value we can see
 
ECID-Context: 1.005ZMgd6^_005ej6ub5B97q2OayhmZMG000FoR00015X;

Need to check this value only ECID   005ZMgd6^_005ej6ub5B97q2OayhmZMG000FoR00015X

Find the ecid in logfile
 grep -e STUCK -e ECID $FMW_HOME/user_projects/domains/EBS_domain/servers/oacore_server9/logs/oacore_server9.out|grep ECID|awk {'print $2'}|sed 's/^.\{2\}//'|sed 's/.\{5\}$//' 

Find the ecid details using v$session using below queries

select 'alter system kill session '''||sid||','||serial#||''' immediate;', program,trunc(last_call_et/60) last_call_et_minutes,module,
action,ecid,username,schemaname,client_identifier, process, machine, sql_id, status,blocking_session_status,event,state
from v$session where program like 'oacore_server6'
--and ecid like '%%'
-- and action like '%%'
order by program,last_call_et desc;

'ALTERSYSTEMKILLSESSION'''||SID||','||SERIAL#||'''IMMEDIATE;'
--------------------------------------------------------------------------------
alter system kill session1234,56789 immediate;
After killing SID, verify if "WARNING" message is gone and health is showing "OK" now
Reference:
EBS 12.2 Oacore JVMs Getting Into Warning State Because Of STUCK Threads (Doc ID 2061045.1)

Thanks,
Srini