Overview of Concurrent Processing

Concurrent Managers

Concurrent Processing is an Oracle E-Business Suite feature that allows these non–interactive and potentially long-running functions to be executed efficiently alongside interactive operations. It uses operating system facilities to enable background scheduling of data- or resource-intensive jobs, via a set of programs and forms.To ensure that resource-intensive concurrent processing operations do not interfere with interactive operations, they are run on a specialized server, the Concurrent Processing server.

Processes that run on the Concurrent Processing server are called concurrent requests. When you submit such a request, either through HTML-based or Forms-based applications, a row is inserted into a database table specifying the program to be run. A concurrent manager then reads the applicable requests in the table, and starts the associated concurrent program.

Concurrent Manager Characteristics

Concurrent managers are fundamental to concurrent processing. Acting as a job scheduling and execution system, a concurrent manager:  

1. Is an executable that is registered as a program library within Oracle E-Business Suite, and which runs in its own operating system process

2. Runs operating system processes called target processes (often referred to as workers), each of which can start one concurrent program at a time

3. Can optionally run an immediate program that runs as part of the concurrent manager’s own operating system process

4. Can be allowed to run any concurrent program, or be specialized to run certain programs

5. Operates during the days and times defined by a work shift


There are a number of specialized concurrent managers 

The Internal Concurrent Manager (ICM) controls all other concurrent managers. It administers the startup and shutdown of managers as defined by their work shift, monitors for process failure, and cleans up if a failure occurs. The ICM does not process concurrent requests itself (except for queue control requests, such as ACTIVATE, DEACTIVATE, or ABORT).

While the basic ICM definition should not be changed, you can if required modify the sleep time (number of seconds the ICM waits between checking for new concurrent requests), PMON (process monitor) cycle time (number of sleep cycles the ICM waits between checking for failed workers), and queue size (duration between checks for number of active workers, measured in PMON cycles). If Parallel Concurrent Processing is being used, you can also set some options for this.

The Conflict Resolution Manager (CRM) enforces rules designed to ensure that incompatible concurrent requests do not run in the same conflict domain  (an abstract representation of the groupings used to partition data). As with the Internal Concurrent Manager, the basic CRM definition should not be changed, but you can modify the sleep time for each work shift, as well as some Parallel Concurrent Processing options.

The Standard Manager  as shipped with Oracle E-Business Suite will accept and run any concurrent requests, as it has no specialization rules that would restrict its activities. Consequently, the definition of the Standard Manager should not be altered without careful planning, otherwise some programs might not be able to run at all. Jobs should only be excluded from the Standard Manager after ensuring they can be run by an alternative manager, such as a product-specific manager or user-defined manager.

Transaction Managers support synchronous request processing, whereby a pool of server processes responds to requests from client programs. Instead of polling the concurrent requests table to obtain instructions, a transaction manager waits to be signaled by a client. An example is approval of an order, where execution of the request must take place quickly.

The relevant transaction manager program runs on the server, transparently to the client. All transaction programs for a given manager process run in the same database session. Communication between the client and the server is conducted synchronously via Advanced Queueing.

Multiple managers can be run on multiple nodes using Parallel Concurrent Processing

A concurrent request has a life cycle, which consists of three or possibly four phases:

Phase	Activity
Pending	The request is waiting to be run
Running	The request is running
Completed	The request has finished
Inactive	The request cannot be run

A concurrent program library  contains concurrent programs that can be called by a concurrent manager. An important example is the Oracle Application Object Library program library (FNDLIBR), which contains Oracle E-Business Suite immediate concurrent programs, and is assigned to the standard concurrent manager.

Various database tables are employed by the concurrent processing architecture:

Table	Content
FND_CONCURRENT_REQUESTS	Details of user requests, including status, start date, and completion date
FND_CONCURRENT_PROGRAMS	Details of concurrent programs, including execution method, whether the program is constrained, and whether it must be run alone.
FND_CONCURRENT_PROCESSES	Cross-references between concurrent requests and queues, and a history of concurrent manager processes
FND_CONCURRENT_QUEUES	Information about each of the concurrent manager queues

NOTE

Do not update these tables manually. You can (subject to your organization's archiving requirements) periodically run the "Purge Concurrent Requests and/or manager data" program to prevent these tables growing too large.

Concurrent Processing Operations

Because the Internal Concurrent Manager controls all the other managers, it must be running before any other manager can be activated. Once the ICM has been activated, it starts a Service Manager on each node that is enabled for concurrent processing. Acting as an agent of the ICM, the Service Manager starts the concurrent managers on its node, excluding any managers that have been deactivated, or that have no current work shift. 

The ICM can be activated and deactivated from the operating system prompt, or Oracle Applications Manager. It can also be deactivated (but not activated) from the Administer Concurrent Managers form.

When the ICM is initiated on UNIX, the $FND_TOP/bin/startmgr program is invoked. This calls $FND_TOP/bin/batchmgr, which then:

1. Starts a shell process
   
2. Starts the ICM process using the command FNDLIBR, with startup parameters FND, CPMGR, and FNDCPMBR
   
3. Creates log files (std.mgr and wnnn.mgr) in $APPLCSF/$APPLLOG
   

Normally, startmgr is run by the user account that owns the application software (for example, applmgr). This account must have write privileges to the log and out directories where the log and output files respectively are written.

The ICM starts up a Service Manager on each node that is enabled for concurrent processing, by instructing the node's Applications Listener (which is dedicated to Concurrent Processing) to spawn a process running the Service Manager executable (FNDSM). The Applications Listener must be configured to source the Oracle E-Business Suite environment file before FNDSM is spawned. Following startup, the Service Manager acts as an agent of the ICM to start and stop concurrent managers on that node, according to their defined work shifts.

Note: 

The Service Manager is a component of the Generic Service Management (GSM) architecture rather than Concurrent Processing, although GSM and Concurrent Processing are closely integrated.

Concurrent manager processes on a specific node can be seen by running the UNIX commands:

ps –ef | grep FNDLIBR

ps –ef | grep FNDSM

The Service Manager PID seen in the output of the second command can then, if desired, be used to locate all concurrent manager and service processes on the node, since the Service Manager is the parent process for them:

ps –ef | grep <sm_pid>

On Windows, the Task Manager can be used to locate concurrent manager processes. An FNDLIBR process runs for the Internal Concurrent Manager and each standard manager. The ICM can be distinguished by additional details being displayed, including some of the parameters it was started with.

For every process that was successfully started at operating system level, the ICM inserts a row into FND_CONCURRENT_PROCESSES. It then updates the RUNNING_PROCESSES column to reflect the actual running processes as shown in FND_CONCURRENT_QUEUES.

Viewing Concurrent Processing Output

The output from a concurrent processing job goes through several stages before being displayed to the user.

1. The Concurrent Processing server communicates with the database server via Oracle Net.

2. The log or output file associated with a concurrent request is passed back to the Report Review Agent, also known as the Web Review Agent.

3. The Report Review Agent passes a file containing the entire report to the Forms services.

4. The Forms services pass the report back to the user's browser one page at a time.

Parallel Concurrent Processing

Parallel Concurrent Processing (PCP) allows concurrent processing activities to be distributed across multiple nodes in an Oracle Real Application Clusters (Oracle RAC) environment or similar cluster system. By distributing concurrent processing in this way, hardware resources can be fully utilized, maximizing throughput and providing resilience to node failure, while retaining a central point of control.

Parallel Concurrent Processing enables you to:

• Run concurrent processes on multiple nodes to improve concurrent processing throughput
  
• Continue running concurrent processes on the remaining nodes when one or more nodes fail
  
• Administer concurrent managers running on multiple nodes from any node in the cluster
  

One or more concurrent managers can be specified to run on one or more nodes, to best suit your processing needs and fully utilize available hardware resources.

Parallel Concurrent Processing is enabled by default, so PCP is always available for use in environments where one or more concurrent processing nodes exist.

Note:

PCP does not require an Oracle RAC environment. Conversely, you do not have to use PCP in an Oracle RAC environment, although it typically makes sense to do so.

Managing Concurrent Processing

From the command line, two commands can be entered to control the Internal Concurrent Manager:

startmgr, which starts the ICM; and concsub, which is used to stop or abort the ICM, or request the ICM to check on the operating system process for each manager.

In addition, an AutoConfig-enabled environment provides a number of scripts for starting and stopping application tier services from the command line.

The script for concurrent processing startup and shutdown is INST_TOP/admin/scripts/adcmctl.sh.

The various components of the concurrent processing system can be managed from various forms, such as Concurrent Manager: Administer, or from Concurrent Managers or Concurrent Requests under Oracle Applications Manager (OAM).


CONSUB : Utility allows user to submit concurrent request from operating system level

IMP Links:
---------
Concurrent Processing - Product Information Center (PIC) (Doc ID 1304305.1) - https://support.us.oracle.com/oip/faces/secure/km/DocumentDisplay.jspx?id=1304305.1&recomm=Y

https://support.us.oracle.com/oip/faces/secure/km/DocumentDisplay.jspx?id=1057802.1

NOTE:1389261.1 - PCP Concurrent Manager Failover/Failback Does not work When Application Listener is down on Primary Node
NOTE:1060707.1 - Purge Concurrent Requests/Manager Data, FNDCPPUR, Not Removing Files From Filesystem
NOTE:466752.1 - Concurrent Processing - What is FNDREVIVER and How Is It Set?
NOTE:551895.1 - Concurrent Processing - Failover Of Concurrent Manager Processes Takes More than 30 Minutes
NOTE:822368.1 - Concurrent Processing - How To Run the Purge Concurrent Request FNDCPPUR, Which Tables Are Purged, And Known Issues Like Files Are Not Deleted From File System or Slow Performance
NOTE:104452.1 - Concurrent Processing - Troubleshooting Concurrent Manager Issues (Unix specific)
NOTE:1075684.1 - Concurrent Managers are consuming high CPU and memory
NOTE:1360118.1 - Performance: Concurrent Requests Hang in Pending Status For Long Time
NOTE:1060736.1 - Deadlock Error During Concurrent Request Termination
NOTE:866298.1 - Concurrent Processing - ORA-00060: Deadlock Detected - UPDATE FND_CONCURRENT_QUEUES
NOTE:1399454.1 - Tuning Output Post Processor (OPP) to Improve Performance
NOTE:844976.1 - Concurrent Processing - Concurrent Reports Failing With Errors REP-0004,REP-0082 and REP-0104
NOTE:1492893.1 - R12: Performance Issue When Standard Managers Waiting for "enq: TX - row lock contention" Held By ICM


FND Concurrent Manager FAQ's

https://support.oracle.com/oip/faces/secure/km/DocumentDisplay.jspx?id=105133.1

1. When would one be required to bounce (stop and restart) the Concurrent Manager?

When you modify the Printer Driver you have to restart the Manager which runs the request which is attached to that Printer Driver, however,if you do not know which manager then you have to restart the Internal manager because the printer driver can be used by multiple managers and multiple requests. If only a concurrent program definition is modified, running a verify on the Internal Manager will pick up the changes without the need for bouncing the manager.


2. Does the Internal manager schedule requests to be run or does it put requests  into queues to be run by other managers?

This is a very common misconception. The ICM really does not have any such scheduling responsibilities. It has NOTHING to do with scheduling requests, or deciding which manager will run a particular request. Its function is only to run 'queue control' requests, which are requests to startup or shutdown other managers. It is responsible for startup and shutdown of the whole concurrent processing facility, and it also monitors the other managers periodically, and restarts them if  they should go down. It can also take over the Conflict Resolution manager's job, and resolve incompatibilities.

If the ICM itself should go down, requests will continue to run normally, except for 'queue control' requests. You can restart it with 'startmgr', you do not need to kill the other managers first.


3. How can I check to see if a concurrent manager is running?

One way to see if a manager is running is to use the 'Administer
Concurrent Managers' form. Navigate to Concurrent->Managers->Administer.

You will see two columns labeled 'Actual' and 'Target'. The Target column lists the number of processes that should be running for each manager for this particular workshift. The Actual column lists the number of processes that are actually running. If the Actual column is zero, there are no processes running for this manager. If the Target column is zero, then either a workshift has not been assigned to this manager, or the current workshift does not specify any target processes. If the target column is not zero, then the manager processes have either failed to start up, or gone down.

You should check the manager's logfile and the ICM logfile. You can also search for OS processes using the 'ps' command. It is possible for the form to be inaccurate, i.e. it may show actual processes even though they are not really running. When in doubt, check for processes at the OS level. On NT, you can check to see if the Concurrent Manager service is running using the Services control panel.


4. Where do concurrent request or manager logfiles and output files go?

The concurrent manager first looks for the environment variable
$APPLCSF. If this is set, it creates a path using two other
environment variables: $APPLLOG and $APPLOUT

It places log files in $APPLCSF/$APPLLOG, output files go in
$APPLCSF/$APPLOUT

So for example, if you have this environment set:
 $APPLCSF = /u01/appl/common ($FND_TOP/log)
 $APPLLOG = log
 $APPLOUT = out


5. What are the logfile and output file naming conventions?

Request logfiles: l.req

Output files: If $APPCPNAM is not set:  .
       If $APPCPNAM = REQID:     o.out
       If $APPCPNAM = USER:      .out

Where:  = The request id of the concurrent request
And:  = The id of the user that submitted the request

Manager logfiles:

ICM logfile: Default is std.mgr, can be changed with the mgrname
startup parameter
Concurrent manager log:                 w.mgr
Transaction manager log:                t.mgr
Conflict Resolution manager log:        c.mgr

Where: is the concurrent process id of the manager


6. Can I delete a concurrent manager?

You can disable the manager by checking the 'Enabled' checkbox, or you can simply Terminate the manager and it will not run again unless you reactivate it.

If it is really necessary, you can query the manager in the
'Define Manager' form, and delete the row. (It is recommended that you DO NOT do this)


7.  What is the function of the 'Conflict Resolution Manager'?

Concurrent managers read requests to start concurrent programs running. The Conflict Resolution Manager checks concurrent program definitions for incompatibility rules.

If a program is identified as Run Alone, then the Conflict Resolution Manager prevents the concurrent managers from starting other programs in the same conflict domain.

When a program lists other programs as being incompatible with it, the Conflict Resolution Manager prevents the program from starting until any incompatible programs in the same domain have completed running.


8. What is the 'Internal Scheduler/Prereleaser' manager?

The short name for this manager is FNDSCH. It is also known as the Advanced Scheduler/Prereleaser Manager. This manager is intended to implement Advanced Schedules. Its job is to determine when a scheduled request is ready to run. Advanced Schedules were not fully implemented in Release 11.0, they are implemented in Release 11.5, but are not widely used by the various Apps products. General Ledger uses FNDSCH for financial schedules based on different calendars and period types. It is then possible to schedule AutoAllocation sets, Recurring Journals, MassAllocations, Budget Formulas, and MassBudgets to run according to the General Ledger schedules that have been defined. 

If financial schedules in GL are not being used then it is not a
problem to deactivate this manager.


9. What is the 'Internal Monitor' manager/service?

This manager/service is used to implement Distributed Concurrent Processing. It monitors whether the ICM is still running, and if the ICM crashes, it will restart it on another node. You do not need to run this manager/service unless you are using Distributed
Concurrent Processing.


10. How do I check/set the PMON method?

To check the PMON method:
1) cd $FND_TOP/sql
2) sqlplus apps/apps @afimchk.sql

This will tell whether the internal manager is running, what the PMON method is, and where the log file is.

To set the PMON method:
1) first shut the concurrent managers down
2) cd $FND_TOP/sql
3) sqlplus apps/apps @afimpmon.sql LOCK  (or RDBMS)


11. How do I enable/disable the Conflict Resolution Manager?

Use the system profile option 'Concurrent: Use ICM'.
Setting this to 'No' (which is the default) allows the CRM to be started. Setting it to 'Yes' causes the CRM to be shutdown and the Internal Manager (ICM) will take over the conflict resolution duties. If the CRM will not start (it is started automatically by the ICM), check this profile option.

Note that using the ICM to resolve conflicts is not recommended.
The CRM's sole purpose is to resolve conflicts, while the ICM has
other functions to perform as well. Only set this option to 'YES'
if you have a good reason to do so.


12. How do I clean out the Concurrent Manager tables?

Cleaning out the tables is a useful method of making sure that there are no invalid statuses that can prevent the managers from starting. Previously, this has been done by truncating fnd_concurrent_processes and/or fnd_concurrent_requests. Truncation of the tables is a little drastic, and can cause problems later when trying to purge requests, not to mention losing all of the request information.

Run the script, cmclean.sql, article note 134007.1 CMCLEAN.SQL - Non Destructive Script to Clean Concurrent Manager Tables
It will make sure the relevant status codes are valid without
deleting any information.


13. How do I tell concurrent manager processes apart at the OS level?

Use: pf -ef | grep FNDLIBR

This will produce output like:

vd11 13703 13660  0   May 11 ?            0:01 FNDLIBR FND Concurrent_Processor
MANAGE OLOGIN="APPS/94A491A1000000000000000000
n1070161 24936 24927  0   Apr 29 ?        0:05 FNDLIBR FND Concurrent_Processor
MANAGE OLOGIN="APPS_APPDEMO/94C4B1C10000000000
n1070161 24938 24927  0   Apr 29 ?        0:06 FNDLIBR FND Concurrent_Processor
MANAGE OLOGIN="APPS_APPDEMO/94C4B1C10000000000
n1070161 24927 24922  0   Apr 29 ?        2:03 FNDLIBR FND CPMGR FNDCPMBR sysmgr
=""  sleep=60 pmon=20  diag=N logfile=/u16/app

The last process, #24927, shows 'FNDLIBR FND CPMGR', this one is the Internal Manager (ICM). Notice that it gives some of the parameters it was started with, the other processes showing 'Concurrent_Processor' are Standard manager processes. Notice that the ICM process is the parent process of the Standard managers. (processes 24936 and 24938)

Other managers will have the name of the executable, like ARLIBR or
INVLIBR:

$ ps -ef | grep ARLIBR
vd11 13683 13660  0   May 11 ?        0:20 ARLIBR APPS/82A2A4940000000000000
000000000000000000000000000000000000000 AR ART

The Conflict Resolution manager will look like:

$ ps -ef | grep FNDCRM
n1070161 24941 24927  0   Apr 29 ?        1:17 FNDCRM APPS_APPDEMO/84BFBEB900000
0000000000000000000000000000000000000000000000


14. Why am I seeing pinging entries like this in the ICM logfile?

PING  (0.0.0.0): 56 data bytes                                             
64 bytes from 192.75.91.2: icmp_seq=0 ttl=255 time=0.705 ms                
64 bytes from 192.75.91.2: icmp_seq=1 ttl=255 time=1.120 ms                    
      Process monitor session ended : 29-FEB-2000 10:38:43                                            
64 bytes from 192.75.91.2: icmp_seq=2 ttl=255 time=0.985 ms                
64 bytes from 192.75.91.2: icmp_seq=3 ttl=255 time=1.006 ms

Pinging other machines is used in Distributed Concurrent Processing. This means you have DCP turned on, using the environment variable APPLDCP. Set APPLDCP to OFF and restart the managers.


15. I hit the Restart button to start the Standard manager, but it still did not start?

Telling a manager to restart just sets the status to Restart. The ICM will start it the next process monitor session or the next time the ICM starts. Use Activate to start a manager immediately.
When a manager is deactivated manually, the ICM will not restart
it, you will need to set it to Restart, or activate it manually.


16. How many rows are in FND_CONCURRENT_REQUESTS and FND_CONCURRENT_PROCESSES tables?

Depending on the specification of the system it has been seen that when tables reach above 3000-4000 rows, the performance begins to diminish, however, there could be 30000-40000 rows in the tale before the performance begins to degrade.

You may want to run the Purge Concurrent Request and/or Manager Data on a regular basis, dependant on the amount of requests being run.

The Purge Concurrent Requests job can be used to purge:
Requests, Mgr logs, and All requests depending on what is chosen.

Use the following options: Enter = All, Mode = AGE, Mode Value = 15

The std.mgr log continuously grows where it may good to
archive it regularly.


17. Any processes pending in Internal or Conflict Resolution Manager?

Best course of action before starting the Concurrent Managers is to cancel any "Deactivate" or "Verify" jobs pending in the Internal Manager and place any other pending jobs on hold.


18. How do I turn on transaction manager diagnostics?

Set the profile option 'Concurrent:Debug Flags' to 'TCTM1' at the site level. This will cause transactions to make debug entries in the FND_CONCURRENT_DEBUG_INFO table. Truncate this table before running a tranasction, then select the entries from the table.
Starting the managers with diag=Y will also produce more information in the transaction manager logfile.


19. How do I process more concurrent requests concurrently?

The Concurrent Manager parameters, (Query the concurrent manager by Login as Sysadmin, navigate -> Concurrent -> Manager -> Define and Query for the relevant concurrent manager), should be modified to handle more concurrent requests concurrently, this can be done in two steps:

(i) Increase the Number of Target processes for the manager

(ii) Change the cache size of the concurrent manager  as this determines how many requests will be evaluated by a manager at a time and should match the target (process) value as set above.


20. When you try to submit a request like Active users or
Active responsibilities, request gets submitted.

When we view the help requests, you find that it is
inactive / nomanager.

Within 12 to 15 seconds, you refresh-it gets completed.

Initially, you could find only inactive and we look at
the diagnostic- the concurrent manager assigned is not
picking up.

There is no specialization rules in any managers except
the include program this sourceSolution....

Most often when this occurs where a request goes
"inactive/no manager" and is then processed a short time
later, the solution is to either increase the cache size
for your Standard manger, or increase the actual number of
Standard manager processes.

Cache Size is set on the CONCURRENT/MANAGER/DEFINE form.  Basically,this regulates how many requests a manager will pick up for each sleep cycle.


Thanks 
Srini