Oracle 19C RAC Startup sequence step by step process

 Dear All,

In this post i have explained the step-by-step oracle 19C RAC startup sequence.

The startup sequence in Oracle 19c RAC is complex due to the multi-instance, multi-node nature of RAC.

 

1.  Start Oracle Grid Infrastructure (Clusterware) – crsctl

2.  Start Oracle ASM (Automatic Storage Management)  - srvctl / as part of cluster start

3.  Start Oracle RAC Database Instances using srvctl

4.  Start Oracle Net Listener

5.  Start Oracle Services (optional)

6.  Start ASM (if not already done)

This process ensures that all instances across nodes are properly synchronized, ensuring high availability and data consistency in a shared-disk environment like Oracle RAC. Proper startup ensures that all Oracle processes, database services, and storage are correctly initialized, providing a stable environment for your applications and database users.

 

Oracle 19c RAC Startup Sequence: Detailed Explanation

In Oracle 19c Real Application Clusters (RAC), the startup sequence refers to the steps Oracle follows to bring the database and its associated services into an operational state. The startup process is different in a RAC environment compared to a single-instance Oracle database because it involves starting and coordinating multiple instances across several nodes, ensuring they can work together in a shared-disk environment.

The startup sequence in Oracle RAC is divided into the following stages:

---

1. Starting Oracle Grid Infrastructure (CRS) and Cluster ware

Before any Oracle RAC database instance starts, Oracle Cluster ware (Grid Infrastructure) must be started on all nodes in the cluster.

1: Start the Cluster ware Stack (CRS) 

> Oracle Cluster ware manages the nodes in the cluster and ensures they function as a single logical unit. This includes monitoring the health of all instances, providing failover mechanisms, and managing resources like voting disks and OCR (Oracle Cluster Registry).

> Cluster ware services (CRS) must be running on each node to manage the cluster's health.

Commands:

$ crsctl start crs        # Start Oracle Clusterware (CRS) on all nodes

When you issue this command, the following services are started:

Clusterware Daemons: These include crsd, evmd, and cssd.

crsd (Cluster Ready Services Daemon): Manages Oracle Clusterware resources.

evmd (Event Manager Daemon): Collects and processes events in the cluster

cssd (Cluster Synchronization Services Daemon): Manages cluster membership and coordinates node status and interconnect activity.

Start Oracle Clusterware (Oracle ASM) 

>     If you're using Oracle ASM (Automatic Storage Management) for your RAC database, it must also be started as part of the Clusterware stack.

> ASM is responsible for managing the storage for Oracle databases in a RAC environment. ASM organizes data into disk groups and provides striping and mirroring for high availability.

Commands:

$ crsctl start asm

This command starts the ASM instance, and it will mount the ASM disk groups that contain the Oracle database files, including voting disks and OCR (Oracle Cluster Registry).

---

 Starting Oracle RAC Database Instances

Once Clusterware is up and running, the next step is to start the Oracle RAC database instances. This step starts each instance of the RAC database on each node in the cluster.

>> Start the Database Instances Using srvctl

To start a database in Oracle RAC, you typically use the srvctl command. This command ensures that all instances of the RAC database are properly started, ensuring correct inter-node communication and Cache Fusion functionality.

> srvctl automatically starts all required Oracle background processes, including LGWR (Log Writer), DBWn (Database Writer), and PMON (Process Monitor), and ensures proper interconnect activity between nodes.

Commands:

$ srvctl start database -d <dbname>

ex: <dbname> is the name of the RAC database that you want to start.

$srvctl start database -d racsdb

2. Oracle Instance Startup Sequence

Once the srvctl command is issued, Oracle begins the following instance startup sequence for each node in the cluster:

Process Initialization:

        > The instance processes like SMON, PMON, and others are started. These processes ensure the instance runs in a healthy state.

Instance Registration with the Listener:

> Each instance registers itself with the Oracle Net Listener for client connections. It ensures that the listener knows which instance is available to service requests for that particular instance.

Shared Memory Allocation: 

       >SGA (System Global Area) and PGA (Program Global Area) are allocated for each instance ,  SGA is a shared memory region that contains data and control information for the database. 

     > PGA is a private memory region used by Oracle background processes and user sessions.

Global Enqueue Initialization:

Oracle RAC uses a shared global cache to manage locks and latches on data blocks across all nodes. The instances communicate and synchronize via Global Enqueue Services (GES) and Global Cache Services (GCS). 

> The Global Cache layer ensures consistency of data across RAC nodes.

Database Opening:  After the instance processes and memory structures are initialized, the Database itself is opened. Oracle opens the database in READ ONLY mode to ensure that it can be accessed by all instances of the RAC database, and parallel execution is available across nodes. 

  >The redo log files are opened to allow transaction processing.

Cache Fusion Initialization: 

> Cache Fusion ensures that data blocks are shared across instances. Oracle will initiate Global Cache Synchronization to synchronize the shared memory structures and data blocks between instances.

> This is key in Oracle RAC to ensure all nodes are consistent with the data they share.

Database Processes Start: Database processes, such as LGWR (Log Writer), DBWn (Database Writer), SMON (System Monitor), and PMON (Process Monitor), are started. These processes are responsible for maintaining the health of the database and managing transactional and background operations.

 Verify Database Instance Startup

You can verify the status of your RAC database instances and ensure that they are up and running with the following command:

$ srvctl status database -d <dbname>

The output will indicate which instances are running and their status across all RAC nodes.

---

3. Starting Oracle Net Listener

The Oracle Net Listener is a process that listens for incoming client connection requests and directs those requests to the appropriate database instance in the cluster.

Start Oracle Listener

Use the srvctl utility to start the listener for Oracle RAC:

Commands:

$ srvctl start listener -l <listener name>

This command starts the listener for your Oracle RAC database, allowing client connections to be established.

 

4. Start Oracle Services -- Optional

Oracle services allow you to manage the load balancing and high availability of the database in a RAC environment. For example, Oracle can use Service Names to direct clients to the appropriate instance based on load or availability.

 Start Oracle Services Using srvctl

$ srvctl start service -d <dbname> -s <service_name>

This command starts the specified service on the available instances. In a RAC setup, Oracle can configure services to run on specific nodes or across all nodes in the cluster.

Verify Service Status

To check the status of the services, you can use the following command:

$ srvctl status service -d <dbname> -s <service_name>

 

5. Starting Oracle Automatic Storage Management (ASM)

ASM (Automatic Storage Management) is responsible for managing database storage in RAC environments. It needs to be started to manage disk groups and handle file operations.

Step 1: Start ASM Using srvctl

You can start ASM using the srvctl command as follows:

$ srvctl start/status/stop asm

 

6. Database and Instance Synchronization in RAC

Once all instances are up, and the Oracle Grid Infrastructure is running, Oracle RAC ensures that all instances are synchronized via Cache Fusion. This synchronization allows multiple instances to access the same set of database files (e.g., data files, control files, and redo log files) in a shared storage configuration.

Monitor Cluster Synchronization

You can monitor Cache Fusion performance using the v$gc_stats view:

crsctl check ctss

crsctl check css

crsctl check cluster

 

ways to check cache fusion in Oracle RAC:

GCS statistics: GCS statistics for current and cr blocks, such as gc current blocks received and gc cr blocks received, can be used to analyze the effect of cache fusion. 

GCS wait events: GCS wait events, such as gc current block 3-way and gc cr grant 2-way, can be used to analyze the effect of cache fusion. 

V$SYSSTAT view: The V$SYSSTAT view can be queried to monitor global cache statistics. 

Cache Fusion is a feature in Oracle RAC that allows multiple instances to share and read the same data block concurrently. It uses the Global Cache Service (GCS) to check if another instance has a copy of a data block before accessing it. If another instance has the block, Cache Fusion transfers it directly between the instances without writing it to disk

Thanks,
Srini