ODI Clustering for Load Balancing, High Availability, and Failover

by Keith Mascarenhas

Oracle Data Integrator Clustering for High Availability and Failover

This article profiles the use of Oracle Data Integrator (ODI) from a high availability and fail-over perspective. Before doing this, let’s see how ODI works from a single node installation point of view. 

Single Node ODI Installation

ODI provides a unified run-time architecture using Java components.  The run-time architecture consists of the following components:

1. Repository

The Repository consists of the design-time objects (interfaces, packages) and run-time objects (load plans, scenarios). 

2. Run-time Agents

Run-time agents are the components that execute a session (load plan/scenarios).  Run-time agents can be standalone agent or Java EE agents.

3. ODI Console

The ODI Console is a web application user interface (UI) that allows users to browse, manage and monitor the components of the repository.

The figure shown below (referenced from oracle documentation), describes a single node installation of an ODI application. 

ODI agents process a load plan execution instance as a session and a separate java thread is created for the execution of each load plan/scenario.

When a scenario is executed on an agent:

• The agent creates a session in the repository corresponding to the scenario instance.
• Then creates a connection to the master repository to obtain the credentials to the work repository.
• Gets the work repository information that is required for the execution of the thread and executes accordingly.
• Then writes the result (return code, duration, rows processed, etc.) back to the repository.

The scenario mentioned above is a simple description of Oracle Data Integrator in a single node installation. Now, let’s move over to using the Oracle Data Integrator in a high availability and a failover environment.

Two-Node ODI Cluster for Load Balancing, High Availability, Failover

The figure below (referenced from Oracle documentation), shows a two-node ODI cluster installation for load balancing, high availability and failover.

The main features of this setup are:

• ODI runs on a two clustered Weblogic managed servers.
• Duplicate schedule processing is avoided by configuring only one agent to behave like a scheduler.
• Coherence cache is used to handle scheduler service uniqueness.
• The requests to the ODI agent in a cluster go through the load balancer/HTTP server.
• ODI’s master and work repository database is configured with Oracle Real Application Clusters, most commonly known as Oracle RAC, to avoid database failures.

There are three possible instances that we can identify in case of failures.  This is where a high-availability configuration kicks in:

1. Weblogic Server Crash

• When the Weblogic server (say on Application Node A) crashes, the node manager attempts to restart it locally.
• In case the restart fails, the WLS infrastructure attempts to perform a server migration to other node (say Application Node B) in the cluster
• Now, the other node (Application Node B) becomes the scheduler and is able to read, compute and execute the schedule for all work repositories
• As mentioned earlier, a coherence cache is used to handle the scheduler lifecycle and maintains the uniqueness of the schedules as well.

2. Repository Database Failure

• ODI repositories are protected against database failures by using multi data sources that are configured while running the configuration wizard that allows to define multiple connection pools during the initial set up of the system.
• This guarantees that when an instance that hosts the repositories on an Oracle RAC fails, the connections are re-established with other available database instances.

3. Scheduler Node Failure

• When the scheduler node fails, another node in the Weblogic server cluster takes over as the scheduler node.
• This node, then reinitializes all the schedules and executes the scheduled load plans/scenarios from that point onwards.

High-level steps of configuring high availability for Oracle Data Integrator:

1. Create the ODI master and work repositories in an Oracle RAC database using the Oracle Fusion Middleware Repository Creation Utility (RCU)

2. Install and Configure the first ODI Node (Application Node A)

• Install Oracle Weblogic server on Application Node A.
• Install Oracle Data Integrator on Application Node A.
• Create the high availability Weblogic domain by using the configuration wizard (config.sh/config.bat based on the operating system used).
  • In this step, you would define the managed servers and cluster during the Configure Managed Servers and Configure Clusters page options and then assign these servers to the defined cluster.
• Start the administration server and configure the credential store either using WLST or the enterprise manager that was configured with the domain.
• Configure the default agent.
  • This would be done using the ODI Studio by connecting to the Master repository.
• Configuring coherence for the cluster to enable communication among the cluster members.
• Configure the node manager using the setNMProps.sh and start the managed server on Node A and verify that the agent created earlier is running using the following URL in a web browser: http://<host_name_node_a> :<port_configured>/<web_application_context>

3. Install and Configure the second ODI Node (Application Node B)

• Install Oracle Weblogic Server on Application Node B using the same directory paths as used on Node A.
• Pack and Unpack the domain that was configured on Application Node A to Application Node B.  This can be done using the pack.sh and unpack.sh scripts that sit in the MW_HOME/oracle_common/common/bin folder.
• Configure the node manager using the setNMProps.sh and start the managed server on Application Node B and verify that the agent created earlier is running using the following URL in a web browser: http://<host_name_node_b> :<port_configured>/<web_application_context>

4. Install and configure the Oracle HTTP server on two other hosts (Web Node A and Web Node B) 

5. Configure the load balancer  for round robin requests across the two HTTP servers and verify the ODI agent is running by using the following URL in a web browser: http://<load_balancer_host>:<port_configured>/<web_application_context>

6. Once the load balancer is configured, you would need to configure the default agent by connecting to the ODI Studio and pointing the agent properties to the load balancer URL properties. 

Keith Mascarenhas is a data warehousing and business intelligence enthusiast. As a senior-level consultant for KPI Partners, Keith has participated in some of the most complex deployments of OBIEE, Oracle BI Applications, and Informatica in the world. His expertise also includes Oracle Data Integrator (ODI), Cognos, and Oracle Warehouse Builder (OWB). Check out Keith’s blog at KPIPartners.com.