Professional Documents
Culture Documents
August 2013
Executive Overview
Oracle Real Application Clusters (Oracle RAC) and Oracle Clusterware allow Oracle Database
to run any packaged or custom application across a set of clustered servers. This capability
provides the best availability for node and instance failures and most planned maintenance
activities, and the most flexible scalability. If a clustered node fails, the Oracle database
continues running on the surviving nodes. When more processing power is needed, another
node can be added without interrupting user access to data.
Oracle Clusterware is a cluster manager that is designed specifically for the Oracle database.
In an Oracle RAC environment, Oracle Clusterware monitors all Oracle resources (such as
database instances and listeners). If a failure occurs, then Oracle Clusterware automatically
attempts to restart the failed resource. During outages, Oracle Clusterware relocates the
processing performed by the inoperative resource to a backup resource. For example, if a
node fails, then Oracle Clusterware relocates the database services being used by the
application to a surviving node in the cluster.
This white paper describes best practices for configuring Oracle GoldenGate to work with
Oracle RAC, Oracle Clusterware and Oracle Database File System (DBFS). Oracle
GoldenGate is instrumental for many reasons, including the following:
As part of an application architecture that requires Oracle RAC plus the flexible availability
features provided by Oracle GoldenGate, such as active-active database for data distribution
and continuous availability, and zero or minimal downtime during planned outages for
system migrations, upgrades, and maintenance
To capture from an OLTP application running on Oracle RAC to support further downstream
consumption such as a SOA type integration
This paper focuses on configuring Oracle GoldenGate to run on Oracle RAC, which can act as
the source database, as the target database, or in some cases as both source and target
databases for Oracle GoldenGate processing.
Configuration Overview
This section introduces Oracle GoldenGate, Oracle RAC, Oracle Clusterware, and Oracle Database
File System (DBFS). For more information about these features, see the References section at the end
of this white paper.
Oracle GoldenGate
Oracle GoldenGate provides real-time, log-based change data capture and delivery between
heterogeneous systems. This technology enables you to construct a cost-effective and low-impact realtime data integration and continuous availability solution.
Oracle GoldenGate moves committed transactions with transaction integrity and minimal overhead on
your existing infrastructure. The architecture supports multiple data replication topologies such as oneto-many, many-to-many, cascading, and bidirectional. Its wide variety of use cases includes real-time
business intelligence; query offloading; zero-downtime upgrades and migrations; and active-active
databases for data distribution, data synchronization, and high availability. Figure 1 shows the Oracle
GoldenGate architecture.
Oracle Real Application Clusters (Oracle RAC) enables multiple instances that are linked
by an interconnect to share access to an Oracle database. In an Oracle RAC environment,
Oracle Database runs on two or more systems in a cluster while concurrently accessing a
single shared database. The result is a single database system that spans multiple hardware
systems, enabling Oracle RAC to provide high availability and redundancy during failures
in the cluster. Oracle RAC accommodates all system types, from read-only data
warehouse systems to update-intensive online transaction processing (OLTP) systems.
Oracle Clusterware
Oracle Clusterware enables servers to communicate with each other, so that they appear to function as
a collective unit. This combination of servers is commonly known as a cluster. Although the servers are
standalone servers, each server has additional processes that communicate with other servers. In this
way the separate servers appear as if they are one system to applications and end users.
Oracle Clusterware provides the infrastructure necessary to run Oracle RAC. Oracle Clusterware also
manages resources, such as virtual IP (VIP) addresses, databases, listeners, services, and so on.
There are APIs to register an application and instruct Oracle Clusterware regarding the way an
application is managed in a clustered environment. You use the APIs to register the Oracle
GoldenGate Manager process as an application managed through Oracle Clusterware. The Manager
process should then be configured to automatically start or restart other Oracle GoldenGate processes.
Substitute the size parameters with your required trail file storage size.
The LOB segment used by DBFS should be configured with the storage options NOCACHE
LOGGING which is the default:
-- Connect to the DBFS database
SQL> connect system/<passwd>@<dbfs_tns_alias>
-- View current LOB storage:
SQL> SELECT table_name, segment_name, logging, cache
FROM dba_lobs WHERE tablespace_name='DBFS_GG_SOURCE_TBS';
-- More than likely it will be something like this:
--- TABLE_NAME
SEGMENT_NAME
LOGGING CACHE
LOB_SFS$_FST_73
YES
NO
SEGMENT_NAME
LOGGING CACHE
LOB_SFS$_FST_73
YES
NO
Starting with Oracle GoldenGate release 11.2.1 the dirtmp directory can be placed on DBFS. With
earlier releases of Oracle GoldenGate, the dirtmp directory was normally on the local file system or
another non-DBFS shared file system since data stored in dirtmp is transient and not required for any
Oracle GoldenGate startup. However, by placing dirtmp on DBFS, you get the additional benefit of
larger storage potential.
If dirtmp is on DBFS, use a NOCACHE NOLOGGING tablespace. For example:
Substitute the size parameters with your required trail file storage size.
Create the file system:
% cd $ORACLE_HOME/rdbms/admin
% sqlplus dbfs_user/dbfs_password
SQL> start dbfs_create_filesystem dbfs_gg_dirtmp_tbs gg_dirtmp
The LOB segment used by DBFS should be configured with the storage options NOCACHE
NOLOGGING which is the default when the tablespace is created with the NOLOGGING option:
-- Connect to the DBFS database
SQL> connect system/<passwd>@<dbfs_tns_alias>
-- View current LOB storage:
SQL> SELECT table_name, segment_name, logging, cache
FROM dba_lobs WHERE tablespace_name='DBFS_GG_DIRTMP_TBS';
------
Follow the instructions in My Oracle Support note 1054431.1 for configuring the newly created DBFS
file system so that the DBFS instance and mount point resources are automatically started by Cluster
Ready Services (CRS) after a node failure. When registering the resource with Oracle Clusterware, be
sure to create it as a cluster_resource instead of a local_resource as specified in the My
Oracle Support note. The reason for using cluster_resource is so the resource can only be
running on a single node preventing the accidental mounting of DBFS from concurrent nodes.
Example command to register the resource:
crsctl add resource $RESNAME \
-type cluster_resource \
-attr "ACTION_SCRIPT=$ACTION_SCRIPT, \
CHECK_INTERVAL=30,RESTART_ATTEMPTS=10, \
START_DEPENDENCIES='hard(ora.$DBNAMEL.db)pullup(ora.$DBNAMEL.db)',\
STOP_DEPENDENCIES='hard(ora.$DBNAMEL.db)',\
SCRIPT_TIMEOUT=300"
Once the file system is mounted, create directories in the newly created file system for storing the
Oracle GoldenGate files.
Example:
% cd /mnt/gg_source/goldengate
% mkdir dirchk
% mkdir dirpcs
% mkdir dirprm
% mkdir dirdat
% mkdir BR
Create symbolic links for the directories that are not controlled by Oracle GoldenGate parameters:
% ln s /mnt/gg_source/goldengate/dirprm $GG_HOME/dirprm
% ln s /mnt/gg_source/goldengate/dirchk $GG_HOME/dirchk
% ln s /mnt/gg_source/goldengate/dirpcs $GG_HOME/dirpcs
% ln s /mnt/gg_dirtmp $GG_HOME/dirtmp
The Bounded Recovery (BR) feature was added to Extract in Oracle GoldenGate version 11.1.1. This
feature guarantees an efficient recovery after Extract stops for any reason, planned or unplanned, no
matter how many open (uncommitted) transactions there were at the time that Extract stopped and no
matter how old they were. Bounded Recovery sets an upper boundary for the maximum amount of
time that it would take for Extract to recover to the point where it stopped and then resume normal
processing. The Bounded Recovery checkpoint files should be placed on a shared file system such that
in an event of a failover when there are open long running transactions, Extract can use Bounded
Recovery to reduce the time to taken to perform recovery. Starting in Oracle GoldenGate version
11.2.1 the Bounded Recovery files are supported and it is recommended that you place them on DBFS.
With earlier releases the Bounded Recovery files need to be stored on NFS storage such as Oracle ZFS
Storage Appliance. It is possible to store the checkpoint files on the local file system, but when Extract
performs recovery after a node failure, the standard checkpoint mechanism will be used until new local
Bounded Recovery checkpoint files are subsequently created. This will only be noticeable if there are
long running transactions at the time of the failure.
To set the Bounded Recovery file directory use the following Extract parameter:
BR BRDIR /mnt/gg_source/goldengate/BR
For more information on Bounded Recovery refer to the Oracle GoldenGate Windows and UNIX Reference
Guide:
http://docs.oracle.com/cd/E35209_01/doc.1121/e29399.pdf
The location of the Extract/Data Pump trail file directory is specified during process creation. For
Extract it is also specified in the parameter file with the EXTTRAIL.
Target Environment (Replicat)
On the target environment, where the Replicat processes read the trail files and apply the data to the
target database there is a requirement for two separate DBFS file systems to separate the different I/O
requirements of the trail and checkpoint files.
Trail files are written to by the Collection Server process on the target host using consecutive serial
I/Os from the start to the end of the file, sized according to your Data Pump configuration. The same
trail files are read by each Replicat process, also using consecutive serial I/O requests. Once a portion
of the trail is read by a Replicat process it will not normally be read a second time by the same process.
When using multiple Replicat processes reading from the same trail files, it is rare that they remain in
sync, reading from the same portion of the trail file at the same time. Because of this, the best
configuration for DBFS is with NOCACHE LOGGING storage options. This is described above in
configuring the source environment.
The checkpoint files are small (approximately 4KB) but written to frequently, overwriting previous
data. The file doesnt grow in size and is only read during process startup to determine the proper
starting point for recovery or initiation. Because the checkpoint file is written to over and over,
performance is best when the file is stored in DBFS with the CACHE LOGGING storage option.
Setting the CACHE option causes the small amount of data being written to the checkpoint files to be
written into the buffer cache of the DBFS instance, and not issuing direct writes to disk causing higher
waits on I/O. In testing this has shown to increase checkpoint performance by a factor of 2 to 5 times
compared to using the NOCACHE configuration with DBFS.
Create the second DBFS file system for the checkpoint files in much the same way as the file system
on the source environment (above). Some important notes:
The file system is only for checkpoint files, so it can be sized less than 100 MB.
Create the file system using the same user as the first file system created. It is important to make sure
the same user creates both file systems.
-- TABLE_NAME
SEGMENT_NAME
LOGGING CACHE
-- ------------------ ---------------------- ------- ----------- T_GOLDENGATE2
LOB_SFS$_FST_75
YES
NO
SQL> ALTER TABLE DBFS.<TABLE_NAME> MODIFY LOB (FILEDATA)
(NOCACHE LOGGING);
-- View the new LOB storage:
SQL> SELECT table_name, segment_name,logging
FROM dba_lobs WHERE tablespace_name='DBFS_GG_CKPT_TBS';
TABLE_NAME
SEGMENT_NAME
LOGGING CACHE
------------------ ---------------------- ------- ---------T_GOLDENGATE2
LOB_SFS$_FST_75
YES
YES
Note: If you are using an Oracle GoldenGate Data Pump process to transfer the trail files from a
source host on the database machine using DBFS, then contact Oracle GoldenGate Support to obtain
the fix to Bug 10146318. This bug fix improves trail file creation performance on DBFS by the Oracle
GoldenGate server/collector process. This only affects Oracle GoldenGate versions earlier than
11.1.1.0.5.
Download the Oracle GoldenGate software from Oracle Technology Network (OTN) at:
http://www.oracle.com/technetwork/middleware/goldengate/downloads/index.html
2.
Install Oracle GoldenGate locally on the primary source and target nodes in the Oracle RAC
configuration. Make sure the installation directory is the same on all nodes.
3.
Once you have successfully configured Oracle GoldenGate on the primary source and/or
target nodes, shut down Extract/Replicat and copy the entire Oracle GoldenGate home
directory to the other source and target nodes.
4.
Follow the generic installation instructions for the source and target machine installations
available in Chapter 2, Installing GoldenGate at:
http://docs.oracle.com/cd/E35209_01/doc.1121/e35957.pdf
10
1411356.1. It can also be used to capture changes from Oracle versions starting with 10.2.0.4 in a
downstream mining deployment.
Extract can still be configured to capture directly from the redo logs for any supported Oracle version.
This configuration is now called classic capture mode.
1.
2.
Use the default Oracle Automatic Storage Manager (Oracle ASM) naming convention for
the archived redo log files.
b. Configure the Oracle GoldenGate Extract parameter for the newer Oracle ASM log read
API.
Oracle GoldenGate release 11.1.1 introduces a new method of reading log files stored in
Oracle ASM. This new method uses the database server to access the redo and archived
redo log files, instead of connecting directly to the Oracle ASM instance. The database
must contain the libraries with the API modules. The libraries are currently included with
Oracle Database release 10.2.0.5, 11.2.0.2 and 11.2.0.3.
To successfully mine the Oracle archived redo log files located on the storage cells that
are managed by Oracle ASM, configure the Oracle GoldenGate Extract parameter as
follows:
Set the TRANLOGOPTIONS parameter to specify use of the new log read API. For
example:
TRANLOGOPTIONS DBLOGREADER
3.
11
2.
To configure Oracle GoldenGate trail files on DBFS for the source database:
1.
2.
3.
After creating the Extract, use the same EXTTRAIL parameter value to add the local trail:
% ggsci
GGSCI (ggtest.oracle.com) 1> ADD EXTTRAIL
/mnt/dbfs/goldengate/dirdat/aa, EXTRACT ext_db, Megabytes 500
12
Further instructions about creating the Extract are available in the Oracle GoldenGate Windows
and UNIX Administration Guide at
http://docs.oracle.com/cd/E35209_01/doc.1121/e29397.pdf
To configure Oracle GoldenGate trail files on DBFS for the target database:
1.
Make sure the DBFS directory is already created on the target environment.
2.
3.
When adding the Replicat, use the same EXTTRAIL parameter value:
% ggsci
GGSCI (ggtest.oracle.com) 1> ADD REPLICAT rep_db1, EXTTRAIL
/mnt/dbfs/goldengate/dirdat/aa
Do not place trail files on the local file system because it will lengthen restart times in the
event of a node failure, reducing availability.
To configure Data Pump between a source and target database:
1.
2.
Set the RMTHOST Data Pump parameter to the IP or hostname that will be used for
connecting to the target. In Step 7: Oracle Clusterware Configuration, the Application Virtual
IP address is created with Cluster Ready Services (CRS) so that a single IP address can be
moved between compute nodes, so that Data Pump can continue to connect to the target
host when it moves from a failed node to a surviving node:
RMTHOST gg_dbmachine, MGRPORT 8901
3.
Set the RMTTRAIL Data Pump parameter to the trail file location on the target host:
RMTTRAIL /mnt/dbfs/goldengate/dirdat/aa
13
4.
Create a Data Pump process using the local trail file location on the source host:
% ggsci
GGSCI (ggtest.oracle.com) 1> ADD EXTRACT dpump_1, EXTTRAILSOURCE
/mnt/dbfs/goldengate/dirdat/aa
5.
Use the ADD RMTTRAIL command to specify the remote trail file location on the target
host:
% ggsci
GGSCI (ggtest.oracle.com) 1> ADD RMTTRAIL
/mnt/dbfs/goldengate/dirdat/aa EXTRACT dpump_1, MEGABYTES 500
Further instructions about creating the Data Pump process are available in the Oracle
GoldenGate Administration Guide at
http://docs.oracle.com/cd/E35209_01/doc.1121/e29397.pdf
14
If required, instead of using the Oracle GoldenGate process name wildcard (*), explicitly name the
processes you want to be restarted automatically.
Example:
AUTOSTART EXTRACT EXT_1A
AUTOSTART EXTRACT DPUMP_1A
AUTORESTART EXTRACT EXT_1A
AUTORESTART EXTRACT DPUMP_1A
To create the application VIP, run the following as the root user:
$GRID_HOME/bin/appvipcfg create -network=1 \
-ip=10.1.41.93 \
-vipname=gg_vip_source \
-user=root
In the example:
$GRID_HOME is the Oracle home in which Oracle 11g Release 2 Grid
infrastructure components have been installed (for example:
/u01/app/grid).
network is the network number that you want to use. With Oracle Clusterware
release 11.2.0.1, you can find the network number using the following command:
crsctl stat res -p |grep -ie .network -ie subnet |grep -ie
name -ie subnet
15
c.
d. To validate whether the VIP is running and on which node it is running, execute:
$GRID_HOME/bin/crsctl status resource gg_vip_source
See the Oracle Clusterware documentation for further details about creating an Application
VIP: http://docs.oracle.com/cd/E11882_01/rac.112/e16794/crschp.htm#CHDGHGJA
2.
Oracle GoldenGate
11.2.1.+
Oracle Database
10.2.0.5, 11.2.0.2+
The bundled agent software should be downloaded from the following location:
16
http://www.oracle.com/technetwork/products/clusterware/downloads/index.html
Follow the installation instructions provided in the readme.txt file.
Example configuration of the bundled agent:
a.
Configure DBFS as detailed in My Oracle Support note 1054431.1 and test to make
sure CRSCTL can be used to mount and unmount the file system. Mounting the file
system should also start the DBFS instance if it is not already running.
NOTE: It is important that the DBFS resource is registered with Oracle
Clusterware for mounting and unmounting DBFS. If the Oracle Clusterware
resource is not registered and tested, as described in MOS note 1054431.1, it
will not be possible to use the Bundled Agent to automatically mount and
unmount DBFS.
NOTE: The dbfs_mount value for the --filesystems parameter is the name of
the resource registered with Oracle Clusterware. This should have been
carried out as described in My Oracle Support note 1054431.1.
Once the Oracle GoldenGate processes have been added to a bundled they should
only be started and stopped using AGCTL. The bundled agent starts the Oracle
GoldenGate processes by starting Manager which in turn will automatically start the
processes (Extract, Data Pump, Replicat) that have been configured to autostart. If
an Oracle GoldenGate process aborts due to a problem, as long as the manager
process is still running it is okay to use GGSCI to restart the failed process.
To check the status of Oracle GoldenGate:
17
GGCSI can be used to check the status of individual Oracle GoldenGate processes.
To start Oracle GoldenGate manager, and all processes that have autostart enabled:
% agctl start goldengate GG_Target [--serverpool serverpool_name
| --node node_name]
Note: Oracle GoldenGate will start up on the node you issue the command from,
unless a node name or serverpool name is specified.
To stop all Oracle GoldenGate processes:
% agctl stop goldengate GG_Target [--serverpool serverpool_name
| --node node_name]
Note: The Oracle GoldenGate resource MUST be running before relocating it.
To view the configuration parameters for the Oracle GoldenGate resource:
% agctl config goldengate GG_Target
18
19
Must be able to accept five parameter values: start, stop, check, clean and
abort.
Must be stored in the same location on all nodes. In this example, it is stored in the
Grid Infrastructure ($GRID_HOME) ORACLE_HOME/crs/script directory.
Must be owned by the Oracle Grid Infrastructure user and have execute permissions.
See Appendix B: Example Agent Script for an example agent script starts and stops the
Oracle GoldenGate Manager, Extract, Data Pump, and Replicat processes.
It is important to manually test the agent script to start and stop the Oracle GoldenGate
processes before moving onto the next step.
2.
20
1.
Determine the name of the DBFS or source/target database resource for the start
dependency:
crsctl status resource | grep -i dbfs|source/target DB name
2.
Use the Oracle Grid Infrastructure user (oracle, in this example) to execute the
following:
$GRID_HOME/bin/crsctl add resource GG_Source \
-type cluster_resource \
-attr
"ACTION_SCRIPT=/u01/app/11.2.0/grid/crs/script/11gr2_gg_action.scr,
CHECK_INTERVAL=30,
START_DEPENDENCIES='hard(gg_vip_source,dbfs_mount,ora.ggs.db)
pullup(gg_vip_source)', STOP_DEPENDENCIES=hard(gg_vip_source),
SCRIPT_TIMEOUT=300"
If the source and target databases are within the same cluster, make sure the Extract and
Replicat is restricted to the designated cluster nodes:
$GRID_HOME/bin/crsctl add resource GG_Source \
-type cluster_resource \
-attr
"ACTION_SCRIPT=/u01/app/11.2.0/grid/crs/script/11gr2_gg_action.scr,
CHECK_INTERVAL=30, START_DEPENDENCIES='hard(gg_vip_source,dbfs_mount)
pullup(gg_vip_source)', STOP_DEPENDENCIES=hard(gg_vip_source),
HOSTING_MEMBERS=dbm01db05 dbm01db06, PLACEMENT=restricted,
SCRIPT_TIMEOUT=300"
For more information about the CRSCTL add resource command and its options, see
the Oracle Clusterware Administration and Deployment Guide at
http://docs.oracle.com/cd/E11882_01/rac.112/e16794/toc.htm
3.
21
For example:
% crsctl status resource GG_Source
NAME=GG_Source
TYPE=cluster_resource
TARGET=ONLINE
STATE=ONLINE on dbm01db05
4.
5.
Stop Oracle GoldenGate (login as the Oracle Grid Infrastructure (oracle) user):
22
Ensure that the DBFS database has instances running on all the database nodes involved in the
Oracle RAC configuration.
This action provides access to Oracle GoldenGate if it is restarted after a node failure.
Ensure that the DBFS file system is mountable on all database nodes in the Oracle RAC
configuration. Mounting DBFS using a Clusterware resource to prevent the Extract or Replicat
processes from being started on multiple nodes concurrently, mount the file system only on the
node where Oracle GoldenGate is running. Use the same mount point names on all the nodes to
ensure seamless failover.
23
24
### Changed default from local3 to user for Solaris default support on 17FEB-2012
### This will allow us to log messages to the syslog
###
LOGGER_FACILITY=user
###########################################
### No editing is required below this point
###########################################
### determine platform
UNAME_S=`uname -s`
if
logit () {
### type: info, error, debug
type=$1
msg=$2
if [ "$type" = "info" ]; then
echo $msg
$LOGGER -p ${LOGGER_FACILITY}.info "$msg"
elif [ "$type" = "error" ]; then
echo $msg
$LOGGER -p ${LOGGER_FACILITY}.error "$msg"
elif [ "$type" = "debug" ]; then
echo $msg
$LOGGER -p ${LOGGER_FACILITY}.debug "$msg"
fi
}
25
26
EOF`
}
stop_everything () {
# Before starting, make sure everything is shutdown and process files are
removed
#attempt a clean stop for all non-manager processes
logit info "${SCRIPTNAME}(stop_everything) - Stopping all processes"
call_ggsci 'stop er *'
#ensure everything is stopped
call_ggsci 'stop er *!'
#in case there are lingering processes
call_ggsci 'kill er *'
#stop Manager without (y/n) confirmation
call_ggsci 'stop manager!'
#Remove the process files:
rm -f $GGS_HOME/dirpcs/MGR.pcm
rm -f $GGS_HOME/dirpcs/*.pce
rm -f $GGS_HOME/dirpcs/*.pcr
}
case $1 in
'start')
# stop all GG processes and remove process files
logit info "${SCRIPTNAME} - start - Starting all processes"
stop_everything
sleep ${start_delay_secs}
#Now can start everything...
#start Manager
logit info "${SCRIPTNAME} - start - Starting Manager, autostarting
processes"
call_ggsci 'start manager'
27
28
#exit success
exit 0
;;
'abort')
# stop all GG processes and remove process files
logit info "${SCRIPTNAME} - abort - Stopping all processes"
stop_everything
#exit success
exit 0
;;
esac
29
References
30
Copyright 2013, Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only and the
contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other
August 2013
warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or
fitness for a particular purpose. We specifically disclaim any liability with respect to this document and no contractual obligations are
formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any
means, electronic or mechanical, for any purpose, without our prior written permission.
Oracle Corporation
World Headquarters
Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.
AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices.
U.S.A.
Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license
Worldwide Inquiries:
Phone: +1.650.506.7000
Fax: +1.650.506.7200
oracle.com
and are trademarks or registered trademarks of SPARC International, Inc. UNIX is a registered trademark licensed through X/Open
Company, Ltd. 1010