Professional Documents
Culture Documents
Fundamentals
Student Guide
Version 10.4
October 2009
Oracle GoldenGate Fundamentals Student Guide, version 10.4
Copyright © 1995, 2009 Oracle and/or its affiliates. All rights reserved.
This software and related documentation are provided under a license agreement containing restrictions on use and
disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement
or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute,
exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or
decompilation of this software, unless required by law for interoperability, is prohibited.
The information contained herein is subject to change without notice and is not warranted to be error-free. If you
find any errors, please report them to us in writing.
If this software or related documentation is delivered to the U.S. Government or anyone licensing it on behalf of the
U.S. Government, the following notice is applicable:
U.S. GOVERNMENT RIGHTS Programs, software, databases, and related documentation and technical data
delivered to U.S. Government customers are "commercial computer software" or "commercial technical data"
pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such,
the use, duplication, disclosure, modification, and adaptation shall be subject to the restrictions and license terms
set forth in the applicable Government contract, and, to the extent applicable by the terms of the Government
contract, the additional rights set forth in FAR 52.227-19, Commercial Computer Software License (December 2007).
Oracle USA, Inc., 500 Oracle Parkway, Redwood City, CA 94065.
This software is developed for general use in a variety of information management applications. It is not developed
or intended for use in any inherently dangerous applications, including applications which may create a risk of
personal injury. If you use this software in dangerous applications, then you shall be responsible to take all
appropriate fail-safe, backup, redundancy, and other measures to ensure the safe use of this software. Oracle
Corporation and its affiliates disclaim any liability for any damages caused by use of this software in dangerous
applications.
Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their
respective owners.
This software and documentation may provide access to or information on content, products, and services from third
parties. Oracle Corporation and its affiliates are not responsible for and expressly disclaim all warranties of any
kind with respect to third-party content, products, and services. Oracle Corporation and its affiliates will not be
responsible for any loss, costs, or damages incurred due to your access to or use of third-party content, products, or
services.
Contents
Our Business
Real-Time Access
to
Real-Time Information
We offer a robust yet easy platform for moving real-time transactional data between
operational and analytical systems to enable both:
- High Availability solutions, and
- Real Time Integration solutions
• Real-Time Access -- meaning your critical data is accessible and available
whenever you need it, 24x7.
At the same time…
• Real-Time Information -- meaning that the data available is as current as
possible – not 24 hours old, not even 4 hours old.
Oracle GoldenGate Success
Rapid Growth in
Strategic Partners
Our partnerships are rapidly increasing with major technology players, including
database and IT infrastructure, packaged applications, business intelligence, and
service providers.
And because our software platform supports a variety of solution use cases… Our
more than 500 customers are using our technology for over 4000 solutions around the
world. What we typically find that once an initial solution is implemented and the
benefits achieved, our customers then find additional areas across the enterprise
where we can further drive advantages for them.
Oracle GoldenGate Fundamentals Student Guide
7
TRANSACTIONAL
DATA INTEGRATION
•High Availability
Live Standby for an immediate fail-over solution that can later re-synchronize with
your primary source.
Active-Active solutions for continuous availability and transaction load distribution
between two or more active systems.
•Live Reporting
Feeding a reporting database so that you don’t burden your source production
systems.
9
Oracle GoldenGate for High Availability & Disaster Tolerance
Active-Active
Improved Uptime
Higher Performance
Zero-Downtime Operations for:
Faster Recovery
Upgrades
Minimized Data Loss
Migrations
Maintenance
Lower TCO
For High Availability and Disaster Tolerance solutions, it’s about “real-time” or
CONTINUOUS access to your data via your critical applications.
Benefits:
Eliminate unplanned downtime
Reduce data loss and isolate corrupt data
Re-synchronize backup and primary systems
Remove distance constraints
Automate switchovers
Improve ROI with active standby available for reporting
The customer’s Return on Investment can be further increased by using the live
standby system for reporting or testing – Oracle GoldenGate allows the standby
database to be open, so it does not have to sit idle and can be put to work!
11
High Availability: Active-Active
Benefits:
Achieve continuous availability
Enable transaction load distribution (with built-in conflict resolution)
Improve performance
Lower TCO
The active-active solution allows data updates and changes (“write” activity) to occur
on two or more active databases supporting live applications. Oracle GoldenGate
synchronizes the two active databases by replicating the data between each at a
logical level, and allows load distribution to improve system performance. In the case
of an outage of one system, there is no downtime for the end user because the other
active system continues with operations.
Because Oracle GoldenGate is an asynchronous solution, conflict management is
required to ensure data accuracy in the event that the same row is changed in two or
more databases at (or about) the same time. Oracle GoldenGate provides capabilities
to detect and resolve conflicts as well.
Benefits:
Eliminate “planned downtime” during hardware, database, OS and/or
application upgrades and migrations
Minimize risk with fail-back contingency
Improve success with phased user migrations
Oracle GoldenGate captures all the changed data in the primary system while the new
system is initiated and prepared. Once the second or the new system is upgraded or
migrated Oracle GoldenGate applies all the changed data to the new system. Oracle
GoldenGate then keeps the two environments synched with our real-time replication.
Often with such projects, there is always concern about what will happen once you
switchover to the new environment. Oracle GoldenGate alleviates many of those risks
with our fail-back capabilities -- after switchover Oracle GoldenGate captures the
changes that happen in the new system so that the old system is kept up to date in
case there is a need for fail back to the old environment.
13
Oracle GoldenGate for Real-Time Data Integration
Real-Time Information
Live Reporting
For our Real-Time Data Integration solutions, it’s about “real-time information” or
access to CURRENT operational data.
Benefits:
Use real-time data for better, faster decision making
Remove reporting overhead on source system
Reduce cost-to-scale as user demands and data volumes grow
Leverage cost-effective systems for reporting needs
Oracle GoldenGate feeds real-time data from the source to a secondary reporting-only
database such as an operational data store (ODS). This allows reporting activity to be
off-loaded from the production database. This secondary database can be a different
database and/or platform from the production database, to lower the total cost of
ownership and allow organizations to leverage emerging open source technologies.
The solution also helps increase scalability as user demands and data volumes grow.
15
Operational Business Intelligence
Benefits:
Use real-time data for better, faster decision making
Eliminate batch window dependency
Reduce overhead on source system
Maintain referential integrity for data quality
Leverage its flexibility for transformations and integration with ETL
18
In each of these architectures combining real-time change data capture with ETL
decreases data latency to real time or near real-time and eliminates the batch window
dependency.
Oracle GoldenGate Fundamentals Student Guide
Benefits:
Easily integrate large volumes of real-time data
between transaction processing systems
Reduce overhead; Eliminate batch windows
Improve scalability
Enhance SOA and EDA environments (delivery to
JMS-based messaging systems)
19
Oracle GoldenGate provides real-time data integration between OLTP systems non-
intrusively and with minimal impact. Distributed databases and the applications they
support can continuously access, utilize, and act on the most current operational data
available. The solution can also integrate with JMS-based messaging systems to
enable event driven architecture (EDA) and to support service oriented architecture
(SOA).
17
Technology Overview
Source Target
Database(s) Target Source Database(s)
Delivery Capture
Trail Trail
Bi-directional
Oracle GoldenGate consists of decoupled modules that are combined to create the best
possible solution for your business requirements.
Routing:
•Data is sent via TCP/IP to the target systems. Data compression and encryption are
supported. Thousands of transactions can be moved per second, without distance
limitations.
•The Delivery (Replicat) process applies transactional data to the designated target
systems using native SQL calls.
Bi-directional:
•In bi-directional configurations/solutions, this process runs the same in reverse, to
concurrently synchronize data between the source and target systems.
Capture:
Oracle Windows 2000, 2003, XP
DB2 Linux
Microsoft SQL Server Sun Solaris
Sybase ASE HP NonStop
Ingres HP-UX
Teradata HP TRU64
Enscribe
HP OpenVMS
SQL/MP
IBM AIX
SQL/MX
IBM z/OS
Delivery:
All listed above
MySQL, HP Neoview, Netezza, and any
ODBC compatible databases
ETL products
JMS message queues
We support log-based Capture of changed data from nearly all major database
vendors.
We can Deliver that data to an even wider range of targets – including open source
databases, several data warehouse appliances, ETL servers, and JMS message queues
to support Service Oriented Architectures (SOA) and Event-Driven Architectures
(EDA).
19
Oracle GoldenGate Advantages
Oracle GoldenGate Director supports all platforms and databases supported by Oracle
GoldenGate.
Oracle GoldenGate Fundamentals Student Guide
Benefits:
Reduce financial/legal risk exposure
Speed and simplify IT work in
comparing data sources
No disruption to business systems
Improved failover to backup systems
Confident decision-making and
reporting
21
Architecture
Primarily used for change data capture and delivery from database
transaction logs
Can optionally be used for initial load directly from database tables
Especially useful for synchronizing heterogeneous databases
Database-specific methods may be preferable for homogeneous
configurations
Source
Database
Network
(TCP/IP) Target
Database
Extract Server Replicat
Collector Trail
Transaction
Log
Manager Manager
Manager processes on both systems control activities such as starting, monitoring and
restarting processes; allocating data storage; and reporting errors and events.
Source
Database
Network
(TCP/IP) Target
Database
Extract Server Replicat
Collector Remote
Trail
Transaction
Log
Local Data
Trail Pump
Manager Manager
Manager processes on both systems control activities such as starting, monitoring and
restarting processes; allocating data storage; and reporting errors and events.
23
Initial Load
Source Network
Database (TCP/IP) Target
Tables Database
Extract Replicat
Or DB Bulk
Server Load Utility
Collector Files
Manager Manager
Manager processes on both systems control activities such as starting, monitoring and
restarting processes; allocating data storage; and reporting errors and events.
Oracle GoldenGate Fundamentals Student Guide
Change data capture & delivery can be run either continuously (online)
or as a special run (batch run) to capture changes for a specific period of
time.
Checkpointing - Extract
For change data capture, Extract and Replicat save checkpoints to a
checkpoint file so they can recover in case of failure
Extract maintains:
2 input checkpoints
1 output checkpoint for each trail it writes to
Input:
Transaction Log
Checkpoints
Checkpoints are used during online change synchronization to store the current read
and write position of a process. Checkpoints ensure that data changes marked for
synchronization are extracted, and they prevent redundant extractions. They provide
fault tolerance by preventing the loss of data should the system, the network, or a
GoldenGate process need to be restarted.
25
Checkpointing - Replicat
Input:
GoldenGate Trail
Checkpoints
27
Configuring Oracle GoldenGate
2. Change Capture
3. Initial Load
4. Change Delivery
Note: You can run log-based change capture after the initial data load if you set the
extract begin time to the start of the longest running transaction committed
during the initial data load.
Source
Database
Transaction
Log
Local Data
Extract Pump Remote Replicat
Trail Trail
4. Change Delivery
2. Change Capture
Transaction
Log
Data
Extract Local Pump Remote Replicat
Trail Trail
4. Change Delivery
2. Change Capture
29
Installing Oracle GoldenGate installs all of the components required to run and
manage GoldenGate processing, and it installs the GoldenGate utilities.
Manager must be running on each system before Extract or Replicat can be started,
and must remain running while those processes are running so that resource
management functions are performed.
The source definitions file contains the definitions of the source tables and is required
on the target system in hetereogeneous configurations. Replicat refers to the file to
when transforming data from the source to the target.
Identify the proper release of GoldenGate for your source and target environments
Database and version
Operating system and version
For Windows: Do not install Oracle GoldenGate into a folder that contains spaces in
its name, for example “GoldenGate Software.” The application references path
names, and the operating system does not support path names that contain spaces,
whether or not they are within quotes.
Syntax:
INSTALL <item> [<item> …]
Example:
C:\GGS> INSTALL ADDEVENTS ADDSERVICE
31
this way, Manager will stop when the user logs out. By using install, you can install
Manager as a Windows service so that it can be operated independently of user
connections and can be configured to start either manually or when the system starts.
You can configure the Manager service to run as the Local System account or as a
specific named account. The configuration of a service can be changed by using the
Services applet of the Windows Control Panel and changing the service Properties.
AUTOSTART Specifies that the service be started at system boot time (the default).
MANUALSTART Specifies that the service be started only at user request (with GGSCI
or the Control Panel Services applet).
USER Specifies a user name to logon as when executing Manager. If specified, user
name should include the domain name, a backward slash, and the user name.
PASSWORD Specifies the user’s password for logon purposes. This can be changed
using the Control Panel Services applet.
INSTALL ADDSERVICE
Reads the GLOBALS MGRSERVNAME for the service name
If no GLOBALS setting, uses default service name GGSMGR
A GLOBALS file stores parameters that relate to the GoldenGate instance as a whole,
as opposed to runtime parameters for a specific process. This file is referenced when
installing the Windows service, so that the correct name is registered.
Oracle GoldenGate Fundamentals Student Guide
gzip –d {filename}.tar.gz
For UNIX, z/OS, or Linux: Use the gzip and tar options appropriate for your
system. If you are installing GoldenGate into a cluster environment, make certain
that the GoldenGate binaries and files are installed on a file system that is available
to all cluster nodes. After installing GoldenGate, make certain to configure the
GoldenGate Manager process within the cluster application, as directed by the
vendor’s documentation, so that GoldenGate will fail over properly with the other
applications. The Manager process is the master control program for all GoldenGate
operations.
The ggmxinstall script SQL compiles the Extract program and installs the VAMSERV
program in the NonStop Guardian space.
33
The command to run it is:
OSS> ggmxinstall /G/<Guardian vol>/<Guardian subvol>
Directory Contents
dirchk GoldenGate checkpoint files
dirdat GoldenGate trail and extract files
dirdef Data definitions produced by DEFGEN and used to
translate heterogeneous data
dirpcs Process status files
dirprm Parameter files
dirchk
Contains the checkpoint files created by Extract and Replicat processes, which store
current read and write positions to support data accuracy and fault tolerance. Written
in internal GoldenGate format. Do not edit these files.
dirdat
The default location for GoldenGate trail files and extract files created by Extract
processes to store records of extracted data for further processing, either by the
Replicat process or another application or utility. Written in internal GoldenGate
format. Do not edit these files.
dirdef
Oracle GoldenGate Fundamentals Student Guide
The default location for data definitions files created by the DEFGEN utility to contain
source or target data definitions used in a heterogeneous synchronization
environment. Written in external ASCII.
File name format is a user-defined name specified in the DEFGEN parameter file.
These files may be edited to add definitions for newly created tables. If you are unsure
of how to edit a definitions file, contact technical support. Example: defs.dat
dirpcs
Default location for status files. File name format is <group>.<extension> where
<group> is the name of the group and <extension> is either pce (Extract), pcr
(Replicat), or pcm (Manager).
These files are only created while a process is running. The file shows the program
name, the process name, the port, and process ID that is running. Do not edit these
files. Examples: mgr.pcm, ext.pce
dirprm
The default location for GoldenGate parameter files created by GoldenGate users to
store run-time parameters for GoldenGate process groups or utilities. Written in
external ASCII format.
dirrpt
The default location for process report files created by Extract, Replicat, and Manager
processes to report statistical information relating to a processing run. Written in
external ASCII format.
dirsql
The default location for SQL scripts.
dirtmp
The default location for storing large transactions when the size exceeds the allocated
memory size. Do not edit these files.
35
Oracle GoldenGate Documentation
● Oracle GoldenGate Quick Install Guide: Describes the structure of the media pack
and where to find installation instructions.
● Oracle GoldenGate Installation and Setup Guides: There is an installation guide
and setup guide for each database that is supported by Oracle GoldenGate. These
include database-specific configuration information.
● Oracle GoldenGate Administration Guide: Introduces Oracle GoldenGate
components and explains how to plan for, configure, and implement Oracle GoldenGate
on the Windows and UNIX platforms.
● Oracle GoldenGate Reference Guide: Provides detailed information about Oracle
GoldenGate parameters, commands, and functions for the Windows and UNIX
platforms.
● Oracle GoldenGate Troubleshooting and Tuning Guide: Provides suggestions for
improving the performance of Oracle GoldenGate in different situations, and provides
solutions to common problems.
Oracle GoldenGate Fundamentals Student Guide
Parameter file
mgr.prm file in GGS ./dirprm directory
Manager Parameters
Enter Manager parameters in the dirprm/mgr.prm file, under the GoldenGate
installation directory. If no mgr.prm file exists, default management parameters are
used.
37
Prepare Environment: Manager - Configuration
Starting Manager
You must start Manager before most other configuration tasks performed in GGSCI.
Use either START MANAGER or START MGR.
On Windows systems, you can also start and stop Manager through the standard
Windows services control applet (in Control Panels).
PORT 7809
DYNAMICPORTLIST 8001, 8002, 9500–9520
AUTOSTART ER *
AUTORESTART EXTRACT *, WAITMINUTES 2, RETRIES 5
LAGREPORTHOURS 1
LAGINFOMINUTES 3
LAGCRITICALMINUTES 5
This parameter file has the Manager listening on PORT 7809. Ports 8001, 8002, and
those in the range 9500 to 9520 will be assigned to the dynamic processes started by
Manager.
This manager process will recycle GoldenGate trails that match the file name of
/ggs/dirdat/aa* and /ggs/dirdat/bb*. It will only recycle the trail once all Extracts
Oracle GoldenGate Fundamentals Student Guide
and Replicats have a checkpoint beyond the file (USECHECKPOINTS), however bb* trails
will not be purged until there has been no activity for 5 days.
The manager will automatically start any Extract and Replicat process at startup and
will attempt to restart any Extract process that abends after waiting 2 minutes, but
only up to 5 attempts.
The manager will report lag information every hour, but only for processes that have
3 and 5 minutes of latency. The message will be flagged informational for lags of 3
minutes and critical for any process that has a lag greater than 5 minutes.
The problem
Understanding source and target layouts across disparate
systems and databases
The Problem
When Capturing, Transforming, and Delivering data across disparate systems and
databases, you must understand both the source and target layouts. Understanding
column names and data types is instrumental to GoldenGate’s data synchronization
functions.
When transformation services are required on the source system, Extract can use a
definition file containing the target, rather than source, layouts.
39
Note: The user should never modify the DEFGEN output.
Unix Example:
defgen paramfile /ggs/dirprm/defgen.prm
reportfile /ggs/dirrpt/defgen.rpt
Windows Example:
defgen paramfile c:\ggs\dirprm\defgen.prm
reportfile c:\ggs\dirrpt\defgen.rpt
Parameter Specifies
DEFSFILE The output definitions file location and name
Database access
You need to assign a database user for each of the GoldenGate processes, unless the
database allows authentication at the operating system level. While not required,
GoldenGate recommends creating a user specifically for the GoldenGate application.
To ensure that processing can be monitored accurately, do not permit other users or
processes to operate as the GoldenGate user.
In general, the following permissions are necessary for the GoldenGate user:
• On the source system, the user must have permissions to read the data dictionary or
catalog tables.
• On the source system, the user must have permissions to select data against the
tables.
• On the target system, the user must have the same permissions as the GoldenGate
user on the source system plus additional privileges to perform DML on the target
tables.
41
Prepare Environment: Source Database
Oracle
Add minimal supplemental logging at database level
ADD TRANDATA to mark tables for replication
DB2
Enter DATA CAPTURE CHANGES at the column for LOB data type
ADD TRANDATA to mark tables for replication
Sybase
Set the secondary truncation point in the logs
ADD TRANDATA to mark tables for replication
NonStop SQL/MX
Special installation steps but no special database preparation
Oracle logs - On UNIX, GoldenGate reads the online logs by default, or the archived
logs if an online is not available. On the Windows platform, GoldenGate reads the
archived logs by default, or the online logs if an archive is not available. GoldenGate
recommends archive logging be enabled, and that you keep the archived logs on the
system for the longest time possible. This prevents the need to resynch data if the
online logs recycle before all data has been processed.
DB2 - In addition to enabling logging at a global level, each table to be captured must
be configured to capture data for logging purposes. This is accomplished by the DATA
CAPTURE CHANGE clause in the CREATE TABLE statement.
Sybase –To capture database operations for tables that you want to synchronize with
GoldenGate, each one must be marked for replication. This can be done through the
database, but GoldenGate recommends using ADD TRANDATA.
NonStop SQL/MX
During the installation of SQL/MX, the script ggmxinstall sets a pointer to the VAM
that will work with Extract to capture changes from the TMF audit trail.
Oracle GoldenGate Fundamentals Student Guide
ODBC Data Source Administrator is used to configure and define the ODBC
connection and to define the data source.
SQL Server Enterprise Manager is used to set full recovery mode and back up the
database.
SQL Server Query Analyzer is used to access the database to turn off log
truncation and non-logged bulk copy.
43
Prepare Environment: Source Database – SQL Server 2005
The Distributor database must be used only for source databases to be replicated by
GoldenGate. One Distributor can be used for all of these databases. GoldenGate does
not depend on the Distributor database, so transaction retention can be set to zero.
Because GoldenGate does not depend on the Distributor database, but rather reads
the logs directly, the GoldenGate extraction process can process at its customary high
speed. For instructions on installing the replication component and creating a
Distributor database, see the GoldenGate for Windows and UNIX Administrator
Guide.
MANAGESECONDARYTRUNCATIONPOINT
Required TRANLOGOPTIONS parameters control whether or not GoldenGate maintains
the secondary truncation point. Use the MANAGESECONDARYTRUNCATIONPOINT option
if GoldenGate will not be running concurrently with SQL Server replication. Use the
NOMANAGESECONDARYTRUNCATIONPOINT option if GoldenGate will be running
concurrently with SQL Server replication. Allows SQL Server replication to manage
the secondary truncation point.
Primary Key Requirement - The requirement that all tables to be captured from
SQL Server 2005 source databases must have a primary key is a requirement of the
Microsoft replication component, which is utilized by GoldenGate as part of the log-
based capture process.
Connecting - Use SQL Server Management Studio for SQL Server 2005/2000 to
connect to a MS SQL Server 2005 or 2000 database. Use Enterprise Manager only for
MS SQL Server 2000.
Oracle GoldenGate Fundamentals Student Guide
1. edelivery.oracle.com
2. Starting and stopping processes; monitoring processes; reporting lag, errors and
events; purging trail files.
45
GoldenGate Command Interface
Golden Gate Software Command Interface (GGSCI) provides on-line help for all
commands. The following is an example of the information returned when you enter
HELP STATUS EXTRACT:
Use STATUS EXTRACT to determine whether or not Extract groups are running.
Syntax:
STATUS EXTRACT <group name>
[, TASKS]
[, ALLPROCESSES]
<group name> is the name of a group or a wildcard (*) to specify multiple groups.
ALLPROCESSES displays status of all Extract processes, including tasks.
TASKS displays status of all Extract tasks.
Examples:
STATUS EXTRACT FINANCE, STATUS EXTRACT FIN*
Oracle GoldenGate Fundamentals Student Guide
GGSCI Commands
MANAGER EXTRACT REPLICAT ER EXTTRAIL RMTTRAIL TRANDATA CHECKPOINT TRACE
TABLE TABLE
ADD X X X X X X X
ALTER X X X X
CLEANUP X X X
DELETE X X X X X X X X
INFO X X X X X X X X X
KILL X X X
LAG X X X
REFRESH X
SEND X X X X
START X X X X
STATS X X X
STATUS X X X X
STOP X X X X
Objects
Manager, Extract, Replicat GoldenGate processes.
ER Multiple Extract and Replicat processes.
EXTTRAIL Local trail.
RMTTRAIL Remote trail.
TRANDATA Transaction data (from transaction logs).
CHECKPOINTTABLE Checkpoint table (on target database).
TRACETABLE Oracle trace table (on target database).
Commands
ADD Creates an object or enables TRANDATA capture.
ALTER Changes the attributes of an object.
CLEANUP Deletes the run history of a process or removes records from a checkpoint
table.
DELETE Deletes an object or disables TRANDATA capture.
INFO Displays information about an object (status, etc).
KILL Forces a process to stop (no restart).
LAG Displays the lag between when a record is processed by the process and the
source record timestamp.
REFRESH Refreshes Manager parameters (except port number) without stopping
Manager.
SEND Sends commands to a running process.
START Starts a process.
STATS Displays statistics for one or more processes.
STATUS Displays whether a process is running.
STOP Stops a process gracefully.
47
GGSCI Commands (cont’d)
Commands
Parameters SET EDITOR, EDIT PARAMS, VIEW PARAMS
Database DBLOGIN, ENCRYPT PASSWORD, LIST TABLES
DDL DUMPDDL [SHOW]
Miscellaneous !command, CREATE SUBDIRS, FC, HELP, HISTORY, INFO ALL,
OBEY, SHELL, SHOW, VERSIONS, VIEW GGSEVT, VIEW REPORT
Parameter commands
SET EDITOR Changes the default text editor for the current GGSCI session from
Notepad or vi to any ASCII editor.
EDIT PARAMS Edits a parameter file.
VIEW PARAMS Displays the contents of a parameter file.
Database commands
DBLOGIN Establishes a database connection through GGSCI.
ENCRYPT PASSWORD Encrypts a database login password.
LIST TABLES List all tables in the database that match a wildcard string.
DDL commands
DUMPDDL Saves the GoldenGate DDL history table to file.
SHOW option displays the DDL information in standard output format.
Miscellaneous commands
!command Executes a previous GGSCI command without modification.
CREATE SUBDIRS Creates default directories within the GoldenGate home directory.
FC Edit a previously issued GGSCI command.
HELP Displays information about a GGSCI command.
HISTORY List the most recent GGSCI commands issued.
INFO ALL Displays the status and lag for all GoldenGate processes on a system.
OBEY Runs a file containing a list of GGSCI commands.
SHELL Run shell commands from within GGSCI.
SHOW Displays the GoldenGate environment.
VERSIONS Displays OS and database versions.
VIEW GGSEVT Displays the GoldenGate event/error log.
VIEW REPORT Displays a process report for Extract or Replicat.
Oracle GoldenGate Fundamentals Student Guide
GGSCI Examples
START MGR
ADD EXTRACT myext, TRANLOG, BEGIN NOW
ADD EXTTRAIL /ggs/dirdat/rt, EXTRACT myext
START EXTRACT myext
49
Running GoldenGate from the OS Shell
You can also start GoldenGate processes from the OS command shell when running
a batch job or initial load, for example:
This is especially useful to schedule GoldenGate batch jobs to run during off-peak
hours using a command-line capable scheduler
<filepath> specifies the fully qualified name of the parameter and report files.
Transaction
Log
Local Data
Extract Pump Remote Replicat
Trail Trail
4. Change Delivery
2. Change Capture
Capture data directly from source tables for initial data load
51
Change Capture - Tasks
To configure Extract to capture changes from transaction logs, perform the following
steps:
Set up a parameter file for Extract with the GGSCI EDIT PARAMS command.
Set up an initial Extract checkpoint into the logs with the GGSCI ADD EXTRACT
command.
Optionally, create a local trail using the GGSCI ADD EXTTRAIL command and a data
pump Extract (and parameter file) reading from the local trail.
Set up a remote trail using the GGSCI ADD RMTTRAIL command.
Start the Server Collector process on the target system or let the Manager start the
Server Collector dynamically.
Start Extract using the GGSCI START EXTRACT command. For example: GGSCI>
START EXTRACT FINANCE
GGSCI sends this request to the Manager process, which in turn starts Extract.
Manager monitors the Extract process and restarts it, when appropriate, if it goes
down.
Add the initial Extract checkpoint with the GGSCI command ADD EXTRACT:
SOURCEISTABLE Creates an Extract task that extracts entire records from the
database for an initial load. If SOURCEISTABLE is not specified, ADD EXTRACT
creates an online change-synchronization process, and one of the other data source
options must be specified. When using SOURCEISTABLE, do not specify service
options. Task parameters must be specified in the parameter file.
53
Change Capture - ADD EXTRACT <starting point>
Several parameters specify the position with the log or trail to begin processing:
LSN <value>
Valid for SQL Server or Ingres. Specifies the LSN in a SQL Server or Ingres
transaction log at which to start capturing data. The LSN specified should exist in a
log backup or the online log.
- For SQL Server, an LSN is composed of three hexadecimal numbers separated by
colons. The first is the virtual log file number, the second is the segment number
within the virtual log, and the third is the entry number.
- For Ingres, an LSN is two, 4-byte unsigned integers, separated by a colon. For
example, to specify an LSN of 1206396546,43927 (as viewed in an Ingres utility), you
would enter 1206396546:43927.
- An alias for this option is EXTLSN.
55
Change Capture - ADD EXTRACT <processing options>
Create a data-pump Extract group named “finance” that reads from the
GoldenGate trail c:\ggs\dirdat\lt.
EXTRACT ODS
USERID GoldenUser, PASSWORD password
RMTHOST serverx, MGRPORT 7809
RMTTRAIL /ggs/dirdat/rt
TABLE SALES.ORDERS;
TABLE SALES.INVENTORY;
Examples:
ADD EXTTRAIL c:\ggs\dirdat\aa, EXTRACT finance, MEGABYTES 10
<trail name> The fully qualified path name of the trail. The actual trail name can
contain only two characters. GoldenGate appends this name with a six-digit sequence
number whenever a new file is created. For example, a trail named /ggs/dirdat/tr
would have files named /ggs/dirdat/tr000001, /ggs/dirdat/tr000002, and so forth.
<group name> The name of the Extract group to which the trail is bound. Only one
Extract process can write data to a trail.
MEGABYTES <n> The maximum size, in megabytes, of a file in the trail. The
default is 10.
57
Change Capture - Start Extract
Trail /ggs/dirdat/rt000000
/ggs/dirdat/rt000001
ADD RMTTRAIL
Identifies the GoldenGate trail on the target system and the name of the EXTRACT
group.
/ggs/dirdat/rt000000
Trail
/ggs/dirdat/rt000001
SOURCEDB <dsn> supplies a data source name as part of the connection information.
USERID <login>, PASSWORD <pw> supplies database credentials.
RMTHOST <hostname> specifies the target system.
MGRPORT <port> specifies the port where Manager is running.
RMTTRAIL <trail id> specifies the GoldenGate trail on the target system.
TABLE <table name> specifies a source table for which activity will be extracted.
ADD RMTTRAIL
Identifies the GoldenGate trail on the target system and the name of the EXTRACT
group.
59
Data Pumps - Overview
For most business cases, it is best practice to use a data pump. Some reasons for using
a data pump include the following:
• Protection against network and target failures: In a basic GoldenGate
configuration, with only a trail on the target system, there is nowhere on the source
system to store data that Extract continuously extracts into memory. If the network
or the target system becomes unavailable, the primary Extract could run out of
memory and abend. However, with a trail and data pump on the source system,
captured data can be moved to disk, preventing the abend. When connectivity is
restored, the data pump extracts the data from the source trail and sends it to the
target system(s).
• You are implementing several phases of data filtering or transformation. When
using complex filtering or data transformation configurations, you can configure a
data pump to perform the first transformation either on the source system or on the
target system, and then use another data pump or the Replicat group to perform the
second transformation.
• Consolidating data from many sources to a central target. When synchronizing
multiple source databases with a central target database, you can store extracted data
on each source system and use data pumps on each system to send the data to a trail
on the target system. Dividing the storage load between the source and target systems
reduces the need for massive amounts of space on the target system to accommodate
data arriving from multiple sources.
• Synchronizing one source with multiple targets. When sending data to multiple
target systems, you can configure data pumps on the source system for each one. If
network connectivity to any of the targets fails, data can still be sent to the other
targets.
Oracle GoldenGate Fundamentals Student Guide
Trail
Primary Data
Trail Trail
Extract Pump
Trail
A data pump can be set up to duplicate or selectively route the data to multiple trails.
However, if the trails are on multiple target systems and the communication to one of
the systems goes down, the Extract may exhaust its retries and shut down, causing
the updates to all targets to stop.
Data
Trail
Pump 1
Primary Data
Trail Trail
Extract Pump 2
Data Trail
Pump 3
61
Change Capture – Discussion Points
1. What does Extract do?
2. Where does Extract capture transactional changes from?
3. What parameters tell Extract where to send data?
4. What commands are used to create and start an Extract group?
5. What command option is used to set how large a GoldenGate trail
file may get before it rolls to the next file?
1. Captures incremental changes from database transaction logs. It can also save
source data from the tables themselves or other GoldenGate trails. Writes the
captured data to GoldenGate trails or files.
2. From transaction logs (or archive logs) – except for Teradata.
3. EXTTRAIL, EXTFILE
RMTHOST with RMTTRAIL, RMTFILE or RMTTASK
4. EDIT PARAMS
ADD EXTRACT
ADD {EXTTRAIL | RMTTRAIL |EXTFILE | RMTFILE}
START EXTRACT
5. The MEGABYTE <megabytes> option in the ADD EXTTRAIL or ADD RMTTRAIL
commands.
EXTRACT <primary>
<login for your database>
EXTTRAIL ./dirdat/<trailid>
<table statements as required>;
EXTRACT <datapump>
PASSTHRU
RMTHOST <target >, MGRPORT <port>
RMTTRAIL ./dirdat/<rmttrail>
<table statements as required>;
Add a data pump (source is the local trail from the primary Extract )
PASSTHRU parameter is used on a data pump if you do not need to do any data
transformations or user exit processing.
Add the data pump extract with a local trail as source and remote trail as destination.
1. A secondary Extract process that reads from a local trail and distributes that data to a
remote system.
2. Allows a local trail on the source system, which is useful for recovery if the network or
target system fails.
3. To send to multiple target systems (so if one goes down, they don’t all); to separate
out different tables; for parallel processing (faster).
4. RMTHOST is used to identify the name or IP address of the remote system and the
port that is being used.
5. The PASSTHRU parameter is used on a data pump (unless you need to perform data
transformation or user exit processing).
6. Review Architecture slide for change capture & delivery using a data pump.
63
Step 3. Initial Load
Transaction
Log
Initial Load
Can use GoldenGate TDM methods:
Notes:
Run a test initial load early on for timing and sizing
Run the actual initial load after starting change capture on the source
Initial Load
An initial load takes a copy of the entire source data set, transforms it if necessary,
and applies it to the target tables so that the movement of transaction data begins
from a synchronized state.
The first time that you start change synchronization will be during the initial load
process. Change synchronization keeps track of ongoing transactional changes while
the load is being applied.
Oracle GoldenGate Fundamentals Student Guide
Break mirror
Break from database mirroring.
Array fetch
GoldenGate 10.0 and later fetches 1000 rows (except for LOBs) but you can control
this by DBOPTIONS FETCHBATCHSIZE
65
Initial Load: File to Replicat
Manager
Source Target
Database Database
Extract Replicat
Files
File to Replicat
Captures data directly from the source tables and writes to an Extract file for Replicat
to process.
Extract parameters
SOURCEISTABLE instructs Extract to read the source tables directly rather than
the transaction log.
Using Replicat provides the ability to perform additional data transformation prior to
loading the data.
Execution
You can start Extract by the GGSCI command: START EXTRACT <name>
Or from the command shell with syntax:
extract paramfile <command file> [ reportfile <out file> ]
Oracle GoldenGate Fundamentals Student Guide
SQL*
Manager File
Loader
Source Target
Database Database
File BCP
Extract
File SSIS
Extract parameters
SOURCEISTABLE instructs Extract to read the source tables directly rather than
the transaction log.
To format the output for native bulk utilities, such as SSIS, BCP, or SQL*Loader,
use:
RMTFILE
FORMATASCII with appropriate options, like BCP or SQLLOADER
Execution
You can start Extract by the GGSCI command: START EXTRACT <name>
Or from the command shell with syntax:
extract paramfile <command file> [ reportfile <out file> ]
67
Initial Load: Direct Load
Manager Manager
Source Target
Database Database
Extract Replicat
Direct Load
Captures data directly from the source tables and sends the data in large blocks to the
Replicat process.
Using Replicat provides the ability to perform additional data transformation prior to
loading the data.
Extract parameters
Here you have RMTTASK (instead of RMTFILE in the Queue Data method).
RMTTASK instructs the Manager process on the target system to start a Replicat
process with a group name specified in the GROUP clause.
Execution
When you add Extract and Replicat:
•SOURCEISTABLE instructs Extract to read the source tables directly rather than
the transaction log.
•SPECIALRUN on Replicat specifies one-time batch processing where checkpoints are
not maintained.
The initial data load is then started using the GGSCI command START EXTRACT.
The Replicat process will be automatically started by the Manager process. The port
used by the Replicat process may be controlled using the DYNAMICPORTLIST
Manager parameter.
Oracle GoldenGate Fundamentals Student Guide
Manager Manager
Source Oracle
Database Target
The Manager process dynamically starts the Replicat process. Using Replicat provides
the ability to perform additional data transformation prior to loading the data. The
port used by the Delivery process may be controlled using the DYNAMICPORTLIST
Manager parameter.
Extract parameters
Here you have RMTTASK (instead of RMTFILE in the Queue Data method).
RMTTASK instructs the Manager process on the target system to start a Replicat
process with a group name specified in the GROUP clause.
Replicat parameters
The parameter BULKLOAD distinguishes this from the direct load method.
Execution
When you add Extract and Replicat:
•SOURCEISTABLE instructs Extract to read the source tables directly rather than
the transaction log.
•SPECIALRUN on Replicat specifies one-time batch processing where checkpoints are
not maintained.
The initial data load is then started using the GGSCI command START EXTRACT.
The Replicat process will be automatically started by the Manager process.
The port used by the Replicat process may be controlled using the
DYNAMICPORTLIST Manager parameter.
69
Initial Load – Discussion Points
1. File to Replicat (Extract writes to a file for Replicat to load via SQL)
File to database utility (Extract writes to ASCII files formatted for database utilities to
load)
Direct load (Extract writes directly to Replicat which loads via SQL)
Direct bulk load (Oracle only – Extract writes directly to Replicat, which loads through
the SQL*Loader API)
2. ADD EXTRACT with SOURCEISTABLE
ADD REPLICAT with SPECIALRUN
3. HANDLECOLLISIONS; in the Replicat parameter file for change delivery; turn off
after initial load data processed.
Oracle GoldenGate Fundamentals Student Guide
Transaction
Log
Local Data
Extract Pump Remote Replicat
Trail Trail
Replicat can:
Overview
GoldenGate trails are temporary queues for the Replicat process. Each record header
in the trail provides information about the database change record. Replicat reads
these trail files sequentially, and processes inserts, updates and deletes that meet
your criteria. Alternatively, you can filter out the rows you do not wish to deliver, as
well as perform data transformation prior to applying the data.
71
for optimal performance. You can configure multiple Replicat processes for increased
throughput.
When replicating, Replicat preserves the boundaries of each transaction so that the
target database has the same degree of integrity as the source. Small transactions can
be grouped into larger transactions to improve performance. Replicat uses a
checkpointing scheme so changes are processed exactly once. After a graceful stop or a
failure, processing can be restarted without repetition or loss of continuity.
Replicat reads the GoldenGate trail and applies changes to the target database. Like
Extract, Replicat uses checkpoints to store the current read and write position and is
added and started using the processing group name.
Oracle GoldenGate Fundamentals Student Guide
REPLICAT REPORD
TARGETDB dsn USERID ggsuser PASSWORD ggspass
Replicat -- USERID ggsuser, PASSWORD ggspass
ASSUMETARGETDEFS
DISCARDFILE /ggs/dirrpt/REPORD.dsc, APPEND
MAP SALES.ORDERS, TARGET SALES.ORDERS;
MAP SALES.INVENTORY, TARGET SALES.INVENTORY;
In this example:
DBLOGIN USERID and PASSWORD logs the user into the database in order to add the
checkpoint table.
Replicat parameters:
TARGETDB identifies the data source name (not required for Oracle)
USERID and PASSWORD provide the credentials to access the database
ASSUMETARGETS is used when the source and target systems have the same data
definition with identical columns.
DISCARDFILE creates a log file to receive records that cannot be processed.
MAP establishes the relationship between source table and the target table.
ADD REPLICAT names the Replicat group REPORD and establishes a local trail
(EXTTRAIL) with the two-character identifier rt residing on directory dirdat.
If the source database remains active during an initial load, you must
either avoid or handle any collisions when updating the target with
interim changes
Avoiding Collisions
If you can backup/restore or clone the database at a point in time, you
can avoid collisions by starting Replicat to read trail records from a
specific transaction Commit Sequence Number (CSN):
START REPLICAT <group> ATCSN | AFTERCSN <csn>
73
Avoiding Collisions using the CSN
1. Use a standby copy of the source database for the initial load.
2. After the initial load completes, note the highest CSN number of the standby
database. The CSN varies by database, e.g. for Oracle it is the SCN.
3. Start Replicat to read from the next CSN:
ATCSN <csn> Causes Replicat to skip transactions in the trail until it finds a
transaction that contains the specified CSN. <csn> must be in the format
that is native to the database.
AFTERCSN <csn> Causes Replicat to skip transactions in the trail until it
finds the first transaction after the one that contains the specified CSN.
SKIPTRANSACTION Causes Replicat to skip the first transaction in the trail
upon startup. All operations in that first transaction are excluded.
If you cannot avoid collisions by the prior method, you must handle
collisions
HANDLECOLLISIONS processing requires that each target table have a primary key or
unique index. If you cannot create a temporary primary key or unique index through
your application, use the KEYCOLS argument of the TABLE or MAP parameter to
designate columns as a substitute key. Otherwise, the source database must be
quiesced for the initial load.
Oracle GoldenGate Fundamentals Student Guide
Note: Once all of the change data generated during the load has been replicated, turn
off HANDLECOLLISIONS:
1. Reads change data from GoldenGate trails and applies them to a target database via
SQL commands.
2. When the source and target table structures (column order, data type and length) are
identical.
3. It uses the source definitions created by DEFGEN.
4. ADD CHECKPOINTTABLE (optional)
EDIT PARAMS
ADD REPLICAT
START REPLICAT
5. ADD CHECKPOINTTABLE (must be logged into database).
6. Identifies operations that could not be processed by Replicat.
7. Review Architecture slide for change capture & delivery (without data pump).
75
Extract Trails and Files
Introduction
Viewing in Logdump
120
121
When transporting trails via TCP/IP, a Server Collector process on the target
platform collects, writes, and checkpoints blocks of records in one or more extract
files.
- By default, only the primary key and changed columns are recorded
77
Extract Trails and Files - Cleanup
Trail cleanup
If one Replicat is configured to process the trail, you can instruct the Replicat to purge
the data once it has been consumed. As long as Replicat remains current, your
temporary storage requirements for trails can be very low. If multiple Replicat
processes are configured against a single trail, you can instruct the Manager purge
data trail data as soon as all checkpoints have been resolved. As long as replication
processes keep pace, temporary storage requirements can be kept quite low.
Oracle GoldenGate Fundamentals Student Guide
Each trail file has a trail file header and trail records
GoldenGate Trails
Trail files are unstructured files containing variable length records. They are
unstructured and written to in large blocks for best performance.
Checkpoints
Both Extract and Replicat maintain checkpoints into the trails. Checkpoints provide
persistent processing whenever a failure occurs. Each process resumes where the last
checkpoint was saved guaranteeing no data is lost. One Extract can write to one to
many trails. Each trail can then be processed by one or many Replicat processes.
Extract information
GoldenGate version
Group name
Host name
Hardware type
OS type and version
DB type, version and character set
79
GoldenGate Data Format - Compatibility Level
The input and output trails of a data pump must have the same
compatibility level
Note: To ensure that the correct file version is applied, it is best practice to use
RELEASE to match the trail to the version of the Extract or Replicat that will be
reading it, unless the version that you downloaded instructs you otherwise. If full
audit recovery (FAR) mode is not enabled for Extract, the file version number
automatically defaults to 9 (GoldenGate version 9.x). FAR is controlled by the
RECOVERYOPTIONS parameter.
Oracle GoldenGate Fundamentals Student Guide
Each database management system generates some kind of unique serial number of
its own at the completion of each transaction, which uniquely identifies that
transaction. A CSN captures this same identifying information and represents it
internally as a series of bytes. A comparison of any two CSN numbers, each of which
is bound to a transaction commit record in the same log stream, reliably indicates the
order in which the two transactions completed. However, because the CSN is
processed in a platform-independent manner, it supports faster, more efficient
heterogeneous replication than using the database-supplied identifier.
All database platforms except Oracle, DB2 LUW, and DB2 z/OS have fixed-length
CSNs, which are padded with leading zeroes as required to fill the fixed length. CSNs
that contain multiple fields can be padded within each field, such as the Sybase CSN.
Ingres
<LSN-high>:<LSN-low>
Where:
<LSN-high> is the newest log file in the range. Up to 4 bytes padded with leading
zeroes.
<LSN-low> is the oldest log file in the range. Up to 4 bytes padded with leading
zeroes.
The valid range of a 4-byte integer is 0 to 4294967295.
The two components together comprise the Ingres LSN.
Example:
1206396546:43927
Oracle
<system change number>
Where: <system change number> is the Oracle SCN value.
Example: 6488359
81
Sybase
<time_high>.<time_low>.<page>.<row>
Where:
<time_high> and <time_low> comprise a sequence number representing the time
when the transaction was committed. It is stored in the header of each database log
page. <time_high> is 2-bytes and <time_low> is 4-bytes, both without leading
zeroes.
<page> is the data page, without leading zeroes.
<row> is the row, without leading zeroes.
The valid range of a 2-byte integer for a timestamp-high is 0 - 65535. For a 4-byte
integer for a timestamp-low, it is: 0 - 4294967295.
Example: 12245.67330.12.345
DB2 LUW
<LSN>
Where:
<LSN> is the decimal-based DB2 log sequence number, without leading zeroes.
Example: 2008-01-01 10:30:00.1234567890
DB2 z/OS
<RBA>
Where:
<RBA> is a 6-byte relative byte address of the commit record within the transaction
log.
Example: 1274565892
SQL Server
Can be any of these, depending on how the database returns it:
- Colon separated hex string (8:8:4) padded with leading zeroes and 0X prefix
- Colon separated decimal string (10:10:5) padded with leading zeroes
- Colon separated hex string with 0X prefix and without leading zeroes
- Colon separated decimal string without leading zeroes
- Decimal string
Where the first value is the virtual log file number, the second is the segment number
within the virtual log, and the third is the entry number.
Examples:
0X00000d7e:0000036b:01bd
0000003454:0000000875:00445
0Xd7e:36b:1bd
3454:875:445
3454000000087500445
c-tree
<log number>.<byte offset>
Where:
<log number> is the 10-digit decimal number of the c-tree log file, left padded with
zeroes.
Oracle GoldenGate Fundamentals Student Guide
<byte offset> is the 10-digit decimal relative byte position from the beginning of the
file (0 based), left padded with zeroes.
Example: 0000000068.0000004682
SQL/MX
<sequence number>.<RBA>
Where:
<sequence number> is the 6-digit decimal NonStop TMF audit trail sequence
number, left padded with zeroes.
<RBA> is the 10-digit decimal relative byte address within that file, left padded with
zeroes.
Together these specify the location in the TMF Master Audit Trail (MAT).
Example: 000042.0000068242
Teradata
<sequence ID>
Where:
<sequence ID> is the generic VAM fixed-length printable sequence ID.
Example: 0x0800000000000000D700000021
Record Header
Use the Logdump utility to examine the record header. Here is a layout of the header
record:
Hdr-Ind: Always E, indicating that the record was created by the Extract process.
UndoFlag: (NonStop) Normally, UndoFlag is set to zero, but if the record is the
backout of a previously successful operation, then UndoFlag will be set to 1.
RecLength: The length, in bytes, of the record buffer.
IOType: The type of operation represented by the record.
83
TransInd: The place of the record within the current transaction. Values are 0 for the
first record in transaction; 1 for neither first nor last record in transaction; 2 for the
last record in the transaction; and 3 for the only record in the transaction.
SyskeyLen: (NonStop) The length of the system key (4 or 8 bytes) if the source is a
NonStop file and has a system key.
AuditRBA: The relative byte address of the commit record.
Continued: (Windows and UNIX) Identifies (Y/N) whether or not the record is a
segment of a larger piece of data that is too large to fit within one record, such as
LOBs.
Partition: Depends on the record type and is used for NonStop records. In the case of
BulkIO operations, Partition indicates the number of the source partition on which
the bulk operation was performed. For other non-bulk NonStop operations, the value
can be either 0 or 4. A value of 4 indicates that the data is in FieldComp format.
BeforeAfter: Identifies whether the record is a before (B) or after (A) image of an
update operation.
OrigNode: (NonStop) The node number of the system where the data was extracted.
FormatType: Identifies whether the data was read from the transaction log (R) or
fetched from the database (F).
AuditPos: Identifies the position of the Extract process in the transaction log.
RecCount: (Windows and UNIX) Used to reassemble LOB data when it must be
split into 2K chunks to be written to the GoldenGate file.
Oracle GoldenGate Fundamentals Student Guide
Alternative Formats
FORMATASCII
FORMATSQL
FORMATXML
FORMATASCII, BCP
FORMATASCII, SQLLOADER
85
Alternative Formats: FORMATASCII (cont’d)
Default output
Without options, FORMATASCII generates records in the following format.
Line 1, the following tab-delimited list:
• The operation-type indicator: I, D, U, V (insert, delete, update, compressed
update).
• A before or after image indicator: B or A.
• The table name.
• A column name, column value, column name, column value, and so forth.
• A newline character (starts a new line).
Line 2, the following tab-delimited begin-transaction record:
• The begin transaction indicator, B.
• The timestamp at which the transaction committed.
• The sequence number of the commit in the transaction log.
• The relative byte address (RBA) of the commit record within the transaction log.
Line 3, the following tab-delimited commit record:
• The commit character C.
• A newline character.
Oracle GoldenGate Fundamentals Student Guide
FORMATASCII
[, BCP]
[, COLHDRS ]
[, DATE | TIME | TS ]
[, DELIMITER <delimiter> ]
[, EXTRACOLS ]
[, NAMES ]
[, NONAMES ]
[, NOHDRFIELDS ]
[, NOQUOTE ]
[, NOTRANSSTMTS ]
[, NULLISPACE ]
[, PLACEHOLDERS ]
[, SQLLOADER ]
BCP
Formats the output for compatibility with SQL Server’s BCP (Bulk Copy Program) or
SSIS high-speed load utility.
COLHDRS
Outputs the table’s column names before the data. COLHDRS takes effect only when
the extract is directly from the table.
DELIMITER
Use an alternative field delimiter (the default is tab).
EXTRACOLS
Includes placeholders for additional columns at the end of each record. Use this when
a target table has more columns than the source.
NAMES | NONAMES
Excludes column names from the output. For compressed records, column names are
included unless you also specify PLACEHOLDERS.
NOHDRFIELDS
Suppresses output of transaction information, the operation type character, the
before or after image indicator, and the file or table name.
NOQUOTE
87
Excludes quotation marks from character-type data (default is surrounded by single-
quotes).
NOTRANSSTMTS
Excludes transaction information.
NULLISPACE
Outputs NULL fields as empty fields. The default is to output null fields as the word
NULL.
PLACEHOLDERS
Outputs a placeholder for missing fields or columns. For example, if The second and
fourth columns are missing in a four column table, the data might look like:
‘ABC’,,123,,
SQLLOADER
Generates a file compatible with the Oracle SQL*Loader high-speed data load utility.
SQLLOADER produces a fixed-length, ASCII-formatted file.
Note: The last record returns column names for the CUSTNAME and BALANCE columns
because the record is a compressed update and PLACEHOLDERS was not used.
Oracle GoldenGate Fundamentals Student Guide
Scope of a transaction
Every record in a transaction is contained between the begin and commit indicators.
Each combination of commit timestamp and RBA is unique.
FORMATSQL
[, NONAMES ]
[, NOPKUPDATES ]
[, ORACLE ]
NONAMES
Omits column names for insert operations, because inserts contain all column names.
This option conserves file size.
NOPKUPDATES
Converts UPDATE operations that affect columns in the target primary key to a
DELETE followed by an INSERT. By default (without NOPKUPDATES), the output
is a standard UPDATE operation.
ORACLE
89
Formats records for compatibility with Oracle databases by converting date and time
columns to a format accepted by SQL*Plus. For example: TO_DATE('1996-05-
01','YYYY-MM-DD'
B,2008-11-11:13:48:49.000000,1226440129,155,
DELETE FROM TEST.TCUSTMER WHERE CUST_CODE='JANE';
DELETE FROM TEST.TCUSTMER WHERE CUST_CODE='WILL';
DELETE FROM TEST.TCUSTORD WHERE CUST_CODE='JANE' AND
ORDER_DATE='1995-11-11:13:52:00' AND PRODUCT_CODE='PLANE' AND
ORDER_ID='256';
DELETE FROM TEST.TCUSTORD WHERE CUST_CODE='WILL' AND
ORDER_DATE='1994-09-30:15:33:00' AND PRODUCT_CODE='CAR' AND
ORDER_ID='144';
INSERT INTO TEST.TCUSTMER (CUST_CODE,NAME,CITY,STATE) VALUES
('WILL','BG SOFTWARE CO.','SEATTLE','WA');
INSERT INTO TEST.TCUSTMER (CUST_CODE,NAME,CITY,STATE) VALUES
('JANE','ROCKY FLYER INC.','DENVER','CO');
INSERT INTO TEST.TCUSTORD
(CUST_CODE,ORDER_DATE,PRODUCT_CODE,ORDER_ID,PRODUCT_PRICE,PRODUC
T_AMOUNT,TRANSACTION_ID) VALUES ('WILL','1994-09-
30:15:33:00','CAR','144',17520.00,3,'100');
INSERT INTO TEST.TCUSTORD
(CUST_CODE,ORDER_DATE,PRODUCT_CODE,ORDER_ID,PRODUCT_PRICE,PRODUC
T_AMOUNT,TRANSACTION_ID) VALUES ('JANE','1995-11-
11:13:52:00','PLANE','256',133300.00,1,'100');
C,
Syntax
FORMATXML
[, INLINEPROPERTIES | NOINLINEPROPERTIES ]
[, TRANS | NOTRANS ]
INLINEPROPERTIES | NOINLINEPROPERTIES
Controls whether or not properties are included within the XML tag or written
separately. INLINEPROPERTIES is the default.
TRANS | NOTRANS
Controls whether or not transaction boundaries and commit timestamps should be
included in the XMLoutput. TRANS is the default.
Oracle GoldenGate Fundamentals Student Guide
91
Viewing in Logdump
Logdump
Logdump overview
Logdump provides access to GoldenGate Trails, unstructured file with a variable
record length. Each record in the trail contains a header, known as the GGS Header
(unless the NOHEADERS Extract parameter was used), an optional user token area, and
the data area.
Shell> logdump
To get help:
Opening a trail
The syntax to open a trail is:
OPEN <file_name>
Where: <file_name> is either the relative name or fully qualified name of the file,
including the file sequence number.
93
Logdump – Setting up a View
usertoken
By default, the name of the token and its length are displayed. Use the USERTOKEN
DETAIL option to show the actual token data.
reclen
Controls how much of the record data is displayed. You can use RECLEN to control the
amount of scrolling that must be done when records are large, while still showing
enough data to evaluate the record. Data beyond the specified length is truncated.
Oracle GoldenGate Fundamentals Student Guide
95
Logdump – Viewing Trail Records
Operation
type and
time the
record was
written
Source
table
Image type:
could be a
before or Record Length of record and Record data,
after image Column data, its RBA position in the in ASCII
information in hex trail file
GoldenGate trail files are unstructured. The GoldenGate record header provides
metadata of the data contained in the record and includes the following information.
The operation type, such as an insert, update, or delete
The transaction indicator (TransInd): 00 beginning, 01 middle, 02 end or 03 whole of
transaction
The before or after indicator for updates
Transaction information, such as the transaction group and commit timestamp
The time that the change was written to the GoldenGate file
The type of database operation
The length of the record
The relative byte address within the GoldenGate file
The table name
The change data is shown in hex and ASCII format. If before images are configured to
be captured, for example to enable a procedure to compare before values in the WHERE
clause, then a before image also would appear in the record.
Oracle GoldenGate Fundamentals Student Guide
Logdump> count
Average of 25 Transactions
Bytes/Trans ..... 22661
Records/Trans ... 193
Files/Trans ..... 8
COUNT output
The basic output, without options, shows the following:
- The RBA where the count began
- The number of records in the file
- The total data bytes and average bytes per record
- Information about the operation types
- Information about the transactions
TCUSTMER
Total Data Bytes 10562
Avg Bytes/Record 55
Delete 300
Insert 1578
FieldComp 12
Before Images 300
After Images 1590
TCUSTORD
Total Data Bytes 229178
Avg Bytes/Record 78
Delete 600
Insert 2324
Field Comp 14
Before Images 600
After Images 2338
COUNT Syntax:
COUNT
[, DETAIL]
[, END[TIME] <time_string>]
[, FILE <specification>]
[, INT[ERVAL] <minutes>]
97
[, LOG] <wildcard>]
[, START[TIME] <time_string>]
COUNT options allow you to show table detail without using the DETAIL command
first, set a start and end time for the count, filter the count for a table, data file,
trail file, or extract file, and specify a time interval for counts.
• FILENAME specifies a SQL table, NonStop data file or group of tables/files using
a wildcard.
• You can string multiple FILTER commands together, separating each one with a
semi-colon, as in:
FILTER INCLUDE FILENAME fin.act*; FILTER RECTYPE 5;
FILTER MATCH ALL
• To avoid unexpected results, avoid stringing filter options together with one
FILTER command. For example, the following would be incorrect:
FILTER INCLUDE FILENAME fin.act*; RECTYPE 5; MATCH ALL
• Without arguments, FILTER displays the current filter status (ON or OFF) and any
filter criteria that are in effect.
Oracle GoldenGate Fundamentals Student Guide
99
Logdump – Saving Records to a New Trail
Use SAVE to write a subset of the records to a new trail or extract file. By saving a
subset to a new file, you can work with a smaller file. Saving to another file also
enables you to extract valid records that can be processed by GoldenGate, while
excluding records that may be causing errors.
Options allow you to overwrite an existing file, save a specified number of records or
bytes, suppress comments, use the old or new trail format, set the transaction
indicator (first, middle, end, only), and clean out an existing file before writing new
data to it.
Syntax
SAVE <file_name> [!] {<n> records | <n> bytes}[NOCOMMENT]
[OLDFORMAT | NEWFORMAT]
[TRANSIND <indicator>]
[TRUNCATE]
Use LOG to start and stop the logging of Logdump sessions. When enabled, logging
remains in effect for all sessions of Logdump until disabled with the LOG STOP
command. Without arguments, LOG displays the status of logging (ON or OFF). An
alias for LOG is OUT.
Syntax
LOG {<file_name> | STOP}
Logdump Commands
Purpose Examples
Viewing information:
COUNT Displays record count information
FILES Displays names of files in the current subvolume
ENV Displays current Logdump settings
NOTIFY Displays the number of records scanned, trail position, and record
timestamp.
SHOW Displays based on the following options:
<OPEN>: open files
<ENV>: current Logdump environment
<RECTYPE>: list of GoldenGate record types
<FILTER>: current filter settings
TIME Displays the current time in local and GMT formats
101
Selecting data and records:
DUMP Displays the specified number of bytes of data from the current
position in the file
FILTER Filters the record display
NEXT Displays the next or <nn> record in the file
SCANFORENDTRANSACTION Finds a record that is the last or only record in a
transaction and displays the next record
SCANFORHEADER Finds the start of the next record header
SCANFORRBA Finds a specified relative byte address
SCANFORTIME Finds the next record with a specified timestamp
SCANFORTYPE Finds the next record of a specified type
SKIP Skips a specified number of records
Making conversions:
COMPUTETIMESTAMP Converts a datetime string to a Julian timestamp
CTIME Converts a C timestamp to an ASCII timestamp
DECRYPT Decrypts data before displaying it
ENCRYPT Encrypts file data
INTERPRETINTERVAL Displays a 64-bit Julian interval as: days-hh:mm:ss:ms:us
INTERPRETTIMESTAMP Displays a 64-bit Julian timestamp in ASCII format
Miscellaneous:
EXIT Exits Logdump
FC Edits a previous command
HELP Shows syntax for Logdump commands
Oracle GoldenGate Fundamentals Student Guide
103
Reversing the Trail Sequence
Reverse - Overview
SPECIALRUN indicates a one-time batch process that will run from BEGIN date and
time until END date and time.
NOCOMPRESSDELETES causes Extract to send all column data to the output, instead of
only the primary key. Enables deletes to be converted back to inserts.
END RUNTIME causes the Extract or Replicat to terminate when it reaches process
startup time.
105
Reverse – Overall Process
1. What is a trail?
2. What formats are Extract trails and files written in?
3. What GoldenGate utility allows you to view trail contents
1. A trail is a series of files on disk where GoldenGate stores data for further processing.
2. GoldenGate trail format, ASCII, SQL, XML
3. Logdump
Oracle GoldenGate Fundamentals Student Guide
Parameters
Parameters - Overview
GLOBALS parameters
Manager parameters
Extract parameters
Replicat parameters
107
GLOBALS versus Process Parameters
Process-wide parameters
These apply to all tables specified in the parameter file for the process. A global
parameter can appear anywhere in the parameter file, and it should only be listed in
the file once. When listed more than once, only the last instance of the parameter is
active. All other instances are ignored.
Table-specific parameters
These control processing for tables specified with a TABLE or MAP statement. Table-
specific parameters enable you to designate one set of processing rules for some
tables, while designating other rules for other tables. Table-specific parameters take
effect in the order that each parameter is listed in the parameter file. There are two
implementations for file-specific parameters:
Toggling the parameter on and off around one or more TABLE or MAP statements.
Adding the parameter within MAP statement so it applies only to that table or file.
Oracle GoldenGate Fundamentals Student Guide
GLOBALS Parameters
GLOBALS Parameters
109
Manager Parameters
PORT 7809
DYNAMICPORTLIST 9001–9100
AUTOSTART ER *
AUTORESTART EXTRACT *, WAITMINUTES 2, RETRIES 5
LAGREPORTHOURS 1
LAGINFOMINUTES 3
LAGCRITICALMINUTES 5
PURGEOLDEXTRACTS /ggs/dirdat/rt*, USECHECKPOINTS
PORT establishes the TCP/IP port number on which Manager listens for requests.
DYNAMICPORTLIST specifies the ports that Manager can dynamically allocate.
AUTOSTART specifies processes that are to be automatically started when Manager
starts.
AUTORESTART specifies processes to be restarted after abnormal termination.
LAGREPORTHOURS sets the interval in hours at which Manager checks lag for Extract
and Replicat processing. Alternately can be set in minutes.
LAGINFOMINUTES specifies the interval at which Extract and Replicat will send a
informational message to the event log. Alternately can be set in seconds or hours.
LAGCRITICALMINUTES specifies the interval at which Extract and Replicat will send a
critical message to the event log. Alternately can be set in seconds or hours.
PURGEOLDEXTRACTS purges GoldenGate trails that are no longer needed based on
the option settings.
Oracle GoldenGate Fundamentals Student Guide
Manager Parameters
Purpose Examples
General COMMENT
Process
management AUTOSTART, AUTORESTART
Maintenance PURGEOLDEXTRACTS
Port Management
DYNAMICPORTLIST Specifies the ports that Manager can dynamically allocate.
DYNAMICPORTREASSIGNDELAY Specifies a time to wait before reassigning a port.
PORT Establishes the TCP/IP port number on which Manager
listens for requests.
Process Management
AUTOSTART Specifies processes to be started when Manager starts.
AUTORESTART Specifies processes to be restarted by Manager after a
failure.
BOOTDELAYMINUTES Determines how long after system boot time MANAGER
delays until performing main processing activities.
USETHREADS | NOUSETHREADS To perform background tasks, USETHREADS
causes a thread to be used on Windows systems or a child
process to be spawned on a UNIX system.
Event Management
DOWNCRITICAL Reports processes that stopped gracefully or abnormally.
DOWNREPORT Controls the frequency of reporting stopped processes.
LAGCRITICAL Specifies a lag time threshold at which a “critical” message
is reported to the event log.
LAGINFO Specifies a lag time threshold at which an informational
message is reported to the event log.
LAGREPORT Sets an interval for reporting lag time to the event log.
UPREPORT Determines how often process heartbeat messages are
reported.
Database login
SOURCEDB Specifies a data source name as part of the login
information.
111
USERID Provides login information for Manager when it needs to
access the database.
Maintenance
CHECKMINUTES Determines how often Manager cycles through
maintenance activities.
PURGEDDLHISTORY Purges rows from the Oracle DDL history table when they are no
longer needed.
PURGEMARKERHISTORY Purges Oracle marker table rows that are no longer needed.
PURGEOLDEXTRACTS Purges trail data that is no longer needed.
PURGEOLDTASKS Purges Extract and Replicat tasks after a specified period
of time.
STARTUPVALIDATIONDELAY[CSECS] Sets the delay time after which Manager
checks the status of a process it started.
Oracle GoldenGate Fundamentals Student Guide
Extract Parameters
113
Sample Extract Parameter File
EXTRACT ODS
USERID GoldenUser, PASSWORD password
RMTHOST manhattan, MGRPORT 7809
RMTTRAIL /ggs/dirdat/rt
TABLE SALES.ORDERS;
TABLE SALES.INVENTORY;
USERID and PASSWORD supply database credentials. (SOURCEDB is not required for
Oracle.)
RMTHOST specifies the target system while the MGRPORT option specifies the port
where Manager is running.
RMTTRAIL specifies the GoldenGate trail on the target system.
TABLE specifies a source table for which activity will be extracted.
Extract Parameters
Purpose Examples
General SETENV, GETENV, OBEY
Processing method BEGIN, END, PASSTHRU
Database login SOURCEDB, USERID
Selecting and
mapping data IGNOREINSERTS, GETUPDATEBEFORES, TABLE
Routing data EXTTRAIL, RMTHOST, RMTTRAIL
Formatting data FORMATASCII, FORMATSQL, FORMATXML, NOHEADERS
Custom processing CUSEREXIT, INCLUDE, MACRO, SQLEXEC
Reporting REPORT, REPORTCOUNT, STATOPTIONS
Error handling DISCARDFILE, DDLERROR
Tuning ALLOCFILES, CHECKPOINTSECS, DBOPTIONS
Maintenance PURGEOLDEXTRACTS, REPORTROLLOVER
Security ENCRYPTTRAIL, DECRYPTTRAIL
General
CHECKPARAMS Performs parameter check, then terminates.
COMMENT Allows insertion of comments in parameter file.
ETOLDFORMAT Generates trails in a format that is compatible with Replicat
versions prior to GoldenGate version 6.0.
GETENV Retrieves variables that were set with the SETENV parameter.
Oracle GoldenGate Fundamentals Student Guide
Processing Method
BEGIN Specifies when to begin a processing run.
DSOPTIONS Specifies Extract processing options when a Vendor Access
Module (VAM) is being used.
END Specifies when to end a processing run. Not required unless
SPECIALRUN is used. Online processing is implied if END is in
the future or unspecified.
EXTRACT Defines an Extract group as an online process.
GETAPPLOPS | IGNOREAPPLOPS
Controls whether or not operations from all processes except
Replicat are captured.
GETREPLICATES | IGNOREREPLICATES
Controls whether or not replicated operations are captured by an
Extract on the same system.
PASSTHRU | NOPASSTHRU Controls whether tables will be processed by a
data-pump Extract in pass-through mode (data definitions not
required).
RMTTASK Creates a processing task on the target system.
SOURCEISTABLE Extracts entire records from source tables.
SPECIALRUN Used for one-time processing tasks that do not checkpoint from
run to run.
VAM Indicates a Vendor Access Module (VAM) is being used to provide
transactional data to Extract.
Database Login
SOURCEDB Specifies the data source as part of the login information.
USERID Specifies database connection information.
115
FETCHOPTIONS Controls certain aspects of how GoldenGate fetches data from the
database, such as what to do when a row is missing. Several of
the options are specific to Oracle databases.
GETDELETES | IGNOREDELETES
Controls the extraction of delete operations.
GETINSERTS | IGNOREINSERTS
Controls the extraction of insert operations.
GETTRUNCATES | IGNORETRUNCATES
Controls the extraction of truncate statements.
GETUPDATEAFTERS | IGNOREUPDATEAFTERS
Controls the extraction of update after images.
GETUPDATEBEFORES | IGNOREUPDATEBEFORES
Controls the extraction of update before images.
GETUPDATES | IGNOREUPDATES
Controls the extraction of update operations.
REPLACEBADCHAR Replaces invalid character values with another value.
SEQUENCE (Oracle) Specifies sequences for synchronization.
TABLE Specifies tables for synchronization. Controls column mapping and
conversion.
TABLEEXCLUDE Excludes tables from the extraction process.
TARGETDEFS Specifies a file containing target table definitions for target
databases that reside on the NonStop platform.
TRIMSPACES | NOTRIMSPACES
Controls whether or not trailing spaces are removed when
mapping CHAR to VARCHAR columns.
WILDCARDRESOLVE Defines rules for processing wildcard table specifications in
a TABLE statement.
Route Data
EXTFILE Specifies an extract file to which extracted data is written on the
local system.
EXTTRAIL Specifies a trail to which extracted data is written on the local
system.
RMTHOST Specifies the target system and Manager port number for a remote
file or trail.
RMTFILE Specifies an extract file to which extracted data is written on the
target system.
RMTTRAIL Specifies a trail to which extracted data is written on a remote
system.
Formatting Data
FORMATASCII Formats extracted data in external ASCII format.
FORMATSQL Formats extracted data into equivalent SQL statements.
FORMATXML Formats extracted data into equivalent XML syntax.
INPUTASCII Accepts a decimal value from 1 to 127, representing an ASCII
character.
NOHEADERS Prevents record headers from being written to the trail.
Custom Processing
CUSEREXIT Invokes customized user exit routines at specified points during
processing.
Oracle GoldenGate Fundamentals Student Guide
Reporting
CMDTRACE Displays macro expansion steps in the Extract report file.
LIST | NOLIST Suppresses or displays the listing of macros in the Extract report
file.
REPORT Schedules a statistical report.
STATOPTIONS Specifies information to include in statistical displays.
REPORTCOUNT Reports the number of records processed.
TLTRACE Traces transaction log extraction activity.
TRACE/TRACE2 Shows Extract processing information to assist in revealing
processing bottlenecks.
WARNLONGTRANS Defines a long-running transaction and controls the frequency of
checking for and reporting them.
Error Handling
DDLERROR Controls error handling for DDL extraction.
DISCARDFILE Contains records that could not be processed.
Tuning
ALLOCFILES Used to control the incremental amount of memory structured
allocated once the initial allocation specified by NUMFILES
parameter is reached. Defaults to 500.
CACHEMGR Controls the virtual memory cache manager. You can set the
CACHEDIRECTORY <path> option, but other options are best
defaulted, unless under direction from GoldenGate Support.
CHECKPOINTSECS Controls how often Extract writes a checkpoint.
DBOPTIONS Specifies database options.
DDLOPTIONS Specifies DDL processing options.
DYNAMICRESOLUTION | NODYNAMICRESOLUTION
Suppresses the metadata lookup for a table until Extract
encounters transactional data for it. Makes Extract start faster
when there is a large number of tables specified for capture.
EOFDELAY | EOFDELAYCSECS
Determines how long Extract delays before searching for more
data to process.
FLUSHSECS | FLUSHCSECS
Determines the amount of time that record data remains buffered
before being written to a trail.
FUNCTIONSTACKSIZE Controls the number of arguments permitted in a
parameter clause.
GROUPTRANSOPS Controls the number of records that are sent to the trail in one
batch.
117
MAXFETCHSTATEMENTS Controls the maximum number of prepared queries that
Extract can use to fetch data from the database.
NUMFILES Used to control the initial amount of memory structured allocate
for the storage of tables structures. Defaults to 1000.
RMTHOSTOPTIONS Specifies connection attributes other than host information for a
TCP/IP connection used by a passive Extract group.
THREADOPTIONS Controls aspects of the way that Extract operates in an Oracle
Real Application Cluster environment. Supports Oracle.
TRANLOGOPTIONS Supplies log-processing options.
TRANSMEMORY (DB2 z/OS and NonStop SQL/MX only)
Controls the amount of memory and temporary disk space
available for caching uncommitted transaction data.
Maintenance
DISCARDROLLOVER Controls when to create a new discard file.
PURGEOLDEXTRACTS Purges trails after they have been consumed.
REPORTROLLOVER Specifies when to create new report files.
ROLLOVER Specifies the way that trail files are aged.
Security
DECRYPTTRAIL Decrypts data in a trail or extract file.
ENCRYPTTRAIL | NOENCRYPTTRAIL Controls encryption of data in a trail or
extract file.
There must be a TABLE statement for each source table from which you will be
extracting data. Use wildcards to specify multiple tables with one TABLE statement,
for example: TABLE acct*;
TARGET <table spec> Specifies a target table to which the source table will be
mapped.
TARGET is required for TABLE statements when Extract must refer to a target
definitions file (specified with the TARGETDEFS parameter) to perform conversions,
and when the COLMAP option is used. Otherwise, it can be omitted. Using TARGET
identifies the extracted data by the target structure, rather than that of the source, to
reflect the structure of the record that is reflected in the definitions file or the column
map.
Column mapping and conversion can be performed on the target system to prevent
added overhead on the source system. Replication from a Windows or UNIX system to
a NonStop system require these functions to be performed on the source, however.
In addition, it may be preferrable to perform the mapping and conversion on the
source when there are multiple sources and one target. In that case, it could be easier
to manage one target definitions file that is transferred to each source, rather than
having to manage source definitions for each source database that must be
transferred to the target, especially when there are frequent application changes that
require new files to be generated.
FETCHCOLS | FETCHCOLSEXCEPT
Enables the fetching of column values from the source database when the values are
not in the transaction record.
119
FETCHMODCOLS | FETCHMODCOLSEXCEPT
Forces column values to be fetched from the database when the columns are present
in the transaction log.
FILTER Selects records based on a numeric value. FILTER provides more flexibility
than WHERE.
TRIMSPACES | NOTRIMSPACES
Controls whether trailing spaces are trimmed or not when mapping CHAR to
VARCHAR columns.
TRANLOGOPTIONS ARCHIVEDLOGONLY
Causes Extract to read from the archived logs exclusively.
Oracle GoldenGate Fundamentals Student Guide
121
Replicat Parameters
Error handling
The Replicat process runs on the target system, reads the extracted data, and replicates
it to the target tables. Replicat reads extract and log files sequentially, and processes the
inserts, updates and deletes specified by selection parameters. Replicat reads extracted
data in blocks to maximize throughput.
Optionally, you can filter out the rows you do not wish to deliver, as well as perform data
transformation prior to replicating the data. Parameters control the way Replicat
processes – how it maps data, uses functions, and handles errors.
You can configure multiple Replicat processes for increased throughput and identify
each by a different group name.
Oracle GoldenGate Fundamentals Student Guide
Replicat supports a high volume of data replication activity. As a result, network activity
is block-based rather than a record-at-a-time. SQL operations used to replicate
operations are compiled once and execute many times, resulting in virtually the same
performance as pre-compiled operations.
Replicat preserves the boundaries of each transaction while processing , but small
transactions can be grouped into larger transactions to improve performance. Like
Extract, Replicat uses checkpoints so that after a graceful stop or a failure, processing
can be restarted without repetition or loss of continuity.
REPLICAT SALESRPT
USERID ggsuser, PASSWORD ggspass
ASSUMETARGETDEFS
DISCARDFILE /ggs/dirrpt/SALESRPT.dsc, APPEND
MAP HR.STUDENT, TARGET HR.STUDENT
WHERE (STUDENT_NUMBER < 400000);
MAP HR.CODES, TARGET HR.CODES;
MAP SALES.ORDERS, TARGET SALES.ORDERS,
WHERE (STATE = “CA” AND OFFICE = “LA”);
REPLICAT names the group linking together the process, checkpoints, and log files.
USERID, PASSWORD provide credentials to access the database.
ASSUMETARGETDEFS specifies that the table layout is the identical on the source and
target.
123
DISCARDFILE identifies the file to receive records that cannot be processed. Records
will be appended or the file will be purged at the beginning of the run depending on the
options.
MAP links the source tables to the target tables and applies mapping, selection, error
handling, and data transformation depending on options.
Replicat Parameters
Purpose Examples
General SETENV, GETENV, OBEY
Processing method BEGIN, END, SPECIALRUN
Database login SOURCEDB, USERID
Selecting, converting COLMATCH, IGNOREUPDATES, MAP, SOURCEDEFS,
and mapping data ASSUMETARGETDEFS
Routing data EXTFILE, EXTTRAIL
Custom processing CUSEREXIT, DEFERAPPLYINTERVAL, INCLUDE,
MACRO, SQLEXEC
Reporting REPORT, REPORTCOUNT, STATOPTIONS
Error handling DISCARDFILE, OVERRIDEDUPS, HANDLECOLLISIONS
Tuning ALLOCFILES, BATCHSQL, GROUPTRANSOPS,
DBOPTIONS
Maintenance PURGEOLDEXTRACTS, REPORTROLLOVER
Security DECRYPTTRAIL
General
CHECKPARAMS Performs parameter check, then terminates.
COMMENT Allows insertion of comments in parameter file.
GETENV Retrieves a value for a variable set by SETENV.
OBEY Processes parameter statements contained in a different
parameter file.
SETENV Specifies a value for a UNIX environment variable from within the
GGSCI interface.
TRACETABLE | NOTRACETABLE
Defines a trace table in an Oracle database to which Replicat adds
a record whenever it updates the target database.
Processing Method
BEGIN Specifies when Replicat starts processing. Required when
SPECIALRUN is used.
BULKLOAD Loads data directly into the interface of Oracle’s SQL*Loader bulk-
load utility.
END Specifies when Replicat stops processing. Required when using
SPECIALRUN.
GENLOADFILES Generates run and control files that are compatible with bulk-load
utilities.
REPLICAT Links this run to an online Replicat process.
SPECIALRUN Used for one-time processing that does not require checkpointing
from run to run.
Oracle GoldenGate Fundamentals Student Guide
Database Login
TARGETDB Specifies the target database. Support SQL Server, DB2, Sybase,
and Informix.
USERID Specifies a database user ID and password for connecting to the
target database.
125
SPACESTONULL | NOSPACESTONULL
Controls whether or not a target column containing only spaces is
converted to NULL on Oracle.
TABLE (Replicat) Specifies a table or tables for which event actions are to take place
when a row satisfies the given filter criteria.
TRIMSPACES | NOTRIMSPACES
Controls if trailing spaces are preserved or removed for character
or variable character columns.
UPDATEDELETES | NOUPDATEDELETES
Changes deletes to updates.
USEDATEPREFIX Prefixes data values for DATE data types with a DATE literal, as
required by Teradata databases.
USETIMEPREFIX Prefixes data values for TIME datatypes with a TIME literal, as
required by Teradata databases.
USETIMESTAMPPREFIX Prefixes data values for TIMESTAMP datatypes with a
TIMESTAMP literal, as required by Teradata databases.
Routing Data
EXTFILE Defines the name of an extract file on the local system that
contains data to be replicated. Used for one-time processing.
EXTTRAIL Defines a trail containing data to be replicated Used for one-time
processing.
Custom Processing
CUSEREXIT Invokes customized user exit routines at specified points during
processing.
DEFERAPPLYINTERVAL Specifies a length of time for Replicat to wait before applying
replicated operations to the target database.
INCLUDE References a macro library in a parameter file.
MACRO Defines a GoldenGate macro.
SQLEXEC Executes a stored procedure, query or database command during
Replicat processing.
Reporting
CMDTRACE Displays macro expansion steps in the report file.
LIST | NOLIST Suppresses or displays the listings of the parameter file to the
report file.
REPORT Schedules a statistical report at a specified date or time.
REPORTCOUNT Reports the number or records processed.
SHOWSYNTAX Causes Replicat to print its SQL statements to the report file.
STATOPTIONS Specifies information to include in statistical displays.
TRACE | TRACE2 Shows Replicat processing information to assist in revealing
processing bottlenecks.
Error Handling
CHECKSEQUENCEVALUE | NOCHECKSEQUENCEVALUE
(Oracle) Controls whether or not Replicat verifies that a target
sequence value is higher than the one on the source and corrects
any disparity that it finds.
DDLERROR Controls error handling for DDL replication.
DISCARDFILE Contains records that could not be processed.
Oracle GoldenGate Fundamentals Student Guide
HANDLECOLLISIONS | NOHANDLECOLLISIONS
Handles errors for duplicate and missing records. Reconciles the
results of changes made to the target database by an initial load
process with those applied by a change-synchronization group.
HANDLETPKUPDATE Prevents constraint errors associated with replicating
transient primary key updates.
OVERRIDEDUPS | NOOVERRIDEDUPS
Overlays a replicated insert record onto an existing target record
whenever a duplicate-record error occurs.
RESTARTCOLLISIONS | NORESTARTCOLLISIONS
Controls whether or not Replicat applies HANDLECOLLISIONS
logic after GoldenGate has exited because of a conflict.
REPERROR Determines how Replicat responds to database errors.
REPFETCHEDCOLOPTIONS Determines how Replicat responds to operations
for which a fetch from the source database was required.
SQLDUPERR When OVERRIDEDUPS is on, specifies the database error
number that indicates a duplicate-record error.
WARNRATE Determines how often database errors are reported.
Tuning
ALLOCFILES Used to control the incremental amount of memory structured
allocated once the initial allocation specified by NUMFILES
parameter is reached. Defaults to 500.
BATCHSQL Increases the throughput of Replicat processing by arranging
similar SQL statements into arrays and applying them at an
accelerated rate.
CHECKPOINTSECS Controls how often Replicat writes a checkpoint when checkpoints
are not being generated as a result of transaction commits.
DBOPTIONS Specifies database options.
DDLOPTIONS Specifies DDL processing options.
DYNAMICRESOLUTION | NODYNAMICRESOLUTION
Makes Replicate start faster when there is a large number of
tables specified for synchronization.
DYNSQL | NODYNSQL Causes Replicat to use literal SQL statements rather than
a compile-once, execute-many strategy.
EOFDELAY | EOFDELAYSECS Determines how many seconds Replicat delays before
looking for more data to process.
FUNCTIONSTACKSIZE Controls the number of arguments permitted when using
GoldenGate column-conversion functions.
GROUPTRANSOPS Groups multiple transactions into larger transactions.
INSERTAPPEND | NOINSERTAPPEND
Controls whether or not Replicat uses an APPEND hint when
applying INSERT operations to Oracle target tables.
LOBMEMORY Controls the memory and disk space allocated to LOB
transactions.
MAXDISCARDRECS Limits the number of discarded records reported to the discard file.
MAXSQLSTATEMENTS Controls the number of prepared SQL statements that can
be used by Replicat.
MAXTRANSOPS Divides large transactions into smaller ones.
NUMFILES Used to control the initial amount of memory structured allocate for
the storage of tables structures. Defaults to 1000.
127
RETRYDELAY Specifies the delay between attempts to retry a failed SQL
operation.
TRANSACTIONTIMEOUT Specifies a time interval after which Replicat will commit its
open target transaction and roll back any incomplete source
transactions that it contains, saving them for when the entire
source transaction is ready to be applied.
TRANSMEMORY (DB2 z/OS and NonStop SQL/MX only)
Controls the amount of memory and temporary disk space
available for caching uncommitted transaction data.
WILDCARDRESOLVE Alters the rules by which wildcard specifications in a MAP
statement are resolved.
Maintenance
DISCARDROLLOVER Specifies when to create new discard files.
PURGEOLDEXTRACTS Purges GoldenGate trail files once consumed.
REPORTROLLOVER Specifies when to create new report files.
Security
DECRYPTTRAIL Decrypts data in a trail or extract file.
The MAP parameter establishes a relationship between one source and one target table.
Insert, update and delete records originating in the source table are replicated in the
target table. The first <table spec> is the source table.
With MAP, you can replicate particular subsets of data to the target table, for example,
WHERE (STATE = “CA”). In addition, MAP enables the user to map certain fields or
columns from the source record into the target record format (“column mapping”). You
can also include a FILTER command to evaluate data provided built-in functions for
more complex filtering criteria.
Oracle GoldenGate Fundamentals Student Guide
A table can appear in multiple maps as either source or target. For example, one might
replicate a “sales” file to either the “east sales” or “west sales” tables, depending on
some value or type of operation.
FILTER Selects records based on a numeric operator. FILTER provides more flexibility
than WHERE.
HANDLECOLLISIONS | NOHANDLECOLLISIONS
Reconciles the results of changes made to the target table by an initial load process with
those applied by a change-synchronization group.
REPERROR Controls how Replicat responds to errors when executing the MAP
statement.
TRIMSPACES | NOTRIMSPACES
Controls whether trailing spaces are trimmed or not when mapping CHAR to VARCHAR
columns.
129
Parameters – Discussion Points
Column mapping
Functions
SQLEXEC
Macros
User tokens
User exits
Oracle Sequences
TABLE selection
The MAP (Replicat) or TABLE (Extract) parameter can be used to select a table.
MAP sales.tcustord, TARGET sales.tord;
ROWS selection
131
The following WHERE option can be used with MAP or TABLE to select rows for “AUTO”
product type.
WHERE (PRODUCT_TYPE = “AUTO”);
OPERATIONS selection
The following can be used with MAP or TABLE to select rows with amounts greater
than zero only for update and delete operations.
FILTER (ON UPDATE, ON DELETE, amount > 0);
COLUMNS selection
The COLS and COLSEXCEPT options of the TABLE parameter allow selection of columns
as shown in the example below. Use COLS to select columns for extraction, and
use COLSEXCEPT to select all columns except those designated by COLSEXCEPT, for
example:
TABLE sales.tcustord, TARGET sales.tord, COLSEXCEPT
(facility_number);
Use the FILTER clause for more complex selections with built-in
functions
Arithmetic operators and floating-point data types are not supported by WHERE.
Only rows where the state column has a value of CA are returned.
WHERE (STATE = “CA”);
Only rows where the amount column has a value of NULL. Note
that if amount was not part of the update, the result is false.
WHERE (AMOUNT = @NULL);
Only rows where the amount was part of the operation and it has
value that is not null.
WHERE (AMOUNT @PRESENT AND AMOUNT <> @NULL);
133
Selection – FILTER Clause
When multiple filters are specified per a given TABLE or MAP entry, the filters are
executed until one fails or until all are passed. The failure of any filter results in a
failure for all filters.
Filters can be qualified with operation type so you can specify different filters for
inserts, updates and deletes.
The FILTER RAISEERROR option creates a user-defined error number if the filter clause
is true. In the following example error 9999 is generated when the BEFORE
timestamp is earlier than the CHECK timestamp. This also selects only update
operations.
Syntax
<column specification>
<field conversion function>
<ON INSERT | UPDATE | DELETE >
<IGNORE INSERT | UPDATE | DELETE>
<RAISEERROR <error number> >
FILTER ((PRODUCT_PRICE*PRODUCT_AMOUNT)>10000);
Why is the example above not constructed like the one below? The
filter below will fail!
FILTER(NAME = “JOE”);
135
Data Selection – RANGE Function
Syntax
@RANGE (<my range>, <total ranges>
[, <column> [, ...]])
Example
TABLE SALES.ACCOUNT, FILTER (@RANGE (1,3));
@RANGE helps divide workload into multiple, randomly distributed groups of data,
while guaranteeing that the same row will always be processed by the same process.
For example, @RANGE can be used to split the workload different key ranges for a
heavily accessed table into different Replicat processes.
The user specifies both a range that applies to the current process, and the total
number of ranges (generally the number of processes).
@RANGE computes a hash value of all the columns specified, or if no columns are
specified, the primary key columns of the source table. A remainder of the hash and
the total number of ranges is compared with the ownership range to determine
whether or not @RANGE determines true or false. Note that the total number of
ranges will be adjusted internally to optimize even distribution across the number of
ranges.
Restriction: @RANGE cannot be used if primary key updates are performed on the
database.
Oracle GoldenGate Fundamentals Student Guide
Replicat #1
MAP SALES.ACCOUNT,
TARGET SALES.ACCOUNT, FILTER (@RANGE (1,3));
Replicat #2
MAP SALES.ACCOUNT,
TARGET SALES.ACCOUNT, FILTER (@RANGE (2,3));
Replicat #3
MAP SALES.ACCOUNT,
TARGET SALES.ACCOUNT, FILTER (@RANGE (3,3));
The example above demonstrates three Replicat processes, with each Replicat group
processing one-third of the data in the GoldenGate trail based on the primary key.
RMTTRAIL /ggs/dirdat/bb
TABLE SALES.REP, FILTER (@RANGE (2,3));
TABLE SALES.ACCOUNT, FILTER (@RANGE (2,3,REP_ID));
RMTTRAIL /ggs/dirdat/cc
TABLE SALES.REP, FILTER (@RANGE (3,3));
TABLE SALES.ACCOUNT, FILTER (@RANGE (3,3,REP_ID));
Both of these MAP statements select the first of 2 ranges, so this Replicat is processing
the first range for both the ORDER table and the TRANSACTION table. Another
Replicat would include parameters to process the second of the ranges.
137
Column Mapping
Extract and Replicat provides the capability to transform data between two
dissimilarly structured database tables or files. These features are implemented with
the COLMAP clause in the TABLE and MAP Parameters.
Varchar and character fields can accept other character, varchar, group, and datetime
fields, or string literals enclosed in quotes. If the target character field is smaller than
that of the source, the character field is truncated on the right.
Oracle GoldenGate Fundamentals Student Guide
For COLMAP:
Default Mapping
When you specify USEDEFAULTS, the process maps columns in the source table to
columns in the target with the same name. This can be useful when the source and
target definitions are similar but not identical.
Note: If you set up global column mapping rules with COLMATCH parameters, you
can map columns with different names to each other using default mapping.
139
Column Mapping – Example
This example:
Moves the HR.CONTACT CUST_NAME column value to the HR.PHONE NAME
column
Concatenates the HR.CONTACT AREA_CODE, PH_PREFIX and PH_NUMBER
with quote and hypen literals to derive the PHONE_NUMBER column value
Automatically maps other HR.CONTACT columns to the HR.PHONE columns that
have the same name.
INSERTALLRECORDS
MAP SALES.ACCOUNT, TARGET REPORT.ACCTHISTORY,
COLMAP (USEDEFAULTS,
TRAN_TIME = @GETENV(“GGHEADER”,”COMMITTIMESTAMP”),
OP_TYPE = @GETENV(“GGHEADER”, “OPTYPE”),
BEFORE_AFTER_IND = @GETENV(“GGHEADER”,
“BEFOREAFTERINDICATOR”),
);
COLMAP uses the @GETENV function to get historical data from the GoldenGate
trail header – TRAN_TIME picks up the commit timestamp for the date of the
Oracle GoldenGate Fundamentals Student Guide
Functions
If you require more, you also have the ability to call your own
logic through user exits
Functions - Overview
141
Functions – Example
The example uses @STREXT to extract portions of a string field into 3 different
columns. It takes the first through the third characters from the source’s PHONE-NO to
populate the target’s AREA_CODE, characters 4 through 6 for the PHONE_PREFIX,
and 7 to 10 for the PHONE_NUMBER. The syntax for the @STREXT function is:
@STREXT (<column or literal string>, <begin position>, <end
position>)
Function Description
CASE Allows user to select a value depending on a series of value
tests
Syntax:
@IF (<conditional expression>,
<value if expression is non-zero>,
<value if expression is zero>)
143
Functions – Working with Dates
Function Description
Syntax
@DATE (“<output format>”,
“<input format>”, <source column>
[, “<input format>”, <source column>]
[,...])
1. What DATE expression would you use to convert year, month and
day columns into a date?
Formats that are supported for both input and output are:
CC century;
YYYY four-digit year;
YY two-digit year;
MMM alphanumeric month, such as APR;
MM numeric month;
DDD numeric day of the year (e.g. 001, 365)
DD numeric day of month;
HH Hour;
MI minute;
Oracle GoldenGate Fundamentals Student Guide
SS seconds;
FFFFFF fraction (up to microseconds);
DOW0 numeric day of the week (Sunday = 0);
DOW1 numeric day of the week (Sunday = 1);
DOWA alphanumeric day of the week, (e.g. SUN;
JUL Julian day;
JTSGMT and JTS Julian timestamp;
JTSLCT Julian timestamp that is already local time, or to keep local time when
converting to a Julian timestamp;
STRATUS Application timestamp;
CDATE C timestamp in seconds since the Epoch.
Calculating the century: When a two-digit year is supplied, but a four-digit year is
required in the output, the system can calculate the century (as 20 if the year is <
50), it can be hard-coded or the @IF function can be used to set a condition.
145
Functions – Working with Strings and Numbers
Function Description
COMPUTE Returns the result of an arithmetic expression
Function Description
Syntax
@STRCAT (<string1>, <string2> [, ...])
147
Discussion Point: STREXT Function
Syntax
@STREXT (<column or literal string>,
<begin position>, <end position>)
Functions - Other
Function Description
BINARY Keeps source data in its original binary format in the target when
source column is defined as character.
BINTOHEX Converts a binary string to a hexadecimal string.
GETENV Returns information on the GoldenGate environment, trail file
header, trail record header, last replicated operation and lag. Can
retrieve the commit timestamp in local time or GMT.
GETVAL Extracts parameters from a stored procedure as input to a FILTER or
COLMAP clause.
HEXTOBIN Converts a hexadecimal string to a binary string.
HIGHVAL, Emulate COBOL functions that allow you to set a numeric limit on
LOWVAL string or binary datatypes.
RANGE Divides workload into multiple groups of data, while ensuring the
same row will always be sent to the same process. Range uses a hash
against primary key or user defined columns.
TOKEN Maps environmental values that are stored in the user token area to
the target column.
Further information
GETENV and TOKEN are discussed further in User Tokens.
SQLEXEC
SQLEXEC - Overview
SQLEXEC advantages:
The SQLEXEC option enables both Extract and Replicat to communicate with the user’s
database, either via SQL queries or stored procedures. SQLEXEC can be used to
interface with a virtually unlimited set of the functionality supported by the underlying
database.
Extract and Replicat enables stored procedure capabilities to be leveraged for Oracle,
SQL Server and DB2. Tying together industry-standard stored procedure languages
with extraction and replication functions brings a familiar, powerful interface to virtually
unlimited functionality.
Stored procedures can also be used as an alternative method for inserting data into the
database, aggregating data, denormalizing or normalizing data, or any other function
that requires database operations as input. Extract and Replicat can support stored
procedures that only accept input, or procedures that produce output as well. Output
parameters can be captured and used in subsequent map and filter operations.
149
SQLEXEC – Basic Functionality
Before defining the SQLEXEC clause, a database logon must be established. This is
done via the SOURCEDB or USERID parameter for Extract, and the TARGETDB or
USERID parameter for Replicat.
When using SQLEXEC, a mapping between one or more input parameters and source
columns or column functions must be supplied.
When supplying at least one SQLEXEC entry for a given Replicat map entry, a target
table is not required.
VARCHAR2
DATE
All available numeric data types
LOB data types (BLOB and CLOB) where the length is less than 200 bytes
The ANSI equivalents of the above types
The stored procedure interface for SQL Server currently supports the following input and
output parameter types:
CHAR
VARCHAR
DATETIME
All available numeric data types
Image and text data types where the length is less than 200 bytes
TIMESTAMP parameter types are not supported natively, but you can specify other
data types for parameters and convert the data to TIMESTAMP format within the stored
procedure
The stored procedure interface for DB2 currently supports the following input and output
parameter types:
CHAR
VARCHAR
DATE
All available numeric data types
BLOB data types
The stored procedure interface for Sybase currently supports data types except TEXT,
IMAGE, and UDT.
The stored procedure interface Teradata version 12 and later supports CHAR,
VARCHAR, DATE and all available numeric data types.
As with direct table updates, database operations initiated within the stored procedure
will be committed in the same context as the original transaction.
151
SQLEXEC – Using with Lookup Stored Procedure
When the SQLEXEC parameter is used within a TABLE or MAP statement, the
syntax is:
SQLEXEC (
{ SPNAME <sp name> | QUERY “<sql query>” }
[, ID <logical name>]
{ PARAMS <param spec> | NOPARAMS}
[, BEFOREFILTER | AFTERFILTER]
[, DBOP]
[, EXEC <frequency>]
[, ALLPARAMS <option>]
[, PARAMBUFSIZE <num bytes>]
[, MAXVARCHARLEN <num bytes>]
[, TRACE <option>]
[, ERROR <action>]
)
The SQLEXEC option is specified as an option in TABLE and MAP statements within
EXTRACT and REPLICAT parameter files. Use either SPNAME (for stored
procedure) or QUERY (for SQL query).
153
QUERY “<sql query>”
Is a query to execute against the database. The query must be a legitimate SQL
statement for the database in question.
ID <logical name>
ID is required with the QUERY parameter (to reference the column values returned
by the query) or when you can invoke several instances of a stored procedure (to
reference each instance separately, for example, to invoke the same stored procedure
to populate two separate target columns).
NOPARAMS
Specifies there are no parameters.
BEFOREFILTER
Use BEFOREFILTER to cause the stored procedure or query to execute before
applying filters to a particular map. By default, stored procedures and queries are
executed after filtering logic has been applied.
AFTERFILTER
Use AFTERFILTER (the default) to cause the stored procedure or query to execute
after applying filters to a particular map. This enables you to skip stored procedure
or query processing unless the map is actually executed.
DBOP
Use DBOP if the stored procedure or query updates the database and you want the
process to execute transaction commit logic.
EXEC <frequency>
Determines the frequency of execution for the stored procedure or query.
MAP
Executes the stored procedure or query once for each source-target table map
for which it is specified. MAP renders the results invalid for any subsequent
maps that have the same source table. For example, if a source table is being
synchronized with more than one target table, the results would only be valid
for the first source-target map. MAP is the default.
ONCE
Executes the stored procedure or query once during the course of a
GoldenGate run, upon the first invocation of the associated FILE or MAP
Oracle GoldenGate Fundamentals Student Guide
statement. The results remain valid for as long as the process remains
running.
TRANSACTION
Executes the stored procedure or query once per source transaction. The
results remain valid for all operations of the transaction.
SOURCEROW
Executes the stored procedure or query once per source row operation. Use
this option when you are synchronizing a source table with more than one
target table, so that the results of the procedure or stored procedure or query
are invoked for each source-target mapping.
TRACE [ALL|ERROR]
If TRACE or TRACE ALL is specified, the input and output parameters for each
invocation of the stored procedure or query are output to the report file.
If TRACE ERROR is specified, parameters are output only after an error occurs in the
stored procedure or query.
ERROR <action>
Requires one of the following arguments:
IGNORE Database error is ignored and processing continues.
REPORT Database error is written to a report.
RAISE Database error is handled just as a table replication error.
FINAL Database error is handled as a table replication error, but does not
process any additional queries.
FATAL Database processing abends.
155
SQLEXEC - Syntax as a Standalone Statement
When a SQLEXEC parameter is used at the root level, the syntax is:
SQLEXEC
{“call <sp name> ()” | “<sql query>” | “<database command>”}
[EVERY <n> {SECONDS | MINUTES | HOURS | DAYS}]
[ONEXIT]
Examples:
“<sql query>”
Specifies the name of a query to execute. Enclose the query within quotes. For a
multi-line query, use quotes on each line. For best results, type a space after each
begin quote and before each end quote (or at least before each end quote). Example:
SQLEXEC “ select x from dual ”
“<database command>”
Executes a database command.
ONEXIT
Executes the SQL when the Extract or Replicat process stops gracefully.
Oracle GoldenGate Fundamentals Student Guide
Syntax
@GETVAL (<name>.<parameter>)
Example
<name>
The name of the stored procedure or query. When using SQLEXEC to execute the
procedure or query, valid values are as follows:
Queries: Use the logical name specified with the ID option of the SQLEXEC clause. ID
is a required SQLEXEC argument for queries.
Stored procedures: Use one of the following, depending on how many times the
procedure is to be executed within a TABLE or MAP statement:
- For multiple executions, use the logical name defined by the ID clause of the
SQLEXEC statement. ID is required for multiple executions of a procedure.
- For a single execution, use the actual stored procedure name.
<parameter>
Valid values are one of the following:
157
- The name of the parameter in the stored procedure or query from which the data
will be extracted and passed to the column map.
- RETURN_VALUE, if extracting values returned by a stored procedure or query.
Rules for determining expiration are defined by the SQLEXEC EXEC option. When a
value cannot be extracted, the @GETVAL function results in a “column missing”
condition. Usually this means that the column is not mapped. You can also use the
@COLTEST function to test the result of the @GETVAL function to see if it is
missing, and map an alternative value if desired.
Macros
Macros - Overview
By using GoldenGate macros in parameter files you can easily configure and reuse
parameters, commands, and functions. As detailed in the slide, you can use macros
for a variety of operations to enable easier and more efficient building of
parameters.
Note: Do not use macros to manipulate data for tables being processing by a data
pump Extract in pass-through mode.
Oracle GoldenGate Fundamentals Student Guide
Macros - Creating
Syntax
MACRO #<macro name>
PARAMS (#<param1>, #<param2>, …)
BEGIN
<macro body>
END;
Syntax
<macro name> is the name of the macro. <macro name> must begin with the #
character, as in #macro1. If the # macro character is used elsewhere in the
parameter file, such as in a table name, you can change it to something else with
the MACROCHAR parameter. Macro names are not case-sensitive.
PARAMS (<p1>,<p2>...) describes each of the parameters to the macro. Names must
begin with the macro character, such as #param1. When the macro is invoked, it
must include a value for each parameter named in the PARAMS statement.
Parameter names are optional and not case-sensitive.
BEGIN indicates the beginning of the body of the macro. Must be specified before
the macro body.
<macro body> represents one or more statements to be used as parameter file input.
It can include simple parameter statements, such as COL1 = COL2; more complex
statements that include parameters, such as COL1 = #val2; or invocations of other
macros, such as #colmap(COL1, #sourcecol).
END ends the macro definition.
159
Macros - Invoking
EXTRACT EXSALES
MACRO #make_date
PARAMS (#year, #month, #day)
BEGIN
@DATE(“YYYY-MM-DD”, “CC”, @IF(#year < 50, 20, 19),
“YY”, #year, “MM”, #month, “DD”, #day)
END;
Invoking Macros
The example above demonstrates defining a macro named #make_date, calling the
macro two different times, with each instance sending a different set of source column
values to determine the target column values.
Note that the order and ship dates are determined as the result of calling the
make_date routine to populate the target columns.
Macros - Example
GETINSERTS
GETUPDATES
GETDELETES
INSERTDELETES
IGNOREUPDATES
MAP SALES.SRCTAB, TARGET SALES.TARGTAB;
GETINSERTS
GETUPDATES
GETDELETES
INSERTDELETES
MAP SALES.SRCTAB2, TARGET SALES.TARGTAB2;
Note that the macro’s result is altered by the IGNOREUPDATES parameter for the first
MAP statement.
Macros – Libraries
EXTRACT EXTACCT
INCLUDE /ggs/dirprm/macro.lib
NOLIST
include /ggs/dirprm/large.lib
LIST
Macro libraries
To use a macro library, use the INCLUDE parameter at the beginning of a parameter
file. The syntax is:
You may toggle the listing or suppression of listing of the output of libraries by using
the LIST and NOLIST parameters.
161
Macros – Expansion
Syntax
CMDTRACE [ ON | OFF | DETAIL ]
Default is OFF
Example
EXTRACT EXTACCT
INCLUDE /ggs/dirprm/macro.lib
CMDTRACE ON
MAP SALES.ACCOUNT, TARGET REPORT.ACCOUNT_HISTORY,
COLMAP (USEDEFAULTS,
#maptranfields () );
Macro expansion
The macro processor enables tracing of macro expansion for debugging purposes via
the CMDTRACE command. When CMDTRACE is enabled, the macro processor will
display macro expansion steps in the process’s report file.
The ON option enables tracing, OFF disables it, and DETAIL produces additional details.
Oracle GoldenGate Fundamentals Student Guide
User Tokens
Set token values through a TABLE TOKENS clause and @GETENV functions,
for example:
TABLE SALES.PRODUCT,
TOKENS (TKN1 = @GETENV(“GGENVIRONMENT",“OSUSERNAME"),
TKN2 = @GETENV(“GGHEADER",“COMMITTIMESTAMP") );
163
User Tokens - Environmental Values Available to @GETENV
Note: If a given database, operating system, or GoldenGate version does not provide
information that relates to a given token, a NULL value will be returned.
@GETENV (“JULIANTIMESTAMP”)
@GETENV (“RECSOUTPUT”)
“HOSTNAME” Returns the name of the system running the Extract or Replicat
process.
“OSUSERNAME” Returns the operating system user name that started the process.
“PROCESSID” The process ID that is assigned by the operating system.
165
“LASTRECIOTIME” Returns the time that the last record was written to the trail
file. NULL until the trail file is completed.
“URI” Returns the universal resource identifier of the process that created the trail
file, in the format: <host_name>:<dir>:[:<dir>][:<dir_n>]<group_name>
“URIHISTORY” Returns a list of the URIs of processes that wrote to the trail file
before the current process.
Information about the GoldenGate process that created the trail file
“GROUPNAME” Returns the droup name associated with the Extract process that
created the trail. The group name is that which was given in the ADD EXTRACT
command. For example, “ggext.”
“DATASOURCE” Returns the data source that was read by the process.
“GGMAJORVERSION” Returns the major version of the Extract process that created
the trail, expressed as an integer. For example, if a version is 1.2.3, it returns 1.
“GGMINORVERSION” Returns the minor version of the Extract process that created
the trail, expressed as an integer. For example, if a version is 1.2.3, it returns 2.
“GGVERSIONSTRING” Returns the maintenance (or patch) level of the Extract
process that created the trail, expressed as an integer. For example, if a version is
1.2.3, it returns 3.
“GGMAINTENANCELEVEL” Returns the maintenance version of the process
(xx.xx.xx).
“GGBUGFIXLEVEL” Returns the patch version of the process (xx.xx.xx.xx).
“GGBUILDNUMBER” Returns the build number of the process.
Information about the database that produced the data in the trail file.
“DBNAME” Returns the name of the database, for example findb.
“DBINSTANCE” Returns the name of the database instance, if applicable to the
database type, for example ORA1022A.
“DBTYPE” Returns the type of database that produced the data in the trail file.
“DBCHARSET” Returns the character set that is used by the database that produced
the data in the trail file.
“DBMAJORVERSION” Returns the major version of the database that produced the
data in the trail file.
“DBMINORVERSION” Returns the minor version of the database that produced the
data in the trail file.
Oracle GoldenGate Fundamentals Student Guide
167
User Tokens - Setting
Values are stored in the GoldenGate record header using a TOKENS clause and
@GETENV functions:
EXTRACT EXTDEMO
TABLE SALES.PRODUCT, TOKENS (
TKN-OSUSER = @GETENV (“GGENVIRONMENT", “OSUSERNAME"),
TKN-DOMAIN = @GETENV (“GGENVIRONMENT", “DOMAINNAME"),
TKN-COMMIT-TS = @GETENV (“GGHEADER", “COMMITTIMESTAMP"),
TKN-BA-IND = @GETENV (“GGHEADER", “BEFOREAFTERINDICATOR),
TKN-TABLE = @GETENV (“GGHEADER", “TABLENAME"),
TKN-OP-TYPE = @GETENV (“GGHEADER", “OPTYPE"),
TKN-LENGTH = @GETENV (“GGHEADER", “RECORDLENGTH"),
TKN-DB-VER = @GETENV (“DBENVIRONMENT", “DBVERSION");
Using the TOKENS clause of the TABLE parameter, the user defines a token identifier
(e.g. TKN-OSUSER) and specifies the environment category and value using the
@GETENV function.
specify the token identifier (e.g. TKN-GROUP-NAME) value to use for the target column
specification.
User tokens:
TKN-HOST : jemhadar
TKN-GROUP : EXTORA
TKN-BA_IND : AFTER
TKN-COMMIT_TS : 2003-03-24 17:08:59.000000
TKN-POS : 3604496
TKN-RBA : 4058
TKN-TABLE : SOURCE.CUSTOMER
TKN-OPTYPE : INSERT
TKN-LENGTH : 57
TKN-TRAN_IND : BEGIN
TKN-LAG_SEC : 1
TKN-LAG_MIN : 0
TKN-LAG_MSEC : 1229
TKN-NUMRECS : 8
TKN-DBNAME : ORA901
TKN-DB_USER : GGOODRIC
TKN-DB_VER : 9.0.1.0.0
TKN-INAME : ora901
TKN-ROWID : AAABBAAABAAAB0BAAF
LOGDUMP example
Once environment values have been stored in the trail header, Logdump can display
them when the USERTOKEN ON option is used. The USERTOKEN DETAIL option provides
additional information.
169
User Exits
User exits cannot be used for tables being processing by a data-pump Extract in pass-
through mode.
Oracle GoldenGate Fundamentals Student Guide
171
User Exits - Parameters
EXIT_CALL_TYPE indicates the processing point of the caller and determines the type
of processing to perform. Extract and Replicat call the shell routine with the following
calls:
EXIT_CALL_START - Invoked at the start of processing. The user exit can
perform initialization work.
EXIT_CALL_STOP - Invoked before the caller stops or ends abnormally. The user
exit can perform completion work.
EXIT_CALL_BEGIN_TRANS - In Extract, invoked just before the output of the
first record in a transaction. In Replicat, invoked just before the start of a
transaction.
EXIT_CALL_END_TRANS - In Extract and Replicat , invoked just after the last
record in a transaction is processed.
EXIT_CALL_CHECKPOINT - Called just before an Extract or Replicat checkpoint
is written.
EXIT_CALL_PROCESS_RECORD - In Extract, invoked before a record buffer is
output to an Extract file. In Replicat, invoked just before a replicated
operation is performed. This call is the basis of most user exit processing.
EXIT_CALL_PROCESS_MARKER - Called during Replicat processing when a
marker from a NonStop server is read from the trail, and before writing to the
marker history file.
EXIT-CALL_DISCARD_RECORD - Called during Replicat processing before a
record is written to the discard file.
EXIT_CALL_DISCARD_ASCII_RECORD - Called during Extract processing
before an ASCII input record is written to the discard file.
EXIT_CALL_FATAL_ERROR - Called during Extract or Replicat processing
just before GoldenGate terminates after a fatal error.
EXIT_CALL_RESULT - Set by the user exit routines to instruct the caller how
to respond when each exit call completes (see below).
Oracle GoldenGate Fundamentals Student Guide
EXIT_CALL_RESULT is set by the user exit routines and instructs the caller how to
respond when each exit call completes. The following results can be specified by the
operator’s routines:
EXIT_OK_VAL - If the routine does nothing to respond to an event, OK is
assumed. If the call specified PROCESS_RECORD or DISCARD_RECORD and
OK_VAL is returned, the caller processes the record buffer returned by the user
exit and uses the parameters set by the exit .
EXIT_IGNORE_VAL - Reject records for further processing.
EXIT_STOP_VAL - Instructs the caller to STOP immediately.
EXIT_ABEND_VAL - Instructs the caller to ABEND immediately.
EXIT_PROCESSED_REC_VAL - Instructs Extract or Replicat to skip the
record, but update the statistics that are printed to the eport file for that table
and for that operation type.
EXIT_PARAMS supplies information to the user exit routine, such as the program
name and user-defined parameters. You can process a single data record multiple
times:
PROGRAM_NAME - Specifies the full path and name of the calling process,
for example \ggs\extract or \ggs\replicat. Use this parameter when loading a
GoldenGate callback routine using the Windows API or to identify the calling
program when user exits are used with both Extract and Replicat processing.
FUNCTION_PARAM - Allows you to pass a parameter that is a literal string
to the user exit. Specify the parameter with the EXITPARAM option of a
TABLE or MAP statement. FUNCTION_PARAM can also be used at exit call
startup to pass the parameters that are specified in the PARAMS option of the
CUSEREXIT parameter.
MORE_RECS_IND Set on return from an exit. For database records,
determines whether Extract or Replicat processes the record again. This
allows the user exit to output many records per record processed by Extract, a
common function when converting Enscribe to SQL (data normalization). To
request the same record again, set MORE_RECS_IND to CHAR_NO_VAL or
CHAR_YES_VAL.
173
<result_code> The status of the function executed by the callback routine.
The result code returned by the callback routine indicates whether or not the
callback function was successful.
See the Oracle GoldenGate Reference Guide for function codes and result codes.
You can define the name of the routine, but it must accept the following user exit
parameters:
EXIT_CALL_TYPE
EXIT_CALL_RESULT
EXIT_PARAMS
2. In the source for the DLL/shared object, include the usrdecs.h file. This file
contains type definitions, return status values, callback function codes, and a number
of other definitions.
3. Include callback routines in the user exit when applicable. Callback routines
retrieve record and application context information, and modify the contents of data
records.
Oracle GoldenGate Fundamentals Student Guide
Extract and Replicat export an ERCALLBACK function to be called from the user exit
routine. The user exit must explicitly load the callback function at run-time using the
appropriate Windows/Unix API calls.
175
User Exits - Calling
You can call a user exit from Extract or Replicat by the CUSEREXIT
parameter
Syntax
CUSEREXIT <DLL or shared object name> <routine name>
[, PASSTHRU]
[, INCLUDEUPDATEBEFORES]
[, PARAMS "<startup string>”]
Examples
CUSEREXIT userexit.dll MyUserExit
CUSEREXIT userexit.dll MyUserExit, INCLUDEUPDATEBEFORES, &
PASSTHRU, PARAMS "init.properties"
<routine name>
The name of the exit routine to be executed.
PASSTHRU
Valid only for an Extract data pump. It assumes that no database is required, and
that no output trail is allowed. It expects that the
user exit will perform all of the processing and that Extract will skip the record.
Extract will perform all of the required data mapping before passing the record to the
user exit. Instead of a reply status of EXIT_OK_VAL, the reply will be
EXIT_PROCESSED_REC_VAL. All process statistics are updated as if the records
were processed by GoldenGate.
INCLUDEUPDATEBEFORES
Passes the before images of column values to a user exit. When using this parameter,
you must explicitly request the before image by setting the
requesting_before_after_ind flag to BEFORE_IMAGE_VAL within a callback function
that supports this flag. Otherwise, only the after image is passed to the user exit. By
default, GoldenGate only works with after images.
When using INCLUDEUPDATEBEFORES for a user exit that is called from a data
pump or from Replicat, always use the GETUPDATEBEFORES
parameter for the primary Extract process, so that the before image is captured,
written to the trail, and causes a process record event in the user exit. In a case where
the primary Extract also has a user exit, GETUPDATEBEFORES causes both the
before image and the
after image to be sent to the user exit as separate EXIT_CALL_PROCESS_RECORD
events.
Oracle GoldenGate Fundamentals Student Guide
If the user exit is called from a primary Extract (one that reads the transaction log),
only INCLUDEUPDATEBEFORES is needed for that
Extract. GETUPDATEBEFORES is not needed in this case, unless other GoldenGate
processes downstream will need the before image to be
written to the trail. INCLUDEUPDATEBEFORES does not cause before images to be
written to the trail.
Oracle Sequences
Oracle Sequences
GoldenGate supports the replication of Oracle sequence values
Use the Extract SEQUENCE parameter to extract sequence values from the
transaction log; for example:
SEQUENCE hr.employees_seq;
Use the Replicat MAP parameter to apply sequence values to the target;
for example:
Note: Change this default only if you know there will be no gaps in the sequence updates (e.g.
from a trail corruption or process failure) and you want to improve the performance of
GoldenGate
Note: Gaps are possible in the values of the sequences that GoldenGate replicates
because gaps are inherent, and expected, in the way that sequences are maintained by
the database. However, the target values will always be greater than those of the
source, unless the NOCHECKSEQUENCEVALUE parameter is used.
177
Configuration Options
BATCHSQL
Compression
Encryption
Bidirectional considerations
Event actions
BATCHSQL
Operations containing the same table, operation type (I, U, D), and column list are
grouped into a batch. For example, the following would each be an example of a batch:
Inserts to table A
Inserts to table B
Updates to table A
Updates to table B
Deletes from table A
Deletes from table B
179
Options: BATCHSQL – Syntax
Syntax
BATCHSQL
[BATCHERRORMODE | NOBATCHERRORMODE]
[BATCHESPERQUEUE <n>]
[BATCHTRANSOPS <n>]
[BYTESPERQUEUE <n>]
[OPSPERBATCH <n>]
[OPSPERQUEUE <n>]
[TRACE]
BATCHERRORMODE | NOBATCHERRORMODE
Set the response of Replicat to errors.
•In NOBATCHERRORMODE (default), Replicat aborts the transaction on an error,
temporarily disables BATCHSQL, and retries in normal mode.
•In BATCHERRORMODE, Replicat attempts to resolve errors without reverting to
normal mode.
•Requires HANDLECOLLISIONS to prevent Replicat from exiting on an error.
BATCHESPERQUEUE <n>
Sets a maximum number of batches per queue before flushing all batches. The default
is 50. Note: A queue is a thread of memory containing captured operations waiting to
be batched. By default, there is one buffer queue, but you can change this with
NUMTHREADS.
BATCHTRANSOPS <n>
Controls the size of a batch. Set to the default of 1000 or higher.
BYTESPERQUEUE <n>
Sets the maximum number of bytes to hold in a queue before flushing batches. The
default is 20 megabytes.
OPSPERBATCH <n>
Sets the maximum number of rows that can be prepared for one batch before
flushing. The default is 1200.
OPSPERQUEUE <n>
Sets the maximum number of row operations that can be queued for all batches
before flushing. The default is 1200.
TRACE
Enables tracing of BATCHSQL activity to the console and report file.
Oracle GoldenGate Fundamentals Student Guide
At 100 bytes of data per row change, BATCHSQL has been known
to improve Replicat’s performance from 400 to 500 percent.
Usage restrictions
Some statement types cannot be processed in batches and must be processed as
exceptions. When BATCHSQL encounters them, it flushes everything in the batch,
applies the exceptions in the normal manner of one at a time, and then resumes batch
processing. Transaction integrity is maintained. Statements treated as exceptions
include:
• Statements containing LOB or LONG data.
• Statements containing rows longer than 25k in length.
• Statements where the target table has one or more unique keys besides the primary
key. Such statements cannot be processed in batches because BATCHSQL does not
guarantee correct ordering for non-primary keys if their values could change.
181
Compression
Options: Compression
Example:
The destination Server Collector decompresses the data stream before writing it to
the remote file or remote trail. This typically results in compression ratios of at
least 4:1 and sometimes much better. However, compression can require
significant CPU resources.
Oracle GoldenGate Fundamentals Student Guide
Encryption
Message Encryption
Encrypts the messages sent over TCP/IP
Uses Blowfish, a symmetric 64-bit block cipher from CounterPane™
Internet Security
The data is automatically decrypted by Server Collector before
saving the data to the trail
Message Encryption
(Blowfish)
Parameters Parameters
183
Options: Message Encryption
1. Run the GoldenGate KEYGEN utility to generate random hex keys
C:\GGS> RUN KEYGEN <key length> <number of keys>
Blowfish accepts a variable-length key from 32 to 128 bits
2. Enter key names and values in an ASCII text file named ENCKEYS (upper case, no
file extension) in the GoldenGate install directory
##Key name Key value
superkey 0x420E61BE7002D63560929CCA17A4E1FB
secretkey 0x027742185BBF232D7C664A5E1A76B040
3. Copy the ENCKEYS file to the source and target GoldenGate install directory
4. In the Extract parameter files, use the RMTHOST ENCRYPT and KEYNAME
parameters
RMTHOST West, MGRPORT 7809, ENCRYPT BLOWFISH, KEYNAME superkey
5. Configure a static Server Collector and start it manually with the -ENCRYPT and
-KEYNAME parameters
server -p <port> -ENCRYPT BLOWFISH -KEYNAME <keyname>
If you prefer to use a literal key, then instead of using KEYGEN, enter the literal key
in quotes as the key value in an ENCKEYS file:
KEYGEN
ENCKEYS ENCKEYS
Message Encryption
(Blowfish)
Oracle GoldenGate Fundamentals Student Guide
For example:
185
Options: Password Encryption – Method 2
For example:
2. Enter the key name and value in the ENCKEYS file, for example:
Password Encryption
Oracle GoldenGate Fundamentals Student Guide
Event Actions
187
Options: Event Actions – Examples (cont’d)
EVENT EVENT
PROCESSING PROCESSING
Whenever a record is written to the event table, the trail file is rolled
over.
TABLE | MAP
…
EVENTACTIONS (
[STOP | ABORT | FORCESTOP]
[IGNORE [TRANSACTION [INCLUDEVENT]]
[DISCARD]
[LOG [INFO | WARNING]]
[REPORT]
[ROLLOVER]
[SHELL <command>]
[TRACE <trace file> [TRANSACTION] [PURGE | APPEND]]
[CHECKPOINT [BEFORE | AFTER | BOTH]]
[, ...]
)
Note: You can also use a TABLE parameter in a Replicat to trigger actions without
writing data to target tables
EVENTACTIONS
STOP – Graceful stop
ABORT – Immediate exit
FORCESTOP – Graceful stop if the event record is the last operation in the
transaction, else log warning message and abort
IGNORE [TRANSACTION [INCLUDEVENT]] – Ignore record. Optionally ignore
entire transaction and propagate the event record.
DISCARD – Write record to discard file
LOG [INFO | WARNING] – Log an informational or warning message to the report,
error and systems event files
REPORT – Generate a report file
ROLLOVER – (Extract only) Roll over the trail file
SHELL – Execute a shell command
TRACE – Write trace information to file
CHECKPOINT – Write a checkpoint before and/or after writing the event record
189
Options: Event Actions – Heartbeat Example
Info
EVENT Log
PROCESSING
Warning
TX1
TX2
Target Replicat
Heartbeat
Trail
TX4
Switchover
Application Application
1. User writes an event record at the 2. When Replicat reads the event
planned outage point. This is read by record, it triggers an event action –
Extract through the transaction log. run a custom script to switch the
application to the target database.
Replicat Extract
Trail Trans Log
3. The Extract on the target,
already configured and running,
starts capturing transactions.
Oracle GoldenGate Fundamentals Student Guide
Replicat Extract
Trail Trans Log
3. When the second ETL process is completed, it generates an event record that is read by
Extract on the target. When Replicat on the source receives the event record, it triggers a
custom script to start the application based on the status of the batch process on the source.
191
Bidirectional Considerations
Transaction
Log
Source Target
Target
Source
The top of this illustration shows changes from the left-side database being extracted
and sent over the network to be replicated in the right-side database. The bottom is
the reverse – changes from the right-side database are extracted and sent to the left.
Distributed processing
Bidirectional Capabilities
GoldenGate supports bidirectional replication between two databases, whether the
configuration is for homogeneous or heterogeneous synchronization.
Oracle GoldenGate Fundamentals Student Guide
In a bidirectional configuration, both sides of the database may be live and processing
application transactions at the same time. It also may be that the target application is
standing-by waiting to be used if there is a fail-over. For this, operations are captured
and queued to be synchronized back to the primary database once it becomes
available.
Loop Detection
Detect if GoldenGate or the application performed the operation
Loops
Because there are both Extract and Replicat processes operating on the same tables in
bidirectional synchronization, Replicat’s operations must be prevented from being
sent back to the source table by Extract. If they are re-extracted they will be re-
replicated beginning an endless loop. Loop detection is sometimes called ping-pong
detection.
Conflicts
Because GoldenGate is an asynchronous solution, conflict-management is required to
ensure data accuracy in the event that the same row is changed in two or more
databases at (or about) the same time. For example, User A on Database A updates a
row, and then User B on Database B updates that same row. If User B’s transaction
occurs before User A’s transaction is synchronized to Database B, there will be a
conflict on the replicated transaction.
193
Options: Bidirectional - Loop Example 1
Preventing looping
To avoid looping, Replicat’s operations must be prevented from being sent back to the
source table by Extract. This is accomplished by configuring Extract to ignore
transactions issued by the Replicat user. This slide illustrates why data looping needs
to be prevented.
Methods of preventing data looping are available for the following databases.
• Oracle, DB2, and SQL Server databases using log-based extraction
• Teradata databases using VAM-based extraction
• SQL/MX using the checkpoint table.
Or…
Because of the constraint that the primary key must be unique, the insert fails in this
example so it does not trigger looping. However the failed insert causes the Replicat
Oracle GoldenGate Fundamentals Student Guide
to abend which stops all replication services, so it is another example of the need to
recognize and not extract Replicat transactions.
When the Replicat starts up, it looks for a TRACETABLE parameter, and if one exists,
updates the trace table name as every transaction is committed. If no TRACETABLE
parameter is present, Replicat looks for a default table GGS_TRACE owned by the
logged-in USERID. If it exists, it will automatically update the table. If GGS_TRACE
does not exist, then nothing is updated.
To create the GGS_TRACE table, use the GGSCI command ADD TRACETABLE. To create
a table with a different table name, use ADD TRACETABLE [<owner>].<table_name>. If
<owner> is omitted, it assumes the logged-in USERID. GGSCI also provides INFO
and DELETE commands for the TRACETABLE entity. To use these commands, you must
first login to the database using DBLOGIN.
195
Options: Bidirectional - Loop Detection (cont’d)
SQL Server
By default, SQL Server does not log the ggs_repl transactions
written by Replicat, so no loops arise.
Loop-detection operates by default for SQL Server, NonStop SQL/MP and Enscribe
sources. For other database types, you must handle loop-detection in various ways.
SQL Server
By default, SQL Server logging excludes the ggs_repl transactions written by Replicat.
Sybase
Do nothing and allow Replicat to use the default transaction name
ggs_repl
Or identify the Replicat transaction name in the Extract parameter:
TRANLOGOPTIONS EXCLUDETRANS <trans name>
Or identify the Replicat user name in the Extract parameter:
TRANLOGOPTIONS EXCLUDEUSER <user name>
Oracle GoldenGate Fundamentals Student Guide
NonStop SQL/MX
Replicat transactions identified by the name of the checkpoint
table specified with TRANLOGOPTIONS option:
TRANLOGOPTIONS FILTERTABLE <table>
Extract ignores transactions that include this checkpoint table
PURGEDATA operation is not supported
Teradata
You do not need to identify Replicat transactions that are applied
to a Teradata database.
c-tree
Extract automatically identifies Replicat transactions that are
applied to a c-tree database
NonStop SQL/MX
To prevent data loopback, a checkpoint table is required in a bidirectional
configuration that involves a source or target SQL/MX database (or both).
Because there is not SQL transaction associated with a PURGEDATA operation, this
operation type is not supported for bidirectional replication. Because PURGEDATA
operations are DDL, they are implicit transactions, so GoldenGate cannot update the
checkpoint table within that transaction.
Conflict Detection
Low latency reduces the risk of encountering a conflict where the same record is
updated on both systems. This is the best overall method that we encourage in
bidirectional configurations.
197
Conflict detection can also be addressed at the application or system level by assuring
that records are always updated on one system. For example, all updates to card
holder (or account number) 0 – 1000000 are applied on system A, while updates to
1000001 or higher is applied on system B.
GoldenGate for UNIX and Windows (only) provides the capability to capture the
before and after values of columns so that comparisons may be made on the target
database before applying the values. Additional SQL procedures can be written based
on your application requirements.
INSERTALLRECORDS
MAP SRCTAB, TARGET TARGET_EXCEPT, EXCEPTIONSONLY,
COLMAP (USEDEFAULTS, ERRTYPE = “Conflict Detected”);
Time
The conflict
When the two transactions occur at different times, the deposit transaction is
replicated to New York before the withdrawal transaction is extracted. But when the
two transactions occur at around the same time, they are applied independently,
resulting in an incorrect balance on both systems.
199
Options: Bidirectional - Conflict Resolution by Applying Net Differences
Identity data types are used for a variety of purposes, for example as a field that is
part of a primary key in order to make the row unique. The challenge in a
bidirectional configuration is to ensure that the same identity value is not generated
on both systems causing a conflict.
One technique that addresses this issue is to create the table on each system and set
the seed and increment values so that each system is using a different range of
identity values.
201
Oracle DDL Replication
Object definitions
Source and target object definitions must be identical (ASSUMETARGETDEFS specified
for Replicat). Neither data nor DDL conversion is supported.
DDL and data pumps
When Extract data pumps are being used, tables for which DDL is being replicated
must be configured in pass-through mode. DDL filtering, manipulation, and error
handling is not supported by data pumps.
Oracle GoldenGate Fundamentals Student Guide
Wildcard resolution
Standard GoldenGate asterisk wildcards (*) can be used with certain parameter
options when synchronizing DDL operations. WILDCARDRESOLVE is now set by default
to DYNAMIC and must remain so for DDL support.
User Exits
GoldenGate user exit functionality is not supported for use with DDL synchronization
activities (user exit logic can not be triggered based on DDL operations.) User exits
can be used with concurrent DML processing.
LOB data
With LOB data Extract might fetch a LOB value from a Flashback Query, and Oracle
does not provide Flashback capability for DDL (except DROP). When a LOB is fetched,
the object structure reflects current metadata, but the LOB record in the transaction
log reflects old metadata. Refer to the Oracle GoldenGate Reference Guide for
information on this topic.
User defined types
DDL operations that involve user defined types generate implied DML operations on
both the source and target. To avoid SQL errors that would be caused by redundant
operations, GoldenGate does not replicate those DML operations.
If DML is being replicated for a user defined type, Extract must process all of those
changes before DDL can be performed on the object. Because UDT data might be
fetched by Extract, the reasons for this rule are similar to those that apply to LOB
columns
SQLEXEC
Objects that are affected by a stored procedure must exist with the correct structure
prior to the execution of SQL. Consequently, DDL that affects structure must happen
before the SQLEXEC executes.
Objects affected by a standalone SQLEXEC statement must exist before the
GoldenGate processes start. This means that DDL support must be disabled for these
objects; otherwise DDL operations could change or delete the object before the
SQLEXEC executes.
Long DDL statements
GoldenGate 10.4 supports the capture and replication of Oracle DDL statements of up
to 2 MB in length (including some internal GoldenGate maintenance information).
Extract will skip statements that are greater than the supported length, but the
ddl_ddl2file.sql script can be used to save the skipped DDL to a text file in the
USER_DUMP_DEST directory of Oracle.
To use the new support, the DDL trigger must be reinstalled in INITIALSETUP
mode, which removes all of the DDL history. See the GoldenGate for Oracle
Installation and Setup Guide.
203
Options: DDL Replication - Oracle Requirements/Restrictions
Schema names using Oracle reserved names are ignored
The GETTRUNCATES parameter should not be used with full DDL
support
Table name cannot be longer than 16 characters plus quotation
marks for ALTER TABLE RENAME
ALTER TABLE..MOVE TABLESPACE
Supported when tablespace all SMALLFILE or BIGFILE
GETTRUNCATES
GoldenGate supports the synchronization of TRUNCATEs as a standalone function
(independently of full DDL synchronization) or as part of full DDL synchronization. If
using DDL synchronization, disable standalone TRUNCATE synchronization to avoid
errors caused by duplicate operations.
Table names
The ALTER TABLE RENAME fails if the old or new table name is longer than 18
characters (16 for the name and two for the quotation marks). Oracle only allows 18
characters for a rename because of the ANSI limit for identifiers.
Extract sends all DDL to each trail when writing to multiple trails
Renames
To work around remote permissions issues that may arise when different users are
being used on the source and target, RENAME will always be converted to equivalent
ALTER TABLE RENAME for Oracle.
Oracle GoldenGate Fundamentals Student Guide
New Names
New names must be specified in TABLE/MAP statements in order to:
1) Replicate DML operations on tables resulting from a CREATE or RENAME,
2) CREATE USER and then move new/renamed tables into that schema
/* GOLDENGATE_DDL_REPLICATION */
Renames
To work around remote permissions issues that may arise when different users are
being used on the source and target, RENAME will always be converted to equivalent
ALTER TABLE RENAME.
New Names
New names must be specified in TABLE/MAP statements in order to:
1) Replicate DML operations on tables resulting from a CREATE or RENAME,
2) CREATE USER and then move new/renamed tables into that schema
205
DDL marker table
The DDL Marker table only receives inserts and its rows can be periodically purged.
The name can be set in the GLOBALS parameter file, but if it is not the default name is
GGS_MARKER.
Do not delete the DDL marker table if you plan to continue processing DDL. The
marker table and the DDL trigger are interdependent. If you attempt to remove the
marker table without first removing the trigger, the following error will be generated:
"ORA-04098: trigger 'SYS.GGS_DDL_TRIGGER_BEFORE' is invalid
and failed re-validation"
DDL history table
The DDL History table receives inserts, updates, deletes. It contains the SQL
statement that was issued by the user. Default name is GGS_DDL_HIST. Caution must
be used if purging.
DDL trigger
DDL trigger is installed with some packages. Default trigger name is
GGS_DDL_TRIGGER_BEFORE.
User Role
Default user role name is GGS_GGSUSER_ROLE.
Other supporting database objects:
•Internal setup table
•DUMPDDL tables (for viewing DDL history)
•ddl_pin (pins tracing for performance evaluation)
•sequence used for a column in the marker table
MAPPED
- Is specified in a TABLE or MAP statement
- Operations CREATE, ALTER, DROP, RENAME, GRANT*, REVOKE*
- Objects TABLE*, INDEX, TRIGGER, SEQUENCE*, MATERIALIZED VIW*
* Operations are only for objects with asterisk
UNMAPPED
- Does not have a TABLE or MAP statement
OTHER
- TABLE or MAP statements do not apply
- DDL operations other than those listed above
- Examples are CREATE USER, CREATE ROLE, ALTER TABLESPACE
Oracle GoldenGate Fundamentals Student Guide
Only one DDL parameter can be used in a parameter file, but you can combine
multiple inclusion and exclusion options to filter the DDL to the required level. When
combined, multiple option specifications are linked logically as AND statements. All
criteria specified with multiple options must be satisfied for a DDL statement to be
replicated.
Options
INCLUDE | EXCLUDE Identifies the beginning of an inclusion or exclusion clause.
INCLUDE includes specified DDL for capture or replication. EXCLUDE excludes specified
DDL from being captured or replicated.
The inclusion or exclusion clause must consist of the INCLUDE or EXCLUDE keyword
followed by any valid combination of other options of the DDL parameter. An EXCLUDE
must be accompanied by a corresponding INCLUDE clause. An EXCLUDE takes priority
over any INCLUDEs that contain the same criteria. You can use multiple inclusion and
exclusion clauses.
MAPPED | UNMAPPED | OTHER | ALL applies INCLUDE or EXCLUDE based on the DDL
operation scope.
MAPPED applies to DDL operations that are of MAPPED scope.
UNMAPPED applies to DDL operations that are of UNMAPPED scope.
OTHER applies to DDL operations that are of OTHER scope.
ALL applies to DDL operations of all scopes. DDL EXCLUDE ALL maintains up-to-date
metadata on objects, while blocking the replication of the DDL operations themselves.
OPTYPE <type> applies INCLUDE or EXCLUDE to a specific type of DDL operation. For
<type>, use any DDL command that is valid for the database, such as CREATE, ALTER,
and RENAME.
OBJTYPE ‘<type>’ applies INCLUDE or EXCLUDE to a specific type of database object.
For <type>, use any object type that is valid for the database, such as TABLE, INDEX,
TRIGGER, USER, ROLE. Enclose the object type within single quotes.
OBJNAME “<name>” applies INCLUDE or EXCLUDE to the name of an object, for
example a table name. Provide a double-quoted string as input. Wildcards can be
used. If you do not qualify the object name for Oracle, the owner is assumed to be the
GoldenGate user.
207
When using OBJNAME with MAPPED in a Replicat parameter file, the value for
OBJNAME must refer to the name specified with the TARGET clause of the MAP
statement.
For DDL that creates triggers and indexes, the value for OBJNAME must be the name
of the base object, not the name of the trigger or index.
For RENAME operations, the value for OBJNAME must be the new table name.
INSTR ‘<string>’ applies INCLUDE or EXCLUDE to DDL statements that contain a
specific character string within the command syntax itself, but not within comments.
Enclose the string within single quotes. The string search is not case sensitive
INSTRCOMMENTS ‘<comment_string>’s applies INCLUDE or EXCLUDE to DDL
statements that contain a specific character string within a comment, but not within
the DDL command itself. By using INSTRCOMMENTS, you can use comments as a
filtering agent. Enclose the string within single quotes. The string search is not case
sensitive. You can combine INSTR and INSTRCOMMENTS options to filter on a string in
the command syntax and in the comments.
Where:
‘<search_string>’ is the string in the source DDL statement you
want to replace, in single quotes
‘<replace_string>’ is the replacement string, in single quotes
DDLSUBST Clauses
DDLSUBST ‘<search_string>’ WITH ‘<replace_string>’
[ {INCLUDE | EXCLUDE}
[, ALL | MAPPED | UNMAPPED | OTHER]
[, OPTYPE <type>]
[, OBJTYPE <type>]
[, OBJNAME “<name>”]
[, INSTR ‘<string>’]
[, INSTRCOMMENTS ‘<comment_string>’]
]
[...]
Oracle GoldenGate Fundamentals Student Guide
Extract syntax:
DDLERROR
[, RESTARTSKIP <num skips>]
Replicat syntax:
DDLERROR
{<error> | DEFAULT} {<response>}
[RETRYOP MAXRETRIES <n> [RETRYDELAY <delay>]]
{INCLUDE <clause> | EXCLUDE <clause>}
[, IGNOREMISSINGTABLES | ABENDONMISSINGTABLES]
[, RESTARTCOLLISIONS | NORESTARTCOLLISIONS]
Where <response> can be IGNORE, ABEND, DISCARD
209
NOREPORT reports basic DDL statistics. REPORT adds the parameters being used and a
step-by-step history of the operations that were processed
ADDTRANDATA is valid for Extract. Use ADDTRANDATA to enable supplemental
logging for CREATE TABLE or update supplemental logging for tables affected by an
ALTER TABLE to add or drop columns, are renames, or have unique key added or
dropped.
DEFAULTUSERPASSWORD is valid for Replicat. It specifies a different password for a
replicated {CREATE | ALTER} USER <name> IDENTIFIED BY <password> statement from
the one used in the source statement. The password may be entered as a clear text or
encrypted using the default or a user defined <keyname> from ENCKEYS. When using
DEFAULTUSERPASSWORD, use the NOREPLICATEPASSWORD option of DDLOPTIONS for
Extract.
GETAPPLOPS | IGNOREAPPLOPS are valid for Extract. This controls whether or not
DDL operations produced by business applications except Replicat are included in the
content that Extract writes to a trail or file. The default is GETAPPLOPS.
GETREPLICATES | IGNOREREPLICATES is valid for Extract. It controls whether or not
DDL operations produced by Replicat are included in the content that Extract writes
to a trail or file. The default is IGNOREREPLICATES.
REMOVECOMMENTS is valid for Extract and Replicat. It controls whether or not
comments are removed from the DDL operation. By default, comments are not
removed.
REMOVECOMMENTS BEFORE removes comments before the DDL operation is
processed by Extract or Replicat. AFTER removes comments after they are used for
string substitution.
REPLICATEPASSWORD is valid for Extract. It applies to the password in a {CREATE |
ALTER} USER <user> IDENTIFIED BY <password> command. By default GoldenGate uses
the source password in the target CREATE or ALTER statement. To prevent the source
password from being sent to the target, use NOREPLICATEPASSWORD.
Oracle GoldenGate Fundamentals Student Guide
Trail management
Monitoring
Troubleshooting
211
To implement security for GoldenGate commands, you create a CMDSEC file in the
GoldenGate directory. Without this file, access to all GoldenGate commands is
granted to all users. The CMDSEC file should be created and secured by the user
responsible for central administration of Extract/Replicat.
Comments begin with a pound sign (#), two hyphens (--), or the word COMMENT.
For the command security line, separate each of the following components with spaces
or tabs.
<command name> <command object> <OS group> <OS user>
<YES | NO>
Note: Command names and command objects are not validated for accuracy.
Oracle GoldenGate Fundamentals Student Guide
Can you see the error with the two STOP lines?
213
Since the CMDSEC file is the source of security, it must be secured. The administrator
must grant read access to anyone allowed access to GGSCI, but restrict write and
purge access to everyone but the administrator.
Trail Management
terminates, but the primary Extract group reading from logs or audit data continues
extracting data. It is not good practice to stop the primary Extract group to prevent
further accumulation. The transaction logs could recycle or the audit could be off-
loaded.
• For trails on the target system, data will accumulate because data is extracted and
transferred across the network faster than it can be applied to the target database.
To estimate the required trail space
1. Estimate the longest time that you think the network can be unavailable.
2. Estimate how much transaction log volume you generate in one hour.
3. Use the following formula:
trail disk space = <transaction log volume in 1 hour> x <number of
hours down> x .4
Note: The equation uses a multiplier of 40 percent because GoldenGate estimates
that only 40 percent of the data in the transaction logs is written to the trail. A more
exact estimate can be derived by configuring Extract and allowing it to run for a set
time period, such as an hour, to determine the growth. This growth factor can then be
applied to the maximum down time.
215
Managing: Trail Management – PURGEOLDEXTRACTS Parameter
Manager evaluation process for PURGEOLDEXTRACTS:
Parameter Explanation
FREQUENCYMINUTES | •If set, determines how often Manager purges old files.
FREQUENCYHOURS •If not set, defaults to the maintenance frequency set in the
Manager CHECKMINUTES parameter (default 10).
Syntax:
PURGEOLDEXTRACTS {<trail name> | <log table name> }
[, USECHECKPOINTS | NOUSECHECKPOINTS]
[, <minkeep rule>]
[, <frequency>]
Arguments:
USECHECKPOINTS (Default) Allows purging after all Extract and Replicat processes
are done with the data as indicated by checkpoints, according to any MINKEEP rules.
NOUSECHECKPOINTS Allows purging without considering checkpoints, based on
keeping a minimum of:
Either one file if no MINKEEP rule is used
Or the number of files specified with a MINKEEP rule.
<minkeep rule> Use only one of the following to set rules for the minimum amount of
time to keep data:
MINKEEPHOURS <n>
Keeps an unmodified file for at least the specified number of hours.
MINKEEPDAYS <n>
Keeps an unmodified file for at least the specified number of days.
MINKEEPFILES <n>
Keeps at least n unmodified files, including the active file.
Oracle GoldenGate Fundamentals Student Guide
<frequency> Sets the frequency with which to purge old trail files. The default time
for Manager to process maintenance tasks is 10 minutes, as specified with the
CHECKMINUTES parameter. Every 10 minutes, Manager evaluates the
PURGEOLDEXTRACTS frequency and conducts the purge after the specified
interval. <frequency> can be one of the following:
FREQUENCYMINUTES <n>
Sets the frequency, in minutes, with which to purge old trail files. The default
purge frequency is 60 minutes.
FREQUENCYHOURS <n>
Sets the frequency, in hours, at which to purge old trail files.
For example:
Trail files AA000000, AA000001, and AA000002 exist. Replicat has
been down for four hours and has not completed processing any of
the files.
The result:
The files have not been accessed for 4 hours so MINKEEP rule allows
purging, but checkpoints indicate the files have not been processed
so purge is not allowed.
Additional examples:
Example 1
Trail files AA000000, AA000001, and AA000002 exist. The Replicat has been down for
four hours and has not completed processing.
The Manager parameters include:
PURGEOLDEXTRACTS /ggs/dirdat/AA*, NOUSECHECKPOINTS,
MINKEEPHOURS 2
Result: All trail files will be purged since the minimums have been met.
Example 2
The following is an example of why only one of the MINKEEP options should be set.
Replicat and Extract have completed processing. There has been no access to the trail
files for the last five hours. Trail files AA000000, AA000001, and AA000002 exist.
The Manager parameters include:
PURGEOLDEXTRACTS /ggs/dirdat/AA*, USECHECKPOINTS,
MINKEEPHOURS 4, MINKEEPFILES 4
Result: USECHECKPOINTS requirements have been met so the minimum rules will be
considered when deciding whether to purge AA000002. There will only be two files if
AA000002 is purged, which will violate the MINKEEPFILES parameter. Since both
MINKEEPFILES and MINKEEPHOURS have been entered, however, MINKEEPFILES is
217
ignored. The file will be purged because it has not been modified for 5 hours, which
meets the MINKEEPHOURS requirement of 4 hours.
GETPURGEOLDEXTRACTS
SEND MANAGER option
Displays the rules set with PURGEOLDEXTRACTS
Syntax
SEND MANAGER
{CHILDSTATUS |
GETPORTINFO [DETAIL] |
GETPURGEOLDEXTRACTS |
KILL <process name>}
PurgeOldExtracts Rules
Fileset MinHours MaxHours MinFiles MaxFiles UseCP
S:\GGS\DIRDAT\EXTTRAIL\P4\* 0 0 1 0 Y
S:\GGS\DIRDAT\EXTTRAIL\P2\* 0 0 1 0 Y
S:\GGS\DIRDAT\EXTTRAIL\P1\* 0 0 1 0 Y
S:\GGS\DIRDAT\REPTRAIL\P4\* 0 0 1 0 Y
S:\GGS\DIRDAT\REPTRAIL\P2\* 0 0 1 0 Y
S:\GGS\DIRDAT\REPTRAIL\P1\* 0 0 1 0 Y
OK
Extract Trails
Filename Oldest_Chkpt_Seqno IsTable IsVamTwoPhaseCommit
S:\GGS\8020\DIRDAT\RT 3 0 0
S:\GGS\8020\DIRDAT\REPTRAIL\P1\RT 13 0 0
S:\GGS\8020\DIRDAT\REPTRAIL\P2\RT 13 0 0
S:\GGS\8020\DIRDAT\REPTRAIL\P4\RT 13 0 0
S:\GGS\8020\GGSLOG 735275 1 0
S:\GGS\8020\DIRDAT\EXTTRAIL\P1\ET 14 0 0
S:\GGS\8020\DIRDAT\EXTTRAIL\P2\ET 14 0 0
S:\GGS\8020\DIRDAT\EXTTRAIL\P4\ET 14 0 0
Oracle GoldenGate Fundamentals Student Guide
AUTOSTART
AUTOSTART ER *
AUTOSTART EXTRACT PROD*
Do not specify any batch task groups, such as one-time initial load groups.
AUTORESTART
AUTORESTART ER *, RETRIES 5, WAITMINUTES 3, RESETMINUTES 90
AUTOSTART
Manager parameter used to start one or more Extract or Replicat processes when
Manager starts. This can be useful at system boot time, for example, when you want
synchronization to begin immediately. You can use multiple AUTOSTART statements
in the same parameter file. The syntax is:
AUTOSTART <process type> <group name>
<process type> is one of the following: EXTRACT, REPLICAT, ER (Extract and Replicat)
<group name> is a group name or wildcard specification for multiple groups.
Example
AUTOSTART ER *
Note: Be careful to not include any batch tasks, such as initial load processes.
AUTORESTART
Manager parameter used to specify Extract or Replicat processes to be restarted by
Manager after abnormal termination. You can use multiple AUTORESTART statements
in the same parameter file. The syntax is:
AUTORESTART <process type> <group name>
[, RETRIES <max retries>]
[, WAITMINUTES <wait minutes>]
[, RESETMINUTES <reset minutes>]
<process type> Specify one of: EXTRACT, REPLICAT, ER (Extract and Replicat)
<group name> A group name or wildcard indicating the group names of multiple
processes to start.
219
RETRIES <max retries> is the maximum number of times that Manager should try to
restart a process before aborting retry efforts. The default is 2 retries.
WAITMINUTES <wait minutes> is the amount of time to pause between discovering that
a process has terminated abnormally and restarting the process. Use this option to
delay restarting until a necessary resource becomes available or some other event
occurs. The default delay is 2 minutes.
RESETMINUTES <reset minutes> is the window of time during which retries are
counted. The default is 20 minutes. After the time expires, the number of retries
reverts to zero.
Example
In the following example, Manager tries to start all Extract processes three times
after failure within a one hour time period, and it waits five minutes before each
attempt.
AUTORESTART EXTRACT *, RETRIES 3, WAITMINUTES 5,
RESETMINUTES 60
Error
Response (ABEND or RETRY)
Delay (in centiseconds)
Maximum Retries
Response
Controls whether or not GoldenGate tries to connect again after the defined error.
Valid values are either RETRY or ABEND.
Delay
Controls how long GoldenGate waits before attempting to connect again.
Max Retries
Controls the number of times that GoldenGate attempts to connect again before
aborting.
Oracle GoldenGate Fundamentals Student Guide
Changing TCPERRS
To alter the instructions or add instructions for new errors, open the file in a text
editor and change any of the values in the columns.
Reports overview
Each Extract, Replicat, and Manager process generates a standard report file that
shows:
parameters in use
221
table and column mapping
database information
runtime messages and errors
You can set up additional reports to be generated at a defined interval by using the
Extract and Replicat parameters REPORT, REPORTCOUNT, and
REPORTROLLOVER.
You can request additional reports by sending the request to running Extract and
Replicat processes.
REPORTCOUNT
- REPORTCOUNT EVERY 1000000 RECORDS
- REPORTCOUNT EVERY 30 MINUTES, RATE
- REPORTCOUNT EVERY 2 HOURS
REPORTROLLOVER
- REPORTROLLOVER AT 01:00
REPORT
Use REPORT to specify when Extract or Replicat generates interim runtime statistics in
a process report. The statistics are added to the existing report. By default, runtime
statistics are displayed at the end of a run unless the process is intentionally killed. By
default, reports are only generated when an Extract or Replicat process is stopped.
The statistics for REPORT are carried over from the previous report. For example, if the
process performed 10 million inserts one day and 20 million the next, and a report is
generated at 3:00 each day, then the first report would show the first 10 million inserts,
and the second report would show those plus the current day’s 20 million inserts, totaling
30 million. To reset the statistics when a new report is generated, use the
STATOPTIONS parameter with the RESETREPORTSTATS option.
Syntax
REPORT
{AT <hh:mi> |
ON <day> |
AT <hh:mi> ON <day>}
Where:
Oracle GoldenGate Fundamentals Student Guide
AT <hh:mi> generates the report at a specific time of the day. Using AT without ON
generates a report at the specified time every day.
ON <day> generates the report on a specific day of the week. Valid values are the days
of the week in text (e.g. SUNDAY).
REPORTCOUNT
Use REPORTCOUNT to generate a count of records that have been processed since
the Extract or Replicat process started. Results are printed to the report file and to
screen. Record counts can be output at scheduled intervals or after a specific number of
records. Record counts are carried over from one report to the other. REPORTCOUNT
can be used only once in a parameter file. If there are multiple instances of
REPORTCOUNT, GoldenGate uses the last one.
Syntax
REPORTCOUNT
[EVERY] <count>
{RECORDS | SECONDS | MINUTES | HOURS}
[, RATE]
REPORTROLLOVER
Use REPORTROLLOVER to define when the current report file is aged and a new one
is created. Old reports are renamed in the format of <group name><n>.rpt, where
<group name> is the name of the Extract or Replicat group and <n> is a number that
gets incremented by one whenever a new file is created, for example: myext0.rpt,
myext1.rpt, myext2.rpt, and so forth.
Note Report statistics are carried over from one report to the other. To reset the
statistics in the new report, use the STATOPTIONS parameter with the
RESETREPORTSTATS option.
Either the AT or ON option is required. Both options can be used together. Using AT
without ON generates a report at the specified time every day.
This parameter does not cause new runtime statistics to be written to the report. To
generate new runtime statistics to the report, use the SEND EXTRACT or SEND
REPLICAT command with the REPORT option. To control when runtime statistics are
generated to report files, use the REPORT parameter.
Syntax
REPORTROLLOVER
{AT <hh:mi> |
ON <day> |
AT <hh:mi> ON <day>}
223
Managing: Reporting – SEND and VIEW REPORT Commands
SEND REPORT
Use SEND REPORT to communicate with a running process and generate an interim
statistical report that includes the number of inserts, updates, and deletes output
since the last report. The request is processed as soon as it is ready to accept
commands from users.
Syntax:
REPORT
[HANDLECOLLISIONS [<table spec>] ]
VIEW REPORT
Use VIEW REPORT to view the process report that is generated by Extract or Replicat.
The report lists process parameters, run statistics, error messages, and other
diagnostic information.
The command displays only the current report. Reports are aged whenever a process
starts. Old reports are appended with a sequence number, for example finance0.rpt,
finance1.rpt, and so forth. To view old reports, use the [<n>] option.
Syntax
VIEW REPORT {<group name>[<n>] | <file name>}
<group name> The name of the group. The command assumes the report file named
<group>.rpt in the GoldenGate dirrpt sub-directory.
<n> The number of an old report. Report files are numbered from 0 (the most recent)
to 9 (the oldest).
<file name> A fully qualified file name, such as c:\ggs\dirrpt\orders.rpt.
Oracle GoldenGate Fundamentals Student Guide
Example 1
VIEW REPORT orders3
Example 2
VIEW REPORT c:\ggs\dirrpt\orders.rpt
Statistics overview
Extract and Replicat maintain statistics in memory during normal processing. These
statistics can be viewed online with GGSCI by issuing the STATS command. There are
many options with STATS, such as to reset the counters, or display only a brief totals
only, or to provide a per-table basis statistical report.
225
Managing: Statistics – STATS Command
STATS Command
Use STATS REPLICAT or STATS EXTRACT to display statistics for one or more groups.
Syntax:
STATS EXTRACT | REPLICAT <group name> or just STATS
<group name>
[, <statistic>]
[, TABLE <table>]
[, TOTALSONLY <table spec>]
[, REPORTRATE <time units>]
[, REPORTFETCH | NOREPORTFETCH]
[, REPORTDETAIL | NOREPORTDETAIL]
[, ... ]
<group name> The name of a Replicat group or a wildcard (*) to specify multiple
groups. For example, T* shows statistics for all groups whose names begin with T.
<statistic> The statistic to be displayed. More than one statistic can be specified by
separating each with a comma, for example STATS REPLICAT finance, TOTAL, DAILY.
Valid values are:
TOTAL Displays totals since process startup.
DAILY Displays totals since the start of the current day.
HOURLY Displays totals since the start of the current hour.
LATEST Displays totals since the last RESET command.
RESET Resets the counters in the LATEST statistical field.
TABLE <table>
Displays statistics only for the specified table or a group of tables specified with a
wildcard (*).
Oracle GoldenGate Fundamentals Student Guide
REPORTFETCH | NOREPORTFETCH
(Extract) Controls whether or not statistics about fetch operations are included in the
output. The default is NOREPRTFETCH.
REPORTDETAIL | NOREPORTDETAIL
(Replicat) Controls whether or not the output includes operations that were not
replicated as the result of collision errors. These operations are reported in the
regular statistics (inserts, updates, and deletes performed) plus as statistics in the
detail display, if enabled. For example, if 10 records were insert operations and they
were all ignored due to duplicate keys, the report would indicate that there were 10
inserts and also 10 discards due to collisions. The default is REPORTDETAIL.
227
Managing: Statistics – STATOPTIONS Parameter
REPORTDETAIL | NOREPORTDETAIL
Report statistics for collisions (Replicat)
REPORTFETCH | NOREPORTFETCH
Report statistics on row fetching (Extract)
RESETREPORTSTATS | NORESETREPORTSTATS
Resets statistics when a new report file is create by the
REPORTROLLOVER parameter
STATOPTIONS Parameter
Use STATOPTIONS to specify information to be included in statistical displays
generated by the STATS EXTRACT or STATS REPLICAT command. These options also can
be enabled as needed as arguments to those commands.
Syntax:
STATOPTIONS
[, REPORTDETAIL | NOREPORTDETAIL]
[, REPORTFETCH | NOREPORTFETCH]
[, RESETREPORTSTATS | NORESETREPORTSTATS]
REPORTDETAIL | NOREPORTDETAIL
Valid for Replicat. REPORTDETAIL returns statistics on operations that were not
replicated as the result of collision errors. These operations are reported in the
regular statistics (inserts, updates, and deletes performed) plus as statistics in the
detail display, if enabled. For example, if 10 records were insert operations and they
were all ignored due to duplicate keys, the report would indicate that there were 10
inserts and also 10 discards due to collisions. NOREPORTDETAIL turns off reporting of
collision statistics. The default is REPORTDETAIL.
REPORTFETCH | NOREPORTFETCH
Valid for Extract. REPORTFETCH returns statistics on row fetching, such as that
triggered by a FETCHCOLS
clause or fetches that must be performed when not enough information is in the
transaction record. NOREPORTFETCH turns off
reporting of fetch statistics. The default is NOREPORTFETCH.
RESETREPORTSTATS | NORESETREPORTSTATS
Controls whether or not report statistics are reset when a new process report is
created. The default of NORESETREPORTSTATS continues the statistics from one report
Oracle GoldenGate Fundamentals Student Guide
to another (as the process stops and starts or as the report rolls over based on the
REPORTROLLOVER parameter). To reset statistics, use RESETREPORTSTATS.
GoldenGate provides proactive messaging for process that are not running:
DOWNCRITICAL messages for failed processes
DOWNREPORT reminders for failed processes
Director Client can be configured to send automatic email alerts to operators or email
distribution groups for:
LATENCY ALERTS sends emails when latency thresholds that have been exceeded
MESSAGE ALERTS sends emails when any particular error message is generated
LAG Charts can be displayed to show history of average latency over a period of
time
229
Managing: Monitoring Processes
DOWNREPORT
Whenever a process starts or stops, events are generated to the error log, but those
messages can easily be overlooked if the log is large. DOWNREPORTMINUTES and
DOWNREPORTHOURS sets an interval for reporting on terminated processes. Only
abended processes are reported as critical unless DOWNCRITICAL is specified.
Syntax
DOWNREPORTMINUTES <minutes> | DOWNREPORTHOURS
<hours>
<minutes> The frequency, in minutes, to report processes that are not
running.
<hours> The frequency, in hours, to report processes that are not
running.
Example
The following generates a report every 30 minutes.
DOWNREPORTMINUTES 30
DOWNCRITICAL
Specifies that both abended processes and those that have stopped gracefully are
marked as critical in the ‘down’ report.
UPREPORT
Use UPREPORTMINUTES or UPREPORTHOURS to specify the frequency with which
Manager reports Extract and Replicat processes that are running. Every time one of
those processes starts or stops, events are generated. Those messages are easily
overlooked in the error log because the log can be so large. UPREPORTMINUTES and
UPREPORTHOURS report on a periodic basis to ensure that you are aware of the
process status.
Oracle GoldenGate Fundamentals Student Guide
Source
Database
Manager Manager
Target
Database
Write to Trail
System Time
Source Commit Timestamp
Pump lag
Replicat lag
“end-to-end” latency
Definitions:
Lag
This is the Extract lag or Pump lag in the diagram. It is the difference in time
between when a change record was processed by Extract (written to the trail) and the
timestamp of that record in the data source.
Latency
This is the Replicat lag in the diagram. It is the difference in time between when a
change is made to source data and when that change is reflected in the target data.
231
Managing: Monitoring Lag (cont’d)
LAGREPORT
Manager parameter, LAGREPORTMINUTES or LAGREPORTHOURS, used to specify the
interval at which Manager checks for Extract and Replicat lag. The syntax is:
LAGREPORTMINUTES <minutes> | LAGREPORTHOURS
<hours>
Example
LAGREPORTHOURS 1
LAGINFO
Manager parameter, LAGINFOSECONDS, LAGINFOMINUTES, or LAGINFOHOURS, used to
specify how often to report lag information to the error log. A value of zero (0) forces a
message at the frequency specified with the LAGREPORTMINUTES or LAGREPORTHOURS
parameter. If the lag is greater than the value specified with the LAGCRITICAL
parameter, Manager reports the lag as critical; otherwise, it reports the lag as an
informational message. The syntax is:
LAGINFOSECONDS <seconds> |
LAGINFOMINUTES <minutes> |
LAGINFOHOURS <hours>
Example
LAGINFOHOURS 1
Oracle GoldenGate Fundamentals Student Guide
LAGCRITICAL
Manager parameter, LAGCRITICALSECONDS, LAGCRITICALMINUTES, or
LAGCRITICALHOURS, used to specify a lag threshold that is considered critical and to
force a warning message to the error log when the threshold is reached. This
parameter affects Extract and Replicat processes on the local system. The syntax is:
LAGCRITICALSECONDS <seconds> |
LAGCRITICALMINUTES <minutes> |
LAGCRITICALHOURS <hours>
Example
LAGCRITICALSECONDS 60
Email Alerts
Director Client can be configured to send automatic email alerts to operators or email
distribution groups for:
MESSAGE ALERTS sends emails when any particular error message is generated
LATENCY ALERTS sends emails when latency thresholds that have been exceeded
LAG Charts can be displayed to show history of average latency over a period of time
233
Using the System Alert Setup page, choose to enter a new Alert, or change an existing
one, by selecting from the Select an Alert dropdown list. When the desired Alert is
displayed, complete the following fields:
Alert Name
This is a friendly name that you will use to refer to the Alert, it can be anything up to
50 characters.
Alert Type
Choose between the following options, note that some fields on the page will change,
depending upon what you select here.
Process Lag
This alert type allows you to specify a threshold for checkpoint lag, should lag go over
the threshold, notifications will be generated.
Event Text
This alert type allows you to specify event type or text, if the event matches,
notifications will be generated.
Instance Name
Choose a specific Manager Instance to apply the Alert criteria to, or you may make
the Alert global to all instances.
Process Name
Type the process name to apply the criteria to. You may use a wildcard (*) here to
make partial matches, for example:
'*' - would match all process names
'EXT*' - would match processes beginning with 'EXT'
'*SF*' - would match all processes with 'SF' anywhere in the name
Criteria
When Lag goes above
This field is displayed when you select a Process Lag Alert type. Enter the desired Lag
Threshold here.
When Event Type is
This field is displayed when you select an Event Text Alert type. Select the event
types (ERROR, WARNING) to match here.
When Event Text contains
This field is displayed when you select an Event Text Alert type. Enter the text to
match, leave blank or enter '*' to match any text.
Action
Send eMail to
Enter a comma-separated list of email addresses
Oracle GoldenGate Fundamentals Student Guide
Troubleshooting
Troubleshooting - Resources
GGSCI
The GGSCI command interface provides several helpful commands for
troubleshooting. For syntax, view the GGSCI online help by issuing the HELP
command, or see the alphabetical reference in the Oracle GoldenGate Reference
Guide.
Process Reports
Each Extract, Replicat, and Manager process generates a report file that shows:
• parameters in use
• table and column mapping
• database information
• runtime messages and error
Event/Error log
The error log is a file that shows:
• history of GGSCI commands
• processes that started and stopped
• errors that occurred
• informational messages
235
Discard file
GoldenGate creates a discard file when the DISCARDFILE parameter is used in the
Extract or Replicat parameter file and the process has a problem with a record it is
processing. The discard file contains column-level details for operations that a process
could not handle, including:
• the database error message
• the trail file sequence number
• the relative byte address of the record in the trail
• details of the discarded record
System logs
GoldenGate writes errors that occur at the operating system level to the Event Viewer
on Windows or the syslog on UNIX. On Windows, this feature must be installed.
Errors appearing in the system logs also appear in the GoldenGate error log.
Director
Most of the information viewed with GGSCI commands can also be viewed through
Director Client and Director Web, GoldenGate’s graphical user interfaces. For more
information about Director, see the Director online help.
INFO ALL shows status and lag for all Manager, Extract, and
Replicat processes on the system
The data source can be transaction log or trail/extract file for Extract, or trail/extract
file for Replicat.
Sample INFO DETAIL command output (some redo logs purposely omitted due to space
constraints):
Current directory
C:\GoldenGate\Oracle_10.1.10\win_ora101_v8020_019
Report file
C:\GoldenGate\Oracle_10.1.10\win_ora101_v8020_019\dirrpt\EXT_CTG1
.rpt
Parameter file
C:\GoldenGate\Oracle_10.1.10\win_ora101_v8020_019\dirprm\EXT_CTG1
.prm
Checkpoint file
C:\GoldenGate\Oracle_10.1.10\win_ora101_v8020_019\dirchk\EXT_CTG1
.cpe
Process file
C:\GoldenGate\Oracle_10.1.10\win_ora101_v8020_019\dirpcs\EXT_CTG1
.pce
Error log
C:\GoldenGate\Oracle_10.1.10\win_ora101_v8020_019\ggserr.log
237
Troubleshooting – Show Checkpoints
Read Checkpoint #1
Read Checkpoint #2
Write Checkpoint #1
Sequence #: 2
RBA: 2142224
Timestamp: 2006-06-09 14:16:50.567638
Extract Trail: ./dirdat/eh
Header:
Version = 2
239
Record Source = A
Type = 6
# Input Checkpoints = 2
# Output Checkpoints = 1
File Information:
Block Size = 2048
Max Blocks = 100
Record Length = 2048
Current Offset = 0
Configuration:
Data Source = 3
Transaction Integrity = 1
Task Type = 0
Status:
Start Time = 2006-06-09 14:15:14
Last Update Time = 2006-06-09 14:16:50
Stop Status = A
Troubleshooting – Recovery
Both Extract and Replicat restart after a failure at their last read
checkpoint
SEND EXTRACT STATUS command reports when Extract is
recovering
Checkpoint information is updated during the recovery stage
allowing you to monitor the progress with the INFO command
If an error prevents Replicat from moving forward in the trail, you
can restart Replicat after the ‘bad’ transaction:
To determine the CSN to use, view the Replicat report file with
the VIEW REPORT <group> command or view the trail with the
Logdump utility
Oracle GoldenGate Fundamentals Student Guide
Each Extract, Replicat, and Manager has its own report file that shows:
Banner with startup time
Parameters in use
Table and column mapping
Database and Environmental information
Runtime messages and errors
The report provides initial clues, such as invalid parameter, data mapping
errors, or database error messages.
View with:
VIEW REPORT <group> in GGSCI
Director provides single click to next or previous historical reports
241
Troubleshooting - Event Log (ggserr.log)
View with:
Standard text editor or shell command
GGSCI command VIEW GGSEVT
Oracle GoldenGate Director
The log’s name is ggserr.log, located in the root GoldenGate directory. You can also
locate the file using the INFO EXTRACT <group>, DETAIL command. The location of the
ggserr.log file is listed with the other GoldenGate working directories, as shown
below:
Discard file
The location of the discard file is set in either the Extract or Replicat parameter file
by using the DISCARDFILE parameter:
DISCARDFILE C:\GoldenGate\dirrpt\discard.txt, <OPTION>
Options are:
APPEND: adds new content to old content in an existing file
PURGE: purges an existing file before writing new content
MEGABYTES <n>: sets the maximum size of the file (default is 1 MB)
243
Technical Support
245