Extreme Replication:

Performance Tuning Oracle GoldenGate

UTOUG Fall Symposium 2015
Date: 9/23/2015
 Bobby Curtis
Douglasville, Georgia
Infrastructure Principle
IOUG (RACSIG/DI SIG), RMOUG, GAOUG

@dbasolved Work: bobby.l.curtis@accenture.com

Personal: curtisbl@gmail.com
http://dbasolved.com
Accenture Enkitec Group
Part of Accentures Infrastructure Services:
- 52,000 Oracle professionals
- 17,000 Infrastructure Services professionals

Oracle Database & Engineered Systems Solutions:

- Database Migrations
- Oracle Applications on Engineered Systems
- Cloud-based solutions leveraging Engineered Systems

Oracle Specialized in the following areas:

- Oracle Database, Oracle Database Security, Oracle Real Application Cluster (RAC),
Oracle Data Warehouse, Oracle Database Performance Turning, Oracle Linux
- Engineered Systems: Oracle Exadata, Oracle Exalogic, Oracle Exalytics
Accenture Enkitec Group
Our Accenture Enkitec Group resources are members of several Oracle
Database groups such as Oak Table, Oracle ACE, etc and speak frequently
at events across the globe (Oracle Use Groups (OUG), Oracle Open World
(OOW), etc..
Our team have authored several widely used books specific to Oracle
Database topics ranging from Exadata, performance & tuning to best
practices.
 Oracle GoldenGate Overview
Tuning Areas and Tools
Case Study
Recommendations
Improvements?
Q&A
Oracle GoldenGate Overview
Quick History
Change Data Capture Oracle Advance
Oracle Streams Oracle GoldenGate
(CDC) Replication

Synchronous
Asynchronous
Multi-Master Replication
MV Replication
Hybrid Replication
Capture Messages
Staging Messages
Consumption
Heterogeneous
Multiple Architectures
Multiple Use Cases
What is GoldenGate?
How does GoldenGate work?
 How Oracle GoldenGate Works
Capture (extract): committed transactions are captured (and can be filtered) as they occur by reading the transaction logs. As of
V.11.2.1, GoldenGate offers two options for capture for Oracle; Classic & Integrated Capture
Trail: stages and queues data for routing.
Pump: distributes data for routing to target(s).
Route: data is compressed, encrypted for routing to target(s).

Delivery: applies data with transaction integrity. New with

GoldenGate 12c, Integrated Delivery (replicat).

Trail Trail
Capture Pump
Files Files Delivery

MGR MGR
 How Oracle GoldenGate Works
Capture (extract): committed transactions are captured (and can be filtered) as they occur by reading the transaction logs. As of
V.11.2.1, GoldenGate offers two options for capture for Oracle; Classic & Integrated Capture
Trail: stages and queues data for routing.
Pump: distributes data for routing to target(s).
Route: data is compressed, encrypted for routing to target(s).

Delivery: applies data with transaction integrity. New with

GoldenGate 12c, Integrated Delivery (replicat).

Trail Trail
Capture Pump
Files Files Delivery

Trail Trail
Delivery Files Pump Files Capture
Integrated Extract
Introduced in GoldenGate 11.2.0.2
Integrated Extract for Oracle source databases only
Database release: 11.2.0.3 and later
Works with Logminer
Register Extract (capture) with database/logminer required
Example: GGSCI> register extract [ name ] database container [ (PDB) ]

Oracle Database

Trail
Capture
Files
 Logmining Server
Oracle Database

Preparer
Preparer
1 Trail
Reader 2
Preparer Builder Capture Capture
Files
N

Logmining Server Extract

Reader: Reads logfile and splits into regions Requests LCRs from logmining server
Preparer: Scans regions of logfiles and prefilters Performs mapping and
based on extract parameters transformations
Builder: Merges prepared records in SCN order Writes trail file
Capture: Formats LCR and passes to GG Extract
 Integrated Replicat
Introduced in GoldenGate 12.1.2
Integrated Replicat for Oracle databases only
Database release: 11.2.0.4, 12.1 and later
Leverages database parallel apply servers
Min. changes to replicat configuration
Single Replicat, no partitioning needed (via @RANGE/THREAD or manual
partitioning)
Oracle Database

Trail Delivery
Files
Integrated Replicat
Oracle Database

Apply
Trail Lightweight Streaming 1Apply
Delivery Receiver Preparer Coordinator 2Apply
Files API N

Replicat Inbound Server

Reads the trail file(s)
Constructs logical change records
(LCR)
Uses a lightweight streaming API to
transmit LCR to Oracle database
Receiver: Reads LCR
Preparer: Computes dependencies between
transactions (PK, FK, UK)
Coordinator: Maintains the order between
transactions
Applier: Applies transactions in order, including CDR
and error handling
Tuning Areas and Tools
Why should I care about performance?
What tools are available for performance tuning?
Oracle GoldenGate Performance Areas

Trail Trail
Capture Pump
Files Files Delivery

GoldenGate
GG tools: LAG, REPORTCOUNT GoldenGate

Database - AWR, ASH, UTL_SPADV, TRACE Database

Host - MPSTAT, VMSTAT, IOSTAT, STRACE, TOP Host

GoldenGate Performance Tools
LAG Information

Monitor latency from the Manager process (mgr.prm)

- LAGINFO [ SECONDS | MINUTES | HOURS ]
- LAGREPORT [ MINUTES | HOURS ]
- LAGCRITICAL [ SECONDS | MINUTES | HOURS]

EXAMPLE:
LAGINFOSECONDS 1
LAGREPORTMINUTES 1
LAGCRITICALSECONDS 2

Information output to ggserr.log

 GoldenGate Performance Tools
Report Files
Parameters currently running
Table/Column Mappings
Runtime messages and errors
Runtime statistics
REPORTCOUNT
- REPORTCOUNT EVERY [ # ] [ SECONDS | MINUTES |
HOURS ], RATE
Example: REPORTCOUNT EVERY 15 MINUTES, RATE

Files located in $OGG_HOME/dirrpt

Database Performance Tools
Automatic Workload Repository (AWR)
Good starting point within the database
Help determine potential bottlenecks
Extract/Replicat are given unique SQL Module IDs
- Used in both AWR and ASH
Use Active Session History (ASH) for more details on
identified session
AWR Output Replication Stats
Oracle 12c Database (12.1.0.1 and later)

Note: Will not be in 11.2.x AWR reports

Database Performance Tools
Streams Performance Advisor (Integrated Extract/Replicat Only)

Used to provide information on the performance of integrated

processes
Both Extract and Replicat

Provides real-time statistics

Has to be installed
- Installed under GoldenGate User
- $ORACLE_HOME/rdbms/admin/utlspadv.sql
Database Performance Tools (Views)
Run Time Views Configuration Views
V$GOLDENGATE_CAPTURE DBA_GOLDENGATE_PRIVILEGES
V$GG_APPLY_RECEIVER DBA_GOLDENGATE_SUPPORT_MODE
V$GG_APPLY_READER DBA_CAPUTRE
V$GG_APPLY_COORDINATOR DBA_CAPTURE_PARAMETERS
V$GG_APPLY_SERVER DBA_GOLDENGATE_INBOUND
V$GOLDENGATE_TABLE_STATS DBA_GG_INBOUND_PROGRESS
V$GOLDENGATE_CAPABILITIES DBA_APPLY
V$DBA_APPLY_ERRORS DBA_APPLY_PARAMETERS
DBA_APPLY_REPERROR_HANDLERS
DBA_APPLY_HANDLE_COLLISIONS
DBA_APPLY_DML_CONF_HANDLERS

Associated GoldenGate Monitoring Scripts

Health Checks
SQL based scripts that create HTML output
- Different scripts for database releases
- MOS:1448324.1

Summary
- Overview of environment
- Advice/Warnings of potential issues of configuration

Analysis
- Compare configurations
- Performance Recommendations
- Detail information for diagnostic purposes

Statistics
- Runtime information of Streams (GoldenGate) processing
Host Performance Tools
CPU Performance (mpstat/top)
Looking to see if there are any processes limited by CPU (top)
Looking to see what average cpu spike is (mpstat)
I/O Performance (iostat)
Gather for database disks and trail file locations
Critical before increasing parallelism
Memory Performance (vmstat)
Not needed for integrated processes
Needed if tuning classic or coordinated processes
Case Study
Server/Database Configs

Trail Pump Trail

Capture Delivery
Files Files

Dell PowerEdge T110 Dell PowerEdge T110 II

Intel Core Duo Intel i3
16G DRAM 8G DRAM
Database: Database:
SGA: 6GB SGA: 6GB
Oracle GoldenGate Performance Areas

Trail Trail
Capture Pump
Files Files Delivery

GoldenGate
GG tools: LAG, REPORTCOUNT GoldenGate

Database - AWR, ASH, UTL_SPADV, TRACE Database

Host - MPSTAT, VMSTAT, IOSTAT, STRACE Host

Lag Information (OEM)

Approx. 1 hr. 20 min. behind

JAGENT 12.1.3/GG Plugin 12.1.0.2 is required

Lag Information
ggserr.log
- Reported by Manager Process (mgr.prm)
- Extract (Source)
2015-02-11 15:32:12 INFO OGG-00948 Oracle GoldenGate
Manager for Oracle, mgr.prm: Lag for EXTRACT EXTI is 00:00:01
(checkpoint updated 00:00:07 ago).

- Replicat (Target)
2015-02-11 15:24:14 WARNING OGG-00947 Oracle GoldenGate
Manager for Oracle, mgr.prm: Lag for REPLICAT REPI is 01:18:50
(checkpoint updated 00:00:00 ago).
Process Statistics
Monitor throughput for each process
- REPORTCOUNT EVERY [ SECOND | MINUTES | HOURS ], RATE
- Can impact throughput with aggressive monitoring

EXTI.rpt
1068208 records processed as of 2015-02-11 15:30:25 (rate 57,delta 299)
PMPI.rpt
545130 records processed as of 2015-02-11 15:35:28 (rate 109,delta 281)
REPI.rpt
108483 records processed as of 2015-02-11 15:28:05 (rate 116,delta 120)

Total records Throughput for session Changes since last

processed this session report count
Streams Performance Advisor
Extract
PATH 1 RUN_ID 1 RUN_TIME 2015-JAN-12 15:17:31 CCA Y
|<C> OGG$CAP_EXTI 31 31 0 LMR 99.7% 0% 0.3% "" LMP (2) 199.7% 0%
0.3% "" LMB 99.3% 0% 0.3% "" CAP 99.7% 0% 0.3% "" |<Q>
"GGATE"."OGG$Q_EXTI" 0.01 0.01 0 |<A> OGG$EXTI 0.01 0.01 0 |<B>
NO BOTTLENECK IDENTIFIED

Logminer Reader (LMR): 99.7% idle, no flow control, no top event

Logminer Preparer (LMP): 2, 199.7% idle, no flow control, no top event
Logminer Builder (LMB): 99.3% idle, no flow control, no top event
Capture (CAP): 99.3% idle, no flow control, no top event
Streams Performance Advisor
Replicat
PL/SQL procedure successfully completed.

SQL>

Advisor couldnt gather required statistics

Performance (OEM View)
Trail Pump Trail
Capture Delivery
Files Files
GoldenGate Configuration
Trail Pump Trail
Capture Delivery
Files Files

--CHECKPARAMS --CHECKPARAMS --CHECKPARAMS

EXTRACT EXTI EXTRACT PMPI REPLICAT REPI
SETENV userid ggate password ggate SETENV
(ORACLE_HOME="/oracle/app/product/11.2.0.4/d PASSTHRU (ORACLE_HOME="/opt/oracle/app/product/12.1.0.
bhome_1") RMTHOST 192.168.65.68, MGRPORT 15000, 1/dbhome_1")
SETENV (ORACLE_SID="bc11g") COMPRESS SETENV (ORACLE_SID="oemrep")
USERID ggate, PASSWORD ggate RMTTRAIL ./dirdat/ri USERID ggate, PASSWORD ggate
WARNLONGTRANS 15m, CHECKINTERVAL 5m REPORTCOUNT EVERY 1 MINUTES, RATE ALLOWDUPTARGETMAP
REPORTCOUNT EVERY 5 MINUTES, RATE TABLE SOE.ADDRESSES; ASSUMETARGETDEFS
EXTTRAIL ./dirdat/li TABLE SOE.CARD_DETAILS; REPORTCOUNT EVERY 5 MINUTES, RATE
TABLE SOE.ADDRESSES; TABLE SOE.CUSTOMERS; DISCARDFILE ./dirrpt/REPCI.dsc, append,
TABLE SOE.CARD_DETAILS; TABLE SOE.INVENTORIES; megabytes 200
TABLE SOE.CUSTOMERS; TABLE SOE.LOGON; REPERROR(default, discard)
TABLE SOE.INVENTORIES; TABLE SOE.ORDER_ITEMS; REPERROR(default2, discard)
TABLE SOE.LOGON; TABLE SOE.ORDERENTRY_METADATA; MAP SOE.ADDRESSES, TARGET
TABLE SOE.ORDER_ITEMS; TABLE SOE.ORDERS; SOE.ADDRESSES;
TABLE SOE.ORDERENTRY_METADATA; TABLE SOE.PRODUCT_DESCRIPTIONS; MAP SOE.CARD_DETAILS, TARGET
TABLE SOE.ORDERS; TABLE SOE.PRODUCT_INFORMATION; ..
TABLE SOE.PRODUCT_DESCRIPTIONS; TABLE SOE.WAREHOUSES; MAP SOE.WAREHOUSES, TARGET
TABLE SOE.PRODUCT_INFORMATION; SOE.WAREHOUSES;
TABLE SOE.WAREHOUSES;
AWR Output Replication Stats

LAG: 12.9 Minutes

Database Views
select capture_name, 86400 * (available_message_create_time -
capture_message_create_time) latency_in_seconds
from gv$goldengate_capture;

select r.apply_name, 86400 * (r.dequeue_time - c.lwm_message_create_time)

latency_in_seconds
from gv$gg_apply_reader r, gv$gg_apply_coordinator c
where r.apply# = c.apply#
and r.apply_name = c.apply_name;
Recommendations
 Integrated Extract Parameters
LOGALLSUPCOLS
- Instructs extract to write supplemental logged columns to trail file

UPDATERECORDFORMAT
- Single Logical Change Record (LCR) with BEFORE and AFTER
images
- COMPACT
- reduces the amount of data sent with LCR
 Integrated Extract Parameters
Continued

PARALLELISM
- Controls number of preparers for processing logs
- Increase parallelism if preparers are CPU bound
- EE allows parallelism of preparers (default = 2)
- SE does not allow parallelism of preparers (default = 1)

MAX_SGA_SIZE
- Controls amount of streams pool configured extract
- Min: 1024 (1G)/Max: 3584 (3.5G)
Integrated Replicat Parameters
COMMIT_SERIALIZATION
- Default = DEPENDENT_TRANSACTIONS
- Set to FULL, if apply in source commit order is required
EAGER_SIZE
- Threshold to begin apply of large transactions (default = 9500)
- Serializes apply processing
- Make sure streams_pool_size is configured correctly to avoid
Waiting for Memory issues
MAX_SGA_SIZE
- Controls memory resource for IR
- Default is INFINITE
Integrated Replicat Parameters
Continued

PARALLELISM
- Controls number of appliers (default = 4, actually 50)
- Setting to 1 disables parallism
MAX_PARALLELISM
- Controls maximum number of appliers
- Default is 50 in OGG 12.1.2.1 (30 in OGG 12.1.2.0)
Note: Auto Tuning (parallelism)
- Enabled by default
- Computed over range of 5 intervals (5 seconds)
- Unneeded processes marked INACTIVE, removed after 5 minutes
- MAX_PARALLISM=PARALLELISM, disables auto turning
 Integrated Replicat Parameters
Continued

BATCHSQL
- Controls number of appliers (default = 4)
- Setting to 1 disables parallelism
- Use with caution
- Degradation can be seen in some cases.
Database Configurations
ENABLE_GOLDENGATE_REPLICATION
- New in DB 11.2.0.4/12.1.0.2
- Boolean value. Default to False, set to True to activate
STREAMS_POOL_SIZE
- Size appropriately for number of concurrent IE & IR processes
- Min. 1G per IE/IR then add 25%
- If using MAX_SGA_SIZE
- sum of (MAX_SGA_SIZE * IR) * 25%
 Sizing Streams Pool

(MAX_SGA_SIZE * PARALLELISM) * 25%

(1024M * 2) * 25% = 2560M (2.5G)

Note: Max size of streams_pool_size should not exceed 3.5G

Note: May have to increase SGA size to allocate enough
memory
 Database Object Tuning
GoldenGate uses SQL to apply changes
AWR highlights SQL and SEGMENT issues
ADDM recommendations as needed
Redo Logs, ensure they are large enough for expected
workload
Gather statistics regularly, especially after initial initialization
Target side: Indexes can cause slowness in apply process as
well.
Network Tuning
TCPBUFSIZE and TCPFLUSHBYTES
Use ping to find average round trip time (RTT)
[oracle@oel dirrpt]$ ping fred.acme.com
PING fred.acme.com (192.168.65.68) 56(84) bytes of data.
64 bytes from fred.acme.com (192.168.65.68): icmp_seq=1 ttl=64 time=0.180 ms
64 bytes from fred.acme.com (192.168.65.68): icmp_seq=2 ttl=64 time=0.173 ms
64 bytes from fred.acme.com (192.168.65.68): icmp_seq=3 ttl=64 time=0.199 ms
64 bytes from fred.acme.com (192.168.65.68): icmp_seq=4 ttl=64 time=0.203 ms
^C
--- fred.acme.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3807ms
rtt min/avg/max/mdev = 0.173/0.188/0.203/0.021 ms
 Network Tuning
(megabits / gigabits) * RTT = Bandwidth-Delay Product (BDP in bytes)
BDP * 3 = TCPBUFSIZE/TCPFLUSHBYTES

(8192 / 8) * 0.021 = 21.5 bytes

21.5 bytes * 3 = 64.5 bytes

Since BDP result is less than 1MB, no need to set TCPBUFSIZE or TCPFLUSHBYTES

*Default is 1MB
Host

Evaluate if you need bigger/newer hardware if needed!

Improvements
Configuration
Trail Pump Trail
Capture Delivery
Files Files

EXTRACT EXTI --CHECKPARAMS REPLICAT REPI

SETENV EXTRACT PMPI SETENV
(ORACLE_HOME="/oracle/app/product/11.2.0.4/d userid ggate password ggate (ORACLE_HOME="/opt/oracle/app/product/12.1.0.
bhome_1") PASSTHRU 1/dbhome_1")
SETENV (ORACLE_SID="bc11g") RMTHOST 192.168.65.68, MGRPORT 15000, SETENV (ORACLE_SID="oemrep")
USERID ggate, PASSWORD ggate COMPRESS USERID ggate, PASSWORD ggate
WARNLONGTRANS 15m, CHECKINTERVAL 5m RMTTRAIL ./dirdat/ri ALLOWDUPTARGETMAP
REPORTCOUNT EVERY 1 MINUTES, RATE REPORTCOUNT EVERY 1 MINUTES, RATE ASSUMETARGETDEFS
EXTTRAIL ./dirdat/li TABLE SOE.ADDRESSES; REPORTCOUNT EVERY 1 MINUTES, RATE
TRANLOGOPTIONS EXCLUDEUSER GGATE TABLE SOE.CARD_DETAILS; DISCARDFILE ./dirrpt/REPCI.dsc, append,
TRANLOGOPTIONS INTEGRATEDPARAMS TABLE SOE.CUSTOMERS; megabytes 200
(MAX_SGA_SIZE 2560 PARALLELISM 4) TABLE SOE.INVENTORIES; REPERROR(default, discard)
LOGALLSUPCOLS TABLE SOE.LOGON; REPERROR(default2, discard)
UPDATERECORDFORMAT COMPACT TABLE SOE.ORDER_ITEMS; DBOPTIONS INTEGRATEDPARAMS
TABLE SOE.ADDRESSES; TABLE SOE.ORDERENTRY_METADATA; (MAX_SGA_SIZE 2560, PARALLELISM 4,
TABLE SOE.CARD_DETAILS; TABLE SOE.ORDERS; MAX_PARALLELISM 6,
TABLE SOE.CUSTOMERS; TABLE SOE.PRODUCT_DESCRIPTIONS; COMMIT_SERIALIZATION FULL)
.. TABLE SOE.PRODUCT_INFORMATION; MAP SOE.ADDRESSES, TARGET
TABLE SOE.PRODUCT_INFORMATION; TABLE SOE.WAREHOUSES; SOE.ADDRESSES;
TABLE SOE.WAREHOUSES; ;
MAP SOE.WAREHOUSES, TARGET
SOE.WAREHOUSES;
Performance (OEM View)
Trail Pump Trail
Capture Delivery
Files Files
 Report Counts
EXTI.rpt
1015083 records processed as of 2015-02-14 01:49:21 (rate 90,delta 301)
1034022 records processed as of 2015-02-14 01:50:22 (rate 91,delta 310)
1052135 records processed as of 2015-02-14 01:51:22 (rate 92,delta 301)
PMPI.rpt
1033398 records processed as of 2015-02-14 01:50:20 (rate 91,delta 304)
1052112 records processed as of 2015-02-14 01:51:21 (rate 92,delta 310)
1069906 records processed as of 2015-02-14 01:52:21 (rate 93,delta 294)
REPI.rpt
122370 records processed as of 2015-02-14 01:40:33 (rate 502,delta 884)
188385 records processed as of 2015-02-14 01:41:34 (rate 620,delta 1099)
317619 records processed as of 2015-02-14 01:42:35 (rate 872,delta 2143)

Changes since last

report count
Comparison
2500

2143
2000

1500

Series1 94.4% increase

1000
Series2 in replicat
299 301 performance
500

120
0

EXTI
REPI
STOP
Call for papers open now @ http://gaoug.strikingly.com/
 Further Reading
MOS Note: 1557031.1 patches for integrated capture and replicat
MOS Note: 1448324.1 OGG Integrated Healthcheck Script
MOS Note: 1485620.1 Best Practices for Downstream Capture
MOS Note: 1488668.1 GoldenGate Performance Data Gathering

Oracle GoldenGate Best Practices:

http://www.oracle.com/technetwork/database/availability/maa-gg-performance-
1969630.pdf
Replicat Improvement Ratio

((RUN 2 REP RUN 1 REP) / RUN 2 REP) * 100

((2143 - 120) / 2143) * 100 = 94.40%