Optimizing Oracle E-Business Suite on Exadata

John Clarke
Oracle Architect
Centroid

Now What?

About the Presenter

John Clarke, Sr. Oracle Architect at Centroid Systems, Inc.
Specialized in:
Oracle Exadata
Performance Tuning
Oracle Real Application Clusters
Data Warehousing
Oracle Linux
Oracle Database 11g
john.clarke@centroid.com
www.centroid.com
Blogs at www.centroid.com/knowledgebase
Sporadically adds random Oracle content at
twitter.com/#!/jccclarke

Optimizing EBS on Exadata

One Size
Fits all

Tuning EBS

On Exadata

Why Not?

Best
Practices

Some Performance-Related Things to Think About with

Oracle EBS and Exadata

Todays Goal

Exadata fundamentals
What makes Exadata Different
Exadata Software Features
How the Features work with EBS

Exadata Hardware
Compute Grid
Dual-core x64 servers running
Oracle 11gR2

InfiniBand Network
Redundant 40 Gb/s switches
Unified fabric for storage and
servers

Storage Grid
12 HP or HC SAS drives
384Gb PCIe Flash/cell
Data mirrored across cells with
ASM

Exadata Software
Exadata Software Goals

Fully and evenly utilize all

Computing resources in

Exadata Database Machine

to eliminate bottlenecks and

deliver consistent high

performance

Reduce demand for resources

by eliminating any IO that can
be discarded without impacting

result

Smart Scan
Probably the most important software feature with Exadata
One of several cell offload features in Exadata
cell offload is defined as Exadatas shifting of database work
from the database servers to Exadata storage servers
Goal of Smart Scan is to perform the majority of IO processing on
storage servers and return smaller amounts of data from the
storage infrastructure to the database servers
IO no longer a bottleneck
Provides dramatic performance improvements for eligible SQL
operations (full table scans, fast full index scans, joins, etc.)
Smart Scan is implemented automatically on Exadata
Smart Scan only works on Exadata

Traditional SQL Processing

User submits a query

select flow_status_code,count(*)!
from oe_order_headers!
group by flow_status_code!

Oracle filters rows and

columns, once loaded to buffer
cache, and returns to user

Database instance processes

blocks all required blocks
read from storage system to
buffer cache

The query is parsed and an

execution path is determined.
Extents are identified

IO is issued Oracle retrieves

blocks based on extents
identified. A block is the
smallest unit of transfer from
storage system, contains rows
and columns

Smart Scan Processing

User submits a query

select flow_status_code,count(*)!
from oe_order_headers!
group by flow_status_code!

The database consolidates

results from across each
Exadata cell and returns rows
to the client

Each Exadata cell returns to

the database instance an iDB
message containing the
requested rows and columns.
These are not block images
and are returned to the PGA

An iDB command is
constructed and sent to
Exadata cells

Each Exadata cell scans data

blocks and extracts relevant
rows and columns that satisfy
the SQL query

IO: Non-Exadata

IO: Exadata

Smart Scan
Filtering operations are offloaded to the Exadata storage cell
Column Filtering
Predicate Filtering (i.e., row filtering)
Join Filtering
Only requested rows and columns are returned to the database
server
Significantly less IO transfer over storage network
Less memory and CPU required on database tier nodes
Rows/columns retrieved to users PGA via direct path read
mechanism, not through buffer cache.
Large IO requests dont saturate buffer cache

What Operations are Eligible for Smart Scan?

Single-table full table scans ***

Fast full index scans ***
Scans must qualify for serial or parallel direct reads !!!
Join filtering using Bloom Filters
Look at V$SQLFN_METADATA
Oracle is a C program
kcfis_read is the function
that does smart scan
kcfis_read is called by
direct path read function,
klbldrget
klbldrget is called from full
scan functions

Smart Scans and Oracle EBS

CBO must choose full scan
Must qualify for direct reads
Direct reads are influenced by:
Size of segment being scanned, and
Whether BLOCKS > _small_table_threshold, and
How many blocks in buffer cache (maybe), and
How many dirty blocks in buffer cache
So segments must be large, and not already read into buffer cache
by other means (i.e., large index range scans)
Check out http://www.centroid.com/knowledgebase/blog/direct-reads-and-smart-scan-with-exadata

Digression: The History of Tuning Oracle EBS

Oracle launches
EBS, long time ago

Performance is
largely IO waits

Customers around
the world have
performance issues

Oracle over-indexes
and over-hints to
force index scans

Oracle EBS code is index-heavy and hint-heavy

Oracle code tries to make every problem a single-row access
problem after all, its an OLTP application, right?
On Exadata, IO no longer the problem (or is it?)
But code-base doesnt make it easy for Smart Scan to kick in

Smart Scans and _small_table_threshold

_small_table_threshold = 2% db_cache_size
Direct reads are done in 11g when blocks >
_small_table_threshold
Large cache sizes mean fewer Smart Scan qualifiers
Exadata DBCA templates typically use 16GB SGA_TARGET
DB_CACHE_SIZE usually 10-12GB after cache warming
(10GB * 2% ) / 8k = 26,244 blocks this is how large our
segments need to be for Smart Scan

Smart Scan Qualifiers, Mid-Sized EBS Environment

Smart scan qualifiers = full scans with direct read

Check DBA_HIST_SEG_STAT.TABLE_SCANS_TOTAL
Join to DBA_SEGMENTS, get BLOCKS
Determine your target buffer cache size on Exadata
Test case: 98 segments had full table or index scans
19 of these would qualify for direct reads with 10G buffer cache
35 would qualify for direct reads with 4GB buffer cache
Further analysis required to correlate with programs, sessions,
business application logic, etc.

Smart Scan Qualifiers, Large-Sized EBS Environment

Smart scan qualifiers = full scans with direct read

Check DBA_HIST_SEG_STAT.TABLE_SCANS_TOTAL
Join to DBA_SEGMENTS, get BLOCKS
Determine your target buffer cache size on Exadata
Test case: 760 segments had full table or index scans
227 of these would qualify for direct reads with 10G buffer cache
329 would qualify for direct reads with 4GB buffer cache
Further analysis required to correlate with programs, modules

Smart Scan and Multi-Org

In R12, Multi-Org uses SYS_CONTEXT call and RLS
SYS_CONTEXT not offloadable (check V$SQLFN_METADATA)
Full scans can be offloaded, but row-filtering done via RLS and
handled on compute node

Influencing Smart Scan

DISCLAIMER
Make buffer cache smaller (!)
Test with _serial_direct_read=ALWAYS
Make indexes invisible
Test with optimizer_ignore_hints=TRUE
These are all hacks, except possibly the index tip

Smart Scan: Is hope lost with EBS?

Local buffer cache access isnt such a bad thing, and often
appropriate

Smart Flash Cache can help

Configuring hugepages can help grow even larger
Each compute node has 96GB RAM
Think about mixed workloads and EBS more on this later

Smart Scan Test Case, dropping OLTP indexes

Load 100k rows into MTL_SYSTEM_ITEMS_INTERFACE
Measure inventory/cost transaction worker request timings
Gather AAS/AWR data

Smart Scan Test Case, dropping OLTP indexes

Load 100k rows into MTL_SYSTEM_ITEMS_INTERFACE
Make INV/BOM indexes invisible and run it

Yikes! Force serial direct reads

Smart Scan Test Case, dropping OLTP indexes

Run AWR Compare

Smart Scan Test Case, dropping OLTP indexes

Do we give up?
Make MTL_TRANSACTION_ACCOUNTS_N1 visible

Only 1 index was causing the issue

Dropping the rest saves a ton of space!
But TEST!!!

Smart Scan Test Case, Material Transactions Form

Test: Query material transactions based on a date
Enter a date range in which we know there are a lot of
transactions
Enable trace
Make buffer cache small

Smart Scan Test Case, Material Transactions Form

First Test: Query transactions for 2 different date ranges, one with
many rows, one with few

No Smart Scan

SBR/MBR

Smart Scan Test Case, Material Transactions Form

Second Test: Disable Indexes on MMT and Repeat

Still No Smart Scan !!!

SBR/MBR

Smart Scan Test Case, Material Transactions Form

3rd Test: Run same query from SQL*Plus

Smart Scan hmm

Smart Scan

Smart Scan Test Case, Material Transactions Form

4th Test: Set _serial_direct_read=always, test from Form

Smart Scan!
SBR/MBR

Smart Scan: Forcing Serial Direct Reads

Whats the impact of forcing serial direct reads?

Smart Scan Test Case, Material Transactions Form

What does it all mean?
Making indexes invisible allows full scans
Even though our block count made it eligible for cell offload, serial
direct reads didnt happen
Setting _serial_direct_read=always makes Smart Scan work
Why is this so hard to get the result we want?
the serial direct read algorithms are complicated
Segment size is important
Cached blocks also important
Buffer cache load plays a role
If you force it, be wary of concurrency in Oracle EBS

Measuring Smart Scan

Execution plans have TABLE ACCESS STORAGE FULL or
INDEX STORAGE FAST FULL SCAN, etc
V$SQL.IO_CELL_OFFLOAD_ELIGIBLE_BYTES > 0
Statistics from V$SYSSTAT, AWR
cell physical IO bytes eligible for predicate offload
cell physical IO bytes returned by smart scan
See
http://www.centroid.com/knowledgebase/blog/exadata-smartscan-processing

Hybrid Columnar Compression

The Basics:
Only available on Exadata ***
Data is compressed by column, not row, and stored in
Compression Unit (CU)
CU = set of blocks
Standard or Advanced compression unit = database block
HCC compression unit = CU
Compression ratios are much better than standard or OLTP
compression
Better compression = less disk space = sometimes better
performance

Hybrid Columnar Compression

What You Need to Know:
HCC compressed when loaded via direct path loads
HCC achieves higher compression because:
Data is grouped into arrays upon direct path loads the input
set is larger, giving Oracle more data to consider for dupes
Compression algorithms are more aggressive than other
Oracle compression flavors
DML against HCC tables
Does a row migration for current row and inserts into a OLTPcompressed block
Locks the entire CU
Selects retrieve column per CU, not entire block
When retrieved via SS, decompression happens on cell. When
done via SBR, decompression occurs on database server

Hybrid Columnar Compression

Hybrid Columnar Compression

What it Means for EBS

You only want to HCC-compress read-only data

think about old partitions and ILM policies
Dont compress with HCC if youre doing DML
Target candidates that full-scan via Smart Scan otherwise, CPU
penalty on compute node

Hybrid Columnar Compression and EBS

HCC can save massive amounts of disk space

HCC can improve performance
Partition large EBS tables according to ILM policies
HCC-compress old partitions
Examples
Partition GL_BALANCES and GL_JE_LINES by
PERIOD_NAME
Partition GL_IMPORT_REFERENCES by JE_HEADER_ID
Composite-Partition XLA_AE_HEADERS, XLA_AE_LINES,
and XLA_EVENTS by
(APPLICATION_ID,ACCOUNTING_DATE)
.. And more

HCC Case Study

Partition and HCC-compressed based on previous example
HCC compress for query high for all partitions containing data
from 1995 through 2009
Leaving 2010 and 2011 partitions uncompressed
Results
10x disk space savings
4x PIO savings for Trial Balance, Journal Entries Detail, and
Create Accounting concurrent programs
6x PIO savings for drill-down month end reports
CPU/LIO savings =~ 10-20%
Concurrent request improvement =~ 40% *** FND_FILE.PUT
SQL performance improvement =~ 70%

IORM

IORM = IO Resource Management

Means to meter IO to Exadata cells
Works with DBRM
If consolidating, IORM is imperative
If not consolidating, DBRM/IORM still makes sense!

Dont be afraid

Why IORM?

Many databases
Different types of users
Different workloads
Different SLAs
Over-provision storage ?

but how economical is this?

IORM is the answer

IORM
Some basics
Resource consumer groups map sessions by workload
Combo of consumer groups and session mappings = resource
plan
Consumer groups assigned to categories key for IORM
An Exadata IORM plan is based on a database plan + a category
plan

IORM and EBS

DESIGN and PLANNING are critical consider example
PROD, DEV, and TEST
PROD should take priority
PROD is comprised of:
OLTP/online users => EBS_OLTP CG
Reporting users = EBS_REPORTING
DEV and TEST take back-seat
Same types of consumers as PROD
Different % allocations per consumer

IORM and EBS: Step 1:

IORM and EBS: Step 2:

IORM and EBS: Step 3:

IORM and EBS: Things to Think About

Do you have a dedicated reporting environment?
How robust is it?
How much time did it take to build and test?
Dont you wish you didnt have one?
IORM = ability to run production reporting in production and not
worry about it!
Same can be said about DBRM? Maybe

Storage Indexes
Storage indexes <> normal indexes. Not at all
Goal: Eliminate unnecessary IO
Not by directing you specifically to table rows/blocks, but
By telling you with absolute certainty where not to look

Storage Indexes
Data stored in ASM Allocation Unit
Each 4M AU comprised of four 1MB storage regions
Storage regions track high and low values for columns in region
indexes
Region indexes are your storage indexes

Storage Indexes
When blocks access via Smart Scan, Exadata automatically
tracks high and low values in region index
When queries access blocks, Exadata cellsrv checks these
region indexes to see if any rows are contained in the storage
region if not, IO to storage region is skipped

Things to Know about Storage Indexes

Maintained over time as data is accessed via Smart Scan
Region indexes maintain high and low values for up to 8 columns
on a table
Storage indexes are automatically maintained by Exadata
theres nothing you can do to directly influence them
Theyre just memory structures not persistent across cell
reboots
Can return false-positives, never false negatives
Can provide dramatic performance under the right conditions:
Need Smart Scan
Data needs to be well-ordered to offer elimination candidates

Storage Indexes and Oracle EBS

Need Smart Scan (serial direct reads, no index lookups, etc.)
Since data needs to be well-ordered, FIFO columns like
TRANSACTION_DATE, ORDER_DATE, ACCOUNTING_DATE,
etc. may work well
Probably shouldnt rely on them for optimization difficult to
predict and not deterministic

Storage Index Example

Build a copy of MTL_MATERIAL_TRANSACTIONS and order by
ORGANIZATION_ID
Ensure Smart Scan/Direct Reads

Smart Flash Cache

Smart Flash Cache

Smart Flash Cache is both HW and SW
Each Storage cell has 384GB of PCIe Flash
The storage server software determines what to cache in Flash
Cache
Frequently accessed data or index block
Control file reads and writes
File header blocks
Anything a DBA/developer chooses to KEEP
Smart Flash Cache wont cache anything it shouldnt:
ASM mirror copies
Backup-related IO
Data Pump stuff
Goal cache what makes sense!

Smart Flash Cache What you Need to Know

Smart Flash Cache is a write-through cache
Write performance not hurt or helped by Flash Cache
Writes not cushioned by Flash Cache
Smart Flash Logging is an interesting improvement
Data in Flash Cache persists across instance/database bounce
By default, the entire capacity of PCI Flash Cards is dedicated to
Smart Flash Cache look at this as a secondary, high speed
cache
You can KEEP or UNKEEP specific segments from Flash Cache
You can place permanent objects in Flash Cache by building
flash grid disks

Smart Flash Cache Writes and Reads or Un-cached

Data

Smart Flash Cache Reads from Cached Data

Smart Flash Cache and Oracle EBS

Configure all of your PCI flash for Smart Flash Cache
Dont KEEP segments in Smart Flash Cache with storage
(flash_cache keep) unless specific testing shows benefits
Dont put applications tablespaces in flash; i.e. dont permanently
store objects in ASM disk groups based on flash grid disks
In short let Flash Cache automatically do its thing unless
specific throughput requirements mandate a deviation

Exadata Smart Flash Logging

Starting with Oracle 11.2.0.3 and Cell version 11.2.2.4, Oracle
introduced Smart Flash Logging
LGWR writes to Flash and Disk simultaneously
By Disk I mean cell server DRAM cache, = 512MB
The first one done sends write acknowledgement to LGWR
On latest versions, dont put redo logs on Flash Grid Disks
Potentially a very good improvement

So What in EBS Will Work Well on Exadata?

Long-running batch jobs on large tables that do full scans

Think month-end, year-end close
Things like MRP, Create Accounting, BIS Refresh, Payroll, etc.
Use the tools youve got to evaluate
Create a SQL Tuning Set on your current platform
Use SQL Performance Analyzer use the Exadata Simulation
Look at AWR reports for historical SQL doing full-scans

So What in EBS Will Work Well on Exadata?

Optimizing Oracle E-Business Suite on Exadata

Questions?