DW Implementation Boot Camp - Student Manual

1
For Oracle employees and authorized partners only. Do not distribute to third parties.
2008 Oracle Corporation Proprietary and Confidential
The following is intended to outline our general
product direction. It is intended for information
purposes only, and may not be incorporated into any
contract. It is not a commitment to deliver any
material, code, or functionality, and should not be
relied upon in making purchasing decisions.
The development, release, and timing of any
features or functionality described for Oracles
products remains at the sole discretion of Oracle.
Safe Harbor Statement
Use of this Site (Site) or Materials constitutes agreement with the following terms and conditions:
1. Oracle Corporation (Oracle) is pleased to allow its business partner (Partner) to download and
copy the information, documents, and the online training courses (collectively, Materials") found on this
Site. The use of the Materials is restricted to the non-commercial, internal training of the Partners
employees only. The Materials may not be used for training, promotion, or sales to customers or other
partners or third parties.
2. All the Materials are trademarks of Oracle and are proprietary information of Oracle. Partner or other
third party at no time has any right to resell, redistribute or create derivative works from the Materials.
3. Oracle disclaims any warranties or representations as to the accuracy or completeness of any
Materials. Materials are provided "as is" without warranty of any kind, either express or implied,
including without limitation warranties of merchantability, fitness for a particular purpose, and non-
infringement.
4. Under no circumstances shall Oracle or the Oracle Authorized Delivery Partner be liable for any loss,
damage, liability or expense incurred or suffered which is claimed to have resulted from use of this Site
of Materials. As a condition of use of the Materials, Partner agrees to indemnify Oracle from and against
any and all actions, claims, losses, damages, liabilities and expenses (including reasonable attorneys'
fees) arising out of Partners use of the Materials.
5. Reference materials including but not limited to those identified in the Boot Camp manifest can not be
redistributed in any format without Oracle written consent.
Oracle Training Materials Usage
Agreement
<Insert Picture Here>
Title of Presentation
PresentersName
PresentersTitle
Data Warehouse course introduction
2
Module Agenda
Introductions
Introduction to data warehousing
Creating business value from IT
Data warehouse architecture & design
Preparing for the hands-on labs
Introductions
The goals of this course
Provide background to understand technical
challenges presented by data warehouses
Provide a technical overview of the Oracle Database
11g Data Warehousing features with extensive
hands-on labs
Guidelines on sizing of Data Warehouse hardware
configurations
Share best practices for deploying Oracle Data
Warehouse architectures and platforms
Introductions
Who are you?
Name
Company
Role
What is your prior experience?
Database administration
Data warehouse experience
10g or 11g Database experience
How do you expect to benefit from this course?
3
Introductions
Who am I?
Course modules
Introduction to data warehousing
Introduction to Oracle data warehouse features
Parallel execution - Lab
Partitioning - Lab
Result Cache - Lab
OLAP - Lab
Advanced compression - Lab
Data integration and ETL- Demo
Course modules
Continued
Sizing Oracle data warehouse platforms
Exadata overview
Best practices for performance
Introduction to data
warehouses
4
All about data warehouses
What is a data warehouse?
A data warehouse is
A repository of data
From multiple sources
Used for analysis
Strategic value is realized from data aggregation
Why have a data warehouse?
Increase the value inherent in data
Extract strategic information from tactical data
Drive IT value into business results
How is a data warehouse different from OLTP?
Read-mainly
Typically, real-time not required
Large data sets
Complex queries based on many factors
How is a data warehouse the same as OLTP?
Modern data warehouses
Near real time
Mixed workloads
ETL
OLTP
Write operations make OLTP issues relevant
Performance
Locking
5
Using a data warehouse
Data
Warehouse
Customers
Web
Servers
Personalization
Content
Servers
Fulfillment
Centers
Supply
Chain
Planning
Apps
Financial
Analysis
Campaign
Generation
Email
Servers
Internal
End Users
Partners
Using a data warehouse Amazon.com
0
100
200
300
400
500
600
700
T
e
r
a
b
y
t
e
s
2
0
0
0
2
0
0
1
2
0
0
2
2
0
0
3
2
0
0
4
2
0
0
5
2
0
0
6
2
0
0
7
2
0
0
8
2
0
0
9
2
0
1
0
NA
FE
EU
Data Volume has grown 2x year over year for the past 4 years.
Projecting higher growth in upcoming years*:
- additional lines of business / product lines supported
- huge standard reporting growth with more partners supported
Creating business value
from IT
IT PRIORITIES BUSINESS PRIORITIES
Reduced Cost Business Value
Improved Security Ease of Access
Improved System
Performance
Results in Time
for Deadlines
Manageability /
Consistency
Good Enough Infrastructure
6
Increasing Return on Investment for information
producers by
Reducing current costs of analysis for lines of business
Empower LOBs with tools, applications, and automated
analysis rather than manually creating custom reports
Reducing ongoing costs of analysis for IT
Faster and simpler development of analytic applications
Centralized management
Increasing Return on Investment for information
consumers in lines of business by
Enabling top-line growth
New customers/markets, best customers,
React faster than the competition
Enabling bottom-line savings
Optimize operations, mitigate risk,
Better manage the business than the competition
Data warehouse
architecture and design
Data warehouse architecture and design
Architecture
Data warehouse
Operational data store
Data marts
Federated data marts
7
Data design
Analysis performed on a set of facts
Examined through different dimensions
Usually has time dimension
Queries typically examine facts with many dimensions
Design options
Star schema
Snowflake schema
OLAP cubes
Hybrid organization
Design options Star schema
Sales Sales
Distributor Distributor
OEM OEM
Channel Channel Geography Geography
Type Type
Size Size
Color Color
Product Product Time Time
Year Year
Quarter Quarter
Month Month
Region Region
District District
Branch Branch
Fact Table: Fact Table:
Sales Sales
Transactions Transactions
Design options Snowflake schema
8
Design options OLAP cubes
Product
Sales Cube
Inventory Cube
SALES FACT SALES FACT
TIME
INVENTORY INVENTORY
REGION
CUST
ITEM
Time
Geography
Product
Time
Design options Hybrid organization
Combination of structures
Star schema
OLAP cubes
Third normal form
Oracle query optimizer can handle appropriately for
all structures
Lab Setup
Lab setup - Connection
Instructions for dialing in to hosted image
9
Lab setup - Testing
Login to the VM Linux environment with oracle/oracle
Open a new command window
Start a SQL Plus session as SH/SH
Lab setup VM environment overview
All passwords for OS & Oracle logins are same as the
username
Except - root, SYS, SYSTEM, and SYSMAN are
oracle
All the lab scripts are stored in the
/home/oracle/wkdir directory
Few shortcuts:
Command
Window
Firefox
Browser
Analytic
Workspace
Manager
SQL
Developer
Summary
A data warehouse is a way to provide
additional value from existing data
Data warehouse architectures can include
operational data stores, federated data
marts and data marts
Data warehouses can use star schemas,
snowflake schema, OLAP cubes, third
normal form or a mixture of different
structures
10
PresentersName
PresentersTitle
Data Warehouse and Business Intelligence
overview
Module Agenda
Why Oracle for data warehousing?
Core data warehousing features
Oracle 11g data warehousing features
Oracle hardware data warehouse solutions
Analytics and data mining
Oracle Business Intelligence Platform (OBIEE)
11
Why Oracle for data
warehousing?
Why Oracle for data warehousing?
Best-of-breed for data warehouses and data marts
Consolidation platform
Eliminates the costs and inefficiencies of multiple data stores
Complete, integrated ELT and data-quality
capabilities
Unmatched analytic capabilities
Embedded OLAP, data-mining and statistics all accessible via
SQL
Optimized for grid computing
Next-generation BI with integrated OLTP and DSS
Optimized hardware solutions
Foundation of Oracle Business Intelligence solutions
Oracle - #1 in data warehouse market
share
Core data warehousing
features
12
Oracle data warehouse platform
DW Services
Administration
Workload Mgmt
Self-Tuning
Security
Virtual Private
Database
Database Vault
Data Integration
Bulk ETL + Real-Time
ETL
Data Quality
Metadata Management
Extensible
Repository
Lineage and Impact
Analysis
Brainy Software
Analytics
Multi-
Dimensional
Calculations
Statistics Data Mining
Intelligent Query Processing
Materialized
Views
Bitmap
Indexes
Partition
Elimination
Star Query
Optimization
Brawny Hardware
Scalable Data Management
Parallel
Execution
Partitioning RAC
Automatic
Storage
Mgmt
Compression
Scalable Hardware Infrastructure
Reference
Configurations
Optimized
Warehouse
Exadata
Database
Machine
Exadata
Storage Server
Core data warehouse features
Bit-mapped indexes
PARTS
table
001 001 GREEN GREEN MED MED 98.1 98.1
002 002 RED RED MED MED 1241 1241
003 003 RED RED SMALL SMALL 100.1 100.1
004 004 BLUE BLUE LARGE LARGE 54.9 54.9
005 005 RED RED MED MED 124.1 124.1
006 006 GREEN GREEN SMALL SMALL 60.1 60.1
... ... .... .... ..... ..... ... ...
size size = = SMALL SMALL 0 0 1 0 1 1 0 1 0 1 0 0 0 1 0 1 0 0 1 0 1 1 0 1 0 1 0 0 0 1 0 1
size size = = MED MED 1 0 0 0 0 0 1 0 0 0 0 1 0 1 0 0 1 0 0 0 0 0 1 0 0 0 0 1 0 1 0 0
size size = = LARGE LARGE 0 1 0 1 0 0 0 0 1 0 1 0 1 0 1 0 0 1 0 1 0 0 0 0 1 0 1 0 1 0 1 0
Index on COLOR
color color = = BLUE BLUE 0 1 0 1 0 0 1 0 1 0 1 0 0 0 1 0 0 1 0 1 0 0 1 0 1 0 1 0 0 0 1 0
color color = = RED RED 0 0 1 0 0 0 0 1 0 0 0 0 1 0 0 1 0 0 1 0 0 0 0 1 0 0 0 0 1 0 0 1
color color = = GREEN GREEN 1 0 0 0 1 1 0 0 0 1 0 1 0 1 0 0 1 0 0 0 1 1 0 0 0 1 0 1 0 1 0 0
0
1
0
0
1
0
0
1
0
0
1
0
partno partno color color size size weight weight
Index on SIZE
SELECT count(*)
FROM parts
WHERE
size = MED a AND
color = RED
Parallel bit-mapped star joins
Dramatic performance gains
Innovative use of bitmap indexes
Ideal for complex star schemas
Multiple fact tables
Many dimensions
Unconstrained dimensions
Large dimension tables
Optimized for sparse fact tables
Parallel execution
Optimizer modifies an SQL query for faster
performance with star schema
Bitmapped indexes used to select rows from fact
tables
Replaces multiple joins
Results are joined to dimension tables for additional
information
Must enable in database and conform to constraints
Star transformation
13
Materialized views
What were sales What were sales
in the West and in the West and
South regions for South regions for
the last three the last three
quarters? quarters?
x
(2 GB)
SALES SALES SALES SALES
SALES SALES
PER PER
MONTH MONTH
BY REGION BY REGION
SALES SALES
PER PER
MONTH MONTH
BY REGION BY REGION
Stored summaries are
maintained for faster
warehouse query
processing
Materialized views query rewrite
SELECT p.brand, r.country, t.month,
SUM(s.amt) tot_sales
FROM sales s, region r,
time t, product p
WHERE s.region_id = r.region_id
AND s.sdate = t.curdate
AND s.prod_code = p.prod_code
GROUP BY p.brand, r.country, t.month
HAVING SUM(s.amt) > 5000000;
SELECT brand, country, month, tot_sales
FROM sales_sumry
WHERE tot_sales > 5000000;
Optimizer
transparently
rewrites query
on detailed
data ...
Optimizer
transparently
rewrites query
on detailed
data ...
to access
data in the
materialized
view
to access
data in the
materialized
view
Materialized views data refresh
Maintain the timeliness of stored
result to preference
Several options for refreshing data
in materialized views
Manage the effect of performance
on operational data
No impact on load performance
SUM SUM
ORDERS ORDERS
by by
QUARTER QUARTER
COUNT
ORDERS
by
PRODUCT
AVG
ORDERS
by
REGION
Insert Insert
Update Update
Delete Delete
Insert Insert
Update Update
Delete Delete
Deferred Deferred
Refresh Refresh
Full Full
Refresh Refresh
ORDERS ORDERS
Incremental Incremental
Refresh Refresh
Parallel execution
select c.cust_last_name
, sum(s.amount_sold)
from customers c, sales s
where c.cust_id = s.cust_id
group by c.cust_last_name;
Data on Disk Parallel Servers
scan scan
scan scan
scan scan
aggregate aggregate
Scanners
Coordinator
join join
join join
join join
aggregate aggregate
aggregate aggregate
Joiners Aggregators
RAC aware!
14
Partitioning
Core functionality Performance Manageability
Oracle8 Range partitioning
Global range indexes
Static partition
pruning
Basic maintenance
operations: add,
drop, exchange
Oracle8i Hash and composite
range-hash partitioning
Partition-wise joins
Dynamic pruning
Merge operation
Oracle9i List partitioning Global index
maintenance
Oracle9i R2 Composite range-list
partitioning
Fast partition split
Oracle10g Global hash indexes Local Index
maintenance
Oracle10g R2 1M partitions per table Multi-dimensional
pruning
Fast drop table
Oracle
Database 11g
More composite choices
REF Partitioning
Virtual Column Partitioning
Interval Partitioning
Partition Advisor
Data compression
Tables and indexes can be compressed
Can be specified on a per-partition basis
Typical compression ratio 4:1
Requires more CPU to load data
Decompression minimal resource cost
Compress for all DML operations
Less data on disk
Requires less time to read
Little or no impact on query performance
Completely transparent
Especially applicable to data warehouses
4X
Compression
Data compression real world examples
0
500
1000
1500
2000
2500
Table Scans
0
0.1
0.2
0.3
0.4
2.5x Faster
DML Performance
0
10
20
30
40
< 3% Overhead
3x Saving
Data Storage
Database Resource Manager
High Priority Sales Analysis
20 users
(DOP 10)
Medium Priority Ad Hoc Reports
200 users
(DOP 4)
Low Priority ETL Jobs
200 users
(DOP 4)
Protect the system pro-actively
Maximum number of concurrent operations
Control CPU and IO consumption based on Consumer Groups
15
SQL Access Advisor
Based on workload
Materialized view advice
Index advice
Partition advice
SQL Access Advisor how it works
Indexes Materialized
views
Materialized
views log
SQL Access
Advisor
Hypothetical
SQL cache
Filter
Options
STS
Complete
Workload
Partitioned
objects
Hash
partitions?
Interval
partitions?
Oracle Warehouse Builder
Design Target Warehouse Design Target Warehouse 1
2 Define Sources Define Sources
Relational
Flat Files
Applications
Mainframe
Map Source to Target Map Source to Target 3
Oracle or flat files
Generate Code Generate Code 4
Instantiate Warehouse Instantiate Warehouse 5
66 Extract & transform data
Oracle Warehouse Builder - Repository
Multi-user repository
Single update, multi-read
Object locking
Synchronize
Metadata import / export &
interchange
Bridges to Oracle tools, ERwin,
PowerDesigner
Bridges to other metadata available
from Metadata Integration Technology
Incorporated
16
Oracle Data Integrator
Next Generation Architecture
E-LT E-LT
Load Extract
Transform Transform
Conventional ETL Architecture
Extract Load Transform
Performance
Heterogeneous E-LT
Productivity
Declarative design
Flexibility
Active integration design
Easily extendable
Knowledge modules
Real Application Clusters
Query Query Query Query
ETL ETL ETL ETL
Data Warehouse
Scale across cluster
Query Query
ETL ETL
Data Warehouse
Dedicate nodes to
specific work
Query Query
ETL ETL
Data Warehouse
Dedicate nodes to
specific work
Scale out when
workload increases
Query
17
Spreads database files across
all available storage for optimal
performance and resource
utilization
Enables simple, online resource
allocation and provides
automatic data rebalancing
Up to 3-way mirroring
Can leverage external hardware
RAID mirror and parity protection
Automatic Storage Management (ASM)
Oracle OLAP Option
Embedded, manageable, enterprise-ready
A multidimensional calculation and aggregation engine
Multidimensional data types: Cubes and dimensions
An OLAP API for cube definition and multidimensional queries
A SQL interface to OLAP Cubes and dimensions
Embedded in the Oracle Database
Runs within Oracle instance on same server
OLAP cubes are stored in Oracle data files
OLAP metadata in the Oracle Data Dictionary
All key database functionality extends to OLAP
Security: Object and data security
Scalability: Real Application Clusters
Availability: Backup-Recovery, Disaster Recovery, RAC
Administration: Enterprise Manager
Why Oracle OLAP is so Fast
Data Access Method
Data stored in dense arrays
Offset addressing no joins
More powerful analysis
Better performance
Time
Product
CD
Profit
DVD
Audio
Q1 Q2 Q3
SF
West
USA
Market
Compare Profit this Quarter versus Last Quarter
What is a Products Profit Share of its Category?
Current adoption
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
DB Res Mgr
RMAN
ASM
Read Only
VPD
MV Use
Compression
Parallel Exec
Partitioning
18
Oracle 11g data
warehousing features
Oracle 11g data warehouse features
Data Result Cache
Automatically caches results of queries, query blocks, or PL/SQL
function calls
Cache is shared across statements and sessions on server
Significant speed up for read-only / read-mostly data
Full consistency and proper semantics
Cache refreshed when any underlying table updated
join
join
Table1 Group-by
join
Table2 Table3
Q2: Uses result
transparently cache
Data Results Cache
Retail customer data (~50 GB)
Concurrent users submitting queries randomly
Executive dashboard with 12 heavy analytical queries
Cache results only at in-line view level
12 queries run in random, different order 4 queries cached
Measure average, total response time for all users
447 s
267 s
186 s
No cache
334 s
201 s
141 s
Cache
25%
25%
24%
Improvement
8
4
2
# Users
SQL PIVOT/UNPIVOT
Rotate rows into columns and vice versa
Common operation for crosstab queries
PROD QUARTER AMOUNT
Shoes Q1 2000
Shoes Q2 1000
Jeans Q1 1000
Jeans Q3 500
Jeans Q3 100
Jeans Q4 1000
PROD Q1 Q2 Q3 Q4
Shoes 2000 1000 Null Null
Jeans 1000 Null 600 1000
SELECT * FROM t
PIVOT (sum(amount)
FOR quarter in
Q1,Q2,Q3,Q4);
19
Additional features
Enhanced partitioning options
Advanced compression
OLAP-based materialized views
Priority-dependent maximum degree of parallelism in
Database Resource Manager
Oracle data warehousing
hardware
Custom Custom
Complete Flexibility
Any OS, any platform
Easy fit into a
companys IT
standards
Oracle DW Hardware Solutions
The evolution of hardware infrastructure
Custom Custom
Easy fit into a
companys IT
standards
Documented best-
practice
configurations for
data warehousing
Reference
Configurations
Reference
Configurations
20
Custom Custom
Easy fit into a
companys IT
standards
Documented best-
practice
configurations for
data warehousing
Optimized
Warehouse
Optimized
Warehouse
Scalable systems
pre-installed and pre-
configured: ready to
run out-of-the-box
Reference
Configurations
Reference
Configurations
Custom Custom
Easy fit into a
companys IT
standards
Documented best-
practice
configurations for
data warehousing
Optimized
Warehouse
Optimized
Warehouse
Scalable systems
pre-installed and pre-
configured: ready to
run out-of-the-box
Highest performance
Pre-installed and pre-
configured
Sold by Oracle
Reference
Configurations
Reference
Configurations
Sun Oracle
Database
Machine
Sun Oracle
Database
Machine
Build From Scratch
with Components
OWI Reference
Configurations
Take delivery of OWI
configuration
Weeks to Months
Pre-implementation
System sizing
Acquisition of
components
Installation and
configuration
Acquisition of
components
Installation and
configuration
Testing and
Validation
Testing and
Validation
Weeks to Months
Sun Oracle
Database Machine
Faster deployment
Lower Risk
< 1 Week
Database Machine Benefits
Accelerate Implementations & Lower Risk
Analytics and data
mining
21
SQL analytics
Bring Algorithms to the Data - Not Data to
the Algorithms
Analytic computations
done in the database
SQL analytics
Statistics
OLAP
Data Mining
Scalability
Security
Backup & Recovery
Simplicity
OLAP
Data Mining
Statistics
Why Data Mining?
Do you wish to ..
Identify factors most associated with a business problem?
Predict customer behavior ?
Predict or estimate a value ?
Find profiles of targeted people or items?
Segment a population ?
Determine important relationships/market baskets within the
population ?
Find fraudulent or rare events ?
(Attribute Importance)
(Classification)
(Regression)
(Decision Trees)
(Clustering)
(Associations)
(Anomaly Detection)
Statistics & SQL Analytics
Ranking functions
rank, dense_rank, cume_dist, percent_rank, ntile
Window Aggregate functions
(moving and cumulative)
Avg, sum, min, max, count, variance, stddev,
first_value, last_value
LAG/LEAD functions
Direct inter-row reference using offsets
Reporting Aggregate functions
Sum, avg, min, max, variance, stddev, count,
ratio_to_report
Statistical Aggregates
Correlation, linear regression family, covariance
Linear regression
Fitting of an ordinary-least-squares regression line
to a set of number pairs.
Frequently combined with the COVAR_POP,
COVAR_SAMP, and CORR functions.
Descriptive Statistics
average, standard deviation, variance, min, max, median (via
percentile_count), mode, group-by & roll-up
DBMS_STAT_FUNCS: summarizes numerical columns of a
table and returns count, min, max, range, mean, stats_mode,
variance, standard deviation, median, quantile values, +/- n
sigma values, top/bottom 5 values
Correlations
Pearsons correlation coefficients, Spearman's and Kendall's
(both nonparametric).
Cross Tabs
Enhanced with % statistics: chi squared, phi coefficient,
Cramer's V, contingency coefficient, Cohen's kappa
Hypothesis Testing
Student t-test , F-test, Binomial test, Wilcoxon Signed Ranks
test, Chi-square, Mann Whitney test, Kolmogorov-Smirnov
test, One-way ANOVA
Distribution Fitting
Kolmogorov-Smirnov Test, Anderson-Darling Test, Chi-
Squared Test, Normal, Uniform, Weibull, Exponential
Pareto Analysis
80:20 rule, cumulative results table
Oracle Data Mining Option
Embedded, manageable, enterprise-ready
A complete data-mining solution
10+ data-mining algorithms: addresses every business scenario
Generalized Linear Models (Logistic & Multiple Regression)
Oracle Data Miner: Graphical user-interface for analysts
A Data Mining PL/SQL & Java API for developers
Embedded in the Oracle Database
Runs within Oracle instance on same server
Data Mining algorithms execute directly on database tables
Data Mining models and metadata stored within Oracle database
All key database functionality extends to Data Mining
Scalability: Parallelism, Real Application Clusters
Security: Object and data security
Administration: Enterprise Manager
Portability: All major platforms
22
Oracle Data Mining
Oracle in-Database Mining Engine
Oracle Data Miner (GUI)
Simplified, guided data mining
Spreadsheet Add-In for
Predictive Analytics
1-click data mining from a
spreadsheet
PL/SQL API & Java (JDM) API
Develop advanced analytical
applications
Wide range of algorithms
Oracle Business
Intelligence
OLTP & ODS
Systems
Data
Warehouse
Data Mart
Oracle Business Intelligence EE
Data warehouse components
OLTP & ODS
Systems
Data
Warehouse
Data Mart
SAP, Oracle
PeopleSoft, Siebel,
Custom Apps
Files
Excel
XML
Business
Process
Full data foundation
23
OLTP & ODS
Systems
Data
Warehouse
Data Mart
SAP, Oracle
PeopleSoft, Siebel,
Custom Apps
Files
Excel
XML
Business
Process
Oracle BI Server
Common Enterprise Information Model
Common information model
OLTP & ODS
Systems
Data
Warehouse
Data Mart
SAP, Oracle
PeopleSoft, Siebel,
Custom Apps
Files
Excel
XML
Business
Process
Oracle BI Server
Reporting
&
Publishing
Ad hoc
Analysis
Proactive
Detection
and Alerts
Microsoft
Office
Interactive
Dashboards
Advantages:
Consolidate and
standardize BI
tools
Unified BI
metadata and
infrastructure
Seamless BI user
experience
Intelligent
Caching Services
Data Federation
Intelligent Alerting
Comprehensive, integrated BI suite
Oracle BI Enterprise Edition Plus
(OBIEE+)
OLTP & ODS
Systems
Data
Warehouse
Data Mart
SAP, Oracle
PeopleSoft, Siebel,
Custom Apps
Files
Excel
XML
Business
Process
Performance
Management
Applications
Insight Performance Action
Set Goals
Plan
Monitor
Analyze
Report
Align
Hyperion
Essbase
Oracle BI Server
Reporting & Publishing
Financial Reporting
BI Publisher
Interactive Reporting
SQR Production Reporting
Web Analysis
Ad hoc
Analysis
Proactive
Detection
and Alerts
Microsoft
Office
Interactive
Dashboards
E
x
t
e
n
d
O
D
B
C
Summary
Oracle has many features to enhance data
warehouse operations
Oracles data warehouse features are
implemented transparently to developers
Oracles data warehouse features are
integrated with the Oracle database
Oracle has evolved the hardware
infrastructure of data warehouses to
preconfigured optimized warehouses
Oracle offers a complete set of analytic and
business intelligence capabilities
24
PresentersName
PresentersTitle
Parallel execution
25
Module Agenda
Parallel execution overview
Parallel execution setup
Parallel joins
Parallel execution with RAC
Parallel execution
overview
What is parallel execution?
With parallel execution, a task is
Broken into smaller sub-tasks, which are executed independently in
parallel, with the result joined at completion
SELECT c.cust_last_name, sum(s.amount_sold)
FROM customers c, sales s
WHERE c.cust_id = s.cust_id
GROUP BY c.cust_last_name;
Data on Disk Parallel Servers
scan scan
scan scan
scan scan
aggregate aggregate
Coordinator
join join
join join
join join
aggregate aggregate
aggregate aggregate
SQL Parallel Execution
QC is the QC is the user session that initiates the parallel SQL statement user session that initiates the parallel SQL statement
& it will & it will distribute the work to parallel servers distribute the work to parallel servers
Parallel servers Parallel servers - - individual sessions that perform individual sessions that perform
work in parallel They are allocated from a work in parallel They are allocated from a pool of pool of
globally available parallel server processes and globally available parallel server processes and
assigned to a given operation assigned to a given operation
26
Dynamic parallelism
Advantages
Uses granules
Advantages
Uses optimal queueing technique
Avoids convoy effect
Adaptable to changing conditions
Structural
Runtime
Allow partitioning based on business requirements
Parallel execution
Parallel scans
Data is partitioned into granules
(block range or partition)
Dynamic Granules Computed
during runtime by the database
Each scan parallel process is
assigned multiple granules
No two scanners ever contend
for the same granule
Granules are assigned so that
the load is balanced across all
parallel scan processes
PQ #2
PQ #3
PQ #1
Parallel Execution
Servers
Parallel execution
Intra- and inter-operation parallelism
PQ 1
PQ 2
PQ 3
PQ 4
Sales table Full Scan Parallel Sort
A - C
D - F
G - M
N - O
PQ 5
PQ 6
PQ 7
PQ 8
S
T - V
W - Z
P - R
SELECT ... FROM product p, sales s where p.prod_id = s.prod_id ORDER BY p.prodname;
User
Process
Query
Coordinator
Intra-Parallelism Intra-Parallelism
DOP = 8
(Intra-Parallelism)
DOP = 16
(Inter-Parallelism)
Inter-Parallelism
Parallel execution
Parallel scans of multiple tables
SELECT c.cust_name, s.date,
s.amount
FROM sales s, customers c
WHERE s.cust_id = c.cust_id;
Query
coordinator
SALES
Table
CUSTOMERS
Table
27
Parallel execution
s.amount
Query
coordinator
P1 P2 P3 P4
SALES
Table
CUSTOMERS
Table
Begin query
with
DOP of 4
Allocate producers
and consumers
P8
P7
P6
P5
Parallel execution
s.amount
Query
coordinator
P1 P2 P3 P4
SALES
Table
CUSTOMERS
Table
Access the
smaller
CUSTOMERS
table
in parallel
P8
P7
P6
P5
Parallel execution
s.amount
Query
coordinator
P1 P2 P3 P4
SALES
Table
CUSTOMERS
Table
P8
P7
P6
P5
Access the
SALES
table
in parallel
Parallel execution
s.amount
Query
coordinator
P1 P2 P3 P4
SALES
Table
CUSTOMERS
Table
Parallel
join
28
Parallel execution
s.amount
Query
coordinator
P1 P2 P3 P4
SALES
Table
CUSTOMERS
Table
Assembly
Degree of Parallelism = number of parallel execution servers
associated with a parallel operation
Assigned for database, object, session or query
DOP is calculated at run-time
Default DOP (when not specified) =
CPU_COUNT x PARALLEL_THREADS_PER_CPU
DOP may be reduced depending on the availability of parallel
execution servers and system load
Database Resource Manager can throttle DOP per resource
consumer groups
Parallel execution
Degree of Parallelism (DOP)
DOP applies to intra-operation parallelism
In inter-operation pallelism, parallel execution servers
can be twice the number specified by the DOP
No more than two sets of parallel execution servers
can be used for one parallelized statement at a time
Parallel execution
Degree of Parallelism (DOP) - continued
SQL Access Methods
Table Scans
Fast Full Index Scans
Partitioned Index Range Scans
Sorts, Aggregations, Set Operations
External Table access
Joins
Nested Loop, Sort-Merge
Hash, Star, Partition-Wise
DDL
CTAS, CREATE INDEX, REBUILD INDEX
DML
INSERT-SELECT, UPDATE, DELETE, MERGE
Parallel execution
Parallelizable operations
29
Parallel Execution Server Pool
Server pool size is determined by PARALLEL_MIN_SERVERS
and PARALLEL_MAX_SERVERS
Dynamic based on demand
Adaptive Multiuser and DOP
PARALLEL_ADAPTIVE_MULTI_USER = TRUE (default)
Throttles DOP to prevent overloading of server
PARALLEL_MIN_PERCENT is the minimum parallel execution
servers requested for the operation
Default 0 Disabled
Operation not executed if the MIN % of DOP servers not available
Configuration
Enabling Parallel DML, DDL and Query
ALTER SESSION (ENABLE | DISABLE | FORCE) PARALLEL
(DML | DDL | QUERY) (PARALLEL n);
FORCEPARALLEL overrides default DOP
Hints overrides a forced DOP
Parallel Query & Parallel DDL is enabled if ANY of
below is TRUE
Object was created with PARALLEL clause
ALTER SESSION [ENABLE|FORCE]PARALLEL
Parallel hints are specified in the query
Parallel DML enabled if both are TRUE
ALTER SESSION [ENABLE|FORCE]PARALLEL
Object was created with PARALLEL clause OR Parallel
Hints are specified in DML
Execution
Parallel joins
30
Serial Join
scan scan
join
Products Sales
Sales.prod_id = products.prod_id
Scan of both tables
in serial
Compares (joins) all
records from sales
with products
Basic Parallel Join
scan scan
join
scan scan
join
Scan of both tables
in parallel
records from sales
with products
Products Sales

Parallel Join with Broadcasting
scan scan
join
scan
join
Scan of both tables
in parallel/serial
records from sales
with products
Products Sales
Smaller table is
broadcast to all
processes
Parallel Join with Broadcasting

scan scan
join
scan
join
Scan of both tables
in parallel/serial
records from sales
with products
Smaller table is
broadcast to all
processes
No data distribution
for large table
Products Sales
31
Parallel processes
Data Redistribution Methods
HASH
Standard redistribution for parallel processing
Example JOIN, GROUP BY operations
KEY
Ensures the logical clumping of KEY values
RANGE
Used for combined GROUP BY and ORDER BY operations
BROADCAST
Sends smaller data sets to all participating processes to
optimize data shipping
ROUND ROBIN
Final operation to send data to requesting process
Variance LOCAL as RAC optimization
Partition-wise Join
scan scan
join
Products Sales
Scan of both tables
in serial
records from sales
with products
Equi-partitioning of
tables will be
leveraged
Partition-wise Parallel Join
scan scan
join
scan scan
join
Products Sales
Scan of both tables
in serial
records from sales
with products
tables will be
leveraged
required
Parallel execution with
RAC
32
Parallel Query
PQ 1
PQ 2
PQ 3
PQ 4
A - C
D - F
G - M
N - O
PQ 5
PQ 6
PQ 7
PQ 8
S
T - V
W - Z
P - R
User
Process
Query
Coordinator
Internode Parallel Query
PQ 1
PQ 2
PQ 3
PQ 4
A - C
D - F
G - M
N - O
PQ 5
PQ 6
PQ 7
PQ 8
S
T - V
W - Z
P - R
User
Process
Query
Coordinator
Node 1
Node 2
Many large-scale DWs have many concurrent jobs
Multiple small-to-medium size queries
Degree of parallelism < CPUs-per-node
With Oracle, queries will automatically run on a single node,
eliminating traffic over the interconnect
RAC and Parallel Execution
Q1 Q2 Q4 Q3
Q5 Q7 Q6 Q8
Q9 Q12 Q11 Q10
Very large queries utilize all resources on the
cluster
Large Query
33
RAC and parallel execution
Allocating parallel execution servers
If no DOP has been set for the object, session or
through a hint, RAC uses parallel servers from across
all nodes.
Else, if DOP < 2 times the number of CPUs on a
node, RAC will first attempt to use parallel execution
processes on owner node
If DOP is too high or if enough local parallel server
process are not available, processes spread over
multiple nodes
P1 P1 P2 P2 P3 P3 P4 P4
Node 1 Node 2
PX1 PX2 PX3 PX4
Sample optimizations
Classical co-located partition-wise join
A partition-wise parallel join is used if the DOP is either equal to
the number of partitions, or an even divisor of the number of
partitions
If there were 16 partitions, a partition-wise parallel join would
work with a DOP of 16, 8, 4 or 2
Example shown 2 nodes, 4 partitions on 2 tables, DOP 4
P1 P1 P2 P2 P3 P3 P4 P4
Node 1 Node 2
Sample optimizations
Without partition-wise parallel join, data redistribution
must occur, which can stress interconnect
Sample join of two tables, each with 4 partitions, on 2 nodes,
DOP 8
PX1 PX2 PX3 PX4 PX5 PX6 PX7 PX8
Parallel Operations with RAC
Using services
PARALLEL_INSTANCE_
GROUPS and
INSTANCE_GROUPS no
longer used to control
parallelism across RAC
nodes
Services specify which
nodes can participate in
parallel execution
Connection of a process
to a service through
sql*net can contain the
parallel execution
Define service thru OEM
or (DB and srvctl) then
add to tnsnames.ora
34
Summary
Oracle uses dynamic parallelism
The degree of parallelism is dynamically
calculated, based on several factors
Parallel partition-wise joins are the
performance goal
Parallelism is properly supported with RAC
Instance groups or services can limit
parallelism to specific nodes with RAC
35
PresentersName
PresentersTitle
Partitioning
Module Agenda
Partitioning benefits
Partitioning options
Index partitioning
Partition operations
Dividing a table or index into smaller portions,
based on a partition key
Partitioning transparent to applications and users
Partitioning
Definition
36
Partition pruning
Optimizer eliminates unneeded partitions for data access
Improves query performance by several orders (FTS .vs. FPS)
Prunes range or list partitioning for LIKE, equality, and IN-list
Prunes hash partitioning for equality and IN-list
Static and dynamic partition pruning
Partition-wise join
Used when two tables are joined together and when both the tables are
partitioned on the join key
Breaks the large join into smaller joins that occur between each of the
partitions
Significantly reduces response time and improves the use of both CPU and
memory resources
Performance
Maintenance operations can address smaller
partitions rather than full table or index
Maintenance operations like DROP, SPLIT,
COALESCE make it easy to adjust partitions
Partitions support a 'rolling window load or purge
process
Manageability
Partitioned database objects provide partition
independence
Important part of a high-availability strategy
Reduce scheduled downtime
Availability
Partitions can be stored on different storage tiers
Archived partitions can be stored on cheaper storage
devices
Cost
37
Partition options
Range
Hash
List
Classic
Composite partitioning
Table SALES
RANGE(order_date)-RANGE(ship_date)
Jan 2006
... ...
Feb 2006 Mar 2006 Jan 2007
... ...
... ...
...
...
...
...
Jan
2006
Feb
2006
Jan
2007
Data is
partitioned along
two dimensions
(A,B)
A distinct value
pair for the two
dimensions
uniquely
determines the
target partitioning
Table SALES
RANGE(order_date)-RANGE(ship_date)
Jan 2006
... ...
Feb 2006 Mar 2006 Jan 2007
... ...
... ...
...
...
...
...
Jan
2006
Feb
2006
May
2006
Mar 2006
All records with
order_date in
March 2006
AND
ship_date in
May 2006
May
2006
38
Composite partitioning options
11g 11g 11g List
8i 9iR2 11g Range
Hash List Range
Interval partitioning
Extension to range partitioning
Full automation for equi-sized range partitions
Partitions are created automatically as data arrives
No need to create new partitions
Local indexes are created and maintained as well
Table SALES
Jan 2006
... ...
Feb 2006 Mar 2006 Jan 2007 Oct 2009 Nov 2009
...
Interval partitioned table can have classical range and
automated interval section
Automated new partition management plus full partition
maintenance capabilities: Best of both worlds
Table SALES
Jan 2006
... ...
Feb 2006 Mar 2006 Jan 2007 Oct 2009 Nov 2009
...
MERGE and move old partitions for ILM
Range partition section
39
Table SALES
Jan 2006 Feb 2006 Mar 2006 Jan 2007 Oct 2009 Nov 2009
MERGE and move old partitions for ILM
Range partition section
... ... ...
Interval partition section
VALUES ('13-NOV-2009') Insert new data
- Automatic segment creation
Table SALES
...
2005 Q1 2006 Oct 2006
Range partitioned tables can be extended into interval
partitioned tables
Simple metadata command
Investment protection
Q2 2006
...
Table SALES
...
2005 Q1 2006 Oct 2006 Q2 2006
ALTER TABLE sales (order_date DATE, ...)
SET INTERVAL(NUMTOYMINTERVAL(1,'month');
Range partitioned tables can be extended into interval
partitioned tables
Simple metadata command
Investment protection
...
New monthly
Interval partitions
Old range partition table
...
Related tables benefit from same partitioning strategy
Sample 3NF order entry data model
Redundant storage of the same information solves
this problem
Data overhead
Maintenance overhead
Business Problem
Solution
Oracle Database 11g introduces REF Partitioning
Child table inherits the partitioning strategy of parent table
through PK-FK relationship
Intuitive modelling
Enhanced performance and manageability
REF partitioning
40
Table ORDERS
Jan 2006
... ...
Feb 2006
Table LINEITEMS
Jan 2006
... ...
Feb 2006
RANGE(order_date)
Primary key order_id
RANGE(order_date)
Foreign key order_id
Redundant storage of order_date
Redundant maintenance
Before REF partitioning
Table ORDERS
Jan 2006
... ...
Feb 2006
Table LINEITEMS
Jan 2006
... ...
Feb 2006
RANGE(order_date)
Primary key order_id
RANGE(order_date)
RANGE(order_date)
PARTITION BY REFERENCE
Partitioning key inherited through
PK-FK relationship
With REF partitioning
Want to create extended schema attributes are fully
derived and dependent on existing common data
Redundant storage or extended view definitions are
solving this problem today
Requires additional maintenance and creates overhead
Business Problem
Solution
Oracle Database 11g introduces virtual columns
Purely virtual, meta-data only
Treated as real columns except no DML
Virtual columns can have statistics
Virtual columns are eligible as partitioning key
Enhanced performance and manageability
Virtual column-based partitioning
ORDERS
ORDER_ID ORDER_DATE CUSTOMER_ID...
---------- ----------- ----------- --
9834-US-14 12-JAN-2007 65920
8300-EU-97 14-FEB-2007 39654
3886-EU-02 16-JAN-2007 4529
2566-US-94 19-JAN-2007 15327
3699-US-63 02-FEB-2007 18733
REGION requires no storage
Partition by ORDER_DATE, REGION
REGION AS (SUBSTR(ORDER_ID,6,2))
------
US
EU
EU
US
US
Virtual column-based partitioning
41
Index partitioning
Created as equi-partition with the
underlying table; created with the
LOCAL attribute
the index partition key ranges and
table partition key ranges are
identical
Managed automatically by server to
ensure equi-partitioning
Easier administration
Enables parallelism of index scans
Reduction of scheduled downtime
Only one index partition affected
by maintenance operation on an
underlying table partition Partition
Q1
Partition
Q2
Partition
Q3
Partition
Q4
SALES07 Table
Index
Partition
Index
Partition
Index
Partition
Index
Partition
CREATE INDEX order_idx
ON sales07 (order_no)
LOCAL
(PARTITION INDEX_Q1,
PARTITION INDEX_Q2 ,
PARTITION INDEX_Q3,
PARTITION INDEX_Q4);
CREATE INDEX order_idx
ON sales07 (order_no)
LOCAL
(PARTITION INDEX_Q1,
PARTITION INDEX_Q2 ,
PARTITION INDEX_Q3,
PARTITION INDEX_Q4);
Local partitioned indexes
41
Not equi-partitioned with the
underlying table
The index is partitioned by the index
key, not by the table partition key
OLTP applications may get
better performance because
they minimize the number of
index probes
Harder to manage than local
indexes
Can be unique or non-unique
Partition
Q1
Partition
Q2
Partition
Q3
Partition
Q4
SALES Table
Partitioned by Quarter
CREATE INDEX product_idx
ON sales07 (product_id)
PARTITION BY RANGE(product_id)
GLOBAL
(PARTITION PIDX1 VALUES LESS THAN (1000),
PARTITION PIDX2 VALUES LESS THAN (2000),
PARTITION PIDX3 VALUES LESS THAN (MAXVALUE));
CREATE INDEX product_idx
ON sales07 (product_id)
PARTITION BY RANGE(product_id)
GLOBAL
(PARTITION PIDX1 VALUES LESS THAN (1000),
PARTITION PIDX2 VALUES LESS THAN (2000),
PARTITION PIDX3 VALUES LESS THAN (MAXVALUE));
Index Partition
Products
0-999
Index Partition
Products
1000-1999
Index Partition
Products
2000 +
Global partitioned indexes
All index data within one schema
object
Treated the same as global indexes
Administration advantages gained
when table is partitioned are lost if
associated indexes are not
partitioned
Why? If recovering a partition of a table,
it will only be usable once the entire non-
partitioned Index is recovered and usable
Non-Partitioned Index
Partition
Q1
Partition
Q2
Partition
Q3
Partition
Q4
SALES07 Table
CREATE INDEX sales07_idx
ON sales97 (sales_date);
CREATE INDEX sales07_idx
ON sales97 (sales_date);
Non-partitioned indexes
42
Partition operations
Partition-wise join
scan scan
join
Products Sales
Scan of both tables
in serial
rows from sales with
products
tables will be
leveraged
Partition-wise parallel join
scan scan
join
scan scan
join
Products Sales
Scan of both tables
in serial
records from sales
with products
tables will be
leveraged
required with RAC
Rolling window operations
Q106 Q206 Q306 Q406
Order Table
(partitioned by quarter)
Drop
Other data & queries not affected
Q107
Add
43
EXCHANGE PARTITION
The basic technique of bulk-loading new data into a
temporary load table, which is then indexed, analyzed,
and then published all at once to end-users using the
EXCHANGE PARTITION operation
Should be the default load technique for all large tables in
a data warehouse
22-Feb
2008
23-Feb
2008
24-Feb
2008
(empty)
Composite-partitioned
table TXN
2. Bulk
Loads
5. EXCHANGE PARTITION
3. Create
Indexes
4. Table
/ Col Stats
EXCHANGE PARTITION
1. Create
Temp Table
25-Feb
2008
Hash-partitioned
table TXN_TEMP
Summary
Partitioning improves performance and
manageability
Oracle supports many types of partitioning,
including range, list, hash, interval, virtual
column and reference partitions
Oracle supports composite partitions
Performance goal is partition-wise joins
Indexes can be local or global
Partition exchange can be used to add
data to a warehouse with very high
availability
44
PresentersName
PresentersTitle
Result Caches
Module Agenda
SQL Result Cache
PL/SQL Function Cache
45
SQL Result Cache
Analyze data across large data sets
Reporting
Forecasting & trend analysis
Data mining
Use parallel execution for good performance
Result
Very I/O intensive workload direct reads from disk
Memory is less important
PGA memory more sensitive based on sort or aggregations
SQL Result Cache
Data warehouse workloads
SQL Result Cache
Data warehouse sample query
Accesses many rows
Returns few rows
select p.prod_category
, sum(s.amount_sold) revenue
from products p
, sales s
where s.prod_id = p.prod_id
and s.time_id
between to_date('01-JAN-2006','dd-MON-yyyy')
and to_date('31-DEC-2006','dd-MON-yyyy')
group by rollup (p.prod_category)
SQL Result Cache
What it does
Caches results of queries, query blocks, or
PL/SQL function calls
Read consistency is enforced
DML/DDL against dependent database objects invalidates
cache
Bind variables parameterize cached result with
variable values
46
SQL Result Cache
How it works
Table 1
Table 2 Table 3
join
join
Group by
query 1
executes
SQL Result Cache
How it works
Table 1
Table 2 Table 3
join
join
Group by
Table 1
Table 2 Table 3
join
join
Group by
cached
result
result is
cached
SQL Result Cache
How it works
Table 1
Table 2 Table 3
join
join
Group by
cached
result
Table 5 Table 6
join
Table 4
join
Group by
join
Query 2 uses cached
result transparently
RESULT_CACHE_MODE initialization parameter
MANUAL, use hints to populate and use
FORCE, queries will use cache without hint
RESULT_CACHE_MAX_SIZE initialization parameter
Default is dependent on other memory settings
(0.25% of MEMORY_TARGET or 0.5% of SGA_TARGET
or 1% of SHARED_POOL_SIZE)
0 disables result cache
Never > 75% of shared pool (built-in restriction)
/*+ RESULT_CACHE */ hint in queries
SQL Result Cache
Enabling SQL Result Cache
47
SQL Result Cache
SQL hint example
Use RESULT_CACHE hint
select /*+ RESULT_CACHE */ p.prod_category
from products p
, sales s
and s.time_id
group by rollup (p.prod_category)
SQL Result Cache
View hint example
Use RESULT_CACHE hint in view definition
select prod_subcategory
, revenue
from
(select /*+ RESULT_CACHE */ p.prod_category, p.prod_subcategory
from products p
, sales s
and s.time_id
group by rollup (p.prod_category, p.prod_subcategory)
)
where prod_category = 'Women'
/
Similar query for different prod_category can use cached result
SQL Result Cache
Execution plan for query using result cache
------------------------------------------------------------------
| Id | Operation | Name |
------------------------------------------------------------------
| 0 | SELECT STATEMENT | |
| 1 | RESULT CACHE | fz6cm4jbpcwh48wcyk60m7qypu |
| 2 | SORT GROUP BY ROLLUP | |
|* 3 | HASH JOIN | |
| 4 | PARTITION RANGE ITERATOR| |
|* 5 | TABLE ACCESS FULL | SALES |
| 6 | VIEW | index$_join$_001 |
|* 7 | HASH JOIN | |
| 8 | INDEX FAST FULL SCAN | PRODUCTS_PK |
| 9 | INDEX FAST FULL SCAN | PRODUCTS_PROD_CAT_IX |
------------------------------------------------------------------
Depends... based on
Query repetitiveness
Query execution times
DML activity (cache invalidation frequency)
Remember data warehouse workload
Query may run 30 minutes
Query may return 5 rows
Query served from result cache would take a split second
SQL Result Cache
Opportunity for benefit
48
Use DBMS_RESULT_CACHE package to manage
Use V$RESULT_CACHE_* views to monitor
SQL Result Cache
Management
Result cache is disabled for queries containing
Temporary or dictionary tables
Non-deterministic PL/SQL functions
Sequence CURRVAL and NEXTVAL
SQL functions current_date, sysdate, sys_guid, etc.
Result cache for distributed queries
Set RESULT_CACHE_REMOTE_EXPIRATION > 0
0 means distributed queries are not cached
Default is 0
DML/DDL on remote database will not expire cached results
SQL Result Cache
Restrictions
Result cache does not automatically release memory
Grows until maximum size is reached
DBMS_RESULT_CACHE.FLUSH purges memory
Bind variables
Cached result is parameterized with variable values
Cached results can only be found for same variable values
Cached result will not be built if
Query is built on a non-current version of data (read consistency
enforcement)
Current session has outstanding transaction on table(s) in query
Flashback queries can be cached
SQL Result Cache
Caveats
Retail customer data (~50 GB)
Concurrent users submitting queries randomly
Executive dashboard application with 12 heavy analytical queries
Cache results only at in-line view level
12 queries run in random, different order
4 queries benefiting from the cache
Measure average, total response time for all users
# Users No cache Cache Improvement
2 186 s 141 s 24%
4 267 s 201 s 25%
8 447 s 334 s 25%
SQL Result Cache
Internal benchmark
49
Calculate a complex derived metric like the ratio of
the highest median income grouped by state to the
lowest median income grouped by state over the
whole population
You need a PL/SQL function
Like the results stored in the SQL Result Cache, the
data changes fairly slowly but the query is repeated
fairly often
The challenge
function f2 return t1%rowtype
is
...
begin
select a, m into r1.a, r1.b from ...;
select a, m into r2.a, r2.b from ...;
r.a := r1.a + r2.a;
r.b := r1.b + r2.b;
return r;
end f2;
~ 2,000 milliseconds for each new call
Function implementation
function f2 return t1%rowtype
result_cache relies_on(t1, t2)
is
...
begin
select a, m into r1.a, r1.b from t1;
select a, m into r2.a, r2.b from t2;
r.a := r1.a + r2.a;
r.b := r1.b + r2.b;
return r;
end f2;
~ 0 milliseconds for each new call
Using the function cache
50
Both are cross-session and RAC interoperable
Set RESULT_CACHE_MAX_SIZE > 0 on all instances, or = 0
on all instances
Memory pool is instance-specific
Both build on the same infrastructure
Same RESULT_CACHE_MAX_SIZE initialization parameters
Same DBMS_RESULT_CACHE management package
Same V$RESULT_CACHE_* performance views
Similarities with SQL Result Cache
Summary
The Result Cache saves results of queries,
reducing overhead needed on repeated
use
Result cache is invalidated if there is DML
or DDL on cached tables
Result Cache can be used to satisfy part of
a query
PL/SQL Function Cache can cache
function results
51
PresentersName
PresentersTitle
Oracle OLAP
Module Agenda
Oracle OLAP Option
Cube Organized Materialized Views
Querying OLAP Cubes
Oracle OLAP Option
52
A full featured multidimensional OLAP
server
Excellent query performance for ad-hoc /
unpredictable query
Enhances the analytic content of business
intelligence application
Fast, incremental updates of data sets
A summary management solution for SQL
based business intelligence applications
An alternative to table-based materialized
views, offering improved query performance
and fast, incremental update
Oracle OLAP Option
Overview
One cube can be used as
A full-featured multidimensional cube, servicing dimensionally
oriented business intelligence applications
A summary management solution to SQL-based business
intelligence applications as cube-organized materialized
views
A analytically rich data source to SQL-based business
intelligence applications as SQL cube-views
Oracle OLAP Option
Overview
Oracle OLAP Option
Single source of truth
SQL Query
OLAP Query
Metadata
Data
Business Rules
Extract, Load
& Transform (ELT)
Centrally managed data, meta data and business rules
An embedded OLAP solution
Runs within Oracle Database
Enterprise Edition
Data are stored in Oracle data
files
Meta data in the Oracle Data
Dictionary
Fully compatible with RAC / Grid
computing
Oracle OLAP Option
Features
53
A secure solution
Oracle users are OLAP users
SQL GRANT / REVOKE on
OLAP cubes and dimensions
Compatible with Virtual Private
Database
Fine grained cube security
Oracle Authentication
SQL Cube Access Control
Virtual Private Database
Fine Grained Cube Security
Oracle OLAP Option
Features
An open solution
Oracle cubes and dimensions
are queried using
SQL
PL / SQL
Oracle OLAP API
Transparent access as cube-
organized materialized view
SQL
SELECT
time,
product,
customer,
sales_ytd
FROM
sales_cube
Oracle OLAP Option
Features
A rich solution
Rich aggregations
Time series
Indices and market shares
Rankings
Forecasting
Allocations
Statistics
Calculations are embedded
in the database
Centrally managed for consistency
Accessible by any application
Oracle OLAP Option
Features
Predictable query environment
Predefined reports
Predefined calculations
Less exploration of data
Exploratory query environment
Users define reports
Users access any data
Users define calculations
More users amplify this effect
OLAP cubes are optimized for ad-hoc, exploratory
usage patterns
Static Reporting
Self Service Reporting
and Analysis
Oracle OLAP Option
Performance
54
OLAP cubes offer excellent
performance for unpredictable
query patterns
Appropriate for both
static and exploratory
reporting
Advantages increase
as reporting becomes
more as hoc
Oracle OLAP Option
Performance
OLAP Cubes offer fast,
incremental updates of data
sets
Manage all summaries in a
single database object
Fast, incremental
materialized view refresh
Incremental / fast
aggregation
Cost-based
aggregation
Oracle OLAP Option
Performance
Oracle OLAP in the data warehouse
Use Oracle OLAP to enhance your data warehouse
Simplified summary management
Speed of thought query performance
Advanced time series analysis and analytic content
Centralized management of data, meta data, calculations and
security
Embedded Oracle OLAP is preferred by IT to external
solutions
Use the database you already own
Use the BI tools you already own
Use Oracle skills you already have
Embedded Oracle OLAP is secure and enterprise ready
Oracle OLAP in the data warehouse
55
Querying OLAP cubes
OLAP Cube views
SQL Query of Oracle Cubes
OLAP Cube Includes
All levels of summarization
Rich analytical calculations
Cube is represented as star
schema of relational views
Dimension and fact views
Detail and summary fact rows
Rich analytic fact columns
OLAP Cube views
Simple SQL queries access cube content
SELECT cu.long_description customer,
f.profit_rank_cust_sh_parent,
f.profit_share_cust_sh_parent,
f.profit_rank_cust_sh_level,
f.profit,
f.gross_margin
FROM time_calendar_view t,
product_primary_view p,
customer_shipments_view cu,
channel_primary_view ch,
units_cube_view f
WHERE t.level_name = 'CALENDAR_YEAR'
AND t.calendar_year = 'CY2006'
AND p.dim_key = 'TOTAL'
AND cu.parent = 'TOTAL'
AND ch.dim_key = 'TOTAL'
AND t.dim_key = f.TIME
AND p.dim_key = f.product
AND cu.dim_key = f.customer
AND ch.dim_key = f.channel;
Application Express on Oracle OLAP
220
OLAP Cube views
OBIEE access
Oracle Business
Intelligence Enterprise
Edition querying time series
calculations directly from
an Oracle cube using SQL.
Oracle cubes can make any
BI tool smarter and faster.
56
SQL issued by Oracle BI EE
against views of Oracle
cube and dimensions.
New Joined Cube Scan row
source pushes joins into
the cube and accesses
summary data and
calculations.
OLAP Cube views
OBIEE access
Cube-organized
materialized views
SQL Query of OLAP Cubes
BI Application
Cube
Materialized
Views
SQL
Automatic
Query
Rewrite
BI Application
Cube Views
SQL
Oracle Cube
Transparently enhance the query performance of BI
applications
Data is managed in an Oracle cube
Fast query
Fast refresh
Manage a single cube instead of 10s, 100s or 1,000s of
table-based materialized views
Applications query base / detail relational tables
Oracle automatically rewrites SQL queries to OLAP cubes
Access to summary data in the cube is fully transparent
Cube-organized materialized views
Overview
57
Summary Data: Collections of Materialized Views
Materialized Views
Typical MV Architecture Today
Fact Table: Sales by Day, Item, Customer and Channel
BI
Application
SELECT
SUM(sales)
GROUP BY
quarter,
brand,
region,
channel
Automatic
Query
Rewrite
Users expect excellent
query response for all
summary queries
Might require 10s, 100s or
even 1,000s of materialized
views
Difficult to manage
Longer build and update times
Cube-Organized Materialized Views
Automatic Query Rewrite
BI
Application
SELECT
SUM(sales)
GROUP BY
quarter,
brand,
region,
channel
Automatic
Query Rewrite
A single cube manages
summaries for all
groupings in the model
A cube can be represented
as a cube-organized
materialized view
Oracle automatically
rewrites summary queries
to the cube
A singe cube can replace
10s, 100s or 1,000s of
materialized views
227
Typical query issued by
Oracle Business
Intelligence Enterprise
Edition.
Query is automatically
rewritten by Oracle to
access summary data in the
cube-organized materialized
view.
Cube-Organized Materialized Views
Fast, Incremental MV Refresh
A single cube is refreshed
using MV refresh system
Fast, incremental
update from MV logs.
Fast, incremental
aggregation within the
cube.
Efficient management
of sparse data sets.
Replaces 10s, 100s
or even 1,000s of
table-based MVs
MV Refresh
58
An excellent summary management solution for
business intelligence tools such as BI EE,
MicroStrategy, Cognos and Business Objects
Cube organized materialized views are similar to
materialized views on pre-built tables
Cube organized materialized views are meta data only they
do not store data; data comes from the cube
A common implementation will be to leave detail data
in tables and create the cube at aggregate levels
E.g. tables with day, customer and cube with month, zip code
Usage
Compares performance of table-based materialized
views with cube-organized materialized views with
goals of:
Improving query performance of SQL-based BI tools
Reducing build/update times
Source data
Fast moving consumer goods company data
7 dimensions
20 million fact rows
Case study
Methodology
Indexes and materialized views were created as per Oracle
SQL Access Advisor recommendations.
124 materialized views
198 indexes
Oracle cube and cube-organized materialized views were
created by DBA.
1 compressed cube
Pre-aggregated to 20%
1469 test queries
Case study
Measurements
Time to load data and prepare it for query
MVs: Create MVs, create indexes and compute statistics
Cube: Load data and aggregate.
Query performance
Run the same 1469 queries against MVs and cube.
Case study
59
0
20
40
60
80
100
120
140
160
180
Table MVs Cube MV
Load and Aggregate
Query
Time in minutes to
Case study results
Summary
Oracle OLAP is a way to store data in
cubes
Oracle OLAP cubes deliver fast and
consistent performance for ad-hoc
analytical queries
automatically aggregate along all cube
dimensions
Cube-organized materialized views can
provide significant performance
advantages in terms of loading and query
vs. materialized views
60
PresentersName
PresentersTitle
Oracle 11g Advanced Compression
Module Agenda
Data growth challenges
Data compression
Unstructured data compression
Backup compression
61
Explosion in data volume
Government regulations (Sarbanes-Oxley, etc)
User generated content (Web 2.0)
Application consolidation
Limited technology budgets
Need to optimize storage consumption
Also maintain acceptable application performance
Intelligent and efficient compression technology can
help address these challenges
Data compression
Table Compression
Introduced in Oracle Database 9i Release 2
Compression during bulk load operations (Direct Load, CTAS)
Data modified using conventional DML not compressed
Optimized compression algorithm for relational data
Improved performance for queries accessing large
amounts of data
Fewer I/Os
Buffer Cache efficiency
Data is compressed at the database block level
Compression enabled at either the table or partition level
Completely transparent to applications
Advanced Compression Option
Oracle Database 11g introduces a comprehensive set
of compression capabilities
Structured/relational data compression
Unstructured data compression
Compression for backup data
Network transport compression
Reduces resource requirements and costs
Storage
Network bandwidth
Memory usage
Ideal for Information Life Cycle (ILM)
62
OLTP Table Compression
Oracle Database 11g extends table compression for
OLTP data
Support for conventional DML Operations
(INSERT, UPDATE)
New algorithm significantly reduces write overhead
Batched compression ensures no impact for most OLTP
transactions
No impact on reads
Reads may actually see improved performance due to fewer
I/Os and enhanced memory efficiency
OLTP Table Compression Process
Initially
Uncompressed
Block
Compressed
Block
Partially
Compressed
Block
Compressed
Block
Empty
Block
Legend
Header Data
Free Space
Uncompressed Data
Compressed Data
Block-Level Batch Compression
Patent pending algorithm minimizes performance overhead and
maximizes compression
Individual INSERT and UPDATEs do not cause recompression
Compression cost is amortized over several DML operations
Block-level (local) compression keeps up with frequent data changes in
OLTP environments
Competitors use static, fixed size dictionary table thereby compromising
compression benefits
Extends industry standard compression algorithm to databases
Compression utilities such as GZIP and BZ2 use similar adaptive, block
level compression
Smith John 3
Doe Jane 4
Doe Jane 2
Doe John 1
LAST_NAME FIRST_NAME ID
Employee Table
Initially Uncompressed
Block
INSERT INTO EMPLOYEE
VALUES (5, Jack, Smith);
COMMIT;
1JohnDoe 2Jane
Doe 3JohnSmith 4
Jane Doe
Free Space
Header
63
Block
John=|Doe=|Jane=|Smith=
Header
Doe Jane 4
Smith John 3
Smith Jack 5
Doe Jane 2
Doe John 1
LAST_NAME FIRST_NAME ID
Employee Table
1JohnDoe 2Jane
Doe 3JohnSmith 4
Jane Doe
Free Space
1 2 3 4
5Jack
Free Space
Compressed Block
Local
Symbol Table
1JohnDoe 2Jane
Doe 3JohnSmith 4
Jane Doe 5Jack
Smith Free Space
Header
Uncompressed Block
John=|Doe=|Jane=|Smith=
Header
1JohnDoe 2Jane
Doe 3JohnSmith 4
Jane Doe
Free Space
1 2 3 4
5Jack
Free Space
Compressed Block
Local
Symbol Table
More Data
Per Block
Table Compression Syntax
OLTP Table Compression Syntax:
CREATE TABLE emp (
emp_id NUMBER
, first_name VARCHAR2(128)
, last_name VARCHAR2(128)
) COMPRESS FOR ALL OPERATIONS;
Direct Load Compression Syntax (default):
CREATE TABLE emp (
emp_id NUMBER
, first_name VARCHAR2(128)
, last_name VARCHAR2(128)
) COMPRESS [FOR DIRECT_LOAD OPERATIONS];
Best Practices
Compress your 10 largest tables
80/20 rule 20% of your tables consume 80% of your space
Bigger block sizes may lead to better compression
ratios
B*-tree index compression
Validate index and review INDEX_STATS
INDEX_STATS.OPT_CMPR_COUNT
INDEX_STATS.OPT_CMPR_PCTSAVE
Bitmap indexes are already highly compressed
Good for low and medium cardinality
64
Getting Started with Compression
Compression Advisor
Shows projected compression ratio for uncompressed tables
Reports actual compression ratio for compressed tables
Getting Started with Compression
Monitoring Compression
DBA_TABLES
COMPRESSION (ENABLED/DISABLED)
COMPRESS_FOR (ALL OPERATIONS / DIRECT LOAD OPERATIONS)
Data from Oracles implementation of Oracle Applications
Compressed the 10 largest tables
Two copies of each table: compressed and uncompressed
Oracle Enterprise Linux
Oracle Database 11g Release 1
Test Queries
Full table scan
Index range scan
DML operations (INSERT, UPDATE, DELETE)
Real world compression example
Overview
Storage Utilization
MB
0
500
1000
1500
2000
2500
No Compression Compression
More than 70%
Storage Savings
Results
65
Table Scan Performance
Time
(seconds)
0
0.1
0.2
0.3
0.4
Compression is
2.5x Faster
Results
0
2000
4000
6000
8000
10000
12000
14000
Table Scan Reads
Physical
Reads
Compression
performs
3.5x fewer reads
Results
0
0.01
0.02
0.03
Index Range Scan Performance
Time
(seconds)
No Performance
Impact on
Index Scans
Results
DML Performance
Time
(seconds)
0
5
10
15
20
25
30
35
Compression
has less than 3%
overhead
Results
66
Unstructured data
compression
Introduction to SecureFiles
SecureFiles is a new 11g feature designed to break the
performance barrier keeping file data out of databases
Next-generation LOB
Superset of LOB interfaces allows easy migration from LOBs
Transparent deduplication, compression, and encryption
Enables consolidation of file data with associated relational data
Single security model
Single view of data
Single management of data
Scalable to any level using SMP scale-up or grid scale-out
SecureFiles deduplication
Secure Hash
Enables storage of a single physical image for duplicate data
Significantly reduces space consumption
Dramatically improves writes and copy operations
No adverse impact on read operations
May actually improve read performance for cached data
Duplicate detection happens within a table, partition or sub-partition
Specially useful for content management, email applications and
data archival applications
SecureFiles compression
Huge storage savings
Industry standard compression algorithms
2-3x compression for typical files (combination of doc, pdf, xml)
Allows for random reads and writes to compressed SecureFile data
Can be specified at a partition level
Automatically detects if SecureFile data is compressible
Skips compression for already compressed data
Auto-turn off compression when space savings are minimal or zero
Two levels of compression provide different compression ratios
SecureFiles compression is independent of table or index
compression
67
SecureFiles compression syntax
Compression Syntax
CREATE TABLE docs (
doc_id NUMBER,
doc BLOB)
LOB(image) STORE AS SECUREFILE
(TABLESPACE lob_tbs COMPRESS);
Deduplication Syntax
CREATE TABLE images (
image_id NUMBER,
image BLOB)
LOB(image) STORE AS SECUREFILE
(TABLESPACE lob_tbs DEDUPLICATE);
Backup compression
Metadata compression available since Oracle Database 10g
Oracle Database 11g extends compression to table data during
exports
No need to decompress before import
Single step compression of both data and metadata
Compressed data directly hits disk resulting in reduced disk space
requirements
Internal tests reduced dump file size up to 75%
Compression factor comparable to GNU gzip utility
Application transparent
Complete Data Pump functionality available on compressed files
Data Pump compression
Overview
Performance cost: ~10% overhead
Compression Ratio: comparable to gzip
Compression Method OE/SH Spatial Table
expdp compression=none 6.0 MB 26.6 MB
expdp compression=all 1.5 MB
(74.7%)
9.9 MB (62.7%)
gzip cv1 1.1 MB
(82.7%)
11.4 MB (57.1%)
gzip cv6 835 KB
(86.2%)
10.2 MB (61.7%)
gzip cv9 818 KB
(86.5%)
10.1 MB (62.0%)
Compress 1.6 MB
(74.2%)
13.8 MB (48.1%)
Results
68
Syntax
DataPump syntax
PROMPT> expdp hr FULL=y
DUMPFILE=dpump_dir:full.dmp COMPRESS
Fast RMAN Compression
Compresses the backup set contents before writing them to disk or
tape
No extra decompression steps are required during recovery when
you use RMAN compression
High performance, industry standard compression algorithm
40% faster backup compression versus Oracle Database 10g
Suitable for fast, incremental daily backups
Reduces network usage
Recovery Manager (RMAN) compression
Overview
Fast RMAN Compression Configuration
RMAN> CONFIGURE COMPRESSION ALGORITHM zlib;
RMAN Compression Syntax
RMAN> backup as COMPRESSED BACKUPSET database
archivelog all
Syntax
Data from Oracles implementation of Oracle Applications
3.5 GB Database
Oracle Enterprise Linux
Oracle Database 11g Release 1
Test 1: Slow I/O (16 MB/s)
11g RMAN without Compression
10g RMAN with Compression
11g RMAN with FAST Compression
Test 2: Fast I/O (200 MB/s)
11g RMAN without Compression
10g RMAN with Compression
11g RMAN with FAST Compression
Example
69
Backup Size Comparison
GB
0
500
1000
1500
2000
2500
3000
3500
No Compression 10g Compression 11g Compression
Compression
reduced backup
size by 6x
Results
Backup Speed Comparison
Slow I/O (Tape)
0
50
100
150
200
250
11g Compression
reduces backup
time by almost 3x
Results
Slow I/O (Tape)
0
50
100
150
200
250
11g Compression
is almost 2.5x
faster than 10g
Compression
Results
Fast I/O (Disk)
0
50
100
150
200
250

11g Compression
is almost 2.5x
faster than 10g
Compression
Results
70
Summary
Oracle Database 11g OLTP compression
compresses data on load and DML
operations
OLTP compression does not impact the
performance of real-time write operations
SecureFiles offer compression and
deduplication for large objects
Backup compression improves backup
speed and reduces space requirement
71
PresentersName
PresentersTitle
Oracle Data Integration
Module Agenda
Historical overview
Modern data integration
Oracle data integration strategy
Oracle Warehouse Builder
Historical overview
Data Warehouse
Ancient History
1985 1995 Controlled Chaos
Fragmented strategy for marts vs. warehouse
No practical notion of Enterprise Data Warehouse
Data Integration:
Hand-coded scripts (external to DB)
Not optimized
Procedural transformations (PL/SQL etc)
Few Data Integration tools
No formal methodology, metrics or governance
72
Data Warehouse
Good Old Days
1995 2005 Formal Methods and Discipline
Strategic choices for marts vs. warehouse
Top-down (Inmon) vs. Bottom-up (Kimball)
Formal notion of Enterprise Data Warehouse
Data Integration:
Tool-based Data Integration solutions
Optimized, parallel server-based transforms
Formal methodology, metrics or governance
Reduced reliance on hand-coded scripts and procedural
transformations (PL/SQL etc)
Data Warehouse
Revival Period
2005 2015 Specialized Warehouse Solutions
Technology-driven choices for high-end DWs
Commodity H/W vs. optimized appliances
Relational/star vs. columnar (vs. Cubes/OLAP)
Database + BI vs. distributed analytic apps (Hadoop etc)
EDW as a source of truth vision morphs and expands to
MDM as a distinct problem domain
Data Integration is still stuck in the Good Old Days
Good Old Days Modern Alternative
Hub-based Runtime Agent-based Runtime
Centralized ETL Server Optimized E-LT (DW Appliance)
Mainly Batch Mainly Real Time / Trickle Feed
Modern data integration
Traditional ETL + CDC
Invasive capture on OLTP systems
using complex adapters
Transformations in ETL engine on
expensive middle tier servers
Bulk load to the data warehouse with
large nightly/daily batch
Continuous feeds from operational
systems
Non-invasive data capture
Thin middle tier with transformations
on the database platform (target)
Mini-batches throughout the day or
bulk processing nightly
Modern Data Integration Approach
Heterogeneous, Real-time, High Performance E-LT
Modern E-LT + Real-time
Staging
Trickle
Lookup
Data
Load
Extract
Lookup
Data
Xform Xform
Bulk
A
g
e
n
t
A
g
e
n
t
Heterogeneous
73
ETL Batch Integration
Good old days
Stage Prod
Lookup
Data Sources
ETL engines
require BIG
H/W and heavy
parallel tuning
Extract Transform Load Lookups/Calcs Transform Load
ETL Engine(s)
Meta Lookup
Data
ETL Metadata
Development,
QA, System(etc)
Environments
Good tools, but:
Expensive environments, performance
bottlenecks, too many data hops,
proprietary skills w/vendor lock-in, and
heavy optimization in complex situations
Wont scale w/new generation of DWs
Sources
Meta
Stage Prod
Lookup
Data
E-LT
Agent
Data Movement
Set-based SQL
transforms
typically faster
SQL Load
inside DB is
always faster
Development,
QA, System(etc)
Environments
Same good tools you expect, plus:
Reduce Data Center costs, de-commission servers
Open frameworks, non-proprietary SQL skills
Deploys seamlessly alone or within SOA servers
Scales linearly with modern DW servers
Data Transformation
Agent-based integration
Modern ETL
Real Time Replication
Good old days
Stage Prod
Lookup
Data Sources
CDC Hub(s)
Good tools, but:
Arcane capture process, sometimes invasive
Okay for data integration Changed Data Capture, but:
Not used for Active-Active / ZDT migrations
Not used for high availability or disaster recovery
ETL Engine(s)
Transaction Apply
Mgmt Server
Agent-based
Modern Real Time Replication
Sources Stage Prod
Lookup
Data Data Movement
Capture
Agent
Replicat
Agent
Same good tools you expect, but:
Not dependent on hardware for replication
Capable of heterogeneous, Active-Active deployments
Suitable for Zero Downtime migrations
Point-in-time recovery
74
Data Capture Architecture
On-Disk Logs
Log Tables
Triggers Updates
Inserts
Deletes
Oracle
IBM DB2
MSFT SQL Server
Sybase
Teradata
Enscribe
Next Generation capabilities
Non-invasive, heterogeneous, disk-based log access
Suitable for CDC + high availability & Active-Active
Bi-directional and high performance
Check-pointing and simple trail/queue management
Data Integration
Good old days
Monolithic & expensive environments
Fragile, hard to manage
Difficult to tune or optimize
Stage Prod
Lookup
Data Sources
ETL engines
require BIG
H/W and heavy
parallel tuning
ETL Engine(s)
Meta Lookup
Data
ETL Metadata
Development,
QA, System(etc)
Environments
CDC Hub(s)
Transaction Apply
Mgmt Server
Data Integration
Modern architecture
Lightweight, inexpensive environments agents
Resilient, easy to manage non-invasive
Easy to optimize and tune uses DBMS power
Sources
Meta
Stage Prod
Lookup
Data Data Transformation
E-LT
Agent
Bulk Data Movement
Set-based SQL
transforms
typically faster
SQL Load
inside DB is
always faster
Development,
QA, System(etc)
Environments
Capture
Agent
Replicat
Agent
Oracles data integration
product strategy
75
The solution for enterprise-wide real-time data
Dramatically improve the accessibility, reliability, and
quality of critical data across enterprise systems
Data Quality
Databases
Distributed
systems
Legacy systems
OLAP systems
OLTP systems
Mission critical
systems and data
Business
Intelligence,
Performance
Management
Reporting for
customers and
partners
Real-time Data
Data Access
Best-in-class leader in real-time data
integration solutions, combined with high
availability
500 customers, and 4,000+ implementations in
Financial Services, Communications,
Healthcare, Public Sector, Retail & Utilities
industries:
Top 4 of 5 largest commercial banks
Top 3 of 3 busiest ATM networks
Top 8 of 10 financial data services
companies
Top 4 of 5 telecommunications providers
Top 3 of 5 largest retail companies
Oracle GoldenGate
Best-in-class real-time data integration
Best-in-class Real-time Data Integration
Oracle Data Quality and
Data Profiling
Oracle Data Integrator EE
MDM
Applications
SOA
Platforms
Oracle
Applications
Business
Intelligence
Activity
Monitoring
Fusion
Applications
Data Cleansing
Data Profiling
File System
Real-Time
Data
OLTP
System
Data Warehouse/
Data Mart OLAP Cube
Data Migration Services Bulk Data Services Data Quality Services Data Access Services
ELT/ETL
Oracle GoldenGate
Zero Downtime Migration
Data Replication
Real-time CDC
Data Services Layer
Enterprise Service Bus BPELProcess Manager Data Services
Data Transformation
Bulk Data Movement Match and Merge
Oracle Data Integration Solution
10x performance over traditional ETL
Fastest transformation in the industry on
Exadata v2 platform
Knowledge Modules provide a foundation
for extensibility
Heterogeneous sources and targets
Lowest TCO of ELT/ETL vendors
Oracle Data Integrator Enterprise Edition
Bulk data movement and transformation
Data Integrator
Enterprise Edition
76
Fastest performance of changed data
capture in the industry
Real-time, bi-directional data replication
for zero-down time migrations and
disaster recovery
Support for heterogeneous sources and
targets
Guaranteed delivery of transactions
Oracle GoldenGate
Data replication for high availability
Data Integrator
Enterprise Edition
Oracle GoldenGate
Integrated seamlessly into Oracle Data
Integrator for faster implementations
Best-in-class leader for pre-built data
integration rules
Improves confidence in data sources and
targets
Oracle Data Quality
Data quality and profiling for integration processes
Oracle Data Profiling,
Data Quality
Data Integrator
Enterprise Edition
Oracle GoldenGate
Oracle Warehouse
Builder
OWB Product Line
As listed on price list
OWB in
ODI-
EE
license
Add-
ons
Base ETL Tool Warehouse Builder in Database
(OWB in DB)
Mapping Editor and Debugger
Basic dependency management
Robust scripting language
Fuzzy match/merge name and address
cleansing
Warehouse Builder Flexible, heterogeneous connectivity
Extensible metadata framework
Dimensional modeling
Interactive lineage and impact analysis
Adapters for
Application Integration
eBS
Siebel
PeopleSoft
SAP
Reveals business views of application data
Natively access SAP R/3 hosted on any
database
read from and write to eBusiness Suite,
PeopleSoft, or Siebel
Data Quality Tools Data Profiling and Quality Profile table, file, and application data in place or
stage the data
Auto-detect data rules.
Audit, auto-correct incoming data.
Data Watch and Repair Data quality for Oracle MDM: Siebel UCM, PIM,
CDH
Write back corrected data directly to Siebel UCM
77
ODI
2009 2010 2011
Data Integration Platform
2006 2008 2007 2005
OWB
Unified platform is a
superset of both products
OWB/ODI investments
are fully protected
No forced migrations
Easy upgrade path
No regressions
in functionality
Training and support
continues
for both products
common pricing
unified dev team
11gR2
OWB in DB
Lifetime support
of OWB functionality
includedin
Database license
11g
11gR1 10gR2
Oracle data integration
Product direction
PresentersName
PresentersTitle
Sizing Oracle Data Warehouses
Module Agenda
RAC for data warehousing
Sizing hardware for data warehousing
Sizing RAC for data warehousing
Scaling hardware for data warehousing
Oracle Reference Configurations
Storage considerations
78
RAC for data warehousing
RAC for Data Warehousing
Shared everything paradigm
Runs all applications transparently
Manageable
Virtual resource pool
Scale out of a node for large operations
Transparent and adaptive inter-node parallelism
Add low cost capacity on demand
Dynamic workload provisioning
Logical services concept to map workloads to resources
Reduction of TCO
Node autonomy for high availability and to enforce physical
boundaries for different jobs
Potential framework for physical workload separation
With Oracle, queries will automatically run on a single
node, eliminating traffic over the interconnect
Q1 Q2 Q4 Q3
Q5 Q7 Q6 Q8
Q9 Q12 Q11 Q10
With Oracle, queries will automatically run on a single
node, eliminating traffic over the interconnect
Very large queries utilize all resources on the cluster
Large Query
79
ETL server is busy using ETL outside the database
Moderate load on database
No load on end user server
11:00 PM
ETL DB End Users
Data warehouse
Varying workload by time
No load on ETL server
Canned reporting
Aggregation and potential data mart maintenance
No load on end user server
ETL DB End Users
3:00 AM
Data warehouse
No load on ETL server
End user server is busy using analysis outside the
database
ETL DB End Users
9:00 AM
Data warehouse
Maximum total workload used for system sizing
Workload max total < workload max components
All workload is evenly spread across all nodes at any
point in time
ETL+
DB+
End Users
RAC and the data warehouse
An optimal solution
80
If workload
autonomy is used
to physically
separate workloads
Data Warehouse DB
RAC
Services and workload autonomy
If workload
autonomy is used
to physically
separate workloads
Data Warehouse DB
Only one workload
(service) is affected
by a failure
RAC
If workload
autonomy is used
to physically
separate workloads
Data Warehouse DB
Only one workload
(service) is affected
by a failure
And can even be
migrated to
another node
RAC
Data warehousing was the first type of
application to leverage the grid
Static workload provisioning used since
Oracle OPS version 6.0
Oracle RAC introduced true transparency,
enabling the grid for every application
Oracle will be widely deployed in large
commercial computing grids in the future
The future already started
Information grid
The next step
81
What works for a single data warehouse
ETL+
DB+
End Users
Information grid
The next step
ETL
DB
End user tool
works in a larger environment as well
Information grid
The next step
ETL
DB
End user tool
Applications ERP
CRM
OLAP
Mining
works in a larger environment as well
... and for the whole enterprise information grid
Information grid
The next step
Sizing hardware for data
warehousing
82
Sizing Data Warehouses
Database CPUs Memory Actuators LUNs Disks Raid
Database CPUs Memory Actuators LUNs Disks Raid
An Unbalanced configuration
A Balanced configuration
100%
Possible
Efficiency
100%
Possible
Efficiency
100%
Achieved
Efficiency
< 50%
Achieved
Efficiency
Sample Customer Configuration
FC-Switch1 FC-Switch2
H
B
A
1
H
B
A
2
H
B
A
1
H
B
A
2
H
B
A
1
H
B
A
2
H
B
A
1
H
B
A
2
4 x P690 x 32-CPUs
4 x 18 x 2Gb HBA
EMC
6 x DMX 2000
P2 frames
Max IO throughput: 25GB/sec
Max IO throughput: 18GB/sec
72% capacity, 25%degradation
Max IO throughput: 5 6GB/sec
20 -24% capacity, 76 80% degradation
Disk
Array 1
Disk
Array 2
Disk
Array 3
Disk
Array 4
Disk
Array 5
Disk
Array 6
Disk
Array 7
Disk
Array 8
H
B
A
1
H
B
A
2
H
B
A
1
H
B
A
2
H
B
A
1
H
B
A
2
H
B
A
1
H
B
A
2
CPU Quantity and Speed dictate
number of HBAs
capacity of interconnect
HBA Quantity and Speed dictate
number of Disk Controllers
Speed and quantity of switches
Controllers Quantity and Speed dictate
number of Disks
Speed and quantity of disks
Disk Quantity and Speed
Balanced configuration
The weakest link defines the throughput
Disk
Array 1
Disk
Array 2
Disk
Array 3
Disk
Array 4
Disk
Array 5
Disk
Array 6
Disk
Array 7
Disk
Array 8
H
B
A
1
H
B
A
2
H
B
A
1
H
B
A
2
H
B
A
1
H
B
A
2
H
B
A
1
H
B
A
2
Sample Balanced Unit
Each machine has 2 CPUs
All four servers drive about
2 * 200MB/s * 4 = 1600 MB/s
83
Disk
Array 1
Disk
Array 2
Disk
Array 3
Disk
Array 4
Disk
Array 5
Disk
Array 6
Disk
Array 7
Disk
Array 8
H
B
A
1
H
B
A
2
H
B
A
1
H
B
A
2
H
B
A
1
H
B
A
2
H
B
A
1
H
B
A
2
2 * 200MB/s * 4 = 1600 MB/s
Each machine has 2 Gb HBAs
All 8 HBAs can sustain
8 * 200MB/s = 1600 MB/s
Disk
Array 1
Disk
Array 2
Disk
Array 3
Disk
Array 4
Disk
Array 5
Disk
Array 6
Disk
Array 7
Disk
Array 8
H
B
A
1
H
B
A
2
H
B
A
1
H
B
A
2
H
B
A
1
H
B
A
2
H
B
A
1
H
B
A
2
2 * 200MB/s * 4 = 1600 MB/s
8 * 200MB/s = 1600 MB/s
Each switch needs to support
800MB/s to guarantee a total
system throughput of 1600 MB/s
Disk
Array 1
Disk
Array 2
Disk
Array 3
Disk
Array 4
Disk
Array 5
Disk
Array 6
Disk
Array 7
Disk
Array 8
H
B
A
1
H
B
A
2
H
B
A
1
H
B
A
2
H
B
A
1
H
B
A
2
H
B
A
1
H
B
A
2
2 * 200MB/s * 4 = 1600 MB/s
8 * 200MB/s = 1600 MB/s
Each disk array has one
2Gbit controller
All 8 disk arrays can sustain
8 * 200MB/s = 1600 MB/s
Disk
Array 1
Disk
Array 2
Disk
Array 3
Disk
Array 4
Disk
Array 5
Disk
Array 6
Disk
Array 7
Disk
Array 8
H
B
A
1
H
B
A
2
H
B
A
1
H
B
A
2
H
B
A
1
H
B
A
2
H
B
A
1
H
B
A
2
2 * 200MB/s * 4 = 1600 MB/s
8 * 200MB/s = 1600 MB/s
Each disk array has one
2Gbit controller
All 8 disk arrays can sustain
8 * 200MB/s = 1600 MB/s
Interconnect Bandwidth
should >= 1600 MB/s
84
Build a balanced hardware configuration
Total throughput = # cores X 100-200 MB (depends on chip set)
Total HBA throughput = Total core throughput
If total core throughput =1.6GB will need 4 x 4Gb HBAs
Use 1 disk controller per HBA Port (throughput capacity must be
equal)
Switch must be same capacity as HBA and disk controllers
Max of 10 physical disks per controller (Use smaller drives 146 or 300
GB) and Min of 4
Minimum of 4GB of memory per core (8GB if using
compression)
Interconnect bandwidth should equal IO bandwidth (Infiniband)
Use ASM with RAID 1 mirroring for redundancy
Data warehouse hardware
Building a balanced configuration
Components
890 MB/s Infiniband
80 MB/s* 2 Gbit/s GigE NIC (interconnect)
2 Gbit/s
2 Gbit/s
8 * 2 Gbit/s
1/2 Gbit/s
-
Hardware spec
200 MB/s Disk controller
200 MB/s Fiber channel
1,200 MB/s 16 port switch
100/200 MB/s 1/2 Gbit HBA
100 - 350 MB/s CPU core
Sustained Component
* Limited by the protocol
30 - 50 MB/s Disk (spindle)
10 Gbit/s
Sizing RAC for data
warehousing
Analytic Server
Maximum workload for each component used for system sizing
ETL server must handle maximum ETL load
Database must handle ETL load plus end user load
Be prepared for the overlap of workload patterns
End user server must handle maximum end user load
Seasonal effects can change workload dramatically
DB ETL
Data warehouse hardware
Sizing for peak workload
85
Few Large Nodes
Many Small Nodes
or
?
Many nodes are more difficult to manage:
Increased maintainance (e.g. upgrades)
Performance problems are probably harder to diagnose
However, computing power loss during rolling upgrades
has less impact:
16 x 2 grid 6% less power
4 x 8 grid 25% less power
Few large or many small?
Manageability
Both alternatives are easy scale out
Recommendation: Add only nodes with similar performance
characteristics: CPUs, HBAs, NICs etc.
Scale-out increment is one node
Many small nodes add smaller increment for scale-out
Scale out
Number of nodes does not matter
Record breaking TPC-H numbers with Oracle Grid 4, 8 and 12-
Node Grids
Performance differences between Few-Large-Node and
Many-Small-Node grids:
Many-Small-Node grids might require more interconnect
bandwidth for parallel operations
Performance
86
Scaling hardware for data
warehousing
Common Sense Guidelines
A RAC system has to be configured as carefully as
any other system
Ensure a balanced system to start with
Many small versus few large nodes
Verify the I/O throughput of your system
Oracle cannot be faster than the OS
Data Warehousing business
is I/O business
Scalability
Requirement for more resources
Stable performance when a system grows
More data
More users accessing the system
More complex business requirements
Better performance for a stable system with more
resources
What is scalability?
With x% more resources response times reduce x%
With x% more resources I can support x% more users
Constant response times
With x% more resources I can handle x% more data
Constant number of users
Constant response times
If you want to measure scalability, define what you mean
87
If you start with a balanced system
All Data All Data
then keep it balanced
Increase computing resources
Scale-up: more CPUs, memory, HBAs
Scale-out: add servers
Increase IO capacity
Increase storage capacity
Data redistribution for optimal IO
automatically
Data Data
All Data All Data
You must keep the right
balance between CPU and
IO resources to ensure
good performance
Scaling a Data Warehouse Configuration
Build a balanced system to begin with
Today our customers can buy any combination of hardware components
to build a data warehouse.
Some percentage of customers inevitably end up with poorly configured
data warehouses
Grow a system in an incremental balanced manner
All components must be increased in balance to achieve scalability
The Oracle Optimized Warehouse Initiative helps to deploy and
grow balanced systems
Phase I: Optimized Warehouse Reference Configurations
Best practices and reference configurations
Phase II: Sun Oracle Database Machine
Pre-packaged, pre-configured, pre-installed, calibrated & validated
system a high performance data warehousing appliance
Oracle Reference
Configurations
88
Full Range of DW Solution Options
Partitioning
RAC
Reference
Configurations
Reference
Configurations
Documented best-practice
configurations for data
warehousing
Benefits:
High performance
Simple to scale; modular
building blocks
Industry-leading database
and hardware
Available today with HP,
IBM, Sun, EMC/Dell
Custom Custom
Flexibility for the most
demanding data warehouse
Benefits:
High performance
Unlimited scalability
Completely customizable
Industry-leading database
and hardware
Partitioning
RAC
Oracle
Optimized Warehouse
Oracle
Optimized Warehouse
Scalable systems pre-
installed and pre-configured:
ready to run out-of-the-box
Benefits:
High performance
Simple to buy
Fast to implement
Easy to maintain
Competitively priced
Partitioning
RAC
Flexibility
Pre-configured, Pre-installed, Validated
Oracle Optimized Warehouse
Reference Configurations
What is it?
Documented balanced system configurations for
pre-defined DWBI environments
Starting point for sizing a system
Balanced system consists of CPU, memory, I/O, and cabling
Sizing factors are raw data, avg. concurrent users and
workload complexity
Leverages scalable, modular components
Enables incremental growth (scale-in, scale-out)
Mitigates implementation risks
What is it?
Balanced system configurations are built on modular
balanced building blocks
Add building blocks as you grow
Each building block is a balanced unit
Disks Disks Disk Disk Disk Disk Disks Disks Disks Disks Disks Disks Disks Disks Disks Disks Disk Disk Disk Disk Disk Disk Disk Disk
Disks Disks
Different building
blocks provide guided
choices
Many small nodes
versus few large nodes
What is a balanced unit ?
The weakest link defines
the throughput
Each building block is a
balanced unit
Components to consider:
CPU: Quantity and speed
HBA (Host Bus Adapter):
Quantity and speed
Switch speed
Interconnect
Controller: Quantity and
speed
Disk: Quantity and speed
Disk
Array 1
Disk
Array 2
Disk
Array 3
Disk
Array 4
Disk
Array 5
Disk
Array 6
Disk
Array 7
Disk
Array 8
H
B
A
1
H
B
A
2
H
B
A
1
H
B
A
2
H
B
A
1
H
B
A
2
H
B
A
1
H
B
A
2
89
V
a
r
y
i
n
g

w
o
r
k
l
o
a
d
Varying system size
Spreadsheet-like guidance
Choice of workload and data volume
Choice of hardware and operating system
Oracle Optimized Warehouse Initiative
Sample Reference Configuration
Sample workload
Raw data volume
Sizing and pricing
starting point
(option 1)
Sizing and pricing
starting point
(option 2)
Hardware
Vendor
Model Status
IBM pSeries, xSeries Revision 4, Last updated for p6 chip set
Sun T class, E class, X class Revision 1
Dell/EMC Dell2900, EMC Clariion
Revision 3; last updated for multi-core
chips
HP Rx, Integrity, Proliant Revision 2, last updated for Montecito
AIX P-Series Clusters
Addition of AMD severs Mar, 2007 Revision 3
Updated for mutli-core P5 chips Oct, 2006 Revision 2
IBM System p for Oracle Data Warehousing White paper
Jan, 2008 (est) Revision 5
In progress Test 5 - 20 TB
Next steps
http://www-
03.ibm.com/support/techdocs/atsmastr.nsf/fe582a1e48
331b5585256de50062ae1c/296cb1445e1713fa862570
ce004fb4c3?OpenDocument
IBM.com webpage
http://www.oracle.com/solutions/business_intelligence/
quick-sizer.html
O.com webpage
Collateral
Updated for P6 chipset Jun, 2007 Revision 4
Nov, 2005 Revision 1
Deliverables
Linux X-Series Clusters
AIX P-Series SMP Hardware systems
Scope
Reference Configurations Status
Updated for CLARiiON CX-3 Nov, 2006 Revision 2
Oct, 2005 Revision 1
Dell-Oracle-EMC Information Appliance
Foundation Reference Configurations
White paper
Nov, 2007 (est) Revision 4
Finished September 2007 Optimized warehouse
Next steps
Oracle Information Appliance Foundations on
Dell and EMC (May 2007)
Press release
http://www.emc.com/solutions/oracle/10g_rac/pdf/H2382.
3_emc_sol_oracle_rac_10g_so_ldv.pdf
EMC.com webpage
www.oracle.com/solutions/business_intelligence/emc.ht
ml
O.com webpage
Collateral
Updated for Dell 2900 multi-core May.2007 Revision 3
Deliverables
EMC CLARiiON CX-3 disk arrays
Linux Dell 2900 SPM & Clusters Hardware systems
Scope
90
Sun large and extra large reference
configurations for Oracle Data Warehouses
White paper
Aug, 2007 (est) Revision 2
Fall 2007 Testing of 5TB SMP
and Cluster systems
Next steps
Oracle Announces Oracle-Sun Information
Appliance Found (March 20007)
Press release
http://www.sun.com/third-
party/global/oracle/bidw/sun_oracle_rc_ds.pdf
SUN.com webpage
www.oracle.com/solutions/business_intelligence/sun.htm
l
O.com webpage
Collateral
Feb, 2007 Revision 1
Deliverables
Sun StorageTek arrays
E Class servers SMP & Clusters
X Class servers Clusters
T Class servers Clusters Hardware systems
Scope
HP Reference Configuration for Oracle 10g
DW
White paper
Dec, 2007 (est) Revision 3
In progress Test 1TB HP-UX cluster
Next steps
Oracle and HP Unveil Reference
Configurations to Streamline Data Warehouse
Design and Deployment (July 2006)
Press release
http://h71028.www7.hp.com/enterprise/cache/150368-0-
0-0-121.html
HP.com webpage
www.oracle.com/features/hp/data-warehousing.html O.com webpage
Collateral
Updated for Montecito Mar, 2007 Revision 2
June, 2006 Revision 1
Deliverables
Linux Proliant Clusters
Linux Integrity Clusters
HP-UX Clusters
HP-UX SMP Hardware systems
Scope
Storage considerations
The ideal storage configuration
S.A.M.E.
Stripe And Mirror Everything
Optimize throughput across as many physical disks as
possible stripe across all devices
Automatic Storage Manager (ASM)
Implements S.A.M.E. per disk group (mirroring optional)
Simplifies and automates database storage management
Automatic rebalancing
Separate disk groups for different storage tiers
91
ASM
Optimal Performance No Space Wasted
Balance I/O
Across disks
Across disk arrays
LUN A LUN A1 1 LUN A LUN A2 2 LUN A LUN A3 3 LUN A LUN A4 4
LUN A LUN A5 5 LUN A LUN A6 6 LUN A LUN A7 7 LUN A LUN A8 8
Disk Disk12 12
Disk Disk8 8
Disk Disk16 16
Disk Disk4 4
Disk Disk11 11
Disk Disk7 7
Disk Disk15 15
Disk Disk3 3
Disk Disk10 10
Disk Disk6 6
Disk Disk14 14
Disk Disk2 2
Disk Disk9 9
Disk Disk5 5
Disk Disk13 13
Disk Disk1 1
ASM stripe ASM stripe
1 1 2 2 3 3 4 4
5 5 6 6 7 7 8 8
8 8
1 1 2 2 3 3 4 4
5 5 6 6 7 7
ASM Diskgroup g ASM Diskgroup g1 1
LUN A LUN A1 1
LUN A LUN A2 2
LUN A LUN A3 3
LUN A LUN A4 4
LUN A LUN A5 5
LUN A LUN A6 6
LUN A LUN A7 7
LUN A LUN A8 8
S.A.M.E. architecture
RAID-1 and ASM
Summary
RAC can solve the problem of over-
allocation of resources for different parts of
the data warehouse workload
The key to good performance in a data
warehouse is a balanced configuration
Reference configurations can deliver value
faster, eliminating testing and configuration
steps
92
PresentersName
PresentersTitle
Exadata and Sun Oracle Database Machine
Overview
Module Agenda
Challenges and solutions
Sun Oracle Database Machine
Exadata features
Exadata capacity, performance and
licensing
Summary
93
CHALLENGES ORACLE STRATEGY
Fragmented Solutions Information Consolidation
Difficulty of Self-Service BI Flexible, Guided, Easy-to-Use, Automated
Data Not Current Solutions for Just-in-Time understood data
Time to ROI /
Development Time
Horizontal and industry pre-built solutions,
appliance-like solutions
Rapidly Growing Data &
User Communities
Enterprise class solutions serving 1000s of
users and providing 100s TB of data
Deployment Manageability,
Security & Expense
Integrated solutions that are centrally managed
with advanced security / governance
IT Challenges
Oracle Database 11g
The Leading Database for Data Warehousing
Key Data Warehousing
Capabilities
Flexible Model Deployment
Embedded Analytics
Single Point of Management
Secure
24X7 Availability
Optimal Storage Management
Scale to hundreds of Terabytes
& large analyst communities
BI Applications
EPM Workspace
(Oracle Business Intelligence Enterprise Edition Plus: Desktop, Web, Search, Office, Alerts, Mobile)
BI Applications
(Sales, Service, Mkting,
Order Mgmt, HR, Fin, Projects)
Performance Management
(Strategy Mgmt, Business Planning, Profit Mgmt,
Financial Reporting & Compliance)
Business Intelligence Foundation
(Oracle Business Intelligence Enterprise Edition: BI Server, Essbase, Real-time Decisions)
Middleware
(BPEL, Oracle Data Integrator, MDM)
Database
(Oracle Database 11g Enterprise Edition, RAC, Partitioning, Exadata Software, OLAP, Data Mining
Advanced Compression, Advanced Security, Real Application Testing, Packs)
Database Machine / Storage
(Sun Oracle Database Machine, Exadata Storage)
Industry
Bundles
Oracle Data Warehouse components
Complete, Open, Integrated, Best-of-Breed
Sun Oracle Database Machine / Exadata
The Extreme ROI Platform
Fast Predictable Performance
Lower Ongoing Costs
The Fastest Time to Value & Lowest Risk
94
When it comes to speed, Oracle Exadata technology has
changed the game completely..
Grant Salmon
CEO
LGR Telecommunications
from Profit Magazine, February 2009
Exadata features
Traditional Scan Processing
With traditional storage, all
database intelligence
resides in the database
hosts
Very large percentage of
data returned from storage
is discarded by database
servers
Discarded data consumes
valuable resources, and
impacts the performance of
other workloads

I/Os Executed:
1 terabyte of data
returned to hosts

DB Host reduces
terabyte of data to 1000
customer names that
are returned to client

Rows Returned

SELECT
customer_name
FROM calls
WHERE amount >
200;

Table
Extents
Identified

I/Os Issued
Exadata Smart Scan Processing
Only the relevant columns
customer_name
and required rows
where amount>200
are are returned to hosts
CPU consumed by predicate
evaluation is offloaded
Moving scan processing off
the database host frees host
CPU cycles and eliminates
massive amounts of
unproductive messaging
Returns the needle, not the
entire hay stack

2MB of data
returned to server

Rows Returned

Smart Scan
Constructed And
Sent To Cells

Smart Scan
identifies rows and
columns within
terabyte table that
match request

Consolidated
Result Set
Built From All
Cells

SELECT
customer_name
FROM calls
WHERE amount >
200;
95
Additional Smart Scan Functionality
Join filtering
Star join filtering is performed within Exadata storage cells
Dimension table predicates are transformed into filters that are
applied to scan of fact table
Backups
I/O for incremental backups is much more efficient since only
changed blocks are returned
Create Tablespace (file creation)
Formatting of tablespace extents eliminates the I/O associated with
the creation and writing of tablespace blocks
2009 Oracle Corporation
50X
Up To
New
Exadata Hybrid Columnar
Compression
Data is stored by column
and then compressed
Query Mode for data
warehousing
Optimized for speed
10X compression ratio is
typical
Scans improve proportionally
Archival Mode for
infrequently accessed data
Optimized to reduce space
15X compression is typical
Up to 50X for some data
0
200
400
600
800
1000
1200
1400
I/O Resource Manager
OLTP Only
OLTP & Report
No IORM
OLTP & Report
IORM
OLTP Performance
in TPS
> 4X Improvement!
Allocate I/O bandwidth based on category, database
and storage disk
Guarantee QoS for critical operations
Enforce efficient utilization of disk resources
Multi-level, flexible resource plans
Exadata Storage Layout Example
ASM Mirroring and Failure Groups
ASM mirroring is used protect against disk failures
ASM failure groups are used to protect against cell
failures
Exadata Cell Exadata Cell
Hot Hot Hot Hot Hot Hot
Cold Cold Cold Cold Cold Cold
ASM
Disk Group
ASM
Failure Group
ASM
Failure Group
96
Data Protection Solutions
All single points of failure eliminated by the architecture
Hardware Assisted Resilient Data (HARD) built into storage
Prevent data corruption before it happens
Data Guard provides disaster protection & data corruption
protection
Automatically maintained second copy of database
Flashback provides human error protection
Snapshot-like capabilities to rewind database to before error
Recovery Manager (RMAN) provides backup to disk
Archiving and corruption protection
Can be used with Oracle Secure Backup (OSB) or third party tape
backup software
These work just as they do for traditional non-Exadata storage
Users and database administrator use familiar tools
Exadata Enhancements
Oracle Database 11g Enterprise Edition Release 2
Improved Exadata offload capabilities
Smart Scan offload for new Hybrid Columnar Compressed
Tables
Smart Scan offload for tables with more than 255 columns
Smart Scan offload for encrypted tablespaces and columns
Offload of Data Mining Model Scoring
Interleaved Intelligent Data Placement
Storage Indexes
New with R2
SI helps filter out data on disk (whereas
index helps find data)
Array of in-memory entries (called
region indexes)
Each region index stores column
summaries (e.g. min/max) for 1MB
region on disk
Transparent to the DB; and maintained
automatically
Min/Max can help eliminate IOs if data
cannot match the where clause
8
8
9
2
1
3
D C B A
5
7
6
Min B =
1
Max B
=3
Min B = 8
Max B =9
Min B = 5
Max B =7
Select * from Table where B=6
Ready-to-Load Database
Delivers a single, fully functioning database system
All servers properly configured and networked
All software configured (CRS, RAC, DB, Exadata)
Pre-configured optimized database created
Performance and functionality validated
High availability out-of-the-box
97
Exadata benefits
Exadata Benefits
Fast Predictable Performance
More predictable timeliness of
results
Faster results by moving Oracle
database intelligence to disk storage
Properly configured out-of-the-box
Ready to run - plug it in
More capabilities to support more
business analysts
Scale to support an enterprise
After carefully testing several data warehouse platforms, we chose the
Oracle Database Machine. Oracle Exadata was able to speed up one of
our critical processes from days to minutes. The Oracle Database Machine
will allow us to improve service levels and expand our service offerings.
Brian Camp
SVP, Infrastructure Services
KnowledgeBase Marketing
Reduce operational costs
Leverage Oracle skills investment and reduce its management
Protect your software investment
Always open to platforms from other vendors
Always price competitive
Open state-of-the-art components
Exadata Benefits
Lower ongoing costs
98
its fully supported so you wont have to worry about
the finger-pointing between vendors that might occur if
you build your own solution.
Carl Olofson
Research Vice President
IDC
Build From Scratch
with Components
OWI Reference
Configurations
Take delivery of Sun Oracle
Database Machine
Weeks to Months
Pre-implementation
System sizing
Acquisition of
components
Installation and
configuration
Acquisition of
components
Installation and
configuration
Testing and
Validation
Testing and
Validation
Weeks to Months
Sun Oracle
Database Machine
Faster deployment
Lower Risk
Database pre-configured
< 1 Week after Delivery
Exadata Benefits
Fastest time to benefit and lowest risk
You can easily remove six months of the
implementation cycle
we estimate theres up to a 70 percent reduction in
terms of cost of ownership compared to custom
solutions, just in terms of the personnel savings.
Paul Hartley
General Manager
LGR Telecommunications
Exadata hardware
99
Grid is the architecture of the future
Highest performance, lowest cost, redundant, incrementally scalable
Sun Oracle Database Machine delivers the first and only complete
grid architecture for all data management needs
Exadata Storage Server Grid
14 High-performance low-cost
storage servers
100 TB raw SAS disk storage
or
336 TB raw SATA disk storage
5TB flash storage!
RAC Database Server Grid
8 High-performance low-cost
compute servers
2 Intel quad-core Xeons each
InfiniBand Network
40 Gb/sec fault-tolerant unified
server and storage network
Same architecture as previous Database Machine
Same number and type of Servers, CPUs, Disks
Plus Flash Storage!
New
Latest Technologies
Faster
Better
80% Faster CPUs
33% More SAS Disk Capacity
100% More SATA Disk Capacity
50% Faster Disk Throughput
100% Faster Networking
125% More Memory
200% Faster Memory
100% More Ethernet Connectivity
Xeon 5500 Nehalem
600 GB SAS Disks
2 TB SATA Disks
6 Gb SAS Links
40 Gb InfiniBand
72 GB per DB Node
DDR3 DRAM
4 Ethernet links per DB Node
New
Hardware improvements
8 Sun Fire X4170 DB per rack
8 CPU cores 2x performance
72 GB memory 2.5x increase
Redundant HCA path to 2
switches
Fully redundant power and
cooling
Database Server
14 Sun Fire X4275 per rack
5x faster than conventional
storage
2x more storage capacity
Simplifies storage to eliminate
complex SAN architectures
Sun FlashFire Technology
turbocharges applications
Exadata Storage Server
100
Sun FlashFire Technology
Extreme Performance Accelerator
10x better I/O response time
5.25 Terabytes Flash per rack
1,000,000 IOPS per rack
20x IOPS speedup for Oracle
Integrated super caps for data
retention
New
InfiniBand Network
High Bandwidth, Low Latency
Sun Datacenter InfiniBand Switch 36
Fully redundant non-blocking I/O paths from servers to storage
2.88 Tb/sec bi-sectional bandwidth per switch
40 Gb/sec QDR, Dual port QSFP per server
8 Sun FireX4170 Oracle Database servers
14 Exadata Storage Servers (All SAS or all
SATA)
3 Sun Datacenter InfiniBand Switch 36
36-port Managed QDR (40Gb/s) switch
Admin Cisco Ethernet switch
Keyboard, Video, Mouse (KVM) hardware
Redundant Power Distributions Units (PDUs)
Single Point of Support from Oracle
3 year, 24 x 7, 4 Hr On-site response
Add more racks for additional scalability
Pre-configured for Scalability
Start Small and Grow
Quarter
Rack
Half
Rack
Full
Rack
101
Exadata Product Capacity
Quarter Rack Half Rack Full Rack
Raw Disk
1
SAS 21 TB 50 TB 100 TB
SATA 72 TB 168 TB 336 TB
Raw Flash
1
1.1 TB 2.6 TB 5.3 TB
User Data
2
(assuming no
compression)
SAS 6 TB 14 TB 28 TB
SATA
21 TB 50 TB 100 TB
1 Raw capacity calculated using 1 GB = 1000 x 1000 x 1000 bytes and 1 TB = 1000 x 1000 x 1000 x 1000 bytes.
2 - User Data: Actual space for end-user data, computed after single mirroring (ASM normal redundancy) and after
allowing space for database structures such as temp, logs, undo, and indexes. Actual user data capacity varies by
application. User Data capacity calculated using 1 TB = 1024 * 1024 * 1024 * 1024 bytes.
Exadata Product Performance
Quarter Rack Half Rack Full Rack
Raw Disk Data
Bandwidth
1,4
SAS 4.5 GB/s 10.5 GB/s 21 GB/s
SATA 2.5 GB/s 6 GB/s 12 GB/s
Raw Flash Data Bandwidth
1,4
11 GB/s 25 GB/s 50 GB/s
Max User Data Bandwidth
2,4
(10x compression & Flash)
110 GB/s 250 GB/s 500 GB/s
Disk IOPS
3,4
SAS 10,800 25,000 50,000
SATA 4,300 10,000 20,000
Flash IOPS
3,4
225,000 500,000 1,000,000
Data Load Rate
4
1 TB/hr 2.5 TB/hr 5 TB/hr
1 Bandwidth is peak physical disk scan bandwidth, assuming no compression.
2 - Max User Data Bandwidth assumes scanned data is compressed by factor of 10 and is on Flash.
3 IOPs Based on IO requests of size 8K
4 - Actual performance will vary by application.
Exadata Performance Scales
Exadata delivers brawny
hardware for use by Oracles
brainy software
Performance scales with size
Result
More business insight
Better decisions
Improved competitiveness
Table Scan Time
Table Size 1TB 10 TB 100TB
1 Hour
10 Hour
5 Hour
Typical
Warehouse
Exadata
Performance
Why is Oracle Faster?
DB Processing in Storage
Better Compression (10x)
Smart Flash Cache
Faster Interconnect (40Gb/sec)
More Disks
Faster Disks (15K RPM)
2009 Oracle Corporation
21
10
11.4
7.5
0
10
20
30
40
50
60
Query Throughput Query Throughput with Flash
50
HITACHI
USP V
TERADATA
2550
NETEZZA
TwinFin 12
SUN ORACLE
Database Machine
102
Exadata Licensing
Required Products
Oracle Database 11g Enterprise Edition
Oracle Exadata Storage Server Software
RAC
Partitioning Option
Other Recommended Software
Advanced Compression Option
Enterprise Manager Packs: Diagnostics, Provisioning, Tuning
OLAP Option
Data Mining Option
Advanced Security Option
Real Application Testing
Oracle Business Intelligence Enterprise Edition Plus
Exadata summary
Sun Oracle Database Machine Summary
Extreme Performance for all Data Management
Best for Data Warehousing
Smart scan of 10x compressed tables
Parallel query on in-memory data
Overall up to 5x faster than 11.1 for Warehousing
Best for OLTP
Only database that scales real-world applications on grid
Smart Flash Cache for 20x IOPs or 20x fewer disks
Smart Flash cache can hold entire working set
Up to 50x compression for archival data
Secure, fault tolerant
Best for Consolidation
Only database machine that runs and scales all workloads
Predictable response times in multi-database, multi-application, multi-user
environments
103
PresentersName
PresentersTitle
Data Warehousing Best Performance Practices
Module Agenda
OLTP vs. data warehouse
Hardware
Logical model
Data loading
Physical model
System management
104
OLTP vs. data warehouse
OLTP vs. Data Warehouse
Effect of I/O bottleneck
OLTP workloads DW workloads
105
Hardware
106
Disk
Array 1
Disk
Array 2
Disk
Array 3
Disk
Array 4
Disk
Array 5
Disk
Array 6
Disk
Array 7
Disk
Array 8
H
B
A
1
H
B
A
2
H
B
A
1
H
B
A
2
H
B
A
1
H
B
A
2
H
B
A
1
H
B
A
2
CPU Quantity and Speed dictate
number of HBAs
capacity of interconnect
HBA Quantity and Speed dictate
number of Disk Controllers
Controllers Quantity and Speed dictate
number of Disks
Disk Quantity and Speed
The weakest link defines the throughput
Hardware configuration - best practices
Build a balanced hardware configuration
Total throughput = # cores X 100-200 MB (depends on chip set)
Total HBA throughput = Total core throughput
If total core throughput =1.6GB will need 4 4Gb HBAs
Use 1 disk controller per HBA Port (throughput capacity must be equal)
Switch must be same capacity as HBA and disk controllers
Max of 10 physical disks per controller(Use smaller drives 146 or 300 GB)
Minimum of 4GB of Memory per core (8GB if using compression)
Interconnect bandwidth should equal IO bandwidth (Infiniband)
Use ASM with RAID 1 mirroring for redundancy
Create two ASM diskgroups (1 for data, 1 for flash recovery area)
Use ATTRIBUTE 'au_size to increase allocation unit
Logical models
Why schema modeling is important
Model according to your business needs
Don't get lost in academia
Ignore the physical DB, hardware or end-user tools that will
eventually be used
Two main data warehouse models
Third Normal Form (3NF)
Minimizes data redundancy through normalization
Typically has a large number of tables
Star Schema
Simplest model
One fact table surrounded by multiple dimension tables
Upon completion model should be
Easy to map to fact and dimension tables in physical database
Show clearly how information in operational systems will fit in data
warehouse
107
3
rd
Normal Form
Preserve a detailed record of each transaction without any data redundancy
Allows for rich encoding of attributes & all relationships between data
elements
Users typically require a solid understanding of the data in order to navigate
Star Schema
Dimensional approach that simplified the data model to facilitate access
Drill paths, hierarchy and query profile are embedded in the data model itself
rather than the data
Easier for inexperienced users to navigate
Forrester:
3
rd
Normal Form is the self-less model (Neutral)
Star Schema is the selfish model (Subject oriented)
Best practices
When to use each model
Power
Partitioning
Parallelism
Optimizing 3
rd
normal form
Power to handle I/O
Optimizing 3
rd
normal form
Power
Three advantages
Partition pruning
Efficient parallel joins
Efficient data loading
Optimizing 3
rd
normal form
Partitioning
108
Partitioning
Fact table or large tables
Range partition large fact tables typically on date column
Consider data loading frequency
Is an incremental load required?
How much data is involved, a day, a week, a month?
Partition pruning for queries
What range of data do the queries touch - a quarter, a year?
Subpartition by hash to improve join performance
between fact tables and / or dimension tables
Pick the common join column
If all dimensions have different join columns use join column for
the largest dimension or most common join in the queries
Number of hash partitions should be power of 2 (#CPU X 2)
Sales Table Sales Table
May May 22 22nd nd 2008 2008
May 23 May 23rd rd 2008 2008
May May 24 24th th 2008 2008
May 18 May 18th th 2008 2008
May May 19 19th th 2008 2008
May 20 May 20th th 2008 2008
May May 21 21st st 2008 2008
Select sum(sales_amount)
From SALES
Where sales_date between
to_date(05/20/2008,MM/DD/YYYY)
And
to_date(05/23/2008,MM/DD/YYYY);
Q: What was the total
sales for the weekend of
May 20
-
22 2008?
Only the 3
relevant
partitions are
accessed
Partition Pruning
Select sum(sales_amount)
From
SALES s, CUSTOMER c
Where s.cust_id = c.cust_id;
Both tables have the same
degree of parallelism and are
partitioned the same way on
the join column (cust_id)
Sales Sales
Range Range
partition May partition May
18 18th th 2008 2008
Sub part Sub part 2 2
Customer Customer
Range Range
partition May partition May
18 18th th 2008 2008
Sub part 4 Sub part 4
A large join is divided into
multiple smaller joins,
each joins a pair of
partitions in parallel
Partition Wise join
Execution plan for partition-wise join
ID ID Operation Operation Name Name Pstart Pstart Pstop Pstop TQ PQ Distrib
0 SELECT STATEMENT
1 PX COORDINATOR
2 PX SEND QC (RANDOM) :TQ10001 Q1,01 QC (RAND)
3 SORT GROUP BY Q1,01
4 PX RECEIVE Q1,01
5 PX SEND HASH :TQ10000 Q1,00 HASH
6 SORT GROUP BY Q1,00
7 PX PARTITION HASH ALL 1 128 Q1,00
8 HASH JOIN Q1,00
9 TABLE ACCESS FULL Customers 1 128 Q1,00
10 TABLE ACCESS FULL Sales 1 128 Q1,00
Partition Hash All above the join & Partition Hash All above the join &
single PQ set indicate partition single PQ set indicate partition- -wise join wise join
109
Parallel Execution should be used
Instead of one process doing all the work multiple processes
working concurrently on smaller units
Parallel degree should be power of 2
Goal is parallel partition wise joins
Optimizing 3
rd
normal form
Parallelism
Star Schema
What does it look like?
Called star schema because diagram resembles a star
The center of the star consists of one or more fact tables
The points of the star are the dimension tables
Select SUM(quanity_sold) total_umbrellas_sold_in_Boston
From Sales s, Customers c, Products p, Times t
Where s.cust_id = c.cust_id
And s.prod_id = p.prod_id
And s.time_id = t.time_id
And c.cust_city = BOSTON
And p.product = UMBRELLA
And t.month = MAY
And t.year = 2008;
Q: What was the total
number of umbrellas sold
in Boston during the month
of May 2008 ?
Star Query
Optimizing a Star schema
Create bitmap index on foreign key columns in fact table
Set STAR_TRANSFORMATION_ENABLED to TRUE
Goal is star transformation
Powerful optimization technique that rewrites or transform SQL
Executes the query in two phases
The first phase retrieves necessary rows (row set) from the fact table
Bitmap joins between bitmap indexes on all of the foreign key columns
The second phase joins this row set to the dimension tables
The join back to the dimension tables done using a hash join
110
Star Transformation in detail
Select SUM(quanity_sold)
From Sales s, Customers c, Products p, Times t
Where s.cust_id = c.cust_id
And s.prod_id = p.prod_id
And s.time_id = t.time_id
And c.cust_city = BOSTON
And p.product = UMBRELLA
And t.month = MAY
And t.year = 2008;
Select SUM(quanity_sold)
From Sales s
Where s.cust_id IN
(Select c.cust_id From Customers c Where c.cust_city = BOSTON)
And s.prod_id IN
(Select p.prod_id From Products p where p.product = UMBRELLA)
And s.time_id IN
(Select t.time_id From Times t Where t.month =MAY And t.year =2008);
Step 1: Oracle rewrites / transforms the
query to retrieve only the necessary rows from
the fact table using bitmap indexes on foreign
key columns
Step 2: Oracle joins
the rows from fact table
to the dimension tables
Phase 1 Phase 1
Phase Phase 2 2
Execution plan for Star Query
ID ID Operation Operation Name Name Rows Rows Pstart Pstart Pstop Pstop
0 SELECT STATEMENT 1
1 SORT GROUP BY NOSORT 1
2 HASH JOIN 3
3 TABLE ACCESS FULL PRODUCTS 2
4 HASH JOIN 1
5 TABLE ACCESS FULL TIMES 1
6 PARTITION RANGE SUBQUERY 44144 1 16
7 TABLE ACCESS BY LOCAL INDEX ROWID SALES 44144 1 16
8 BITMAP CONVERSION TOROWIDS
9 BITMAP AND
10 BITMAP MERGE
11 BITMAP KEY ITERATION
12 BUFFER SORT
13 TABLE ACCESS FULL TIMES 1
14 BITMAP INDEX RANGE SCAN SALES_TIME_BIX 1 16
15 BITMAP MERGE
17 BUFFER SORT
18 TABLE ACCESS FULL CUSTOMERS 1
19 BITMAP INDEX RANGE SCAN SALES_TIME_BIX 1 16
20 BITMAP MERGE
22 BUFFER SORT
23 TABLE ACCESS FULL PRODUCTS 2
24 BITMAP INDEX RANGE SCAN SALES_PROD_BIX 1 16
Physical models
Physical model
Blueprint of physical layers in a data warehouse
111
Physical model
What does this buy us?
The more data, the greater the
strategic value realized
The more data, the greater the
overhead of changes
Analysis needs change at the
speed of business
Separating the layers allows
greater aggregation of data while
maintaining flexibility for access
and performance
Data loading
External Tables
Allows flat file to be accessed via SQL or PL/SQL as if it was a table
Enables complex data transformations & data cleansing to occur on the fly
Avoids space wastage
Pre-processing
Ability to specify a program that the access driver will execute to read the
data
Specify gunzip to decompress a .gzip file on the fly while its being
Direct Path in parallel
Bypasses buffer cache and writes data directly to disk via multi-block async
IO
Use parallel to speed up load
Remember to use Alter session enable parallel DML
Range Partitioning
Enables partition exchange loads
Data loading
Best practices
How to load?
SQL Loader or External Tables
And the winner is => External Tables
Why:
Full usage of SQL capabilities directly on the data
Automatic use of parallel capabilities (just like a table)
No need to stage the data again
Better allocation of space when storing data
High watermark brokering
Autoallocate tablespace will trim extents after the load
112
Tips for External Tables
File locations and size
When using multiple files the file size should be similar
List largest to smallest in LOCATION clause if not similar in size
File Formats
Use a format allowing position-able and seek-able scans
Delimitate clearly and use well known record termination to allow for
automatic granulation
Always specify the character set if its different to the database
Consider compressing data files and uncompressing during
loading
Run all queries before the data load to populate column usage
for histogram creation during statistics gathering
May May 22 22nd nd
2008 2008
May 23 May 23rd rd
2008 2008
May 24 May 24th th
2008 2008
May 18 May 18th th
2008 2008
May 19 May 19th th
2008 2008
May 20 May 20th th
2008 2008
May 21 May 21st st
2008 2008
DBA
1. Create external table
for flat files
5. Alter table Sales
exchange partition
May_24_2008 with table
tmp_sales
2. Use CTAS command
to create non-
partitioned table
TMP_SALES
Tmp_ sales Tmp_ sales
Table Table
May 22 May 22nd nd
2008 2008
May 23 May 23rd rd
2008 2008
May 24 May 24th th
2008 2008
May May 18 18th th
2008 2008
May 19 May 19th th
2008 2008
May 20 May 20th th
2008 2008
May 21 May 21st st
2008 2008
Sales
table now
has all the
data
3. Create indexes
4. Gather Statistics
Tmp_ sales Tmp_ sales
Table Table
Partition Exchange loading
System management
Parallel Execution
Use common sense to apply parallelism only where it will help
performance and not hinder it
Resource Manager
If your data warehouse is CPU bound protects critical tasks from
interference from non-critical tasks
Always have accurate optimizer statistics
Use INCREMENTAL statistic maintenance or copy_stats to keep large
partitioned fact - table up to date
Set only the initialization parameters that you need to
Avoid tuning spiral of death by not tuning
System management
Keeping the lights on for your data warehouse
113
I have a poorly performing query
I will add new structures (indexes, materialized views, etc.) to
provide better performance
Ooops, now my ETL process takes 20% longer
I will drop some structures to speed up ETL
Repeat as necessary
System management
Tuning spiral of death
Parallel Execution
Use common sense to apply parallelism only where it will help
performance and not hinder it
Resource Manager
If your data warehouse is CPU bound protects critical tasks from
interference from non-critical tasks
Always have accurate Optimizer statistics
Use INCREMENTAL statistic maintenance or copy_stats to keep large
partitioned fact - table up to date
Set only the initialization parameters that you need to
Avoid tuning spiral of death by not tuning
Workload Monitoring
Take hourly AWR or statspack report
Use EM to do real-time system monitoring
New Parallel Execution and SQL Monitoring screen in 11g
System management
Keeping the lights on for your data warehouse
SQL Parallel Execution
Messages Messages QC connection QC connection Parallel server connection Parallel server connection
QC is the QC is the user session that initiates the parallel SQL statement user session that initiates the parallel SQL statement
& it will & it will distribute the work to parallel servers distribute the work to parallel servers
Parallel servers Parallel servers - - individual sessions that perform individual sessions that perform
work in parallel They are allocated from a work in parallel They are allocated from a pool of pool of
globally available parallel server processes and globally available parallel server processes and
assigned to a given operation assigned to a given operation
Parallel servers communicate among themselves & the Parallel servers communicate among themselves & the
QC using messages that are passed via memory buffers QC using messages that are passed via memory buffers
in the shared pool in the shared pool
Parallel Servers Parallel Servers
do majority of the work do majority of the work
Query Coordinator Query Coordinator
SQL Parallel Execution Plan
ID ID Operation Operation Name Name TQ TQ IN IN--OUT OUT PQ PQ
Distribution Distribution
0 SELECT STATEMENT
1 PX COORDINATOR
2 PX SEND QC {RANDOM} Q1,01 P->S
3 HASH JOIN Q1,01 PCWP
4 PX RECEIVE Q1,01 PCWP
5 PX SEND BROADCAST Q1,01 P->P BROADCAST
6 PX BLOCK ITERATOR Q1,01 PCWP
7 TABLE ACCESS FULL CUSTOMERS Q1,01 PCWP
8 PX BLOCK ITERATOR Q1,01 PCWP
9 TABLE ACCESS FULL SALES Q1,01 PCWP
SELECT c.cust_name, s.purchase_date, s.amount
114
Create two services
Srvctl add service d database_name
-s ETL
-r sid1, sid2
Srvctl add service d database_name
-s AHOC
-r sid3, sid4
ETL
Ad-Hoc queries
Note: Prior to 11g use init.ora parameters instance_groups and
parallel_instance_group to control PQ on RAC
Controlling PQ on RAC
Use services
Common sense with parallel execution
Parallel execution provides performance boost but requires more
resources
General rules of thumb for determining the appropriate DOP
objects smaller than 200 MB should not use any parallelism
objects between 200 MB and 5GB should use a DOP of 4
objects beyond 5GB use a DOP of 32
Settings may vary on your system- either in size range or DOP - and
highly depend on your target workload & hardware configuration
If your data warehouse is CPU bound, consider using
CPU Resource Management
Protects critical tasks from interference from non-critical tasks
Allows CPU to be utilized according to a specific ratio
Prevents thrashing and system instability that can occur with
excessive CPU loads
Managing your workload
Oracle Database Resource Manager
To use Resource Manager:
1. Define Consumer Groups for each type of workload
Priority DSS consumer group
Normal DSS consumer group
Maintenance consumer group
2. Create rules to dynamically map sessions to consumer groups,
based on session attributes
3. Define Resource Plan: policy for managing workloads
Priority DSS
Normal DSS
Maintenance
service = PRIORITY
Oracle username = LARRY
Oracle username = Maria
client program name = ETL
function = BACKUP
query has been running > 1 hour
Mapping Rules Consumer Groups
115
Resource Manager can manage Degree Of Parallelism (DOP)
Specify maximum DOP for each consumer group
OLTP consumer group: max DOP = 0 (serial queries only)
Low-Priority consumer group: max DOP = 4
Critical Reports consumer group: max DOP = unlimited
Actual DOP determined by
Hints and parallel execution parameters -> proposed DOP
MIN(proposed DOP, consumer groups max DOP) -> actual DOP
Efficiency Statistics Management
How do I gather accurate statistics?
.. Compute statistics gives accurate results but takes too long ..
.. Sampling is fast but not always accurate ..
.. AUTO SAMPLE SIZE does not always work with data skew ..
Gathering statistics on one partition (e.g. after a bulk load)
causes a full scan of all partitions to gather global table
statistics Extremely time and resource intensive
Use incremental statistics
Gather statistics for touched partition(s) ONLY
Table (global) statistics are built from partition statistics
New groundbreaking implementation for AUTO SAMPLE SIZE
Faster than sampling
Accuracy comparable to compute statistics
Incremental Global Statistics - 1
May 22 May 22nd nd 2008 2008
May 23 May 23rd rd 2008 2008
May 18 May 18th th 2008 2008
May 19 May 19th th 2008 2008
May 20 May 20th th 2008 2008
May 21 May 21st st 2008 2008
Sysaux Tablespace
1. Partition level stats are
gathered & synopsis
created
2. Global stats generated by
aggregating partition
synopsis
Incremental Global Statistics - 2
May 22 May 22nd nd 2008 2008
May 23 May 23rd rd 2008 2008
May 24 May 24th th 2008 2008
May 18 May 18th th 2008 2008
May 19 May 19th th 2008 2008
May 20 May 20th th 2008 2008
May 21 May 21st st 2008 2008
Sysaux Tablespace
3. A new partition
is added to the
table & Data is
Loaded
May 24 May 24th th 2008 2008
4. Gather partition
statistics for new
partition
4. Retrieve synopsis for
each of the other
partitions from Sysaux
5. Global stats generated by
aggregating the original
partition synopsis with the
new one
116
Db_block_size
16 K (Larger size may help with compression ratio)
Db_file_multiblock_read_count
1024/db_block_size
SGA_target & PGA_Aggregate_target
Shared_pool needs to be size to accommodate PQ message buffers
Parallel_min_servers
DOP X Avg # Concurrent Queries
Parallel_max_servers
DOP X Max # Concurrent Queries
Parallel_execution_message_size
16K
Parallel_adaptive_multi_user
False
Initialization parameters
Only set what you really need
Workload monitoring
Take hourly AWR or statspack snapshots
Establish baseline performance
Use V$ performance views for command-line real-time monitoring
V$_session, (G)V$_PQ, (G)V$_PX, (G)V$PX_PROCESS
New view (G)V$SQL_MONITOR
Enables real-time monitoring of long-running or parallel SQL statements
Or use Oracle Enterprise Manager Database Control 11g for
New Parallel Execution screens
Identifies all parallel execution activity
Displays how and where parallel resources are being used
SQL Monitoring screens
Visually identifies which parts of an execution plan are expensive relative to overall
cost of statement
Provides information about the parallel server sets & work distribution
SQL Monitoring Screens
Click on the
SQL ID for
more info
SQL Monitoring Screens - PWJ
Only one set
of parallel
servers
117
More Information
Accompanying White Paper
http://www.oracle.com/technology/products/bi/db/11g/pdf/twp_dw_best_practies_11g11_2008_09.pdf
Get Specialized!
118
Get Specialized!
Begin your path to Specialization today and become an Data Warehousing 11g Certified
Implementation Specialist!
The Data Warehouseng 11g Implementation Specialist Certification identifies professionals that
are skilled in implementing solutions based using the Oracle Database as a data warehouse.
The certification covers skills such as: data warehouse fundamentals, parallelism,
partitioning, the Result Set Cache, Advanced Compression, using Oracle Data Mining, data
integration with ODI and OWB, the Exadata Storage Server and the Sun Oracle Database
Machine and best practices for implementing data warehouses.
Take the Data Warehousing 11g Essentials 1Z0 515 exam today! Exam appointments are
currently open for scheduling at authorized Pearson Vue testing centers.
This new credential gives you the opportunity to gain the differentiation and competitive edge
you need to stand out in the marketplace. OPN Specialized partners are recognized by
Oracle, and preferred by customers.
Quick Links:
OPN Specialized program
OPN Certified Specialist FAQ

DW Implementation Boot Camp - Student Manual

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

DW Implementation Boot Camp - Student Manual

Uploaded by

Copyright:

Available Formats

1

Parallel Join with Broadcasting

You might also like