T5AP OBrien Data Modeling For Modern BI and Analytics

1
DATA MODELING
FOR MODERN BI
AND ANALYTICS
13th TDWI European Conference | 1719 June, 2013
John OBrien
@obrienjw @radiantadvisors
john.obrien@radiantadvisors.com
Copyright 2013 Radiant Advisors. All Rights Reserved
#modernBI
v2.10.000
Principal and Founder, Radiant Advisors
JOHN OBRIEN
@obrienjw @radiantadvisors
john.obrien@radiantadvisors.com
With over 25 years of experience delivering value through data warehousing and business
intelligence (BI) programs, John OBrien's unique perspective comes from the combination of
his roles as a practitioner, consultant, and vendor in the BI industry. His knowledge in
designing, building, and growing enterprise BI systems and teams brings real world insights to
each role and phase within a BI program.
Today, through Radiant Advisors John provides research and advisory services that guide
companies in meeting the demands of next generation information management, architecture,
and emerging technologies.
Instructor 10+ years
Experienced
Education
As a recognized thought leader in

BI, John has been publishing
articles and presenting at
conferences in North America
and Europe for the past 10 years.
John has also developed and
presented many of his own
courses that now comprise the
initial Radiant Advisors Learning
Catalog.
In 2005, John co-founded and

became CTO of data warehouse
appliance company, Dataupia, that
raised several rounds of venture
capital financing and has many global
production customers. As CTO,
Johns primary role was to focus
product development and BI market
strategy.
John has a B.S. in Mechanical

Engineering from California State
University with an emphasis in
control systems and
instrumentation and an Executive
M.B.A. from University of Colorado.
#modernBI
v2.10.000
Data Modeling for Modern BI and Analytics
OUTLINE
Section 1
Analysis and Data Modeling in BI
Section 2
Semantic Context and Data Virtualization
Section 3
Advanced BI Data Modeling Techniques
Section 4
Understanding Hadoop, Key NoSQL Data Stores
#modernBI
v2.10.000
The Need for Information and Business Intelligence
ANALYSIS AND
INFORMATION
MODELING
#modernBI
v2.10.000
Analysis and Information Modeling
INDUSTRY EVOLUTION
Value: Gaining Insight
Value: Directly Achieving Goals

Performance
Management
2005 2010
Business
Intelligence
2000 - 2005
Data
Warehousing
1980s -1990s
Value: Leveraging Data
Value: Integrating Data
#modernBI
Business
Analytics
2010+
v2.10.000
DECADES OF BI EVOLUTION
1970s-1980s Bill Inmons Building Strategic Data Warehouses
Subject Oriented and Business Model Oriented
Build it & they will come or One version of the truth
80% of data warehouse projects fail
1990s Ralph Kimballs Architected Data Marts

Business Scope Oriented with first Incremental Delivery
Delivered Business Value quickly and with less risk
2000s Shifts to BI/Performance Management & Real-Time

The Operational Data Store (ODS)
Monitoring with Dashboards and Scorecards
2010+ Focus on Data Science

Advanced Analytics, Data Mining, Big Data
Think Google, Facebook, Linked In Social / Behavioral data
#modernBI
v2.10.000
BI EVOLVED AS CUSTOMERS DID

The evolution of data warehousing followed the
evolution of how organizations and people
learned to leverage and consume information
Reporting
Analytics
Monitoring
Prediction
what happened?
why did that happen?
whats happening now?
whats going to happen?
#modernBI
v2.10.000
NEED FOR BUSINESS INTELLIGENCE

Achievement: BI is about informing the business
on how it is performing and achieving its goals
Knowledge: BI provides information for people
to make informed business decisions
Insight and Understanding: BI allows the
business to work with information in ways to
understand and explore
#modernBI
v2.10.000
BUSINESS ARCHITECTURES
Functionally Oriented
Business Oriented
Line of Business
CEO
Operations
Inventory
Product
Development
Line of Business
CEO
Finance
Sales
Line of Business
Customer
Support
Global Finance
Business fundamentally organized by function or business line

Goals and metrics are set about the company as a whole
Business definitions do not exist in functional systems
#modernBI
v2.10.000
10
BUSINESS SYSTEMS
Operational/Transactional Systems (OLTP)

Applications developed for functional organizations
Optimized for transaction processing
Current data with functional perspectives
Decision Support/Analytical Systems (OLAP)

Applications for analysis, understanding, insight
Optimized for query access and analytic functions
All historical and detail data needed for decisions
#modernBI
v2.10.000
11
FUNCTIONAL ISLANDS OF DATA

Order Mgmt
Marketing
Sales data
Customer data
Order data
Customer data
Product data
Customer data
Close the Sale
Process Orders
Opportunities
Prospecting
Customer rating
Customer budget
Customer Discounts Customer Patterns

Customer Address
Customer Products
Customer Since Date Customer Industry
Answers
Data
Sales
Perspective
Apps
Functional Systems create duplicate, inconsistent data
Billing
Order data
Customer data
Support
Order data
Customer data
Collect Revenue
Support/Retain
Customer Balance
Customer Last Pmt
Customer Bill Term
Customer Value
Customer Warranty
Customer Status
How many customers do we have?

What is the definition of customer?
Who are my most valuable customers?
What is my profit margin per customer by product for the past 5 years?
#modernBI
v2.10.000
12
BUSINESS ALIGNMENT
Align People
Align Goals
Align Information
Goals and Strategies of the company

Different organizations must have consistent metrics and definitions
#modernBI
v2.10.000
13
BI FEEDBACK LOOP
Quote
Support
Quoting
App
Order
Business
Processes
Customer
Care Sys
Ship
Ordering
System
Operational
Systems
Shipping
App
Invoice
Finance
Accounting
Business Processes
Executed
Operational Systems
Capture Business Events
Integrate
Cleanse
Acquire
Act
Business
Intelligence
Analyze
Transform to
Business Context
Actionable
Business Data
Improve Processes
Achieve Goals
Decide & Act
#modernBI
Evaluate & Analyze

Metrics and Goals
v2.10.000
Publish
14
Understanding How Businesses Work with Information
MODELING
#modernBI
v2.10.000
15
Modeling
MODELING GENERALIZED
What is modeling?
Understanding something
Analysis, Learning, Investigating, Testing Assumptions
Find a way to represent and communicate it

Diagrams, Documentation, Notation, Terminology
So what is data modeling?

#modernBI
v2.10.000
15
16
Modeling
BUSINESS MODELING
Business subjects and their relationships
Customer places Order
Order contains Product

Products in Inventory
Customer pays Finance
Employee build Product
Product requires Material
Insurance: Claims, Policies

Healthcare: Prescription
eCommerce: Clicks
Telecom: Calls, Net packets
Entertainment: Shows,
Commercials
Supplier ships Material

Finance pays Suppliers
#modernBI
v2.10.000
16
17
Modeling
SUBJECTS TO NORMALIZED
Product
Customer
Finance
Order
#modernBI
v2.10.000
17
18
Modeling
BUSINESS SUBJECT MODELING

Normalized Business Subjects
Customer, Invoice, Product, Finance

Dimensional Business Subjects
Sales organization and processes
Finance organization and processes
Customer Relationship Management (CRM)
Operations organization and processes
#modernBI
v2.10.000
18
19
Modeling
BILL INMON
a subject oriented, nonvolatile, integrated, time variant collection of
data in support of management's decisions.
Bill Inmon is known as the Father of Data

Warehousing for the first accepted definitions
He is still active today and has written over 50

books and 650 articles
His classic definition still holds true today
His approach is synonymous with Top Down

and Enterprise Approach
#modernBI
v2.10.000
20
Modeling
RALPH KIMBALL
A data warehouse is merely the union of all its constituent data
marts.
Created Dimensional Modeling

Author of The Data Warehouse Toolkit and Data Warehouse
Lifecycle and Still very active today and focuses on the
Kimball University
An original Data Warehouse Architect focused
Business understandability and usage

Quick deployments
Metrics driven approach
Synonymous with Data Marts and Business Focused Value

Synonymous with Bottom Up and Tactical although wrongly
so
#modernBI
v2.10.000
21
Modeling
PERCEPTIONS
Are you Kimball or Inmon?
Departmental Focused or Enterprise Driven?
Are you Tactical or Strategic?
Wrongful but common assumptions:

Inmon Architecture is slow, enterprise driven and
does not deliver immediate business value
Kimball Architecture is quick, tactical and creates
silos of inconsistent department data marts
#modernBI
v2.10.000
22
Modeling
INMONS DATA WAREHOUSE

Source
1
Source
2
Source
3
Source
4
Source
5
Data Warehouse
ODS
Staging
Layer
Normalized
Subject
Areas
Conformed
Dim &
Metrics
Operations
& Supply
#modernBI
Finance
v2.10.000
Order to
Cash
CRM
23
Modeling
HUB AND SPOKE (INMON)

Data Marts
ERP
database
Operational
database
Extract
data
Data
Warehouse
Distribute
data
Customer
Data Mart
Sales
Data Mart
CRM
database
Transform
Code /
Business
Rules/
Cleansing
Flat
Flat
File
File
Financial
Cube
Consistency across data marts enforced by data warehouse persistence

#modernBI
v2.10.000
23
24
Modeling
INMON DATA WAREHOUSE

Purpose:
Centralized hub of consistent enterprise data which all data marts are
dependent on
Subject Oriented
Business subject areas are business entities and relationships
(Customer, Order, Locations, Inventory, Product, Accounting, etc.)

Implies Normalized
3rd Normal Form
Business
Modeling
Business
subjects and
relationships
Subject areas
modeled for data
integrity
Integrated
Operational data is standardized into a single data model and made

consistent in meaning and codes
Data is typically atomic detail level and summarized as needed
Nonvolatile
Once loaded into the data warehouse, the data is not updated or
changed
Time Variant
Stores near current data (since the last acquisition process) and all
historical data and changes for analysis
#modernBI
v2.10.000
25
Modeling
TOP DOWN APPROACH

Enterprise Scope
Enterprise Modeling & Architecture

Incremental Development Planning
Data Warehouse Design &
Development
Department Scope
Data Mart Design & Development

Incremental Deployment
#modernBI
v2.10.000
26
Modeling
HUB AND SPOKE (INMON)

Pros
Cons
Data consistency ensured

through dependence
Focus on enterprise model

jeopardize business needs
Scalable architecture
Adds additional work to data mart

development from dependence
Supports many types of data mart

needs
#modernBI
v2.10.000
Extra work to design and code

ETL for DW
27
Modeling
NORMALIZED DATA MODELING

Entities:
Employee
Department
Skill
Expert In
Office
Attributes of Employee:
Employee ID (unique
Id)
First name
Last name
Home address
Relationships:
Employee manages zero, one or many Departments
Department is headed by one and only one Employee
Modeling Data to be stored based on Business Rules through the use of

Normalization Rules for Entities, Relationships and Attributes
#modernBI
v2.10.000
28
Modeling
BUS ARCHITECTURE (KIMBALL)

Dimensional
Data Warehouse
ETL Bus
ERP
database
Business
Rules
Customer
Data Mart
Load
Data
Operational
database
Extract
Data
Transform
Code
Conformed
Dimensions
Sales
Data Mart
CRM
database
Stage
Area
Flat
Flat
File
File
Based upon Ralph Kimball Architected Data Marts Approach

Financial
Cube
Consistency across data marts enforced by ETL Business in code

#modernBI
v2.10.000
29
Modeling
DIMENSIONAL MODELING
Answers Business Questions:
1. How much Sales were last
month by Sales Person and
Product Category?
2. Are Sales Quantities of
Product Category A
increasing each month for
the past year?
3. What Products do
Customers in City A buy
most of this month?
4. Who are our repeat buy
customers?
Modeling Data to be stored based on Business Defined Metrics or Facts

through the use of Dimensional Rules Facts and Dimensions
#modernBI
v2.10.000
30
Modeling
BOTTOM UP APPROACH
Enterprise Scope
Operations & Support

Data Mart Deployment
Data Mart Design & Development
Identify Business Area Scope
Department Scope
#modernBI
v2.10.000
31
Modeling
ARCHITECTED DATA MARTS (KIMBALL)

Pros
Cons
Quicker Deliveries
Departmental focus risk

enterprise consistency
Focused on Department
Information Needs
Definitions driven by department

users over enterprise
Dimensional Models easily

understood by business
Poor communications can lead

teams to build silos of data
Designed for analytics
#modernBI
v2.10.000
32
Modeling
DIMENSIONAL MODELING
DATE
TRADE
JOB
Facts related to
Qualifiers
Facts are numeric
Facts have standard
units
Every Fact has all
Qualifiers
Modeling Answers to
business questions
#modernBI
v2.10.000
32
33
Understanding How Businesses Work with Information
BI CAPABILITIES AND
DATA MODELING
#modernBI
v2.10.000
34
BI Capabilities and Data Modeling
ANALYTIC CULTURES GROW UP
#modernBI
v2.10.000
34
35
GENERATIONS OF CAPABILITIES
Next:
Opportunities
Manage Risk
Now:
Whats happening
Operational
Understand
Insight
Make Decisions
What happened:
Last Year
Last Month
Last Week
Yesterday
#modernBI
v2.10.000
36
CAPABILITY NEEDED BY USERS

Executives,
V.P.s
Scorecards
Performance
Dashboard
Analytics
Director
Manager
Reports
Director
Manager
Team
#modernBI
Analyst
Knowledge
Worker
Manager
Knowledge
Worker
v2.10.000
36
37
Informa@on
Consumers
Standard Reports
Knowledge Worker
Parameterized
Reporting
Business Analysts
Analytics
Managers &
Directors
Performance
Dashboards
Executives
Scorecards
Statisticians
Data Mining
Customers
Partners & Suppliers
External
Operational
Data Mart
BI Applica@ons
Internal
APPLICATIONS DELIVER CAPABILITIES
Understand how the user needs to work with information and experience level
#modernBI
v2.10.000
38
BI SERVICES AND CAPABILITIES
#modernBI
v2.10.000
38
39
FIVE TYPES OF BI CAPABILITIES

1. Static or Perfect Reports
2. Interactive Reporting
3. Analytics & Forecasting
4. Monitoring
5. Predicting and Data Mining
#modernBI
v2.10.000
40
1 STATIC REPORTING
What happened or What to do

The ability to produce reports for the business
Departmental or Integrated data reports
One-directional Reports pushed to user or location

Basic value of access to information
Pixel Perfect Reports, Forms, Filings
PDFs, images, documents, spreadsheets, files
#modernBI
v2.10.000
40
41
2 INTERACTIVE REPORTING
What I want to know
Customizing user sets parameters
Enhanced value to access, select, filter, sort
Introduces UI (User Interface) to allow control
Avoids many reports of same data
#modernBI
v2.10.000
41
42
3 ANALYTICS AND FORECASTING

Answers: Why did that happen?
Understanding, Planning, Comparing
Analysis is mostly a behavior
High interactivity (1-2 second results, think Google)

Drilling down to details (summary data to details)
Breaking down data sets quickly
Typically Metrics driven with Qualifiers
Sales by month, by region, by product, by person
#modernBI
v2.10.000
42
43
4 MONITORING
Whats happening now? or How am I doing?

Operational oriented: for intra-day refreshing
Performance oriented: Metrics/Goals tracking
Observe and quickly see what needs attention
May invoke analysis or change
Tracking so issues are dealt with before too late

#modernBI
v2.10.000
43
44
5 PREDICTIVE ANALYTICS
What is going to happen? or Likely to happen?
I dont know what question to ask the data
Data asks the person Is this relevant?
Algorithms analyze data for hidden patterns, clusters

and dependencies
Models built to answer Can you predict this?
Statisticians and small specialized teams
#modernBI
v2.10.000
44
45
MULTIDIMENSIONAL
ANALYSIS
Dimensional Modeling Paradigm
#modernBI
v2.10.000
46
TOPICS
Data Models for Dimensional
Process for Requirements Gathering
Data Modeling for Dimensional Analysis
Physical Modeling for Dimensional Analysis
#modernBI
v2.10.000
47
DIMENSIONAL
MODELING
#modernBI
v2.10.000
48
Dimensional Modeling
ANSWERING BUSINESS QUESTIONS

BI requirements are typically reports and
information needs
BI customer has a business goal to achieve and
needs information to make decisions
Approach: Model the Information Needs
Ensure each business question is fully answered

Facts and metrics are business rules for integration
Analysts translate questions into dimensional models
#modernBI
v2.10.000
48
49
FACTS AND QUALIFIERS

What are company sales for 1st quarter by sales
person, by product?
Fact: Sales Quantity, Sales Amount
Qualifier: Quarter, Person, Product

#modernBI
v2.10.000
49
50
METERS AND REFINEMENTS

Business Ques@ons List

1.
2.
3.
4.
5.
6.
7.
8.
9.
.
.
.
.
.
.
100.
Meter: Protability
Fact
Fact
Fact
Fact
Fact
Facts
Meter: Revenue
Fact
Fact
Fact
Fact
Fact
#modernBI
v2.10.000
50
51
LOGICAL MODELING
#modernBI
v2.10.000
51
52
ADDITIVE FACTS
Additive
Useful and meaningful when summarized along any set of
dimensions
Semi-Additive
Useful to summarize using some dimensions but not all
dimensions
Non-Additive
Impractical to sum along any set of dimensions
Example: Fact Employee Count can be summarized across departments, over different
job trades but not time for quarter
#modernBI
v2.10.000
52
53
FACTLESS FACT TABLES
#modernBI
v2.10.000
53
54
PHYSICAL STAR
SCHEMAS
#modernBI
v2.10.000
55
Physical Star Schemas
STAR SCHEMA
#modernBI
v2.10.000
55
56
ROLAP FOREIGN KEYS

employee
time
location
location_code
location_name
location_address
location_city_name
location_state_abbr
location_zip_code
location_key
labor organization
union_id_number
union_name
union_group_code
trade_code
trade_SOC_code
trade_name
contract_start_date
contract_end_date
labor_org_key
date_yyyymm
fiscal_yyyymm
fiscal_yyyyqq
time_key
employee satisfaction
job_change_count
employment_length_months
complaint_count
resignation_count
termination_count
promotion_count
demotion_count
disciplinary_action_count
satisfaction_score
time_key
emp_key
location_key
employment_org_key
labor_org_key
emp_age
emp_gender
#modernBI
emp_id_number
emp_age
emp_gender
emp_name
emp_hire_date
emp_term_date
emp_status_code
emp_term_reason
emp_key
employment
organization
dept_number
dept_name
dept_abbr
job_id_number
job_title
job_shift_code
job_shift_name
emplmt_org_key
v2.10.000
56
57
SEMANTIC STAR SCHEMA

Semantic Layer
Advanced topic
Users interact
with
Stars Schemas
BI Tool Layer:
Universes
Catalogs
Stars mapped
to Tables
Database Layer:
Views
Data stored in
Normalized
Tables
Physical Database
Structures
#modernBI
v2.10.000
57
58
SNOWFLAKE SCHEMA
#modernBI
v2.10.000
58
59
SLOWLY CHANGING DIMENSIONS (SCD)

A fact is a measure of interest typically
associated with a point in the time dimension.
What happens when attributes within the other dimensions change or

get updated later?

Type 1: update the dimension attribute (lose history)
Type 2: insert new dimension record (time stamped)
Type 3: have two columns (current attribute, previous attr)
#modernBI
v2.10.000
59
60
ROLAP VS. MOLAP AND HOLAP

OLAP On-Line Analytical Processing (Dimensional)
R OLAP in a Relational database
MOLAP in a Multi-dimensional database (Cube)
HOLAP Hybrid combination

Summarized data in cubes for speed
Detail data in relational db for scalability
Needs ability to drill from cube to RDBMS
#modernBI
v2.10.000
60
61
Delivery Architecture for Adaptability and Speed
MULTI-DIMENSIONAL
OLAP (CUBES)
#modernBI
v2.10.000
61
62
Multi-Dimensional Cubes
OVERVIEW
The Multi-Dimensional Database (MDDB) is a database designed for

dimensional structures with an optimized index and storage
structures
The Service Capability for Analytics can be defined by its user
behavior:
Interactive
Understanding
Discovery
Pre-defined or framed business questions
Pre-Calculation is a Pay Now or Pay Later

#modernBI
v2.10.000
63
45+ YEARS
Some of the highlights:
1970 First OLAP product Express introduced by Information Resources

1982 Comshare System W launched first OLAP for financials
1984 Metaphor launched first ROLAP
1990 Cognos PowerPlay launched
1992 Essbase launched ("Extended Spread Sheet dataBASE)
1993 Codd white paper coined the OLAP term
1994 MicroStrategy DSS Agent launched (multi-pass SQL)
1995 Oracle acquires Express
1996 Business Objects 4.0 launches DOLAP
1998 IBM DB2 OLAP Server released
1998 Hyperion Solutions formed from Arbor and Hyperion merger
1999 Microsoft OLAP Services shipped (Analysis Services in 2000)
2000 XML for Analysis announced

#modernBI
v2.10.000
64
OLAP HISTORY CONTINUED

2002 Oracle ships integrated OLAP server (integration of Express)
(Not usable until 2006, 11 years later after Express purchase)
2003 Consolidation starts
Business Objects buys Crystal Decisions

Hyperion Solutions buys Brio Software
Cognos buys Adaytum
GEAC buys Comshare
2004 Excel add-ins go mainstream

2005 Pentaho buys Mondrian (Open source BI environment)
2006 Palo launched
---------------------------------------------------------------------------------------------------
2007 Oracle buys Hyperion (just when it delivers 11g embedded OLAP)
2008 IBM buys Cognos
2008 SAP buys Business Objects
#modernBI
v2.10.000
65
Products
MOLAP CUBES
Metric: Sales Quantity
Clothing
50
66
Hats
Sweaters
20
20
Jackets
25
40
U.S.
West
Hawaii
Alaska
Nov Nov Dec Dec

30 Total 1 2
#modernBI
66 clothing items
sold in Alaska
on December 2nd
40 items of jackets
sold in Alaska
on December 2nd
Time
v2.10.000
65
66
ADVANTAGES & DISADVANTAGES

Advantages
Disadvantages
Pre-Calculations
Latency
All SUM/GROUP BY values are

calculated as part of the LOAD
process
Loading cubes are CPU and

memory intensive
Scalability
Hierarchies
Ability to define multiple hierarchies
on a dimension
matrix storage of summary data for

all dimensions and deep hierarchies
quickly explodes
Sparsity
Significantly less data is actual, but
every cell assigned a value
#modernBI
v2.10.000
67
SPARSITY SOLUTIONS
Cube Farming or Cube Chunking
Horizontal Partitioning
Vertical Partitioning
Hybrid OLAP
#modernBI
v2.10.000
68
CUBE FARMING
LOB 1
LOB 2
Q1
Asia
Europe
North America
Companies will start to isolate data by user group
(This is why Base Facts & Dimensions are needed)

LOB Quarterly cubes for 4 years
Divisional annual cubes at monthly level detail
#modernBI
v2.10.000
69
HORIZONTAL PARTITIONING
Horizontal typically manages data policies;

Level of Detail with Length of History
Detail for 30 days
Weekly for 1 year
Monthly for 25 months

Quarterly for 4 years
May lose drill down capabilities

Must know how to delete 31st day, 53rd week, 26th month, 17th
Quarter
#modernBI
v2.10.000
70
VERTICAL PARTITIONING
OEM Business
Retail Business
Asia
Europe
North America
Vertical typically manages data policies

Independent Lines of Business
North America Business Unit and President
Global Retail Business Operations
May lose Enterprise summary data

Individual cubes may not be able to join due to fact/dim
capability
#modernBI
v2.10.000
71
CHUNKING CUBES
Chunk cubes by data distributions rather than users

Will need the ability to union all chunks as one
#modernBI
v2.10.000
72
Delivery Architecture for Adaptability and Speed
OLAP ARCHITECTURE
#modernBI
v2.10.000
72
73
OLAP Architecture and Technologies

Business Activity Data
Information
Data Flow
Source
DM
Source
DM
Source
Stage
EDL
DM
Source
DM
Source
DM
Extract Engine
(Little to no code)
Transformation
The Real Work
#modernBI
v2.10.000
Load Engine
(Select, Filter, Target)
Consistent Enterprise Information
OLAP ARCHITECTURE GOAL
74
BASIC DW ARCHITECTURE
#modernBI
v2.10.000
75
DIMENSIONAL DATA ARCHITECTURE

Atomic level / Detail data
Enterprise Conformity
Conformed dimensions
Conformed facts
Departmental Specialization
Localized and sub-dimensions
Local facts and derivations
Tailored Delivery
Subsets for user groups; Summaries, Subsets, Materialized

Views
#modernBI
v2.10.000
76
DIMENSIONAL ARCHITECTURE
#modernBI
v2.10.000
77
TRANS FACTS/ATOMIC LEVEL

Considered to be Data Warehouse layer
Lowest level of detail in facts and dimensions
Can be very high data volume
Contains full history data
Considered to be Base layer of a DW
Typically will not be accessed by end user queries
#modernBI
v2.10.000
78
ENTERPRISE DIMENSIONAL LAYER

Conformed Dimensions:
Dimensions used across fact tables
Can be derived from Enterprise Subject
Area
Enterprise Facts/Metrics:
Facts used across multiple data marts /
departments
Strategic Enterprise Metrics
#modernBI
v2.10.000
79
DEPT DIMENSIONAL LAYER

Localized for Departmental Use
Both Enterprise Facts and Localize Facts
Conformed Dimensions and Localized Dimensions
Star, Snowflake, Denormalized as needed
Maybe combination of OLAP Technologies
#modernBI
v2.10.000
80
ENABLING TECHNOLOGIES
#modernBI
v2.10.000
81
Blending Data Virtualization for Agile Integration and Access
SEMANTIC
VIRTUALIZATION
#modernBI
v2.10.000
82
Semantic Virtualization
Unstructured
Structured
WHERE DOES CONTEXT LIVE?
More Rigid
#modernBI
v2.10.000
More Agile
83
PERSISTING SEMANTICS IN MODELS

Structured
Context leveraged
BI Tools
Direct access
Unstructured
Context in structures
More Rigid
#modernBI
v2.10.000
More Agile
84
PERSISTING SEMANTICS VIRTUALLY

Structured
Context leveraged
Context(s) leveraged
BI Tools
Direct access
Context in abstraction
Unstructured
More Rigid
#modernBI
v2.10.000
More Agile
85
Centralized
Context in
abstraction
PIG
Individual
Context
with Data
Scientists
HIVE
Unstructured
Structured
PERSISTING SEMANTICS IN CODE
MapReduce
Hadoop HDFS
More Rigid
#modernBI
v2.10.000
More Agile
86
Individual
Context
with Data
Scientists
Centralized
Context in
abstraction
MapReduce
Hadoop HDFS
DB
HCatalog
MapReduce
Hadoop HDFS
More Rigid
PIG
Context in
Data
Scientists
Hive
PIG
Centralized
Context in
abstraction
HIVE
Unstructured
Structured
PERSISTING SEMANTICS VIRTUALLY
#modernBI
v2.10.000
More Agile
87
MANAGING ALL SEMANTIC OPTIONS

Structured
Context leveraged
BI Tools
Direct access
Individual
Context
with Data
Scientists
Centralized
Context in
abstraction
MapReduce
Hadoop HDFS
M/R
HCatalog
MapReduce
Hadoop HDFS
More Rigid
PIG
Context in
Data
Scientists
Hive
PIG
Centralized
Context in
abstraction
Hive
Unstructured
#modernBI
v2.10.000
More Agile
88
MANAGING ALL SEMANTIC OPTIONS

Structured
Context leveraged
BI Tools
Direct access
Individual
Context
with Data
Scientists
Centralized
Context in
abstraction
MapReduce
Hadoop HDFS
M/R
HCatalog
MapReduce
Hadoop HDFS
More Rigid
PIG
Context in
Data
Scientists
Hive
PIG
Centralized
Context in
abstraction
Hive
Unstructured
#modernBI
v2.10.000
More Agile
89
UNLOCKING UNSTRUCTURED VALUE

More
Analysts
Value
PIG
MapReduce
Hadoop HDFS
Power Users
Users Involved
Hive
Yesterday
constrained
Very Few
Data Scientists
#modernBI
v2.10.000
90
UNLOCKING UNSTRUCTURED VALUE

More
Analysts
Many Many
Consumers
Very Few
Data Scientists
HCatalog
MapReduce
Hadoop HDFS
#modernBI
v2.10.000
PIG
Power Users
Hive
DB
Tomorrow
Unleashed
BI Tool
Value
Analysts &
Casual Users
Users Involved
91
DISCOVERY IN BI PROCESSES
Many More Analysts
Iterate
More
Analysts/Modelers
M/R
HCatalog
Very Few
Data Scientists
Hadoop HDFS
2.
Defined Context
Available to
Structured Database
#modernBI
v2.10.000
BI
Tool
Verify
BI
Tool
Discover
Context
PIG
Migrate
Test
Hive
Discover
Many Many Consumers
Few
Analysts/
Modelers
92
REVIEW
Understand context in processes and architectures
Realize that value is unlocked with more users
Discovery is a powerful BI process to operationalize
Modern BI Architectures are integrating Hadoop
#modernBI
v2.10.000
93
Blending Data Virtualization for Agile Integration and Access
ROLE OF DATA
VIRTUALIZATION
#modernBI
v2.10.000
94
The Role of Data Virtualization
KEY TAKE AWAYS

Data Virtualization
What and Why - data virtualization
Important aspects for Data Integration Framework

Important aspects for Modern Data Platforms
#modernBI
v2.10.000
95
VIRTUALIZAITON DEFINED
Virtualization refers to technologies that provide a layer of
abstraction in between technology stacks and allows for a single
logical view rather than physical view.
RS
BO
COG
HYP
REP
REP
AS
DB Layer
SS
ORA
ORA
ORA
PG
ORA
SS
OS Layer
MS
LNX
SOL
MS
LNX
MAC
MS
Server Layer
x86
x86
x86
x86
x86
x86
X86
BI Layer
Disks Layer
#modernBI
v2.10.000
96
TOP 3 DRIVERS FOR VIRTUALIZATION

1. Managing and tracking all data assets in a centralized yet
distributed approach
2. Allowing client applications to interact with disparate databases
on the back end in a re-usable, non-hard coded manner. This
also provides flexibility in managing change in back end
systems.
3. Allowing companies corporate merger and acquisition (M&A)
activities to function as a new single business entity
#modernBI
v2.10.000
97
BENEFITS
Similar to the benefits of Centralization

Agility in working with data
Abstraction in separating app and data development
Closer to real-time without latency of data transport
Easier to manage many assets
Business Agility - combining computing resources into

reusable pools of resources are better positioned to
respond to changing demands the business places of
those resources.
#modernBI
v2.10.000
98
DATABASE LAYER VIRTUALIZATION

Data Virtualization and BI technologies:
Allows for a single virtual database that holds the meta data related
to all underlying heterogeneous databases and tables.
user
user
user
user
BO
COG
HYP
MS
Virtual Database

VIEW VIEW VIEW VIEW VIEW VIEW VIEW

Databases
ORA
MS
#modernBI
DB2
PG
v2.10.000
XML
LOG
FILE
99
BENEFITS FROM ABSTRACTION

A data abstraction layer allows you to pull the curtains
on databases from the outside world.
This allows you to:
Work on data clean up and migrations
Consolidate databases
Decommission databases
Reorganize data subject areas
By managing the Virtual Layers metadata, developers and

consumers no longer worry about where the data comes from
simply as long as their fetch interface is available.
#modernBI
v2.10.000
100
DATA LEGACIES
There are so many data stores in the company and no time or

resources to consolidate or decommission them properly.
Heterogeneous database technologies make it difficult to view
information. This typically leads to extracts and transformations
loaded into data warehouses for reporting. This also has latency.
A Federated Data Architecture approach is being favored in many
organizations today.
#modernBI
v2.10.000
101
DATA INTEGRATION REQUIREMENTS

Applications need to be synchronized (EAI)
Databases need to be synchronized (EAI)
Query data that spans several data sources (DV)
Analyze historical data (ETL)
Query a current snap view of data (DV/ODS)
Data transformation on an as-needed basis (DV)
Interactive data used in analysis tools (ETL)
#modernBI
v2.10.000
102
BLENDING DATA INTEGRATION

In order to manage data architecture so that integrity, flexibility, and
performance are maintained, a solution of integrated technologies
such as ETL, EAI, and DV would be used.
These three technologies, when used in a complimentary fashion
along with a strong foundation of data modeling techniques and data
management, lead to an optimized overall data architecture.
#modernBI
v2.10.000
103
COMPLIMENTARY SOLUTIONS
ETL:
Focused on data movement
Data warehouse is another set of data existing in operational systems
Operates normally in batch mode
EAI:
Focused on moving data with corresponding business logic between systems
Guaranteed delivery
Transactional in nature
Data Virtualization:
Virtualizes all data in a single view
Persists no data, meta model driven
Most current to real time
#modernBI
v2.10.000
104
DATA MOVEMENT
Client
App
EAI
ERP
Extract
Client
Client
EAI
App
Extract
EII Virtualized Database

Environment
Extract
ETL
Load
DW
Only DV does not move data, except when returning result sets
#modernBI
v2.10.000
105
KEY TAKE AWAYS
Benefits of Data Virtualization
Complimentary and blended with ETL and EAI
Serves as data abstraction in many scenarios

Has value in being virtual but also trade-offs
#modernBI
v2.10.000
106
DV ARCHITECTURE FOR ACCESS

Applications
Data Services
Query Tools
SOAP Interface
Virtual DB
Cust
Oracle
JDBC
Transform
SQL
Server
#modernBI
Other Access
Top
Cust
DB2
v2.10.000
ODBC
Virtual
Tables
Order
Sybase
Logs
XML
Text
107
DISTRIBUTED QUERY
The original intent of data virtualization tools was
to solve the query across multiple database
scenario
Today these tools can query across:
Multiple database vendors
Multiple data formats
Incorporate legacy APIs
Data on multiple system platforms
#modernBI
v2.10.000
108
PERFORMANCE
The biggest differentiators between DV vendors will be their ability

to optimize the incoming requests and their ability to scale.
There should be minimal overhead to a parse and build optimize
plan (a couple hundred milliseconds).
Memory tends to be the bottleneck for DV servers since they
perform all of their operations in memory after fetching data.
Depending on your DV tool, time will be spent on defining
metadata and hand tuning SQL.
#modernBI
v2.10.000
109
CROSS DB REFERENTIAL INTEGRITY

Another important aspect of DV environments is their ability to
enforce cross-database referential integrity constraints.
If a single virtualized table is represented as a joined pair of tables
in two databases, an insert would be prevented or rolled back if
the referential integrity constraint was not met.
For existing cross-database referential integrity, outer joins reports
could surface data which is not foreign key related in the other
system.
#modernBI
v2.10.000
110
CACHING
Another challenge that some DV vendors have been asked to solve

is that of Caching results.
If retrieving a customers information is needed several times
during a long running business process, will the DV environment
cache the results?
There is tremendous benefit if a long executing query running
several times can be avoided.
Be sure to ask your vendor, and test

#modernBI
v2.10.000
111
CONFIGURATION MANAGEMENT
Strong configuration management best practices will
have to be followed for storing versions
All virtual tables
All transformation
Virtual relationships
Mapping to databases
Change management will need to be utilized to ensure

that changes take place in a well organized and
communicated fashion.
#modernBI
v2.10.000
112
JOB ROLE
DV is meta driven, a role and responsibility will exist:

Data analyst
Data modeler
Data steward
This person will have the responsibility to map strategic

data domains and models to existing data elements.
Maintain all metadata and data services to developers
and consumers.
#modernBI
v2.10.000
113
KEY TAKE AWAYS
Data Virtualization Differentiators
Access, Update and Caching
Key factors for complete product

New roles and processes to manage
#modernBI
v2.10.000
114
TIME VARIANT
DV serves as a view into multiple systems

If the tables that are virtualized are operational, the DV virtual
table represents current data
If the tables that are virtualized contain history from data
warehouses, the DV virtual table represents historical data
Remember that Data Virtualization is a data abstraction layer
A common misconception is that DV is for operational or ODS
applications.
#modernBI
v2.10.000
115
TRANSFORMATIONS
DV meta descriptions are not just limited to joins and

where clauses between tables.
A very powerful aspect of DV tools is their ability to join/retrieve
data across disparate databases, then perform a transformation
on the result set, then perhaps join to another table.
This transformation logic can also be developed using standard
components in the DV too or through the use of embedded Java
code for complex algorithms.
#modernBI
v2.10.000
116
TRANSFORMATIONS
Virtual
Table
All describe in
Virtual Table
Metadata
Join
#modernBI
Agg
Java
Code
Result
v2.10.000
SOAP/JDBC Interface
Join
Virtual
Table
Client
SQL
117
TRANSFORMATION ARCHITECTURE
Data Virtualization Server / Virtual Database
Client
SOAP/
JDBC
Interface
Joined
Table
Joined
Table
Table
Table
Agg
Table
Table
Table
JDBC
ODBC
RDBMS
RDBMS
#modernBI
v2.10.000
Joined
Table
Table
Table
Legacy
API
Text
All tables
represented
as 1 for 1
118
CONFORMED DIMENSIONS PATTERN

Could DV be used to solve or enhance the problem of
independent data marts and lack of conformed
dimensions?
1. Virtualize every dimension table into the DV environment
2. Analyze and determine the proper conformed dimension rules
and data
3. Implement the transformation logic into the DV environment
4. Virtualize all fact tables into the DV environment
5. Query facts joined to dimensions in DV
#modernBI
v2.10.000
119
OPERATIONAL DATA STORES?

Current integrated view of systems and part of the CIF
ODS workload still exists on operational systems
Better option over data replication
Real-real time or near-real time
ODS challenge of tight synchronization
Requires extract, transform, then load
DV will parse, transform, query
#modernBI
v2.10.000
120
KEY TAKE AWAYS
Data Virtualization and DI Patterns
Time Variance is captured in persistence
Common DI patterns can be virtualized

Virtualization challenges some architecture and
replication approaches like the ODS
#modernBI
v2.10.000
121
Next Generation Analytic Databases in Modern BI Architectures
DATA MODELING AND

ANALYTIC DATABASES
#modernBI
v2.10.000
121
122
Data Modeling and Analytic Databases
ANALYTIC TECHNOLOGY CATEGORIES
#modernBI
v2.10.000
123
Next Generation Analytic Databases in Modern BI Architectures
COLUMNAR SQL
DATABASES
#modernBI
v2.10.000
124
Columnar SQL Databases
COLUMNAR DATABASES
Transactional OLTP databases are built around row management
which is what OLTPs need
Analytics works with data sets and more specifically selecting, filter
and grouping by columns
Therefore, Columnar databases that store data on disk in a columnar
orientation will perform much better than reading rows of data to find
columns of interest
#modernBI
v2.10.000
125
AVOID DATA BLOCK MOVEMENTS
#modernBI
v2.10.000
126
Columnar Databases
COLUMNAR ORIENTATION
1
John
OBrien
CO
50000
Betty
Smith
CA
55000
Sue
Hughes
TX
60000
Tom
Jones
NV
65000
David
Saunders
CO
70000
Row Serialization: (Written on disk)
1,John,OBrien,CO,5000,2,Betty,Smith,CA,55000,3,Sue,Hughes,TX,60000,4,Tom,Jones,NV,65000
Columnar Serialization: (Written on disk)
1,2,3,4,5,John,Betty,Sue,Tom,David,50000,55000,60000,65000,70000
SELECT State, SUM(Sales) FROM Table1 GROUP BY state ORDER BY 2;
#modernBI
v2.10.000
127
ROW VS. COLUMN ORIENTATION

Reading Rows of Data for Columns
#modernBI
Reading only Columns you need
v2.10.000
128
New Databases Enable New Forms of Valuable Analytics
NOSQL DATABASES
FOR ANALYTICS
#modernBI
v2.10.000
129
NoSQL Landscape
CONCEPTS AND DEFINITIONS

Not Only SQL a system with a non-SQL component
(XML databases, graph stores, object stores, in-memory caches)
Inspirations:
2004 November Google MapReduce Paper
2006 November Google BigTable Paper
2006 W3C XQuery 1.0 specification
2007 October Amazon Dynamo Paper
2008 Yahoo/Apache Hadoop
NoSQL is an accidental neologism
No generally accepted definition or authority
#modernBI
v2.10.000
130
NoSQL Landscape
CONCEPTS AND DEFINITIONS

Typically associated with open-source, distributed,
non-relational databases
Running on clusters of servers
Schema-less: add fields to database records without
having to define any changes in structure first
Not using relational model - impedance mismatch of
object-to-relational apps
#modernBI
v2.10.000
131
NoSQL Landscape
BENEFITS OF NOSQL SYSTEMS
Application development much faster
Deals better with changes and extensibility
Fault tolerant redundancy and low failure rates
Scales easily and as needed across servers
Does not require data modeling expertise or phase
#modernBI
v2.10.000
132
NoSQL Landscape
LOOKING AHEAD
Relational Databases are not going away
Not ACID like RDBMS, options for consistency and
distribution
not only enables ecosystem of storage options so you
must understand your data better
Polyglot Persistence is using different data stores in different
circumstances
Its more of movement than a technology

#modernBI
v2.10.000
133
Hadoop and Big Data are Transforming BI
UNDERSTANDING
NOSQL COLUMNAR
#modernBI
v2.10.000
134
NoSQL Columnar Databases
DATABASES AVAILABLE
Cassandra
Hbase
HyperTable
Amazon DynamoDB
#modernBI
v2.10.000
135
GOOD FOR
Horizontally Scalable and good for Big Data

Typically built in support for Compression
Compression maximized by column data type
Suitable Use Cases:
Event Logging
Content Management Systems, Blogging Platforms
Counters
Expiring Usage expiring columns delete rows automatically
Cassandra is highly available with no master node
#modernBI
v2.10.000
136
NOT SO GOOD FOR
Design schema ahead of time based on query plan

Not good for ad-hoc query situations
Hbase does not scale down (5 node minimum)
Not Suitable For:
ACID transactions for writes and reads
Cannot Sum or Avg must pull back all rows to client
#modernBI
v2.10.000
137
UNDERSTANDING
NOSQL GRAPHS
#modernBI
v2.10.000
138
NoSQL Graph Databases
DEFINITIONS AND CONCEPTS

For relationship analysis
Stores Entities and Relationships
Entities are Nodes which have properties
Node is an instance of an object
Relationships are Edges which have properties

Have directional significance
Nodes are organized by Relationships to find patterns

between nodes
Queries are traversing the graph
#modernBI
v2.10.000
139

Typeless and Schemaless
Supports 34.4 billion nodes and 34.4 billion relationships
(Thats 42 nodes for Facebooks 800 million users in a
single graph)
Provides Lucene for fast lookups
Language extension Gremlin and REST interface
Enterprise edition provides highly available and high readtraffic sites by Neo4J HA
#modernBI
v2.10.000
140

Relationships with properties
BigCo
f
yee_o
Emplo hitect
Arc
Role= =Feb 04
Hired
Anna
Carol
Barbara
friend
Since=2005
Elizabeth
friend
Since=1989
Share=[books,movies,tweets]
#modernBI
v2.10.000
Dawn
141
DATABASES AVAILABLE
Neo4J
Infinite Graph
OrientDB
FlockDB
#modernBI
v2.10.000
142
UNDERSTANDING
NOSQL KEY VALUES
#modernBI
v2.10.000
143
Key-Value Stores
KEY-VALUE STORES
Simple data structure of two attributes: a key and value
Simple programming interface:
Store key-value pair
Retrieve value given a key
Simplicity leads to scalability

Great for storing binary large objects (BLOBs)
#modernBI
v2.10.000
144
Key-Value Stores
KEY-VALUE STORES
Simplest of NoSQL data stores
Key-value is a simple hash table
Used when all access to database is via
primary key
#modernBI
v2.10.000
145
Key-Value Stores
TERMINOLOGY
Oracle
Riak
Database instance
Riak cluster
Table
Bucket
Row
Key-value
Rowid
Key
#modernBI
v2.10.000
146
Key-Value Stores
TERMINOLOGY
<Bucket = userData>
<key = sessionID>
<Value = Object>
UserProfile
SessionData
ShoppingCart
CartItem
CartItem
#modernBI
v2.10.000
147
Key-Value Stores
DATABASES AVAILABLE
Riak
Redis (referred to as Data Structure server)
Memcached DB
Berkeley DB
HamsterDB (for embedded use)
Amazon DynamoDB (not open-source)
Project Valdemort (open-source version of Amazon
DynamoDB)
#modernBI
v2.10.000
148
Key-Value Stores
GOOD FOR
Little to no need to maintain indexes

Horizontally scalable
Extremely fast
Suited for non-related data like web user session data
and unrelated to other uses
Use Cases:
Storing Session Information all session data in single put/get
User Profiles, Preferences single get via user key
Shopping Care Data
#modernBI
v2.10.000
149
Key-Value Stores
NOT SO GOOD FOR

Lacking indexes and scanning capabilities therefore not
able to perform queries
Basic CRUD (Create, Read, Update, Delete) functions
Use Cases:
Relationships Among Data
Multi-operation Transactions saving multiple keys and
failure occurs
Query by Data
Query by Sets Operation limited to one key at a time
#modernBI
v2.10.000
150
Key-Value Stores
Column-oriented
Column family
Very similar to key-value databases
Values are queried by matching keys
Values are groups of zero, one or more columns
Each row is capable of populating however many it wants
Easy to add columns
Hbase adds versioning to key-values
#modernBI
v2.10.000
151
Key-Value Stores
HBASE
Deployed part of Hadoop ecosystem

HBase does not scale down (5 node minimum)
No sorting or indexing capability aside from row keys
No datatypes array of bytes (app must determine)
Great built-in versioning
Built for scalability
Compression
Garbage collection
In-memory tables
#modernBI
v2.10.000
152
Key-Value Stores
TERMINOLOGY
RDBMS
Cassandra
database instance
cluster
database
keyspace
table
column family
row
row
column (same for all rows)
column (different per row)
#modernBI
v2.10.000
153
Key-Value Stores
TERMINOLOGY
Column family
ROW
Row
KeyX
Column1
Name1:value1
Column2
Name2:value2
ColumnN
NameN:valueN
ROW
Row
KeyY
Column1
Name1:value1
#modernBI
Column9
Name9:value9
v2.10.000
ColumnN
NameN:valueN
154
UNDERSTANDING
HADOOP KEY VALUES
#modernBI
v2.10.000
155
Hadoop, MapReduce and BI
BIG DATA CHALLENGE

Big Data requires large amounts of servers and storage
Scaling costs
Must leverage commodity hardware
More Cores & Spindles (Cheaper & Faster)
Density computers per rack (Power & Cooling)
Design software to handle hardware failures
#modernBI
v2.10.000
156
SCALE OUT ARCHITECTURE

MPP Scale-out:
Hadoop Scale-out:
Shared Nothing architecture
Nodes communicate with each

other to complete Map and
Reduce functions
Separate architecture for

Distributed File System for
loaders and Name node meta
data
Each node has database engine
Data distribute by column with

value, hash or round-robin
Separate architecture for

execution and accessing data
Data access in Master node
Each data node communicates

with other nodes
Database includes SQL access
Separate layers for SQL

translation (PIG, Hive)
#modernBI
v2.10.000
157
BUSINESS BENEFITS FROM NOSQL
New systems can be built much faster
Advanced data modeling and entity-relationship

modeling does not hold up development
Highly scalable without redesign
Affordable and low failure rates
#modernBI
v2.10.000
158
Modern Data Platforms in BI
The Apache Hadoop software library is a framework

that allows for the distributed processing of large
data sets across clusters of computers using a
simple programming model.
It is designed to scale up from single servers to

thousands of machines, each offering local
computation and storage.
Rather than rely on hardware to deliver highavailability, the library is designed to detect and
handle failures at the application layer, delivering a
highly-availability service on top of a cluster of
computers, each of which may be prone to failures.
Hadoop Sub Projects
Hadoop Related Projects
Hadoop Common (Core)

Hadoop Distributed File System (HDFS)
HCatalog
Hive
Cassandra
Mahout
Hadoop MapReduce
Pig
Chukwa
HBase
ZooKeeper
#modernBI
v2.10.000
Big Data Boot Camp NYC May 2013
WHAT IS HADOOP?
159
MOVING CODE, NOT DATA

Accessing data
with MapReduce Programs
Data stored in
Hadoop Distributed File System
#modernBI
v2.10.000
160
SCHEMA-LESS
Web Logs / App Logs

Data which is best stored without a fixed schema (table design)
since the data keeps changing
Applications which are constantly changing or use self-describing
data formats such as XML
Event Data
Data which contains standard or fixed data structures which allows
it to be stored easily in RDBMS
However, high volume Event Data presents the challenges of
loading, managing, and accessing this data
#modernBI
v2.10.000
161
ARCHITECTURE COMPONENTS
Master / Slave Architecture
NameNode
DataNode
Secondary NameNode
JobTracker
TaskTracker
#modernBI
v2.10.000
162
TaskTracker
JobTracker
DataNode
TaskTracker
#modernBI
DataNode
TaskTracker
v2.10.000
DataNode
TaskTracker
Backup
Secondary NameNode
Slave
Slave
DataNode
Slave
Client
NameNode
Slave
Master
ARCHITECTURE TOPOLOGY
163
UNDERSTANDING MAPREDUCE
MapReduce is a framework for processing huge datasets
on certain kinds of distributable problems using a large
number of computers (nodes)
Collectively referred to as a cluster (if all nodes use the
same hardware) or as a grid (if the nodes use different
hardware)
Computational processing can occur on data stored either
in a file system (unstructured) or within a database
(structured)
It takes some time to get used to
#modernBI
v2.10.000
164
MAPREDUCE DATA FLOW
#modernBI
v2.10.000
165
MAP STEP
The master node takes the input, partitions it up into

smaller sub-problems, and distributes those to worker
nodes
A worker node may do this again in turn, leading to a
multi-level tree structure
The worker node processes that smaller problem and
passes the answer back to its master node
#modernBI
v2.10.000
166
REDUCE STEP
The master node then takes the answers to all the

sub-problems and combines them in some way to get
the output the answer to the problem it was originally
trying to solve
#modernBI
v2.10.000
167
HADOOP KEY/VALUE PAIRS

Fundamental data unit of Hadoop
Instead of relational tables
Flexible to work with less structured data types
#modernBI
v2.10.000
168
HADOOP DATA TYPES

You can create your own custom type as long as it implements
the Writable (or WritableComparable<T>) interface
Reference: Hadoop in Action, Hadoop Data Types

#modernBI
v2.10.000
169
PIG
Simplifies Hadoop Programming with a high-level data

processing language
Two Major Components:
1. Pig Latin high level data processing language
2. A Compiler compiles to an evaluation mechanism, such as
Hadoop
Knowing how to write UDFs is a big part of learning to use Pig
Pig can operate on data that is relational, nested,

semi-structured or unstructured
The Pig interactive shell is known as Grunt
Yahoo has >40% Pig programs

#modernBI
v2.10.000
170
HIVE
A SQL like data warehouse infrastructure with HiveQL

Hive is a data warehousing package built on top of Hadoop
Features:
Overview:
Separates the user from the

complexity of MapReduce
Separates the user from the

complexity of MapReduce
Target user is the data analyst

more comfortable with SQL for
ad-hoc queries, summarization
and data analysis
Structured uses familiar

concepts of tables, rows,
columns and schema
Partitions data with use of

directory structures
Connectivity with JDBC, Web

GUI or Command Line Interface
Hive is a metastore for storing

schema information
Hive began at Facebook processing user and log data

#modernBI
v2.10.000
171
HBASE
A distributed, column-oriented database using HDFS

Overview:
Targeted for random real-time read/write access of (fairly)
structured data
Designed to support large tables of billions of rows and
millions of columns
HDFS delivers fully distributed and highly available
Hbase is modeled after Googles BigTable

#modernBI
v2.10.000
172
HADOOP ECOSYSTEM
#modernBI
v2.10.000
173
HORTON WORKS MOTIVATION

Quite simply, these three things:
1. Architecting the Future of Big Data We believe Apache
Hadoop is the ideal platform for storing, managing and
analyzing big data. We have already invested a lot of blood,
sweat and tears in Apache Hadoop and we take tremendous
pride in helping to define its future.
2. Solving Real-World Business Challenges While at Yahoo!,
the core Hortonworks development team gained extensive
experience applying Apache Hadoop technology to deliver real
business value. We want to share this knowledge and
experience with other organizations, large and small.
3. Open Source We believe that open source licensing via
the Apache Software Foundation is the best way to solve the
big data challenge. Open source allows the largest possible
user community to try Apache Hadoop for free, to validate that
our code does what we say it does and to know that our work
will have longevity beyond that of any one company or
contributor.
#modernBI
v2.10.000
174
CLOUDERA & MAPR
Cloudera Distribution
Including Apache Hadoop
Packages CDH ecosystem

(Cloudera Distributed Hadoop)
100% Apache-licensed
components
Subscription offering
Simplified installation and update
Tested and compatible version

Contributes to Apache Hadoop
MapR Technologies Inc.

Distribution
Think of Red Hat for Linux

#modernBI
v2.10.000
May 2011- announced the

availability of an alternate file
system for Hadoop
Replaced the HDFS file system
with a full random-access read/
write file system
Advanced features like snapshots
and mirrors
Gets rid of the single point of
failure issue of the default HDFS
NameNode.
175
Hadoop and Big Data are Coming Together to Transform BI
MODERN DATA
PLATFORM IN BI
#modernBI
v2.10.000
176
Modern Data Platform in BI
POLYGLOT PERSISTENCE
Different databases are designed to solve
different problems
Disparate Data Storage Needs
Hybrid Approach to Persistence
Similar to todays RDBMS and MOLAP
combination
#modernBI
v2.10.000
177
Modern Data Platform in BI
POLYGLOT PERSISTENCE
In 2006, Neal Ford coined the term
polyglot programming
Idea that applications should be written in a mix
of languages to take advantage of being suitable
for tackling different problems
#modernBI
v2.10.000
178
Polyglot Persistence
TYPICAL ARCHITECTURE
E-commerce
platform
Shopping cart
data
Shopping cart
data
Completed
Orders
RDBMS
store
#modernBI
v2.10.000
BI/DW
179
POLYGLOT ARCHITECTURE
Key-Value store
E-commerce
platform
Relational store
Shopping cart
data
Key-Value
store
Completed
Orders
RDBMS
store
Offload session and shopping car data storage

#modernBI
v2.10.000
Session data
Key-Value
store
180
POLYGLOT ARCHITECTURE
E-commerce
platform
Key-Value store
Document store
Shopping cart
and session
data
Graph store
Relational store
Inventory and
Item price
Customer
social graph
Completed
orders
Key-Value
store
#modernBI
Document
store
v2.10.000
RDBMS
store
Graph
store
181
EXPANDING SERVICE USAGE

E-commerce
platform
Shopping cart
and session
data
Inventory and
Item price
Completed
orders
Inventory and
Price service
Session Storage
service
Key-Value
store
Order persistence
service
RDBMS
store
Document
store
#modernBI
v2.10.000
Customer
social graph
Friends bought
these products
service
Graph
store
182
ROLE OF ANALYTIC DATABASES

Data persistence for optimized BI workloads
2-tier versus 3-tier debate
Why 3-tier will be next generation
Integrating semantics in or above data

Cross database versus data virtualization debate
Why a evolving combination will be next generation
#modernBI
v2.10.000
183
MIXED WORKLOAD CAPABILITIES

3-Tier BI Architecture
Key Value Store (Hadoop)
Discovery Oriented
Analytic
Database
Technologies
Highest Scalability
Lowest Cost
Schema-less
Without Context
Accessibility:
Programming
Workload:
Flexible, Scalable Analytic Optimized
Reference Data Mgmt
Maturity:
Emerging
Mature
#modernBI
SQL, MDX, UDF
EDW
RDBMS
Accepted
v2.10.000
SQL Access
184
3 TIER ARCHITECTURE EXTENDED

Tier 1 - Flexible
Tier 2 - Optimized
Cluster
Tier 3 - Reference
HCatalog /
Hive-QL
MapReduce
Highly Optimized for BI
MPP
Columnar
Integration
Links
Gateways
askdjfl
kasjdfl
iuyuiio
Document
Stores
Discovery, Scalable,
Programmable
Projections
MOLAP
Highly Specialized for BI
HCatalog /
Hive-QL
MapReduce
Cluster
In-memory
Graphs
Analytics Oriented
#modernBI
v2.10.000
Data
Warehouses
Master
Reference
Data
Text
Analytics
Stable, Context, SQL
185
HADOOP STAGING AREA PATTERN

Hadoop:
Massive Scalability
Lowest Cost
Handles Complexity
Data Warehouse:
Optimized Work Loads
Operational
Benefit from Context
Internet,
Sensor
data
Operational Systems
Insulate Change or Direct to

Staging
Migrate History
or ETL Acquire
Staging
Very Few
Data
Scientists
MapReduce
ETL
or ETL
PIG
Hadoop HDFS
Hive
HCatalog
ETL
Data Marts
Data Marts
Data Marts
Persisted or Virtual
Many Many Consumers

#modernBI
v2.10.000
Few
Analysts/
Modelers
186
SEMANTIC INTEGRATION - SQL

BI tools
(today)
HCatalog / Hive-QL
MapReduce
SQL
Integration
Semantic Projections
Links
Gateways
text
Analytic DBMS
Hadoop
#modernBI
v2.10.000
Columnar storage
In-memory access
Document stores
Text Analysis
Graph Analysis
ROLAP/MOLAP
EDW
(RDBMS)
187
SEMANTIC INTEGRATION ABOVE DATA

Where should
semantic knowledge
live in the
architecture?
Future BI tools
HCatalog
Services
SQL / Data Virtualization
MapReduce
text
In-memory
Semantic
Discovery
Analytic DBMS
Hadoop
#modernBI
v2.10.000
EDW
188
SEMANTIC INTEGRATION - SERVICES

Future BI tools
Services
HCatalog
SQL / Data Virtualization
MapReduce
text
In-memory
Analytic DBMS
Hadoop
#modernBI
v2.10.000
EDW
189
SUMMARY
Polyglot persistence is about using different data storage

technologies to handle varying storage needs
Can apply across an enterprise or within a single application
Encapsulating data access into services reduces the impact
of data storage choices on other parts of a system
Adding more data storage technologies increases
complexity in programming and operation
Advantages must outweigh complexity
#modernBI
v2.10.000
190
NoSQL Landscape
GREAT RESOURCES
NoSQL Distilled, A brief guide to the Emerging World on
Polyglot Persistence. Pramod J. Sadalage, Martin Fowler,
Addison-Wesley, copyright 2013
Seven Databases in Seven Weeks, A guide to Modern
Database and the NoSQL Movement. Eric Redmond, Jim R.
Wilson, The Pragmatic Bookshelf, copyright 2012
#modernBI
v2.10.000
191
Rediscovering BI
Todays BI environment is all about rethinking how we do BI and
imagining new, innovative ways to approach BI.
Rediscovering BI is a free, monthly eMagazine that challenges readers
to rethink, reexamine, and rediscover the way they approach business
intelligence.
We publish pieces that provide thought leadership, foster innovation,
challenge the status quo, and inspire you to rediscover BI.
www.RadiantAdvisors.com/RediscoveringBI
#rediscoveringBI
#modernBI
v2.10.000
192
Subscribe
#modernBI
v2.10.000
193
For more information

www.RadiantAdvisors.com
Twitter:
@RadiantAdvisors #ModernBI #RediscoveringBI
RSS:
feed://radiantadvisors.com/feed/
Email us at:
info@RadiantAdvisors.com
Linked IN:
www.linkedin.com/company/radiant-advisors
Subscribe:
Rediscovering BI monthly newsletter

www.radiantadvisors.com.rediscoveringbi
THANK YOU!
#modernBI
v2.10.000

T5AP OBrien Data Modeling For Modern BI and Analytics

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

T5AP OBrien Data Modeling For Modern BI and Analytics

Uploaded by

Copyright:

Available Formats

1

Copyright 2013 Radiant Advisors. All Rights Reserved

Principal and Founder, Radiant Advisors

Instructor 10+ years

As a recognized thought leader in

In 2005, John co-founded and

John has a B.S. in Mechanical

Copyright 2013 Radiant Advisors. All Rights Reserved

Data Modeling for Modern BI and Analytics

Analysis and Data Modeling in BI

Semantic Context and Data Virtualization

Advanced BI Data Modeling Techniques

Understanding Hadoop, Key NoSQL Data Stores

Copyright 2013 Radiant Advisors. All Rights Reserved

The Need for Information and Business Intelligence

Analysis and Information Modeling

Value: Gaining Insight

Value: Directly Achieving Goals

Copyright 2013 Radiant Advisors. All Rights Reserved

Value: Leveraging Data

Value: Integrating Data

Analysis and Information Modeling

1990s Ralph Kimballs Architected Data Marts

2000s Shifts to BI/Performance Management & Real-Time

2010+ Focus on Data Science

Copyright 2013 Radiant Advisors. All Rights Reserved

Analysis and Information Modeling

BI EVOLVED AS CUSTOMERS DID

Analysis and Information Modeling

NEED FOR BUSINESS INTELLIGENCE

Analysis and Information Modeling

Business fundamentally organized by function or business line

Copyright 2013 Radiant Advisors. All Rights Reserved

Analysis and Information Modeling

Operational/Transactional Systems (OLTP)

Decision Support/Analytical Systems (OLAP)

Copyright 2013 Radiant Advisors. All Rights Reserved

Analysis and Information Modeling

FUNCTIONAL ISLANDS OF DATA

Close the Sale

Customer Discounts Customer Patterns

Functional Systems create duplicate, inconsistent data

How many customers do we have?

Copyright 2013 Radiant Advisors. All Rights Reserved

Analysis and Information Modeling

Goals and Strategies of the company

Copyright 2013 Radiant Advisors. All Rights Reserved

Analysis and Information Modeling

Evaluate & Analyze

Understanding How Businesses Work with Information

Find a way to represent and communicate it

So what is data modeling?

Order contains Product

Insurance: Claims, Policies

Supplier ships Material

Copyright 2013 Radiant Advisors. All Rights Reserved

Copyright 2013 Radiant Advisors. All Rights Reserved

BUSINESS SUBJECT MODELING

Copyright 2013 Radiant Advisors. All Rights Reserved

Bill Inmon is known as the Father of Data

He is still active today and has written over 50

His approach is synonymous with Top Down

Copyright 2013 Radiant Advisors. All Rights Reserved