Oow2012 Oeid For Obiee Devs

Endeca Information Discovery for BI/DW Devs
Mark Rittman, Technical Director, Rittman Mead

Oracle Openworld 2012, San Francisco, October 2012
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Saturday, 29 September 12
Mark Rittman
Mark Rittman, Co-Founder of Rittman Mead

Oracle ACE Director, specialising in Oracle BI&DW
14 Years Experience with Oracle Technology
Regular columnist for Oracle Magazine
Author of forthcoming Oracle Press book on OBIEE 11g
Writer for Rittman Mead Blog :
http://www.rittmanmead.com/blog
Email : mark.rittman@rittmanmead.com
Twitter : @markrittman
Who Are Endeca? Why Did Oracle Acquire Them?
Recent (2012) acquisition by Oracle, aimed at strengthening ERP and E-Commerce offerings
Endeca was best known for search and guided navigation features for e-commerce sites
HMV.com
Autotrader.com
Sony e-commerce sites
etc
Key feature is searching across arbitrary
selections of attributes, fast response,
supports discovery and exploration
Faceted Search
Enabled through a hybrid search/analytic
database called the Endeca Server
Endeca Faceted
Search feature
What is Oracle Endeca Information Discovery?
As well as their search/guided navigation product, Endeca also had a BI platform

Originally called Endeca Latitude, now rebranded Oracle Endeca Information Discovery (EID)
A BI platform built around the same Endeca Server, and focused on search/discovery of data
Complementary to OBIEE and Essbase
Separately licensed to BI Foundation,
long-term plans for tech platform integration
Information Discovery vs. Reporting & Analysis
Data volume, variety and growth presents issues in finding answers to business questions
Unstructured data, social network data, call centre logs as well as regular structured data sets
Datasets change, dont always fit dimensional models, and arrive quickly
Users want self-service access to data with minimal setup time
Reporting and Analysis is great for accurate answers to known questions ...
... Data discovery provides fast answers to new questions
Guiding principle : Quickly explore all relevant data
Quickly Explore All Relevant Data
Relationships Advanced search Structured

undefined or Contextual Semi-structured
unknown navigation Unstructured
No up-front data Analytics Even messy data is OK
modelling Not in the data
Rapid, iterative warehouse, yet
change
Search-First Interface
Traditional BI tools (OBIEE, Discoverer etc)

are focused on reporting and delivering Value Search
Oracle EID takes a search first approach, across all
attributes
building on e-commerce and web search
user experiences
Search + Contextual Navigation Breadcrumb list
+ Visual Analysis showing all filters
applied so far
Split-second response times to

support and encourage data exploration
Guided
Navigation,
free-form filtering
across all
atttributes
Data Discovery Scenarios
Product Quality Reports / Warranty Analysis

Type of Customer Traditional BI Issues Data Discovery Solution
Most useful information is in free- Easy linking of disparate sources
Manufacturers, retailers form documents and reports that only share limited commonality
Consumer bodies Success comes from correlating All data considered, with ability to
Insurance companies information from many sources parse and detect meaning in docs
Focus on reporting and numbers Unlimited exporing across all attribs.
Online TV Shopping Sales Sentiment Analysis

Online and TV-based retailers Sales reporting only covers what Combine unstructured social
youve sold. not why youve sold it networking feeds with sales data
E-commerce operations Consumer sentiment is found on Content acquisition from non-
B2C companies with vocal, blogs, Facebook and Twitter, not traditional sources
online customer base easily brought into BI datasets Analyze consumer sentiment
Police Crime Report & Evidence Analysis

By the time youve fitted it to a Minimal up-front data modeling
Police forces, CID etc conformed dimensional model, No requirement for a common model
Intelligence agencies (MI5 etc) youve missed the opportunity Support for unstructured, semi-
Private investigators, journalists Evidence spread across disparate structured and structured sources
documents, sources and formats Search is the primary user interface
Oracle Endeca Information Discovery Platform Overview
A suite of tools to capture, load and then analyze all types of data
Built around the Oracle Endeca Server,
a hybrid search/analytic database Oracle Endeca Information
Discovery Studio
Full ETL capabiities plus web-based
dashboard and data exploration tools
Additional tools can be licenced for
Oracle Endeca Server
content acquisition and text enrichment /
sentiment analysis
Oracle Endeca Information

Discovery Integrator
Documents, PDFs Call centre

Spreadsheets RDBMS Data Master Data Social Media
and other unstructured transcripts
and budget data and Web feeds
sources
Oracle Endeca Server : A Hybrid Search/Analytic Database
Key to these capabilities is the Oracle Endeca Server and its datastores (databases)
Proprietary database engine focused on search and analytics
Data organized as records, made up of attributes stored as key/value pairs
No over-arching schema,
no tables, self-describing attributes
Every record can have its own unique
set of attributes, with the overall data model
emerging over time as data is loaded
Endeca Server hallmarks:
Minimal upfront design
Support for jagged data
Administered via web service calls
No data left behind
Load and Go
Inside an Endeca Server Datastore
Attributes unique to the sales data mart / warehouse Shared attributes common to all data sources
e.g. Salesperson, Customer, Channel, Response Type eg. Time, City, Brand, Key Account
Sales transaction data Attributes unique to just this data source,

from a data mart or data warehouse for example Customer Segment, Category, Socio Class
External data from market

research or customer surveys /
demographics
Jagged data, where each row may

Forecast data
use different sets of attributes e.g. customer
from Essbase
demographic and behavioral attributes
Supporting data
and comments from
social media, e.g.
Facebook, Twitter
Endeca Server Search and Document Analysis Capabilities
Search and unstructured data support adds powerful document analysis capabilities
Keyword search, boolean search, parametric search, wildcard search, dimension search and
dimension filters
Dimension precedence rules
Numeric range, geospatial, date/time and security filters
Spell correction/suggestion,
and do you mean-type alternative presentation
Find similar, and 1 and 2-way synonyms
Stemming and lemmatisation
Keyword-in-context snippeting
Results clustering, relevance ranking, sorting and paging
Multi-language support
Oracle EID Integrator and Studio
Data is loaded into Oracle Endeca Server datastores using Oracle EID Integrator
Data Integration (ETL) tool build on open-source CloverETL tool (Eclipse framework)
Oracle EID functionality provided through components that call Endeca Server web services
User Interface created and delivered using Oracle EID Studio, 100% web-based
Create dashboards made up of search, navigation and data analysis components
Also provides Endeca Server / Studio admin features
Endeca Information Discovery Platform & Software Requirements
64-bit application, benefits from multiple CPUs/cores and lots of RAM

Oracle Information Discovery 2.3 available for Windows and Linux platforms
Windows Server 2008 SP1 64-bit
Oracle Linux / RHEL 5 64-bit
Downloadable from Oracle Software Delivery Cloud (Edelivery) (full or trial license),
OTN later on (distribution restrictions due to embedded third-party components)
Endeca Server licensed per-CPU, Oracle EID named-user plus (per-CPU maybe later)
Oracle EID and Oracle BI Foundation
Complementary technology to OBIEE and Essbase

Extends BI capabilities to unstructured and semi-structured data
Provides an alternative search and contextual navigation UI for users
Oracle Endeca Server enables load and go data analysis with minimal up-front modeling
Query, reporting Interactive dashboard, ad-hoc queries,

drilling enabled by a ROLAP engine
OBIEE
and analysis
BI Multi-dimensional
Foundation Multi-dimensional engine, what-if? Essbase
OLAP analysis, scenario modeling, planning
Suite
Production High-volume, pixel-perfect layout, BI Publisher

Reporting printing
Endeca
Information Data Discovery Search and contextual navigation for Endeca Information
structured and unstructured data Discovery
Discovery
BI Foundation Suite + Endeca Information Discovery = Complete BI Solution
An OEID 2.3 Development Scenario
Data to be sourced from structured and semi-structured sources

Sales data supplemented by customer survey responses
Loaded into the search/analysis-focused Endeca Server
Analyzed using a search/guided navigation-led dashboard
Based on the Quickstart demo/dataset provided with Oracle EID 2.3
Oracle Oracle Oracle

Endeca Endeca Endeca
Information Server Information
Discovery Discovery
Structured, Integrator Studio
Semi-Structured
& Unstructured
Data Sources
End-User Analytic/
Search Application
High-Level Oracle EID Development Process (Simplified)
1. Create the Endeca Server datastore (if required)

2. Use Integrator to create the initial set of records, assign a spec (unique ID) and review
3. Add additional data sources, including unstructured and semi-structured
4. Configure datastore attributes, place into groups and provide user-friendly names
5. Configure search interfaces as required, to enable record search
6. Create the Studio user interface, prototype and repeat steps 3-6 as required
7. Deploy out to user community
The Oracle EID Integrator Development Environment
Buttons to
run graphs, run
in debug mode
etc
Project elements
(graphs, in/out
data, params etc)
Additional graph
components
Graph, containing a (readers, writers,
a single component joiners + OEID
(Web Service call to components)
create a new Endeca
Server datastore)
Properties,
metadata etc for
selected object
Integrator Graphs, Components and Edges
Data mappings in Integrator are called graphs

Graphs contain components that are joined by edges that connect to component ports
Edges are assigned metadata, which define the colums and expressions used to move data
Graphs are then then run and execute row-based, pipelining data through the process
Graph
Port
Input Filter Sort
Join Load
Edge
Input
Component
Information Discovery-Specific Components in Integrator
Integrator comes with a number of specific to Oracle EID, for example:

Create a new datastore, or reset existing ones
Incrementally load datastore records, or bulk-load them
Add attributes via key/value pairs
Tag and enrich text
Import and export datastore configurations
All except bulk-load call Endeca Server web service APIs
Bulk-load uses a separate, direct-load interface OEID-Specific
Components
Creating a New Endeca Server Datastore
The Endeca Server runs as a server process, with datastores being created, then attached
New datastores are either created using the endeca-cmd Java command-line utility
cd c:\Oracle\Endeca\Server\2.3.0\endeca-cmd
endeca-cmd create-ds bikestore
Or through a graph component that calls the same Endeca Server web service API
Uses a configuration file to obtain
server host name, datastore name, port etc
for the Endeca Server being called
Endeca Server web

service API call to
create a datastore, using
datastore name in
parameter file
Loading the Initial Record Set
The first stage in creating the dataset is to load the initial set of records
In this example, loaded from a relational fact table export, contains measures and keys
Records require a spec(ification), a unique key that uniquely identifies each record
In the example below, the spec is created using a transformation component that concatenates
the SalesOrderID and SalesOrderLineID
Transform (Reformat)
Reader component, Writer component,
component,
reading fact table writing records to the
concatenating input
rows (records) from Endeca Server data
columns to create
a flat file export store
record spec (PK)
Transformations and Expressions
Data flowing through graphs can be transformed at various stages, for example
As it flows through an edge
During a join
Expression syntax is CTL1/CTL2
Clover Transformation Language
Similar to Java expressions, source
and target-technology neutral
Original CTL1 version now being
superceded by CTL2
Expression used to
create the spec
column,
concatenation two
input columns
Viewing the Initial Record Set
The initial record load can then be viewed in Studio

One record per row of input data, at this point mostly measures and dimension keys
Each input column becomes an attribute, with values stored internally in key/value pairs
Later on, we will enhance these records by
Adding additional sources,
for more useful attribute values
Configuring the attribute datatypes
and names, grouping them and
enabling them for search
An example
record, showing
values for loaded
attributes
Initial set of
record attributes
Loading Additional Structured and Semi-Structured Sources
Additional sources can be loaded using the same graph, joining on common IDs
Unstructured/semi-structured sources with at least a matching ID can be brought in
Continue adding sources until full set of required data is loaded into the datastore
Joiner, joining the

sales fact and
employee data on a
common
EmployeeKey
column
Joiner, joining the

combined sales +
employee data with
survey data, on
common
SalesOrderNumber
Text Parsing, Whitelisting, Enrichment and Sentiment Analysis
Text attributes can be analyzed, enriched and parsed using standard and additional features
Build in features include text tagging component
and various CTL functions
Oracle Endeca Text Enrichment module
Text analysis capabilities for extraction
of people, pleaces, themes etc
Automatic summarized abstracts etc
Uses Salience Engine from Lexalytics, Inc.
Additional cost option
Oracle Endeca Text Enrichment with Whitelist of terms to
be tagged using the
Sentiment Analysis module Text Tagger - Whitelist
component
As above, but with ability to calculate
aggregate sentiment scores Whitelist text
Determine positive or negative sentiment tagger, using the
whitelist of terms in
in comments, surveys, feedback the
SurveyWhiteList file
Demonstrations
Oracle EID Integrator
Creating the Studio Interface
Once you have some data loaded, you can start to create the Studio web-based interface
Create the initial Studio pages early, use as part of the prototyping process
Over time, add more visualizations and data views to aid discovery
Once the model and attribute set are fairly stable, you can perform additional configuration
Rename and relabel attributes, arrange into groups, configure for searching
Studio Individual
application menu datastore record,
using default
attribute names
from file metadata
User details, and tabs for

Studio dashboard pages Flat list of
attributes, using
names inherited
from file metadata
Typical Studio Development Tasks
Adding value and record search components, and guided navigation

Adding tables and cross tabs, and configuring EQL queries for their data sets
Adding charts, maps and other visualizations
Creating Pages, Configuring Layouts and Adding Components
Studio uses pages with layouts to organize content, similar to OBIEEs dashboards
Components are then added to the pages, to create the dashboard content
Guided Navigation
component being
dropped onto page
layout template
Preferences and
Configuration buttons
for the component
Buttons to add new

pages, new components
Components that can be and configure layouts
added to Studio pages
Performing Value Searches (Type-ahead Searching of Records)
By default, the Search Box component provides

type-ahead value search within records
Lists all occurences of a particular value,
broken down by attribute
Does not require any prior configuration
Useful for identifying which attributes should
then be included in search interfaces in order to
enable record searches
Creating and Enabling Search Interfaces
Search interfaces create sets of attributes that are then available for record searches
When values are returned, they are used to filter the record set
Requires search interfaces to be set up and enabled using
an Endeca Server web service API and a configuration file/source Graph data flow that
configures the selected
attributes as searchable,
using an Endeca Server
WS API
Reader
component,
reading in the
Search Index
configuration file
A second graph data

flow, that sets up and
enables the required
search interfaces, again
using the WS API
Selecting and Enabling Search Interfaces in Studio
Studio components, such as Search Box, have configuration setting pages

To enable record search, one or more search interfaces need to be selected and enabled
Also provides ability to disable and enable type-ahead value search
Before record search

can be used, it needs to
be configured to use one Create a new
or more search box
search interfaces configuration, and
select search
interfaces that it
will use
Enable / disable type-

ahead value-search
Performing a Record Search
Record searches filter the record set by the specified search term, within the attributes in the
selected search interface
Record search is only performed with the search button is pressed
Default search interface is used, unless another one is specifically selected
Record set is filtered

by the search term
Select search
interface
Search term(s) appear in

the Breadcrumb list
Press Search
button
Configuring Attribute Naming and Grouping through Studio
Attribute metadata can be updated to provide display names, change sort order etc
Attributes can also be placed into groups, hidden from guided navigation menus
Configurable through Studio, with the
ability to export the new configuration
for use in Integrator List of attribute groups,
and their current
members
Studio Control
Panel, including
menu item for
managing
datastore
Attribute Settings
List of
unassigned
attributes,
currently in the
Other group
Selecting an
attribute group to
add the selected
attributes to
Configuring Attribute Naming and Grouping through a Graph
Attribute configuration is applied through another Endeca Server web service API
Exported configuration from Studio can then be used in a graph to programatically apply
configuration to other / reset data stores
Graph ETL process to

create attribute groups
and apply them to the
datastore, using the
Endeca Server WS API
Graph ETL process to

load attribute metadata
(name, datatype etc)
from a flat file and apply
to datastore using an
Endeca Server WS API
Results Tables, Cross Tabs and Data Explorer Components
Results Table component displays selected attribures for a set of records

Cross Tab component displays record set in crosstab matrix
Data Explorer component displays attribute values for an individual record
Defining the Data Source for a Cross Tab Component
Cross tab components (and optionally, table components) need to be given a data source
Datasources for these components are defined using EQL (Endeca Query Language)
Similar to SQL, assumes a single table of records, implied SELECT with GROUP BY
clauses, and other variations suited to the Endeca Server record / attribute data model
EQL Statement
defining Cross Tab
data set
EQL Statement Types and Examples
RETURN statement, returns a result set to the calling component
Return SalesTotal
AS
SELECT SUM(FactSales_SalesAmount) as TotalSales
GROUP BY DimDate_FiscalYear, DimDate_FiscalQuarter,
DimSalesTerritory_SalesTerritoryCountry
DEFINE statement, creates a temporary table with the result set, used to create views
DEFINE RegionTotals
AS
SELECT
SUM(Amount) AS Total
GROUP BY Region
Configuration and Viewing of Cross Tab Component
Once EQL query has been defined, component can then be configured and deployed on page
Cross Tab component

after it has been
successfully configured
List of available Attributes assigned to

dimension row and column
attributes and dimensions, and the
metrics metrics area
Charting and Other Data Visualization Components
Other visualization components can be used, including

Charts
Maps
Metrics Bars
Tag Clouds
Alerts
All use EQL queries
for their data source,
apart from charts
which use views
Creating EQL Views
Views are stored EQL queries, used for configuring charts

Provide an abstraction layer over the datastore, easier for end-users to work with
Can be used to create aggregates and subqueries, useful for complex analyses and
multi-step calculations
Administered using Studio, can
also be applied using a graph
(both call the same Endeca Server User Interface
and Visualizations
web service API)
View (Abstraction) Employees Products Transactions

Model View View View
Physical
Model Oracle Endeca Server
Creating Views and Using with Chart Components
Views are typically created using the View Manager is Studio, then exported for use in graphs
Once defined, they can be used (amongst other things) as data sources for charts
Chart configuration
options
List of available views,

already defined
List of previously-
defined views, available
as datasets for the
component
The view definition,
expressed as an EQL
DEFINE statement
The Finished Studio Application
Demonstrations
Oracle EID Studio
Getting Started with Oracle Endeca Information Discovery
Endeca Development Series on the Rittman Mead Blog

http://www.rittmanmead.com/category/oracle-endeca/
Getting Started with Oracle Endeca Information Discovery screencast series
http://www.oracle.com/technetwork/middleware/endeca/learnmore/index.html
Use Rittman Mead to deliver a fixed-price Data Discovery Pilot
Fixed-price of 5k + VAT + expenses, agreed scope initial pilot deployment
Up to four structured and one un/semi-structured source
Delivered in one week with an initial Studio web application
Contact us at info@rittmanmead.com
for more details!
More Information
Thank you for attending this presentation

More information can be found at http://www.rittmanmead.com
Contact us at info@rittmanmead.com or mark.rittman@rittmanmead.com
Look out for our book, Oracle Business Intelligence Developers Guide due Q3 2012
Follow-us on Twitter (@rittmanmead) or Facebook (facebook.com/rittmanmead)
Endeca Information Discovery for BI/DW Devs
Mark Rittman, Technical Director, Rittman Mead
Oracle Openworld 2012, San Francisco, October 2012

Oow2012 Oeid For Obiee Devs

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Oow2012 Oeid For Obiee Devs

Uploaded by

Copyright:

Available Formats

Endeca Information Discovery for BI/DW Devs

Mark Rittman, Technical Director, Rittman Mead

Mark Rittman, Co-Founder of Rittman Mead

As well as their search/guided navigation product, Endeca also had a BI platform

Quickly Explore All Relevant Data

Relationships Advanced search Structured

Traditional BI tools (OBIEE, Discoverer etc)

Split-second response times to

Product Quality Reports / Warranty Analysis

Online TV Shopping Sales Sentiment Analysis

Police Crime Report & Evidence Analysis

Type of Customer Traditional BI Issues Data Discovery Solution

Oracle Endeca Information

Documents, PDFs Call centre

Sales transaction data Attributes unique to just this data source,

External data from market

Jagged data, where each row may

64-bit application, benefits from multiple CPUs/cores and lots of RAM

Complementary technology to OBIEE and Essbase

Query, reporting Interactive dashboard, ad-hoc queries,

Production High-volume, pixel-perfect layout, BI Publisher

BI Foundation Suite + Endeca Information Discovery = Complete BI Solution

Data to be sourced from structured and semi-structured sources

Oracle Oracle Oracle

1. Create the Endeca Server datastore (if required)

Data mappings in Integrator are called graphs

Input Filter Sort

Integrator comes with a number of specific to Oracle EID, for example:

Endeca Server web

The initial record load can then be viewed in Studio

Joiner, joining the

Joiner, joining the

User details, and tabs for

Adding value and record search components, and guided navigation

Buttons to add new

By default, the Search Box component provides

A second graph data

Studio components, such as Search Box, have configuration setting pages

Before record search

Enable / disable type-

Record set is filtered

Search term(s) appear in

Graph ETL process to

Graph ETL process to

Results Table component displays selected attribures for a set of records

RETURN statement, returns a result set to the calling component

Cross Tab component

List of available Attributes assigned to

Other visualization components can be used, including

Views are stored EQL queries, used for configuring charts

View (Abstraction) Employees Products Transactions

List of available views,

Endeca Development Series on the Rittman Mead Blog

Thank you for attending this presentation

You might also like