Government Agencies & The Big Data Phenomenon

The Bigger The Haystack, The Clearer The Needle

Tim Paydos, WW Govt Big Data Leader tpaydos@us.ibm.com, 860 478 4670
2
Quick Summary
Federal Big Data Commission
Sponsored by the White House to bring
together experts from Government,
Academia and Industry
Seeks to De-Mystify Big Data, and focus
on the business and mission value it will
deliver
Frame the discussion through the lens of
early best practices, and key lessons
learned
Extract recommendations on how to
assess potential operational business
value; Provide a practical, experience
based roadmap for getting started
Analytics & The Big Data Phenomona
Casual Observations, And Some Advice And Guidance
Veracity
Variety
Volume
Velocity
Something Profound Is Happening
Big Data is not a technology
Big Data is nothing new
Big Data will be transformational and revolutionary
The path to harnessing Big Data is iterative, and evolutionary
What This Means is.
Do not fall for the Big Data Head Fake
Start with the business or mission requirement
Understand the Art of the Possible
Find the compelling use case
Assess Current capabilities against whats required
Get the initial project, move iteratively
Governments Must Lead In The Face of Global Challenges
Key Drivers Of Transformation Creating New Expectations
Key Drivers of Transformation
Required Business
Outcomes
Improved citizen
services & experience
Strengthen public
safety and defense
Transparency and
accountability
Share information across and
between agencies
Manage Resources
Effectively, Reduce
Waste & Fraud
Maximize compliance
& revenue
Changing Demographics
Rapidly Developing Role of
Social Media
Evolving Societal
Relationships
Growing Threats to Social
Stability and Order
Expanding Impact of
Technology
Accelerating Globalization
E
X
T
R
E
M
E

B
U
D
G
E
T

P
R
E
S
S
U
R
E
Intensifying Information
Requirements
Provide transparency to an
ever more demanding public
Leverage improved insight to
manage effectiveness, predict
future demands and requirements
Recognize and optimize service
to the citizen, soldier, veteran,
and patient at any point of
contact
Enabling defense, intelligence and
law enforcement organizations to
achieve situational awareness
Pro-actively recognize and
mitigate patterns of fraud, abuse
and non-compliance
5
What Is Big Data, Really?
Big Data is all data from everywhere - the next natural resource
We have for the first time an economy based on a key resource [Information] that is not only
renewable, but self-generating. Running out of it is not a problem, but drowning in it is
-- John Naisbitt
1 in 3 business leaders
dont trust the information
they use to make decisions
80% of the worlds
data is unstructured
Veracity
30 billion RFID sensors
and counting
Variety
Volume
25 zettabytes worth of
data will exist by 2020
Velocity
harvesting this resource requires mining, refining and delivering
Big Data Creates A Challenge And an Opportunity
The Bigger The Haystack, The Clearer The Needle
Look At All The Data Look At Even Dirty Data
Let Data Lead the Way Leverage Data as it is Captured
7
Leading The Way
Government Agencies Harnessing The Value of Big Data
Smarter Water
$120M in savings, reduction in reportable
incidents
Insider Threat
Perpetual credentialing and vetting
across branches and bases
Improved Public Safety And Mass
Casualty Incident Management
60X Acceleration in Active Wild Fire
behavioral analysis, with greater accuracy
Reduced traffic congestion, shorter travel
times, enhance emergency services
Smarter Cities
Maritime Emergency Management
Early and accurate detection and
notification of marine resource adverse
events
Optimized wind turbine energy production
with longer life span and reduced
maintenance
Smarter Energy:
Wind Infrastructure
Optimized Neo-Natal care through early
detection, prevention, testing hypotheses
Smarter Social Services:
Smarter Health Care Analytics
Leading The Way
Government Agencies Harnessing The Value of Big Data
Greatly reduced frequency and severity of
Traumatic Brain Injury
Warfighter Care
8
Smarter Law Enforcement
30% Reduction in Violent Crime
Efficiently read, process and analyze large
volumes of political debate-related, public
feedback in real-time
Social Media Analytics
Enhanced collaboration across the
electrical grid for greater demand and
maintenance efficiency
Smarter Energy
Early and accurate detection and
notification of events effecting watershed
Smarter Resource Management:
Watershed Intelligence
Leading The Way
Government Agencies Harnessing The Value of Big Data
Enhanced Citizen Outcomes
Enhanced juvenile and family services
Advanced return review analytics
through real time Fraud Analytics
Smarter Revenue:
Tax Compliance
In Four Hours Identified $140M in
Improper Payments
Smarter Social Services
10
Leading the Way
Maritime Intelligence & Law Enforcement
What We Have Learned
A Deliberate Approach Is Required
These experiences reveal a great irony -- that while the impact of Big Data will be transformational,
the path to effectively harnessing it is not. The journey is evolutionary versus revolutionary,
incremental and iterative
Demystifying Big Data, TechAmerica Report, October 2012
1. Understand the Art of the Possible.
2. Start with a clear mission or business requirement,
and fully define a discrete set of use cases.
3. Take inventory and understand your data assets.
4. Assess your current set of capabilities and
technical architecture against what is required to
support your initial use cases.
5. Explore which data assets can be exposed for
public consumption, to drive innovation and the
development of Big Data solutions.
6. Focus on developing skills and talent
7. Do not seek to develop new policies on
governance, security and privacy. Rather, help
agencies understand how current policies apply
Staging area
Actionable insight
Enterprise
warehouse
Data mart
Data sources
Transaction and
application data
Archive
Reporting and
analysis
Predictive analytics
and modeling
What We Have Learned
Successful Big Data Initiatives Require A Shift From Traditional data environments.
Actionable insight
Enterprise
warehouse
Data mart
To The Next Generation Analytic Environment
Provide dedicated analytics processing for faster, deeper analysis and modeling
Analytic
appliances
Data sources
Archive
Transaction and
application data
Staging area
Reporting and
analysis
Predictive analytics
and modeling
Actionable insight
Exploration,
landing and
archive
Enterprise
warehouse Data mart
To The Next Generation Analytic Environment
Leverage Hadoop for exploration in native format, Plus archive
Analytic
appliances
Data sources
+
+
Transaction and
application data
Enterprise
content
Social data
Image and video
Third-party data
Archive
Predictive analytics
and modeling
Reporting,
analysis, content
analytics
Discovery and
exploration
Predictive analytics
and modeling
Decision
management
Exploration,
landing and
archive
Enterprise
warehouse
Data mart
To The Next Generation Analytic Environment
Leverage data in motion
Analytic
appliances
Data sources
+
+
Real-time analytics
Transaction and
application data
Machine and
sensor data
Enterprise
content
Social data
Image and video
Third-party data
Actionable insight
Reporting,
analysis, content
analytics
Discovery and
exploration
Actionable insight
Exploration,
landing and
archive
Enterprise
warehouse
Data mart
To The Next Generation Analytic Environment
Extend transformation, matching and security capabilities to ALL data
Analytic
appliances
Data sources
+
+
Information governance
Real-time analytics
Transaction and
application data
Machine and
sensor data
Enterprise
content
Social data
Image and video
Third-party data
Predictive analytics
and modeling
Decision
management
Reporting,
analysis, content
analytics
Discovery and
exploration
What We Have Learned
A holistic & integrated approach to Big Data & Analytics leads to success
Assemble and combine relevant mix of
information
Discover and explore with smart visualizations
Analyze, predict and automate for more
accurate answers
Take action and automate processes
Optimize analytical performance and IT costs
Reduced infrastructure complexity and cost
Manage, govern and secure information
Enabling agencies to
BI /
Reporting
BI /
Reporting
Exploration /
Visualization
Functional
App
Industry
App
Predictive
Analytics
Content
Analytics
Analytic Applications
Big Data Platform
Systems
Management
Application
Development
Visualization
& Discovery
Accelerators
Information Integration & Governance
Hadoop
System
Stream
Computing
Data
Warehouse
Cloud I Mobility I Security
The Parting Shot
The Key Lies in Building Trusted Information Is Critical -- On Multiple Levels
Insightful
Comprehensive analytics to drive
new meaning from your data
Governed & Secure
Accurate & Complete
Complex and disparate data transformed, cleansed,
reconcile and delivered
Can I Trust The Information?
Rules are in place and tools are deployed to limit visibility,
secure sensitive information, and protect privacy
Can I Trust My Partner and My Own Agency?
Can I extract new Insight to make it all worthwhile?
Government Agencies & The Big Data Phenomenon
The Bigger The Haystack, The Clearer The Needle
Tim Paydos, WW Govt Big Data Leader tpaydos@us.ibm.com, 860 478 4670