Raghavendra Prabhu (RVP)

Zen:
Pinterests Graph Storage Service
Given how robust the messenger is on
day one, its surprising to learn that
Pinterest built the entire product
in three months. The Verge
What does it take to do this consistently?
Many di!erent things
#
Hire the best
#
Culture
#
Focus
Infrastructure that doesnt get in the way
Persistent Storage
Even with a distributed database, app needs to deal with:
#
Schema design
#
Fault tolerance
#
Capacity management
#
Performance tuning
Solution 1: UserMetaStore
Storage-as-a-Service: Key-value thrift API on top of HBase
Features:
#
Key partitioning to balance load
#
Master-slave clusters, semi automatic failover
#
Speculative execution
#
Multi-tenancy with tra$c isolation
Storage-as-a-service is a great step forward,
but can we do better?
Example: Messages Data Model
Conversation
Message 1
Message 2
Message N
User
User
Participates Contains
Realization
#These object models closely resemble a graph
#Objects are nodes, edges represent relationships
#Typical needs:
# retrieve data for a node or edge
# get all outgoing edges from a node
# get all incoming edges from a node
# count incoming or outgoing edges for a node
Enter Zen!
#Provides a graph data model instead of key-value
#Automatically creates necessary indexes
#Materializes counts for e$cient querying
#Implemented on top of HBase, but can plug in other backends
Why the name Zen?
#Data model inspired by Facebooks TAO
#But internally a very di!erent system
#Zen:
# evolution of Buddhism under Taoist conditions
# simplied version of Taoism
# basically Pinterests take on the TAO idea :)
What Zen is NOT
#NOT a full edged graph database
#NO advanced graph operations
#Basically an object-relationship data model on top of existing
databases to simplify app development
Zen API
Nodes:
#
addNode, removeNode, getNode
#
Node id: globally unique 64-bit integer
ID 123
Prop 1 Val 1
Prop 2 Val 2
Zen API
Edges:
#
addEdge, removeEdge, getEdge
#
Edge Ref: (edgeType, fromId, toId)
Edge Ref 120, 123, 4567
Prop 1 Val 1
Prop 2 Val 2
Zen API
Edge Queries:
#
getEdges, countEdges, removeEdges
struct EdgeQuery {

1: required NodeId nodeId;

2: required EdgeDirection direction;

3: optional TypeId edgeType;

}
Zen API
Property Indexes
#
Unique index
#
Ensures a property value is unique across all nodes of a type
#
Non-unique index
#
Allows retrieval by property value
#
Works for both nodes and edges
Illustration: Messages on Zen
Id:1234 Id:2345 Id:3456
Type: Participates Type: Contains
Type: Conversation
Started: 12 Aug 2014 08:00
Header: Great pin!
Pin Id: 10001 [non-unique]
Type: User
Name: Ben Smith [unique]
Status: Active
Type: Message
Sent: 12 Aug 2014 08:00
Text: Great pin!
Zen: Current Usage
Products:
# smart feed, messages, network news, interest graph and other
upcoming features

Numbers:
# ~10 clusters
# 100,000+ requests per second at peak
# Over 5 million HBase operations per second
Xun Liu
Internals and Production Learnings
Internals - Property
Data
type name score distance
node-12345 10 Ben Smith
edge-12345-20-67890 1000 1 mile
Internals - Property Index
Data
ID
<hash>-unique-10-name=ben smith 12345
<hash>-nonuniq-10-lastname=smith-12345
<hash>-nonuniq-10-lastname=smith-67890
Internals - Edge Score Index
Data
12345-out-20-1000-567890
12345-out-20-1001-567891
567890-in-20-1000-12345
567891-in-20-1000-12345
Internals - Edge Count
Data
Count
12345-out-20 2
12345-in-30 4
Status - Soft Delete
New Features
Built-in Cache
New Features
Zen
Cache
HBase Client Zen HBase Client
Cache
Before After
Namespace
New Features
Node
Namespace 1
Edge Index Node
Namespace 2
Edge Index
New Features
#Online type schema change
#Optional reverse edge
#Optional edge count
#Retrieval of subset of properties
#Descending edge score
Performance Work
Demanding work load needs special tuning
# Inserting 1 million edges per second
# Excessive HLog (WAL) ushes
Performance Work
Batching
# Client Side Batching bulk edge insertion
# Zen Server Side Batching bu!er edits across clients & ush
together
# Reduce HLog (WAL) ushes by orders of magnitude
Performance Work
Tuning storage engine
# Bloomlter, Blocksize, Encoding, Compression etc.

Zen production setup
# Dedicated Zen cluster
# Namespace
Distributed transaction or not?
Data Consistency
Stay on top of data inconsistencies
# Manual rollback in Zen server
# O&ine jobs (Dr Zen) to scan and x inconsistencies
# Tools to debug and x one-o! inconsistency
Future Work
#Dr Zen (make it more e$cient)
#Other backends: MySQL, Redis, etc
#Distributed transactions
#Open source!