Resilient Applications with Akka

Persistence
Bjrn Antonsson
@bantonsson

Konrad Malawski
@ktosopl

Patrik Nordwall
@patriknw

Reactive Applications

Akka Persistence ScalaDays 2014

Resilient
Embrace Failure
Failure is a normal part of the application lifecycle

Self Heal
Failure is detected, isolated, and managed

Akka Persistence ScalaDays 2014

The Nave Way

Write State to Database
Transactions Everywhere
Problem Solved?
Not Scalable, Responsive, Event-Driven!

Akka Persistence ScalaDays 2014

Command and Event Sourcing

Command and Event Sourcing

State is the sum of Events
Events are persisted to Store
Append only
Scales well

Akka Persistence ScalaDays 2014

Command v.s. Event

Command
What someone wants me to do
Can be rejected
Event
Something that has already happened
An immutable fact
Akka Persistence ScalaDays 2014

Commands can Generate Events

If I accept a Command and change State
Persist Event to Store
If I crash
Replay Events to recover State

Akka Persistence ScalaDays 2014

Persist All Commands?

If I crash on a Command
I will likely crash during recovery
Like the Army
Don't question orders
Repeat until success

Akka Persistence ScalaDays 2014

Only Persist Events

Only accepted Commands generate Events
No surprises during recovery
Like a dieting method
You are what you eat

Akka Persistence ScalaDays 2014

Achievement Unlocked?
Resilient
State is recoverable
Scalable
Append only writes
Something Missing?
Queries
Akka Persistence ScalaDays 2014

CQRS
Command Query Responsibility Segregation

CQRS
Separate Models
Command Model
Optimized for command processing
Query Model
Optimized data presentation

Akka Persistence ScalaDays 2014

Query Model from Events

Source the Events
Pick what fits
In Memory
SQL Database
Graph Database
Key Value Store
Akka Persistence ScalaDays 2014

Command
Store

Service
Command
Model

Client
Query
Store

Query
Model

Akka Persistence ScalaDays 2014

PersistentActor

Akka Persistence ScalaDays 2014

PersistentActor

Replaces:

Processor & Eventsourced Processor

in Akka 2.3.4+

super quick domain modelling!

Commands - what others tell us;

not persisted

sealed trait Command!

case class GiveMe(coins: Int) extends Command!
case class TakeMy(coins: Int) extends Command

Events - reflect effects, past tense;

persisted

sealed trait Event!

case class BalanceChangedBy(coins: Int) extends Event!

State - reflection of a series of events

case class Wallet(coins: Int) {!
def updated(diff: Int) = State(coins + diff)!
}

PersistentActor
var state = S0
!
def processorId = a
!

Command

!
!

Journal

PersistentActor
var state = S0
!
def processorId = a
!

Generate
Events

!
!

Journal

PersistentActor
var state = S0
!
def processorId = a
!

Generate
Events
E1

!
!

Journal

PersistentActor
var state = S0
!
def processorId = a
!

ACK
persisted
E1
!
!

Journal

PersistentActor
var state = S1
!
def processorId = a
!

E1

Apply
event
E1
!
!

Journal

PersistentActor
var state = S1
!
def processorId = a
!

E1

Okey!

E1
!
!

Journal

PersistentActor
var state = S1
!
def processorId = a
!

E1

Okey!

E1
!
!

Journal

PersistentActor
var state = S1
!
def processorId = a
!

E1

Ok, he got my $.
E1
!
!

Journal

PersistentActor

class BitCoinWallet extends PersistentActor {!

!
var state = Wallet(coins = 0)!
!
def updateState(e: Event): State = {!
case BalanceChangedBy(coins) => state.updatedWith(coins)!
}!
!
// API:!
!
def receiveCommand = ??? // TODO!
!
def receiveRecover = ??? // TODO!
!
}!

persist(e) { e => }

PersistentActor
def receiveCommand = {!
!
case TakeMy(coins) =>!
persist(BalanceChangedBy(coins)) { changed =>!
state = updateState(changed) !
}!
!
!
async callback
!
!
!
!
}

PersistentActor: persist(){}
def receiveCommand = {!
!
!
!
!
!
!
case GiveMe(coins) if coins <= state.coins =>!
persist(BalanceChangedBy(-coins)) { changed =>!
state = updateState(changed) !
sender() ! TakeMy(coins)!
}!
async callback
}
Safe to mutate

the Actors state

PersistentActor
def receiveCommand = {!
!
!
!
!
!
!
case GiveMe(coins) if coins <= state.coins =>!
persist(BalanceChangedBy(-coins)) { changed =>!
state = updateState(changed) !
sender() ! TakeMy(coins)!
}!
}

Safe to access
sender here

persist(){} - Ordering guarantees

var state = S0

C3

C2

C1

!
def processorId = a
!

E1
!
!

Journal

persist(){} - Ordering guarantees

var state = S0

C3

C2

C1

Commands get stashed until

processing C1s events are acted upon.

!
def processorId = a
!

E1
!
!

Journal

persist(){} - Ordering guarantees

var state = S0

C3

C2

C1

!
def processorId = a
!

E1
E2

events get
applied in-order

E1 E2
!
!

Journal

persist(){} - Ordering guarantees

and the cycle
repeats
var state = S0

C2
C3

!
def processorId = a
!

E2

E1

E1 E2
!
!

Journal

persistAsync(e) { e => }

persistAsync(e) { e => }
+
defer(e) { e => }

PersistentActor: persistAsync(){}
def receiveCommand = {!
!
!
!
case Mark(id) =>!
sender() ! InitMarking!
persistAsync(Marker) { m =>!
// update state...!
}!
!
!
will NOT force
!
!
}

persistAsync

stashing of commands

PersistentActor: persistAsync(){}
def receiveCommand = {!
!
!
!
case Mark(id) =>!
sender() ! InitMarking!
persistAsync(Marker) { m =>!
// update state...!
}!
!
defer(Marked(id)) { marked =>!
sender() ! marked!
}!
}

execute once all

persistAsync handlers done

NOT persisted

persistAsync(){} - Ordering guarantees

var state = S0

C3

C2

C1

!
def processorId = a
!

!
!

Journal

persistAsync(){} - Ordering guarantees

var state = S0

C2

!
def processorId = a
!

C3

!
!

Journal

persistAsync(){} - Ordering guarantees

var state = S0
!
def processorId = a
!

C3

!
!

Journal

persistAsync(){} - Ordering guarantees

var state = S0
!
def processorId = a
!

C3

E1
E2

!
!

Journal

persistAsync(){} - Ordering guarantees

var state = S0

C3

!
def processorId = a
!

E1

E1
E2

!
!

Journal
Akka Persistence ScalaDays

persistAsync(){} - Ordering guarantees

var state = S1
!
def processorId = a
!

E1
E2

E1
E2

E3
!
!

Journal
Akka Persistence ScalaDays

persistAsync(){} - Ordering guarantees

d
e
r
e
g
g
i
r
t
s
r
e
l
d hand

e
r
r
e
f
de

M1

M2

var state = S2

!
def processorId = a
!

E1

E2

E1
E2

E3
!
!

Journal
Akka Persistence ScalaDays

Recovery

Akka Persistence ScalaDays

Eventsourced, recovery

/** MUST NOT SIDE-EFFECT! */!

def receiveRecover = {!
case replayedEvent: Event => !
state = updateState(replayedEvent)!
}

re-using updateState, as seen in

receiveCommand

Akka Persistence ScalaDays

Views

Akka Persistence ScalaDays

Views
Journal

(DB)

Processor
!

def processorId = a

!
!

polling

View
!

def processorId = a

Akka Persistence ScalaDays

!
!
!

Views
Journal

(DB)

Processor
!

def processorId = a

!
!
!

different ActorPath,
same processorId
!

polling
!

polling

View
!

def processorId = a

Akka Persistence ScalaDays

!
!
!

View
!

def processorId = a

!
!
!

View

class DoublingCounterProcessor extends View {!

var state = 0!

override val processorId = "counter"!

!
def receive = {!
case Persistent(payload, seqNr) =>!
// state += 2 * payload !

!
}!
}

subject to
change!

Akka Persistence ScalaDays

Views,
as Reactive Streams

Akka Persistence ScalaDays

View, as ReactiveStream

// Imports ...!
!
import org.reactivestreams.api.Producer!
!
import akka.stream._!
import akka.stream.scaladsl.Flow!
!
import akka.persistence._!
import akka.persistence.stream._!

early preview

pull request

by krasserm

val materializer = FlowMaterializer(MaterializerSettings())!

Akka Persistence ScalaDays

View, as ReactiveStream

early preview

pull request

by krasserm
// 1 producer and 2 consumers:!
val p1: Producer[Persistent] = PersistentFlow.!
fromProcessor(processor-1").!
toProducer(materializer)!
!
Flow(p1).!
foreach(p => println(s"consumer-1: ${p.payload})).!
consume(materializer)!
!
Flow(p1).!
foreach(p => println(s"consumer-2: ${p.payload})).!
consume(materializer)

Akka Persistence ScalaDays

View, as ReactiveStream

early preview

pull request

by krasserm
// 2 producers (merged) and 1 consumer:!
val p2: Producer[Persistent] = PersistentFlow.!
fromProcessor(processor-2").!
toProducer(materializer)!

val p3: Producer[Persistent] = PersistentFlow.!

fromProcessor(processor-3").!
toProducer(materializer)!
!
Flow(p2).merge(p3). // triggers on either!
foreach { p => println(s"consumer-3: ${p.payload}") }.!
consume(materializer)!

Akka Persistence ScalaDays

Usage in a Cluster
distributed journal (http://akka.io/community/)
Cassandra
DynamoDB
HBase
MongoDB
shared LevelDB journal for testing
single writer
cluster singleton
cluster sharding

Akka Persistence ScalaDays 2014

Cluster Singleton
A
C

B
D

Akka Persistence ScalaDays 2014

Cluster Singleton
role: backend-1

role: backend-1

A
B

role: backend-2

role: backend-2

C
D

Akka Persistence ScalaDays 2014

Cluster Sharding
A

Akka Persistence ScalaDays 2014

Cluster Sharding
region
node-3
GetShardHome:17
id:17

coordinator
17 -> node2

sender

region
node-1
id:17

ShardHome:17 -> node2

region
node-2

Akka Persistence ScalaDays 2014

Cluster Sharding
region
node-3
coordinator
17 -> node2
sender

region
node-1
id:17
17 -> node2

ShardHome:17 -> node2

GetShardHome:17

id:17
region
node-2
id:17

Akka Persistence ScalaDays 2014

Cluster Sharding
region
node-3
coordinator
17 -> node2
sender

region
node-1

17 -> node2

region
node-2
id:17

17 -> node2

id:17
Akka Persistence ScalaDays 2014

17

Cluster Sharding
region
node-3
id:17

coordinator
17 -> node2

sender

region
node-1

17 -> node2

region
node-2

17 -> node2
17

Akka Persistence ScalaDays 2014

Cluster Sharding
region
node-3
coordinator
17 -> node2
sender

region
node-1

17 -> node2

region
node-2

17 -> node2
17

id:17
Akka Persistence ScalaDays 2014

Cluster Sharding

val idExtractor: ShardRegion.IdExtractor = {

case cmd: Command => (cmd.postId, cmd)
}
val shardResolver: ShardRegion.ShardResolver = msg => msg match {
case cmd: Command => (math.abs(cmd.postId.hashCode) % 100).toString
}
ClusterSharding(system).start(
typeName = BlogPost.shardName,
entryProps = Some(BlogPost.props()),
idExtractor = BlogPost.idExtractor,
shardResolver = BlogPost.shardResolver)

val blogPostRegion: ActorRef =

ClusterSharding(context.system).shardRegion(BlogPost.shardName)

val postId = UUID.randomUUID().toString

blogPostRegion ! BlogPost.AddPost(postId, author, title)

Lost messages

sender

destination

Akka Persistence ScalaDays 2014

At-least-once delivery - duplicates

Re-send
$

sender

destination

ok
ok

Akka Persistence ScalaDays 2014

At-least-once delivery - unordered

Re-send
M3

M2
M2 M1

sender

destination
M1
M3
M2
ok 1

ok 3

Akka Persistence ScalaDays 2014

ok 2

At-least-once delivery - crash

M3
M2
M3 M2 M1

sender
1. Sent M1
2. Sent M2
3. Sent M3

destination

ok 1

ok 2

4. M1 Confirmed
5. M2 Confirmed
6. M3 Confirmed

Akka Persistence ScalaDays 2014

ok 3

M1
M2
M3

PersistentActor with AtLeastOnceDelivery

case class Msg(deliveryId: Long, s: String)
case class Confirm(deliveryId: Long)
sealed trait Evt
case class MsgSent(s: String) extends Evt
case class MsgConfirmed(deliveryId: Long) extends Evt

class Sender(destination: ActorPath)

extends PersistentActor with AtLeastOnceDelivery {

!
!

def receiveCommand: Receive = {

case s: String
=> persist(MsgSent(s))(updateState)
case Confirm(deliveryId) => persist(MsgConfirmed(deliveryId))(updateState)
}
def receiveRecover: Receive = { case evt: Evt => updateState(evt) }
def updateState(evt: Evt): Unit = evt match {
case MsgSent(s) =>
deliver(destination, deliveryId => Msg(deliveryId, s))
case MsgConfirmed(deliveryId) => confirmDelivery(deliveryId)
}

Next step
Documentation
http://doc.akka.io/docs/akka/2.3.3/scala/persistence.html
http://doc.akka.io/docs/akka/2.3.3/java/persistence.html
http://doc.akka.io/docs/akka/2.3.3/contrib/cluster-sharding.html

Typesafe Activator
https://typesafe.com/activator/template/akka-sample-persistence-scala
https://typesafe.com/activator/template/akka-sample-persistence-java
http://typesafe.com/activator/template/akka-cluster-sharding-scala

Mailing list
http://groups.google.com/group/akka-user

Migration guide from Eventsourced

http://doc.akka.io/docs/akka/2.3.3/project/migration-guide-eventsourced-2.3.x.html

Akka Persistence ScalaDays 2014

Typesafe 2014 All Rights Reserved