Exploring Reactive Integrations with

akka streams,
Alpakka and Kafka

Konrad `ktoso` Malawski

Konrad `ktoso` Malawski

Akka Team,
Reactive Streams TCK,
Persistence, HTTP, Remoting / Cluster

Make building powerful concurrent &

distributed applications simple.
Akka is a toolkit and runtime
for building highly concurrent,
distributed, and resilient
message-driven applications
on the JVM

Whats in the toolkit?

Actors simple & high performance concurrency
Cluster / Remoting location transparency, resilience
Cluster tools and more prepackaged patterns
Streams back-pressured stream processing
Persistence Event Sourcing
HTTP complete, fully async and reactive HTTP Server
Oicial Kafka, Cassandra, DynamoDB integrations, tons
more in the community
Complete Java & Scala APIs for all features

Stream

has many meanings

akka streams
Asynchronous back pressured stream processing

Source

Sink

Flow

akka streams
Asynchronous back pressured stream processing

Source

(possible)
asynchronous
boundaries

Flow

Sink

akka streams
Asynchronous back pressured stream processing

Source
10 msg/s

Sink

Flow

1 msg/s
OutOfMemoryError!!

akka streams
Asynchronous back pressured stream processing

Source
10 msg/s
1 msg/s
hand me 3 more

Sink

Flow

1 msg/s
hand me 3 more

akka streams
Not only linear streams

Source

Flow

Flow

Source

Flow

Sink
Sink

Reactive
And the many meanings it carries.

The many meanings of Reactive

reactivemanifesto.org

The many meanings of Reactive

Not-quite-Reactive-System
The reason we started researching
into transparent to users flow control.

Reactive Streams
Reactive Streams is an initiative to provide a standard for
asynchronous stream processing with non-blocking back
pressure. This encompasses eorts aimed at runtime
environments as well as network protocols

http://www.reactive-streams.org

Reactive Streams
A buiding-block of Reactive Systems, not the entire story.

Reactive Streams
Reactive Streams is an initiative to provide a standard for
asynchronous stream processing with non-blocking back
pressure. This encompasses eorts aimed at runtime
environments as well as network protocols

http://www.reactive-streams.org

Part of JDK 9

java.util.concurrent.Flow

http://openjdk.java.net/projects/jdk9/

JEP-266 soon!
public final class Flow {
private Flow() {} // uninstantiable
@FunctionalInterface
public static interface Publisher<T> {
public void subscribe(Subscriber<? super T> subscriber);
}

public static
public void
public void
public void
public void
}

interface Subscriber<T> {
onSubscribe(Subscription subscription);
onNext(T item);
onError(Throwable throwable);
onComplete();

public static interface Subscription {

public void request(long n);
public void cancel();
}
public static interface Processor<T,R> extends Subscriber<T>, Publisher<R> {
}
}

Reactive Streams

RS Library A

RS library B

async
boundary

Reactive Streams

RS Library A

RS library B

async
boundary
Make building powerful concurrent &
distributed applications simple.

Akka Streams

The API

Complete and awesome

Java and Scala APIs
(Just like everything in Akka)

Akka Streams in 20 seconds:

Source<Integer, NotUsed> source = null;

Flow<Integer, String, NotUsed> flow =

Flow.<Integer>create().map((Integer n) -> n.toString());

Sink<String, CompletionStage<Done>> sink =

Sink.foreach(str -> System.out.println(str));

RunnableGraph<NotUsed> runnable =
source.via(flow).to(sink);

runnable.run(materializer);

Akka Streams in 20 seconds:

CompletionStage<String> firstString =
Source.single(1)
.map(n -> n.toString())
.runWith(Sink.head(), materializer);

Akka Streams in 20 seconds:

Source.single(1).map(i -> i.toString).runWith(Sink.head())

// types:
_
Source<Int, NotUsed>
Flow<Int, String, NotUsed>
Sink<String, CompletionStage<String>>

Akka Streams in 20 seconds:

Source.single(1).map(i -> i.toString).runWith(Sink.head())

// types:
_
Source<Int, NotUsed>
Flow<Int, String, NotUsed>
Sink<String, CompletionStage<String>>

Materialization

Gears from GeeCON.org,(its an awesome conf)

What is materialization really?

What is materialization really?

What is materialization really?

Introspection, and more coming

Highly sophisticated stream introspection capabilities.

Alpakka

A community for Streams connectors

http://blog.akka.io/integrations/2016/08/23/intro-alpakka

Alpakka a community for Stream connectors

Threading & Concurrency in Akka Streams Explained (part I)
Mastering GraphStages (part I, Introduction)
Akka Streams Integration, codename Alpakka
A gentle introduction to building Sinks and Sources using GraphStage APIs
(Mastering GraphStages, Part II)
Writing Akka Streams Connectors for existing APIs
Flow control at the boundary of Akka Streams and a data provider
Akka Streams Kafka 0.11

Alpakka a community for Stream connectors

http://developer.lightbend.com/docs/alpakka/current/

Alpakka a community for Stream connectors

o
m
e
D

Akka Streams & HTTP

streams
& HTTP

Akka Streams / HTTP

A core feature not obvious to the untrained eye!
Quiz time!
TCP is a ______ protocol?

Akka Streams / HTTP

A core feature not obvious to the untrained eye!
Quiz time!
TCP is a STREAMING protocol!

Streaming in Akka HTTP

HttpServer as a:
Flow[HttpRequest, HttpResponse]

http://doc.akka.io/docs/akka/2.4/scala/stream/stream-customize.html#graphstage-scala
Framed entity streaming
http://doc.akka.io/docs/akka/2.4/java/http/routing-dsl/source-streaming-support.html

Streaming in Akka HTTP

HttpServer as a:
Flow[HttpRequest, HttpResponse]
HTTP Entity as a:
Source[ByteString, _]

http://doc.akka.io/docs/akka/2.4/scala/stream/stream-customize.html#graphstage-scala
Framed entity streaming
http://doc.akka.io/docs/akka/2.4/java/http/routing-dsl/source-streaming-support.html

Streaming in Akka HTTP

HttpServer as a:
Flow[HttpRequest, HttpResponse]
HTTP Entity as a:
Source[ByteString, _]
Websocket connection as a:
Flow[ws.Message, ws.Message]
http://doc.akka.io/docs/akka/2.4/scala/stream/stream-customize.html#graphstage-scala
Framed entity streaming
http://doc.akka.io/docs/akka/2.4/java/http/routing-dsl/source-streaming-support.html

Resource efficiency and dynamic fan-out

Resource efficiency and dynamic fan-out

Its turtles buffers all the way down!

xkcd.com

Streaming from Akka HTTP

Streaming from Akka HTTP

Streaming from Akka HTTP (Java)

public static void main(String[] args) {
final ActorSystem system = ActorSystem.create();
final Materializer materializer = ActorMaterializer.create(system);
final Http http = Http.get(system);

final Source<Tweet, NotUsed> tweets = Source.repeat(new Tweet("Hello world"));

final Route tweetsRoute =
path("tweets", () ->
completeWithSource(tweets, Jackson.marshaller(), EntityStreamingSupport.json())
);

final Flow<HttpRequest, HttpResponse, NotUsed> handler =

tweetsRoute.flow(system, materializer);
http.bindAndHandle(handler,
ConnectHttp.toHost("localhost", 8080),
materializer
);
System.out.println("Running at http://localhost:8080");
}

Streaming from Akka HTTP (Java)

public static void main(String[] args) {
final ActorSystem system = ActorSystem.create();
final Materializer materializer = ActorMaterializer.create(system);
final Http http = Http.get(system);

final Source<Tweet, NotUsed> tweets = Source.repeat(new Tweet("Hello world"));

final Route tweetsRoute =
path("tweets", () ->
completeWithSource(tweets, Jackson.marshaller(), EntityStreamingSupport.json())
);

final Flow<HttpRequest, HttpResponse, NotUsed> handler =

tweetsRoute.flow(system, materializer);
http.bindAndHandle(handler,
ConnectHttp.toHost("localhost", 8080),
materializer
);
System.out.println("Running at http://localhost:8080");
}

Streaming from Akka HTTP (Scala)

object Example extends App
with SprayJsonSupport with DefaultJsonProtocol {
import akka.http.scaladsl.server.Directives._
implicit val system = ActorSystem()
implicit val mat = ActorMaterializer()
implicit val jsonRenderingMode = EntityStreamingSupport.json()
implicit val TweetFormat = jsonFormat1(Tweet)
def tweetsStreamRoutes =
path("tweets") {
complete {
Source.repeat(Tweet(""))
}
}
Http().bindAndHandle(tweetsStreamRoutes, "127.0.0.1", 8080)
System.out.println("Running at http://localhost:8080");
}

Next steps for Akka

Completely new Akka Remoting (goal: 700.000+ msg/s (!)),
(it is built using Akka Streams, Aeron).

More integrations for Akka Streams stages, project Alpakka.

Reactive Kafka polishing with SoftwareMill, Krzysiek Ciesielski
Akka Typed progressing again, likely towards 3.0.
Akka HTTP 2.0 Proof of Concept in progress.
Collaboration with Reactive Sockets

Ready to adopt on prod?

Totally, go for it.

Akka <3 contributions

Easy to contribute tickets:
https://github.com/akka/akka/issues?q=is%3Aissue+is%3Aopen+label%3Aeasy-to-contribute
https://github.com/akka/akka/issues?q=is%3Aissue+is%3Aopen+label%3A%22nice-to-have+%28low-prio%29%22

Akka Stream Contrib

https://github.com/akka/akka-stream-contrib

Mailing list:
https://groups.google.com/group/akka-user
Public chat rooms:
http://gitter.im/akka/dev developing Akka
http://gitter.im/akka/akka using Akka

More resources:
Read more tutorials and deep-dives:
http://blog.akka.io/
https://www.lightbend.com/resources

Reactive Platform

Reactive Platform
Reactive Platform

Further reading:
Reactive Streams: reactive-streams.org
Akka documentation: akka.io/docs
Free OReilly report bit.ly/why-reactive
Example Sources:
ktoso/akka-streams-alpakka-talk-demos-2016

Contact:
Konrad ktoso@lightbend.com Malawski
http://kto.so / @ktosopl

Thanks!
Questions?

@ktosopl konrad.malawski@lightbend.com
@apnylle johan.andren@lightbend.com