Deep Dive into

Highly Available
OpenStack Architecture
Arthur Berezin,
Sr. Technical Product Manager,
Red Hat

OpenStack Summit
Vancouver May 2015

Agenda
HA Enabling Services
Pacemaker and HAProxy

Shared Services
MariaDB w/Galera, RabbitMQ w/Mirrored Queues

OpenStack Services
Keystone, Nova, Neutron, Glance, Cinder, Horizon

Topologies
Controller, Compute, Network, Storage

cc: Morio2015 Source: https://www.wikiwand.com/en/Scuderia_Ferrari

Losing Your Controller

https://www.youtube.com/watch?v=Kb43Nxuwc4I

High Availability
Minimize downtime by avoiding SPOF
Create service redundancy
Active-Active When possible
Stateless services or HA internal support
Active-Passive if nothing else is applicable

Scale out Architecture

HA Enabling Technologies
Pacemaker, HAProxy

Pacemaker
Cluster Resource Manager
Uses Corosync for cluster communication
Monitor and Control Resources:
Floating Virtual IP Address (VIP)
SystemD/LSB/OCF Services
Cloned Services(Active/Active)

STONITH - Fencing with Power Management

Important for ensuring data consistency

Pacemaker OpenStack Service

Virtual IP(VIP)
SystemD Cloned Resource
STONITH Fencing

Service
Virtual IP
10.0.0.1

Service

pcsd

Cloned

Service

pcsd

Node 1 - 192.168.1.1

Node 2 - 192.168.1.2

STONITH

STONITH

HAProxy Load Balancer

Load Balancing and Proxy for HTTP/TCP
Mature and popular with web applications
Health Checking
Load Distribution

HAProxy Load Balancer

Load Distribution

HAProxy

Round Robin,
Stick-Table

Node 1

API Isolation
Failure Detection
Service
Node 2

Service
Node 3

Avoiding SPOFs
A day in a Highly Available Service Life

Give Me Horizon
Web UI NOW!

Horizon
Controller

Give Me Horizon
Web UI NOW!

Single Point Of Failure

Horizon
Controller

Give Me Horizon
Web UI NOW!

HAProxy
Controller 1

Horizon
Controller 1

Horizon
Controller 2

Horizon
Controller 3

Give Me Horizon
Web UI NOW!

Single Point Of Failure

HAProxy
Controller 1

Each Could Fail

Horizon
Controller 1

Horizon
Controller 2

Horizon
Controller 3

Give Me Horizon
Web UI NOW!

Single Point Of Failure

HAProxy
Controller 1

Pacemaker Cloned Horizon Service

Horizon
Controller 1

Horizon
Controller 2

Horizon
Controller 3

Give Me Horizon
Web UI NOW!

Pacemaker Cloned HAProxy Service

HAProxy
Controller 1

HAProxy
Controller 2

HAProxy
Controller 3

Horizon
Controller 2

Horizon
Controller 3

Pacemaker Cloned Horizon Service

Horizon
Controller 1

Give Me Horizon
Web UI NOW!
Horizon

Pacemaker Cloned HAProxy Service

HAProxy
Controller 1

VIP

HAProxy
Controller 2

HAProxy
Controller 3

Horizon
Controller 2

Horizon
Controller 3

Pacemaker Cloned Horizon Service

Horizon
Controller 1

Shared Components
Database, Messaging

Galera with MariaDB

Active-Active MultiMaster Synchronous
Replication
Auto Node Joining
Row level parallel replication
Native with MariaDB
MariaDB

MariaDB

MariaDB

wsrep

wsrep

wsrep

DB Node 1

GALERA
DB REPLICATION
Node 2

DB Node 3

RabbitMQ Clustering
RabbitMQ Clustering with Mirrored Queues

RabbitMQ

RabbitMQ Node1

RabbitMQ Mirrored Queue

RabbitMQ Node1

RabbitMQ

RabbitMQ Node1

OpenStack Services
Keystone, Glance, Cinder, Nova, Neutron, Horizon

Keystone

Keystone

API Call

Service:
httpd/Keystone
API
Assignments
Identities
LDAP
SQL

HTTPD
Keystone

SQL: Assignments

SQL: Identities

LDAP: Identities

Keystone

API Call

Keystone

VIP

Cloned Stateless
HTTPD Service
Same SSL Certs
on all nodes
Cache is local
on each host

HAProxy

Cloned

HAProxy

HTTPd/
Keystone

Cloned

HTTPd/
Keystone

pcsd

pcsd

Node 1

Node 2

STONITH

SQL: Assignments

STONITH

SQL: Identities

LDAP: Identities

Keystone

API Call

Keystone

VIP

Cloned Stateless
HTTPD Service
Same SSL Certs
on all nodes
Cache is local
on each host

HAProxy

Cloned

HAProxy

HTTPd/
Keystone

Cloned

HTTPd/
Keystone

pcsd

pcsd

Node 1

Node 2

STONITH

SQL: Assignments

STONITH

SQL: Identities

LDAP: Identities

Glance

Glance
Service:
Glance-API

Ceilometer
Notifications

API
Storage Calls

Glance-Registry
Keeps images
registry at the
Database

RabbitMQ
Glance-API

Storage

HTTP

Glance
Registry

SQL

Glance
Both services are
Cloned Active/Active
Both services are
LB and VIP

Glance
API

Glance
Registry

VIP

VIP

HAProxy

Cloned

HAProxy

Glance-API

Cloned

Glance-API

Glance
Registry

Cloned

Glance
Registry

pcsd

pcsd

Node 1

Node 2

STONITH

STONITH

Images Store

SQL

Glance
Both services are
Cloned Active/Active
Both services are
LB and VIP

Glance
API

Glance
Registry

VIP

VIP

HAProxy

Cloned

HAProxy

Glance-API

Cloned

Glance-API

Glance
Registry

Cloned

Glance
Registry

pcsd

pcsd

Node 1

Node 2

STONITH

STONITH

Images Store

SQL

Glance
Both services are
Cloned Active/Active
Both services are
LB and VIP

Glance
API

Glance
Registry

VIP

VIP

HAProxy

Cloned

HAProxy

Glance-API

Cloned

Glance-API

Glance
Registry

Cloned

Glance
Registry

pcsd

pcsd

Node 1

Node 2

STONITH

STONITH

Images Store

SQL

Cinder

Cinder
Cinder-API

Cinder-API

API

Cinder-Scheduler

SQL

Volumes placement

Cinder-Volume
Manages Storage

Cinder
Scheduler

RabbitMQ

Cinder
Volume

Cinder
Backup

Driver

Cinder-Backup
VM

Storage
Data Path

Storage

Cinder
API

VIP

Cinder
Cinder-API is
Stateless Cloned
LB and VIP
Cinder-Volume is A/P
due it potential
races
Cinder-Backup is A/P

HAProxy

Cloned

HAProxy

Cinder-API

Cloned

Cinder-API

Scheduler

Cloned

Scheduler

Volume

Node 1
STONITH

Driver

pcsd

A/P

Volume

Node 2

Driver

pcsd

STONITH

Storage

Nova

Nova
Nova-API

Nova-API

API

Nova-Scheduler
VM placement

Nova-Conductor
Updates DB on
Computes behalf

Nova-Compute
Runs VM Instances

SQL

Nova
Scheduler

RabbitMQ

Nova
Conductor

Nova
Compute
libvirt/KVM
VM

VM

Nova
Nova-API

Nova-API

API

Nova-Scheduler
VM placement

Nova-Conductor
Updates DB on
Computes behalf

Nova-Compute
Runs VM Instances

SQL

Nova
Scheduler

RabbitMQ

Nova
Conductor

Nova
Compute
libvirt/KVM

Controller
Services

VM

Compute

VM

Nova-API

VIP

Nova
Controller Services
Nova-API configured
with LB and VIP
Nova-API,
Nova-Scheduler and
Nova-Conductor
are Stateless A/A
Cloned services

HAProxy

Cloned

HAProxy

Nova-API

Cloned

Nova-API

Scheduler

Cloned

Scheduler

Conductor

Cloned

Conductor

Node 1

pcsd

STONITH

Node 2
STONITH

SQL

RabbitMQ

pcsd

Nova
Compute Service
Each host is independent
Nova-compute watched locally
by SystemD
VM HA not supported(yet),
Probably Liberty

Nova
Compute
libvirt/KVM
VM

VM

Compute1

Nova
Compute
libvirt/KVM
VM

Compute2

Nova VM HA
Compute Service
Probably supported in Liberty
Each host is independent
Nova-compute watched locally
by SystemD
Liberty Blueprint: Mark Host Down

pacemaker_remote
Nova
Compute
libvirt/KVM
VM

VM

Compute1
STONITH

pacemaker_remote
Nova
Compute
libvirt/KVM
VM

Compute1
STONITH

VM

Neutron

Neutron
Neutron Server

L2 Agent(s)
Open vSwitch

API and Management

Neutron L2 Agent
L2 Traffic on compute

Neutron L3 Agent

L3 Agent
Neutron
Server

DHCP
Agent

Network Routing

DHCP Agent
LBaaS Agent

RabbitMQ

SQL

LBaaS
Agent

Neutron
Server

L2 Agent(s)
Open vSwitch

L3 Agent

DHCP
Agent

LBaaS
Agent

R1
SQL
Controller

RabbitMQ

Neutron
Network
Node

VM

Internet

VM

VM

VM

L2 Agent(s)
Compute1

Open vSwitch

VM

VM

L2 Agent(s)
Compute2

Open vSwitch

Neutron
API

VIP
L2 Agent
HAProxy
Neutron-API

Cloned

Cloned

HAProxy
Neutron-API
RPC

pcsd

Controller1

L2 Agent

LBaaS Agent

A/P

LBaaS Agent

DHCP Agent

Cloned

DHCP Agent

L3 Agent

Cloned
+
VRRP

L3 Agent

pcsd

Controller2

R1

R1
Network
Node2

Network
Node1

Compute1

Neutron
Kilo
L3 Agent HA with VRRP
DHCP Agent HA
Liberty
L3 Agent - DVR
DVR + VRRP
Longer Term
Distributed DHCP on compute nodes

Horizon

Browser

Horizon
Service:
httpd/OpenStackDashboard

Horizon

Django web app

Uses services APIs
Keystone
API

Nova
API
Glance
API

Neutron
API
Cinder
API

Horizon
Horizon

VIP

Cloned Stateless
HTTPd Service
Same SSL Certs
on all nodes
Cache is local
on each host

HAProxy

Cloned

HAProxy

HTTPd/
Horizon

Cloned

HTTPd/
Horzon

pcsd

pcsd

Node 1

Node 2

STONITH

STONITH

Topologies
Controller, Compute,Network, Storage

Active - Active Controller Cluster

Controller 1

Controller 2

Controller 3

Packemaker

Packemaker

Packemaker

HAProxy

HAProxy

HAProxy

Keystone

Keystone

Keystone

Neutron

Neutron

Neutron

Cinder

Cinder

Cinder

...

...

....

RabbitMQ Mirrored Queues

MariaDB

MariaDB
Galera Multi-master replication

MariaDB

Controller Cluster
Controller 1

Public
Tenant
Management

Controller 2

Controller 3

Controller Services

Controller Services

Controller Services

Controller Services

Controller Services

Controller Services

Nova Compute

Compute1

Nova Compute

Compute2

Controller Cluster
Controller 1

Controller 2

Public
Tenant
Management

Network Cluster
Controller 3

Nova Compute

Compute1

Neutron
Network
Node1

Nova Compute

Compute2

Neutron
Network
Node2

Neutron
Network
Node3

Controller Cluster
Controller 1

Controller 2

Storage Cluster
Controller 3

Cinder
Glance
Node1

Cinder
Glance
Node2

Volume Storage
Nova Compute

Storage
Management

Compute1

Nova Compute

Compute2

Cinder
Glance
Node3

Image Store

Resources

Resources
RDO HA Ref Arch
https://github.com/beekhof/osp-ha-deploy
Layer 3 High Availability - VRRP DVR DHCP
http://assafmuller.com/2014/08/16/layer-3-high-availability/
DVR
http://assafmuller.com/2015/04/15/distributed-virtual-routingoverview-and-eastwest-routing/
Creating a Highly Available Red Hat OpenStack Platform
Configuration (OSP5 and RHEL 7)
https://access.redhat.com/articles/1150463
About High Availability with OpenStack Platform
https://access.redhat.com/articles/1274203

New nova API call to mark nova-compute down

https://review.openstack.org/#/c/169836/
The Different Facets of OpenStack HA

http://blog.russellbryant.net/2015/03/10/the-different-facets-ofopenstack-ha/
Implementation of Pacemaker Managed OpenStack VM Recovery

http://blog.russellbryant.net/2015/04/08/implementation-ofpacemaker-managed-openstack-vm-recovery/

HA Talks during Summit

HA Infrastructure Talks
Pacemaker: OpenStacks PID 1
MariaDB Galera cluster : Best practices
High Availability Architecture

HA Storage Talks
Keeping OpenStack storage trendy with Ceph and containers
DRBD9 for OpenStack
The Road to Enterprise-Ready OpenStack Storage as Service

Deep Dive Into a Highly Available OpenStack Architecture

Dude, where is my volume

Real World Practices

HA Networking Talks

Highly Available OpenStack: From Theory to Reality

Highly Available, Performant, VXLAN Service Node

Lessons learned on upgrades: the importance of HA and

IPv6 impact on Neutron L3 High Availability

automation

High Availability and Resiliency Testing Strategies for OpenStack

Providing OpenStack Service High-Availability Through

Clouds

Anycast Routing

Thank You