Building High Availability

in OpenStack
Created by Sergii Golovatiuk

Logical architecture
Cloud
management

FUEL

Horizon

(Provisioning and deployment)

(Self-service Web UI/Dashboard)

Logical architecture
Cloud
management

Core IaaS

FUEL

Horizon

(Provisioning and deployment)

(Self-service Web UI/Dashboard)

Nova

Cinder

Neutron

Keystone

(Compute)

(Block Storage)

(Networking)

(Identity)

Swift

Glance

Ceilometer

(Object Storage)

(Image Mgmt)

(Telemetry)

Logical architecture
Cloud
management

Core IaaS

PaaS elements

FUEL

Horizon

(Provisioning and deployment)

(Self-service Web UI/Dashboard)

Nova

Cinder

Neutron

Keystone

(Compute)

(Block Storage)

(Networking)

(Identity)

Swift

Glance

Ceilometer

(Object Storage)

(Image Mgmt)

(Telemetry)

Heat

Murano

Sahara

(Orchestration)

(App catalogue)

(Data processing)

Physical architecture

FUEL API

FUEL master
node

Physical architecture
Load Balancer (HAproxy)

FUEL API

FUEL master
node

OpenStack
APIs

OpenStack
APIs

OpenStack
APIs

Controller
node 1

Controller
node 2

Controller
node 3

OS: Linux (CentOS or Ubuntu)

HA setup (HAproxy, Galera, Pacemaker)

Physical architecture
Load Balancer (HAproxy)
OpenStack
APIs

OpenStack
APIs

OpenStack
APIs

Controller
node 1

Controller
node 2

Controller
node 3

Compute
node 1

Compute
node 2

Compute
node 3

FUEL API

FUEL master
node

OS: Linux (CentOS or Ubuntu)

HA setup (HAproxy, Galera, Pacemaker)

OS: Linux (CentOS or Ubuntu)

Hypervisor: KVM

Physical architecture
Load Balancer (HAproxy)
OpenStack
APIs

OpenStack
APIs

OpenStack
APIs

Controller
node 1

Controller
node 2

Controller
node 3

Compute
node 1

Compute
node 2

Compute
node 3

FUEL API

FUEL master
node

Storage
node 1

Storage
node 2

Storage
node 3

OS: Linux (CentOS or Ubuntu)

HA setup (HAproxy, Galera, Pacemaker)

OS: Linux (CentOS or Ubuntu)

Hypervisor: KVM

OS: Linux (CentOS or Ubuntu)

Storage backend: LVM or Ceph

OpenStack High Availability Stack

HA Management - Corosync/Pacemaker
Networking HA
Database - MySQL
AMQP - RabbitMQ
API Services
Cache - Memcached
Storage - Ceph

HA Management - Corosync/Pacemaker

Yes, its complex

HA for Network Connectivity

Link Aggregations:
Round Robin
Active-Passive
XOR
LACP (my favorite)
- Requires switch configuration

Database HA - MySQL / Galera

MySQL 5.6 with patches from Codership
xtrabackup from Percona
HAProxy + xinetd httpcheck

MySQL/Galera - OCF Script

Use Latest GTID info for PC election
from CIB
from grastate.dat

Start PC with empty gcomm://

Clone based

Messaging HA - RabbitMQ
Hard to reassemble RabbitMQ cluster
Each Node tries to connect to previous
queue master

AMQP - OSLO.MESSAGING
multiple rabbit connection included
OSLO.messaging heartbeats

RabbitMQ - OCF Script

API Endpoint Load Balancing

HAProxy Based
Stateless (Active-Active)

Cache HA - Memcached
Keystone stores tokens in memcached
Horizon keeps sessions in memcached

Do we really need HA for memcached?

Storage HA - Ceph
ephemeral storage = live migration
object and image storage shared
internal HA mechanism based on PAXOS

Main Deployment Concepts

Automate! Do not even try to do it manually
based on our pacemaker puppet resource

types/providers for corosync

error handling (retries)
patching based
asymmetric cluster

Testing High Availability

HA testing framework
integrated with CI
destructive tests
performance testing during failure

Results

Controller restart
network partitioning or port failure
DB node failure handling
AMQP node failure handling
API Endpoint service failure handling
Storage node failure handling

Open Issues
High Availability for Neutron L3 Agents:
Virtual Routers (in Fuel 6.1)