Open Source Cloud Management Stacks Comparison:

Eucalyptus Vs. OpenStack

August 10, 2011
1 Introduction
As cloud computing technology matures, many commercial companies and online com-
munities are developing new and interesting concepts for managing the underlying cloud
stacks. This survey tries to compare between two widely used open-sourced management
stacks, and assess the current state-of-the-art state in terms of resource-management, stor-
age overview, scalability and extendability. Cloud providers as well as medium-large busi-
nesses use these Cloud Management Stacks (CMS) to govern data-base infrastructures and
oer not only private cloud service, but also public or hybrid cloud services. Due to the
nature of open-source projects, survival is achieved by popularity. Compatibility to var-
ious VMM platforms (e.g. Xen, KVM, VMware) is key, as these platforms dictates not
only the build and maintenance cost factors but also the features they will enable cost-
saving services and techniques (e.g. smart resource provisioning, high-availability, SLA
monitoring, etc). Table 1.1 states the list of hypervisors each platform currently supports.
As private cloud IaaS features are the basics for every cloud operator, public APIs set a
new and sometimes contradicting set of requirements as Service Level Agreements (SLA)
usually penalise cloud providers in cases of unsatisfactorily performance or low rates of
availability. Thus, the ability to extend the CMS with a business oriented layer is often a
necessity. Next, we briey describe each of the open-source IaaS management stacks, in
terms of high level design and focuses on the key dierences between them. Follows, is
a more in-depth look at the strategies each of the platforms holds in regards to resource
management, storage perception, scalability concerns and extendability development.
2 Architectures Overview
2.1 Eucaliptus
Being a relatively mature player in the market (4 years), Eucalyptus [3] backed by Euca-
lyptus Systems, Inc oers the benet of a relatively open source infrastructure for managing
1
Eucalyptus OpenStack
Xen yes yes
KVM yes yes
VMware ESX/ESXi Only in EE yes
Hyper-V no yes
Table 1.1: Hypervisors support matrix
private/public cloud. One benet of Eucalyptus is that its open source software compo-
nents are used without modication, meaning that they can run on unmodied GNU Linux
kernels with relative ease. Ubuntus baked-in cloud computing edition (Ubuntu Cloud) was
Eucalyptus-based (Until changed to OpenStack in recent versions) and ready to be installed
right after download, making it very convenient. Another major benet is its API compat-
ibility with one of the largest cloud provider market players, Amazon and its EC2 platform
(as well as partial support for S3). That means that a company evaluating EC2 can use
freely available software on freely available operating systems to build a compatible test
lab. That same company, once they are an Amazon customer, can then use Eucalyptus for
development work before pushing to the ever-live world of the real cloud.
A Eucalyptus cloud setup [7, 5] consists of ve types of components. The cloud con-
troller (CLC) and Walrus are top-level components, with one of each in a cloud instal-
lation. The cloud controller is a Java program that oers EC2-compatible WS (SOAP
based) interfaces, as well as a Web interface to the outside world. In addition to handling
incoming requests, the cloud controller performs high-level resource scheduling and system
accounting. Walrus, also written in Java, implements bucket-based storage, which is avail-
able outside and inside a cloud through S3-compatible WS (SOAP) and HTTP (REST)
interfaces.
Top-level components can aggregate resources from multiple clusters (i.e., collections
of nodes sharing a LAN segment, possibly residing behind a rewall). Each cluster needs
a cluster controller (CC) for cluster-level scheduling and network control and a storage
controller (SC) for EBS-style storage [4]. The two cluster-level components would typically
be deployed on the head-node of a cluster. Finally, every node with a hypervisor (Xen,
KVM, etc) will need a node controller (NC) for controlling the hypervisor. CC and NC
are written in C and deployed as Web services inside Apache; the SC is written in Java.
Communication among these components takes place over SOAP with WS-security.
2
Figure 2.1: Eucalyptus High Level Design
It is important to note that while Eucalyptus has an open-source version, it is not 100%
open. A few code modules remain close, and variations of them are in use in the open-source
and the enterprise version of Eucalyptus. Some of them, (e.g. resource management) are
the reason why Eucalyptus is considered limited in scalability. In fact, NASA which at
time, supported Eucalyptus publicly, pulled out from the project (In favor of OpenStack)
because of these very issue.
2.2 OpenStack
Backed by Rackspace and NASA, OpenStack is a collection of open source technologies
written in Python delivering a massively scalable cloud operating system. It currently
encompasses three main projects:
Nova which provides virtual servers upon demand. This is similar to Rackspaces
Cloud Servers or Amazon EC2.
Swift which provides object/blob storage. This is roughly analogous to Rackspace
Cloud Files (from which it is derived) or Amazon S3.
Glance which provides discovery, storage and retrieval of virtual machine images for
OpenStack Nova.
While these three projects provide the core of the cloud infrastructure, OpenStack is
open and evolving constantly, and more projects are lling missing features as time goes
by. In this survey we focus mainly on the architecture of Nova [6] which is the heart of
the OpenStack platform (which can be extended by the Swift, Glance and others). Nova
consists of seven main components, with the Cloud Controller component representing
the global state and interacting with all other components. API Server acts as the Web
3
Services front end for the cloud controller. Compute Controller provides compute server re-
sources, and the Object Store component provides storage services. Auth Manager provides
authentication and authorization services. Volume Controller provides fast and permanent
block-level storage for the compute servers. Network Controller provides virtual networks
to enable compute servers to interact with each other and with the public network. Sched-
uler selects the most suitable compute controller to host an instance. Nova is built on
a shared-nothing, messaging-based architecture. All of the major components, that is
Compute Controller, Volume Controller, Network Controller, and Object Store can be run
on multiple servers. Cloud Controller communicates with Object Store via HTTP, but
it communicates with Scheduler, Network Controller, and Volume Controller via AMQP
(Advanced Message Queue Protocol). To avoid blocking each component while waiting
for a response, Nova uses asynchronous calls, with a call-back that gets triggered when a
response is received. To achieve the shared-nothing property with multiple copies of the
same component, Nova keeps all the cloud system state in a distributed data store. Up-
dates to system state are written into this store, using atomic transactions when required.
Requests for system state are read out of this store. In limited cases, the read results are
cached within controllers for short periods of time (for example, the current list of system
users).
Figure 2.2: OpenStack Nova High Level Design
4
3 Resources
An important aspect of any cloud infrastructure is the resources granted to a VM. Each VM
deployed in the cloud must be isolated from other VMs on the system in terms of their CPU
access, memory access, network access, and access to persistent storage. A dened Service
Level Agreements (SLAs) or Quality-of-Service (QoS) specications categorize the quality
and guarantee of resources. Farthermore, cloud resource allocations are on-demand
(resources are returned to the user when they are requested without queuing delay) and
atomic (an entire resource allocation request is typically satised without the possibility of
partial allocation failure). Many design and implementation issues are relevant and have
an eect on the infrastructure utilization, user customization and exibility:
Guarantees - Reservation and limit are xed resources guarantees that specify the
minimum and maximum amount a VM may consume. Relative resources such as
shares set VMs priority and in case of resources contention specify how to divide the
hosts resource between the VMs.
Overcommit - Assuming that the VMs do not always use all the resources assigned
to them, overcommmit approach exploit it to better utilize hardware by hosting
more resources than physically available but requires a consistent monitoring and
predication models to keep the guarantees valid.
Flexibility - VM resources may be oered to the users as xed, i.e. predened VM
sizes the user can choose from in terms of cpu, memory, etc. or allow a higher degree
of freedom and customization in which the user can specify his specic resources
requirement. With predened instances types, the user is not free to allocate the
combination of resources that best t his application, but force to choose a specic
resources setup. For cloud providers, this simplies the scheduling and resource
allocation guarantees enforcement. Another exibility issue is the ability to modify
resources dynamically, e.g. increasing memory on the y for a specic VM (i.e.
vertical elasticity).
EC2 parlance exposes pre-dened instances type (called avors in Rackspace) that
determine the size of the instance in terms of resources, used by both OpenStack and
Eucalyptus (See Table 3.1)
5
Type Memory [MB] CPU [cores] Disk [GB]
tiny 512 1 0
small 2048 1 20
medium 4096 2 40
large 8192 4 80
xlarge 16384 8 160
Table 3.1: Supported instance types in Eucalyptus/OpenStack
Both compared implementations do not implement EC2s Modify-Instance-Attribute
that supports exible resources (change the instance type after VM launch dynamically).
4 Storage & Image support
Images repository and images query service are important aspect of a cloud implementation.
Ecient and scalable images management can save a lot of disk space, increase reliability,
allow quick instances deployment, customized images and share in a secured way images
among users and groups. Both Eucalyptus and openstack has their own images store
service and can use any S3 compatible implementation. Images are a triplet of virtual disk
image(s), kernel and ramdisk images as well as an xml le containing meta data about
the image. Eucalyptus includes Walrus, a data storage controller service that its interface
is compatible with Amazons Simple Storage Service (S3) and provide a mechanism for
storing and accessing virtual machine images and user data.Walrus provides two types of
functionality:
Stream data into/out of the cloud as well as from instances.
Storage service for VM images.
Walrus controller compresses the images, encrypt them using user credentials, and split
them into multiple parts that are described in a image description le (called the manifest in
EC2 parlance). Walrus is entrusted with the task of verifying and decrypting images that
have been uploaded by users. When a node controller (NC) requests an image from Walrus
before instantiating it on a node, it sends an image download request that is authenticated
using an internal set of credentials. Then, images are veried and decrypted, and nally
transferred. As a performance optimization, and because VM images are often quite large,
Walrus maintains a cache of images that have already been decrypted.
In OpenStack, there are two methods for managing images. Images can be served
through the OpenStack Image Service, a project that is named Glance that provides discov-
ery, storage and retrieval of images meta data, or directly from a nova-objectstore service.
The image store can be a number of dierent object stores, including OpenStack Swift
6
(provides object/blob storage, long-term storage tool that provides reliability and avail-
ability through redundancy, analogous to Rackspace Cloud Files or Amazon S3). With
an OpenStack Image Service server in place, the Image Service fetches the image on to
the host machine and then OpenStack Compute boots the image from the host machine.
Openstack compute relies on using the euca2ools command-line tools distributed by the
Eucalyptus Team for adding, bundling, and deleting images.
Clients can register new virtual disk images with the Image Service, query for infor-
mation on publicly available disk images, and use the Image Services client library for
streaming virtual disk images. Being A multi-format image registry, OpenStack Image
Service allows uploads of private and public images in a variety of formats, including:
Raw
Machine (kernel/ramdisk outside of image, a.k.a. AMI)
VHD (Hyper-V), also VHD images that include customer data and kernel in one
unied image
VDI (VirtualBox)
qcow2 (Qemu/KVM)
VMDK (VMWare)
OVF (VMWare, others)
While Eucalyptuss Walrus acts both as an VM image storage and as the image man-
agement service, those two roles were divided into expendable two separate building block
components (Glance and Swift) in OpenStack that provides higher degree of implementa-
tion freedom and scalability.
Openstack notates images as private or public only, there is no way to share an im-
age between a dened users group while Eucalyptus imposes its groups security policy
along all its objects including images. Images are not used by several Virtual Machines
simultaneously (ramdisk and kernel disks may).
The two open source clouds have many future plans related to images store such as
adding image replicas, ability to share images between specic users/groups, etc. That
would enhance image store & image services.
5 Scheduling
Resource scheduling is the process of nding the physical machine in which a new virtual
machine will be hosted. The physical machine (AKA host) must have enough free resource
capacity (i.e. vCPU, Memory, local-storage, network bandwidth, etc.) to allow the new
7
VM (and others which are already hosted on it) to perform within the performance bound-
aries specied by their Service Level Agreements (SLA). Cloud providers usually also have
some kind of a global utilization function which tries to place the hosted VMs in such
location which will benet a certain cost aspect (e.g. physical machines consolidation, load
balancing, etc.). Obviously, these kind of problems are usually NP-Hard problems, which
means that nding an optimal solution is an extremely lengthy procedure (impractical for
most cloud scales), and so a good cloud management stack usually balance between the
quality of the solution, and the time it takes nding it. This is one of the areas where the
design dierence between Eucalyptus and OpenStack may very well be a decisive factor
when one come to choose the CMS of his choice.
As Eucalypsus Cloud Controller (CLC) is stateful, it can choose the one of the Cluster
Controller (CC) either by round-robin, or the one with the largest total free capacity, to
delegate the new VM assignment. The CC can chose whether to comply to the request.
If the CC decides the request is legit, it assigns the VM to the node with the largest free
capacity using a two-face commit mechanism, rst the CC reserves the resources, next the
resources are committed after the NC complete provisioning the VM.
As one can see this heuristic is quite simplied, and may very well result in under-
utilized hardware. While it is possible to replace the scheduling code, this must be done,
both in the cluster level (CC), and also in the system-wide level (CLC). Also, currently it
is impossible for cluster to communicate with each other, and so VM migration between
clusters is complicated and demands the CLC involvement. Thus, any solution which tries
to use VM migration (i.e. existing placement optimization) must be centric, as it must be
implemented completely in the CLC.
While OpenStack does oer a similar set of schedulers to chose from (chance scheduler
- randomly selects a host, zone scheduler - pick a random host which is up in a specic
availability zone, simple scheduler - pick the least loaded host), it also adds an additional
two for cases of large scale which are almost completely distributed. When a request for VM
instantiation arrives, the central scheduler module asks all availability zones (recursively)
to assess the cost of assigning the VM for each node which resides within them. This cost
is currently eected by the amount of free space left after the assignment, and additional
optional dynamic weight. The results are then sorted and the most appropriate candidate
is chosen.
As this is a completely distributed model, theres no central bottleneck on the perfor-
mance of the scheduler. Also, other heuristics can be easily deployed by changing the
VM-Node cost model, and by dynamically changing the weights to prefer certain hosts as
time goes by (e.g. preferring the most reliable machines, or those with the lowest applica-
tion loads, etc).
8
6 Extendability
While Eucalyptus is built as an hierarchical architecture, aiming to ease researchers and
developers to replace certain modules with their own versions [1, 2], the current releases
dont support a plug-in extensions. This means, that modules re-use and distribution is
dicult and complex, and sometimes even impossible (due to the fact that Eucalyptus is
not 100% open-source). Also, since Eucalyptus can be considered as an EC2-extension to
the private cloud (implementing EC2 & S3 APIs), the development team is no susceptible
to add support for new features in hypervisors (e.g. Live-Migration). Adding manual
support for such features is considered hard, since the state is distributed among various
components (e.g. network, EBS, public IPs, security group).
Its hierarchical design has also practical implications into the feasibility of features,
and/or their performance. For example, since the Cloud Controller acts as a single aggre-
gation point of resource allocation and instance provisioning it is in fact a bottle-neck for
any smart-placement algorithm (being of centric design). Even in that in mind, the actual
instance allocation is ultimately in the hands of the Cluster Controller, which operates a
very simplistic allocation algorithm (First-Fit) which can override the Cloud Controller
placement algorithm. The core problem is Eucalyptus design as components centric Vs
services oriented. Instead of helping the creation of new services, a predetermined hier-
archical design constrain the behavioral ow and thus burdens the extendability of this
model.
To tackle extendability issues OpenStack implements several APIs which form a two-
dimension communication multi-path. On the one hand user/developer-oriented APIs
(EC2, Rackspace APIs) and on the other hand extendability APIs to bridge between
OpenStack extensions. The main design motto in OpenStack is share-nothing. This
approach comes to reality in the execution ow of requests (i.e. star-design), and com-
munication between the dierent modules. The assumption is that a component simply
advises the central execution unit (i.e. Cloud Controller), thus it is relatively easy to
replace a module since it has virtually no dependent modules. For example, it is easy to
replace the default Scheduler module, since it does not change the state of other modules.
Its high cohesion and the fact that it is decoupled from the complexity of other modules
make the job of writing and debugging such a component much easier.
7 Discussion
Weve surveyed topics concerning system design and there implications in each of the sys-
tems. While certainly theres some technology advantage to OpenStack, it is important to
note that while OpenStack clearly claims to be designed for every purpose clouds, Euca-
lyptus on the other hand, started its way, and in a sense still is, a research or sand-box
oriented cloud; Designed to enable quick development and testing in lab conditions,
before deploying to EC2. While the open-source version of Eucalyptus has remained un-
9
changed, in that regard, the Enterprise Edition includes several advanced features (e.g.
quota and accounting management, scalable DB supprt, windows guests, etc).
While in both systems, setup is easy right out-of-the-box, the amount of work, one needs
to put in-order to customize the dierent modules is dependant on the type of research,
algorithms and scale of their work.
References
[1] Rick Bradshaw and Piotr T. Zbiegiel. Experiences with eucalyptus: deploying an open
source cloud. In Proceedings of the 24th international conference on Large installation
system administration, pages 116. USENIX Association, 2010.
[2] E. Caron, F. Desprez, D. Loureiro, and A. Muresan. Cloud computing resource man-
agement through a grid middleware: a case study with diet and eucalyptus. In IEEE
International Conference on Cloud Computing (CLOUD), pages 151154. IEEE, 2009.
[3] Eucalyptus open source edition. http://open.eucalyptus.com.
[4] Amazon Inc. Elastic Block Storage. http://aws.amazon.com/ebs/.
[5] Eucalyptus Inc. Eucalyptus online guide. http://open.eucalyptus.com/book/
export/html/1398.
[6] OpenStack Nova. http://nova.openstack.org.
[7] D. Nurmi, R. Wolski, C. Grzegorczyk, G. Obertelli, S. Soman, L. Youse, and
D. Zagorodnov. The Eucalyptus open-source cloud-computing system. In IEEE/ACM
International Symposium on Cluster Computing and the Grid (CCGrid), pages 124
131. IEEE, 2009.
10