Virtual Machine Monitors

Bibliography
1.
2.
3.
4.
5.

6.
7.

Virtual Machine Monitors: Current Technology And Future

Trends, Mendel Rosenblum and Tal Garfinkel, IEEE Computer,
May 2005
Xen and the Art of Virtualization, P. Barham, R. Dragovic, K.
Fraser, S. Hand, T. Harris, A Ho, R. Neugebauer, I. Pratt, A.
Warfield, SOSP 03.
The Definitive Guide to the Xen Hypervisor, David Chisnall,
Prentice Hall, 2008.
Scale and Performance in the Denali Isolation Kernel, Andrew
Whitaker, Marianne Shaw, and Steven D. Gribble, in System
Design and Implementation (OSDI), Boston, MA, Dec. 2002.
Denali: Lightweight virtual Machines for Distributed and
Networked Applications, Andrew Whitaker, Marianne Shaw, and
Steven D. Gribble, Proc. USENIX annual Technical Conference,
June 2002.
Xen Homepage: http://www.cl.cam.ac.uk/research/srg/netos/xen/
VMWare: http://www.vmware.com/products/esx/

Outline
Overview
What is a virtual machine?
What is a virtual machine monitor (VMM)?
System or application (process) virtual machines

History of Virtual Machines

Benefits of Virtual Machines
Issues and Implementation
Examples

What is it? (1)

What is virtualization? an abstraction or
simulation of hardware resources
e.g., virtual memory

A virtual machine is an isolated

environment that appears to be a whole
computer, but actually only has access to
a portion of the computers resources.

What is it? (2)

A virtual machine monitor (VMM) is the software
layer that supports one or more virtual machines
Each VM appears to run on bare hardware, giving the
appearance of multiple instances of the same
computer, but all run on a single machine.
VMM is also called a hypervisor

Guest operating system: an operating system

that runs on a VMM rather than directly on the
hardware.

System & Process VMs

http://en.wikipedia.org/wiki/Virtual_machine

System virtual machine (hardware virtual

machine) See previous definitions
Provides a complete system
Each VM can run its own OS, which in turn can run
multiple applications

Process or application virtual machine; e.g., JVM

Runs inside (under the control of) a normal OS
Provides a platform-independent host for a single
application

System Virtual Machines

Traditional: VMM is a thin software layer that
runs directly on the host machine hardware
Main advantage/objective: performance
VMWare ESX, ESXi Servers, Xen, OS370, Denali
Also called a bare metal VMM

Hosted: VMM runs on top of an existing OS.

Main advantage: easier to build; easier to install
Examples: User-mode Linux

Hybrid: shares the hardware with existing OS

Example: VMWare Workstation

VM1

VM2

VM3

Application

Application

Application

Guest OS1

Guest OS2

Guest OS3

Virtual machine layer - VMM

Hardware layer

Traditional VMM

Hybrid

VM1

VM2

Rosenblum & Garfinkel Fig. 2

App
I/O
App App
VMM

Host OS

VMM
Operating system

Guest OS

Hardware layer

VMM

Hardware Layer

Hosted

Hosted/Hybrid versus Non-hosted

VMM
Hosted has 3 advantages [1]
VMM is no harder to install than any other
application
The VMM can use the host OS scheduler,
pager, etc. and focus primarily on isolation
I/O support is better: the VMM can use the
device drivers that are designed to work with
the host OS rather than having to provide its
own.

Hosted versus Non-hosted VMM

Disadvantage [1]
I/O overhead is greatly increased: requests
go from guest OS to VMM to host OS and
down eventually to the device driver.
Too inefficient for servers

More difficult to provide complete isolation,

so not appropriate for servers from a
security perspective.

Hosted v Non-hosted VMM

Conclusion:
Hosting is a good approach for individual work
stations; reduces effort needed to get VMM up
and running.
Hosting is not advisable for servers. Security
issues are the most important concern,
followed by added overhead for I/O.

VM How They Work (1)

VMM runs in kernel mode (replacing
tradtional OS)
Guest OS runs in user mode
Some modern hardware has a third mode for
the guest OS

For the most part, applications run

normally and execute machine code
directly (direct execution)
What about system calls?

VM How They Work (2)

The guest OS runs in user mode how
can it execute privileged code?
It cant. When it tries to execute a
privileged instruction, the VMM traps the
operation, and performs the system call in
place of the guest OS
e.g., when a guest OS appears to execute an
I/O system call, the VMM is actually in charge.

Virtualization versus Emulation

Virtualization presents multiple copies of the
same hardware system.
Direct execution of code on the hardware

Emulation presents a model of another

hardware system
Instructions are emulated in software much
slower than virtualization
Example: Microsofts VirtualPC could run on
other chipsets than the x86 family; used on Mac
hardware until Apple adopted Intel chips

Full Virtualization versus

Paravirtualization
Full virtualization: each virtual machine
runs on an exact copy of the actual
hardware.
Paravirtualization: each virtual machine
runs on a slightly modified copy of the
actual hardware
Because some aspects of the hardware cant
be virtualized (see examples later)
To present a simpler interface; improve
performance.

History - Why VMMs?

Early computers were large (mainframes)
and expensive
VMM approach allowed the machine to be
safely multiplexed among many different
applications
An alternative to multiprogramming

Virtual Machines - History

Early example: the IBM 370
VM/370 is the virtual machine monitor
As each user logs on, a new virtual machine
is created
CMS, a single-user, interactive OS was
commonly run as the OS

Separation of powers:
Virtual machine interacts with user
applications
Virtual machine monitor manages hardware
resources

History 1980s & 1990s

As hardware got cheaper and operating
systems became better equipped to
handle multitasking, the original motivation
went away.
Hardware platforms gradually eliminated
hardware support for virtualization.
And then

History late 90s to today

Massively parallel processors (MPPs) were
developed during the 1990s; they were hard to
program and did not support existing operating
systems
Researchers at Stanford used virtualization to
make MPPs look more like traditional machines
Other research groups explored different
approaches to VMs
Result: today, virtual machines are very common

Example Virtual Machine Systems

VMware: commercial products, derived
from research done at Stanford
Xen: open source, Cambridge University,
widely used in research and academia;
xen.org
Denali: University of Washington, focused
on support for Internet services

VMware
VMware, a publicly held company, founded by
Stanford developers
Two lines of products:
Desktop : a range of products; advertised as a way
for corporations to migrate and upgrade operating
systems from a centralized IT center
VMware ESXi Server is the most recent product in
this line; is a bare-metal hypervisor

Xen
Xen: open-source VM system for x86, Itanium,
ARM & others
Originated at Cambridge University Computer Lab
Now supported as an open-source product that
has destktop, server, and cloud capabilities
(Amazon uses it for its cloud services.)
Designed to support execution of Linux, other
Unix-like systems (Solaris, BSD), Windows
simultaneously on the same platform
Objective of original project: efficient hosting of up
to 100 virtual machines

Denali
Research project U of Washington
Time frame ~ 2001-2004.

Problem addressed: hosting Internet services

economically
Goal: to allow new, untrusted, services to be
hosted on third-party servers.
Protection provided by VM concept lets servers safely
host multiple different services.
Encapsulation lets services be swapped in and out of
memory easily so multiple services can share one
machine

Reasons for Adopting VMMs

Flexibility in choice of operating system
Encapsulation: A VM collects together an
operating system, a complete (virtual)
computer system, and one or more
applications into a single unit that can be
treated like any other software application.
Can be saved to a file, for example

Security and isolation: provided by

encapsulation

Security and Isolation

Applications running on a virtual machine
are more secure than those running
directly on hardware machines.
VMM controls how guest operating systems
use hardware resources; what happens in
one VM doesnt affect any other VM.
OS level security is more vulnerable than VM
security

OS Flexibility

Support several operating systems at the

same time on a single hardware platform
Ability to experiment with new operating
systems, or modifications of existing
systems, while maintaining backward
compatibility with existing systems.

Encapsulation
Conventionally, servers ran on dedicated machines.
Protects against another server/application crashing the OS
But wasteful of hardware resources

VMM technology makes it possible to support multiple

servers, each running on its own VM, on a single
hardware platform
Rosenblum and Garfinkel [1] point out that this makes it
possible to suspend and resume entire virtual machines;
even move to other platforms
For load balancing, system maintenance, etc.

Desirable Qualities
A good VMM
Doesnt require applications to be modified
Doesnt severely affect performance
Is not complex/error prone

Implementation Issues
Virtualize CPU
Guest OS runs as if it is executing directly on
the hardware CPU, but it isnt

Virtualize memory
Guest OS thinks it is managing memory
directly, but it isnt

Paravirtualization versus binary translation

Hardware-assisted virtualization

CPU Virtualization
Basic technique: direct execution
As long as it is executing unprivileged
instructions the virtual machine (guest OS +
applications) executes hardware instructions
directly.
If the guest OS tries to execute a privileged
instruction the CPU traps to the VMM which
executes the privileged operation.

VMM runs in privileged (kernel) mode,

guest OS runs in user mode.

Example: Disable Interrupts [1]

If a guest OS tries to disable interrupts,
the instruction is trapped by the VMM
which makes a note that interrupts are
disabled for that virtual machine
If interrupts arrive for that machine, they
are buffered at the VMM layer until the
guest OS enables interrupts.
Other interrupts are directed to VMs that
have not disabled them.

Direct Execution Not Always

Possible
Modern CPUs, esp. x86 architectures,
have not been designed for virtualization.
Example: POPF (pop CPU flags from
stack)
If executed in user mode, no trap its just
ignored by the hardware
In this case, direct execution fails Guest OS
assumes flags have been popped, but they
havent been because the VMM isnt notified.

Two Ways to Handle Nonvirtualizable Instructions

Paravitualization
Xen, Denali

Binary Translation
VMware

Both use the same basic approach: catch

non-virtualizable instructions and emulate
them in software at the VMM level.

Paravirtualization
Rewrite portions of the guest OS to replace nonvirtualizable instructions with a trap the VMM,
which emulates the instruction on behalf of the
guest OS
e.g., remove POPFs; substitute something else

Paravirtualization affects the guest OS, but not

applications that run on it the API is unchanged
Paravirtualization is also used sometimes to
replace inefficient operations with more efficient
ones.

Binary Translation
Instead of modifying the OS, detect these
instructions at runtime.
VMwares approach: The DBT (dynamic
binary translator) controls execution of
kernel code - replaces non-virtualizable
instructions with equivalent code that can be
virtualized.
Once translated, code is saved and used again
if needed.

Comparison
Paravirtualization changes the source code of a
guest OS; binary translation changes the binary
code as it executes.
Paravirtualization is more efficient, but requires
modification to the guest OS
Paravirtualization also allows more efficient
interfaces, in some cases

Binary translation is backward-compatible but

has some extra overhead of run-time translation
the first time an instruction is encountered.

Hardware-assisted Virtualization
AMD-V and Intel VT are architecture extensions to
support virtualization.
New execution modes
Allows guest OS to run in execution ring 0 and VMM in yet a
higher privileged mode

Flags to indicate if running in this mode

Essentially, the trap and emulate mode used in
paravirtualization or binary translation is now done in
hardware.

Does away with need to modify guest OS; is faster

than binary translation.

Memory Virtualization
VMM maintains a shadow page table for
each virtual machine.
When the guest OS makes an entry in its
own page table, the VMM makes the
same entry in the shadow table.
Shadow page table points to actual page
frame
The hardware MMU uses the shadow page
table when it translates virtual addresses.

Challenges
Let the guest OS decide which of its pages to
swap out
VMwares ESX Server uses the concept of a
balloon process, running inside the guest OS [1].
When the VMM wants to swap out pages from a
VM it notifies the balloon process to allocate
more memory to itself.
The guest OS must page out unused portions
of other processes to its virtual disk.
The VMM now knows which pages the guest OS
thinks it can do without.

Other Virtual Memory Challenges

To share or not to share pages across VM
boundaries:
VMware tracks duplicate pages in different
virtual machines & stores only one copy of the
actual page with pointers from the shadow
page tables in sharing processes.
Copy-on-write policy

Xen focuses on total isolation of each

virtual machine, which means no sharing

Summary & Review (1)

A virtual machine is a copy of a real machine
Applications dont know if they are running on real or
virtual hardware, other than having fewer resources.

A virtual machine is isolated: if several VMs

execute on the same hardware they do not
interact with each other directly or indirectly.
The performance of a virtual machine should be
about the same as that of the actual hardware.
So most instructions should be directly executed by
the hardware as opposed to being emulated.

Summary and Review (2)

Process virtual machines (JVM) virtualize at a
higher level, do not necessarily even correspond
to real machines.
System virtual machines virtualize at the level of
the hardware-software interface
Variations of classic system virtual machine:
Hosted (run on another operating system
Emulation (provides virtual hardware and OS, as in
Virtual PC) not really a virtual machine

Summary & Review (3)

Virtual Machine Monitor (hypervisor) runs on a bare
machine, implements one or more virtual machines.
The VMM allocates resources and controls resource
sharing among all VMs
Operation:

Each VM runs a guest OS

VMM runs in kernel mode
Guest OS and applications run in user mode
Privileged instructions trap to the VMM
Hypercalls (the VMM equivalent of system calls) may be used by
a guest OS to request service from the VMM

Summary & Review (4)

Benefits of VM technology for non-hosted VMs
Isolation and security
Multiple servers on a single machine

Encapsulation of an entire environment: OS and

application for the purpose of
Migration
Checkpointing
Supporting system maintenance

Running several OSs concurrently

Older versions, experimental systems, Linux & Windows,

For hosted VMs, the major advantage is the

ability to run two or more OSs at once

Appendix Examples
Xen
Denali
Hardware Virtual Machines

Xen Intro
Claim: virtualization is better than multitasking as a way to share hardware.
CPU requests, memory demand, disk
accesses, other resource needs of one
process impact the performance of other
processes
Xen solution: multiplex resources at the OS
level instead of the process level.

VM1

Domain 0
Guest

VM2
Application

VM3
Application

Domain U
Guest OS2

Domain U
Guest OS3

Xen
Hardware layer

Xen implementation of VMM

Domain 0 guest
has privileged
access to the
Xen hypervisor
and can be used
by the system
administrator to
manage the
system.
Separation of
powers
Xen only has to
worry about
multiplexing
hardware to
multiple guests

Xen Design Principles

Virtualize all architecture features that are
required by standard binary interfaces.
To support existing applications without
modification

Support multi-application guest operating

systems
Use paravirtualization to get improved
performance and resource isolation

Xen HVM (Hardware Virtual

Machine)
Some versions of Xen are designed to run
on Intel VT and AMD-V chips with special
virtualizing hardware.
Able to run un-modified (no paravirtualization) operating systems. This
implementation is known as a hardware
virtual machine.
Windows requires an HVM environment;
Linux, Solaris, and BSD systems dont.

Xen Memory Management

Unlike VMWare and Denali, Xen expects the
guest OSs to manage their own hardware page
tables.
To support this, each VM receives a fixed
allocation of page frames which it can use as it
wishes.
New page tables must be registered with Xen
and updates must be validated by Xen.
Make the page table write protected.

Xen CPU Management

Xen is designed for the X86 architecture which
supports 4 rings, or privilege levels.
Traditional OSs execute in ring 0 (most privileged)
and applications in ring 3 (least)
Xen executes in ring 0 (only level that can execute
privileged instructions)
Guest OS runs in ring 1, which isolates it from
applications.
Note: since this paper was written there have been
some modifications to X86 to better support
virtualization.

Xen CPU Management

Privileged instructions must be validated
(is it OK?) and executed by Xen
Exceptions (page faults, system calls,
other traps to OS) are handled as much as
possible by the guest OS.
Exception handlers are registered & validated
with Xen
System calls stop at the guest OS; Xen is
involved only if the OS executes a privileged
instruction.

Denali Isolation Kernel

Authors define Denali as a small-kernel
operating system with similarities to
microkernels and exokernels
Once thought to be inefficient, modern
hardware has improved performance of this
kernel architecture

They expected Denali to support multiple

(up to 10,000) untrusted applications that
are virtually independent.

Isolation Kernel Design Principles

Expose low-level resources rather than
high-level abstractions for greater security
Avoid layer-below attacks

Prevent direct sharing by exposing only

private, virtualized namespaces
Keeps one VM from even naming the
resources of another VM, let alone modifying
them. [4]

Isolation Kernel Design Principles

Design for scalability
Be able to support a work load that has a few popular
services and many that are accessed infrequently.

Modify the virtualized architecture for simplicity,

scale and performance.
Paravirtualization for reasons other than necessity.
They do not believe isolation depends on providing an
exact copy of hardware so they provide a hardware
version that is modified to be more efficient and
secure.

Zipfs Law
Given a table that ranks something on the basis
of its frequency of occurrence, Zipfs law states
that the most frequent item occurs about twice
as often as the next most frequent item, which in
turn occurs twice as often as the next item, and
so on.
Zipf made this observation about words in a
natural language. Here, were talking about
accesses to various web services.

Statistically Multiplexing Services

Studies showed that the popularity of most
network services (server requests, document
searches, etc) followed a Zipfian distribution.
Implications:
Most requests go to a small number of services
Most services arent popular, but the total number of
requests for unpopular services is non-trivial
With isolation it can be safe and efficient to run
hundreds or even thousands of services concurrently
on a single platform.

Proof-of-concept
Denali is the virtualized architecture
Yakima: a VMM which was designed to run in
ring 0 on x86 hardware.
Ilwaco: a simple prototype guest OS which
provides a full set of abstractions to its
applications while hiding the Denali architecture
Reasonable performance in tests
1.4 sec to 9 sec context switch time, depending on
number of VMs
End-to-end run times of network apps were
comparable to those of a traditional operating
system.

Conclusion
The Denali research project terminated in
the mid-2000s.
The Denali research group was right in
supposing that virtual machine technology
would be most useful today to enable
efficient use of server hardware.
Multi-core computing the MPP of the
future? How useful will VMM concepts be?