How Does Open Source VMM Helps To

Reduce The Risk Of HVM Rootkits That Can

Effect Your Cloud Computing And Other VM
Deployments
Azril Azam Abdul Rahim, GCFA, CEI, ECSP, CEH
Malaysia Open Source Conference 2010
July 1st 2010
Agenda

•  Introductions
•  Cloud computing and virtualization
•  Virtualization 101
•  HVM Rootkits
•  Demo: From subverting an OS to subverting a VMM
•  Reduced The Risk Via Open Source VMM
•  Conclusion
•  References

2
Introduction
Introduction
About Azril Azam, GCFA, CEI, ECSP, CEH
Currently working as Team Leader for Malware System Development and Analysis at the Global Response
Center – International Multilateral Partnership Against Cyber Threats (IMPACT)

An average computer hardware and software programmer with a huge interest

Computer Security (Penetration Testing, Trusted Computing, Computer/Network Forensics)
System Kernel and Virtualization
Open Source Software

Has written and published many research papers, patent disclosures and open source software on his
interest research areas, and has won a few awards
- 2006 Malaysia Best Open Source Software – Computer Forensics
- 2006 ITEX Gold Award for Innovation – FIRST
- 2006 Malaysia Minister Of Science Research Appreciation Recipient Award

Participated on global scale on virtualization research work

XEN and KVM development groups, EMCSB and OpenTC.

Currently, he is in the midst of completing his own low overhead x86 virtualization system that is based on
his 2007 linux ottawa symposium virtualization summit extended abstarct (at his spare time of course)

Published his OSS through his own domain @ diligentcode.com

4
Introduction
About IMPACT
Non-Profit

International Upper End of

Team of Cyber
Experts Threats

IMPACT

Global
Public- International
Private & Multilateral
Partnership

5
Introduction
Launch of IMPACT
1.  IMPACT was launched on May 2008 by the 4th Prime Minister of Malaysia, Tun Abdullah Ahmad Badawi
2.  IMPACT’s Global Head Quarters in Cyberjaya, Malaysia was launched on 20 March 2009
3.  Event was witnessed by the Secretary General of ITU, Dr. Hamadoun Touré

6
Introduction
IMPACT partnership with UN/ITU

  IMPACT HQ - physical home for ITU’s Global Cybersecurity Agenda (GCA)

  IMPACT to operationalise UN/ITU’s global cybersecurity initiatives to all 191
countries
  GCA – framework for international cooperation to enhance confidence and security
in the information society

7
Introduction
The Global Cybersecurity Agenda

8
Introduction
Terminologies & Color Codes Guidelines
HAL: Hardware Abstraction Layer
RING  3   VM: stands for virtual machine. A collection of
(User  space  /  ApplicaBons)   emulated devices that works together to give a fake
HAL view for OS to execute without calling the
RING  1  -­‐  3   actual devices
(Drivers,  core  libraries)  
VMM: is an entity that controls a VM by allocating
resources and managing memory
RING  0  
(Kernel  Level)   Hypervisor: is a special VMM that provides
emulated devices

RING  -­‐1   Host OS: is an OS or Kernel that can call privilege

(Hardware  Level)   instructions and talk to the actual devices within
virtualized or non virtualized environment

Guest OS: is an OS or Kernel within virtualized

environment that has been de-privilege

9
Cloud Computing And
Virtualization
Cloud Computing & Virtualization
What is Cloud Computing
Cloud Computing is actually an Internet Based Computing services similar to hosting service, server co-
located and server farming.

The different between cloud computing and traditional internet based computing are for

Vendor
- Managed all services with different OS requirement all in 1 server instead of many.
- Reduced cost
- Easy to managed

Customer
- Isolated and scalable resources
- Cheaper solution

Cloud Computing Logical Diagram

11
Cloud Computing & Virtualization
Cloud Computing Layers

Client  

ApplicaBon   Google Earth API, Google Map API

SoFware  As  A  Service  (SAAS)  

PlaHorm   Data Storage, Bandwidth, Processing Power

PlaHorm  As  A  Service  (PAAS)  

Infrastructure   Virtualization
Infrastructure  As  A  Service  (IAAS)  

Server  

12
Virtualization 101
 Virtualization 101
The Fundamental: Von Nuemann Architecture

Memory (RAM)

Input
Arithmetic Logic
Device
Unit
Control
Unit Accumulator
Output
Device
CPU

Von Nuemann Architecture

Von Nuemann Architecture is the foundation on modern computer that utilizing Alan Turing “The Turing
Machine” binary concept

14
 Virtualization 101
The Fundamental: Von Nueman Architecture

VM
Memory (RAM)
Applications

Input
Arithmetic Logic
Unit
Device Guest OS
Control
Unit Accumulator
Output
Virtual Virtual Virtual
Device RAM I/O CPU
CPU

Von Nuemann Architecture

•  To create a virtual machine, all components of Von Nuemann Arch must be emulated
•  Compare to other CPU architecture, x86 is the hardest to be virtualized
•  Therefore, to achieve virtualization state, all components are emulated by software codes

15
Virtualization 101
Earlier Implementation
•  Early implementation on x86 virtualization runs as VM
an application on top of underlying OS
Applications
•  This approach seems to be inefficient due to
Guest OS
- Sharing resources with other apps
- Unable to schedule priority for emulations
Virtual Virtual Virtual
RAM I/O CPU
User Space Applications (Word, Excel)

Virtual Memory (STACK, HEAP, BSS) 4GB Virtual Memory (STACK, HEAP, BSS) 4GB

Operating System

Memory (RAM)

Input
Arithmetic Logic
Device
Unit Earlier VMM is just simply a 32bit
Control
Unit Accumulator protected memory managed by OS and
programmed by using OS API running
Output
at RING 3 with no HAL
Device
CPU

16
Virtualization 101
Earlier Implementation
•  More efforts were made to make CPU emulation more VM
by bringing it closer to actual CPU
Applications
•  The VMM must be a part of the OS/kernel or running by
Guest OS
itself exactly at RING 0 for better HAL
Virtual Virtual Virtual

User Space Applications (Word, Excel)

RAM I/O CPU
✔
Virtual Memory (STACK, HEAP, BSS) 4GB Virtual Memory (STACK, HEAP, BSS) 4GB

Virtual
Operating System CPU

Memory (RAM)

Input
Arithmetic Logic
Device
Unit
Control
Unit Accumulator
Output
Device
CPU

17
Virtualization 101
A new concept of VMM
•  A new concept called virtual machine monitor (VMM) VM 1 VM 2
called hypervisor was introduced
Applications Applications
•  In the VMM/hypervisor concept, VMM runs as the
same level of OS/kernel at RING 0. This will allow
Guest OS Guest OS
the VMM to control the HW directly
Virt Virt Virt Virt Virt Virt
•  VMM runs on its on real memory instead on RAM I/O CPU CPU RAM I/O
protected memory previously

•  VMM controls the VM resources and trapped CPU Hypervisor / VMM

instructions calling. All non-privilege instructions are
reroute and execute by the actual CPU. The privilege TRAP ALL CPU
INSTRUCTIONS
instructions are emulated.

•  Increase performance almost to near native speed YES

Privilege
•  There 2 types of architecture Instructions
- Para virtualization
- Full virtualization NO
- RING 0 (Software VMM)
- RING -1 (HW VMM)
ACTUAL CPU

18
HVM Rootkits
Para virtualization
VM Domain 0 VM Guest Domain
•  VMM is now running at RING 0

Host Applications
•  Host OS runs in special Domain with
Apps
Direct interface with the Hardware
Device  
Modified
drivers  
Guest OS
•  Guest Domain can make calls to VMM Modified
via using VMM Instruction Set. Therefore Host OS
no legacy OS can run in this setup.

•  VMM then decides whether VMM API calling Hypervisor / VMM

are privilege or otherwise
TRAP ALL CPU
INSTRUCTIONS
Virtual Virtual
•  Example RAM I/O
BASED ON VMM
INS
- XEN
- MICROSOFT HYPER-V
YES Privilege
Virtual Instructions
CPU

NO

HW CPU

19
HVM Rootkits
Full Virtualization: Software VMM
•  VMM is now running at RING 0 VM 1 VM 2

Applications Applications
•  VMM runs as drivers for Host OS

•  This setup allows legacy OS Guest OS Guest OS

without modifications
Host Apps
Virt Virt Virt Virt Virt Virt
RAM I/O CPU CPU RAM I/O
•  Example
- VMWARE
- Qemu with KQemu Hypervisor / VMM
- VirtualBox*
TRAP ALL CPU
Device  drivers   INSTRUCTIONS

Host OS YES
Privilege
Instructions

NO

ACTUAL CPU

20
HVM Rootkits
Full Virtualization: Hardware VMM VMX Non-root

•  VMM is now running at RING -1 by the CPU Apps

itself
VMX Root Guest OS
•  Similar like XEN, HVM will create a special
privilege domain for host software and
Host OS or
unprivileged domain for guest OS minimalistic
Virtual Virtual Virtual
CPU I/O RAM
kernel
•  Host OS provides devices drivers
Device  drivers   HAL
•  HAL layer provides the calling of the
actual machine device drivers
HVM VMM
•  This setup allows legacy OS without TRAP ALL CPU
modifications INSTRUCTIONS

•  Example
- KVM
YES
- VMWARE 64bits Privilege
Instructions
- XEN-CITRIX HVM extension*

NO: Execute
HW CPU

21
HVM Rootkits
HVM Rootkits
What is HVM Rootkits
HVM rootkits is a piece of malicious code embedded in the OS kernel as driver and uses
AMD-V or Intel-VT HVM extension for its operation

The birth of HVM rootkit idea is actually was based on a paper called SubVirt, Malicous
kernel module modifies boot sequence to load original OS inside Virtual PC by a group
researcher from the University of Michigan and Microsoft Research.

The objective of HVM rootkit is to subvert a host OS by putting it into a less privilege
environment and reassert itself as the host OS. By doing this, everything happened in original
host OS can be trapped by the rootkit

In the case of subvert OS make privilege calls from the less privilege domain, AMD-V and
Intel-VT will ignore it, and reroute to VMM (which is in the case the rootkit) for emulation.

The rootkit is not a VMM and it does not have all the resources to run the emulation. When it
received signal from AMD-V or Intel-VT for emulation, it just reroute it back to the actual CPU

23
HVM Rootkits
What is HVM Rootkits

The end game of HVM root is not about virtualization, but hiding it self
in privilege domain container to hide its process.

It does not matter whether you are running VMWARE ESX, XEN or
MICROSOFT Hyper-V, as long the CPU processor supports HVM
extension, the rootkit can work out itself

24
 HVM Rootkits
Subverting an OS
VM Root VM Non-root

Applications Applications 1 Applications

Host OS
Host OS Host OS
(Win, Linux, Mac)
(Win, Linux, Mac) (Win, Linux, Mac) HVM  
ROOTKIT  
HVM  
ROOTKIT   2
HW CPU HW CPU HW CPU
V/VT V/VT V/VT

Host OS Host OS infected Privilege calls are reroute to HVM

Runs Normal by HVM Rootkit rootkit. Then it relay back to the CPU

Since HVM rootkit has direct access to HW allows it to call the AMD-V or Intel-VT HVM
extension to create the root and non-root domain

HVM rootkit then carved at bit of the host OS info, and then set it run into non-root
domain. The rootkit also reinstate the HVM extension by notifying that it as VMM
running in root domain

25
HVM Rootkits
Where can I find HVM Rootkits
•  At current stage, there are available HVM rootkits for AMD-V and Intel-VT HVM extension.

•  And the HVM rootkits works on major operating system such Windows, Linux and Mac kernel

•  Available HVM Rootkits

- Bluepill (AMD & VT)

- Developed by Joanna Rutkowska, COSEINC
- Runs as Windows 64bit driver

- Hyperjack (VT)
- Developed by Nate Lawson, Matasano Security
- Runs as Linux kernel driver

- Virtriol (VT)
- Developed by Dino A. Dai Zovi, Matasano Security
- Similar to Hyperjack except it convert into Mac / FreeBSD kernel module

26
HVM Rootkits
How Does BLUEPILLS works?

27
HVM Rootkits
How Does Hyperjack / Virtriol Works?

28
Demonstration:
From Subverting An OS To
Subverting A VMM
DEMO!
A few notes before conceptual the conceptual demo
•  The HVM rootkits is to subvirt host OS into non-root domain so the HVM rootkit can monitor (or intervine
maybe) the calls made by the host OS to the hardware

•  Among the HVM rootkits, BluePill shows termendous effort on Nested VMM (HVM rootkits running
infected HVM rootkit OS)

•  By using the idea of Joanna’s BluePill nested VMM and Nested KVM, it is possible to create a
powerful HVM rootkit to subvirt a VMM!

•  Recall: VMM is a kernel layer code that controls VMs

•  Currently, there are no actual codes have been released or any proven from theory-to-POC has been
made to prove whether is feasable to subvirt a VMM

•  However, I currently in the mist of coming with the code of a HVM rootkit capable of being a
VMM. I have not giving it a name yet, but being a Malaysian, I may called it PadanMuka or
something else


30
DEMO!
How Should The PADANMUKA Scenarios Works?
•  The demo might be a live demo by the presenter

•  Or a recorded simulation. Please Get the copy from conference organizer if available

31
DEMO!
How does PadanMuka End Game Should Work?
VM Root VM Non-root VM Non-root

Apps Apps

VMM Guest OS Guest OS Malicious code can

PADAN  
MUKA   call PADANMUKA
extension to BLUE
HW CPU PILL other VM
VMCS

VM Root VM Non- VM Non-root VM Non-root

Root
Apps Apps
Download via internet by PADAN VMM
PADAN MUKA rootkit MUKA PADAN   Guest OS Guest OS
MUKA  
VMM ****  

Could be modified XEN vmm VMCS

HW CPU

32
How To Reduced The Risk Via
Open Source VMM
Reduced The Risk Via OSS VMM
The best thing is to prevent HVM Rootkit infections at the beginning.

•  By using VMM gives better security to your OS from malicious activities

•  VMM also prevents HVM rootkit to access the HVM extension at all to subvert
your OS

•  However, if PADANMUKA scenario does exist? Can the VMM defend itself from
being subvert?

34
Reduced The Risk Via OSS VMM
Recommended VMM with its advantages and disadvantages.

•  XEN – XEN.ORG (www.xen.org)

• Advantage: XEN VMM is a micro-kernel and does not allow drivers insertion during runtime. All
drivers insertion are done through special non-root VM which pretty much useless for a HVM rootkit

• Disadvantage: XEN requires guest OS to be modified to accept XEN API. Pretty much does not
work with WINDOWS except in HVM mode which performance are more or less like qemu. Best
works with XEN-LINUX

•  TURAYA – EMSCB (www. http://www.emscb.com/content/pages/turaya.htm)

• Advantage: Similar to XEN

• Disadvantage: Only run L4-LINUX

•  MESINMAYA – DiligentCode Computing (www.diligentcode.com)

•  Advantage: Similar to XEN and TURAYA that does not allow additional drivers insertion during
runtime. All changes must recompiled before used. Support full virtualization with almost all legacy
OS with modified KVM extension. Supports Trusted Computing modules

•  Disadvantage: VM Does not support extended HW capabilities like graphic acceleration and DMA
for plug-n-play

35
Reduced The Risk Via OSS VMM
And why not the commercial VMM

•  VMWARE ESX
- Using Linux Kernel and allows drivers insertion during run-time

•  MICROSOFT HYPER-V
- Windows kernel?
- With windows architecture? Go figure

•  RED HAT KVM

- VMM on User-space environment. Cool Idea
- But still similar to VMWARE ESX runs on top of configurable run-time kernel

36
Conclusions
Conclusions
•  HVM Rootkit uses HVM extension to subvert an OS not VMM

•  PADANMUKA scenario shows more advance HVM rootkit can subvert a

VMM and allows malicious code from non-root to manipulate the original VMM
operations.

•  PADANMUKA scenario fit the current deployment of Cloud Computing and if it

to be deployed, it will seriously interrupt the operation

•  If you read on the internet, HVM rootkit can be detected by various tedious
way. In fact, advance anti-virus may be able to do that. But it is unlikely in
PADANMUKA scenario. In fact no anti-virus can run at RING 0 VMM.

•  The best way to reduced the risk either effecting of HVM rootkit or PADANMUKA
scenario, is via running your OS with static kernel VMM such as XEN, TURAYA or
MesinMaya

38
References
References

J. Rutkowska, Subvirting Vista Kernel For Fun And Profit, Black HAT USA, 2006

Dino A. Dai Zovi, Hardware Virtualization Rootkits, Matasano.com

N. Lawson, Dont tell joanna, the virtualized rootkits is dead, Matasano.com

Yu Ke, Intel Virtualization Technology Overview, Intel System Software Division

D. Chisnall, The definitive Guide to XEN hypervisor, Prentice Hall

40
Thank You

IMPACT T +60 (3) 8313 2020

Jalan IMPACT63000 F +60 (3) 8319 2020
Cyberjaya E contactus@impact-alliance.org
Malaysia impact-alliance.org © Copyright 2010 IMPACT. All Rights Reserved.