Major Project-1 Mid semester Report

Live migration of virtual machines for dynamic provisioning in a data-intensive cloud

Submitted By: Abhinit Modi - 09CO02 Ayushkrity Anuj - 09CO19 Dan Van Duy 09CO26

Dated 3rd October, 2012 Department of Computer Science and Engineering National Institute of Technology Karnataka Surathkal

Abstract
Processing large volumes of data to drive their core business has been the primary objective of many firms and scientific applications these days. Cloud computing being a large-scale distributed computing paradigm in which a pool of resources are delivered on demand to external customers can be used to cater to the needs of data intensive applications. We need to effectively manage, control and monitor application workloads across such computing systems. Dynamic Provisioning of resources is one of the mechanisms for workload management. There are many approaches under dynamic provisioning. Scaling, migration and Surge Computing are the ones commonly used. Migration at the virtual machine level will be our focus in the project. Live migration of virtual machines has been shown to have performance advantages in the case of computation-intensive workloads. Virtual machine environments such as Xen or VMware, provide a function to migrate a running virtual machine between two hosts. Majority of the migration techniques proposed so far, use shared memory approach (Network Attached Storage) where the same data store has to be available from both locations. These do not cater to for migrating VM local storage. However, for wide-area network (WAN) migration, it is crucial to not just transfer the VMs image but also transfer its local persistent state (its file system) and its on-going network connections. Through this project, we try to address this, using a block-level solution with pre-copying and write throttling. We show that we can transfer an entire running VM, including its local persistent state, with minimal disruption.

Table of Contents
1. 2. 3. 4. 5. 6. 7. Introduction ........................................................................................................................ 4 Literature Survey ................................................................................................................ 5 Problem Definition ............................................................................................................. 7 Work Done ......................................................................................................................... 8 Results and Analysis:........................................................................................................ 14 Conclusion and Future work: ........................................................................................... 16 References ........................................................................................................................ 17

1. Introduction
The volume of data in the IT sector is increasing at an exponential rate and it has become very difficult to manage and process such huge amount of data. It has been foreseen that the feasibility of the current methods is very low. Most of the storage and computing till today was managed on large repositories and computer clusters respectively. A strong need for methods transform raw data into tangible information thus arises. The promise of infinite resources given by the cloud computing paradigm has led to recent interest in exploiting clouds for large-scale data-intensive computing. Workload management, an important component of systems management, is the discipline of effectively managing, controlling and monitoring application workload across computing systems. Provisioning, a key workload management paradigm, is the process of dynamically allocating resources for the execution of a work-unit. Migration is a provisioning workload management technique used in clouds where an application execution is moved to a more appropriate host. Migration of an entire OS and all of its applications as one unit avoids many of the difficulties faced by process-level migration approaches. Second, migrating at the level of an entire VM means that in-memory state can be transferred in a consistent and efficient fashion. In practical terms this means that, for example, it is possible to migrate an on-line game server or streaming media server without requiring clients to reconnect: something not possible with approaches which use application level restart. Migration approaches majorly assume the existence of a Network Attached Storage (NAS) which eliminated the concept of local persistent storage and hence there is no need of migrating it. But there are scenarios where local persistent storage plays a vital role and has significant impact (discussed in the next section). This project focuses on such scenarios and addresses the issue by implementing a migration mechanism where VMs with local memory can be considered. The remaining part of the report is divided into 5 sections. The first section, Literature Survey dives us a brief description of what has been done in this field and related fields with relevant references. The next section, problem statement describes a mathematical model of the problem at hand. It gives us an idea as to what our specific objective is. The work done section reveals the work done by the group till now in terms of what they have learnt and analysed. The results section describes the conclusions to which we came after a detailed literature survey. The last section summarizes the report and suggests the next set of tentative procedures.
4

2. Literature Survey
Characterization, Scheduling, Provisioning and Monitoring are the aspects of workload management. Scheduling and Provisioning are the key areas. Scheduling controls the order of executing the individual requests in workloads according to some specified objectives where as provisioning is the dynamic allocation and de-allocation of resources in order to provide load balancing. [1] Xen hypervisor partitions the resources of a computer between domains running guest operating systems. The virtualization approach taken by Xen is extremely efficient. Operating systems such as Linux and Windows XP can be hosted simultaneously for a negligible performance overhead. [2] A good migration policy should minimize downtime, minimize total migration time, and maximize abstraction [4]. Using NAS downtime can be brought down to the order of milliseconds [1] but for a local persistent storage the downtime will be significantly higher. A number of reasons for why a migration at the VM level is useful are quoted in [6]. It states that it is always advisable to do a migration at the VM level than at the process level. It is also quoted that live migration of VMs allows a separation of concerns between the users and providers of a data center (or a cloud). Users have complete control regarding the software and services they run within their VMs, and need not provide the operator with any OS-level access. Similarly the providers need not be concerned with the details of what is happening inside a VM. Instead they can simply migrate the entire VM and its attendant processes as a single unit. [6] IaaS (Infrastructure-as-a-Service) is an emerging concept of cloud computing, which allows users to obtain hardware resources from virtualized data centres. Although many commercial IaaS clouds have recently been launched, dynamic virtual machine (VM) migration is not possible among service providers; users are locked into a particular provider, and cannot transparently relocate their VMs to another one for the best cost-effectiveness. An advanced storage mechanism for live migration has been discusses in [8]. Migrating CPU state, migrating memory content, migrating storage content, migrating network connections are the important steps to be followed in migrating a VM from source host do destination host. Conservation of energy and load balancing are the driving factors to use live migration. [5] Previous memory-to-memory approaches demonstrate the effectiveness of live VM migration in local area networks (LAN), but they would cause a long period of downtime in a
5

wide area network (WAN) environment. Haikun Liu [22] describes the design and implementation of a novel approach, which adopts checkpointing/recovery and trace/replay technologies to provide fast, transparent VM migration for both LAN and WAN environments. Matthias Schmidt [22] several approaches for implementing and synchronizing persistent storage during live migration of virtual machines in Cloud infrastructures are presented. It also enables users to migrate swap space, which is currently not possible on most virtual machine hypervisors. Measurements regarding disk synchronization, migration time and possible overheads are presented. Constantine P. Sapuntzakis [21] developed techniques to reduce the amount of data sent over the network: copy-on-write disks track just the update to capsule disks, ballooning zeros unused memory, demand paging fetches onlyneeded blocks, and hashing avoids sending blocks that already exist at the remote end. Bogdan Nicolae [19] proposes a memory-migration independent approach that addresses this challenge. It relies on a hybrid active push / prioritized pre-fetch strategy, which makes it highly resilient to rapid changes of disk state exhibited by I/O intensive workloads. Michael R. Hines [15] facilitates the use of post-copy with adaptive pre-paging techniques to minimize the number of page faults across the network. Kejiang Ye [13] focuses on the live migration strategy of multiple virtual machines with different resource reservation methods. Jie Zheng [12] presents a novel storage migration scheduling algorithm that can greatly improve storage I/O performance during wide-area migration. Our algorithm is unique in that it considers individual virtual machines storage I/O workload such as temporal locality, spatial locality and popularity characteristics to compute an efficient data transfer schedule.

3. Problem Definition
Consider a virtual machine M running on a host A. Let B be another host to which M needs to be migrated. A becomes the source host and B becomes the destination host to which the guest VM M is migrated. Develop a migration technique to migrate M from A to B with minimum downtime. Any changes which occur to the memory of A when the migration is in progress should be reflected in the copy of the corresponding memory at B. Order of changes should be preserved to maintain consistency. Once M is copied to B and the memories at A and B are consistent, network redirection should be done at B so that there is no loss of data during communication.

Objectives
Our prime targets Implementing live migration with the transfer of local persistent state (file system) at the VM level with minimum downtime. Incorporating a network redirection scheme in the implementation to maintain connection consistency and transparency after migration.

4. Work Done The proposal of the project was followed by a detailed study about the topic and its related areas. The entire task of live migration of virtual machine with local persistent storage was understood to be made up 3 main goals to be satisfied: Migration of run time state, migration of local persistent state and synchronization phase. Many papers and research journals were gone through to understand the approaches being followed to satisfy these goals. Few software and tools which could prove useful in the task were identified. Different approaches to achieve migration, synchronization were analyzed and a relative comparison was done. Advantages, disadvantages, improvements, working conditions of each approach for each target were elaborated. The primary concepts which were focused are elaborated below followed by the results and conclusions.

Migration of run-time state Xen live virtual machine migration: Xen migration monitor migrating whole OS with running applications (kernel-internal state and application-level) rather than single processes. It avoids residual dependencies, provides load balancing, hardware maintenance. Xen uses a pre-copy phase iteratively copying modified pages in rounds, then stops VM and copies remaining memory pages (stop-and-copy phase). It uses dynamic rate-limiting algorithm to minimize resource contention and decide when to end pre-copy phase and enter stop-and-copy phase. Migration process is viewed as a transactional interaction between two hosts. VM is not exposed more to system failure than when running on original single host. VMotion: VMware VMotion enables the live migration of running virtual machines from one physical server to another with zero downtime, continuous service availability, and complete transaction integrity. VMotion is a key enabling technology for creating the dynamic, automated, and self-optimizing datacenter. VMotion keeps the transfer period imperceptible to users by keeping track of on-going memory transactions in a bitmap. Once the entire memory and system state has been copied over to the target host, VMotion suspends the source virtual machine, copies the bitmap to the target host, and resumes the virtual machine on the target host.

Migration of local memory Freeze and copy: Virtual Machine execution is stopped while its address space is being transferred.

Fig 1

Pre-copy: The address space is transferred while the VM is still running on the source node. Therefore, once the decision has been made to migrate, it continues to run on its source node until its address space has been transferred to the destination node. Precopying is done as an initial transfer of the complete address space followed by repeated transfers of the pages modified during the previous transfer until the number of modified pages is relatively small or until no significant reduction in the number of modified pages (detected using dirty bits) is achieved. The remaining modified pages are retransferred after the process is frozen for transferring its state information. The second transfer moves only those pages of the address space that were modified during the first transfer, thus taking less time and presumably allowing fewer modifications to occur during its execution time. Thus subsequent transfer operations have to move fewer and fewer pages.

Fig 2

On demand copy: In this method, the VMs address space is left behind on its source node, and as the relocated VM executes on its destination node, attempts to reference memory pages results in the generation of requests to copy in the desired blocks from their remote locations. Therefore in this demand-driven copy-on-reference approach, a page of the migrant VM's address space is transferred from its source node to its destination node only when referenced. The switching time is very short.

Fig 3

Synchronization paradigms: Using deltas: When a VM migration is requested through the migration client on the source host, the client process forks, creating a separate listener process, and signals to enter record mode. Once in this mode, the driver copies the details of any writes made to the listener process, which transfers them to the migration daemon at the destination. As the VM continues to run at the migration stage, we need to ensure that
10

any changes to its disk images are forwarded to the destination and applied to the version there. Thus we intercept write operations that occur on the source file system, and generate deltas - communication units that consist of the written data, the location of the write on the disk, and the size of the written data. We then record deltas in a queue at the destination host for later application to the disk images there. After the bulk transfer stage is finished, each delta queued at the destination is applied to the disk image there.

Block bitmap: A bitmap is used to record the location of dirty disk storage data during migration. A bit in the bitmap corresponds to a unit in disk storage. 0 denotes that the unit is clean and 1 means it is dirty. After the bitmap has been migrated to the destination, and a read request occurs, the bitmap is queried. If the corresponding block is
clean block is immediately granted read access. Else the destination asks the source to push the dirtied block and makes necessary changes in its bitmap before granting read access. The pulling occurs and only occurs when the VM submits a read access to a dirty block. So the destination must intercept all I/O requests from VM and check if a block must be pulled. When a write request is encountered the access is granted immediately and necessary changes are made in the bitmap.

Iterative log transferring: This approach makes use of ReVirt tool which logs enough information to replay a long-term execution of the virtual machine instructionby-instruction. Unlike memory pre-coping algorithms, this method employs the target hosts computation capability to synchronize the migrated VMs state. What gets copied is the execution log of the source VM but not the dirtied memory pages. The
system state (virtual main memory, CPU registers, memory from external devices and virtual disk) at the current instant are saved to an image file in a copy-on-write fashion.

By iterative we mean that synchronization occurs in rounds, in which the log file to be transferred during round n are those that generated during round n-1 (the checkpoint file is transferred in the first round). After several rounds of iteration, the last log file transferred in the stop-and-copy phase may be reduced to a negligible size so that the downtime can be decreased to an unperceived degree. The source VM is suspended and the remaining log file is transferred to host B. After the last log file is replayed, there is a consistent replica of the VM at both A and B.

11

Optimizations: Inverted binary hash tree for pre-copying: For each data block an Inverted Binary Hash tree is generated. These inverted binary hash trees for all the grain sectors are created and a linked list of that is generated and transferred to the target virtual machine. At the target virtual machine similar inverted binary hash tree list is generated and is compared sequentially with the list of inverted binary hash tree that was sent from source virtual machine. This method ensures efficient and quick identification of data blocks which are different and need retransferring. But it also raises issues of space constraints and network contention. Ballooning: It is observed that clever algorithms that eliminate or page out the less useful data in a system have already been implemented in the OSs virtual memor y manager. The OS is made to reclaim physical memory from existing processes by running a balloon program that asks the OS for a large number of physical pages. The program then zeros the pages, making them easily compressible. Ballooning reduces the size of the compressed memory state and thus reduces the start-up time. This technique works especially well if the memory has many freed pages whose contents are not compressible. There is no reason to transfer such data, and these pages are the first to be cleared by the ballooning process. Dynamically choosing parameters: Using a static target downtime as a condition for switching from iterative migration to stop-and-copy leads to sub-optimal behaviour. A new technique where switching is decided by matching memory update patterns that indicate that no beneficial progress, reduction in downtime, is achieved by continuing live migration was analyzed. The probability of further reduction in downtime, without requiring any static estimates was gone through. Dynamic determination of the number of iterations in the pre-copy phase based on dirtying frequency and network resources was found suitable. Virtual Memory approach: Use Xen live migration to transfer the run time state of the VM. Randomly select few pages (say x) from the storage and transfer them to the destination (This will be your downtime). Use DynDNS to announce change of address of the virtual machine. Resume the execution if the VM on the destination. Also the x pages of storage present on source and destination are consistent. When a read access to a page y occurs and if y is present at the destination, just go ahead. Else

12

pull y from source and either replace (high residual dependency and less performance) or add to the set x.

Proposed Approach The Proposed approach is an extension of the delta approach described above. The memory migration is done using any of the existing approaches like Xen-live migration. The local disk is divided into blocks of suitable range. Each block will uniquely be represented by a block-id. For the migration of the local disk storage pre-copy mechanism with a restricted queue is used. Instead of generating deltas and queuing them for each and every change that occurs to the disk image, we restrict the queuing rate. Once the delta is generated, we use a hash function on the location of the change made (will be mentioned in the delta). The result of the hash function will be the block-id to which the changed location belongs. A map consisting of <block-id, count> is maintained where count represents the number of changes made in that particular block. We predefine a threshold t called the write-intensive factor. Every time the hash function returns a block-id we increment the counter corresponding to the block-id. If the count is greater than t we do not queue the delta whereas if it is less we queue the delta. Precopy mechanism is followed to transfer these restricted write intensive blocks to the destination after the background copying.

13

5. Results and Analysis:

Migration of run time state: Xen-live migration Advantages 1. Minimize down time, total migration time and disruption of services as seen by a client. 2. It provides hardware maintenance and separation of concern between users and operators. VM Motion 1. Improve availability by conducting maintenance without disrupting business operations 2. Move virtual machines within server resource pools to continuously align the allocation of resources to priorities.

Disadvantages 1. Wide-area network migration not supported. 2.Local-disk storage not migrated

Migration of local memory: Freeze and copy Advantages 1. Simple and easy to implement. Pre-copying 1. Reduced freezing time. 2. Minimal interference of migration with process interaction On demand copy 1. Switching time of the process from its source node to its destination node is very short. 2. It is virtually independent of the size of the address space 1. This method imposes a continued load on the process's source node and results in failure of the process if the source node fails or is rebooted.

Disadvantages 1. VM is suspended for a long time during Migration, so timeouts may occur.

Environment

Cannot be used if the scenario is highly interactive. Results in long delays.

1. It may increase the total time for migration due to the possibility of redundant page transfers. Redundant pages are pages that are transferred more than once during Pre-transferring because they become dirty while the pretransfer operation is being performed. Cannot be used if the scenario is highly write-intensive over a wide address space.
14

This method is based on the assumption that processes tend to use only a relatively small

part of their address spaces while executing.

Synchronization paradigms: Using deltas Block bitmap Iterative log transfer Advantages 1. Space consumption of a delta is relatively less. 2. Relatively less dependent of the network parameters. 3. Relatively less I/O overhead 1. Simple to understand and implement. 2. Works well even in the case of write intensive applications 1. Great decrease in the data transferred while synchronizing the two VMs running state. 2. This approach reduces the downtime 3. The migration does not disrupt other active services residing in the same host through resource contention (e.g., CPU, network bandwidth). 1. The log transfer rate should be faster than the log growth rate. 2. The log replay rate must be faster than the log growth rate. 3. Centralized dependency on the log file

Disadvantages 1. Queue may start overflowing if the application is highly write intensive. 2. Data compression and avoiding transfer of unnecessary blocks is not taken care of increases migration time.

1. Residual dependency on the source node. 2. Delay in granting read request. 3. Not very transparent. 4. Network contentions play an important role

Proposed Approach Advantages 1. Avoids unnecessary queuing of deltas By preventing queue overflow. 2. Lesser the queue size, lesser will be the performance drop due to network contention. 3. Gives good performance in case of write intensive applications Disadvantages 1. Very complex implementation. 2. Hashing should be done effectively. 3. Not suitable for less write intensive cases

15

6. Conclusion and Future work:

We have done a detailed relative analysis of various approaches for both migration and synchronization. We have found the xen-live migration is the most widely migration mechanism for migration of run time state of the VM. We also concluded that the delta approach for synchronization is more advanced and suitable in most of the scenarios when compared to the bitmap approach. Total freezing technique for memory migration is out dated. Pre copy and on demand approaches are both used depending on the requirement. Few optimization techniques were also studied. Finally we Proposed an approach (partial) for live migration and will be proposing the complete version soon. Our next step would be to soon conclude with an appropriate platform for our purpose. Initially we would like to run basic cloud algorithms on the selected platform to understand the working, the architecture and implementation details of the chosen platform. Later we would like to run existing migration algorithms to understand how to evaluate the parameters which would rate the performance of the migration algorithm. In the final stage we would evaluate those parameters for our proposed algorithm which would have been fine tuned in parallel to the above pipeline.

16

7. References
[1] Rizwan Mian and Dr. Patrick Martin: Managing Data-Intensive Workloads in a Cloud. Ph.D depth paper, in Queens University, Kingston, September 2011. [2] Barham, P., Dragovic, B., Fraser, K., Hand, S., Harris, T., Ho, A, Neugebauer, R., Pratt, I., and Warfield, A: Xen and the Art of Virtualization. In SOSP (2003). [3] Christopher Clark, Keir Fraser, Steven Hand, Jacob Gorm Hanseny, Eric July, Christian Limpach, Ian Pratt and Andrew Wareld: Live Migration of Virtual Machines. [4] Matus Harvam: Xen Live Migration. In Networks and Distributed Systems Seminar, April 2006. [5] Pradeep Padalas blog: Understanding Live migration of virtual machines. http://ppadala.net/blog/2010/06/understanding-live-migration-of-virtual-machines/ [6] Clark, S. (2000). Teradata NCR. Retrived 22.2.11, from

http://www.teradata.com/library/pdf/butler_100101.pdf [7] Jianhua Gu, Guofei Sun and Tianhai Zhao: A Scheduling Strategy on Load Balancing of Virtual Machine Resources in Cloud Computing Environment. In Parallel Architectures, Algorithms and Programming (PAAP), 2010 Third International Symposium by IEEE. [8] Ogawa H, Nakada H, Itoh S and Sekiguchi S: A Live Storage Migration Mechanism over WAN for Relocatable Virtual Machine Services on Clouds. In Cluster Computing and the Grid, 2009. CCGRID '09. 9th IEEE/ACM International Symposium. [9] Yi Zhao: Adaptive Distributed Load Balancing Algorithm Based on Live Migration of Virtual Machines in Cloud. In INC, IMS and IDC, 2009. NCM '09. Fifth International Joint Conference. [10] Michael Nelson, Beng-Hong Lim, and Greg Hutchins: Fast Transparent Migration for Virtual Machines, Proceedings of USENIX 05: General Track Anaheim, California, USA April 10-15, 2005. [11] Hai Jin, Li Deng, Song Wu, Xuanhua Shi, Xiaodong Pan: Live Virtual Machine Migration with Adaptive Memory Compression, Huazhong University of Science and Technology, Wuhan, China. [12] Jie Zheng, T. S. Eugene Ng, Kunwadee Sripanidkulchai: Workload-Aware Live Storage Migration for Clouds.

17

[13] Kejiang Ye, Xiaohong Jiang, Dawei Huang, Jianhai Chen, Bei Wang: Live Migration of Multiple Virtual Machines with Resource Reservation in Cloud Computing Environments, College of Computer Science, Zhejiang University, China. [14] Christian Vecchiola, Dileban Karunamoorthy, and Jessie Yi Wei: Aneka Windows installation guide. [15] Michael R. Hines, Umesh Deshpande, and Kartik Gopalan: Post-Copy Live Migration of Virtual Machines, Computer Science, Binghamton University. [16] Ajay Surie, H. Andrs Lagar-Cavilla, Eyal de Lara, M. Satyanarayanan: LowBandwidth VM Migration via Opportunistic Replay, University of Toronto. [17] Brendan Cully, Geoffrey Lefebvre, Dutch Meyer, Mike Feeley, Norm Hutchinson, and Andrew Warfield: Remus: High Availability via Asynchronous Virtual Machine Replication, University of British Columbia. [18] Jun Zhu, Zhefu Jiang, Zhen Xiao: Optimizing the Performance of Virtual Machine Synchronization for Fault Tolerance, IEEE TRANSACTIONS ON COMPUTERS, DECEMBER 2011. [19] Bogdan Nicolae, Franck Cappello: A Hybrid Local Storage Transfer Scheme for Live Migration of I/O Intensive Workloads, 21st International ACM Symposium. [20] Ashwin Sancheti, Prof.Randal Burns: Replica Synchronization for Virtual Machines. [21] Constantine P. Sapuntzakis, Ramesh Chandra Ben Pfaff Jim Chow, Monica S. Lam Mendel Rosenblum: Optimizing the Migration of Virtual Computers, Proceedings of the 5th USENIX Symposium on Networked Systems Design and Implementation. [22] Hai Jin, Haikun Liu, Xiaofei Liao, Liting Hu, Peng Li: Live Migration of Virtual

Machine Based on Full-System Trace and Replay, Huazhong University of Science and Technology, Wuhan, China. [23] Katharina Haselhorst, Matthias Schmidt, Roland Schwarzkopf, Niels Fallenbeck, Bernd Freisleben: Efficient Storage Synchronization for Live Migration in Cloud Infrastructures, University of Marburg, Germany. [24] Yingwei Luo, Binbin Zhang, Xiaolin Wang, Zhenlin Wang, Yifeng Sun, Haogang Chen: Live and Incremental Whole-System Migration of Virtual Machines Using Block-Bitmap, Peking University, China. [25] Takahiro Hirofuchi, Hirotaka Ogawa, Hidemoto Nakada, Satoshi Itoh, and Satoshi Sekiguchi: A Live Storage Migration Mechanism over WAN and its Performance Evaluation, National Institute of Advanced Industrial Science and Technology.

18