Professional Documents
Culture Documents
Sarl Virtual Open Systems - Registered at 529 212 987 RCS Grenoble - www.virtualopensystems.com - Page 1/14 -
Embedded Systems Virtualization Services by Virtual Open Systems - KVM porting on ARM Cortex-A15
Summary
1 Introduction 2 Simulation platform 2.1 Building a simulation platform 3 The host Linux system 3.1 Cross Compiler 3.2 The host Linux kernel 3.3 Device tree support 3.4 Bootloader 3.5 Host file system 3.6 Testing the host system 4 Preparing the system to boot a guest 4.1 QEMU 5 The guest system 5.1 Guest kernel 5.2 Guest file system 5.3 Running the guest system 6 Optional steps 6.1 Fast Models Networking with a network bridge 6.2 Custom cross compiler 6.3 Booting without semihosting 7 Conclusion 8 About Virtual Open Systems Golden Guides 9 About Virtual Open Systems 9.1 Virtual Open Systems KVM-on-ARM porting Services 3 4 4 5 5 5 6 6 7 7 8 8 9 9 9 10 11 11 12 12 13 14 14 14
Sarl Virtual Open Systems - Registered at 529 212 987 RCS Grenoble - www.virtualopensystems.com - Page 2/14 -
Embedded Systems Virtualization Services by Virtual Open Systems - KVM porting on ARM Cortex-A15
1 Introduction
In the last decade virtualization has been established as a very powerful tool expanding the capabilities of servers, and enabling disruptive technologies like cloud computing. At the same time, virtualization has also been proven as a powerful tool for end users, system administrator's, security researchers, and system developers. Virtualization has only started to show it's capabilities on mobile and embedded platforms, however likewise to the desktop and server world, a very wide range of new use cases can be supported. The Linux Kernel Virtual Machine (Linux KVM) is one of the most successful and powerful Virtualization solutions available, enabling the Linux kernel to boot guest Operating Systems under a process. Linux KVM has been designed to be portable, and has proven itself in a number of architectures, like Intel VT-x, AMD SVM, PowerPC and IA64, and is now on it's way for the ARM Cortex-A15 platform. This document describes how to set up a development environment for KVM/ARM on Cortex-A15. A working recent Linux system is assumed to be used by the user for development. The instructions provided assume an installation of a recent version of Ubuntu (11.10 at the time of this writting), but could be adjusted for other modern distributions. After following this guide the user will be able to boot a Cortex-A15 simulation platform, with a working KVM virtualization setup able to boot a Linux guest. This document will guide the reader through a number of steps in order to setup the development environment for KVM development on Cortex-A15: Installation and setup of the simulation platform. We will use the ARM Fast Models platform to generate a Cortex-A15 environment. Setup of a host ARM Linux system. We will compile an ARM Linux kernel which we can boot on our simulation platform; we will use a basic file system image to boot a working system. Build KVM and QEMU on our setup. We will enable KVM support in the host Linux kernel, and built QEMU with KVM/ARM support in order to boot a guest system. Build a guest system and boot it under KVM/ARM. We will prepare a standard Linux system and boot it as a guest under KVM/ARM. The final system, running with a guest under our host Linux system, can be illustrated as follows:
For more information on the hardware architecture see the ARM Architecture Reference Manual for v7-a processors. Other related documentation includes the Virtualization Extensions documentation as well as the Large Physical Address Extensions (LPAE). These can be found on the ARM Infocenter.
Sarl Virtual Open Systems - Registered at 529 212 987 RCS Grenoble - www.virtualopensystems.com - Page 3/14 -
Embedded Systems Virtualization Services by Virtual Open Systems - KVM porting on ARM Cortex-A15
2 Simulation platform
There is currently no hardware readily available with the Cortex-A15 core on it, so we use the FAST Models simulator from ARM. An evaluation version is available from ARM's web site. Click the Download now button and proceed to find the Fast Models Evaluation Linux. A registration with ARM is necessary to be able to proceed with the download of the software. The latest version of the platform at the time of this writing is 7.1. When downloading the software one can also receive a license file for a 45 day evaluation. Unpack the software and read the installation instructions in Installation_Guide.txt. The official instructions claim support for gcc 3.4.3 or gcc 4.1.2, however in our experience more recent versions offered by modern distributions usually tend to work better. At this point we need to make sure the system has installed all the standard development tools. If not, we can install them using the distribution's package management tools, e.g. on Debian and Ubuntu:
$ sudo apt-get install build-essential xutils xutils-dev $ sudo apt-get install ia32-libs gcc-multilib # required for 64 bit systems
Run ./setup.bin and choose an installation directory and the location of the license file; e.g. /home/user/ARM/. It is also advised to edit the current user's .bashrc file and add a line like this:
$ source ~/ARM/FastModelsTools_7.0/etc/setup_all.sh
In the opposite case this command will need to be run manually each time we need to use the Fast Models simulation platform.
Tip
Network interface In som distributions the license manager might not find eth0 or assume a HOSTID of 000000000000. In that case we need to configure the Linux system used to name the first network interface eth0; check the distribution's documentation, or see the instructions for Fedora.
Tip
On missing lstdc++ If the build fails on a 64 bit system, complaining that lstdc++ cannot be found, then try the following:
Sarl Virtual Open Systems - Registered at 529 212 987 RCS Grenoble - www.virtualopensystems.com - Page 4/14 -
Embedded Systems Virtualization Services by Virtual Open Systems - KVM porting on ARM Cortex-A15
We can now compile the model by going to Project > Build System. Afterwards, use a terminal to navigate to the project directory and run the model:
The simulation of the platform should start; however we have provided no software for it, therefore nothing interesting can happen yet.
Tip
64 bit systems You can use model_shell64 instead of model_shell on 64 bit systems, though we advice against it due to minor performance degradation.
$ sudo apt-get install gcc-4.6-arm-linux-gnueabi binutils-arm-linux-gnueabi \ libc6-armel-cross linux-libc-dev-armel-cross \ gcc-arm-linux-gnueabi libc6-dev-armel-cross cpp-arm-linux-gnueabi
In case arm-linux-gnueabi-gcc isn't available on your linux distribution, it is possible to compile your own cross compiler, with instructions in the Custom cross compiler chapter.
Also, mkimage from the u-boot package is required to build a kernel uImage; under Debian or Ubuntu we can do the following to install it:
Sarl Virtual Open Systems - Registered at 529 212 987 RCS Grenoble - www.virtualopensystems.com - Page 5/14 -
Embedded Systems Virtualization Services by Virtual Open Systems - KVM porting on ARM Cortex-A15
Configuring the kernel for cross compilation does not differ much than usual, however it is important to set the ARCH and CROSS_COMPILE variables. We can download a premade configuration for the kernel:
Make sure that our kernel configuration file (.config), includes the following: CONFIG_ARCH_VEXPRESS_DT=y CONFIG_ARM_APPENDED_DTB=y CONFIG_ARM_ATAG_DTB_COMPAT=y On our kernel source root:
The resulting dtb file should be used with the host bootloader and QEMU, as instructed on the following chapters.
3.4 Bootloader
We don't use u-boot to run Cortex-A15 yet, but instead a small custom bootloader, with the necessary hypervisor monitor API used to initialize KVM. We can clone a copy of this bootloader from the Virtual Open Systems repository:
Sarl Virtual Open Systems - Registered at 529 212 987 RCS Grenoble - www.virtualopensystems.com - Page 6/14 -
Embedded Systems Virtualization Services by Virtual Open Systems - KVM porting on ARM Cortex-A15
Make sure an appropriate directory is exported in /etc/exports with the right settings. For example:
/srv/nfsroot 192.168.0.0/255.255.0.0(rw,sync,no_root_squash,no_subtree_check,insecure)
The above configuration defines an NFS share in /srv/nfsroot, accessible by any machine on the local network with an IP address of 192.168.x.x. Any file system compatible with the processor can be used. A starting point can be one of the file system images from http://www.arm.com/community/software-enablement/linux.php We need to extract the tar in our exported NFS share:
$ sudo mount -o loop -t cramfs filesystem_bin_alip-ael-armv7-thumb-min-debug.cramfs \ /mnt $ sudo cp -R /mnt/* /srv/nfsroot/ $ sudo umount /mnt
Kernel: --kernel /path-to/uImage Optional initrd: --initrd /path-to/initrd Optional device tree file: --fdt /path-to/dtb Kernel arguments: followed by a double dash and a space "-- "
In our case, to boot the host kernel, the semihosting arguments should be (in one contiguous line):
Sarl Virtual Open Systems - Registered at 529 212 987 RCS Grenoble - www.virtualopensystems.com - Page 7/14 -
Embedded Systems Virtualization Services by Virtual Open Systems - KVM porting on ARM Cortex-A15
We can connect to a serial output of the simulated platform in order to see Linux booting:
After a while, we should be able to login as root, greeted by a Busybox shell for us to use. Optionally you can check the Booting without semihosting chapter, in case you don't need the semihosting feature of the bootloader.
Tip
Use a precompiled binary of qemu A binary of qemu can be downloaded from our website, which can be used to skip this section and the next one.
QEMU requires a few dependencies, we can use the tools provided by Ubuntu, to easily get all of them for the cross compiled ARM build of QEMU:
deb [arch=armel] http://ports.ubuntu.com/ubuntu-ports precise main restricted universe multiverse deb-src [arch=armel] http://ports.ubuntu.com/ubuntu-ports precise main restricted universe multiverse
4.1 QEMU
We need to build QEMU to emulate devices and drive KVM from user space. We can clone the latest version with KVM on ARM support from the Virtual Open Systems repository:
Embedded Systems Virtualization Services by Virtual Open Systems - KVM porting on ARM Cortex-A15
This will create an ARM QEMU binary in arm-softmmu/qemu-system-arm, which we will be able to use in our host system to boot a guest.
Tip
Skip building a guest kernel A kernel suitable to be used as a guest can alternatively be downloaded from our website.
To compile the guest kernel we can use the same instructions and repository of the host, but with a slightly different configuration:
Then deselect: The entire Virtualization submenu (CONFIG_VIRTUALIZATION) Support for the Large Physical Address Extension under System Type (CONFIG_ARM_LPAE) Alternatively, we can download a guest configuration file and compile:
Sarl Virtual Open Systems - Registered at 529 212 987 RCS Grenoble - www.virtualopensystems.com - Page 9/14 -
Embedded Systems Virtualization Services by Virtual Open Systems - KVM porting on ARM Cortex-A15
Once inside the KVM host, we can finally attempt to boot the guest:
$ ./qemu-system-arm \ -enable-kvm \ -kernel zImage \ -sd guest.cramfs \ -dtb ./rtsm_ve-cortex_a15x1.dtb \ -m 512 -M vexpress-a15 -cpu cortex-a15 -nographic \ -append "console=ttyAMA0 mem=512M root=/dev/mmcblk0 rw"
Another Linux system should now boot as a guest inside our host Linux system. We can now login to the guest and use it like any usual Linux system.
Tip
Boot the guest with initrd It is highly advised to boot any guest, with initrd (ramdisk) and not MMC. Due to performance issues of Fast Models (different workload), initrd booting is several times faster than MMC.
$ $ $ $ $ $
mkdir mnt1 mnt2 sudo mount -o loop filesystem_bin_alip-ael-armv7-thumb-min-debug.cramfs mnt1 sudo cp -a ./mnt1/* ./mnt2/ cd ./mnt2 sudo ln -s sbin/init init find ./ | cpio -o -H newc | gzip > ./initrd.cpio.gz
Then, simply replace the -sd and --append lines in the QEMU command with:
Sarl Virtual Open Systems - Registered at 529 212 987 RCS Grenoble - www.virtualopensystems.com - Page 10/14 -
Embedded Systems Virtualization Services by Virtual Open Systems - KVM porting on ARM Cortex-A15
6 Optional steps
In this chapter we will describe complementary functionality that is no longer mandatory to setup a fully working system.
Inside the FastModelsPortfolio_7.0 directory, there should be a sub directory called ModelNetworking containing the add_adapter_32.sh and add_adapter_64.sh scripts. These are designed for a RHEL system, however they can usually be made to run on other distributions. They will setup a network bridge for us and create an activate/deactivate script in /etc/init.d/FMnetwork for our convenience.
Tip
Modification of setup.sh The scripts provided by the Fast Models to setup the ethernet bridge might hang on modern distributions with the message - Changing '/dev/net/tun' permissions.... In that case we need to modify the file setup.sh in the same directory, and comment out the lines which update the permissions (215-220 for the Fast Models released at the time of this writing). You will need to take care of any permissions that might need to be changed manually, if any.
From now on we can use sudo /etc/init.d/FMNetwork start to enable the network bridge for the Fast Model and sudo /etc/init.d/FMNetwork stop when we are done using it. We can now run our model with the following in our params file:
motherboard.hostbridge.interfaceName="ARMbridge" motherboard.smsc_91c111.enabled=1
Tip
On Network Manager When using the init.d scripts we have installed, Network Manager may in modern Linux systems report that the system is offline; re-enabling the network connection will cause Network Manager on the other hand to override part of the configuration we need for the network bridge. If such problems arise a workaround is to kill nm-applet when planning to activate the network bridge, or to completely disable Network Manager, e.g. on Ubuntu:
Sarl Virtual Open Systems - Registered at 529 212 987 RCS Grenoble - www.virtualopensystems.com - Page 11/14 -
Embedded Systems Virtualization Services by Virtual Open Systems - KVM porting on ARM Cortex-A15
Applications like Firefox might report as being offline, however they can access the web normally.
Tip
crosstool-ng dependencies crosstool-ng has a number of dependencies. We can install those on Debian and Ubuntu systems using:
$ sudo apt-get install gperf mercurial libncurses5-dev curl libtool \ automake texinfo flex bison gawk
Replace the Cortex-A9 setting with Cortex-A15 and make sure to select a gcc-4.6 compiler and that bare-metal is selected under Operating System -> Target OS. Otherwise follow the instructions from the above guide. Once built, the current user's .bashrc can be edited to add the installed toolchain in the path, e.g.:
$ PATH=$PATH:$HOME/bin:$HOME/.local/bin:$HOME/x-tools/arm-unknown-eabi/bin
Tip
unset LD_LIBRARY_PATH If crosstool-ng returns an error about LD_LIBRARY_PATH being set, then unset it before building the cross compiler:
$ unset LD_LIBRARY_PATH
The bootloader has to be recompiled as described previously, in order for the uImage to be updated. The zImage can be also used with QEMU, by removing the -dtb argument when starting it. Lastly, we have to start the model with the proper elf application:
Sarl Virtual Open Systems - Registered at 529 212 987 RCS Grenoble - www.virtualopensystems.com - Page 12/14 -
Embedded Systems Virtualization Services by Virtual Open Systems - KVM porting on ARM Cortex-A15
Notice the missing -semi suffix in the elf image. In this boot method, the uImage and optionally an initrd filesystem, can be embedded into the application itself. To change the kernel boot arguments, we have to make a copy of the file config-default.mk under the name config.mk. Inside config.mk you can change various aspects of the bootloader: USE_INITRD: Choose between embedding an initrd on the elf image or not. FILESYSTEM: Location of the initrd file. NFS_ROOT: Location of the NFS folder. KCMD (under versatile express): Kernel boot arguments. Every time our kernel or config.mk changes, we have to recompile the bootloader to update it properly.
7 Conclusion
After having followed successfully the previous steps of the guide, the user should now be able to experiment with Linux KVM under an ARM Cortex-A15 simulation platform. After booting a guest operating system, another Linux system, it is up to the user to proceed with what he wants to do; one may find the source code of the KVM ARM port under linux-kvm-arm/arch/arm/kvm. One can adapt the source code to his needs, or use what is provided to try to boot his own software platforms under the Linux KVM ARM port.
Sarl Virtual Open Systems - Registered at 529 212 987 RCS Grenoble - www.virtualopensystems.com - Page 13/14 -
Embedded Systems Virtualization Services by Virtual Open Systems - KVM porting on ARM Cortex-A15
Sarl Virtual Open Systems - Registered at 529 212 987 RCS Grenoble - www.virtualopensystems.com - Page 14/14 -