Commit 7a182f6e authored by Vivek Kumar Gautam's avatar Vivek Kumar Gautam Committed by Thomas Abraham
Browse files

infra/common: document virtualization support

Add a new document that explains the procedure to build software stack,
boot the system and launch multiple guests using kvmtool.

Change-Id: I8b898caeeb4de11562d48165381cd41bf0d8b4be
Signed-off-by: Vivek Kumar Gautam's avatarVivek Gautam <>
parent 37ce1dd7
Virtualization using KVM
.. contents::
What is KVM?
Kernel Virtual Machine (KVM) is a virtualization module built in the linux
kernel which lets the user to turn linux into a hypervisor to allow hosting
single/multiple isolated guests or virtual machine. In brief, KVM is a type-2
hypervisor that requires a host OS to boot first, and the KVM module runs on
top of that.
KVM requires a processor with hardware virtualization extensions. Some of the
architectural features in Arm v8-a profile that support hardware virtualization
are -
- A dedicated Exception level (EL2) for hypervisor code.
- Support for trapping exceptions that change the core context or state.
- Support for routing exceptions and virtual interrupts.
- Two-stage memory translation, and
- A dedicated exception for Hypervisor Call (HVC).
Currently, KVM is part of linux kernel. Some of the features of KVM are:
- Over-committing: KVM allows to allocate more virtualized CPU or memory for the
virtual machine than that of the host.
- Thin provisioning: KVM allows to allocate and optimize the flexible storage
for the virtual machines.
- Disk throttling: KVM allows to set limits for disk I/O requests.
- Virtual CPU hot plug: KVM allows ability to increase the CPU count of the
virtual machine during run time.
Virtualization on Neoverse Reference Design Platforms
Virtualization using KVM hypervisor is supported on the Neoverse reference
design plaforms. The subsequent sections below provide detailed instructions
about booting up of two or more instances of guest OS's (or Virtual Machines,
VMs) using `lkvm tool`_. Each of these guests can support upto NR_CPUS as
vcpus, where NR_CPUS is the number of CPUs booted up by the host OS. There are
instructions on using hardare virtualization features on the platform and enable
use of virtualized devices, such as console, net, and disk etc.
Overview of Native Linux KVM tool
kvmtool is a lightweight tool for hosting KVM guests. As a pure virtualization
tool it only supports guests using the same architecture, though it supports
running 32-bit guests on those 64-bit architectures that allow this.
The kvmtool supports a range of arm64 architectural features such as support for
GIC-v2, v3, and ITS. It also supports device virtualization using emulated
devices such as virtio device support for console, net, and disks, and using
VFIO to allow PCI pass-through or direct device assignment.
Booting multiple guests
Virtualization using KVM hypervisor requires a root filesystem from which
kvmtool can be launched. Buildroot root filesystem supports the kvmtool package.
It fetches the mainline kvmtool source and builds the kvmtool binary out of it.
Detailed description on buildroot based booting ia available in `buildroot guide`_.
Follow all the instructions in that document for building the platform software
stack and booting upto buildroot before proceeding with the next steps.
To boot two or more virtual machines on the host kernel with a kernel image and
an initrd or a disk image, KVMtool virtual machine manager (VMM) (also called as
lkvm tool) is used. Check help for 'lkvm run' command for options to launch
Launching multiple guests using lkvm:
- Mount grub disk-image:
The buildroot filesystem required to perform kvm test is packaged in such a
way that the kernel image, and buildroot ramdisk image are copied to the
second partition of grub disk image that gets probed at /dev/vda2 in the host
kernel. After booting the platform this partition can be mounted as:
mount /dev/vda2 /mnt
- Launch VMs using lkvm:
For launching multiple VMs, 'screen' tool can be used to multiplex console
outputs so that one can switch between multiple workspaces. This tool helps by
providing a new console output pane for each guest. Use the following command
to launch guests using kvmtool with the available kernel and ramdisk images.
screen -md -S "<screen_name>" lkvm run -k <path-to-linux-image> -i <path-to-ramdisk-image> --irqchip gicv3-its -c <nr-cpus> -m <allowed-mem> --console virtio --params "earltprintk=shm console=hvc0 root=/dev/vda
For example, to run the kernel available in mounted disk at /mnt as above use
the following command:
screen -md -S "virt1" lkvm run -k /mnt/Image -i /mnt/ramdisk-buildroot.img --irqchip gicv3-its -c 4 -m 512 --console virtio --params "earltprintk=shm console=hvc0 root=/dev/vda"
- Launch couple of more guests by repeating the above command and updating the
The launched screens can be viewed from the target by using the following
screen -ls
- Select and switch to the desired screen to view boot-up logs from guest. Use
the following command to go to a specific screen:
screen -r <screen_name>
- For example, list of screens are shown below:
# screen -ls
There are screens on:
214.virt1 (Detached)
200.virt2 (Detached)
- Jump to the screen using:
screen -r virt1
- Switch between multiple running guests to view the bootup logs of various
guests executing.
- Perform simple cpu hotplug test to validate that guest kernel is functional.
Use the following command to do that:
echo 0 > /sys/devices/system/cpu/cpu1/online
echo 0 > /sys/devices/system/cpu/cpu2/online
echo 1 > /sys/devices/system/cpu/cpu1/online
echo 1 > /sys/devices/system/cpu/cpu2/online
The CPUs should go offline and come back online with the above set of
- Jump back to the host by exiting the screen using 'Ctrl-a d', and use the
following command to see how many guests are managed by lkvm tool:
# lkvm list
309 guest-309 running
276 guest-276 running
- Power-off the guests by jumping to the right screen, and the guests would
shutdown fine with the following log:
# KVM session ended normally.
This completes the procedure to launch multiple VMs and terminate them.
PCIe pass-through based device virtualization
PCIe pass-through (also called as direct device assignment) allows a device to
be assigned to a guest such that the guest runs the driver for the device
without intervention of the hypervisor/host. This is one of the device
virtualization technique besides para-virtualization.
PCIe pass-through is achieved using frameworks in linux kernel, such as VFIO,
virtio, IOMMU, and pci. A smmu-test-engine (smmute) device that is available on
the platform is used as a test device for this virtualization technique. The
smmu-test-engine is a PCIe exerciser that generates DMA workloads and it uses
arm-smmu-v3 to provide dma isolation. This device first probed in the host
kernel can be assigned to the guest and the smmu-test-engine driver in the guest
kernel can then manage the device directly.
PCI pass-through using multiple guests and smmu test engine:
- Boot the platform by following the `buildroot guide`_, and then ensure that
the smmu test engine device is probed correctly. Use the lspci command to
check for smmu test engine devices with pci BDF ids - 07:00.0, 07:00.3,
08:00.0 and 08:00.1.
- Verbose output of lspci will show the last four devices with above mentioned
pci BDF ids are managed by 'smmut-pci' kernel driver.
- Also check that the smmute-pci driver has probed the smmu test engine devices
properly, and a device extry exists for each of the four smmute devices.
ls -l /dev/smmute*
- Use one of the smmute devices (e.g. device 0000:08:00.1) to perform the PCI
pass-through. Detach the pcie device from its class driver and attach to
vfio-pci driver, as also explained in the `kernel doc`_.
echo 0000:08:00.1 > /sys/bus/pci/devices/0000:08:00.1/driver/unbind
echo vfio-pci > /sys/bus/pci/devices/0000:08:00.1/driver_override
echo 0000:08:00.1 > /sys/bus/pci/drivers_probe
- The kernel and ramdisk images to launch VMs are available in the second
partition of grub disk image that gets probed at /dev/vda2 in the host.
Mount this to use the images.
mount /dev/vda2 /mnt
- This mounted partition can also be shared with guest using 9p virtual
filesystem. A binary to run tests over smmute device is also available
in this partition. So after sharing the filesystem with a guest, tests
can be run on assigned smmute device to verify pci pass-through.
- Launch VMs using lkvm tool that supports virtio-iommu and vfio drivers to
allow pci pass-through.
screen -md -S "virt1" lkvm run -k /mnt/Image -i /mnt/ramdisk-buildroot.img --irqchip gicv3-its -c 2 -m 512 --console virtio --9p /mnt,hostshare --params "earlyprintk=shm console=hvc0 root=/dev/vda" --vfio-pci 0000:08:00.1 --viommu vfio;
- Jump to the right screen to view boot-up logs from guest. Use following command
to go to a specific screen:
screen -r virt1
- After the guest boots up, mount the 9p filesytem to a mount point in the guest.
For example, use the following command to mount at /tmp
mount -t 9p -o trans=virtio hostshare /tmp/
cd /tmp
- Check that the smmu test engine is probed in the guest. The device will show a
different pci BDF id here in guest as compared to the id shown in host kernel.
# lspci
00:00.0 Unassigned class [ff00]: ARM Device ff80
# ls -l /dev/smmute*
crw------- 1 root root 235, 0 Jan 1 00:00 /dev/smmute0
- From /tmp directory that contains the 'smmute' binary, run the test.
./smmute -s 0x100 -n 10
- Jump back to the host by exiting the screen using 'Ctrl-a d', and check the
stats for the kvm session.
lkvm debug -a -i stat
- Launch couple more guests by repeating the above commands and updating the
screen_name, and device. For example,
echo 0000:08:00.0 > /sys/bus/pci/devices/0000:08:00.0/driver/unbind
echo vfio-pci > /sys/bus/pci/devices/0000:08:00.0/driver_override
echo 0000:08:00.0 > /sys/bus/pci/drivers_probe
screen -md -S "virt2" lkvm run -k /mnt/Image -i /mnt/ramdisk-buildroot.img --irqchip gicv3-its -c 2 -m 512 --console virtio --9p /mnt,hostshare --params "earlyprintk=shm console=hvc0 root=/dev/vda" --vfio-pci 0000:08:00.0 --viommu vfio;
- Perform test over smmu test engine in this second screen by mounting the 9p
filesystem and executing the 'smmute' binary.
- Jump back to the host by exiting the screen using 'Ctrl-a d' and use the
following command to list the guests that are managed by lkvm tool.
# lkvm list
309 guest-309 running
276 guest-276 running
- Check the memory stat for the guests as well.
lkvm debug -a -i stat
- Power-off the guest by jumping to the appropriate screen and the guests would
shutdown gracefully with the following log:
# KVM session ended normally.
*Copyright (c) 2021, Arm Limited. All rights reserved.*
.. _buildroot guide: docs/infra/common/buildroot-boot.rst
.. _lkvm tool:
.. _kernel doc:
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment