infra/common: document virtualization support

Add a new document that explains the procedure to build software stack,
boot the system and launch multiple guests using kvmtool.

Change-Id: I8b898caeeb4de11562d48165381cd41bf0d8b4be
Signed-off-by: Vivek Gautam <vivek.gautam@arm.com>

docs/infra/common/virtualization.rst

+Virtualization using KVM
+========================
+
+.. contents::
+
+
+What is KVM?
+------------
+
+Kernel Virtual Machine (KVM) is a virtualization module built in the linux
+kernel which lets the user to turn linux into a hypervisor to allow hosting
+single/multiple isolated guests or virtual machine. In brief, KVM is a type-2
+hypervisor that requires a host OS to boot first, and the KVM module runs on
+top of that.
+
+KVM requires a processor with hardware virtualization extensions. Some of the
+architectural features in Arm v8-a profile that support hardware virtualization
+are -
+
+- A dedicated Exception level (EL2) for hypervisor code.
+- Support for trapping exceptions that change the core context or state.
+- Support for routing exceptions and virtual interrupts.
+- Two-stage memory translation, and
+- A dedicated exception for Hypervisor Call (HVC).
+
+Currently, KVM is part of linux kernel. Some of the features of KVM are:
+
+- Over-committing: KVM allows to allocate more virtualized CPU or memory for the
+  virtual machine than that of the host.
+- Thin provisioning: KVM allows to allocate and optimize the flexible storage
+  for the virtual machines.
+- Disk throttling: KVM allows to set limits for disk I/O requests.
+- Virtual CPU hot plug: KVM allows ability to increase the CPU count of the
+  virtual machine during run time.
+
+
+Virtualization on Neoverse Reference Design Platforms
+-----------------------------------------------------
+
+Virtualization using KVM hypervisor is supported on the Neoverse reference
+design plaforms. The subsequent sections below provide detailed instructions
+about booting up of two or more instances of guest OS's (or Virtual Machines,
+VMs) using `lkvm tool`_. Each of these guests can support upto NR_CPUS as
+vcpus, where NR_CPUS is the number of CPUs booted up by the host OS. There are
+instructions on using hardare virtualization features on the platform and enable
+use of virtualized devices, such as console, net, and disk etc.
+
+
+Overview of Native Linux KVM tool
+---------------------------------
+
+kvmtool is a lightweight tool for hosting KVM guests. As a pure virtualization
+tool it only supports guests using the same architecture, though it supports
+running 32-bit guests on those 64-bit architectures that allow this.
+
+The kvmtool supports a range of arm64 architectural features such as support for
+GIC-v2, v3, and ITS. It also supports device virtualization using emulated
+devices such as virtio device support for console, net, and disks, and using
+VFIO to allow PCI pass-through or direct device assignment.
+
+
+Booting multiple guests
+-----------------------
+
+Virtualization using KVM hypervisor requires a root filesystem from which
+kvmtool can be launched. Buildroot root filesystem supports the kvmtool package.
+It fetches the mainline kvmtool source and builds the kvmtool binary out of it.
+Detailed description on buildroot based booting ia available in `buildroot guide`_.
+Follow all the instructions in that document for building the platform software
+stack and booting upto buildroot before proceeding with the next steps.
+
+To boot two or more virtual machines on the host kernel with a kernel image and
+an initrd or a disk image, KVMtool virtual machine manager (VMM) (also called as
+lkvm tool) is used. Check help for 'lkvm run' command for options to launch
+guests.
+
+Launching multiple guests using lkvm:
+
+- Mount grub disk-image:
+  The buildroot filesystem required to perform kvm test is packaged in such a
+  way that the kernel image, and buildroot ramdisk image are copied to the
+  second partition of grub disk image that gets probed at /dev/vda2 in the host
+  kernel. After booting the platform this partition can be mounted as:
+
+   ::
+
+     mount /dev/vda2 /mnt
+
+- Launch VMs using lkvm:
+  For launching multiple VMs, 'screen' tool can be used to multiplex console
+  outputs so that one can switch between multiple workspaces. This tool helps by
+  providing a new console output pane for each guest. Use the following command
+  to launch guests using kvmtool with the available kernel and ramdisk images.
+
+   ::
+ 
+     screen -md -S "<screen_name>" lkvm run -k <path-to-linux-image> -i <path-to-ramdisk-image> --irqchip gicv3-its -c <nr-cpus> -m <allowed-mem> --console virtio --params "earltprintk=shm console=hvc0 root=/dev/vda
+
+  For example, to run the kernel available in mounted disk at /mnt as above use
+  the following command:
+
+  ::
+
+    screen -md -S "virt1" lkvm run -k /mnt/Image -i /mnt/ramdisk-buildroot.img --irqchip gicv3-its -c 4 -m 512 --console virtio --params "earltprintk=shm console=hvc0 root=/dev/vda"
+
+- Launch couple of more guests by repeating the above command and updating the
+  screen_name.
+
+  The launched screens can be viewed from the target by using the following
+  command:
+
+  ::
+
+    screen -ls
+
+- Select and switch to the desired screen to view boot-up logs from guest. Use
+  the following command to go to a specific screen:
+
+  ::
+
+    screen -r <screen_name>
+
+  - For example, list of screens are shown below:
+
+  ::
+
+    # screen -ls
+    There are screens on:
+        214.virt1       (Detached)
+        200.virt2       (Detached)
+
+  - Jump to the screen using:
+
+   ::
+
+     screen -r virt1
+
+  - Switch between multiple running guests to view the bootup logs of various
+    guests executing.
+
+- Perform simple cpu hotplug test to validate that guest kernel is functional.
+  Use the following command to do that:
+
+   ::
+
+     echo 0 > /sys/devices/system/cpu/cpu1/online
+     echo 0 > /sys/devices/system/cpu/cpu2/online
+
+     echo 1 > /sys/devices/system/cpu/cpu1/online
+     echo 1 > /sys/devices/system/cpu/cpu2/online
+
+   The CPUs should go offline and come back online with the above set of
+   commands.
+
+- Jump back to the host by exiting the screen using 'Ctrl-a d', and use the
+  following command to see how many guests are managed by lkvm tool:
+
+  ::
+
+    # lkvm list
+    PID NAME                 STATE
+    ------------------------------------
+    309 guest-309            running
+    276 guest-276            running
+
+- Power-off the guests by jumping to the right screen, and the guests would
+  shutdown fine with the following log:
+
+   ::
+
+      # KVM session ended normally.
+
+This completes the procedure to launch multiple VMs and terminate them.
+
+
+PCIe pass-through based device virtualization
+---------------------------------------------
+
+PCIe pass-through (also called as direct device assignment) allows a device to
+be assigned to a guest such that the guest runs the driver for the device
+without intervention of the hypervisor/host. This is one of the device
+virtualization technique besides para-virtualization.
+
+PCIe pass-through is achieved using  frameworks in linux kernel, such as VFIO,
+virtio, IOMMU, and pci. A smmu-test-engine (smmute) device that is available on
+the platform is used as a test device for this virtualization technique. The
+smmu-test-engine is a PCIe exerciser that generates DMA workloads and it uses
+arm-smmu-v3 to provide dma isolation. This device first probed in the host
+kernel can be assigned to the guest and the smmu-test-engine driver in the guest
+kernel can then manage the device directly.
+
+PCI pass-through using multiple guests and smmu test engine:
+
+- Boot the platform by following the `buildroot guide`_, and then ensure that
+  the smmu test engine device is probed correctly. Use the lspci command to
+  check for smmu test engine devices with pci BDF ids - 07:00.0, 07:00.3,
+  08:00.0 and 08:00.1.
+
+  ::
+
+    lspci
+
+- Verbose output of lspci will show the last four devices with above mentioned
+  pci BDF ids are managed by 'smmut-pci' kernel driver.
+
+  ::
+
+    lspci
+
+- Also check that the smmute-pci driver has probed the smmu test engine devices
+  properly, and a device extry exists for each of the four smmute devices.
+
+  ::
+
+    ls -l /dev/smmute*
+
+- Use one of the smmute devices (e.g. device 0000:08:00.1) to perform the PCI
+  pass-through. Detach the pcie device from its class driver and attach to
+  vfio-pci driver, as also explained in the `kernel doc`_.
+
+  ::
+
+    echo 0000:08:00.1 > /sys/bus/pci/devices/0000:08:00.1/driver/unbind
+    echo vfio-pci > /sys/bus/pci/devices/0000:08:00.1/driver_override
+    echo 0000:08:00.1 > /sys/bus/pci/drivers_probe
+
+- The kernel and ramdisk images to launch VMs are available in the second
+  partition of grub disk image that gets probed at /dev/vda2 in the host.
+  Mount this to use the images.
+
+   ::
+
+     mount /dev/vda2 /mnt
+
+- This mounted partition can also be shared with guest using 9p virtual
+  filesystem. A binary to run tests over smmute device is also available
+  in this partition. So after sharing the filesystem with a guest, tests
+  can be run on assigned smmute device to verify pci pass-through.
+
+- Launch VMs using lkvm tool that supports virtio-iommu and vfio drivers to
+  allow pci pass-through.
+
+   ::
+
+     screen -md -S "virt1" lkvm run -k /mnt/Image -i /mnt/ramdisk-buildroot.img --irqchip gicv3-its -c 2 -m 512 --console virtio --9p /mnt,hostshare --params "earlyprintk=shm console=hvc0 root=/dev/vda" --vfio-pci 0000:08:00.1 --viommu vfio;
+
+- Jump to the right screen to view boot-up logs from guest. Use following command
+  to go to a specific screen:
+
+   ::
+
+     screen -r virt1
+
+- After the guest boots up, mount the 9p filesytem to a mount point in the guest.
+  For example, use the following command to mount at /tmp
+
+   ::
+
+     mount -t 9p -o trans=virtio hostshare /tmp/
+     cd /tmp
+
+- Check that the smmu test engine is probed in the guest. The device will show a
+  different pci BDF id here in guest as compared to the id shown in host kernel.
+
+   ::
+
+     # lspci
+     00:00.0 Unassigned class [ff00]: ARM Device ff80
+
+     # ls -l /dev/smmute*
+     crw-------    1 root     root      235,   0 Jan  1 00:00 /dev/smmute0
+
+- From /tmp directory that contains the 'smmute' binary, run the test.
+
+   ::
+
+     ./smmute -s 0x100 -n 10
+
+- Jump back to the host by exiting the screen using 'Ctrl-a d', and check the
+  stats for the kvm session.
+
+   ::
+
+      lkvm debug -a -i stat
+
+- Launch couple more guests by repeating the above commands and updating the
+  screen_name, and device. For example,
+
+   ::
+
+     echo 0000:08:00.0 > /sys/bus/pci/devices/0000:08:00.0/driver/unbind
+     echo vfio-pci > /sys/bus/pci/devices/0000:08:00.0/driver_override
+     echo 0000:08:00.0 > /sys/bus/pci/drivers_probe
+
+     screen -md -S "virt2" lkvm run -k /mnt/Image -i /mnt/ramdisk-buildroot.img --irqchip gicv3-its -c 2 -m 512 --console virtio --9p /mnt,hostshare --params "earlyprintk=shm console=hvc0 root=/dev/vda" --vfio-pci 0000:08:00.0 --viommu vfio;
+
+- Perform test over smmu test engine in this second screen by mounting the 9p
+  filesystem and executing the 'smmute' binary.
+
+- Jump back to the host by exiting the screen using 'Ctrl-a d' and use the
+  following command to list the guests that are managed by lkvm tool.
+
+  ::
+
+    # lkvm list
+    PID NAME                 STATE
+    ------------------------------------
+    309 guest-309            running
+    276 guest-276            running
+
+- Check the memory stat for the guests as well.
+
+   ::
+
+      lkvm debug -a -i stat
+
+- Power-off the guest by jumping to the appropriate screen and the guests would
+  shutdown gracefully with the following log:
+
+   ::
+
+      # KVM session ended normally.
+
+
+--------------
+
+*Copyright (c) 2021, Arm Limited. All rights reserved.*
+
+.. _buildroot guide: docs/infra/common/buildroot-boot.rst
+.. _lkvm tool: https://github.com/lkvm/lkvm
+.. _kernel doc: https://www.kernel.org/doc/Documentation/driver-api/vfio.rst