• Jean-Philippe Brucker's avatar
    vfio-pci: add MSI-X support · c9888d95
    Jean-Philippe Brucker authored
    
    
    Add virtual MSI-X tables for PCI devices, and create IRQFD routes to let
    the kernel inject MSIs from a physical device directly into the guest.
    
    It would be tempting to create the MSI routes at init time before starting
    vCPUs, when we can afford to exit gracefully. But some of it must be
    initialized when the guest requests it.
    
    * On the KVM side, MSIs must be enabled after devices allocate their IRQ
      lines and irqchips are operational, which can happen until late_init.
    
    * On the VFIO side, hardware state of devices may be updated when setting
      up MSIs. For example, when passing a virtio-pci-legacy device to the
      guest:
    
      (1) The device-specific configuration layout (in BAR0) depends on
          whether MSIs are enabled or not in the device. If they are enabled,
          the device-specific configuration starts at offset 24, otherwise it
          starts at offset 20.
      (2) Linux guest assumes that MSIs are initially disabled (doesn't
          actually check the capability). So it reads the device config at
          offset 20.
      (3) Had we enabled MSIs early, host would have enabled the MSI-X
          capability and device would return the config at offset 24.
      (4) The guest would read junk and explode.
    
    Therefore we have to create MSI-X routes when the guest requests MSIs, and
    enable/disable them in VFIO when the guest pokes the MSI-X capability. We
    have to follow both physical and virtual state of the capability, which
    makes the state machine a bit complex, but I think it works.
    
    An important missing feature is the absence of pending MSI handling. When
    a vector or the function is masked, we should rewire the IRQFD to a
    special thread that keeps note of pending interrupts (or just poll the
    IRQFD before recreating the route?). And when the vector is unmasked, one
    MSI should be injected if it was pending. At the moment no MSI is
    injected, we simply disconnect the IRQFD and all messages are lost.
    
    Reviewed-by: default avatarPunit Agrawal <punit.agrawal@arm.com>
    Signed-off-by: default avatarJean-Philippe Brucker <jean-philippe.brucker@arm.com>
    Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
    c9888d95