Run Windows in KVM with dedicated GPU

This is also known as GPU or PCIe passthrough. All notes based and tested on Debian Buster.

Activate IOMMU in /etc/default/grub. On intel platforms replace amd_iommu with intel_iommu.

1
2
3

...
GRUB_CMDLINE_LINUX_DEFAULT="... amd_iommu=on ..."
...

and apply changes with

1

$ sudo update-grub && sudo reboot

Some tutorials suggest to blacklist the graphics driver, e.g. nouveau, however if the host GPU uses the same driver this is not a viable option. Instead, use the vfio driver for the VM GPU. Run

1

$ lspci -vnn

to identify the Vendor ID and Device ID of the devices for the VM. Some graphics cards use two or even more devices, usually one VGA controller, one Audio device and sometimes USB devices. Create and edit /etc/modprobe.d/vfio.conf and add the VID:DID pairs, e.g.

1

options vfio-pci ids=10de:1f82,10de:10fa

If for any reason another driver claims the device before vfio-pci is able to, check if the module is compiled into the kernel or not, e.g. if xhci_hcd is claiming the device instead check

1

$ grep -r CONFIG_USB_XHCI_HCD /boot/config-*

If the module is compiled in CONFIG_USB_XHCI_HCD=y then search for a unbind/rebind solution or recompile your kernel. If not CONFIG_USB_XHCI_HCD=m then add

1

softdep xhci_hcd pre: vfio-pci

to /etc/modprobe.d/vfio.conf

Add the following modules to /etc/initramfs-tools/modules

1
2
3
4
5
6

...
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
...

and apply changes with

1

$ sudo update-initramfs -u -k all && sudo reboot

Create and run iommu_info.sh (Source: https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF)

1
2
3
4
5
6
7
8

#!/bin/bash
shopt -s nullglob
for g in /sys/kernel/iommu_groups/*; do
        echo "IOMMU Group ${g##*/}:"
        for d in $g/devices/*; do
                echo -e "\t$(lspci -nns ${d##*/})"
        done;
done;

Check if the desired devices are listed in a separate IOMMU group, on most consumer grade hardware this will not be the case. If yes, skip the ACS override patch section. If no, kernel needs to be compiled with ACS override patch from https://queuecumber.gitlab.io/linux-acs-override/

See https://kernel-team.pages.debian.net/kernel-handbook/ch-common-tasks.html for some hints on how to quickly patch build a kernel on Debian.

Once the kernel with ACS override patch is installed edit /etc/default/grub

1
2
3

...
GRUB_CMDLINE_LINUX_DEFAULT="... pcie_acs_override=downstream ..."
...

and apply changes with

1

$ sudo update-grub && sudo reboot

Then add PCIe devices to Windows VM, run it and install drivers for graphics card. Nvidia drivers will refuse to start inside a VM throwing error 43 (see https://mathiashueber.com/fighting-error-43-nvidia-gpu-virtual-machine/).

Therefore it might be necessary to edit the VM config to hide KVM from guest

1

$ virsh edit <name_of_vm>

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

<features>
  <hyperv>
    ...
    <vendor_id state='on' value='1234567890ab'/>
  </hyperv>
  ...
  <kvm>
    <hidden state='on'/>
  </kvm>
  ...
  <ioapic driver='kvm'/>
  ...
</features>

Now the GPU driver should start as expected.