Updated: February 17, 2018

Introduction: What is VGA passthrough?

Answer: Attaching a graphics card to a Windows virtual machine, on a Linux host, for near-native graphical performance. This is evolving technology.

VGA passthrough demonstration with Sonic Generations

This webpage reflects my experience running my VGA passthrough setup for several years. It is not intended as a complete step-by-step guide, but rather a collection of notes to supplement the existing literature (notably, Alex Williamson's VFIO blog) given my specific configuration and objectives. In particular, I am interested in achieving a smooth gaming experience, maintaining access to the attached graphics card from the Linux host, and staying as close to a stock Fedora configuration as possible. Hopefully, my notes will be useful to someone.

Objectives

Hardware

My target platform is a 2014-era Haswell Refresh desktop.

Motherboard ASRock H97M Pro4
Processor Intel Core i5-4690 (4 cores/4 threads)
Memory 24GB DDR3
Integrated graphics Intel HD Graphics 4600
Passthrough graphics Nvidia GeForce GTX 1060 6GB
Storage 240GB SSD (host /, guest C:)
1TB HDD (host /home)
1TB HDD (guest E:)
Displays 3x Dell 1905FP (1280x1024, 1x VGA/1x DVI-D)

Software

My motherboard is configured to boot using integrated graphics. On the ASRock H97M, this setting is buggy and I also have to turn on CSM support.

Host Fedora 27 x86_64
Desktop environment KDE 5
Hypervisor libvirt
Guest Windows 10 64-bit

Host configuration

Use Alex Williams's VFIO passthrough guides as a reference. The first few entries begin with determining the capabilities of your hardware and configuring your kernel for VGA passthrough.

Some notes from my setup:

Guest configuration

Use Alex Williamson's "Our first VM" chapter as a reference, assuming your graphics card features EFI mode. Williamson also has a chapter for BIOS mode that is slightly more complicated. In addition, here are some tips I've discovered to achieve better and smoother performance.

CPU performance and stuttering

As described on Williamson's blog, turn on and use hugepages for slightly better CPU performance. As of November 2017, Fedora 27's selinux rules block libvirt from launching guests with hugepages. Active bug report.

Virtual machines stutter due to context switches and other CPU activity on the host, which is particularly noticeable in resource-intensive games like Grand Theft Auto IV. To some extent, this is a limitation of the technology. However, there are some things you can do to mitigate it:

  1. Pin your virtual CPU's to physical CPU's in a 1:1 ratio.

    <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='4' threads='1'/>
    </cpu>
    <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='1'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='3'/>
    </cputune>
  2. Assign maximum real-time priority to the guest's IO thread.

    <iothreads>1</iothreads>
    <cputune>
    <iothreadsched iothreads='1' scheduler='fifo' priority='99'/>
    </cputune>
  3. In the Windows guest, force MSI interrupts on for all PCI devices using a utility like this one. Williamson on why this helps.

A word on CPU shielding

If you want to stop the virtual machine from stuttering completely, the only complete solution is to dedicate entire cores to the guest. You can do this at boot-time, using the isolcpus parameter, or at runtime, using cgroups and a helper script like cpuset.

Neither solution is acceptable to me given that I only have 4 physical cores. (In hindsight, it would have been worth paying more for a Core i7.) In my case, the virtual CPU pinning and scheduling tweaks reduced the stuttering to a comfortable level.

Disk performance

For much-improved IO performance, turn on the data plane implementation in virtio-scsi. This seems to fix slow, high-latency disk writing for tasks like Steam downloads.

Audio configuration

libvirt's emulated ich6/ich9 sound card works fine for sound output and is actively developed. Stick with it.

You have several options for routing the audio from the guest to the host:

Using pulseaudio

I choose to connect to pulseaudio, which I'll describe in detail here.

  1. Configure a virtual input in pulseaudio by opening an anonymous socket. This should work regardless of your DE.
    # in ~/.config/pulse/default.pa
    .include /etc/pulse/default.pa
    load-module module-native-protocol-unix auth-anonymous=1 socket=/tmp/ryan-pulse
  2. Add the required Qemu options to libvirt.
    <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
    ...
    <qemu:commandline>
    <qemu:env name='QEMU_AUDIO_DRV' value='pa'/>
    <qemu:env name='QEMU_PA_SERVER' value='unix:/tmp/ryan-pulse'/>
    </qemu:commandline>
  3. On Fedora, selinux prohibits this configuration out of the box. You can let Qemu fail to start and then add your own exception to whitelist it.
    setsebool -P virt_use_xserver 1
    ausearch -c 'qemu-system-x86' --raw | audit2allow -M my-qemusystemx86
    semodule -X 300 -i my-qemusystemx86.pp

Networking

For simplicity, I use macvtap in bridge mode attached to the host's Ethernet interface. For host-to-guest communication, I have a second virtual NIC attached to a standalone bridge. This preserves compatibility with NetworkManager.

Sharing the keyboard and mouse

The emulated mouse provided by virt-manager is imprecise and produces too much CPU stuttering for gaming, so you'll probably need a way to share your keyboard and mouse between the host and the guest.

For an easy, off-the-shelf solution, you can purchase Synergy. Be sure to enable relative mouse movements so you can play first-person shooters.

Synergy produces some weird behavior in a few games, like freezing in Halo 2 Vista, so I rolled my own script to attach/detach my USB keyboard and mouse to/from the guest.

#!/bin/bash

MOUSE='046d c52b'
KEYBOARD='2516 0009'

DOMAIN=$1
if [[ -z "$DOMAIN" ]]; then
        echo Usage: $0 "DOMAIN [switch back timer]"
        exit 0
fi
SWITCHTIME=$2
DEVICE_FILE=''
export LIBVIRT_DEFAULT_URI='qemu:///system'

function make_device()
{
        vendor=$1
        product=$2
        file="/tmp/hostdev-$vendor:$product.xml"
        echo "<hostdev mode='subsystem' type='usb'>" >$file
        echo '  <source>' >>$file
        echo "    <vendor id='0x$vendor'/>" >>$file
        echo "    <product id='0x$product'/>" >>$file
        echo '  </source>' >>$file
        echo '</hostdev>' >>$file
        DEVICE_FILE=$file
}
function attach_usb()
{
        vendor=$1
        product=$2
        make_device $vendor $product
        virsh attach-device "$DOMAIN" $DEVICE_FILE
}

function detach_usb()
{
        vendor=$1
        product=$2
        make_device $vendor $product
        virsh detach-device "$DOMAIN" $DEVICE_FILE
}

function usb_is_attached()
{
        vendor=$1
        product=$2
        virsh dumpxml "$DOMAIN" | grep -q "<vendor id='0x$vendor'/>" && \
                virsh dumpxml "$DOMAIN" | grep -q "<product id='0x$product'/>"
}

if usb_is_attached $KEYBOARD; then
        detach_usb $MOUSE
        detach_usb $KEYBOARD
elif [[ -n "$SWITCHTIME" ]]; then
        attach_usb $MOUSE
        attach_usb $KEYBOARD
        sleep $SWITCHTIME
        detach_usb $MOUSE
        detach_usb $KEYBOARD
else
        attach_usb $MOUSE
        attach_usb $KEYBOARD
fi

exit 0

In Windows, I have an AutoHotkey script to send the "switch back" command over ssh.

Sharing files

Besides Windows file sharing, you may be interested in Swish, an SFTP plugin for Windows Explorer.