My VGA Passthrough Notes

tech  ·  · 5 min read

Introduction: What is VGA passthrough?

Answer: Attaching a graphics card to a Windows virtual machine, on a Linux host, for near-native graphical performance. This is evolving technology.

This post reflects my experience running my VGA passthrough setup for several years. It is not intended as a complete step-by-step guide, but rather a collection of notes to supplement the existing literature (notably, Alex Williamson’s VFIO blog) given my specific configuration and objectives. In particular, I am interested in achieving a smooth gaming experience, maintaining access to the attached graphics card from the Linux host, and staying as close to a stock Fedora configuration as possible. Hopefully, my notes will be useful to someone.

Objectives

Hardware

My target platform is a 2014-era Haswell Refresh desktop.

Software

My motherboard is configured to boot using integrated graphics. On the ASRock H97M, this setting is buggy and I also have to turn on CSM support.

Host configuration

Use Alex Williams’s VFIO passthrough guides as a reference. The first few entries begin with determining the capabilities of your hardware and configuring your kernel for VGA passthrough.

Some notes from my setup:

Guest configuration

Use Alex Williamson’s “Our first VM” chapter as a reference, assuming your graphics card features EFI mode. Williamson also has a chapter for BIOS mode that is slightly more complicated. In addition, here are some tips I’ve discovered to achieve better and smoother performance.

CPU performance and stuttering

As described on Williamson’s blog, turn on and use hugepages for slightly better CPU performance. As of November 2017, Fedora 27’s selinux rules block libvirt from launching guests with hugepages. Active bug report.

Virtual machines stutter due to context switches and other CPU activity on the host, which is particularly noticeable in resource-intensive games like Grand Theft Auto IV. To some extent, this is a limitation of the technology. However, there are some things you can do to mitigate it:

  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='4' threads='1'/>
  </cpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='1'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='3'/>
  </cputune>
<iothreads>1</iothreads>
<cputune>
  <iothreadsched iothreads='1' scheduler='fifo' priority='99'/>
</cputune>

A word on CPU shielding

If you want to stop the virtual machine from stuttering completely, the only complete solution is to dedicate entire cores to the guest. You can do this at boot-time, using the isolcpus parameter, or at runtime, using cgroups and a helper script like cpuset.

Neither solution is acceptable to me given that I only have 4 physical cores. (In hindsight, it would have been worth paying more for a Core i7.) In my case, the virtual CPU pinning and scheduling tweaks reduced the stuttering to a comfortable level.

Disk performance

For much-improved IO performance, turn on the data plane implementation in virtio-scsi. This seems to fix slow, high-latency disk writing for tasks like Steam downloads.

Audio configuration

libvirt’s emulated ich6/ich9 sound card works fine for sound output and is actively developed. Stick with it.

You have several options for routing the audio from the guest to the host:

Using PulseAudio

I choose to connect to PulseAudio, which I’ll describe in detail here.

First, configure a virtual input in PulseAudio by opening an anonymous socket. This should work regardless of your DE.

# in ~/.config/pulse/default.pa
.include /etc/pulse/default.pa
load-module module-native-protocol-unix auth-anonymous=1 socket=/tmp/ryan-pulse

Then add the required Qemu options to libvirt.

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  ...
  <qemu:commandline>
    <qemu:env name='QEMU_AUDIO_DRV' value='pa'/>
    <qemu:env name='QEMU_PA_SERVER' value='unix:/tmp/ryan-pulse'/>
  </qemu:commandline>

On Fedora, selinux prohibits this configuration out of the box. You can let Qemu fail to start and then add your own exception to whitelist it.

setsebool -P virt_use_xserver 1
ausearch -c 'qemu-system-x86' --raw | audit2allow -M my-qemusystemx86
semodule -X 300 -i my-qemusystemx86.pp

Networking

For simplicity, I use macvtap in bridge mode attached to the host’s Ethernet interface. For host-to-guest communication, I have a second virtual NIC attached to a standalone bridge. This preserves compatibility with NetworkManager.

Sharing the keyboard and mouse

The emulated mouse provided by virt-manager is imprecise and produces too much CPU stuttering for gaming, so you’ll probably need a way to share your keyboard and mouse between the host and the guest.

For an easy, off-the-shelf solution, you can purchase Synergy. Be sure to enable relative mouse movements so you can play first-person shooters.

Synergy produces some weird behavior in a few games, like freezing in Halo 2 Vista, so I rolled my own script to attach/detach my USB keyboard and mouse to/from the guest.

#!/bin/bash

MOUSE='046d c52b'
KEYBOARD='2516 0009'

DOMAIN=$1
if [[ -z "$DOMAIN" ]]; then
        echo Usage: $0 "DOMAIN [switch back timer]"
        exit 0
fi
SWITCHTIME=$2
DEVICE_FILE=''
export LIBVIRT_DEFAULT_URI='qemu:///system'

function make_device()
{
        vendor=$1
        product=$2
        file="/tmp/hostdev-$vendor:$product.xml"
        echo "<hostdev mode='subsystem' type='usb'>" >$file
        echo '  <source>' >>$file
        echo "    <vendor id='0x$vendor'/>" >>$file
        echo "    <product id='0x$product'/>" >>$file
        echo '  </source>' >>$file
        echo '</hostdev>' >>$file
        DEVICE_FILE=$file
}
function attach_usb()
{
        vendor=$1
        product=$2
        make_device $vendor $product
        virsh attach-device "$DOMAIN" $DEVICE_FILE
}

function detach_usb()
{
        vendor=$1
        product=$2
        make_device $vendor $product
        virsh detach-device "$DOMAIN" $DEVICE_FILE
}

function usb_is_attached()
{
        vendor=$1
        product=$2
        virsh dumpxml "$DOMAIN" | grep -q "<vendor id='0x$vendor'/>" && \
                virsh dumpxml "$DOMAIN" | grep -q "<product id='0x$product'/>"
}

if usb_is_attached $KEYBOARD; then
        detach_usb $MOUSE
        detach_usb $KEYBOARD
elif [[ -n "$SWITCHTIME" ]]; then
        attach_usb $MOUSE
        attach_usb $KEYBOARD
        sleep $SWITCHTIME
        detach_usb $MOUSE
        detach_usb $KEYBOARD
else
        attach_usb $MOUSE
        attach_usb $KEYBOARD
fi

exit 0

In Windows, I have an AutoHotkey script to send the “switch back” command over ssh.

Sharing files

Besides Windows file sharing, you may be interested in Swish, an SFTP plugin for Windows Explorer.