My VGA Passthrough Notes

Introduction: What is VGA passthrough?

Answer: Attaching a graphics card to a Windows virtual machine, on a Linux host, for near-native graphical performance. This is evolving technology.

This post reflects my experience running my VGA passthrough setup for several years. It is not intended as a complete step-by-step guide, but rather a collection of notes to supplement the existing literature (notably, Alex Williamson’s VFIO blog) given my specific configuration and objectives. In particular, I am interested in achieving a smooth gaming experience, maintaining access to the attached graphics card from the Linux host, and staying as close to a stock Fedora configuration as possible. Hopefully, my notes will be useful to someone.

Objectives

  • Play Windows games with good performance and minimal virtualization-induced stuttering.
  • Use integrated graphics on the Linux host. However, make my graphics card available as a secondary device for CUDA, cryptocurrency mining, etc. when the guest isn’t running.
  • Use the facilities provided by Fedora for virtual machine and system administration, including libvirt and systemd.
  • Make all CPU cores available to the guest.
  • Make all CPU cores available to the host at all times.
  • Integrate with KDE’s PulseAudio instance and use software audio mixing.

Hardware

My target platform is a 2014-era Haswell Refresh desktop.

MotherboardASRock H97M Pro4
ProcessorIntel Core i5-4690 (4 cores/4 threads)
Memory24GB DDR3
Integrated graphicsIntel HD Graphics 4600
Passthrough graphicsNvidia GeForce GTX 1060 6GB
Storage 240GB SSD (host /, guest C:)
1TB HDD (host /home)
1TB HDD (guest E:)
Displays3x Dell 1905FP (1280×1024, 1x VGA/1x DVI-D)

Software

My motherboard is configured to boot using integrated graphics. On the ASRock H97M, this setting is buggy and I also have to turn on CSM support.

HostFedora 27 x86_64
Desktop environmentKDE 5
Hypervisorlibvirt
GuestWindows 10 64-bit

Host configuration

Use Alex Williams’s VFIO passthrough guides as a reference. The first few entries begin with determining the capabilities of your hardware and configuring your kernel for VGA passthrough.

Some notes from my setup:

  • Since I only have one graphics card, IOMMU groups are not a problem on my consumer board. Thus, I do not need the ACS override patch.
  • The 1060 is an EFI-capable card, so I can boot the guest in EFI mode. This means I do not have to deal with VGA arbitration.
  • I do not bind my 1060 to pci-stub because I want to use its compute capabilities on the host. In this case, the proprietary nvidia driver will happily be bound/unbound automatically by libvirt.
    • Sometimes nouveau is stubborn about loading, even when it is blacklisted, so I also boot with nouveau.modeset=0, which allows me to modprobe -r nouveau at runtime.
    • Unbinding nvidia while processes are still running on it will cause the card to unrecoverably lock up.
  • Use options kvm ignore_msrs=1 to avoid fatal CPU exceptions in the Windows guest in certain games.

Guest configuration

Use Alex Williamson’s “Our first VM” chapter as a reference, assuming your graphics card features EFI mode. Williamson also has a chapter for BIOS mode that is slightly more complicated. In addition, here are some tips I’ve discovered to achieve better and smoother performance.

CPU performance and stuttering

As described on Williamson’s blog, turn on and use hugepages for slightly better CPU performance. As of November 2017, Fedora 27’s selinux rules block libvirt from launching guests with hugepages. Active bug report.

Virtual machines stutter due to context switches and other CPU activity on the host, which is particularly noticeable in resource-intensive games like Grand Theft Auto IV. To some extent, this is a limitation of the technology. However, there are some things you can do to mitigate it:

  • Pin your virtual CPU’s to physical CPU’s in a 1:1 ratio.
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='4' threads='1'/>
  </cpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='1'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='3'/>
  </cputune>
  • Assign maximum real-time priority to the guest’s IO thread.
<iothreads>1</iothreads>
<cputune>
  <iothreadsched iothreads='1' scheduler='fifo' priority='99'/>
</cputune>

A word on CPU shielding

If you want to stop the virtual machine from stuttering completely, the only complete solution is to dedicate entire cores to the guest. You can do this at boot-time, using the isolcpus parameter, or at runtime, using cgroups and a helper script like cpuset.

Neither solution is acceptable to me given that I only have 4 physical cores. (In hindsight, it would have been worth paying more for a Core i7.) In my case, the virtual CPU pinning and scheduling tweaks reduced the stuttering to a comfortable level.

Disk performance

For much-improved IO performance, turn on the data plane implementation in virtio-scsi. This seems to fix slow, high-latency disk writing for tasks like Steam downloads.

Audio configuration

libvirt’s emulated ich6/ich9 sound card works fine for sound output and is actively developed. Stick with it.

You have several options for routing the audio from the guest to the host:

  • Keep the virt-manager graphical monitor open, even though the guest does not even use the emulated display. Simple and easy.
  • Configure Qemu to connect to your desktop environment’s PulseAudio instance. Slightly harder to set up, but no monitor needed.
  • Passthrough a PCI device, whether that’s the graphics card’s audio controller or the integrated audio controller. You lose access to that device on the host.
  • Passthrough a USB audio controller and use a loopback cable. Needs extra hardware and is kind of a kludge, if you ask me.
  • Send audio over the network using something like JACK. Crazy difficult to set up between Windows and Linux.

Using PulseAudio

I choose to connect to PulseAudio, which I’ll describe in detail here.

First, configure a virtual input in PulseAudio by opening an anonymous socket. This should work regardless of your DE.

# in ~/.config/pulse/default.pa
.include /etc/pulse/default.pa
load-module module-native-protocol-unix auth-anonymous=1 socket=/tmp/ryan-pulse

Then add the required Qemu options to libvirt.

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  ...
  <qemu:commandline>
    <qemu:env name='QEMU_AUDIO_DRV' value='pa'/>
    <qemu:env name='QEMU_PA_SERVER' value='unix:/tmp/ryan-pulse'/>
  </qemu:commandline>

On Fedora, selinux prohibits this configuration out of the box. You can let Qemu fail to start and then add your own exception to whitelist it.

setsebool -P virt_use_xserver 1
ausearch -c 'qemu-system-x86' --raw | audit2allow -M my-qemusystemx86
semodule -X 300 -i my-qemusystemx86.pp

Networking

For simplicity, I use macvtap in bridge mode attached to the host’s Ethernet interface. For host-to-guest communication, I have a second virtual NIC attached to a standalone bridge. This preserves compatibility with NetworkManager.

Sharing the keyboard and mouse

The emulated mouse provided by virt-manager is imprecise and produces too much CPU stuttering for gaming, so you’ll probably need a way to share your keyboard and mouse between the host and the guest.

For an easy, off-the-shelf solution, you can purchase Synergy. Be sure to enable relative mouse movements so you can play first-person shooters.

Synergy produces some weird behavior in a few games, like freezing in Halo 2 Vista, so I rolled my own script to attach/detach my USB keyboard and mouse to/from the guest.

#!/bin/bash

MOUSE='046d c52b'
KEYBOARD='2516 0009'

DOMAIN=$1
if [[ -z "$DOMAIN" ]]; then
        echo Usage: $0 "DOMAIN [switch back timer]"
        exit 0
fi
SWITCHTIME=$2
DEVICE_FILE=''
export LIBVIRT_DEFAULT_URI='qemu:///system'

function make_device()
{
        vendor=$1
        product=$2
        file="/tmp/hostdev-$vendor:$product.xml"
        echo "<hostdev mode='subsystem' type='usb'>" >$file
        echo '  <source>' >>$file
        echo "    <vendor id='0x$vendor'/>" >>$file
        echo "    <product id='0x$product'/>" >>$file
        echo '  </source>' >>$file
        echo '</hostdev>' >>$file
        DEVICE_FILE=$file
}
function attach_usb()
{
        vendor=$1
        product=$2
        make_device $vendor $product
        virsh attach-device "$DOMAIN" $DEVICE_FILE
}

function detach_usb()
{
        vendor=$1
        product=$2
        make_device $vendor $product
        virsh detach-device "$DOMAIN" $DEVICE_FILE
}

function usb_is_attached()
{
        vendor=$1
        product=$2
        virsh dumpxml "$DOMAIN" | grep -q "<vendor id='0x$vendor'/>" && \
                virsh dumpxml "$DOMAIN" | grep -q "<product id='0x$product'/>"
}

if usb_is_attached $KEYBOARD; then
        detach_usb $MOUSE
        detach_usb $KEYBOARD
elif [[ -n "$SWITCHTIME" ]]; then
        attach_usb $MOUSE
        attach_usb $KEYBOARD
        sleep $SWITCHTIME
        detach_usb $MOUSE
        detach_usb $KEYBOARD
else
        attach_usb $MOUSE
        attach_usb $KEYBOARD
fi

exit 0

In Windows, I have an AutoHotkey script to send the “switch back” command over ssh.

Sharing files

Besides Windows file sharing, you may be interested in Swish, an SFTP plugin for Windows Explorer.

Leave a Reply