Virtual Reality VM: Part 1 - Setup and Configuration

The last time I attempted to passthrough a video card was about 5 years ago. It did not work. I remember trying a dozen things, getting confused, and eventually giving up.

Hoping to create a giant VM for my workstation, I wanted to do like what I had done with the storage server. That would let me have the advantages of virtualization, from easy recovery and changes, to being able to reclaim and use some of the processing power for something else, to snapshots and backups that didn’t have large associated costs of enterprise software.

I have never found Windows backups particularly effective. Rarely do I lose files, but almost never seem to be able to get a complete state repair. I just reinstalled Windows, copied the files and went. That’s not true for enterprise backup, which has successfully repaired state, but I do not usually opt for that on home stuff.

When I started down this path I was convinced this time would be different, and it was, I did get it working, but there were quite a few bumps on the way. Virtual Reality is basically a Windows product, not that there is no support at all anywhere else, but it is limited, and was originally designed for Windows. I will be proceeding with a Windows 10 install.

I tend to do complete step by step instructions on these walkthroughs. They can get cumbersome (like it will with a Windows install), but I do this so people can observe exactly which steps I am doing. Whenever I am working through an issue with configuration, I often find “guides” that are simply half complete.

I run into an issue half way through a guide, it tells me to do something and I can’t. It references a screen no longer there. It said to click next and I got an error. I take a lot of screenshots and go through as many of them as I can, even if they are trivial decisions, because I think it helps people who are trying to work through a problem to see exactly what I saw. It may not solve their issue, but at least they won’t be left wondering if the problem is that I was not detailed enough in my description.

The first thing I did was toggle passthrough for the NVidia 1080 GTX. I went to the Manage screen, then the Hardware tab. Next, I selected the NVidia GP104 and GP104 High Definition Audio Controller.

Then, I clicked toggle passthrough (It succeeded).

I navigated to the Virtual Machine Section and clicked on Create / Register VM. This is not the first VR Virtual Machine I have built. The other one listed is the one I have been actively working on. This is a walkthrough for demonstration purposes.

Then I am at the create new virtual machine screen. I clicked next for a new one.

Type in the name and select Windows 10 as the operating system type. Then I clicked next.

I selected the primary datastore and clicked next.

I selected 16 CPUs and 64 GB of memory followed by clicking next. In the other VR VM I also added 1 TB of hard drive space.

I reviewed everything before clicking finish.

Next I navigated to the VR screen and clicked edit.

Here I clicked add another device and then selected PCI device.

I did this twice to create two PCI devices. They defaulted to the Ethernet controller and USB controller.

I selected the GP104 and its High Definition Audio Controller as my two devices. Then clicked save.

Next I needed to add the Windows 10 installation ISO. There are a couple of ways I know to do this. Microsoft will let you do this with their Media Creation Tool [1]. Here is a guide to get around that if you just want to skip the tool [2]. It is also possible to take an old installation DVD and rip the ISO. I downloaded the ISO from the guide. I needed to upload the installation ISO to the datastore. So I started by navigating to the storage section.

Then I selected Primary, and clicked on the datastore browser in the top left.

Here there is an upload button that can be used to navigate to a local file and upload it. For my case, the Windows 10 ISO has already been uploaded.

Next I had to configure the VR VM to boot to the installation ISO. For that I started by navigating back to the VR VM and clicked edit.

I configured the CD/DVD drive.

I started by clicking the checkbox next to connect.

Then I clicked the drop down that says Host device. I selected Datastore ISO file.

This will open up the datastore browser.

I navigated to the place where I just uploaded the Windows installer, and selected it.

This selected the ISO. I clicked on the expand next to the CD/DVD drive and click on the checkbox next to Connect at power on.

After this I needed to reserve the memory for this system. PCI devices have what is called DMA, or Direct Memory Access. This is when the operating system allows a PCI device to write directly to memory. As controlling this direct writing is not easy for the Hypervisor without assuming control of the PCI device directly, it makes sense that the memory needs to be reserved from the rest of the system, since a PCI device could be writing to it at any time.

After this I was ready to boot the VM for the first time, so I clicked on the large Windows icon. It is not uncommon to have to go back in and re-select the two checkboxes around the CD/DVD drive. I do not know why, but they tend to get unchecked. If so the system will not actually attempt to boot to the installation medium.

Here it pays to be quick. I had like 3 seconds to click on the VR window (and thus enable keyboard within the VM) and then hit any button. I had to reset once to get it, it caught me sleeping. When I finally hit it correctly, it started to boot the installer.

This is the first installation screen, it auto-selected US for me. I just clicked next.

Then I clicked on install now. In the bottom left there is a repair option. This is for when the install needs to be repaired or fallback to a restore point. It is probably more aptly used on physical rather than virtual machines.

Next it started setup.

Here it wants me to insert my Windows key. I clicked here I don’t have a key. I do have several keys. Since I do not have a Windows datacenter license, I cannot just use the same key over and over, I will need to use one of my personal keys for each VM.

I am mildly in support of this methodology. Windows was designed in an era predating virtualization. I do understand the argument that it is all still just one machine, therefore I should only need one key for each VM. Ultimately, this is a mild preference, but if I am to have two accounts, I pay twice in every other application. Here should be no different.

However, I do select I don’t have a product key here because I don’t want Microsoft to lock my key while I am figuring this out. I have 30 days to activate Windows per install. If I typed in my key here, then if through trying to figure out how to make the VR VM work, I need to reinstall Windows, it will register this key again and again. Usually after about 3 activations of my same key within a day or two, activation will start to reject this key. I then, typically, have to call Microsoft up, explain the situation to them, and then they will release the key again, which can take a day or so. It’s just inconvenient. So I will skip entering my key until I am confident the system is working as I want it to.

On the next screen I select Windows 10 Pro.

I agree to the absurd EULA that I am certain nobody reads. Quick aside, it is a legal fiction that one can agree to terms that are illegal. Microsoft can put things in the EULA that are not allowed, and try to convince people not to sue them, but one cannot agree to illegal things. This is part of what makes law difficult, because it takes a lawyer to have any idea what in the EULA is even enforceable.

That being true, even if it is enforceable, it may have restrictions beyond just a baseline of the corporation wins. I really think that law is getting a bit too far out from how a common person perceives it as working. There needs to be a kind of reset toward common understanding, especially with the legal language, which may as well be a programming language all its own at this point.

I clicked on Custom: Install Windows only.

I then selected New partition on the unallocated space.

I selected Apply on the whole size of the drive

Next, I agreed to let Windows create extra partitions.

Then I clicked next and Windows begins installing

I just waited here.

Windows would have eventually finished and restarted on its own, but I clicked to speed it up since I was just sitting there.

It started to boot up into the new install on its own.

For me it had one reboot in the middle of its first boot. Eventually it opened into the Windows starting screen below.

Then I could complete the install. I selected the United States region, and hit yes.

On the next screen, I selected the US keyboard layout.

Following that screen, I skipped adding a second keyboard layout.

It told me I am not important.

I then selected that this is Set up for personal use. I am not a Windows domain admin, and really don’t want to get into all of that IT setup for a home lab.

It then wanted me to create an account. I, however, did not want to create an account. I clicked on Offline account in the bottom left.

It failed to understand that I didn’t want to create an account.

It tried to manipulate me into creating an account rather heavy handedly, by calling the account-less install ‘Limited Experience’. I clicked on Limited Experience.

It then asked me to create a username

I tried to not create a username, but I think I am now at local permissions, so I created a VR account.

Here I could insert in whatever password I wanted.

It now asks me about the privacy settings I wish for. Personally I deselected all of them. Most of these don’t even work for a VM. Even then, I want to keep my data for myself (hence this whole project!). I clicked Accept.

It asked me if I want to keep my active history across devices. I say no here as well, because I don’t know how well this idea works from VMs

Then it asks me if I want to use Cortana. It won’t really work correctly here, there isn’t a microphone planned for this VM. Even then, I am not interested in more personal digital assistants. I clicked Not Now.

Finally I am taken to the home screen.

The next thing I needed to do was install the VMWare tools. This will allow Windows and ESXi to better communicate what is going on. It also installs most of the best drivers for the virtual devices. I pop out of the VM control window and click on the actions in general information. I select GuestOS and then Install VMWare tools.

Back in the VM, this will show up as a new DVD drive mounting. I clicked the box when it appeared.

It asks what I wanted to do with the new disc. I clicked run setup64.exe. Don’t worry if you missed it. Just open the file browser and double click the DVD icon to get it started.

It then asked permission to allow VMWare Tools to be installed. I clicked yes. Quick aside, anyone else remember when it was possible to just install things in the background without permissions like back in my high school years? Good times. Also, lots of viruses.

Then it asked me to continue installing, I clicked next.

I went with the default of Typical install and clicked next.

It then asked me to review the blank screen. I clicked next.

It finished installing and I clicked the Finish button. Ignore the other DVD, that is the Windows Installation disk being registered. I should just remove it once this is all done.

It asked if I wanted to restart the system to let the configuration take place. I clicked yes.

It rebooted and everyone was happy. Or at least mildly less displeased. Since it installed the video driver I can even resize the window and the screen will increase in size!

This is now the baseline from where I will attempt to get the graphics card to work in passthrough.

Alright, the naïve thing to do here would be to go to the NVidia download page and try to download NVidia drivers [3]. So, of course I did that, and it failed. There are actually two methods of downloading drivers from NVidia. The auto-detect did not work. I later discovered, and suspect, this is intentional because it is in a VM. The second method involves selecting the drivers needed [4]. After filling out the correct card and getting to the download page I have the drivers installing. At first I thought this was just going to work. However, once I got the drivers installed, nothing seemed to happen.

This is where things get weird. I plugged in a 4K TV I intended to serve as the monitor for the Virtual Reality VM. Additionally, I also had the wires laid and plugged in for all of the VR base controllers and transmitters. I am expecting that when the Graphics Card is in passthrough and the monitor is plugged in, Windows will simply recognize that the monitor is there and start using it. What I saw was nothing on the monitor.

I checked the device manager, and the drivers were there. Next, I checked the GeForce experience and it didn’t see the TV. I double and triple checked the cabling to make sure I had it plugged in correctly, it was. I had no idea where I had gone wrong.

So, it was time to do a little research. First let’s start with something a bit higher level. Does VMWare support GPU passthrough? Yes, it does, quite clearly. There is even a 4 part large blog post about all of the available methods for it [5]. I find references to the compatibility guide [6]. This thing is slow to load and doesn’t let me find my GPU. That probably means it is not supported.

But I am certain other people have done what I am trying to do. Here is a long article about doing this exact thing [7]. One thing I did notice, they are using AMD graphics cards, not NVidia. I check back on the VMWare blogs posts and I think I know what is happening. NVidia does not support consumer graphics cards using virtualization. The NVidia cards they do support are the Titans, super expensive cards that have explicit NVidia licenses for virtualization [8].

I’m sitting here thinking, this is crazy. NVidia wants to charge extra for me to use the video card in a VM? Why? I already bought the card. I am not trying to split it between multiple VMs, a dubious claim to needing more licenses as it is, since this isn’t software, its hardware. They should just be happy to let me use the card any way I wish.

I do not have a good explanation for this yet. But I have discovered the issue. This is the code 43 issue [9]. This is apparently fairly well known among hobbyists, but it is not anywhere on the professional blogs. Heck the blog from VMWare explicitly on Video Card passthrough does not mention this anywhere. NVidia’s drivers will explicitly detect if they are being loaded in a virtual machine, and then fail to work correctly, at least for consumer cards.

Talk about AMD having an advantage here. I very much hope AMD manages to knock NVidia down a peg, they need it. This is garbage. I found a few references to various things to try. Let’s go through a quick rundown.

In order to get around the NVidia drivers detecting they are being loaded in a VM, I will need a method for preventing that detection. This definitely exists on QEMU [10]. The VM can be told to report itself as a regular machine. Turns out there is also an option for this on VMWare [11][12]. It involves setting a flag on the vmware configuration:

hypervisor.cpuid.v0 = FALSE

This can be done by downloading the VMX file, adding this flag at the end, then re-uploading it using the datastore browser. Or it can be edited under the VM Options advanced options, which is just the GUI version of editing this file.

Click Edit on the VM Screen, then select VM Options

Add Parameter here can be used to add new configuration options

I tried adding the hypervisor.cpuid.v0 flag, but it did not work. At least, not at first. I tried adding the PCI configuration options that the VMWare blog references [13].

pciPassthru.use64bitMMIO=”TRUE”
pciPassthru.64bitMMIOSizeGB=8

I tried disabling MSI [14].

pciPassthru0.msiEnabled=FALSE

Which *kinda* worked, it blinked, showing the Video Card had tried to boot up, but then ESXi had a kernel panic and the whole system rebooted.

I tried disabling the svga.present option in the configuration. Under the idea that if the VMWare video wasn’t working, then it would try to use the other display.

Ultimately though, I think this person has the best take on it, so thank you bellacarico [15]:

0) ESXi I run
Version: 6.5.0 Update 2 (Build 10390116)
State: Normal (not connected to any vCenter Server)
1) Once you install the GPU make sure the Server BIOS prefers internal video card to any PCIE
If you don’t do this method below is not going not work; ESXi still boots but the progress bar is never to reach the end.
2) Enable pass-through on the card (via system management) and reboot
3) assign the card as PCI to the VM and reserve all the RAM you have assigned. Leave the default sVGA config untouched
4) Modify the parameter of the Windows10 VM adding:
hypervisor.cpuid.v0 = FALSE
pciPassthru0.msiEnabled = FALSE
Ref: https://nvidia.custhelp.com/app/answers/detail/a_id/4135/~/vmware-vdga-%2F-gpu-passthrough-requires-that-msi-is-disabled-on-vms
5) Once you boot into Windows I first had to make the NVIDIA display act as primary. Note, vmwaretools will bypass any attempt to disable the internal sVGA or it’s virtual display. So forget about device manager and head straight to the Windows Graphic Settings (right click on desktop/Display Settings):
a) highlight the display linked to the NVIDIA card
b) select “make this my main display”
c) under “multiple displays” select “show only on 2” (in my particular case)
NOTE: The only negative point since the sVGA is disable in the windows configuration, the ESXi remote console will only show up to the windows loading logo and get stuck there. That’s when I guess Windows disables the sVGA and set the NVIDIA to be the primary. Hence remote console is not working but you can install your own remote admin software I suppose.
NOTE2: Reading up in Internet it seems like pass-through works better with EFI rather than BIOS. I had a pre-existing VM in BIOS so didn’t want to reinstall so all this procedure refer to BIOS only.

The key here, at least for me, was before the NVidia driver completely installs after the native Windows install, I need to set the TV to the main display. After I had completed a fresh Windows install, I installed the NVidia drivers. Before then I could see the TV as a display option. I rolled back to a restore point before I installed the NVidia latest drivers. I then set the TV to the main display. After I installed the NVidia drivers to the latest, it continued to work, whereas before it would fail. I was able to remove the msiEnabled after I got it working, so I am unsure if that is actually important.

Now I still have the VMWare virtual screen, but I did have a working NVidia Graphics card. The next thing I needed to get working was USB. I will go through some of my USB troubles on the Home Assistant VM, which is where I solved them.

However, there is one specific thing that does need to be mentioned here. When passing through HID (I think this stands for Human Interface Device), USB passthrough does not work [16]. This is because ESXi is still running a CLI interface that can be accessed from the raw system. It can be explicitly enabled for a VM, but that did not work for me.

I ended up using PCI passthrough of the entire USB controller, and allowing Windows to pick up the entire chain. That is how I got the USB 3.0 component of VR to work. This is really a minor point, and didn’t take a lot of time to work through. I kinda wanted to do PCI passthrough of the entire controller anyways, because I did not want ESXI to interfere with the potential bandwidth requirements of VR.