Some notes and references on the GPU passthrough settings.
The guide is a sufficient condition for GPU passthrough, not a necessary one. Keep in mind that the passthrough settings highly depend on your Proxmox version and hardware. This note is only expected to work on Proxmox VE 7.3 combined with the exact hardware. For other Proxmox versions or hardwares, please refer to the potential references instead.
Motherboard: ASUS TUF GAMING B550M-PLUS (WI-FI) CPU: AMD Ryzen 3 3100 4-Core Processor GPU: NVIDIA GeForce GTX 1650 SUPER
The GPU is plugged into the primary (top) PCIe slot.
A external monitor is connected to the GPU through HDMI.
Installing Proxmox VE
You can skip this section is you already installed Proxmox VE on your machine.
Follow the official documentation to install Proxmox VE.
Specifically, follow the steps below:
Download the ISO file here.
In my case I downloaded Proxmox VE 7.3:
Burn it to a USB stick with Rufus or
Configure BIOS to boot from USB to start the installation process.
You will need to enter:
- Proxmox Host Password & Your Email
- Hostname (FQDN) such as
- IP Address (CIDR) such as
- Gateway such as
- DNS Server such as
You can connect through SSH:
ssh firstname.lastname@example.org # or optionally with port tunneling: ssh email@example.com -L 8006:127.0.0.1:8006 # or optionally with SPICE support ssh firstname.lastname@example.org -L 8006:127.0.0.1:8006 -L 3128:127.0.0.1:3128
and connect to the GUI through HTTPS (accept unsigned certificate and with username
root), not HTTP. Example:
https://192.168.0.100:8006 # or with port tunneling: https://127.0.0.1:8006
Download Ubuntu ISO through GUI:
Datacenter > pve > local (pve) > ISO Images > Download from URL
Take Ubuntu 22.04 as an example:
URL: https://releases.ubuntu.com/jammy/ubuntu-22.04.2-desktop-amd64.iso File name: ubuntu-22.04.2-desktop-amd64.iso
Select Ubuntu 22.04 ISO
Default; Optionally select
SSD emulationif you are using a SSD instead of a HDD.
Increase Cores; CPU Type can be changed to
Increase Memory; Optionally disable
Add GPU as PCI(e) device:
Hardware > Add > PCI Device
GeForce GTX 1650 SUPER:
q35, you don’t need to check
PCI-Express. I’m not quite sure if checking it will cause issues…
In my vague memory, checking
Primary GPU seems to work, but may require changing the
Hardware > Display to
Virt-IO GPU. I’m not sure about this though.
Take a snapshot and name it as
Start the VM with
Console and go through normal Ubuntu installation.
After Reboot, power off the VM and take a snapshot with name
Note that you can right-click and
Quitthe initial Ubuntu popup.
You can now set the
CD/DVD Drive to
Do not use any media.
If anything go wrong in the future steps, revert to the previous snapshot. You may also want to run
sudo apt install -y openssh-server and connect to the VM through SSH if your Console VNC stuck at some point during boot.
Configure Host BIOS (Press
Make sure virtualization technologies are enabled. The default settings are enabled on this motherboard so you don’t need to change anything by default.
Configure GRUB on Proxmox Host (Terminal/SSH)
/etc/default/gruband change the following line:
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt initcall_blacklist=sysfb_init"
(without new line)
Save and close the file and execute
Configure Modules on Proxmox Host (Terminal/SSH)
/etc/modulesand add the following lines:
vfio vfio_iommu_type1 vfio_pci vfio_virqfd
Save and close the file.
Reboot the host.
The VFIO driver is an IOMMU/device agnostic framework for exposing direct device access to userspace, in a secure, IOMMU protected environment.
Create VM on Proxmox Host (Web GUI)
In our setting, we don’t need to deal with interrupts, roms, etc.
Verify GPU Functions
sudo lspci -v
00:10.0 VGA compatible controller: NVIDIA Corporation TU116 [GeForce GTX 1650 SUPER] (rev a1) (prog-if 00 [VGA controller]) Subsystem: ASUSTeK Computer Inc. TU116 [GeForce GTX 1650 SUPER] Physical Slot: 16 Flags: bus master, fast devsel, latency 0, IRQ 40 Memory at fd000000 (32-bit, non-prefetchable) [size=16M] Memory at e0000000 (64-bit, prefetchable) [size=256M] Memory at f0000000 (64-bit, prefetchable) [size=32M] I/O ports at f000 [size=128] Expansion ROM at 000c0000 [disabled] [size=128K] Capabilities:  Power Management version 3 Capabilities:  MSI: Enable+ Count=1/1 Maskable- 64bit+ Capabilities:  Express Legacy Endpoint, MSI 00 Kernel driver in use: nouveau Kernel modules: nvidiafb, nouveau
Ubuntu Drivers Devices
== /sys/devices/pci0000:00/0000:00:10.0 == modalias : pci:v000010DEd00002187sv00001043sd00008769bc03sc00i00 vendor : NVIDIA Corporation model : TU116 [GeForce GTX 1650 SUPER] driver : nvidia-driver-470 - distro non-free driver : nvidia-driver-525 - distro non-free driver : nvidia-driver-510 - distro non-free driver : nvidia-driver-525-server - distro non-free driver : nvidia-driver-470-server - distro non-free driver : nvidia-driver-450-server - distro non-free driver : nvidia-driver-525-open - distro non-free recommended driver : nvidia-driver-515-open - distro non-free driver : nvidia-driver-515 - distro non-free driver : nvidia-driver-515-server - distro non-free driver : xserver-xorg-video-nouveau - distro free builtin
sudo apt install -y nvidia-driver-525 reboot nvidia-smi
The output should be normal
nvidia-smioutput. If you run
lspcinow, you will find that the driver & modules have been updated:
Kernel driver in use: nvidia Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
Power off the VM and create snapshot:
If you observed the following error, you may forget to reboot:
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.
sudo apt install -y nvidia-cuda-toolkit
This command may uninstall the previously installed driver, causing
nvidia-smito be removed.
You can attempt to reinstall the driver and do not press
sudo apt install nvidia-driver-525
and observe which packages will be removed, in our case:
The following packages will be REMOVED: libnvidia-compute-495 libnvidia-compute-510 libnvidia-ml-dev nvidia-cuda-dev nvidia-cuda-toolkit
So we know we need to install:
sudo apt install -y nvidia-driver-510
nvidia-smishould work after the reboot.
You can skip all this hassle if you know how to run CUDA inside NVIDIA Docker, but this is out of the scope of this note.
~/hello.cuand paste the code from The real “Hello World!” for CUDA.
Compile and run:
cd ~ nvcc ./hello.cu ./a.out
It should output:
We are now all done, power off the VM and create snapshot
Side Notes for Proxmox VE
apt update Proxmox repository is not signed
Full error message:
E: Failed to fetch https://enterprise.proxmox.com/debian/pve/dists/bullseye/InRelease 401 Unauthorized [IP: 220.127.116.11 443] E: The repository 'https://enterprise.proxmox.com/debian/pve bullseye InRelease' is not signed.
Datacenter > $NODE_NAME > Updates > Repositories and disable
Then add a new repository by selecting
- Install Proxmox VE on Debian 11 Bullseye
- The repository ‘https://enterprise.proxmox.com/debian/pve bullseye InRelease’ is not signed
Adding a new HDD
Datacenter > $NODE_NAME > Disks, select the new disk and click
You can now use the disk in one of the following:
Hardware tab, change
SPICE (qxl) and add an Audio device.
Download the SPICE connection file in the
Console tab by
Console > SPICE. Open the file with virt-manager.
Sidenote: SPICE also supports Clipboard sharing.
/dev/mapper/pve-root 100% Full on Host
It may be due to large log files in
/var/log/. You may delete them to save space.
Discard is use to free space on your physical storage, you delete a file inside a guest vm for example. without discard, when you delete a file, the space is still used on the the physical storage, so you can’t use it in another vm.
Ballooning affects performance so we disable it. It allows for the ram to grow as needed but we don’t want that for stability and performance reasons.
pci_hp_register failed with error -16 During VM Boot
In my environment, ACPI causes this problem, just turn it off in OPTIONS and the problem goes away.
In my case, the iso is not signed, and can’t be started using SecureBoot (which is the default on the latest OVMF shipped with Proxmox). If you hit the escape key while it says ‘Startup boot options’ (and before it says the UEFI message about saying hitting escape that doesn’t actually work), then you get into the UEFI menu which you can use to disable Secure Boot and then boot the iso
Hit escape when you see the Proxmox logo on boot. Then Device Manager -> Secure Boot Configuration Make sure Attempt Secure Boot is disabled. If not, disable it and restart system.
Side Notes for GPU Passthrough
lspci -v lsmod | grep nvidia sudo dmesg | grep nvidia cat /var/log/syslog | grep nvidia nvidia-smi
dmesg | grep -i dmar
On Ryzen systems, there is no output from
grep -i dmar. This is normal.
lspci -v Shows
!!! Unknown header type 7f For VM GPU
In the Hardware settings, try uncheck the
SeaBIOS Instead of OVMF (UEFI)
I believe using the VMs with a UEFI BIOS may have been the issue or at least contributed to it. I don’t have much time to look into this at the moment, but can look again if you still have issues.
So I’ve created VM with:
- BIOS: Default (SeaBIOS)
- Machine: Default (i440fx) And it just worked!
CUDA-Compiled Excutables Freeze On Launch
lspci -v and
nvidia-smi all work correctly in this case. This issue also seem to make the VM BIOS very slow and unresponsive (unverified).
Any CUDA-compiled executable should work, but you can try out The real “Hello World!” for CUDA.
This issue may arise from the PCI(e) pass through due to a breaking change introduced in Proxmox VE 7.2.
initcall_blacklist=sysfb_initto the kernel parameters on the Proxmox host. No need to reset the GPU via scripts but do early bind it to vfio-pci.
By the way, I also found better solution on a reddit post. It is adding “
initcall_blacklist=sysfb_init” to kernel parameter. No need “
video=efifb:off” or “
video=simplefb:off” in kernel parameter.
edit your grub, and just add:
Finally got my gtx1070 passed through
Host BIOS Settings
Potentially you can try out the followings:
Advanced\AMD CBS\IOMMU: Enabled Boot\CSM (Compatibility Support Module)\Launch CSM: Disabled Boot\Boot Configuration\Fast Boot: Disabled
and enable other Virtualization Technologies such as
Other Potential References
- Pci passthrough
- Proxmox PCI(e) Passthrough in 2 minutes
- Yet Another Ultimate GPU Passthrough Guide for Beginners(VM)
- The Ultimate Beginner’s Guide to GPU Passthrough (Proxmox, Windows 10)
- Proxmox GPU Passthrough
- Proxmox GPU Passthrough to VM
- 在 VM 裡面使用顯示卡