Ask HN: How are you using Nvidia cards on Linux with its VRAM issues?

2 points

5 months ago

I recently discovered that on Linux, NVIDIA GPUs will not seamlessly access system memory if the GPU's memory gets full where as on Windows it does.

After 25 years on Windows I recently switched to Linux over the holiday break and came across this with my GeForce 750 Ti which only has 2 GB of GPU memory.

After opening a few Firefox and Ghostty terminals (both apps are hardware accelerated), I notice system instability and things crash or my Wayland compositor (niri) starts failing in unpredictable ways. All journalctl logs point to NVIDIA / kernel errors allocating resources.

Turns out it's likely due to the NVIDIA drivers on Linux not using system memory if the GPU memory is overburdened but on Windows this happened automatically. For reference this is with the official proprietary NVIDA drivers (latest open dkms 580 series for this card).

It's why on Windows I was able to open as many apps as I wanted, record x264 videos with OBS and play games without a single crash in literally 10 years.

There is a multi-year thread on the NVIDIA developer forums with hundreds of people reporting this but NVIDIA says it's not a problem https://forums.developer.nvidia.com/t/non-existent-shared-vram-on-nvidia-linux-drivers/260304/.

It made me think, something must be wrong or mis-communicated because this would be a deal breaker for anyone using NVIDIA GPUs on Linux. I mean, if I open literally 8 Firefox windows or 15 Ghostty terminals, my system crashes but I am only using 3 GB out of 16 GB of system memory but my GPU gets ~75% full and that's when issues start happening. Games are out of the question, but those same games run perfectly well on Windows.

You can verify this by running `nvida-smi` to see your current GPU usage. You can also run `nvidia-smi -q` to get more details. Going by a few replies in that thread, the BAR1 memory usage value is how much memory the GPU has access to which would include system memory if your hardware supports it. Allegedly if you do some tweaking and have a 30XX card you can configure that but everyone else is out of luck. I don't really know for sure what it is, I just know my value is very low and I cannot tweak it.

Here is my nvidia-smi -q output, you can see BAR1 is very low:

    FB Memory Usage
        Total                                          : 2048 MiB
        Reserved                                       : 58 MiB
        Used                                           : 923 MiB
        Free                                           : 1068 MiB
    BAR1 Memory Usage
        Total                                          : 256 MiB
        Used                                           : 38 MiB
        Free                                           : 218 MiB

I figured this was worth an Ask HN post because this community is really good with technical problems. How are you using NVIDIA cards on Linux without rebooting every few hours? Especially with any 20XX, 10XX or older cards.

AMD and Intel cards do not have this issue btw, it's isolated to NVIDIA.

4 comments

FB Memory Usage Total : 2048 MiB Reserved : 58 MiB Used : 923 MiB Free : 1068 MiB BAR1 Memory Usage Total : 256 MiB Used : 38 MiB Free : 218 MiB