---
title: "The next purchase order: VRAM or RAM?"
date: 2026-10-15
series: 26
summary: "A 94 GB open model no longer needs 94 GB of GPU memory. llama.cpp merged an expert cache this week that serves it from host RAM at up to 2.2x the decode speed of plain offload."
voice: architect
tags: [moe-expert-offload, host-memory-tiering, vram-vs-ram-capex, expert-lru-cache, small-batch-fallback]
image: "/notes/moe-expert-ram-cache/image.png"
imageAlt: null
linkedin: "urn:li:share:7516475272951644160"
sources:
  - title: "MoE expert LRU cache on GPU, experts in host memory — llama.cpp PR #29887"
    url: https://github.com/ggml-org/llama.cpp/pull/29887
    date: 2026-10-07
  - title: "Multi-GPU support + 2x RTX 4090 numbers — llama.cpp PR #30112"
    url: https://github.com/ggml-org/llama.cpp/pull/30112
    date: 2026-10-08
  - title: "Origin: port of the qvac-fabric MoE cache"
    url: https://github.com/tetherto/qvac-fabric-llm.cpp
dateApprox: false
---

A 94 GB open model no longer needs 94 GB of GPU memory. llama.cpp merged an expert cache this week that serves it from host RAM at up to 2.2x the decode speed of plain offload.

Who this matters for: anyone sizing on-prem or air-gapped boxes for big mixture-of-experts models. Most of an MoE's weights are experts that sit idle on any given token. The new path keeps them in system RAM and caches only the hot ones on the GPU.

Author-measured numbers from the PRs, single stream, Qwen3.8-Flash-Next Q4 (93.7 GB, 65 GB of experts): one RTX 5090 goes 30.8 to 67.8 tokens per second with an 18.6 GB cache at an 89 percent hit rate. One 4090: 25 to 40. Two 4090s: 35 to 60. The author's rule of thumb: cache roughly 10 percent of total expert size.

The price, and it is the deciding line: prompt processing drops to 0.91x, and to 0.73x with the biggest cache. So this fits chatty, decode-heavy, low-concurrency work, a team assistant on an air-gapped box. It is wrong for RAG ingestion and batch pipelines, where prefill is the bill.

The decision it changes: whether the next purchase order buys VRAM or RAM. Host memory costs a fraction of GPU memory per gigabyte, and it just became usable for the cold 90 percent of an MoE.

The trap: the cached path runs only for batches of 32 tokens or fewer. Raise concurrency and you fall back to the slow path exactly when load peaks. Benchmark at your real concurrency, not at 1.

Before the next GPU order, price the same throughput as RAM plus --moe-cache-mib first.