TGI went into maintenance mode. Here's the on-prem 70B picture now.

HuggingFace quietly put Text Generation Inference into maintenance mode in December 2025.

If you're running TGI in production, that's the memo. Here's the picture for anyone serving a ~70B model on their own GPUs right now:

Three things to take from that table:

1. TGI still works - but it stops improving.

The single-user decode is still competitive. But HuggingFace itself is now pointing new deployments at vLLM or SGLang. If TGI is your runtime today, the migration conversation happens in the next two quarters, not later.

2. Ollama is not a production runtime.

A single user on an H100 gets ~24 tok/s. Great for dev boxes and for building your eval harness. But ~1–3 concurrent req/sec means your product dies the minute the second customer shows up. We use Ollama daily - never in production.

3. Per-user decode tok/s is the number nobody benchmarks.

Every framework publishes aggregate throughput because aggregate looks big. What your user feels is per-user streaming speed at batch=1. Below ~30 tok/s the UX feels slow. Above 60 and you've won the perception battle. vLLM on a single H100 with int4 weights sits comfortably above that line.

Our stack for sovereign-AI clients: vLLM for serving, AWQ / GPTQ for quantisation, Ollama for dev boxes, H100 / H200 for production, Prometheus + Grafana for the whole path. All on-prem. All air-gappable. No OpenAI bill.

The "which model" question is the last one I ask a client. The first is: what's your SLO, and what GPUs do you already own?

Numbers in the table are verified as of April 2026 (sources in the comments). Per-user tok/s moves ±30% with context length - benchmark your own.