Your model weights should outlive the process

Your model weights should outlive the process that serves them. SGLang shipped that idea on August 21.

Anyone running big models on-prem knows the restart: for several minutes every in-flight request fails or queues, and no autoscaler hides it.

The numbers, from Ant Group, Alibaba and the SGLang team. Ling-2.6-1T in FP8 on 8x H20: weight loading took about 495s of a 527s boot, 94% of the restart. With the weight cache daemon it drops under a second, and total boot goes from 8.8 minutes to about 32 seconds. Qwen3-235B FP8 goes from about five minutes to under one.

A per-GPU daemon loads and quantizes the weights once and keeps them resident. A new engine initialises the model on the meta device, then swaps each parameter pointer to the daemon's tensor over CUDA IPC. There is one copy in GPU memory and nothing gets copied. If the engine dies, the daemon still holds the weights. If the daemon dies, running engines keep their mapped tensors.

Active-standby then works without parking a full idle GPU replica, and a batch job can share a GPU with an online service, get evicted, and come back in under a second.

The trap: it only supports unquantized and block-wise FP8 today. AWQ, GPTQ and Marlin repack weights, so the daemon raises a hard error rather than serve wrong numerics. Most on-prem 70B boxes I have seen run exactly those formats.

Put engine restart time in your latency SLO and measure it the way you measure p99.