Anyone running big models on-prem knows the restart: for several minutes every in-flight request fails or queues, and no autoscaler hides it.

The numbers, from Ant Group, Alibaba and the SGLang team. Ling-2.6-1T in FP8 on 8x H20: weight loading took about 495s of a 527s boot, 94% of the restart. With the weight cache daemon it drops under a second, and total boot goes from 8.8 minutes to about 32 seconds. Qwen3-235B FP8 goes from about five minutes to under one.

A per-GPU daemon loads and quantizes the weights once and keeps them resident. A new engine initialises the model on the meta device, then swaps each parameter pointer to the daemon's tensor over CUDA IPC. There is one copy in GPU memory and nothing gets copied. If the engine dies, the daemon still holds the weights. If the daemon dies, running engines keep their mapped tensors.

Active-standby then works without parking a full idle GPU replica, and a batch job can share a GPU with an online service, get evicted, and come back in under a second.

The trap: it only supports unquantized and block-wise FP8 today. AWQ, GPTQ and Marlin repack weights, so the daemon raises a hard error rather than serve wrong numerics. Most on-prem 70B boxes I have seen run exactly those formats.

Put engine restart time in your latency SLO and measure it the way you measure p99.

Qwen-MM-Plugins makes an existing agent harness handle images, video, OCR, ASR and segmentation. The README's own definition of a capability: a skill, so the model knows the toolset exists, plus an optional MCP server, the tools themselves. Apache-2.0, no tagged releases yet, about 1.6k GitHub stars in under two weeks.

None of it touches the weights. There is no adapter and no retraining, which for a harness is the right call: you add OCR to a coding agent without owning a vision stack, and the providers stay swappable.

Read the data path anyway. The local plugin reads frames and crops images on your machine. The understanding calls out to a hosted API, DashScope by default. If you are under data residency rules or genuinely air-gapped, that one line is the whole procurement conversation, and repointing it is your work, not theirs.

The trap: telling a model a tool exists is not the same as the model using it well. The failure mode moves from "cannot see the image" to "picked the wrong tool and summarised it confidently." That needs an eval per modality, and the repo ships no benchmarks.

Ask where the pixels go.

AMD launched Helios and, the same day, announced a Cerebras pairing that splits inference across two machines: Helios processes the prompts and large context windows, the Wafer-Scale Engine handles token generation. AMD's own term for it is disaggregated inference.

If you have sized an on-prem cluster you already knew the split. Prefill is compute bound and batches well. Decode is memory bandwidth bound and does not, which is why a single pool sized for both means you overpay on prefill and still miss your latency SLO on decode.

Two numbers came out that day. Keep them apart.

30% more tokens per dollar: Helios against NVIDIA Vera Rubin NVL72. AMD Performance Labs estimate, Kimi K2 Thinking, 32K in and 8K out.

5x tokens per second per watt: AMD plus Cerebras against a Cerebras WSE-only configuration. Different model, Kimi 2.6 1T. Vendor modelling, not measured silicon.

The trap: that 5x is not a win over NVIDIA. It is Cerebras with Helios against Cerebras without it, and the headline says per watt while the footnote says per kilowatt.

The topology is right. The benchmark is still someone else's marketing.

The 2026-07-28 spec drops the initialize/initialized handshake and the Mcp-Session-Id header. Protocol version and capabilities now ride in _meta on every request, and a new server/discover method covers the rest.

What that buys you is a tool server you can put behind a plain round-robin load balancer. No session affinity to maintain, so you can drop the shared session store your horizontal deployment needed. Proxies route on the Mcp-Method header instead of inspecting bodies. Your MCP fleet becomes an ordinary stateless HTTP service, which is the only shape most platform teams will agree to run.

Roots, Sampling and Logging are deprecated in the same release. Annotation-only, still functional through 2027. Logging moves to stderr and OpenTelemetry.

The trap: read the Sampling deprecation carefully. If your tool server calls back through the client for model access, the replacement is a direct provider API from inside the server. In an air-gapped or egress-controlled estate that is a network and procurement change, not a code change.

Your tool servers just became ordinary web services. Deploy them like it.

Sysdig documented JadePuffer this month: an LLM agent that ran the whole chain by itself. RCE into an exposed Langflow box, swept the host for provider keys and cloud creds, moved laterally into a Nacos/MySQL config server, installed a beacon, then encrypted 1,342 config items and wrote its own ransom note.

The entry point was not novel. CVE-2025-3248, patched April 2025. What was new is tempo. When a Nacos admin login failed, the agent diagnosed it and self-corrected in 31 seconds, switching from a subprocess call to a direct bcrypt import. No human triages that fast.

Sysdig caught it on 600+ distinct payloads, each one narrating its own targeting rationale in plain language. That self-narration is the fingerprint.

The trap: the agent's blast radius was exactly the creds and tools its host could reach. Same rule as your defensive agents. Leave provider keys sitting in your orchestration server env and you have scoped the attacker's reach for them.

Attackers went agentic before most defenders did. Your perimeter is now whatever your agent's tools can touch.

Anthropic shipped Sonnet 5 last week at $2 in / $10 out per million tokens intro pricing ($3/$15 from September). Vendor numbers put it just under Opus 4.8 on agentic coding and ahead on some knowledge work. Terminal-Bench went from 67.0 on Sonnet 4.6 to 80.4.

Mid-tier models now clear the bar for most production agent loops. That changes the architecture more than the bill. What I would do with it:

1. Route by task, not by loyalty. Tool-call planning, extraction, classification go mid-tier. Deep multi-file reasoning escalates. Two tiers beat a monolith on cost per task.

2. Meter cost per task, not per token. Token price tells you little once models think for variable lengths. Tag every run with its task class and its dollars. Alert on outliers.

3. Re-run your evals before flipping traffic. The gap to frontier is benchmark-thin, and your workload is not a benchmark. Golden set first, then migrate.

We run the same tiering on-prem with open weights. Same math, different price tags.

The trap: the effort dial is a budget, not a quality knob. Uncapped, the cheap model on a hard task can spend past Opus and hand you the same answer.

Cost per task is the only price that matters in an agent stack.

You AWQ the model to 4-bit, fit it on the GPU, ship it. Then a user pastes a 200-page contract, and per-token KV at 128K quietly eats more memory than the weights you just shrank.

Low-bit KV cache used to be a trap: push it to 2-bit and needle-in-haystack recall falls apart. New work (Block-GTQ, RoPE-aware bit allocation, Jun 2026) says stop quantizing it uniformly. Under RoPE, key energy concentrates in a few frequency blocks. Spend bits where the signal is, starve the rest.

Paper-stated on Llama-3.1-8B, benchmark your own:

• 3.24x KV-cache compression at fp16-comparable quality
• needle recall 70.6 to 97.4 vs naive low-bit
• 1.34x faster than fp16 attention at 128K on one H800

The trap: uniform 2-bit KV looks clean on short prompts and silently wrecks recall past 32K. You won't catch it in eval until a customer's long document returns the wrong clause.

Quantize the weights for the GPU bill. Quantize the KV cache for the context window. That second one is the on-prem win nobody is pricing in yet.

GLM-5.2 shipped MIT-licensed, full weights, 1M context. The numbers (vendor and third-party, so benchmark your own):

• SWE-bench Pro 62.1, up from 58.4 on 5.1.
• FrontierSWE 74.4, within a point of Opus 4.8 at 75.4.
• Terminal-Bench 2.1 climbed from 63.5 to 81.
• MCP-Atlas tool-use sits near Opus.

For on-prem that flips the calculus. Frontier-class long-horizon coding now runs inside the air gap: no egress, no per-call metering, weights you own outright. The model-ownership red line got cheaper to hold.

The trap: on-prem you don't pay per token, you pay in decode tok/s and KV cache. GLM-5.2 is token-hungry, by early reviews one of the least efficient in its class. A model that wins the leaderboard while burning 2-3x the tokens can still lose your GPU budget.

Own the weights. Then benchmark the tok/s, because the leaderboard doesn't pay your GPU bill.

The model sees the system prompt, the user request, and a retrieved document as one token stream. No boundary between instruction and data. That isn't a missing feature. It's the architecture. OWASP's "State of Agentic AI Security" v2.01 (June 11) maps injection to 6 of the 10 agentic Top-10 categories. You don't patch the foundation.

So stop filtering inputs and start budgeting capability. Two rules worth stealing:

• Lethal trifecta (Willison): private data + untrusted content + external comms. One agent holding all three is an exfiltration tool waiting for one poisoned input.
• Rule of Two (Meta): an unsupervised agent gets at most two of those three legs. Want all three? A human signs off. Full stop.

The trap: the trifecta is a property of the session, not the agent. A safe RAG read, a safe DB lookup, and a safe send-email tool each pass review, then co-occur in one context window and arm the bomb. Audit the composition, not the tools.

Guardrail classifiers lower the incidence. They don't close the class. You're always one encoding away.

Design the blast radius. Don't litigate the prompt.

A year into Model Context Protocol, the numbers are ugly. A taint-analysis sweep of ~39,800 open-source MCP servers confirmed 106 zero-days with working exploits and 67 CVE IDs assigned (VIPER-MCP, arXiv). Censys counted ~12,500 internet-exposed MCP services — most with no auth at all. This month the NSA published MCP security design guidance.

The tool list IS the attack surface. MCP's whole job is letting a model call your tools — and that same path is taint-style injection: prompt → vulnerable sink → SQL injection, metadata exfiltration, command execution. The model is now an untrusted input source pointed straight at your systems.

Auth got treated as optional, so teams skipped it. Unauthenticated metadata exfil and CVSS-9.8 command-injection keep surfacing in DB-connected servers.

So treat every MCP server like the RPC boundary it is: auth in front of all of them, allowlist tools instead of "expose everything", scope credentials per tool, and audit-log every call with the reasoning that triggered it.

The trap: the dangerous servers aren't the public ones — they're the internal ones someone stood up with auth off because "it's behind the firewall."

An agent's tools aren't features. They're your new perimeter.

How it actually works:

Vision. Raw images → 48×48 patches → a single matrix multiply into the LLM's embedding space. No standalone ViT, no cross-attention adapter.

Audio. Raw 16kHz frames → linear projection into that same text embedding space. RoPE already handles the timing, so the 12 conformer layers older Gemma needed are simply gone.

Result. Three encoder stacks collapse to one. Apache 2.0, Q4, runs on a 16GB laptop. Text, image, audio and video in native.

Google published no benchmarks at launch. The "near-26B at half the memory" line going around is community-reported — treat it as a hypothesis, then benchmark your own task.

The trap: one weight space means your single Q4 build is the whole multimodal stack, and LoRA now updates vision, audio and text in one pass. Great — until you quantize. There's no separate encoder to keep at higher precision, so a sloppy quant degrades every modality at once.

When the encoders disappear, your quantization config becomes your entire multimodal strategy.

Anthropic shipped Opus 4.8 on May 28. Buried under the benchmarks was the part that matters for anyone building agents: "dynamic workflows" — the model writes its own orchestration script, fans out tens to hundreds of parallel subagents, points adversarial agents at their findings to refute them, and iterates until the answers converge.

That's not a model feature. That's an architecture going mainstream. And you can build it on open weights, air-gapped, today.

Three things I'd tell anyone wiring up fan-out agent systems:

1. Fan-out is cheap. Convergence is the product. Spawning 100 subagents is one loop. Knowing when they've actually agreed — and aren't just echoing the same wrong context — is the hard part. Your orchestrator needs an explicit convergence criterion (K consecutive rounds with no new finding), not a fixed iteration count.

2. Adversarial verifiers beat more generators. Ten agents generating answers gives you ten plausible answers. One agent whose only job is to refute a finding kills the plausible-but-wrong ones. Spend your token budget on skeptics, not on a bigger crowd.

3. The orchestrator is code, not a prompt. Fan-out, barriers, retries, loop-until-dry — that's deterministic control flow wrapped around stochastic agents. If your "orchestration" is one giant prompt asking the model to coordinate itself, you don't have an orchestrator. You have a wish.

The trap: fan-out without a convergence test burns tokens forever — or worse, returns false consensus when every subagent inherited the same poisoned context. Diversity of angle is what makes the votes mean anything.

You don't need a frontier API to run this shape. LangGraph + a self-hosted 70B + a disciplined orchestration script gives you fan-out, adversarial verification, and convergence — inside your VPC, with a token bill that doesn't explode the moment the agent count does.

The model got the headline. The pattern is the product.

For five years, subquadratic attention was a research artifact. In 2025 it shipped.

Three architectures to know:

NSA — Native Sparse Attention (DeepSeek, ACL 2025 best paper) Three parallel paths per query: compressed coarse-grained tokens, selectively retained fine-grained tokens, and a sliding window. Hardware-aligned. Trainable end-to-end — not bolted on after pretraining.

DSA — DeepSeek Sparse Attention (DeepSeek-V3.2-Exp, 2025-09-29) Lightning indexer + fine-grained token selection. The first frontier model where sparse attention is the production path, not an ablation. MMLU-Pro 85.0 (par with V3.1-Terminus dense). AIME 2025: 89.3 vs 88.4. API price cut >50%.

SSA — Subquadratic Selective Attention (Subquadratic, launched 2026-05-05) Content-dependent selection. 1M tokens production, 12M tokens research. Claims 52× speedup at 1M tokens vs FlashAttention-2 on B200s. Independent benchmarks pending — discount accordingly.

What it means for your stack today:

• SGLang v0.5.9 ships TRT-LLM DSA kernels. Flag: --nsa-prefill-backend trtllm. 3–5× speedup on Blackwell.
• A 1M-token system prompt is now a budget question, not a research project.
• Long-context evals get harder, not easier. Sparse models pass needle-in-haystack and still miss multi-hop reasoning at depth — your eval harness needs a long-context reasoning section, not just retrieval.

The trap: subquadratic is a complexity class, not a quality guarantee. Mamba and RWKV were both subquadratic. Both ended up hybrid. Bet on architectures that ship par-quality benchmarks, not ones that ship throughput plots.

A real harness has five sections:

1. Golden set — frozen labeled examples. Pass rate per task. The set has an owner and a last-updated date. Without this, a model swap silently regresses.

2. Schema set — does output parse? Match the contract? Pydantic validation passes? Grammar-gated decoding trends this to 100%, but eval anyway — schema drift catches you when frameworks upgrade.

3. Calibration set — when the model says "0.95 confidence", is it actually right 95% of the time? Brier score or reliability diagram. Most teams skip this. Then they wonder why their auto-route threshold is wrong.

4. Latency budget — P50, P95, P99 per stage (router, retrieval, decode, tool-call). Hard cap per stage. A feature that violates the budget is a regression by definition.

5. Adversarial set — jailbreaks, prompt injection, OOD inputs, ambiguous queries. Pass rate baseline. Run on every model swap. The day you skip this is the day a customer finds your prompt-break on Twitter.

Each section needs three things: a metric, an owner, and a frequency.

The metric without the owner is theatre. The owner without the frequency is a someday-list.

A harness with fewer than five sections can't tell you whether the next deploy is an upgrade or a regression. The model gets better. The harness tells you.

Most production agents I've reviewed ship some version of this pattern:

→ Call the LLM. → Try to json.loads the output. → On JSONDecodeError, re-prompt with "Respond ONLY in valid JSON" and call the LLM again.

It mostly works. ~5–10% of calls retry. Each retry is a second full inference - a tax on throughput, latency, and GPU cost for a problem unrelated to the model's reasoning.

The fix isn't a better prompt. It's grammar-constrained decoding (xgrammar in SGLang, Outlines in vLLM). The decoder masks every token that would produce invalid output. Result is always parseable, by construction.

The biggest payoff isn't on tool-call args. It's on the router. Define a Pydantic schema -

agent: Literal["billing", "support", "sales", "escalate"] confidence: float reason: str

• hand it to the decoder, and the router cannot pick an agent that doesn't exist. Cannot malform confidence. Cannot omit reason. Structural correctness is guaranteed before the first token decodes.

PRODUCTION NUMBERS (Llama-3.3-70B router, H100, SGLang + xgrammar): • Parse-error rate: ~7% → 0% • Per-decode masking overhead: ~5–15ms • Net per-user tok/s: higher - no retry tokens

THE TRAP: over-constrained schemas make the model lie. Always include an "escalate" or null option in your enum, or nonsense queries get routed somewhere wrong with high confidence.

If your agent still wraps every model call in try/except JSONDecodeError on the hot path, you're paying for a problem that's been solved for two years.

If you're running TGI in production, that's the memo. Here's the picture for anyone serving a ~70B model on their own GPUs right now:

Three things to take from that table:

1. TGI still works - but it stops improving.

The single-user decode is still competitive. But HuggingFace itself is now pointing new deployments at vLLM or SGLang. If TGI is your runtime today, the migration conversation happens in the next two quarters, not later.

2. Ollama is not a production runtime.

A single user on an H100 gets ~24 tok/s. Great for dev boxes and for building your eval harness. But ~1–3 concurrent req/sec means your product dies the minute the second customer shows up. We use Ollama daily - never in production.

3. Per-user decode tok/s is the number nobody benchmarks.

Every framework publishes aggregate throughput because aggregate looks big. What your user feels is per-user streaming speed at batch=1. Below ~30 tok/s the UX feels slow. Above 60 and you've won the perception battle. vLLM on a single H100 with int4 weights sits comfortably above that line.

Our stack for sovereign-AI clients: vLLM for serving, AWQ / GPTQ for quantisation, Ollama for dev boxes, H100 / H200 for production, Prometheus + Grafana for the whole path. All on-prem. All air-gappable. No OpenAI bill.

The "which model" question is the last one I ask a client. The first is: what's your SLO, and what GPUs do you already own?

Numbers in the table are verified as of April 2026 (sources in the comments). Per-user tok/s moves ±30% with context length - benchmark your own.

Autonomous agents touching production, behind a 5-second human gate, is a product.

We shipped a LangGraph multi-agent system this quarter that mutates state in an enterprise customer's production environment. Three things I'd tell anyone building agents that touch real systems:

1. Your toolbelt is the product.

The LLM picks. Your tools decide what's possible. If run_command exists, someone will run rm -rf /. We locked the toolbelt before we tuned a single prompt. Every tool has a named blast radius, a rate limit, and a scope the planner cannot widen.

2. Human-in-the-loop is a UX problem, not a safety feature.

If the review gate takes 40 seconds, your operator auto-approves by week two. We built a 5-second review: diff, blast radius, one-click undo. Approval rate dropped from 100% to 62%. That 38% is the product working.

3. The audit log is not a side-effect. It is the product.

Procurement reads it. Insurance reads it. The customer-facing report is generated from it — by a narrator node that cites every claim with an evidence hash. We designed the log schema before we designed the agent graph.

Stack: LangGraph + self-hosted open-weights models + a disciplined tool surface. No OpenAI key in the build. Runs inside the client's VPC. The team sleeps.

If you're building agents that touch anything destructive — infra, code, money, or customers — the same three rules apply. The model is not the hard part.