On-Device Inference in 2026: 160 Tokens a Second on a Laptop, a 2-Bit Model in Every iPhone, and the Bandwidth Arithmetic That Decides What Stays Local
What actually runs on the device now: Apple's 3B model trained at 2 bits and the WWDC26 framework that lets any model plug in, Gemini Nano inside AICore, Windows ML and the quiet replacement of Phi Silica, the NPU TOPS numbers and why they are the wrong metric, the small-model field from Gemma 4 E2B to Qwen3.5-2B to LFM2.5 with licences, measured decode speeds from Google's own tables and an independent phone benchmark, the runtimes (llama.cpp with multi-token prediction, LiteRT-LM, ExecuTorch 1.0, MLX), the memory-bandwidth arithmetic that predicts all of it, and the model-extraction attacks nobody has fixed. With commands to run one today.
14 minSep 7, 2026LLM Inference in 2026: Prefill Is a Different Machine Now, and the KV Cache Has a Disk Tier
vLLM 0.28, SGLang on 400,000 GPUs, NVIDIA Dynamo 1.0 and llm-d in the CNCF all converged on the same architecture: split prefill from decode, route on cached prefixes, page the KV cache to CPU and SSD, and run MoE experts wide across a rack. What changed this year, with the papers, the flags, and the prices.
15 minSep 2, 2026Sandbox Creation Time Benchmarks Measure Different Things
Every AI sandbox matrix has a creation time column spanning 0.79ms to 2.7s. That spread is definitional, not performance. Here's how to make it comparable.
10 minSep 2, 2026I Measured Every Stage of My 179ms Firecracker Boot Path
Where every millisecond of a Firecracker create goes, measured on my own production fleet, plus why I refuse to stop the clock before the guest answers.
9 min