We run a frontier model on a desk. Two NVIDIA DGX Sparks, 128 GB of unified memory each, 273 GB/s of memory bandwidth apiece, both plugged into a standard wall outlet. On them sits GLM-5.3-Flash, a 320B-parameter open-weight model (18B active), quantized to 4-bit EXL3, with a 500,000-token context window. That stack runs everything we do: chat, a voice assistant on a HomePod, scheduled agent jobs, long-document work.
Until October 1 the serving engine was vLLM. It worked fine. Then we swapped the engine to TensorFold and changed nothing else. Same weights, same speculative decoder, same OpenAI-compatible API. Every client (the chat UI, the voice loop, the scheduled agents) kept working without a single config edit. The whole migration was an afternoon: stage the 164 GB checkpoint on both nodes, boot, done. The server was answering in under two minutes.
The numbers
Same two Sparks, same benchmark protocol, temperature 0, same model and drafter:
| Workload | vLLM recipe | TensorFold |
|---|---|---|
| Structured output, 1 stream | 62.9 tok/s | 114.7 tok/s |
| Structured output, 4 streams (aggregate) | 146.5 tok/s | 227.9 tok/s |
| Prose, 1 stream | 36.1 tok/s | 60.4 tok/s |
| Prose, 4 streams (aggregate) | 75.3 tok/s | 108.8 tok/s |
That is 1.4x to 1.8x on every workload we measure, and single-stream structured decoding landed just shy of double.
The engine has kept shipping since. On the current build we measure 78.8 tok/s single-stream structured and roughly 145 tok/s aggregate prose at 4 streams. The 1.25x to 1.9x band over the vLLM recipe has held on every re-run.
What TensorFold actually is
Not vLLM with tweaks. It is a custom inference engine built for EXL3-quantized GLM across multiple Sparks linked over RoCE (the recipe we run is public on GitHub: MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks-TensorFold). Custom quantized-matrix kernels. A DFlash2 speculative drafter, CUDA-graphed, that accepts 86% of its drafts at temperature 0, which is why the tokens keep coming. An fp8 KV cache pooled across four concurrent requests. First token lands in 233 to 451 ms on a cold prompt. On warm follow-up turns the prompt is already parsed and the first token lands in 0.07 to 0.3 seconds. Prefill runs around 1.8 to 2k tok/s on the 8k to 33k prompts we feed it, so a long document starts answering in seconds.
Why we're not going back
The cloud API used to be the fast route here. Now it is the last resort, and it has stayed there. The model on the desk answers every question on hardware we own: no per-token bill, no rate limits, no data leaving the building, and a 500k context window nobody can revoke. Two quiet boxes at a wall outlet replaced the meter.
There is no fine print on the speed either. The model and the quantization are identical in both columns of that table. The entire gain is serving-stack engineering, and it compounds: every update since the swap has shaved more prefill and concurrency overhead.
That is the whole pitch for local AI. Frontier AI on hardware you own, and it keeps getting faster while the bill stays flat.
We build and tune these stacks for customers. If you want to know what your workload would do on a Spark or a private cluster, measure it first. A workload assessment beats a spec sheet every time.
Measured on: 2x DGX Spark (GB10) in tensor parallel over RoCE, GLM-5.3-Flash-EXL3 4bpw with a DFlash2 drafter, October 2026, temperature 0, rate = (completion tokens - 1) / decode seconds.