Running DeepSeek-V4-Flash-0731 on One DGX Spark With Full Quality
A fully local reasoning stack on a single GB10 box, the benchmark numbers, and the memory-leak hunt it took to get there.
TL;DR
I run a fully local AI agent stack on an NVIDIA DGX Spark (GB10, unified memory, ARM Grace CPU). The model is DeepSeek-V4-Flash-0731, quantized to fit the box, served through a custom inference server, with a full agent framework (Hermes) on top handling tool use, cron jobs, personalities, and reasoning effort control. On community tool-use benchmarks it scores ahead of the upstream release. It took several rounds of memory-leak hunting to get stable, but it now runs 24/7, thinking-enabled, at 260k context.
The hardware
NVIDIA DGX Spark, GB10 Grace Blackwell superchip. Unified memory pool (~121Gi usable), ARM Cortex-X925/A725 CPU cores, 20 threads. Ubuntu 24.04 LTS.
There is no discrete VRAM ceiling to fight here, but the unified memory model means the CPU, OS, and GPU workspace all compete for the same pool. That turned out to be the source of most of my pain.
The stack
Inference layer: a custom llama.cpp-derived server (ds4-server, built from a community fork) serving:
DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-0731.gguf
Running with --reasoning-effort max, 260,000 token context, and a hard 10GB memory floor reserved for the OS so the box does not lock up under pressure.
Agent layer: a self-hosted agent framework (Hermes) sitting in front of the model, giving it tool use and toolsets (CLI tools, MCP-style extensions), a cron scheduler running unattended jobs including a daily morning briefing, and reasoning_effort: max wired through end to end so the model actually thinks before answering rather than autocompleting.
Security: the inference API sits behind a locally generated API key. Default binding is loopback and bridge only, and anything I share externally runs through a separate gated endpoint rather than opening the box up.
Does 2-bit hold up?
The obvious objection to a 2-bit quant is that it should gut a reasoning model. The reason it does not here is the layout. The filename spells out the per-part scheme, and only the routed expert weights actually drop to 2-bit:
| Component | Quant |
|---|---|
| Routed-expert gate/up weights | IQ2_XXS (~2-bit) |
| Routed-expert down weights | Q2_K (~2-bit) |
| Dense parts: shared expert, attention projections, output head, router | Q8_0 (~8-bit) |
| LoRA matrices, compressor, indexer | F16 |
| Norms | F32 |
On a MoE that puts most of the parameter count at 2-bit and most of the sensitivity at 8-bit or better. Imatrix calibration does the rest.
Community numbers back it up. Mia (@MiaAI_lab) ran Tool-Eval Bench across 8 trials on 2026-08-01, comparing this build against the upstream deepseek-ai/DeepSeek-V4-Flash-0731 release:
| Metric | Single-Spark | Upstream 0731 |
|---|---|---|
| Mean score | 88.0 | 84.5 |
| Std dev | ±0.0 | ±1.5 |
| Reliability (Pass^8) | 79.8% | 60.7% |
| Quality | 88 / 100 | 83 / 100 |
| Mean points | 148.0 | 142.1 |
| Responsiveness | 29 / 100 | 54 / 100 |
| Median turn time | 5.5s | 2.7s |
The local quant won outright, and it was dramatically more consistent run to run. Zero variance across 8 trials against ±1.5, and reliability 19 points higher.
The category breakdown shows where the gains came from. Structured output +30. Multi-step chains +19. Error recovery +19. Context and state +9. Autonomous planning +8. Code patterns +6. Restraint and refusal +4.
It is not a clean sweep, and the losses are worth stating. Instruction following came in 10 points lower. Safety and boundaries 5 lower. Toolset scale 4 lower. Hard mode 2 lower.
The real cost is latency. Responsiveness scored 29 against 54, with median turn time roughly double. On a single box at max reasoning effort, that is the trade you are making, and for agent work running on a cron schedule it is a trade I will take every time.
Two caveats. This is one eval family, tool use, so it says nothing about knowledge, math, or long-context retrieval. And the summary lists two bench versions across the runs, so treat the margin as directional rather than exact.
Source: Mia (@MiaAI_lab) Tool-Eval Bench
I am running my own eval pass across core enterprise workloads and will publish the full numbers here shortly.
The debugging saga
Getting a MoE reasoning model stable on unified memory hardware was not a download-and-run experience. Rough timeline:
First stable boot at 1M token context. Worked great initially, but under real sustained load the box would slowly bleed available memory until it stalled.
Rolled context back to 262k to buy headroom. Bought some time, but the leak was still there, just slower.
Started tuning concurrency instead of context. Dropped max parallel sequences from 5 to 7 down to 2. Helped, but a slow leak (nicknamed TOPK-BOUND-VIOL internally, after the error signature) kept recurring. Memory would plateau, then suddenly nosedive twice in under an hour.
Root-caused it, mostly, to CUDA graph capture and invalidation churning under batch-shape changes whenever concurrent requests exceeded the configured max sequence count. Fix: disable continuous graph capture (DS4_CONT_CAPTURE=0) and raise the VMM budget. Memory plateaued instead of decaying. Big win.
A different leak showed up on a long single-shot 60k-token thinking generation on the serial-fallback code path. Same symptom, memory collapsing to under 2GB available, different trigger. This one turned out to be specific to the server build I was on.
Reverted the server binary and source one minor version back. Same leak signature never reproduced on the older build, even pushed past 100k tokens. Confirmed clean with a 40-minute soak test under sustained generation.
Eventually moved back to the newer build, since it has other fixes I wanted, but capped context at 260k as a deliberate mitigation rather than a fix, and kept the lower concurrency. That is where it sits today: stable, boots clean, being watched.
Total tuning surface that actually mattered, in order of impact: DS4_CONT_CAPTURE, the CUDA graph capture toggle, was the single biggest fix. Max concurrent sequences (batch size) was a bigger lever than I expected on unified memory. Context length mattered less for leaks than I assumed, more for baseline footprint. DS4_BATCH_VMM_BUDGET_MB and headroom were fine-tuning knobs once the big leaks were gone. And server version pinning, because sometimes the fix genuinely is do not upgrade yet.
Current running config
ds4-serve --cuda -c 260000 --reasoning-effort max --mem-floor-gb 10 DS4_BATCH_FIT_HEADROOM_MB=22528 DS4_BATCH_VMM_BUDGET_MB=2048 DS4_SERVER_SERIAL_MAX_TOKENS=131072 DS4_CONT_CAPTURE=0
Reasoning is on end to end. Both the inference server and the agent layer are configured for max reasoning effort, so every response goes through a real chain-of-thought pass before the final answer, not just a display toggle.
Lessons for unified-memory hardware
Unified memory is forgiving right up until it is not. You do not hit a hard VRAM OOM wall, you get a slow bleed that looks like a leak because it is one, just gradual.
Do not assume the newest server build is the safest one. Keep a known-good previous version around and do not be afraid to revert when a regression shows up under real load rather than synthetic benchmarks.
Concurrency mattered more than context length for stability on this hardware. Tune that knob first.
Soak test for real. A clean boot and a quick prompt tells you nothing about a leak that only appears after 40 minutes of sustained generation.
It is not fully solved. I am still watching logs after each config change. But it has been stable through real daily use, including agent cron jobs, long sessions, and big thinking traces, for the first time in a few weeks of iteration.