A VRAM leak in llama.cpp's Multi-Token Prediction stack could crash servers after repeated sleep cycles. A fix merged on May 21 now makes speculative decoding more dependable for teams running local AI agents over long sessions.
The important detail is not that llama.cpp gained another incremental patch. It is that a production-style server failure showed up exactly where startups are trying to use local models: persistent agents that sleep, wake, and keep working without someone watching the GPU meter.
According to the merged pull request #23461 in the ggml-org/llama.cpp repository, contributor am17an found that the server's destroy() path reset the main model and context but did not free the speculative decoder, draft context, or draft model. For MTP models, those draft-side resources can hold GPU KV cache and compute buffers. When the server entered sleep and later resumed, new resources were allocated while the old ones stayed resident, pushing VRAM higher until the process could crash with an out-of-memory error.
The fix is direct. llama.cpp now explicitly resets the speculative decoder, the draft context, and the draft model before resetting the main initialization state, so the cleanup order does not create a use-after-free problem. Maintainers ggerganov and allozaur approved the change, and it merged into master on May 21 as commit 52fb93a.
The timing matters because MTP support itself only landed in llama.cpp on May 16 through pull request #22673. That earlier change added support for Multi-Token Prediction heads, including testing against Qwen3.6 27B and Qwen3.6-35B-A3B. In the PR notes, am17an reported a typical steady-state acceptance rate around 75 percent with three draft tokens, producing more than a 2x speed-up over baseline in tested workloads.
Why MTP matters for local AI economics
For startups building self-hosted coding assistants, support bots, or internal research agents, speed is only useful if the service stays up. A model that feels fast for a demo but crashes after repeated idle cycles is not production infrastructure. The May 21 patch fixes a narrow bug, but the business meaning is broader: local inference is becoming good enough that uptime, memory hygiene, and release discipline now matter as much as benchmark numbers.
MTP works by letting a model speculate several tokens ahead and then verify those predictions. That reduces the waiting time created by standard autoregressive generation, where every token depends on the one before it. In practice, the speed-up can make a local model feel much closer to a cloud API, especially for coding agents that produce long explanations, patches, and tool traces.
That is why teams are willing to tolerate the complexity. A small company running agents on its own hardware can control data flow, tune models for internal use, and avoid a bill that scales directly with every generated token. But once that setup moves beyond experimentation, reliability stops being a nice-to-have. A slow model is annoying. A leaking server is a pager event.
The infrastructure fragility beneath the speed
The leak also shows how young the local agent stack still is. MTP needs compatible GGUF files with the right prediction heads. The llama.cpp MTP PR points users to ggml-org GGUF builds for Qwen3.6-27B-MTP and Qwen3.6-35B-A3B-MTP, but not every community quant or fine-tune will include the required tensors. If the model file does not carry those heads, the promised speed benefit may not appear.
There are other caveats. The MTP support notes warn that prompt processing can take a hit because of device-to-host embedding transfers. Parallel decoding with MTP is supported, but not fully optimized. These are not reasons to avoid the feature. They are reminders that running local AI well is an operations problem, not just a model download.
A related optimization, pull request #23287, moved MTP draft sampling to the backend and merged on May 20. On an RTX 5090, the author reported roughly an 8 percent improvement in one CUDA test and about 4 percent on Vulkan. The same discussion also made clear that Metal did not see a noticeable gain yet, while tensor-parallel paths can fall back to CPU sampling in some cases. That is the current shape of local AI performance: real progress, uneven across hardware.
What this means for startups building on local AI
The practical takeaway is simple. If a startup is using llama.cpp with MTP and sleep-enabled servers, it should update past the May 21 merge and test the exact lifecycle it expects in production. Do not just run a throughput benchmark. Put the server through sleep, resume, idle, and repeated request patterns while tracking VRAM. The bug was tied to that lifecycle, so a clean short benchmark would not necessarily reveal it.
Teams should also verify the model artifact before designing around MTP. The safe assumption is that an MTP-specific GGUF from a trusted source is required. Treat random quants and renamed uploads with caution, especially if the deployment plan depends on speculative decoding to meet latency targets.
The next thing to watch is not one headline benchmark. It is whether llama.cpp can keep turning these fast-moving patches into stable defaults across CUDA, Vulkan, Metal, and multi-GPU setups. The VRAM leak fix gives local agent servers a stronger base. For founders, that is the real story: local AI is becoming more practical, but it still rewards teams that follow upstream closely and test like they are already in production.
