Models & Open Source
llama.cpp Cuts DeepSeek V4 Prefill by 4.92 Seconds
A Metal kernel removes 4.92 seconds from a 512-token DeepSeek V4 prefill at 30K context, but the 47% headline is not end-to-end speed.
llama.cpp’s new Metal Lightning Indexer removed 4.92 seconds from a 512-token DeepSeek V4 Flash prefill at 30K prior context on an M1 Ultra. The published 47% prompt-throughput gain is real, but the same synthetic prefill-plus-decode sequence improves only 19.6%—a reason for existing high-memory Mac owners to update, not a reason to buy a new fleet.
Long context is where the kernel earns its keep
llama.cpp release b10236 raises 30K-context prompt processing from 33.40 to 49.18 tokens per second, a 47.25% throughput increase. Convert throughput into waiting time: 512/33.40 is 15.33 seconds and 512/49.18 is 10.41 seconds. The decision-relevant result is 4.92 seconds, or 32.1%, removed from prefill.
The effect scales with context. The release reports 153.73→155.19 tok/s at zero prior context, 73.90→86.95 at 10K, and 45.83→62.01 at 20K. Those rates imply only 0.03 seconds saved at zero, 1.04 seconds at 10K, and 2.92 seconds at 20K. Coding and research agents that repeatedly ingest long fresh context are the natural beneficiaries; short chats are not.
PR #25893 identifies the machine and command: one contributor’s M1 Ultra run of llama-bench, with 512 prompt tokens and 128 generated tokens. The tested community quant is a 284.33-billion-parameter, 106.05 GiB IQ3_XXS artifact; its Hugging Face repository documents that footprint. This is effectively a 128GB-plus deployment once runtime and cache headroom enter, not a base-memory Mac benchmark.
Decode supplies the brake. At 30K context, generation rises only from 7.94 to 8.60 tok/s, an 8.3% improvement. A synthetic 512-token prefill plus 128-token decode falls from 31.45 to 25.30 seconds: 6.15 seconds or 19.6% combined, nowhere near 47%. Real agents may spend even more time in tool calls, model deliberation, and repeated generation.
The merged commit implements the Lightning Indexer directly in Metal, including quantized K-cache paths important for long context. That makes the release more than a compiler flag, but not a broad hardware comparison. No same-model H200, MLX, or managed-API benchmark accompanies it.
The arithmetic mirrors today’s Meta Business Agent cost analysis: translate the vendor unit into the unit an operator experiences. It also extends the archive’s DeepSeek V4 launch analysis from capability and price into local serving latency.
Update first; buy hardware only after a trace replay
Existing owners of 128GB-or-larger Apple Silicon systems running DeepSeek V4 through llama.cpp should canary b10236 or newer. The software cost is zero; the operational cost is a roughly 106.05 GiB download, build and rollback time, output-quality evaluation, and enough memory for model plus cache. Replaying production-shaped 10K–30K prompts can settle the decision in hours.
New buyers should wait. Apple’s Mac Studio announcement emphasizes unified-memory capacity for very large on-device models, but its $1,999 base price is not the tested high-memory configuration. The release’s rates imply 56,808 extra prompt tokens per hour at 30K context, yet no valid dollars-per-capacity figure follows because Apple does not price the tested M1 Ultra setup on that page and the benchmark omits power, quality, and concurrency. Capacity and local ownership are Apple’s advantage here, but this patch does not establish fleet economics.
Production alternatives solve a different problem. MLX-LM offers Apple-native generation, quantization, prompt caching, fine-tuning, and serving, yet no same-model DeepSeek V4 comparison is presented. SGLang’s DeepSeek V4 NPU support describes fused kernels, parallelism, disaggregated prefill and decode, and multi-request serving. A single M1 Ultra is an owner-operated workstation; that is a different topology from a production cluster.
The evidence also has an unusual provenance caveat. The contributor said the initial implementation was generated by an AI system and that they were not a Metal-kernel developer. Maintainers reviewed and revised the path, including quantized-K support, but the published M1 Ultra figures remain a single contributor’s benchmark, not an independent replication. Reviewed code is not the same thing as reproduced performance.
Quality can overturn the result. IQ3_XXS may miss a coding, retrieval, or tool-use threshold that official weights or an API clear. Cached prefixes can shrink prefill’s share. Concurrent requests can expose a different bottleneck. The broader optimization branch posted much larger numbers, but it bundled additional hyper-connection and sparse-attention work; those figures do not belong to b10236.
The minimum acceptance test should report p50 and p95 end-to-end latency at one, four, and sixteen sessions; task quality against the current baseline; prompt-cache hit rate; wall energy per request; and memory pressure. The archive’s open-model control-premium analysis explains why local ownership can justify lower raw throughput, but control is valuable only when the workload still passes.
Reverse the update verdict if the same command fails to reproduce, if IQ3_XXS loses material task quality, or if generation and tools dominate enough that end-to-end latency barely changes. Strengthen it when independent M1 Ultra tests reproduce the delta and real agent traces show a double-digit p95 improvement without quality loss.
The 47% headline describes throughput in one stage. 4.92 seconds describes the measured prefill reduction in that stage, not total user latency. That smaller number is still a meaningful free upgrade for the right installed base—and a far more honest buying signal.