InferScale cuts memory-agent latency up to 79%

InferScale cut time to first token by 72%–79% at 50 retrieved memories in tests across three open-weight models on LoCoMo, while delivering 3.7–4.5 times the concurrent throughput and 60.3% accuracy versus Mem0’s 63.3%. The preprint’s vLLM implementation caches each memory fact’s KV representation on the GPU and injects it at serving time, avoiding repeated prompt prefills without modifying the engine or fine-tuning a model. As persistent memory becomes the coordination layer for agents, operators should benchmark reusable KV state before paying the latency tax on the same retrieved context for every request.