Tsinghua University · arXiv 2610.28577
AKV: KV Cache Engineering for LLM Agents
Agents drop and reposition KV cache instead of rewriting their text history.
−27.7% avg cost · +1.7 pts avg score
Context engineering (summarize text)
Re-prefill work: 0 blocks
Cache engineering (AKV drop)
Re-prefill work: 0 blocks
Both agents hold a user prompt, assistant turns and tool results. Press the button to remove old tool result A.
UserAssistantTool resultRecomputedNew turn
Active KV cache (number = position encoding index)
Start state: tokens A, B, C sit at positions 0, 1, 2. Step through the 3 operations.
{"messages": [...], "drop_message": {}, "reposition": []}Model
Benchmark
Pass rate (%) · higher is better
Cost (US cents per task) · lower is better
Concurrency: 14 · GPT-OSS-120B · 80 long-context SWE-bench Verified tasks · 8 A800 GPUs
P95 request latency (s) · lower is better
Output throughput (tokens/s) · higher is better
1.18x
Output throughput, peak vs peak
1.53x
Input throughput, peak vs peak
1.51x
Faster P95 latency, peak vs peak
2.849ms
Full attention per decode step (was 7.082)
- Few supported model familiesDrop and Repos are enabled for Qwen, GPT-OSS and MiniMax-M2 text architectures. Linear attention models are future work.
- Self-hosted onlyYou run the AKV server yourself. It extends SGLang with cache-operation support.
- Self-reported, single hardware setupAll results come from the authors, on 8 A800-80GB GPUs in BF16.
- Different measurement windowsHigh-concurrency AKV runs were measured through the first 30 task completions. Standard used a 1-hour window.
- Not every benchmark improvesMiniMax-M2.7 on BrowseComp-Plus scores 58.9 with AKV, below Text Engineering at 62.3.
- Dropping reasoning cache hurtsOn SWE-bench Verified, dropping reasoning cache fell to 36.5, and dropping both to 34.5, versus 41.0 for tool drop.
Tap a row to expand it.
