"kv cache optimization" Papers
5 papers found
$\text{D}_{2}\text{O}$: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Models
Zhongwei Wan, Xinjian Wu, Yu Zhang et al.
ICLR 2025poster
22
citations
MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding
Ranajoy Sadhukhan, Jian Chen, Zhuoming Chen et al.
ICLR 2025posterarXiv:2408.11049
61
citations
When Attention Sink Emerges in Language Models: An Empirical View
Xiangming Gu, Tianyu Pang, Chao Du et al.
ICLR 2025posterarXiv:2410.10781
90
citations
Bifurcated Attention for Single-Context Large-Batch Sampling
Ben Athiwaratkun, Sujan Kumar Gonugondla, Sanjay Krishna Gouda et al.
ICML 2024poster
QUEST: Query-Aware Sparsity for Efficient Long-Context LLM Inference
Jiaming Tang, Yilong Zhao, Kan Zhu et al.
ICML 2024poster