SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attention

Qianchao Zhu, Jiangfei Duan, Chang Chen, Dahua Lin, Chao Yang

Conference on Machine Learning and Systems 2025 · Day 3 · Session 7: Quantization and Sparsity

This article delves into SampleAttention, a novel approach designed to drastically accelerate inference for **Large Language Models (LLMs)** operating with exceptionally long context windows. Presented at MLSys 2025, the work by Qianchao Zhu, Jiangfei Duan, Chang Chen, Dahua Lin, and Chao Yang tackles the critical bottleneck of **time to first token (TTFT) latency**, which is exacerbated by the quadratic computational cost of the attention mechanism as sequence lengths grow. With LLMs now capable of processing millions of tokens, the traditional full attention computation becomes prohibitively expensive, rendering many interactive long-context applications impractical.

AI review

SampleAttention is a legitimate systems paper solving a real problem — quadratic attention scaling at inference time — with a two-stage adaptive sparse attention algorithm guided by a principled metric (CIA). The 5.29x TTFT reduction at 1M tokens is the kind of number that makes engineers pay attention. But this article reads like a PR summary generated from the abstract and related work section, not a reconstruction of what actually happened in the room. The speaker wasn't an author, there's no code link, no architectural diagram walkthrough, and the 'technical deep dive' repeats the same…