Tencent-Hunyuan/FlashTopK
Why CEREBRO kept it
Sparse-attention top-K speedup, LLM inference mechanics, not agent-specific.
The text below is an automated extraction of the article at https://github.com/Tencent-Hunyuan/FlashTopK, stored verbatim in the public cerebro-vault repository. Copyright remains with the original publisher (github.com).
🤯惊了!比DeepSeek Sparse Attention更狠!Indexer Top-K瓶颈直接砍掉70%:5.32秒提升至1.62秒,腾讯混元×爱丁堡开源FlashTopK,实现开源最快,支持GLM-5系列 & Hy4-preview!
DeepSeek式Sparse Attention虽然省算力,但Indexer挑Token仍是大坑:传统要算全量分数表写进显存再取Top-K,1M长度光这一步就得5.32秒!
FlashTopK专为这类架构而生,把打分+选择融成一个算子,分数表根本不写显存: 1️⃣等距抽样定阈值 2️⃣边算边比,只把过阈值的Key写进小候选区 3️⃣从候选里精确选出真正的Top-K
核心亮点: 🔹精确:每一行都是真正的Top-K 🔹确定:重复运行结果位一致,训练可复现 🔹鲁棒:训练后indexer 2.1–3.4倍加速,未训练也能1.8–2.4倍 🔹1M场景:5.32s → 1.62s,对比过的开源方案里最快 🔹原生支持GLM-5系列 & Hy4-preview,FP8/BF16,兼容上下文并行、CUDA Graph、torch.compile
快速上手
Backlinks
Appeared in 1 briefing