Steven Gonsalvez

Software Engineer

Tencent-Hunyuan/FlashTopK

Why CEREBRO kept it

Sparse-attention top-K speedup, LLM inference mechanics, not agent-specific.

The text below is an automated extraction of the article at https://github.com/Tencent-Hunyuan/FlashTopK, stored verbatim in the public cerebro-vault repository. Copyright remains with the original publisher (github.com).

🤯惊了!比DeepSeek Sparse Attention更狠!Indexer Top-K瓶颈直接砍掉70%:5.32秒提升至1.62秒,腾讯混元×爱丁堡开源FlashTopK,实现开源最快,支持GLM-5系列 & Hy4-preview!

DeepSeek式Sparse Attention虽然省算力,但Indexer挑Token仍是大坑:传统要算全量分数表写进显存再取Top-K,1M长度光这一步就得5.32秒!

FlashTopK专为这类架构而生,把打分+选择融成一个算子,分数表根本不写显存: 1️⃣等距抽样定阈值 2️⃣边算边比,只把过阈值的Key写进小候选区 3️⃣从候选里精确选出真正的Top-K

核心亮点: 🔹精确:每一行都是真正的Top-K 🔹确定:重复运行结果位一致,训练可复现 🔹鲁棒:训练后indexer 2.1–3.4倍加速,未训练也能1.8–2.4倍 🔹1M场景:5.32s → 1.62s,对比过的开源方案里最快 🔹原生支持GLM-5系列 & Hy4-preview,FP8/BF16,兼容上下文并行、CUDA Graph、torch.compile

快速上手

Backlinks

Appeared in 1 briefing

Related

Shares tags: ai/llm-mechanics · cerebro/signal

Also from github.com