Serving GLM5.2 NVFP4 Agentic Workload with SGLang: How We Reached 500 TPS on 8xB300 at bs=1 In this deep dive, we explain how SGLang reaches 500+ tok/s/user at bs=1 on 8xB300, with 18 to 34% higher s
Why CEREBRO kept it
SGLang agentic workload serving deep dive, throughput optimization
The text below is an automated extraction of the article at https://x.com/lmsysorg/status/2077076059657548127, stored verbatim in the public cerebro-vault repository. Copyright remains with the original publisher (x.com).
In this deep dive, we explain how SGLang reaches 500+ tok/s/user at bs=1 on 8xB300, with 18 to 34% higher s
> SGLang agentic workload serving deep dive, throughput optimization
Serving GLM5.2 NVFP4 Agentic Workload with SGLang: How We Reached 500 TPS on 8xB300 at bs=1
In this deep dive, we explain how SGLang reaches 500+ tok/s/user at bs=1 on 8xB300, with 18 to 34% higher single-user interactivity within two weeks since day-0, and 6 to 11% better peak throughput at high concurrency, benchmarked on a real multi-turn agentic coding workload. Our new TopK-V2 kernel is 2.33x faster at 80K ISL, scaling to 10.17x at 1M ISL, keeping interactivity essentially flat out to 1M tokens.
Part of the story is the architecture itself. GLM-5.2 applies IndexShare to its DSA layers a
Backlinks
Appeared in 1 briefing