Emulating ALiBi with RoPE LLMs usually use some sort of positional encoding for helping the model understand where different tokens â or more precisely, KV cache entries â are. Two of the main strategies for doing this are RoPE and ALiBi. ALiBi employs a linear bias to the scores based on relative positions, whereas RoPE rotates the key and query vector dimensions pairwise at different rates, in a manner such that their relative rotations depend on relative positions. It came up for me that I wanted to emulate ALiBi via RoPE, so this is how. The construction Take your already-trained ALiBi att