More severe in smaller models than in larger counterparts (Figure 2). What makes LLMs better than small LMs? Data? Parameters? Geometry might play a role! What makes LLMs better than small LMs? Data? Parameters? Geometry might play a role! Every Transformer layer of a language model represents each input token as a vector in a high-dimensional embedding space. We notice that as those vectors progress through Transformer layers, they often behave as if they were confined to a narrow cone: they point to increasingly similar directions as measured by pairwise cosine similarity. We call this geome