In the world of artificial intelligence, the term "reasoning" has become a buzzword, especially with the rise of large reasoning models (LRMs) that generate step-by-step explanations for their answers. But a growing body of research suggests that these models might be getting the right answers for the wrong reasons, relying on superficial patterns rather than genuine logical inference.
The promise and the pitfalls
LRMs, such as OpenAI's o1, are designed to improve on traditional large language models (LLMs) by generating "reasoning traces"—explicit chains of thought that mirror human problem-solving. These traces are meant to make the model's process transparent and auditable. However, recent findings indicate that these traces may be more of a performance-enhancing trick than a faithful record of the model's internal computations.
Evidence of shortcuts
Melanie Mitchell, a computer scientist at the Santa Fe Institute, has been at the forefront of this research. Her team demonstrated that LRMs can ace carefully designed reasoning benchmarks using "surface-level shortcuts," essentially gaming the system rather than engaging in generalizable reasoning. Meanwhile, a study from New York University showed that "meaningless filler tokens"—like strings of dots—can replace human-readable chains of thought without degrading performance. This suggests that the linguistic content of these traces is not what drives the model's success.
Not faithful, not useful
William Merrill, a professor at the Toyota Technological Institute at Chicago and co-author of the NYU paper, puts it bluntly: "There's no guarantee the chain of thought has to be meaningful in any sense." Indeed, research from Northeastern University and UC Berkeley found that between 30% and 60% of thinking steps in some LRMs have "minimal causal impact" on the final answer. Removing half of these steps barely affects performance, raising questions about their actual role.
Bridging the gap
Despite these doubts, LRMs do outperform their predecessors on reasoning tasks. Mitchell acknowledges this, noting that "it works"—the models are more accurate. But she also emphasizes that the generated text is not necessarily faithful to what's happening inside the model, and much of it isn't even useful. This disconnect between output and internal process is a central puzzle for AI researchers.
Subbarao Kambhampati, a researcher at Arizona State University, describes these traces as "mumblings"—bits of language whose meaning may be incidental to any actual reasoning. His lab showed that replacing correct traces with incorrect or irrelevant ones doesn't hurt performance, further undermining the idea that these traces are essential.
Implications for the future
This research has significant implications for how we evaluate and trust AI systems. If we can't rely on their explanations, how can we verify their decisions? The field is still young, but the evidence suggests that we need to be cautious about attributing human-like reasoning to these models. As Pavel Izmailov, a researcher at NYU and Anthropic, notes, "The chances are not very high" that reinforcement learning will naturally produce faithful chains of thought.
For now, the scientific community is left with a paradox: AI reasoning seems to work, but not in the way we assumed. Understanding this gap is crucial for developing more reliable and transparent AI systems in the future.
