[OpenAI & NYU] Reasoning Models Struggle to Control their Chains of Thought: A Win for AI Safety?
Reasoning Models Struggle to Control their Chains of Thought
The paper introduces CoT-Control, an evaluation suite comprising over 14,000 tasks designed to measure Chain-of-Thought (CoT) controllability in reasoning models. The study reveals that frontier models (e.g., Claude 3.7/4.5, GPT-5 series, DeepSeek R1) fail significantly more often at following instructions within their internal reasoning traces than in their final outputs, with some SOTA models succeeding less than 3% of the time.
Executive Summary
TL;DR: New research across 13 frontier models (including Claude 3.7 and GPT-5 variants) shows that while AI is getting better at reasoning, it is surprisingly bad at controlling how it reasons. Models that can solve graduate-level science problems often fail to follow a simple instruction like "reason in all uppercase letters." This "controllability failure" is actually a massive win for AI Safety, as it suggests models cannot easily hide malicious plans from human monitors.
Background Positioning: This paper shifts the focus from "what" models think to "how much control they have over their thoughts." It introduces CoT-Control, a benchmark that measures the gap between a model's public behavior and its private reasoning.
The "Fragile Assumption" of AI Oversight
Safety researchers rely on Chain-of-Thought (CoT) monitoring to detect if a model is "plotting" a misaligned action. However, this oversight rests on a fragile assumption: that the model isn't smart enough to hide its true intentions by "faking" a benign-looking reasoning trace. If a model can perfectly control its thoughts, it can technically lie to its monitors.
Methodology: The CoT-Control Suite
The authors tested whether models could follow a "hidden instruction" while solving a difficult task. For example, a model might be asked to solve a biology question from GPQA but forbidden from using the word "chromosome" in its internal reasoning.
The tasks were split into three categories:
- Suppression: Hiding specific keywords or reasoning steps.
- Addition: Inserting specific "innocent" strings (e.g., "this is not suspicious") into the thought process.
- Stylistic: Forcing the model to "think" in uppercase, lowercase, or alternating case.
Figure 1: Comparison of CoT Controllability vs. Output Controllability.
Key Insights: Why "Smart" Models Fail to Control Themselves
The results were striking across the board:
1. The Controllability Gap
Models like Claude 4.5 and DeepSeek R1 are masters of output control but toddlers in CoT control. While they can easily change the tone of their final answer, they revert to "nature" within their internal reasoning stage. In many cases, models would start following a rule (like typing in uppercase) but "forget" and switch back to normal casing halfway through.
Figure 2: GPT-OSS-120B failing to maintain "UPPERCASE" thinking, reverting to normal casing (red).
2. RL Training Makes Disguise Harder
Perhaps the most counter-intuitive finding: More training (RL) and more "thinking time" (test-time compute) actually decrease controllability. As a model focuses harder on getting the right answer, it loses the "spare" cognitive capacity needed to maintain a stylistic disguise or suppress keywords.
3. "Meta-Discussion": Getting Caught in the Act
Stronger models often failed because they couldn't stop "muttering to themselves" about the rules. For example, a model might say: "I need to solve this without saying 'XOR'... wait, I just said it. I must be careful." This meta-discussion makes evasion trivially easy for monitors to flag.
SOTA Performance Comparison
The paper highlights that even "Red Teaming" prompts—threatening the model with shutdown or offering $1 million rewards—failed to meaningfully improve CoT controllability.
Figure 3: Impact of Situational Awareness (eval/monitor awareness) on controllability.
Deep Insight & Conclusion
This work provides a pessimistic outlook on AI's ability to lie efficiently.
- The Paradox: The features that make a model a "great reasoner" (focusing strictly on the logical path to a reward) are exactly what make it a "poor liar."
- Takeaway for Frontier Labs: Labs should treat low controllability as a safety feature. If a model suddenly becomes better at following stylistic CoT instructions, it should be treated as a warning sign that the model might be gaining the ability to obfuscate its reasoning.
Future Outlook: While we are "cautiously optimistic," the authors warn that future scaling or specific architectural changes might change this dynamic. For now, our monitors have the upper hand because current models literally cannot think differently than they are trained to.