A research paper by Sayam Goyal, Brad Peters, María E. Granda, Akshath V. Narmadha, Dharunish Yugeswardeenoo, Cole Blondin, Callum S. McDougall, Sean O'Brien, Ashwinee Panda, and Kevin Zhu has been accepted into the Mechanistic Interpretability Workshop at NeurIPS 2025.
Latent reasoning language models aim to improve reasoning efficiency by computing in continuous hidden space rather than explicit text, but the opacity of these internal processes poses major challenges for interpretability and trust.
We present a mechanistic case study of CODI (Continuous Chain-of-Thought via Self-Distillation), a latent reasoning model that solves problems by chaining "latent thoughts." Using attention analysis, SAE based probing, activation patching, and causal interventions, we uncover a structured "scratchpad computation" cycle: even numbered steps serve as scratchpads for storing numerical information, while odd numbered steps perform the corresponding operation.
This research provides important insights into how latent reasoning models process information internally, advancing our understanding of AI interpretability.
