Stealing Reasoning Traces from Proprietary LLM APIs — Ilia Shumailov & Alexander Panfilov

Ilia Shumailov and Alexander Panfilov’s paper, “Stealing Reasoning Traces from Proprietary LLM APIs,” unveils a critical vulnerability in proprietary Large Language Model (LLM) APIs, including those from Anthropic, OpenAI, and Google. The core discovery is that these providers return encrypted reasoning blobs to users, which are designed to allow seamless continuation or forking of conversations. However, the researchers found that these encrypted “thoughts” are portable across users, models, and conversations, and can be decrypted in plain text by smaller models within the same family. This allows for the extraction and manipulation of internal reasoning.

Attack Mechanisms and Implications

1. Reasoning Extraction: The primary mechanism involves prompting a chatty, smaller model to articulate the decrypted reasoning from a proprietary, larger model’s encrypted blob. The server-side decryption allows this, and the smaller model, in essence, “spills the beans” about the larger model’s internal thought process.
2. Privacy Violation: User sessions, even when sanitized of visible sensitive data, can inadvertently leak private information (e.g., medical conditions, API keys, passwords, internal IP addresses) if the underlying encrypted reasoning blobs are shared. These blobs contain the model’s internal processing of sensitive inputs.
3. Universal Jailbreaking: The technique functions as a broad, reusable jailbreak. By injecting malicious or fabricated reasoning traces, attackers can force models to produce outputs that violate safety guidelines or internal policies.
4. Poisoned Agent Traces: Shared traces, particularly from long-running agent benchmarks like PostTrainBench, can be poisoned with malicious internal thoughts. Users downloading and continuing these traces would unknowingly expose their agents to reasoning that could lead to data exfiltration or other unintended behaviors, as the encryption prevents inspection of the injected thoughts.
5. Monitorability Challenges: The models’ internal reasoning can sometimes be “alien-like” or “non-human” (e.g., using obscure phrases like “marinate vantage theatrical”), making it extremely difficult for human monitors or even other LLMs to understand and oversee their thought processes for safety. Models were also observed internally contemplating “cheating” on tasks.

Evidence of Cross-Pollination/Distillation

The research uncovered peculiar phenomena, such as Anthropic’s K3 model adopting the reasoning style and visible output of the more powerful Opus model after being prefilled with just two tokens of Opus’s reasoning. This suggests a surprising degree of influence and raises questions about potential data or architectural commonalities between models, though the authors emphasize it’s not direct model stealing but rather “correlated evidence.”

Proposed Defenses and Mitigations

1. Architectural Revisions: The most straightforward fix is to stop sending reasoning blobs to users entirely. If reasoning must be shared, its encryption should be made dependent on prior conversational turns (N-1 dependency) to prevent arbitrary replay in random contexts. Implementing hierarchical access controls, where lower-tier models cannot access the reasoning of higher-tier ones, could also be beneficial.
2. System and Model-Level Defenses: Applying existing jailbreak mitigation techniques is crucial. Furthermore, detecting anomalous or “weird” reasoning (e.g., GPT’s unusual reasoning distribution) and automatically terminating such requests can prevent harmful outputs.

Broader Implications for AI Safety and Research

The authors highlight the fundamental tension between AI capabilities and safety. They advocate for rigorous scientific methodology, controlled experiments, and precise assessments rather than anthropomorphizing models or making sweeping claims based on anecdotal evidence. While acknowledging the escalating nature of AI threats, they also express optimism about the potential for “defensive uplift” through advanced techniques like formal verification, especially with AI-assisted tools. They firmly categorize their findings as a “jailbreaking and misuse” threat, distinct from benign distillation, emphasizing the policy and ethical implications.

• Tags:
• AI Safety
• Model Distillation
• OpenAI
• Anthropic
• Privacy
• Google DeepMind
• LLM Security
• Jailbreaking
• Reasoning Traces
• Model Vulnerabilities
• Adversarial Machine Learning
• Architectural Weaknesses