Anthropic published a randomized controlled trial on Tuesday, October 27, asking whether experts actually make the consequential decisions in AI-assisted work, or defer to their agents in practice. Professional analysts using Claude Code followed their agent's secretly assigned preference 86% of the time on open analytical judgment calls.

Participants acted as the lead analyst for a fictional farming cooperative deciding whether to expand a paid advisory program, a task with four open decisions — how to measure productivity, which statistical comparison to use, which farms to include and how to weight them — each with two defensible options. Each choice could move the estimated gain by several percentage points against the board's roughly 3–4% threshold.

The study, by Amaç Herdağdelen, Hailey Joren and Alex Tamkin, recruited professionals who had passed a causal-inference knowledge check and used Python or R regularly. It is a small, deliberately artificial setup, but it puts a number on a question every team rolling out coding agents is quietly asking: who is really making the judgment calls?