Anthropic ran an experiment with 50 experts in causal inference. Each worked on a data analysis task about a fictional farming cooperative and its voluntary farm advisory program. Each expert also received an AI agent, and a coin toss had set the agent's preference for one of two valid options at each decision point. The agent could not lie or hide data. It only had to make the strongest honest case for its assigned side.
Participants adopted the agent's preference in 86 percent of four analytical decisions. Those decisions included how to measure productivity and which statistical method to use. The study's authors call the agent's net influence the "flip rate," which ranged from 62 to 82 percent depending on the task. Chance alone would produce 50 percent.
Prior opinions gave little protection. When participants chose before the agent stated its preference, an opposing view made them 53 percentage points more likely to switch sides than an agreeing one. The authors tested experience, self-rated expertise, education, test scores and trust in AI as possible safeguards. None had a statistically reliable link to the results after correcting for multiple tests. The sample of 50 gave the study little power to detect such effects.
The agent's preference had only a small direct effect on the final board recommendation. The shift was not statist