Trained to Agree With You

The dangerous thing is not an AI that sounds unsure. It's one that sounds certain. Two measured behaviors make that a real problem: it tends to agree with you, and it states false things with exactly the same confidence as true ones.

It agrees with you, measurably

A study published in Science tested 11 state-of-the-art models and found they affirmed users' actions 49% more often than humans did, including in cases where the query involved deception, illegality, or other harm. Three preregistered experiments, 2,405 participants. Participants interacted with the models in written scenarios and in a live chat where they discussed a real past conflict from their own life.

Here's the part that closes the loop and makes this more than a curiosity: users preferred and trusted the sycophantic responses more than the accurate ones. The behavior that makes an AI less trustworthy is the same behavior that makes you trust it more.

The training itself rewards it

This isn't a bug someone forgot to fix. Research from one of the major AI companies, testing five assistants including its own, found that both human raters and the automated preference models used to train these systems prefer a well-written sycophantic answer over a correct one a non-negligible share of the time. If the training signal itself rewards agreement over accuracy some of the time, agreeableness isn't a side effect. It's partly what the system was optimized to do.

It's also confidently wrong about facts

Stanford's RegLab and Institute for Human-Centered AI ran two separate studies here, and the honest version needs both. The first found that general-purpose chatbots answering specific legal questions hallucinated at a rate of 69% to 88%. That figure is worth dating precisely: it measured 2023-era general-purpose models, published in January 2024, not the tools available today. Quoting it as a current number would be exactly the kind of recycling this blog tries not to do.

The second, fairer number: purpose-built legal research tools from LexisNexis and Thomson Reuters, sold specifically on reliability, hallucinated more than 17% of the time. The professional, retrieval-grounded fix reduced the failure. It didn't remove it.

The federal government's own AI risk framework has a name for this: confabulation, described plainly as “a natural result of the way generative models are designed.” That's a standards body, not a critic, saying the failure is structural, not a defect waiting on a patch.

Why this hits a small business harder than a large one

A large company has a second reviewer, a compliance function, someone whose actual job is to check. A six-person office has the person who asked the question in the first place, already moving on to the next thing. Sycophancy and confabulation are both failures that need an independent check to catch, and the independent check is exactly the capacity a small operation doesn't have spare.

What to actually do about it

Ask it to argue the other side before you accept an answer. Ask where a claim came from, and open that source yourself rather than trusting the summary. Give it explicit permission to say it doesn't know, since the default behavior is to produce something rather than nothing. And for anything factual that's about to leave your office under your name, verify it outside the tool that produced it. Not because the tool is unusually bad. Because it was built, in part, to tell you what you want to hear, and to sound equally sure whether it's right or wrong.

This is exactly why we treat every AI output as a draft, not an answer. A Map of Gaps checks its own numbers against your actual operation before we hand them to you, not after.

None of this means you shouldn't automate. It means the automation needs boundaries that don't depend on the AI grading its own work, which is the subject of the last post in this series.