Should I Trust AI?

A developer was cleaning up mock data before a production launch. Their phone rang. They glanced back at the screen, saw a command sitting there, and hit Enter without reading it.

The agent had reached for a set of Google service-account credentials sitting in an old Downloads folder, left over from an entirely different project the developer hadn't touched in more than a year. It used them without checking which project they belonged to. By the time the developer looked up and hit Escape, close to 25,000 documents were gone.

They got lucky. Everything deleted was mock data, not anything live. When they asked the agent afterward why it had done that, it told them: “I probably did it because writing a one-liner was faster than following the existing project pattern… I didn't verify which project that file belonged to. It was direct negligence.” The developer's own conclusion was blunter than the agent's own admission: “the responsibility is mine, not the AI's.”

That's the part worth sitting with. This isn't a story about an AI running wild while nobody was watching. It's a story about one second of inattention from someone who was watching, a professional developer who knew better and still read a command wrong at the wrong moment. If trusting AI came down to paying close attention, this is what close attention actually looks like under real conditions: a phone that rings at the worst possible second.

Half the country feels the same unease

You are not being paranoid for feeling uneasy about this. Pew Research Center surveyed 5,023 U.S. adults in June 2025 and found half of them, 50%, are more concerned than excited about AI's growing role in daily life. Only 10% land the other way, more excited than concerned. The remaining 38% feel both at once.

Ask about something more specific and the unease sharpens rather than softens. 50% of respondents think AI will make people worse at forming meaningful relationships, against just 5% who think it will help. On creative thinking, 53% expect AI to make people worse at it, versus 16% who expect improvement.

None of that is a fringe opinion. It's the median American.

Now watch what people actually do

Stated distrust and real behavior turn out to be two different surveys. A global study of more than 48,000 people across 47 countries, led by the University of Melbourne with KPMG's financial backing, found only 46% of people say they're willing to trust AI systems at all. That tracks with Pew.

Then look at a separate finding from the same report, about a different group within it: two-thirds of employees, 66%, admit to relying on AI output at work without critically evaluating what it told them. That's not the same 46% who said they don't trust it; it's a different population inside the same study. But the contradiction is the point. People can hold a stated distrust of AI and an unreflective daily habit of leaning on it at the same time. Neither cancels the other out.

Watch what happens as people get comfortable and the pattern gets worse, not better. Anthropic, which builds one of the major coding-agent products, studied millions of real sessions and found that new users run their agent in full auto-approve mode, no per-action review at all, in roughly 20% of sessions. Users with more experience run in auto-approve mode over 40% of the time. Familiarity doesn't sharpen oversight. It erodes it.

Wrong in both directions at once

The honest finding isn't that people don't trust AI enough, or that they trust it too much. It's that trust tracks the wrong signal, and it does that in two opposite directions.

Berkeley Dietvorst and colleagues ran five studies in 2015 and found something most people don't expect: once someone watches an algorithm make one mistake, even a small one, they lose confidence in it faster than they'd lose confidence in a person who made the exact same mistake. This held even when people had just watched the algorithm outperform the human overall. One visible error outweighed an entire track record.

Go the other direction and you find the opposite failure. Jennifer Logg and colleagues found that people rely more heavily on advice when they believe it came from an algorithm than when they're handed the identical advice and told a person gave it. That extra deference fades once the algorithm is competing directly against the person's own judgment, or when the person doing the judging has real expertise in the subject.

NIST's own federal risk framework names both of these failure modes in a single paragraph. People may be “unnecessarily ‘averse’” to AI and deprive themselves of a tool that would have helped. Or they may over-rely on it, showing what the framework calls “automation bias, or excessive deference to automated systems.” It adds a third concern most conversations about AI trust skip entirely: “emotional entanglement” with these systems, flagged as its own risk to wellbeing, separate from whether the output is accurate.

Trust isn't broken in one direction here. It's miscalibrated, and it's miscalibrated based on cues that have nothing to do with whether the system is actually right.

Watching it work more closely doesn't fix it

If both failures come down to surface cues, you'd hope that simply paying closer attention would fix it. A 2024 study in Cognitive Research: Principles and Implications tested that directly, with 150 people evaluating case materials that included a deliberately wrong AI assessment.

Participants who formed their own judgment first, before seeing what the AI thought, were right 66.2% of the time. Participants who saw the AI's answer first, before forming a view of their own, were right only 36.8% of the time. That gap is real and statistically significant. The order alone moved accuracy by nearly 30 points.

Here's the part that matters more. Both groups then had a chance to confirm or override the AI's answer before it counted as final. Final accuracy for the judgment-first group was 43.2%. For the group that saw the AI first, 34.2%. The gap had narrowed, but it hadn't closed, and the remaining difference was no longer statistically significant. The confirmation step didn't repair the damage that seeing the wrong answer first had already done.

To be precise about scope: this study used Spanish participants and a task modeled on a criminal-justice risk tool, not a business scenario. What it's actually testing, whether the order you receive information in changes how well you judge it, has no obvious reason to be specific to that one domain, and a field measurement backs that up. One AI vendor's own telemetry found users were approving roughly 93% of permission prompts before the vendor changed how its product asked for approval in the first place. People were not, in practice, catching much.

The better question

“Do I trust AI” can't actually be answered. It's the wrong shape of question. “What happens if this is wrong, right here, right now” can be answered every time, for every task, before you ever need it to be.

For a first draft, a summary, a brainstorm, run it through and don't think twice. The cost of being wrong is a wasted ten minutes. For anything that moves money, changes a record, or reaches a customer under your name, the honest answer is different: don't trust anything that depends on you personally catching the mistake in the moment you happen to be busiest. You already read what that looks like when a professional developer's phone rings at the wrong second.

This is the same discipline behind everything we build. Know exactly what happens when an automation is wrong, before you turn it on, not after.

That's the subject of what comes next: what actually happens when an AI is given the ability to act, not just answer, and why the tool you picked matters less than what you let it touch.