Study shows LLM explanations degrade human judgment

Researchers from Harvard Business School, MIT, and the University of Washington investigated how AI recommendations influence human judgment during early-stage innovation screening. The experiment tasked 228 experienced evaluators with reviewing nearly 50 submissions to an MIT challenge. The researchers compared three scenarios: human-only evaluations, black-box AI recommendations with simple pass-fail outputs, and large language model evaluations accompanied by written rationales. Evaluators' choices were measured against a baseline of decisions made by four independent human experts.

The results challenged the common assumption that explaining an AI's reasoning helps humans make better choices. Overall, evaluators accepted the LLM recommendations 67 percent of the time. They agreed with both the black-box and narrative LLM decisions about 75 percent of the time, but only agreed with the baseline human expert decisions 54 percent of the time. Surprisingly, while simple black-box recommendations actually improved the quality of human decisions, recommendations paired with narrative explanations did not. When the LLM provided a written rationale to reject a proposal, evaluators disproportionately agreed, which significantly increased false negatives by weeding out promising ideas.

According to the researchers, narrative explanations suppress what they call "productive overrides," where humans independently verify a model's output before agreeing. Because LLMs are linguistically fluent, they create an illusion of explanatory depth that makes their arguments highly persuasive. This interacts poorly with human negativity bias, as rejecting a proposal feels safer and requires fewer resources. The written AI explanations essentially provide ready-made justifications for evaluators to offload their critical thinking.

For enterprise practitioners designing AI-assisted workflows, these findings suggest that transparency tools can act as unintended behavioral interventions. In high-stakes scenarios like early-stage product screening, organizations should avoid detailed LLM rationales, as simpler or more opaque recommendations may better preserve human discretion. Conversely, narrative explanations might be better suited for conservative tasks like fraud detection or compliance screening where rejecting options is the preferred default.