Cognitive Effects and Biases in Large Language Models

Cognitive effects such as anchoring, positional effect, or confirmation bias are core aspects of human decision making and reasoning. As LLMs increasingly act as communicative partners, reasoning tools, and evaluators, understanding how these cognitive effects influence their behavior and vice versa has become essential. While recent studies have adapted psychological experiments to detect cognitive biases in LLMs, they often use a particular kind of experimental setup from psychology that carries implications for human performance. In addition, current NLP studies often confuse cognitive effects with biases, diverging from their psychological foundations and overlooking potentially functional aspects of these phenomena. In this tutorial, jointly organized by NLP researchers and a cognitive psychologist and decision scientist, we aim to build shared conceptual and methodological ground between the two disciplines. We begin by outlining how cognitive effects and biases are defined, validated, and sometimes debated within psychology, highlighting differences and contradictions in experimental designs. We then bridge these insights to NLP through an overview of key studies examining cognitive biases in LLMs, mapping their methodological parallels and divergences. The tutorial also includes a hands-on component where participants explore the challenges of detecting a single cognitive bias (e.g., positional bias) in multilingual LLMs, illustrating the nuances and pitfalls of such evaluations. We conclude by discussing emerging research directions and open questions at the intersection of cognitive science and large language models.