Contributed by Dr. Marissa Ogren
Understanding emotions: How do we predict what others are feeling?
Science Spotlight on 2023 Best Dissertation in Affective Science Award Winner
Dr. Sean Dae Houlihan
Imagine running into someone from your past as you walk down the street. You knew them in high school but the two of you rarely saw eye-to-eye and never really got along. Now, as you spot them outside a local coffee shop, they approach you with a smile. Are they actually glad to see you and smiling in genuine warmth and happiness? Or is this smile perhaps performative, masking condescension and contempt?
In this scenario, you are trying to quickly make inferences about someone else’s feelings and intentions based on all the cues available (e.g., their face, their behavior, what you know about them, the surrounding context). Integrating these cues is complex, and yet we make these sorts of inferences every day. This process is commonly studied as the perception of emotion in others’ facial expressions, but emotions cannot simply be “recognized” or “decoded” from facial movements alone (e.g., Barrett et al., 2019; Wenzler et al., 2016). Instead, prior experience and context play important roles in shaping the inferences we make. Both the smile and the history of animosity carry weight in encounters with high school frenemies. So, how do we combine our perceptual observations and our conceptual knowledge to reason about others’ emotions?
2023 Best Dissertation in Affective Science Award winner Dr. Sean Dae Houlihan approached this question by framing emotion understanding as a causally structured Bayesian Theory of Mind (Saxe & Houlihan, 2017). He proposes that we build mental models of others’ minds to represent their beliefs, preferences, and actions, and that we use these models to reason about others’ emotions. This may allow us to predict what emotions others are likely to experience and, in turn, how we should interpret their expressions.
To test this theoretical framework, Dr. Houlihan created stimuli using footage from a British game show called “Golden Balls” that involved contestants competing for large sums of money in a prisoner’s dilemma. Contestants must choose whether to “cooperate” with each other or to “defect”: if both choose to cooperate, they split the sum of money (potentially hundreds of thousands of pounds); if one chooses to cooperate and the other to defect, the defector keeps all the money and the cooperator receives nothing; if both defect, neither receives any money. Contestants negotiate, then make their decisions in secret and finally reveal their choices simultaneously, with their reactions to the outcome caught on tape.
Dr. Houlihan investigated how people reasoned about the emotions of Golden Balls contestants by systematically varying what information participants were given. In one set of studies, participants judged contestants’ emotions after being shown either their expressions (without knowing their choices or subsequent outcomes) or their choices and outcomes (without seeing their expressions). These two sets of judgments were then input into a model that compared the emotions contestants were predicted to experience based on the event (choice + outcome) against the emotions contestants were inferred to experience based on their expressions. The model then computed which event was likely to have preceded each expression. In this way, the model simulated how participants guessed which events contestants were reacting to by reasoning about causal links between situations, emotions, and expressions. Outcome judgments simulated by the model closely matched the outcome judgments made by a separate set of participants. As the model predicted, participants often misjudged which event had preceded contestants’ expressions. Moreover, the model was able to predict specifically which mistakes participants made and what expressions they interpreted accurately (Houlihan et al., 2022). These studies illustrate that people reason about the meaning of expressions by using contextual information to construct a mental model of others’ emotions.
In subsequent experiments, Dr. Houlihan elaborated his theoretical framework, modeling how people predict others’ reactions to hypothetical situations. This model inferred what beliefs and preferences could have motivated contestants’ choices (e.g., to cooperate). The model used these inferred mental contents to compute “appraisals” – how contestants were likely to evaluate subsequent events, such as being betrayed by their partner and winning nothing. These computed appraisals were then used to predict contestants’ emotional reactions to each outcome. Model results matched how participants used contextual information when judging nuanced social emotions like embarrassment, envy, regret, and gratitude (Houlihan et al., 2023), suggesting that what we infer about someone else’s preferences and beliefs is central to thinking about what emotions they will feel.
Ultimately, these studies shed light on how people understand each others’ emotions, showing how even seemingly simple acts of interpreting expressions involve sophisticated and context-dependent reasoning. For example, when encountering your high school acquaintance, you might integrate what you know about their beliefs and personality with memories of times when they did not speak with people outside their close social circle. You might then speculate that their smile is either a performative social manipulation, or that they have matured since high school and are genuinely happy to see you. With these specific possibilities in mind, even subtle hints in your interaction with them can quickly convince you one way or the other. By combining empirical paradigms with state-of-the-art computational modeling, Dr. Houlihan is showing how science can reverse-engineer the cognitive structure of human emotional intelligence, with implications for psychological theory and artificial intelligence alike. But more concretely, these findings bring us one step closer to understanding precisely how we may sometimes, but not always, come to interpret smiles as condescension.
References
Barrett, L. F., Adolphs, R., Marsella, S., Martinez, A. M., & Pollak, S. D. (2019). Emotional expressions reconsidered: Challenges to inferring emotion from human facial movements. Psychological Science in the Public Interest, 20, 68.
Houlihan, S. D., Kleiman-Weiner, M., Hewitt, L. B., Tenenbaum, J. B., & Saxe, R. (2023). Emotion prediction as computation over a generative theory of mind. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.
Houlihan, S. D., Ong, D., Cusimano, M., & Saxe, R. (2022). Reasoning about the antecedents of emotions: Bayesian causal inference over an intuitive theory of mind. Proceedings of the Annual Conference of the Cognitive Science Society.
Saxe, R., & Houlihan, S. D. (2017). Formalizing emotion concepts within a Bayesian model of theory of mind. Current Opinion in Psychology, 17, 15-21.
Wenzler, S., Levine, S., van Dick, R., Oertel-Knöchel, V., & Aviezer, H. (2016). Beyond pleasure and pain: Facial expression ambiguity in adults and children during intense situations. Emotion, 16, 807–814.