JOURNAL TRANSCRIPT
www.nature.com/scientificreports
OPEN
Face size biases emotion judgment through eye movement Shuo Wang 1,2
Received: 19 September 2017 Accepted: 16 December 2017 Published: xx xx xxxx
Faces are the most commonly used stimuli to study emotions. Researchers often manipulate the emotion contents and facial features to study emotion judgment, but rarely manipulate low-level stimulus features such as face sizes. Here, I investigated whether a mere difference in face size would cause differences in emotion judgment. Subjects discriminated emotions in fear-happy morphed faces. When subjects viewed larger faces, they had an increased judgment of fear and showed a higher specificity in emotion judgment, compared to when they viewed smaller faces. Concurrent high-resolution eye tracking further provided mechanistic insights: subjects had more fixations onto the eyes when they viewed larger faces whereas they had a wider dispersion of fixations when they viewed smaller faces. The difference in eye movement was present across fixations in serial order but independent of morph level, ambiguity level, or behavioral judgment. Together, this study not only suggested a link between emotion judgment and eye movement, but also showed importance of equalizing stimulus sizes when comparing emotion judgments. Faces are among the most commonly perceived visual stimuli and play a key role in social communication. People often form judgments of others based purely on facial features and trait evaluations from faces can predict important social outcomes. For example, inferences of competence based solely on facial appearance predict the outcomes of elections1 and facial features can influence sentencing decisions2. On the other hand, faces are socially salient stimuli and people preferentially attend to faces3. For example, people detect faces faster than inanimate objects (e.g., plants and artifacts) in the change detection task4 and people orient more and faster to faces in natural scenes5. The way to look at faces has both a developmental6 and genetic7 root, and it often serves as a biomarker for autism, which shows atypical attention to faces5,8. Humans have a dedicated and distributed network of brain regions to process faces. Intracranial field potential studies in neurosurgical patients9 and functional imaging studies10 have both provided evidence that cortical areas in the lateral parts of the inferior occipital gyrus, fusiform gyrus, and superior temporal gyrus are associated with face processing (see11 for a review). In particular, faces signal important information through expressions of emotions, which in turn provide a strong motivating influence on how the environment is perceived12. A large number of brain regions participate in recognizing emotions from facial expressions, including the occipitotemporal cortices, amygdala, orbitofrontal cortex, basal ganglia, and right parietal cortices (see13 for a review), among which the amygdala plays a key role in processing facial emotions: the human amygdala encodes not only fear emotion14,15 and emotions in general16, but also subjective judgment of facial emotions17 and categorical ambiguity of emotions15. A recent proposal argues that emotion should be understood in terms of large-scale network interactions spanning the entire neuro-axis18. Perception of facial expressions is closely related to eye movement. For example, more fixations are directed to the eye region when people view fearful faces whereas relatively more fixations are directed to the mouth region when people view happy faces19. Also, eyes contain more information for fearful faces but the mouth contains more information for happy faces20. Human neuroimaging studies have shown that amygdala activity is specifically enhanced for fearful faces and saccades to the eyes21, and monkey physiological studies using dynamic social videos with various facial expressions have revealed a subset of neurons in the amygdala that respond selectively to fixations on the eyes of other monkeys and to eye contact22. A recent computational framework with novel spatiotemporal analyses of eye movements has provided theoretical insights and empirical evidence for the computational mechanisms underlying perception of facial expressions23. This framework has also revealed culture-specific decoding strategies of facial expressions, arguing against the universality of human facial expressions of emotion24. 1
Department of Chemical and Biomedical Engineering, West Virginia University, Morgantown, WV, 26506, USA. 2Blanchette Rockefeller Neurosciences Institute, West Virginia University, Morgantown, WV, 26506, USA. Correspondence and requests for materials should be addressed to S.W. (email:
[email protected])
SCientifiC REPOrtS | (2018) 8:317 | DOI:10.1038/s41598-017-18741-9
1
www.nature.com/scientificreports/ Most studies up to date focus on the diagnostic facial features for emotion judgment (e.g.,20) and investigate neural correlates of facial expressions by manipulating the emotion contents (e.g.,25). However, it remains unclear whether a simple low-level feature, face size, will influence emotion judgment. In this study, I investigated whether a mere difference in face size would cause different emotion judgments. I employed an emotion judgment task with fear-happy morphed faces. Indeed, when subjects viewed larger faces, they not only had a lower threshold to detect fear on the face, but also showed a higher specificity in emotion judgment. However, subjects showed a similar confidence judgment between face sizes. Concurrent eye tracking further provided insights into the underlying mechanism: more fixations were directed to the eyes when people viewed larger faces whereas there was a wider spatial dispersion of fixations when people viewed smaller faces. This difference was present across fixations in serial order, but independent of the morph level, ambiguity level, and behavioral judgment. Together, this study not only suggested a link between emotion judgment and eye movement, but also showed importance of choosing stimulus size to study emotion judgment.
Results
Emotion judgment. Subjects performed an emotion judgment task (Fig. 1A) with “anchor” (unambiguously happy or unambiguously fearful) and morphed faces (Fig. 1B). Psychometric curves were fitted for each subject (Eq. 1). The proportion of trials identified as fearful monotonically increased as a function of fearful level in the face (Fig. 1C). Two metrics from the fitted psychometric curves were used to compare emotion judgment. First, xhalf, the midpoint of the curve with equal proportions of fearful and happy judgment, shows emotion judgment bias. I found that subjects had a significantly smaller xhalf for large faces compared to small faces (Fig. 1D; large: 48.2 ± 5.09 (mean ± SD), small: 51.3 ± 5.90; paired two-tailed t-test: t(22) = 3.65, P = 0.0014, effect size in Hedges’ g (standardized mean difference): g = 0.56), suggesting that they were more likely to judge faces as fearful when viewing large faces. Second, α, the steepness of the psychometric curve, shows emotion judgment sensitivity and specificity. I found that subjects had a greater α for large faces (Fig. 1E; large: 0.14 ± 0.061, small: 0.12 ± 0.045; t(22) = 2.29, P = 0.032, g = 0.39), showing that they had steeper psychometric curves when viewing large faces, which in turn suggested that subjects were more sensitive and specific in their emotion judgment when they viewed large faces. Together, my results show that different face sizes led to not only different thresholds for judging emotions, but also different sensitivity and specificity of emotion judgment. Confidence judgment. Besides emotion judgment, subjects also provided confidence judgment in their decisions (Fig. 1A). There were three levels of confidence: ‘Very Sure’, ‘Sure’, and ‘Unsure’. First, I found that subjects reported high confidence more often than low confidence (Fig. 2A,J; one-way repeated-measure ANOVA of confidence levels; large: F(2,46) = 11.2, P = 1.07 × 10−4, η2 = 0.33; small: F(2,46) = 12.9, P = 3.57 × 10−5, η2 = 0.36). They judged emotions faster (Fig. 2B,K; large: F(2,43) = 19.6, P = 2.40 × 10−7, η2 = 0.25; small: F(2,45) = 33.5, P = 1.43 × 10−10, η2 = 0.31) and reported confidence faster (Fig. 2C,L; large: F(2,43) = 7.80, P = 9.22 × 10−4, η2 = 0.12; small: F(2,45) = 18.2, P = 6.11 × 10−7, η2 = 0.19) when they had higher confidence. However, this was similarly the case for both large and small faces (two-way repeated-measure ANOVA of face size X confidence level; main effect of face size: all Ps > 0.88; main effect of confidence level: all Ps 0.77). Second, when I analyzed confidence judgment as a function of morph level, I found that subjects reported higher confidence for anchor faces but lower confidence for ambiguous faces (Fig. 2D,M; one-way repeated-measure ANOVA of morph levels; large: F(6,138) = 42.0, P = 7.54 × 10 −29, η 2 = 0.28; small: F(6,138) = 37.2, P = 1.37 × 10−26, η2 = 0.30). I found a similar relationship not only for reaction time (RT) of emotion judgment (Fig. 2E,N; large: F(6,138) = 19.0, P = 4.93 × 10−16, η2 = 0.12; small: F(6,138) = 19.0, P = 4.64 × 10−16, η2 = 0.11), but also RT of confidence judgment (Fig. 2F,O; large: F(6,138) = 2.17, P = 0.050, η2 = 0.0086; small: F(6,138) = 6.33, P = 6.68 × 10−6, η2 = 0.019). However, large faces and small faces had a similar pattern of results (two-way repeated-measure ANOVA of face size X morph level; main effect of face size: all Ps > 0.68; main effect of morph level for explicit confidence rating: P = 1.40 × 10−56, RT of emotion judgment: P = 9.01 × 10−33, and RT of confidence judgment: P = 6.99 × 10−7; interactions: all Ps > 0.41). Third, when I analyzed confidence judgment as a function of ambiguity level (Fig. 1B,I found that subjects reported higher confidence for anchor faces but lower confidence for ambiguous faces (Fig. 2G,P; one-way repeated-measure ANOVA of ambiguity levels; large: F(2,46) = 72.6, P = 5.82 × 10 −15, η 2 = 0.26; small: F(2,46) = 82.8, P = 5.67 × 10−16, η2 = 0.31). I found a similar relationship for both RT of emotion judgment (Fig. 2H,Q; large: F(2,46) = 31.7, P = 2.25 × 10−9, η2 = 0.12; small: F(2,46) = 38.5, P = 1.51 × 10−10, η2 = 0.12), and RT of confidence judgment (Fig. 2I,R; large: F(2,46) = 3.71, P = 0.032, η2 = 0.0082; small: F(2,46) = 9.67, P = 3.12 × 10−4, η2 = 0.018). However, again, large faces had a similar pattern of results as small faces (two-way repeated-measure ANOVA of face size X ambiguity level; main effect of face size: all Ps > 0.62; main effect of ambiguity level for explicit confidence rating: P = 3.45 × 10−30, RT of emotion judgment: P = 1.04 × 10−18, and RT of confidence judgment: P = 2.64 × 10−5; interactions: all Ps > 0.22). Together, I found very similar patterns of confidence judgment between large vs. small faces, suggesting that face size did not influence confidence judgment. Different face sizes led to different spatial distributions of fixations. Could the difference in emotion judgment be attributed to difference in eye movement? To investigate this question, I next analyzed the spatial distribution of fixations. For each subject, I collapsed fixations from all trials during the 1 s stimulus period. I found that when subjects viewed both large and small faces, the fixation density distribution was symmetric along the horizontal dimension. However, I found a significant difference along the vertical dimension: the fixation density distribution for large faces shifted up towards the eyes, whereas the distribution for small faces remained SCientifiC REPOrtS | (2018) 8:317 | DOI:10.1038/s41598-017-18741-9
2
www.nature.com/scientificreports/
Figure 1. Emotion judgment. (A) Task. Subjects viewed a face for 1 second and reported their emotion judgment (fearful or happy). Following emotion judgment, subjects also reported their confidence in emotion judgment (‘1’ for ‘very sure’, ‘2’ for ‘sure’ or ‘3’ for ‘unsure’). (B) Example faces from a female face model. Face stimuli were constructed from44. (C) Psychometric curves for large vs. small faces. Shaded area denotes one SEM across subjects. The top green bar illustrates the points with significant difference between large vs. small faces (paired two-tailed t-test, P