The effects of emoji on cognition: evidence from eye movements, perception, and memory recall

Abstract

The literature investigating emoji, as pictographic visual cues applied in online communication, has historically focused on sender applications and interpretations of receivers in social contexts, despite their increasing usage in other contexts. This thesis aims to address gaps in knowledge within the empirical research by investigating the cognitive impact of emoji on accompanying textual information. Specifically, the syntactic and semantic integration of emoji and sentences, the role of social presence in the text, and memory encoding of emoji-accompanied text were investigated. Experiment 1 aimed to establish baseline findings by investigating the syntactic and perceptual effects of emoji in emotionally neutral, third-person narrative sentences. A sample of native English readers were asked to read sentences, with emotive face emoji in sentence-initial and sentence-final positions, naturally with gaze recorded via eye-tracking. Participant perceptual judgements of sentence valence were then collected. Results of mixed-effects models showed that sentence-final emoji were fixated on for longer than sentence-initial emoji. Likewise, when emoji were in a sentence-final position, readers had longer total fixation durations and higher fixation counts on centre-position target words and total sentence reading durations, indicative of regressive movements in parsing. However, no significant effects of emoji valence were found on the perceived emotional valence of sentences, converse to hypotheses and the predominant literature. Experiment 2 aimed to replicate and extend the research design by including positively and negatively valent third-person narrative sentences in the stimulus sets, and introducing an emotionally ambiguous emoji (i.e., the winking face emoji). Emoji positioning effects were replicated, with longer fixations on emoji in sentence-final, compared to sentence-initial, positioning. Likewise, positioning emoji in sentence-final positions increased the probability of fixating on centre-position target words. A significant emoji position  sentence valence interaction was also detected on probabilities of fixating on target words; there were significantly higher probabilities of fixating on target words when emoji were in sentence-final positions, but only when sentences were positively valent. However, failures to detect expected effects on perceived emotional valence were replicated in Experiment 2. The implications and conclusions drawn from Experiments 1 and 2 indicate that, while emoji instigate relevant semantic and syntactic processing in sentence reading, their perceptual impact on readers is minimal when they are used outside of their most typical context (i.e., communication). Experiment 3 was designed to evaluate the conclusions of Experiments 1 and 2 by reintroducing a basic characteristic of social communication. In addition to sentence valence and emoji valence, the sentence narrative point-of-view was experimentally manipulated to present readers with sentences in first-person or third-person narration. Likewise, control conditions (sentences with no emoji) were also presented to draw appropriate comparisons. Participants were asked to read sentences in an eye-tracking paradigm and then to rate sentences on their perceived emotional valence, believability, sincerity, and predictability. In gaze data, a significant main effect of emoji valence was found on target word probabilities of fixating, with higher probabilities when frowning emoji were included. An emoji valence  sentence valence interaction was also found on emoji region regression path durations (RPDs): incongruent emoji-sentence combinations drew longer RPDs than congruent conditions. This interactive trend was observed in total sentence reading durations of positive and negative sentences, but only when fixations in emoji regions were excluded. For sentence perception ratings, converse to Experiments 1 and 2, a significant emoji valence  sentence valence interaction was found on perceived emotional valence. Simple effect comparisons showed that the accompaniment of congruent smiling emoji in positive sentences elicited more positive valence ratings, compared to control sentences with lower ratings, and incongruent sentences (with a frowning emoji) which received the lowest rating scores on average. This is converse to negative sentences, in which the ratings across emoji conditions were comparatively equivalent. Likewise, a significant emoji valence  sentence valence interaction showed that congruent emoji-sentence combinations were rated as more sincere, believable, and predictable than incongruent combinations, with varying differences in comparisons with control sentences. However, no significant differences in conditions of narrative point-of-view were found. Experiment 4 aimed to assess whether the attentional impact of emoji observed in Experiments 1, 2, and 3 would translate to potential effects on memory encoding and retrieval of information. In an online experiment set during the COVID-19 pandemic, participants were presented with positive and negative sentences, which were accompanied by grinning or frowning emoji, or no emoji as a control condition. Following a distraction task, participants were then asked to recall the affective centre-position target word of each sentence. Results demonstrated significant main effects of sentence and emoji valence on memory recall. Although the main effect of emoji valence was significant, with sentences containing frowning emoji eliciting higher levels of recall based on descriptive statistics, differences were not statistically significant in subsequent pairwise comparisons. Positive sentences had higher proportions of correct recall than control sentences, with negative sentences having the lowest levels of recall. However, the interaction of emoji and sentence valence was marginally non-significant. The shift in detection of the emoji valence  sentence valence interactions between Experiments 1 and 2 compared to Experiment 3, in combination with the marginal effects observed in Experiment 4, contribute to the thesis’ overall conclusions. Firstly, the syntactic effects and integration of emoji are observed at a late stage in sentence processing but may not be the strong effects posited in the prior literature. Secondly, the perceptual impact of emoji on readers may be contingent on whether there are appropriate contextual cues concerning the social nature of text (i.e., whether it originates from an individual directly). Composite findings are discussed in reference to the prior scholarship, and theory and models relevant to syntactic processing, social-, and cyber-cognition.