Assessing the assessment: Measuring personality with Facebook status messages

What do our Facebook posts really say about us? Some dismiss them as just noise, but several research teams are seriously considering social media as a source of psychological data. A common goal of this work is to discover faster or cheaper ways to measure important but elusive variables, like personality, health, and happiness. At the World Well-Being Project, we focus on turning the language from social media into useful new measures.

For example, in a study published last year in PLoS ONE, we searched for traces of age, gender, and personality in a massive amount of social media language: 20 million status updates from 75,000 Facebook users. We found that users’ personality traits could be accurately predicted using only the words in their Facebook status updates. This is consistent with several recent studies [1-6] that suggest that statistical algorithms are surprisingly good at profiling our personalities, especially when they are fed psychologically-rich information like the structure of our Facebook social network or our Facebook likes.

Does this mean that algorithms will replace personality questionnaires?

Not quite. In almost every study of this kind, researchers judge the accuracy of their algorithm by comparing it to how people answer personality questionnaires, not their actual behaviors or other psychological measures. This is a good start, because these algorithms should agree with the way people respond to questionnaires, at a minimum. However, solely relying on this kind of validation leaves some important questions unanswered:

Do the algorithms only predict how we respond to questionnaires, or do they agree with other ways of measuring personality, too? Most of us are familiar with what psychologists call self-report measures, which ask respondents to indicate how well several statements describe them (e.g., “I talk to a lot of different people at parties” or “I get irritated easily”). While self-report measures are very popular, they are just one of many assessment methods. Well-acquainted friends, for example, are surprisingly good judges of our personalities—sometimes better than ourselves [7] — because our friends aren’t affected by the same kind of biases that we have when assessing ourselves. Friend ratings also give us a second way to evaluate an algorithm’s predictions.

Do the algorithms predict personality-relevant behaviors?  Psychologists measure personality traits because they are excellent predictors of other important outcomes [8], like relationship styles, health behaviors, career preferences, etc. So, an algorithm’s personality assessment should also help us make better predictions about other things, too. For example, if an algorithm predicts that a user is high in extraversion, we should expect that this person will also exhibit other features of extraversion, like having a larger circle of friends.

Do the algorithms make consistent predictions over time? Our personalities traits tend to be very stable from year to year [9],  but our language seems to evolve constantly, especially in social media. Think about your own Facebook statuses: are you talking about the same things now as you were six months ago? How about a year ago? A concern with language-based algorithms is that they might be overly sensitive to small, unimportant changes in our word usage. If an algorithmic assessment is valid, it should give consistent results for the same person over time.

In a recent article in the Journal of Personality and Social Psychology, we dove into these nitty-gritty details to see if language-based algorithms can really work as well as traditional assessments. We first developed a prediction algorithm using the status messages from 66,000 Facebook users. We then applied the algorithm to messages from 5,000 separate users and generated predictions of Big Five personality traits for every user. Lastly, we analyzed several measurement properties of the algorithms to answer the questions above. Here’s what we found:

Language-based algorithms are as accurate as friends

For each trait, we quantified accuracy as the correlation between the algorithms’ predictions and questionnaire measures of each trait. Accuracy was quite good: correlations ranged between .35-.46 (a correlation of 0 means predictions are completely random ; correlation of 1 means perfect prediction). To put those numbers into context, we compared the algorithms to ratings made by the users’ actual friends, people who presumably know them well beyond their Facebook messages. Friends’ accuracy ranged between .25-.39 (friend ratings typically range between .3-.6). Algorithms are roughly as accurate as a typical friend; when rating traits of Openness and Agreeableness, the algorithms were actually more accurate than a friend.

Comparings the accuracy of different personality assessments: language-based algorithms, ratings from friends, and a combination of both methods.
Comparing the accuracy of different personality assessments: language-based algorithms, ratings from friends, and a combination of both methods.

For most traits, we found that combining the algorithm with friend ratings yielded even more accurate predictions. Practically, this means that similar algorithms might be a useful way for researchers to boost the accuracy of personality assessments. But, more interestingly, it suggests that algorithms and friends see somewhat different sides of us, and each provides unique information.

Language-based algorithms predict personality-relevant outcomes

We also compared the predictions to several outcomes with well-known connections to personality. Compared to introverts, for example, extroverts typically report greater satisfaction with life [10], have larger social circles [11], and are more likely to share personal information (or “self-disclose” [12]). If an algorithm can truly assess extraversion, it should also predict these extraverted behaviors, at least as well as traditional questionnaires.

Predicting characteristics of extraversion using two kind of extraversion assessments: a language-based algorithm and a traditional questionnaire

For some outcomes, like number of Facebook friends, algorithmic predictions of personality were slightly better than questionnaires; in other cases, as with satisfaction with life, the questionnaires were slightly better. Overall, both methods had very similar patterns relationships with relevant outcomes, suggesting that the algorithms are capturing the right information. We only show  a tiny subset of the results here; see the article for full details (Figure 5 and Appendix C).

Language-based algorithms make consistent predictions over time

Finally, we examined the stability of the algorithms over time. Because language in social media can evolve so rapidly, we wondered if a prediction for a given person would be the same at later times. To test this, we let the algorithms generate multiple personality predictions for the same people, but we used language from different time points over a two-year period. For example, we used a person’s messages from January to June to make one set of predictions, then we made a second set of predictions for the same people using messages from July to December, and so on. We then compared the similarity of predictions across different times.

test_retest
Personality predictions using different language samples from the same people gave similar results over time.

The algorithms were surprisingly consistent over time, roughly on par with traditional questionnaires (average six-month test-retest correlations were r = .70; see Table 5 and Appendix E for full details). How can a language-based algorithm make consistent predictions if our language is always changing? While buzzwords and memes may come and go, our use of the most revealing language—pronouns, cursing, and emotion words—is very stable.

Conclusion: Language-based algorithms are valid personality assessments

We found that our language-based algorithms perform much like traditional personality assessments: they agree with questionnaires and friend ratings, they predict the right kind of outcomes, and they are consistent over time. As researchers mine social media and propose new metrics, establishing these basic psychometric properties will be increasingly important. More validation work is needed to understand how to extend these techniques to other areas, like well-being and mental health. Still, this is a promising sign that status messages and tweets aren’t just noise. With the right techniques, they can yield valid personality assessments and, ultimately, insight into who we are.

References

  1. Golbeck, J., Robles, C., & Turner, K. (2011, May). Predicting personality with social media. In CHI’11 Extended Abstracts on Human Factors in Computing Systems, pp. 253-262.

  2. Sumner, C., Byers, A., Boochever, R., & Park, G. (2012, December). Predicting dark triad personality traits from Twitter usage and a linguistic analysis of tweets. In 11th International Conference on Machine Learning and Applications, pp. 386-393.

  3. Iacobelli, F., Gill, A. J., Nowson, S., & Oberlander, J. (2011). Large scale personality classification of bloggers. In Affective Computing and Intelligent Interaction, pp. 568-577. Springer Berlin Heidelberg.

  4. Argamon, S., Koppel, M., Pennebaker, J. W., & Schler, J. (2009). Automatically profiling the author of an anonymous text. Communications of the ACM, 52(2), pp. 119-123.

  5. Kosinski, M., Stillwell, D., & Graepel, T. (2013). Private traits and attributes are predictable from digital records of human behavior. Proceedings of the National Academy of Sciences, 110(15), pp. 5802-5805.

  6. Youyou, W., Kosinski, M., & Stillwell, D. (2015). Computer-based personality judgments are more accurate than those made by humans. In Proceedings of the National Academy of Sciences, PNAS, 201418680.

  7. Vazire, S., & Carlson, E. N. (2011). Others sometimes know us better than we know ourselves. Current Directions in Psychological Science, 20, pp. 104-108.

  8. Roberts, B. W., Kuncel, N. R., Shiner, R., Caspi, A., & Goldberg, L. R. (2007). The power of personality: The comparative validity of personality traits, socioeconomic status, and cognitive ability for predicting important life outcomes. Perspectives on Psychological Science, 2(4), pp. 313-345.

  9. Roberts, B. W., & DelVecchio, W. F. (2000). The rank-order consistency of personality traits from childhood to old age: a quantitative review of longitudinal studies. Psychological Bulletin, 126(1), pp. 3-25.

  10. Diener, E., Oishi, S., & Lucas, R. E. (2003). Personality, culture, and subjective well-being: Emotional and cognitive evaluations of life. Annual Review of Psychology, 54, pp. 403-425.

  11. Lucas, R. E., Diener, E., Grob, A., Suh, E. M., & Shao, L. (2000). Cross-cultural evidence for the fundamental features of extraversion. Journal of Personality and Social Psychology, 79, pp. 452-468.

  12. Seidman, G. (2013). Self-presentation and belonging on Facebook: How personality influences social media use and motivations. Personality and Individual Differences, 54, pp. 402-407.

Share this on ...Share on Facebook0Tweet about this on Twitter0Share on Google+0Print this pageEmail this to someone

Leave a Reply

Your email address will not be published. Required fields are marked *