Are Web-Based Experiments Reliable? The Data Say 'Yes.'

After a few months, I'm back to the task of getting the Video Test experiments published. As I mentioned last year, the paper had run aground partly due to reviewers' skepticism about Web-based experiments.

I sat down to improve the section of the paper that justifies using Web-based experiments. That required looking for other published experiments. I've done this haphazardly over the years, but this time I was much more systematic. I knew there were a fair number of published Web surveys, but I was surprised to discover there are many, many more published Web-based experiments than I thought. I also turned up a fairly large number of studies in which researchers directly compared Web-based and lab-based studies, typically finding the former to be as reliable as the latter.

In fact, I found so much I almost felt silly writing the justification. It seems strange to be justifying what has become essentially a well-established method. In fact, many researchers who use write up Web-based experiments don't even bother to do so.

The Data
Without further ado, here is a draft of that justification:

Internet-based experiments have become increasingly popular in recent years, with at least 21% of APA journals having published at least one paper relying on Internet-based methods (Skitka & Sargis, 2006). In the cognitive and perceptual research, domains in which the methodology has been particularly productive include face perception (inter alia, Bestelmeyer, Jones, DeBruine, Little & Welling, in press; Boothroyd, Jones, Burt, Cornwell, Little, Tiddeman & Perrett, 2005; Feinberg, DeBruine, Jones & Little, 2008; Feinberg, Jones, DeBruine, Moore, Smith, Cornwell, Tiddeman, Boothroyd & Perrett, 2005; Fessler & Navarrete, 2003; Little, Burriss, Jones, DeBruine & Caldwell, 2008; Little, Jones & Burriss, 2007; Little, Jones, Burt & Berrett, 2007; Little, Jones & DeBruine, 2008; Little, Jones, DeBruine & Feinberg, 2008; Smith, Jones DeBruine & Little, in press; Welling, Jones & DeBruine, 2008; Wilson & Daly, 2004) and reaction-time based studies of implicit social biases (inter alia, Bar-Anan, Nosek & Vianello, in press; Graham, Haidt & Nosek, in press; Lindner & Nosek, 2009; Nosek & Hansen, 2008; Ranganath & Nosek, 2008; Schwartz, Vartanian, Nosek & Brownell, 2006).

A number of researchers have directly compared the results of Internet-based and laboratory-based studies, finding that the former are highly reliable and the two methods produce similar results, both within and between subjects (Buchanan, T., & Smith, J. L., 2000; Gosling, Vazire, Srivastava & John, 2004; Linnman, Carlbring, Ahman, Anderesson & Andersson, 2004; McGraw, Tew, & Williams, 2000; Meyerson & Tryon, 2003; Ollesch, Heineken & Schulte, 2006; Srivastava, John, Gosling & Potter, 2003). Importantly for the present work, a recent study of VWM found converging results from Internet-based and Laboratory-based methods (Hartshorne, 2008).




Bar-Anan, Y., Nosek, B. A., & Vianello, M. (in press). The sorting paired features task: A measure of association strengths. Experimental Psychology.

Bestelmeyer, P. E. G., Jones, B. C., DeBruine, L. M., Little, A. C., & Welling, L. L. M. (in press). Face aftereffects demonstrate interdependent processing of expressions and the invariant characteristics of sex and race. Visual Cognition.

Boothroyd, L. G., Jones, B. C., Burt, D. M., Cornwell, R. E., Little, A. C., Tiddeman, B. P., & Perrett, D. I. (2005). Facial masculinity is related to perceived age but not perceived health. Evolution and Human Behavior, 26, 417-431.

Buchanan, T., & Smith, J. L. (1999). Using the Internet for psychological research: Personality testing on the World Wide Web. British Journal of Psychology, 90, 125-144.

Feinberg, D. R., DeBruine, L. M., Jones, B. C., & Little, A. C. (2008). Correlated preferences for men’s facial and vocal masculinity. Evolution and Human Behavior, 29, 233-241.

Feinberg, D. R., Jones, B. C., DeBruine, L. M., Moore, F. R., Smith, M. J. L., Cornwell, R. E., Tiddeman, B. P., Boothroyd, L. G., Perrett. (2005). The voice and face of woman: One ornament that signals quality? Evolution and Human Behavior, 26, 398-408.

Fessler, D. M. T., & Navarrete, C. D. (2003). Domain-specific variation in disgust sensitivity across the menstrual cycle. Evolution and Human Behavior, 24, 406 – 417.

Gosling, S. D., Vazire, S., Srivastava, S. & John, O. P. (2004). Should we trust web-based studies? A comparitive analysis of six preconceptions about Internet questionnaires. American Psychologist, 49, 93-104.

Graham, J., Haidt, J., & Nosek, B. A. (in press). Liberals and conservatives rely on different sets of moral foundations. Journal of Personality and Social Psychology.

Hartshorne, J. K. (2008). Visual working memory capacity and proactive interference. PloS ONE 3(7): e2716.
Lindner, N. M., & Nosek, B. A. (2009). Alienable speech: Ideological variations in the application of free-speech principles. Political Psychology, 30 67-92.

Linnman, C., Carlbring, P., Ahman, A., Andersson, H., & Andersson, G. (2004). The Stroop effect on the Internet. Computers in Human Behavior, 22, 448-455.

Little, A. C., Burriss, R. P., Jones, B. C., DeBruine, L. M., & Caldwell, C. C. (2008). Social influence in human face preference: men and women are influenced for long-term but not short-term attractiveness decisions. Evolution and Human Behavior, 29, 140-146.

Little, A. C., Jones, B. C., & Burriss, R. P. (2007). Preferences for masculinity in male bodies change across the menstrual cycle. Hormones and Behavior, 52, 633-639.

Little, A. C., Jones, B. C., Burt, D. M., & Perrett, D. I. (2007). Preferences for symmetry in faces change across the menstrual cycle. Biological Psychology, 76, 209-216.

Little, A. C., Jones, B. C., & DeBruine, L. M. (2008). Preferences for variation in masculinity in real male faces change across the menstrual cycle. Personality and Individual Differences, 45: 478-482.

Little, A. C., Jones, B. C., DeBruine, L. M., & Feinberg, D. R. (2008). Symmetry and sexual-dimorphism in human faces: Interrelated preferences suggest both signal quality. Behavioral Ecology, 19: 902-908.

Meyerson, P. & Tryon, W. W. (2003). Validating Internet research: a test of the psychometric equivalence of Internet and in-person samples. Behavior Research Methods, Instruments & Computers, 35, 614-620.

Nosek, B. A., & Hansen, J. J. (2008). The associations in our heads belong to us: Searching for attitudes and knowledge in implicit evaluation. Cognition and Emotion, 22, 553-594.

Heike Ollesch, Edgar Heineken, Frank P. Schulte (2006). Physical or virtual presence of the experimenter: Psychological online-experiments in different settings International Journal of Internet Science, 1 (1), 71-81

Ranganath, K. A., & Nosek, B. A. (2008). Implicit attitude generalization occurs immediately, explicit attitude generalization takes time. Psychological Science, 19, 249-254.

Schwartz, M. B., Vartanian, L. R., Nosek, B. A., & Brownell, K. D. (2006). The influence of one's own body weight on implicit and explicit anti-fat bias. Obesity, 14(3), 440-447

Smith, F. G., Jones, B. C., DeBruine, L. M., & Little, A. C. (in press). Interactions between masculinity-femininity and apparent health in face preferences. Behavioral Ecology.

Srivistava, S., John, O. P., Gosling, S. D., & Potter, J. (2003). Development of personality in early and middle adulthood: Set like plaster or persistent change? Journal of Personality and Social Psychology, 84, 1041-1053.

Welling, L. L. M., Jones, B. C., & DeBruine, L. M. (2008). Sex drive is positively associated with women’s preferences for sexual dimorphism in men’s and women’s faces. Personality and Individual Differences, 44(1): 161-170.

Wilson, M., & Daly, M. (2004). Do pretty women inspire men to discount the future? Proceedings of the Royal Society of London B: Biological Sciences, 271, S177-S179.