Field of Science

Should we trust experiments on the Web?

When I first started doing Web-based experiments, a number of people in my own lab were skeptical as to whether I would get anything valuable out of them. Part of this was due to worries about method (How do you know the participants are paying attention? How do you know they are telling the truth?), but I think part of it was also a suspicion of the Internet in general, which, as we all know, is full of an awful lot of crap.

For this reason, I expected some difficulties getting my Web-based studies published. However, the first of these studies was accepted without much drama, and what concerns the reviewers did raise had nothing to do with the Web (granted that only one of the experiments in that paper was run online). Similarly, while the second study (run in collaboration with Tal Makovski) has run into some significant hurdles in getting published, none of them involved the fact that the experiments were all run online.

Until now. After major revisions and some new experiments, we submitted the paper to a new journal where we thought it would be well-received. Unfortunately, it was not, and many of the concerns involved the Web. Two of the reviewers clearly articulated that they just don't trust Web-based experiments. One went so far as to say that Web-based experiments should never be run unless there is absolutely no way to do the experiment in the lab.

(I would use direct quotes, but the reviewers certainly did not expect their comments to show up on a blog, anonymously or not. So you will have to take my word for it.)

Obviously, I trust Web-based experiments. I have written enough posts about why I think concerns are misguided, so I won't rehash that here. I am more interested in why exactly people have trouble with Web-based experiments as opposed to other methodologies.

Is it because the Web-based method is relatively new? Is it because the Internet is full of porn? Or is it simply the case that for any given method, there are a certain number of people who just don't trust it?

I have been doing street-corner surveying lately (a well-established method), and I can tell you that although it ultimately gives decent results, some very odd things happen along the way. But I suppose if, as a reviewer, I tried to reject a paper because I "just don't trust surveys," the action editor would override me.

6 comments:

Jennifer Talarico said...

IMHO, There are a couple minor, but legitimate, concerns about internet studies. First, difficulty in preventing the same person from participating more than once (undermining the independence of your observations). Second, difficulty verifying self-reported demographic characteristics of your sample (threatening interpretations of generalizability).

Anonymous said...

An additional, off-the-top-of-my-head suggestion, based on a generalization of what jen said: if you have test-takers interacting even at a minimal level with test subjects, it is easier to catch problems with your sampling frame. I think that the biggest weakness comes from vulnerability to intentional sabotage, rather than a lack of independence or sampling bias. If the same undergraduate signed up for 67 time slots to fill out a survey, someone would presumably notice. Such is not the case from someone who wants to scramble their user-agent string and submit bogus test results from a Perl script or post links to your survey on the Something Awful forums or 4chan along with helpful suggestions as to how to feed it bad data.

A valid question is, "How big is the risk of getting bad data because of this sort of thing?" The answer is probably, "Not very," along with the addenda, "You could probably sniff out behavior like this by looking at your dataset anyway," and, "Traditional data-gathering through pen-and-paper surveys is vulnerable to the same types of attacks," so I would also credit a lack of trust in data submitted "anonymously" over the Web as opposed to "anonymously" in person.

This could even be a fruitful line of questioning: are there varying degrees of anonymity, depending on the medium in which one is acting "anonymously?" Does the medium through which others act influence the trust which we place in them when the act under the aegis of anonymity? There is also the question of whether there are different standards of behavior online and off-line and whether that would affect whether someone would mistreat an experiment online but not in the "real world."

Will said...

Jen and anonymous raise legitimate concerns. There are technical methods of overcoming or at least minimizing those problems, for example, by logging the IP address of the computer submitting the survey. However the impression I get is that at least some of the reviewers mentioned are uncomfortable with "the web" in general, and haven't even taken a serious look at the methods employed. In other words, some have a fear and mistrust of technology, which makes me wonder if they are qualified to be academic reviewers. If you, as you suggested, were to reject an article that included a survey taken using clipboards because "you don't trust clipboards", I suspect you wouldn't last very long.

Anonymous said...

Hi,
all those concerns voiced in this discussion have been discussed in the published literature and many have been answered empirically. I urge you to read publications by Ulf Reips (http://tinyurl.com/98qay), Michael Birnbaum and John Krantz.
Regards
Eric

josh said...

Hi Eric,

I think if you look back through previous posts, the empirical research on web-based experiments has been cited. But thank you for bringing it up again.

Yes, the research that has been published so far suggests that web-based research is *at least* as reliable as lab-based research.

Unknown said...

Hi,

I've been using cookies to block multiple submissions and I also take a long hard look at the data.

Contact info for survey incentives helps too and 4 years of doing this tells me it does work.

I also verify data against other known sources (such as forrester research against visitors to the same website to detect any significant bias).

Hey - if I wasn't doing these surveys because I was worried then I'd be lacking some serious insight. The biggest source of skew in surveys that I've seen has been in designing questions badly (not using the internet to collect responses). I always test a small sample first to see whether I've missed a trick in the construction and 90%+ of the work goes into design, not execution.

Craig.