Web Experiment Tutorial: Chapter 10, Recruiting Participants

Several years ago, I wrote a tutorial for my previous lab on how to create Web-based experiments in Flash. This is the final chapter in the original tutorial.

10. Recruiting participants online

1.     Overview

So now you have an experiment implemented on the Web. All you need are participants. Where do you get them?

If you need only very small numbers of subject (50-100), this part is easy. If you want larger numbers of subjects, or if you want to run several experiments under the same URL (so as to prevent the same subject from participating in multiple versions of the experiment), this may be the most challenging part of Web-based experiments.

There are several methods you can use. I recommend using all of them. Each will be discussed below in turn, but briefly: you can list the experiments on experiment portal pages, you can recruit from within your own social network, you can buy ads, you can promote the experiments in online forums, you can blog, you can swap links with other researchers, and you can get media attention.

Media attention, if you can get it, is far more valuable than all those other methods combined.

2.     Experiment portal pages

There are several web sites that list online experiments. By far the one that has provided the most subjects to vacognition is:

http://psych.hanover.edu/research/exponnet.html

The second-most useful is:

http://genpsylab-wexlist.unizh.ch/

Others, much less useful, include:

http://www.onlinepsychresearch.co.uk/

http://www.language-experiments.org/

Another place you can list is:

http://www.psychology.org/

In the first 3 weeks of May, 2007, vacognition (my previous site) received 251 hits from psych.hanover, 67 from genpsylab-wexlist, 15 from onlinepsychresearch, and only 2 from language-experiments.

Here are some other lists I have not used, which may or may not be useful:

http://www.socialpsychology.org/expts.htm

http://www.w-lab.de/lab-united/submit.php

3.     Your social network

Your own friends and family are the most likely to be convinced to do your experiments. Some of them may pass along the URL to their friends and family. Every time I have sent out requests to my F&F, I get about 40-50 participants in various studies.

You can also use Web-based social networking. For instance, I have an account on Facebook.com. My page lists vacognition. A friend of mine created a Facebook “group” called “Harvard Studied My Brain.” We invited all our friends to join (about 200), and anybody on Facebook could in theory join if they found the group in a search. 35 people did join, and many more have clicked on the link.

To make the group more enticing, I created a “certificate of membership,” which members can download. Generally, it is good to think about why anybody would want to participate. What can you do to make it more fun?

Other social networking sites include Stumbleupon.com, Reddit.com and Digg.com. “vacognition” has accounts for all of those. Every time a new webpage mentions your website, it is a good idea to “vote” for that website on Stumbleupon, Reddit and Digg. This increases the likelihood people will surf to that page, and then to your page. However, these services only have an effect if a fairly large number of people vote for the site, and the traffic may or may not be high-quality. At one point, a number of people voted for vacognition on StumbleUpon. In the space of an hour, we suddenly got 150 visitors. However, most did not participate in any experiments, and the traffic died down within 90 minutes. This is likely due to the fact that users of StumbleUpon are randomly sent to the website. In contrast, users of Digg or Reddit know what sort of website they are going to and are more likely to actually be interested. Visitors we have gotten through Digg have been highly likely to participate in an experiment.

You can also add a link to your website as part of your email signature. Ask your labmates to do so as well. Ask your friends to link to your website from their websites.

4.     Purchasing ads

You can also purchase ads. One obvious place to put ads is Google. I have never tried this.

I did, however, buy adds on Facebook. For $5/day, Facebook promised to display my add to at least 10,000 people on the Oberlin network. For another $5/day, it was displayed to another 10,000 people in the Harvard network.

I bought $20 worth of ads as an experiment. Vacognition got about 80 hits. That’s 4/$1. This is not very impressive, but it may be worth it. Also, my ads may not have been very good. (Keep in mind that a hit to the website does not mean that the person completed or even started an experiment!)

5.     Online forums

Another way to recruit participants is to mention the website or a particular experiment in an online forum. Here, it is particularly important to make the post relevant to the forum discussion. Otherwise, you are spamming and may (not unfairly) receive hate mail.

There are many psychology or science forums. It may be perfectly fair to write a post called “Please help me finish this experiment.” Another option is to write about the topic you are studying (“Visual short-term memory is very limited. We are trying to find out exactly hoe limited. Please do this experiment.”). You can also be very oblique about it. Post something interesting about your area of research, and just mention your website (“It turns out 1/100 people have prosopagnosia from a young age, not as a result of stroke. This is something we’ve found through our online experiments at www.faceblind.org. In fact, blah blah blah.”)

You can also pick targeted forums. If you are studying reading, you can post on dyslexia or reading education forums. (“My colleagues and I are trying to better understand reading. The results may eventually help us better understand how to teach reading to children. We need volunteers for our short, 3-4 minute experiments. I thought that participants in this forum might be particularly interested…”)

Because I use Flash for my experiments, I have also posted on forums dealing with Flash programming (“You may be interested in this other application of Flash technology…”). Also, sometimes I have a question about Flash, and I post the question, with a link.

There are also website creation forums where you are encouraged to showcase your website.

The best success I have had with this method is when my experiment has been set up as a type of quiz. My visual short-term memory experiment gives people a score at the end. So I posted the experiment on several forums where people advertise quizzes (“How good is your short-term memory for what you see?”). Normally, a forum post generates only a small amount of traffic (0-10 hits), but these posts on quiz forums produced as many as 100 each.

Vacognition has accounts on many, many message forums.

6.     Swapping links

Reciprocal advertising is an easy-to-use but very limited strategy. Vacognition has a “links” page, where we link to other websites, mostly other Web-based experiments. In return, those websites link to ours.

This serves two purposes. First, visitors to those other sites may click on the link and come to our website. This is extremely rare.

Second, the more links there are to a website, the better its “page rank” – that is, the higher it appears in the list of search results. Swapping links improves your page rank, and thus you are easier to find through Google, etc. My data suggest that visitors that come via Google tend to be low-quality visitors – that is, they tend not to participate in experiments. However, a few do, and it doesn’t hurt.

Usually I arrange these link swaps by emailing the webmasters of websites that I think may be interested. Most do not respond, but some do.

7.     Media attention

By far the most effective method is media attention. Extremely successful online labs (like faceblind.org or the Moral Sense Test) get a lot of mainstream media attention, and they also get huge numbers of participants.

Media attention is hard to orchestrate. Ideally, your research will be so interesting that reporters will come to you. However, you can contact reporters yourself. The university can put out a press release. I got a fair amount of media attention after Georgetown wrote a press release about a paper I wrote.

In the end, though, you have to have work that is interesting to reporters and the public (see “The most important thing,” below).

8.     Blogging

Bloggers are more approachable members of the media. Bloggers of many shades and stripes may be interested in showcasing your experiments. And they are much more likely to respond to an email. Some blogs produce disappointing traffic. I guest-blogged for The New Scientist, whose blog gets a thousand hits a day, but I only got a few dozen hits out of it. However, Skepchic blogged (without my contacting her) about one of my experiments, and I got about 300 hits.

You can also write your own blogs. This will be of minimal help if you don’t attract a following, but even a blog with little following and only one new post every month or so can generate some traffic. The links from the blog to your website can also help your page rank (see “Swapping links”).

9.     Email list

We maintain an email list. On several parts of the website, visitors are encouraged to join a Google Groups email list, which now has over 100 members. The list is emailed when new experiments are posted or results have been posted, although I try to keep this to a minimum. If you overuse an email list, people tend not to read the messages and/or withdraw from the list.

Setting up a Google Groups email list is simple, and it can be set up so that anyone can join. Vacognition’s list can be found here:

http://groups.google.com/group/vacognition/web/home

10.  The most important thing

When recruiting participants, you should always keep in mind one question:

Why would anybody want to participate in this experiment?

Participants are expending resources (time, energy, and sometimes money) in order to participate. What is the product that you are selling them?

This is particularly important when trying to generate media attention – whether newspapers or bloggers. You may get your brother-in-law to blog about your experiment as a favor (mine did), but most bloggers aren’t going to write about something if they don’t find it interesting. Make it interesting. Testmybrain.org is a great example of a site that is fun, and not surprisingly it gets tons of traffic.

However, this issue is important even when using online experiment lists. Anyone who visits an online experiment list is already interested in doing online experiments. However, these lists post many experiments. No visitor is going to do all of them. So how do they choose which one(s) to do? Presumably, this is partly a function of how interesting the experiment looks. Compare:

“This experiment investigates the role of proactive interference in estimates of visual short-term memory capacity.”

with

“How much of what you see can you remember? Probably less than you thought. Take this 5 minute quiz to see how many visual objects you can remember. Typical scores are between 1 and 3 objects.”

Which experiment sounds more interesting? They are the same experiment.

You will want to craft your pitch to your audience. If you are posting on a forum for vision scientists, the 2^nd description above may come across as patronizing. However, if you are posting to a forum about online quizzes and games, the 1^st description will probably get you banned from the forum for spamming.

The design of the website itself also matters. An ugly, unprofessional-looking website will turn away visitors. Many participants are participating because they are interested in science. Make sure they learn something. Post results. Have pages that discuss the research topics. Make sure the debriefing is informative. Many participants find seeing their own results very motivating, so if possible, try to incorporate that into the experiment.

You can also experiment in your advertising. Try different pitches. See which work the best. Modify the website and see if the number of visitors who actually participate in experiments increases or decreases.

11. Where does GamesWithWords.org get it's traffic?

I currently use Google Analytics to track my web traffic. Here is what it shows for the top 10 referrers from Dec 1. 2009 through April 21 2010:

As you can see, the biggest chunk of traffic comes from people simply typing in the name of the site. Word of mouth seems to do a great deal. One thing to consider, however, is also the average time on site and the bounce rate. By these measures, the direct traffic is better than those who come via Google.

I should note that this traffic to dwarfed by weeks in which I get media attention. I can easily get several thousand visits per day when the site is mentioned in a prominent news source (which has not happened in the last few months, unfortunately). Notice also that while there are many other sources of traffic beyond the top 10 listed here, all the rest combined only contributed 1,385 visits.