"Teens surprisingly OK with texting during sex," notes Slate's news aggregator. This seemed like a good lead for a piece I've wanted to write for a while: just how much we should trust claims that 10% of people agree to claim X. In many cases, we probably should put little faith in those numbers.
As usual, Stephen Colbert explains why. In his infamous roast of George W Bush, he notes
Numbers
For instance, many of my studies involve asking people's opinions about various sentences. In a recent one I ran on Amazon Mechanical Turk, I presented people with sentence fragments and asked them which pronoun they thought would likely be the next word in the sentence:
John went to the store with Sally. She/he...
In that case, it could be either pronoun, so I'm trying to get a sense of what people's biases are. However, I put in some filler trials just to make sure people were paying attention:
Billy went to the store with Alfred. She/he...
In this case, it's really, really unlikely the pronoun will be "she," since there aren't any female characters in the story. Even so, over 4% of the time participants still clicked on "she." This wasn't an issue of some of the participants simply being bad. I included 10 such sentences, and nobody only one person got more than 1 of these wrong. However, a lot of people did manage to miss 1 ... probably because they simply were sloppy, made a mistake, were momentarily not thinking ... or because they really thought the next word would be "she."
Those numbers are actually pretty good. In another, slightly harder experiment that I ran on my website, people didn't do so well. This one was shorter, so I included only 4 "catch trials" -- questions for which there was only one reasonable answer. Below is a pie chart of the participants, according to how many of these they got right:
You can see that over half got them all right, but around a quarter missed 1, and a significant sliver got no more than 50% correct. This could suggest many things: my questions weren't as well-framed as I thought, I had a lot of participants who weren't paying attention, some people were deliberately goofing off, etc.
Poll numbers
This isn't a problem specific to experiments. As we all learned in 2000, a certain number of people accidentally vote for the wrong candidate through some combination of not paying attention and poor ballot design.
So there is a difference between a survey finding that 10% of teens say that they think texting during sex is fine and 10% of teens actually thinking that texting during sex is fine. A good survey will incorporate methods of sussing out who is pulling the surveyor's leg (or not paying attention, or having a slip of the tongue, etc.).
Real Surveys
I didn't want to unnecessarily pick on this particular study, so I tried to hunt down the original source to see if they had done anything to protect against the "backwash" factor. Slate linked to a story on mashable.com. Mashable claimed that the research was done by the consumer electronics shopping and review site Retrevo, but only linked to Retrevo's main page, not any particular article. I did find a blog on Retrevo that frequently presents data from surveys, but nothing this year matched the survey in question (though this comes close). I found several other references to this study using Google, but all referenced Retrevo.
If anyone knows how to find the original study, I'd love to see it -- but if it doesn't exist, it wouldn't be the first apocryphal study. So it turns out that the backwash effect isn't the only thing to be careful of when reading survey results.
UPDATE
I have since heard from Revetro. See here.
As usual, Stephen Colbert explains why. In his infamous roast of George W Bush, he notes
Now I know there's some polls out there that say this man has a 32 percent approval rating ... Pay no attention to people who say the glass is half empty .. because 32 percent means it's 2/3 empty. There's still some liquid in that glass, is my point. But I wouldn't drink it. The last third is usually backwash.This was meant as a dig at those who still supported Bush, but there's a deeper point to be made: there's a certain percentage of people who, in a survey, will say "yes" to anything.
Numbers
For instance, many of my studies involve asking people's opinions about various sentences. In a recent one I ran on Amazon Mechanical Turk, I presented people with sentence fragments and asked them which pronoun they thought would likely be the next word in the sentence:
John went to the store with Sally. She/he...
In that case, it could be either pronoun, so I'm trying to get a sense of what people's biases are. However, I put in some filler trials just to make sure people were paying attention:
Billy went to the store with Alfred. She/he...
In this case, it's really, really unlikely the pronoun will be "she," since there aren't any female characters in the story. Even so, over 4% of the time participants still clicked on "she." This wasn't an issue of some of the participants simply being bad. I included 10 such sentences, and nobody only one person got more than 1 of these wrong. However, a lot of people did manage to miss 1 ... probably because they simply were sloppy, made a mistake, were momentarily not thinking ... or because they really thought the next word would be "she."
Those numbers are actually pretty good. In another, slightly harder experiment that I ran on my website, people didn't do so well. This one was shorter, so I included only 4 "catch trials" -- questions for which there was only one reasonable answer. Below is a pie chart of the participants, according to how many of these they got right:
Poll numbers
This isn't a problem specific to experiments. As we all learned in 2000, a certain number of people accidentally vote for the wrong candidate through some combination of not paying attention and poor ballot design.
So there is a difference between a survey finding that 10% of teens say that they think texting during sex is fine and 10% of teens actually thinking that texting during sex is fine. A good survey will incorporate methods of sussing out who is pulling the surveyor's leg (or not paying attention, or having a slip of the tongue, etc.).
Real Surveys
I didn't want to unnecessarily pick on this particular study, so I tried to hunt down the original source to see if they had done anything to protect against the "backwash" factor. Slate linked to a story on mashable.com. Mashable claimed that the research was done by the consumer electronics shopping and review site Retrevo, but only linked to Retrevo's main page, not any particular article. I did find a blog on Retrevo that frequently presents data from surveys, but nothing this year matched the survey in question (though this comes close). I found several other references to this study using Google, but all referenced Retrevo.
If anyone knows how to find the original study, I'd love to see it -- but if it doesn't exist, it wouldn't be the first apocryphal study. So it turns out that the backwash effect isn't the only thing to be careful of when reading survey results.
UPDATE
I have since heard from Revetro. See here.
4 comments:
Your graph is "webkit-fake-url://F73EEFFF-57E1-4DA8-9B4C-81694ECBDFE2/application.pdf" and doesn't show up.
I have met women named Billy - spelled that way, too.
Seems you're right. Sigh. It looked good on my computer when I originally posted it. The image should display now.
I too have met women named Billy. My neighbour included.
For that reason I don't typically use names like "Billy" in the experiment. It was the first thing that came to mind when writing an example for this post, though.
Post a Comment