A number of people have been blogging about a recent, still unpublished study suggesting that "a significant proportion of native English speakers are unable to understand some basic sentences." Language Log has a detailed explanation of the methods, but in essence participants were asked to match sentences to pictures. A good fraction made large numbers of mistakes, particularly those who had been high-school drop-outs.
What's going on here? To an extent, this shouldn't be that surprising. We all know there are people who regularly mangle language. But, as Mark Liberman at Language Log points out, at least some of these data are no doubt ascribable to the "paper airplane effect":
It's probably relevant that the researchers involved in this study are linguists. There are some linguists who run fabulous experiments, but as a general rule, linguists don't have much training in doing experiments or much familiarity with what data looks like. So it's not surprising that the researchers in question -- and the people to whom they presented the data -- weren't aware of the paper airplane effect.
(I should say that psychology is by no means immune to this problem. Whenever a new method is adopted, it takes a while before there's a critical mass of people who really understand it, and in the meantime a lot of papers with spurious conclusions get written. I'm thinking of fMRI here.)
What's going on here? To an extent, this shouldn't be that surprising. We all know there are people who regularly mangle language. But, as Mark Liberman at Language Log points out, at least some of these data are no doubt ascribable to the "paper airplane effect":
At one point we thought we had discovered that a certain fraction of the population is surprisingly deaf to certain fairly easy speech-perception distinctions; the effect, noted in a population of high-school-student subjects, was replicable; but observing one group of subjects more closely, we observed that a similar fraction spent the experiment surreptitiously launching paper airplanes and spitballs at one another.It's worth remembering that, while many participants in an experiment take it seriously and are happy to help out the researcher, some are just there for the money they get paid. Since we're required to pay people whether they pay attention to the experiment or not, they really don't have any incentive to try hard. Does it surprise anyone that high-school drop-outs are particularly likely to be bad at/uninterested in taking tests?
It's probably relevant that the researchers involved in this study are linguists. There are some linguists who run fabulous experiments, but as a general rule, linguists don't have much training in doing experiments or much familiarity with what data looks like. So it's not surprising that the researchers in question -- and the people to whom they presented the data -- weren't aware of the paper airplane effect.
(I should say that psychology is by no means immune to this problem. Whenever a new method is adopted, it takes a while before there's a critical mass of people who really understand it, and in the meantime a lot of papers with spurious conclusions get written. I'm thinking of fMRI here.)
11 comments:
And from a rival blog: "There are some psychologists who write marvelous papers about language, but as a rule, psychologists don't know the first thing about language."
Seriously though, I don't know anything about the author's graduate training, but she seems to have been involved in quite a lot of experimental work. Perhaps it is be unfair to attribute her failure to control for the paper airplane effect (how do people usually control for that anyway?) to the department that writes her paycheck.
Doing a lot of experiments isn't the same as having a lot of training doing experiments. Also, one tends to rely heavily on one's community to make suggestions, catch errors in design, etc. So if you community isn't that familiar with experiments, they can't help much.
There are dozens of well-developed methods to correct for the paper airplane effect -- a product of painful learning over the last 100 years of psychological research. It's a bit like asking a physicist "how can you possibly account for measurement error?" It's one of the first things you learn to do.
"It's probably relevant that the researchers involved in this study are linguists."
That is silly. Looks like the researchers are psycholinguists, which is a highly experimental field within cognitive psychology. Someone being a "linguist" doesn't mean their work only involves compiling word lists and dictionaries...
J -- as a psycholinguist, I flatter myself in knowing something about psycholinguistics. I'm in a psychology department and collaborate with several linugists. Just sayin'.
Dabrowska is a "professor of cognitive linguistics" in the Department of Humanities at her institution, vice president of the UK cognitive linguistics association, and editor of "Cognitive Linguistics." So that means she's probably trained as a linguist.
Yes, some linguists can and do experiments -- and some very good experiments. And some psychologist write linguistic theory. But that's not, for the most part, how we're trained.
Anonymous & J -- comment for both of you:
Basically, this experiment had a pretty glaring and obvious flaw. My hypothesis was that the authors were good linguists who are admirably testing some of their theories with experiments, but due to inexperience interpreted it incorrectly. Your hypothesis seems to be that this is a well-trained team of experimenters who don't know basic experimental design.
I like my hypothesis, but it's an empirical question.
Actually, Liberman does not point out that "some of these data are no doubt ascribable to the 'paper airplane effect'".
Rather, what he writes is that "Overall, I found the experiments fairly convincing. I do worry a bit about what some colleagues and I used to call 'the paper airplane effect'"
He writes this, followed by a colon, directly before the portion you have cited. I'm sure you didn't mean to mischaracterize his remarks, but the perspective of the post (and comments) seems to that be of persons who, having heard of an improbable result, experience a certain dissonance, and attempt to eliminate it by explaining the result in terms of methodological error. Fair enough, but might not one wish to review the actual methodology involved, before allowing one's self to go so far as remove doubt from the picture?
The question is far from moot, I found, as I attempted to find a description of the experiment (not available), and in the course of this learnt that Ewa Dąbrowska happens to teach research methodology to graduate students at Sheffield.
Knowledge of this — evidence of very rudimentary investigation of the topic of discussion — had so far failed to turn up in your discussions. Thus it was that I found myself somewhat annoyed, most especially because I am sure you are all perfectly aware that the methodological and statistical norms of psychology, as a research discipline, remain deeply, deeply unlauded in many quarters.
I do not mean to defend the paper, nor the research practices of linguists, as I have no knowledge as to the stringency of either. But, come on. Data before conclusions, eh?
Anonymous -- to the best of my knowledge, the paper isn't available, so I can't evaluate it. However, LanguageLog posted a fairly detailed description of the methods, which I assume you know since you seem to have read that post. Nothing about that description suggests that any attempt to deal with the paper airplane effect. You're right that it could have been done and simply not reported (or rereported by LL), but that seems unlikely for 2 reasons:
1) Normally the paper airplane effect would be the first thing people would worry about with data like this, so you'd expect it would be saliently presented and LL would know about it.
2) If such a large percentage of people really didn't understand passives, you'd think someone would have noticed by now. Whereas those are exactly the sorts of errors you'd expect from participants who weren't paying attention. Extraordinary claims and all that.
I'm curious what "quarters" you are referring to. There are definitely some improvements to methodology that could be desired (I'd like to see more self-replication prior to publication, for instance). That said, most of the critiques of psychology research methods I've read have come from people who didn't appear to be familiar with research from the last 75 years (they were critiquing Freud, mostly, which is a bit like critiquing the physics of Aristotle). But seriously, I'm always interested in better research methods, so if you've got something post, post it here.
Here's a link to the paper.
The authors don't address the paper airplane effect directly, but here's something they do say:
"Do these differences reflect differences in underlying linguistic knowledge, or could they be attributed to linguistically irrelevant factors such as willingness to cooperate with the experimenter, amount of experience with formal testing or ability to perform the experimental task? In our view, appeals to performance factors as an explanation of the results are highly unsatisfactory. The interviews were conducted at the place where the participants worked and were as informal as possible. Participants had as much time as was necessary to answer the questions (although most completed the task in less than 5 min), and were all extremely co-operative. Second, issues surrounding ‘test-wiseness’ should be evident across all constructions; yet the LAA group performed at ceiling on the control condition (i.e., the actives). Thirdly, as argued earlier, a picture selection task places minimal demands on linguistically irrelevant performance factors, and in fact children as young as two generally succeed on this task when presented with simple sentences that they can understand. Of course, one can always argue that the participants’ difficulties were due to non-linguistic task demands and that they might have succeeded if tested using a different method; however, such an argument is vacuous until it is demonstrated that participants who fail on picture selection succeed on some other task tapping knowledge of the same constructions. We conclude, therefore, that the participants’ relatively poor performance on passives and quantifiers reveals incomplete knowledge of these constructions."
That's not a knock-down argument. They assert the low-attainment participants were as keen on the experiment as the high-attainment participants. I assert that they weren't, but I'd be convinced by data.
They argue that people who are bad test-takers should do equally worse on all conditions. That only follows if you think that "bad test-taking" is an additive effect that does not interact with how hard the question is. I naively think that bad test takers do even worse on hard problems than easy problems, but I'd be convinced by data.
I'm actually on board with point 3. Using the picture-selection task was a good idea.
(Re 12:40)
I wasn't trying to be sarcastic -- having little experimental experience myself, I was truly interested in how people normally deal with this problem. If indeed, as you say, this effect is stronger on harder tasks (which makes a lot of sense), a trivial catch trial such as "have you ever had a fatal heart attack" won't do -- you're going to have to find a catch trial that is "just as hard" as the experimental task.
What I found offensive in your original remark is the fact that you used the author's affiliation to discredit her work, or at least as an explanation of the (presumed) lesser quality of her work. Graduate students at top American universities often get a better training than students at third world countries, but I don't suppose I'll catch you saying, "well, that wasn't a very good paper, that's probably because the guy is from Zambia".
(Re 12:40)
I wasn't trying to be sarcastic -- having little experimental experience myself, I was truly interested in how people normally deal with this problem. If indeed, as you say, this effect is stronger on harder tasks (which makes a lot of sense), a trivial catch trial such as "have you ever had a fatal heart attack" won't do -- you're going to have to find a catch trial that is "just as hard" as the experimental task.
What I found offensive in your original remark is the fact that you used the author's affiliation to discredit her work, or at least as an explanation of the (presumed) lesser quality of her work. Graduate students at top American universities often get a better training than students at third world countries, but I don't suppose I'll catch you saying, "well, that wasn't a very good paper, that's probably because the guy is from Zambia".
I'm not trying to discredit anyone's work based on their affiliation, and I'm sorry it came across that way. The experimental method was flawed, and the conclusions weren't warranted -- and that's quite independent of affiliation.
My interest in affiliation was simply trying to understand how it was that the authors and reviewers didn't notice the problem.
It's true that a simple "have you ever died of a heart attack" manipulation won't work here. The problem with the passive sentences is they're easy to get wrong if you aren't paying attention, since people by default assume a sentence is active (which seems to be what happened in this study; I've thought less about the quantifiers, but it looks like something similar is going on). So you'd probably want some non-linguistic task that you'd get wrong if you aren't paying careful attention. To support their thesis, they'd want no correlation between performance on that task and the linguistic task.
Post a Comment