New Experiment: That Kind of Person

I just got back reviews on one of the pronoun papers. Although the paper already had seven experiments, they want two more. The worst part about it is that they are right.

Luckily, the experiment they asked for can be done online. It takes about 5 minutes. Native English speakers preferred (though I look at all data).

That Kind of Person (takes about 5 minutes)

My target is to post the results for this and the seven previous experiments in 2 weeks ... if I get enough participants quickly. Thank you in advance to everyone who participates.

Revision, Revision, Revision

I have finally been going through the papers in the Frontiers Special Topic on publication and peer review in which my paper on replication came out. One of the arguments that appears in many of these papers (like this one)* -- and many discussions of the review process, is that when papers are published, they should be published along with the reviews.

My experience with the process -- which I admit is limited -- is that you submit a paper, reviewers raise concerns, and you only get published if you can revise the manuscript so as to address those concerns (which may include new analyses or even new experiments). At that stage, the reviews are a historical document, commenting on a paper that no longer exists. This may be useful to historians of science, but I don't understand how it helps the scientific process (other than, I suppose, transparency is a good thing).

So these proposals only make sense to me if it is assumed that papers are *not* typically revised in any meaningful way based on review. That is, reviews are more like book reviews: comments on a finished product. Of my own published work, three papers were accepted more-or-less as is (and frankly I think the papers would have benefited from more substantial feedback from the reviewers). So there, the reviews are at least referring to a manuscript very similar to the one that appeared in print (though they did ask me to clarify a few things in the text, which I did).

Other papers went through more substantial revision. One remained pretty similar in content, though we added a whole slew of confirmatory analyses that were requested by reviewers. The most recent paper actually changed substantially, and in many ways is a different -- and much better! -- paper than what we originally submitted. Of the three papers currently under review, two of them have new experiments based on reviewer comments, and the other one has an entirely new introduction and general discussion (the reviewers convinced me to re-think what I thought the paper was about). So the reviews would help you figure out which aspects of the paper we (the authors) thought of on our own and which are based on reviewer comments, but even then that's not quite right, since I usually get comments from a number of colleagues before I make the first submission. There are of course reviews from the second round, but that's often just from one or two of the original reviewers, and mostly focuses on whether we addressed their original concerns or not.

So that's my experience, but perhaps my experience is unusual. I've posted a poll (look top right). Let me know what your experience is. Since this may vary by field, feel free to include comments to this post, saying what field you are in.

---
*To be fair, this author is describing a process that has actually been implemented for a couple Economics journals, so apparently it works to (at least some) people's satisfaction.

Since one can't be snarky in a response to a review...

I'll do it here. I am currently revising a paper for resubmission. On the whole, the reviews are fairly reasonable, with the exception of one cranky comment from a reviewer who complains that our literature review is woefully incomplete. This incompleteness seems to be our failure to cite one particular study. The reviewer writes

It is possible that this work is flawed, but it really should be discussed.

It does seem to be a relevant study and we would have cited it, had we known about it. Why didn't we know about it? Because it has never been published. It hasn't even been presented at any of the normal psycholinguistics conferences (though it has appeared at some linguistics conferences). Short of emailing every researcher who might be conducting a study that might be relevant, I'm not sure what this reviewer was expecting of us.

I'd also love to know what the folks who are obsessed with only citing studies published in peer-reviewed journals would say. (It's possible that some of these conferences it has been presented at have pretty thorough review procedures; I wouldn't know.)

Ethics

I'm doing my periodic re-certification on research ethics. One of the questions on one of the quizzes is as follows:

TRUE/FALSE: A good alternative to the current peer review process would be web logs (BLOGS) where postings where [sic] papers would be posted and reviewed by those who have an interest in the work.

Apparently, the correct answer is "false". Presumably because we have much better technology for this kind of thing, rather than using a simple blog? 

The pace of review

One of my manuscripts will shortly enter its 7th month of being under review. Apparently one of the three reviewers keeps promising to send in a review but never does. Now the 4+ months a different manuscript languished under review seems speedy.

Ray Kurzweil is convinced that the pace of science is increasing exponentially and will continue to do so. I think he's neglected one rate-limiting step in the process.

And for my next trick, I'll make this effect disappear!

In this week's New Yorker, Jonah Lehrer shows once again just how hard it is to do good science journalism if you are not yourself a scientist.

His target is the strange phenomenon that many high profile papers are failing to replicate. This has been very much a cause celebre lately, and Lehrer follows a series of scientific papers on the topic as well as an excellent Atlantic article by David Freedman. At this point, many of the basic facts are well-known: anecdotally, many scientists report repeated failures to replicate published findings. The higher-profile the paper, the less likely it is to replicate, with around 50% of the highest-impact papers in medicine failing to replicate. As Lehrer points out, this isn't just scientists failing to replicate each other's work, but scientists failing to replicate their own work: a thread running through the article is the story of Jonathan Schooler, a professor at UC-Santa Barbara who has been unable to replicate his own seminal graduate student work on memory.

Lehrer's focus in this article is shrinking effects.

No, not this one.

Some experimental effects seem to shrink steadily over time:

In 2001, Michael Jennions, a biologist at the Australian National University, set out to analyze "temporal trends" across a wide range of subjects in ecology and evolutionary biology. He looked at hundreds of papers and forty-four meta-analyses (that is, statistical syntheses of related studies), and discovered a consistent decline effect over time, as many of the theories seemed to fade into irrelevance.

As described, that's weird. But there is a good explanation for such effects, and Lehrer brings it up. Some results are spurious. It's just one of those things. Unfortunately, spurious results are also likely to be exciting. Let's say I run a study looking for a relationship between fruit-eating habits and IQ. I look at the effects of 20 different fruits. By chance, one of them will likely show a significant -- but spurious -- effect. So let's say I find that eating an apple every day leads to a 5-point increase in IQ. That's really exciting because it's surprising -- and the fact that it's not true is integral to what makes it surprising. So I get it published in a top journal (top journals prefer surprising results).

Now, other people try replicating my finding. Many, many people. Most will fail to replicate, but some -- again by chance -- will replicate. It is extremely difficult to get a failure to replicate published, so only the replications get published. After time, the "genius apple hypothesis" becomes part of established dogma. Remember that anything that challenges established dogma is exciting and surprising and thus easier to publish. So now failures to replicate are surprising and exciting and get published. When you look at effect-sizes in published papers over time, you will see a gradual but steady decrease in the "effect" of apples -- from 5 points to 4 points down to 0.

Where I get off the Bus

So far so good, except here's Lehrer again:

While the publication bias almost certainly plays a role in the decline effect, it remains an incomplete  explanation. For one thing, it fails to account for the initial prevalence of positive results among studies that never even get submitted to journals. It also fails to explaint eh experience of people like Schooler, who have been unable to replicate their initial data despite their best efforts.

Huh? Lehrer seems to be suggesting that it is publication that makes a result spurious. But that can't be right. Rather, there are just lots of spurious results out there. It happens that journals preferentially publish spurious results, leading to biases in the published record, and eventually the decline effect.

Some years ago, I had a bad habit of getting excited about my labmate's results and trying to follow them up. Just like a journal, I was interested in the most exciting results. Not surprisingly, most of these failed to replicate. The result was that none of them got published. Again, this was just a factor of some results being spurious -- disproportionately, the best ones. (Surprisingly, this labmate is still a friend of mine; personally, I'd hate me.)

The Magic of Point O Five

Some readers at this point might be wondering: wait -- people do statistics on their data and only accept a results that is extremely unlikely to have happened by chance. The cut-off is usually 0.05 -- a 5% chance of having a false positive. And many studies that turn out later to have been wrong pass even stricter statistical tests. Notes Lehrer:

And yet Schooler has noticed that many of the data sets that end up declining seem statistically solid--that is, they contain enough data that any regression to the mean shouldn't be dramatic. '"These are the results that pass all the tests," he says. "The odds of them being random are typically quite remote, like one in a million. This means that the decline effect should almost never happen. But it happens all the time!"

So there's got to be something making these results look more unlikely than they really are. Lehrer suspects unconscious bias:

Theodore Sterling, in 1959 ... noticed that ninety-seven percent of all published psychological studies with statistically significant data found the effect they were looking for ... Sterling saw that if ninety-seven per cent of psychology studies were proving their hypotheses, either psychologists were extraordinarily lucky or they published only the outcomes of successful experiments

and again:

The problem seems to be one of subtle omissions and unconscious misperceptions, as researchers struggle to make sense of their results.

I expect that unconscious bias is a serious problem (I illustrate some reasons below), but this is pretty unsatisfactory, as he doesn't explain how unconscious bias would affect results, and the Schooler effect is a complete red herring. 

I wasn't around in 1959, so I can't speak to that time, but I suspect that the numbers are similar today ... but in fact Sterling was measuring the wrong thing. Nobody cares what our hypotheses were. They don't care what order the experiments were actually run in. They care about the truth, and they have very limited time to read papers (most papers are never read, only skimmed). Good scientific writing is clear and concise. The mantra is: Tell them what you're going to tell them. Tell them. And then tell them what you told them. No fishing excursions, no detours. When we write scientific papers, we're writing science, not history.

And this means we usually claim to have expected to find whatever it is that we found. It just makes for a more readable paper. So when a scientist reads the line, "We predicted X," we know that really means "We found X" -- what the author actually predicted is beside the point.

Messing with that Point O Five

So where do all the false positive come from, if they should be less than 5% of conducted studies? There seem to be a number of issues.

First, it should be pointed out that the purpose of statistical tests (and the magic .05 threshold for significance) is to make a prediction as to how likely it is that a particular result will replicate. A p-value of .05 means roughly that there is a 95% chance that the basic result will replicate (sort of; this is not technically true but is a good approximation for present purposes).

But statistics are estimates, not facts. They are based on a large number of idealizations (for instance, many require that measurement error is distributed normally

a normal distribution

meaning that the bulk of measurements are very close to the true measurement and a measurement is as likely to be larger than the true number as it is likely to be smaller. In fact, most data is heavily skewed, with measurements more likely to be too large than too smaller (or vice versa).

For instance, give someone an IQ test. IQ tests have some measurement error -- people will score higher or lower than their "true" score due to random factors such as guessing answers correctly (or incorrectly), being sleepy (or not), etc. But it's a lot harder to get an IQ score higher than your true score than lower, because getting a higher score requires a lot of good luck (unlikely) whereas there are all sorts of ways to get a low score (brain freeze, etc.). 

Most statistical tests make a number of assumptions (like normally distributed error) that are not true of actual data. That leads to incorrect estimates of how likely a particular result is to replicate. The truth is most scientists -- at the very least, most psychologists -- aren't experts in statistics, and so statistical tests are misapplied all the time.

I don't actually think that issues like the ones I just discussed lead to most of the difficulties (though I admit I have no data one way or another). I bring these issues up mainly to point out at that statistical tests are tools that are either used or misused according to the skill of the experimenter. And there are lots of nasty ways to misuse statistical tests. I discuss a few of them below:

Run enough experiments and...

Let's go back to my genius fruit experiment. I ask a group of people to eat an apple and then give them an IQ test. I compare their IQ scores with scores from a control group that didn't eat an apple. Now let's say in fact eating apples doesn't affect IQ scores. Assuming I do my statistics correctly and all the assumptions of the statistical tests are met, I should have only a 5% chance of finding a "significant" effect of apple-eating.

Now let's say I'm disappointed in my result. So I try the same experiment with kiwis. Again, I have only a 5% chance of getting a significant result for kiwis. So that's not very likely to happen either.

Next I try oranges....

Hopefully you see where this is going. If I try only one fruit, I have a 5% chance of getting a significant result. If I try 2 fruits, I have a 1 - .95*.95 = 9.8% chance of getting a significant result for at least one of the fruits. If I try 4 fruits, now I'm up to a 1 - .95*.95*.95*.95 = 18.5% chance that I'll "discover" that one of these fruits significantly affects IQ. By the time I've tried 14 fruits, I've got a better than 50% chance of an amazing discovery. But my p-value for that one experiment -- that is, my estimate that these results won't replicate -- is less than 5%, suggesting there is only a 5% chance the results were due to chance.

While there are ways of statistically correcting for this increased likelihood of false positives, my experience suggests that it's relatively rare for anyone to do so. And it's not always possible. Consider the fact that there may be 14 different labs all testing the genius fruit hypothesis (it's suddenly very fashionable for some reason). There's a better than 50% chance that one of these labs will get a significant result, even though from the perspective of an individual lab, they only ran one experiment.

Data peaking

Many researchers peak at their data. There are good reasons for doing this. One is curiosity (we do experiments because we really want to know the outcome). Another is to make sure all your equipment is working (don't want to waste time collecting useless data). Another reason -- and this is the problematic one -- is to see if you can stop collecting data.

Time is finite. Nobody wants to spend longer on an experiment than necessary. Let's say you have a study where you expect to need -- based on intuition and past experience -- around 20 subjects. You might check your data after you've run 12, just in case that's enough. What usually happens is that if the results are significant, you stop running the study and move on. If they aren't, you run more subjects. Now maybe after you've got 20 subjects, you check your data. If it's significant, you stop the study; if not, you run some more. And you keep on doing this until either you get a significant result or you give up.

It's a little harder to do back-of-the-envelop calculations on the importance of this effect, but it should be clear that this habit has the unfortunate result of increasing the relative likelihood of a false positive, since false positives lead you to declare victory and end the experiment, whereas false negatives are likely to be corrected (since you keep on collecting more subjects until the false negative is overcome). I read a nice paper on this issue that actually crunched the numbers a while back (for some reason I can't find it at the moment), and I remember the result was a pretty significant increase in the expected number of false positives.

Data massaging

The issues I've discussed so real problems but are pretty common and not generally regarded as ethical violations. Data massaging is at the borderline.

Any dataset can be analyzed in a number of ways. Once again, if people get the result they were expecting with the first analysis they run, they're generally going to declare victory and start writing the paper. If you don't get the results you expect, you try different analysis methods. There are different statistical tests that be used. There are different covariates that could be factored out. You can through out "bad" subjects or items. This is going to significantly increase the rate of false positives.

It should be pointed out that interrogating your statistical model is a good thing. Ideally, researchers should check to see if there are bad subjects or items, check whether there are covariates to be controlled for, check whether different analysis techniques give different results. But doing this affects the interpretation of your p-value (the estimate of how likely it is that your results will replicate), and most people don't know how to appropriately control for that. And some are frankly more concerned with getting the results they want than doing the statistics properly (there is where the "borderline" comes in).

Better estimates

The problem, at least from where I stand, is one of statistics. We want our statistical tests to tell us how likely it is that our results will replicate. We have statistical tests which, if used properly, will give us just such an estimate. However, there are lots and lots of ways to use them incorrectly.

So what should we do? One possibility is to train people to use statistics better. And there are occasional revisions in standard practice that do result in better use of statistics.

Another possibility is to lower the p-value that is considered significant. The choice of p=0.05 as a cutoff was, as Lehrer notes, arbitrary. Picking a smaller number would decrease the number of false positives. Unfortunately, it also decreases the number of real positives by a lot. People who don't math can skip this next section.

Let's assume we're running studies with a single dependent variable and one manipulation, and that we're going to test for significance with a t-test. Let's say the manipulation really should work -- that is, it really does have an effect on our dependent measure. Let's say that the effect size is large-ish (Cohen's d of .8, which is large by psychology standards) and that we run 50 subjects. The chance of actually finding a significant effect at the p=.05 level is 79%. For people who haven't done power analyses before, this might seem low, but actually an 80% chance of finding an effect is pretty good. Dropping our significant threshold to p=.01 drops the chance of finding the effect to 56%. To put this in perspective, if we ran 20 such studies, we'd find 16 significant effects at the p=.05 level but only 11 at the p=.01 level. (If you want to play around with these numbers yourself, try this free statistical power calculator.)

Now consider what happens if we're running studies where the manipulation shouldn't have an effect. If we run 20 such studies, 1 of them will nonetheless give us a false positive at the p=.05 level, whereas we probably won't get any at the p=.01 level. So we've eliminated one false positive, but at the cost of nearly 1/3 of our true positives.

No better prediction of replication than replication

Perhaps the easiest method is to just replicate studies before publishing them. The chances of getting the same spurious result twice in a row are vanishingly small. Many of the issues I outlined above -- other than data massaging -- won't increase your replication rate. Test 14 different fruits to see if any of them increase IQ scores, and you have over a 50% chance that one of them will spuriously do so. Test that same fruit again, and you've only got a 5% chance of repeating the effect. So replication decreases your false positive rate 20-fold. Similarly, data massaging may get you that coveted p.05, but the chances of the same massages producing the same result again are very, very low.

True positives aren't nearly so affected. Again, a typical power level is B=0.80 -- 80% of the time that an effect is really there, you'll be able to find it. So when you try to replication a true positive, you'll succeed 80% of the time. So replication decreases your true positives by only 20%.

So let's say the literature has a 30% false positive rate (which, based on current estimates, seems quite reasonable). Attempting to replicate every positive result prior to publication -- and note that it's extremely rare to publish a null result (no effect), so almost all published results are positive results -- should decrease the false positives 20-fold and the true positives by 20%, leaving us with a 2.6% false positive rate. That's a huge improvement.

So why not replicate more?

So why don't people replicate before publishing? If 30% of your own publishable results are false positives, and you eliminate them, you've just lost 30% of your potential publications. You've also lost 20% of your true positives as well, btw, which means overall you've decreased your productivity by 43%. And that's without counting the time it takes to run the replication. Yes, it's nice that you've eliminated your false positives, but you also may have eliminated your own career!

When scientists are ranked, they're largely ranked on (a) number of publications, (b) number of times a publication is cited, and (c) quality of journal that the publications are in. Notice that you can improve your score on all of these metrics by publishing more false positives. Taking the time to replicate decreases your number of publications and eliminates many of the most exciting and surprising results (decreasing both citations and quality of journal). Perversely, even if someone publishes a failure to replicate your false positive, that's a citation and another feather in your cap.

I'm not saying that people are cynically increasing their numbers of bogus results. Most of us got into science because we actually want to know the answers to stuff. We care about science. But there is limited time in the day, and all the methods of eliminating false positives take time. And we're always under incredible pressure to pick up the pace of research, not slow it down.

I'm not sure how to solve this problem, but any solution I can think of involves some way of tracking not just how often a researcher publishes or how many citations those publications get, but how often those publications are replicated. Without having a way of tracking which publications replicate and which don't, there is no way to reward meticulous researchers or hold sloppy researchers to account.

Also, I think a lot of people just don't believe that false positives are that big a problem. If you think that only 2-3% of published papers contain bogus results, there's not a lot of incentive to put in a lot of hard work learning better statistical techniques, replicating everything, etc. If you think the rate is closer to 100%, you'd question the meaning of your own existence. As long as we aren't keeping track of replication rates, nobody really knows for sure where we are on this continuum.

That's my conclusion. Here's Lehrer's:

The decline effect is troubling because it reminds us how difficult it is to prove anything. We like to pretend that our experiments define the truth for us. But that's often not the case. Just because an idea is true doesn't mean it can be proved. And just because an idea can be proved doesn't mean it's true. When the experiments are done, we still have to chose what to believe.

I say it again: huh?

Priest, Altars and Peer Review

David Dobbs at Neuron Culture is complaining about NASA and peer review:

A NASA spokesperson has dismissed a major critique of the Science arsenic bug paper based not on the criticism's merits, but on its venue -- it appeared in a blog rather than a peer-reviewed journal. Apparently ideas are valid (or not) based on their content, or even the reputation of the author, but on where they're published.

I'm not known for my strong endorsement of the fetishism of peer review, but even so I think Dobbs is being somewhat unfair. My reading of history is that scientists have been plugging the peer-review mantra because they're tired of having to respond to ignorant assholes who appear on Oprah spouting nonsense. I mean, yes, you can address wacko claims about vaccines causing autism or the lack of global warming on their merits (they have none), but it gets tiresome to repeat. In any case, relatively few members of the public can follow the actual arguments, so it becomes an issue of who you believe. And that's a hard game to win, since saying "so-and-so doesn't know what they're talking about" sounds elitist even when it's true, and "elitism" (read: "meritocracy") is for some reason unpopular.

Focusing on peer review as a mechanism for establishing authority is convenient, because the public (thinks it) understands the mechanisms. You're not saying, "Believe me because I am a wise scientist," but "Believe the documented record." And since Jenny McCarthy doesn't publish in peer-reviewed journals, you can (try to) exclude her and other nuttos from the conversation.

So I think there are good reasons for a NASA spokesman, when speaking with a reporter, to dismiss blogs. Taking a critique in a blog seriously in public is only going to open the floodgates. I mean, there are a *lot* of blogs out there. That doesn't mean that the scientists involved aren't taking the a series critique by a serious scientist seriously just because the criticism appeared in a blog. I hope that they are, and we don't want to read too much into NASA's official statement.

All that said, I'm not sure focusing on peer-reviewed science has been helping very much. I mean, McCarthy still gets booked on Oprah anyway.

You're Wrong

John Ioannidis has been getting a lot of press lately. He reached the cover of the last issue of The Atlantic Monthly. David Dobbs wrote about him here (and a few years ago, here). This is the doctor known for his claim that around half of medical studies are false -- that is about 80% of non-randomized trials and even 25% of randomized trials. These are not just dinky findings published in dinky journals; of 49 of the most highly-regarded papers published over a 13-year period, 34 of the 45 with that claimed to have found effective treatments had been retested, and 41% of those tests failed to replicated the original result.

Surprised?

Quoting the Atlantic Monthly:

Ioannidis initially thought the community might come out fighting. Instead, it seemed relieved, as if it had been guiltily waiting for someone to low the whistle...

Well, it's not surprising. The appropriate analog in psychology is the randomized trial, of which in medicine 25% turn out to be false according to this research (which hopefully isn't itself false). As Ioannidis has detailed, the system is set up to reward false positives. Journals -- particularly glamour mags like Science -- preferentially accept surprising results, and the best way to have a surprising result is to have one that is wrong. Incorrect results happen: "statistically significant" means "has only a 5% probability of happening by random chance." This means (in theory) that 5% of all experiments published in journals should reach the wrong conclusions. If journals are biased in favor of accepting exactly those 5%, then the proportion should be higher.

There are other factors at work. Some scientists are sloppier than others, and many of the ways in which one can be sloppy lead to significant and/or surprising results. For instance, 5% of experiments have false positives. There are labs that will run the same experiment 6 times with minor tweaks. There is a (1-.95^6) * 100 = 26.5% chance that one of those will have a significant result. The lab may then publish only that final experiments and not report the others. If sloppy results lead to high-impact publications, survival of the fittest dictates that sloppy labs will reap the accolades, get the grant money, tenure, etc.

Keep in mind that often many different labs are trying to do the same thing. For instance, in developmental psychology, one of the deep questions is what is innate? So many labs are testing younger and younger infants, trying to find evidence that these younger infants can do X, Y or Z. If 10 labs all run the same experiment, there's a (1-.95^10) * 100 = 40.1% chance of one of the labs finding a significant result.

Countervailing Forces

Thus, there are many incentives to publish incorrect data. Meanwhile, there are very few disincentives to doing so. If you publish something that turns out to replicate, it is very unlikely that anyone will publish a failure to replicate -- simply because it is very difficult to publish a failure to replicate. If someone does manage to publish such a paper, it will certainly be in a lower-profile journal (which is, incidentally, a disincentive to publishing such work to begin with).

Similarly, consider what happens when you run a study and get a surprising result. You could replicate it yourself to make sure you trust the result. That takes time, and there's a decent chance it won't replicate. If you do replicate it, you can't publish the replication (I tried to in a recent paper submission, and a reviewer insisted that I remove reference to the replication on account of it being "unnecessary"). If the replication works, you'll gain nothing. If it fails, you won't get to publish the paper. Either way, you'll have spent valuable time you could have spent working on a different study leading to a different paper. 

In short, there are good reasons to expect that 25% of studies -- particularly in the high-profile journals -- are un-replicable.

What to do?

Typically, solutions proposed involve changing attitudes. The Atlantic Monthly suggests:

We could solve much of the wrongness problem, Ioannidis says, if the world simply stopped expecting scientists to be right. That's because being wrong in science is fine, and even necessary ... But as long as careers remain contingent on producing a stream of research that's dressed up to seem more right than it is, scientists will keep delivering exactly that.

I've heard this idea expressed elsewhere. In the aftermath of Hausergate, a number of people suggested that a factor was the pressure-cooker that is the Harvard tenure process, and that Harvard needs to stop putting so much pressure on people to publish exciting results.

So the idea is that we should stop rewarding scientists for having interesting results, and instead reward the ones who have uninteresting results? Journals should publish only the most staid research, and universities should reward tenure not based on the number of highly-cited papers you have written, but based on how many papers you've written which have never been cited? I like that idea. I can run a boring study in a few hours and write it up in the afternoon: "Language Abilities in Cambridge Toddlers are Unaffected by Presence or Absence of Snow in Patagonia." That's boring and almost certainly true. And no one will ever cite it.

Seriously, though, public awareness campaigns telling people to be more responsible are great, and sometimes they even help, but I don't know how much can be done without changing the incentive structure itself.


Reputation

I don't have a solution, but I think Ioannidis again points us towards one. He found that papers continue to be cited long after they have been convincingly and publicly refuted. I was discussing this issue with a colleague some time back and mentioned a well-known memory paper that nobody can replicated. Multiple failures-to-replicate have been published. Yet I still see it cited all the time. The colleague said, "Wow! I wish you had told me earlier. We just had a student spend two years trying to follow up that paper, and the student just couldn't get the method to work."

Never mind that researchers rarely bother to replicate published work -- even if they did, we have no mechanism for tracking which papers have been successfully replicated and which papers can't be replicated.

Tenure is awarded partly on how often your work has been cited, and we have many nice, accessible databases that will tell you how often a paper has been cited. Journals are ranked by how often their papers are cited. What if we rewarded researchers and journals based on how well their papers hold up to replication? Maybe it would help, maybe it wouldn't, but without a mechanism for tracking this information, this is at best an intellectual enterprise.

Even if such a database wasn't ultimately useful in decreasing the number of wrong papers, at least we'd know which papers were wrong.

Tables, Charts & Figures

APA format (required for most journals I read/publish in) stipulates that figures and tables should not be included in the parts of the manuscript in which you actually talk about them, but rather they should all come at the end of the manuscript. I understand how this might be of use to the type-setter, but I find it a pain when actually trying to read a manuscript. I know I'm not the only one, because in some of the manuscripts I've submitted for review before I actually violated APA format and put the figures in-line, and the reviewers actually thanked me in their review and suggested that this should become journal policy. (The idea is that after acceptance, you resubmit with the figures and tables in APA format, but that during the review process, you put them in-line.)

With that in mind, I left my figures in situ in my last journal submission. The staff at the journal promptly returned the manuscript without review, saying that they couldn't/wouldn't review a paper that didn't follow APA guidelines on tables and figures.

Obviously I reformatted and resubmitted (the customer/journal is always right), but I put this out to the blogosphere: does anyone actually like having the figures at the end of the manuscript?

Honestly, Research Blogging, Get over yourself

A few years ago, science blog posts started decorating themselves with a simple green logo. This logo was meant to credential the blog post as being one about peer-reviewed research, and is supplied by Research Blogging. As ResearchBlogging.org explains:

ResearchBlogging.org is a system for identifying the best, most thoughtful blog posts about peer-reviewed research. Since many blogs combine serious posts with more personal or frivolous posts, our site offers away to find only the most carefully-crafted about cutting-edge research, often written by experts in their respective fields.

That's a good goal and one I support. If you read further down, you see that this primarily amounts to the following: if the post is about a peer-reviewed paper, it's admitted to the network. If it's not, it isn't. I guess the assumption is that the latter is not carefully-crafted or about cutting-edge research. And that's where I get off the bus.

Peer Review is Not Magic

One result of the culture wars is that scientists have needed a way of distinguishing real data from fantasy. If you look around the Internet, no doubt half or even more than half of what is written suggests there's no global warming, that vaccines cause autism, etc. Luckily, fanatics rarely publish in peer-reviewed journals, so once we restrict the debate to what is in peer-reviewed journals, pretty much all the evidence suggests global warming, no autism-vaccine link, etc. So pointing to peer-review is a useful rhetorical strategy.

That, at least, is what I assume has motivated all the stink about peer-review in recent years, and ResearchBlogging.org's methods. But it's out of place in the realm of science blogs. It's useful to think about what peer review is.

A reviewer for a paper reads the paper. The reviewer does not (usually) attempt to replicate the experiment. The reviewer does not have access to the data and can't check that the analyses were done correctly. At best, the reviewer evaluates the conclusions the authors draw, and maybe even criticizes  the experimental protocol or the statistical analyses used (assuming the reviewers understand statistics, which in my field is certainly not always the case). But the reviewer doesn't can't check that the data  weren't made up, that the experimental protocol was actually followed, that there were no errors in data analysis, etc.

In other words, the reviewer can do only and exactly what a good science blogger does. So good science blogging is, at its essence, a kind of peer review.

Drawbacks

Now, you might worry about the fact that the blogger could be anyone. There's something to that. Of course, ResearchBlogging.org has the same problem. Just because someone is blogging about peer-reviewed paper doesn't mean they understand it (or that they aren't lying about it, which happens surprisingly often with the fluoride fanatics).

So while peer review might be a useful way of vetting the paper, it won't help us vet the blog. We still have to do that ourselves (and science bloggers seem to do a good job of vetting).

A weakness

Ultimately, I think it's risky to put all our cards on peer review. It's a good system, but its possible to circumvent. We know that some set of scientists read the paper and thought it was worth publishing (with the caveats mentioned above). Of course, those scientists could be anybody, too -- it's up to the editor. So there's nothing really stopping autism-vaccine fanatics from establishing their own peer-reviewed journal, with reviewers who are all themselves autism-vaccine fanatics.

To an extent, that already happens. As long as there's a critical mass of scientists who think a particular way, they can establish their own journal, submit largely to that journal and review each other's submissions. Thus, papers that couldn't have gotten published at a more mainstream journal can get a home. I think anyone who has done a literature search recently knows there are a lot of bad papers out there (in my field, anyway, though I imagine the same is true in others).

Peer review is a helpful vetting process, and it does make papers better. But it doesn't determine fact. That is something we still have to find for ourselves.

****
Observant readers will have noticed that I use ResearchBlogging.org myself for it's citation system. What can I say? It's useful.

Peer Review: An Editor Responds

As previously noted, a loosely-translated Hitlerian tantrum about unreasonable reviewers has been making the rounds. Among recent viewers is Gerry Altmann, the editor of Cognition. He is clearly waiting for someone to make a similar video about unreasonable authors. He makes a good case, and it's worth reading (scroll down to the post titled "Peer Review" on Nov. 27th), but as there are more authors than editors, I suspect it wouldn't get as many views on YouTube.

How scientists feel about peer review...

I just had a manuscript rejected. I like to believe I handled it better.

More things you don't have time to read

PLoS One has published over 5,000 papers. Is that a sign of success or failure?

I've worried before on this blog about the exploding number of science publications. Publications represent completed research, which is progress, and is good. But the purpose of writing a paper is not for it to appear in print, the purpose is for people to read it. The more papers are published, it stands to reason, the fewer people read each one. Thus, there is some argument for publishing fewer, higher quality papers. I have heard that the average publication gets fewer than 1 citation, meaning many papers are never cited and thus presumably were not found to be relevant to anybody's research program.

It is in this context that I read the following excited announcement from PLoS ONE, a relatively new open-access journal:

nearly 30,000 of your peers have published over 5,000 papers with us since our launch just over two years ago.

That's a lot of papers. Granted, I admit to being part of the problem. Though I do now have a citation.

Can Peer Review Solve Conflicts of Interest?

As I wrote recently, Stephen Quake has been writing about conflicts of interest in research over at The Wild Side blog. He proposes solving these problems with peer review. I like the article, and he has many thoughtful things to say on the topic, but I don't really understand this proposal.

He doesn't give a lot of details as to how this would work. For instance:

When this bureaucracy asked me for a plan to manage conflicts in my own research, I wrote one that described all of the steps involved in peer review – and the COI committee sent it back as “too much.” In their view the process that scientific publications go through was more rigorous than necessary.

This reads as if the committee thought it would be too much work for him, but it sounds like too much work for the peer reviewers. Peer reviewers are not professional peer reviewers: they are typically volunteers from within the community who review papers partly out of a sense of social responsibility. Any proposal for expanding peer review has to keep in mind that the reviewers might not want the extra responsibilities.

I have not seen his proposal, but I'm not sure if it would work even in principle. As I wrote previously, nobody really knows what the data are but the researcher. Analyses can always be run in many different ways, potentially giving different results -- a problem whether your concern is conscious bias or unconscious bias. It's just not clear what peer review is supposed to do about that.

All Scientists Have Conflicts of Interest (Duh)

In a thoughtful and provocative piece in the The Wild Side blog at NYTimes.com, Stephen Quake takes up the issue of conflicts of interest in research. By "conflicts of interest," Quake means researchers who have a financial interest in the outcome of their research. It is becoming increasingly common for academic researchers to partner with businesses in developing new technology.

He raises many important and interesting questions, some of which I'll write about in the future. One point that deserves much more space was the fact that many researchers have conflicts of interest even when they aren't selling a product:

Interestingly, it is not unusual for basic scientists with no commercial relationships to be dependent on grants for their salaries and therefore have a significant personal financial interest in preserving their grants. Although COI experts have assured me that that this is not a conflict that needs to be managed, I must confess that I have some difficulty with the distinction they are trying to draw. Who is under greater temptation to bias the results of their research: the financially comfortable academic entrepreneur, or the ivory tower scientist who may not be able to pay his mortgage if his grant is not renewed?

What may not be clear to the casual reader is that research agencies like the NIH prefer to give grants to researchers with a history of success. Last time they gave you a grant, did you publish a series of important papers, or did all your projects end in failure? In the latter case, you may be out of a job.

Nobody Sees Your Data But You

A baseball player is paid largely on his ability to perform on the field. If you are known as a power-hitter, you better produce home-runs.

The major difference between a baseball star and a research star, is that a baseball player's performance is public. Everybody at the game knows whether he hit a home run or struck out.

In contrast, the only person with access to a researcher's data is the researcher. It is as if a baseball player went to an automatic batting cage where nobody was looking, took a few swings, came out and told the team management how many home runs he hit and was paid accordingly.

A number of non-scientists I've talked to seem to be under the impression that during peer review, the reviewers check the data. They don't. They can't, really. In my experiments, I ask people questions and mark down whether they got it right or wrong. The reviewers weren't in the testing room with me, so they simply have to take my word for it.

Now let's say you are a researcher and it's time to renew your grant. You've been swinging and missing -- your data are uninterpretable or simply uninteresting. The easiest way to make the data more interesting is to "fix" them.

All Scientists Have Conflicts of Interest

Ultimately, all scientists have a conflict of interest, because our promotions and salaries are ultimately based on the research we have produced, whether directly or indirectly. And there are many non-financial incentives. Nobody wants to be seen as a failure.

I don't believe many people are simply making up their data (though it happens). A larger concern is selectively reporting results. Suppose you are in the situation in which you have run four different experiments. Three of them support one conclusion, but the fourth supports the opposite conclusion.

You simply can't publish that. No journal will take a paper showing conflicting results. You can either give up on the project and admit to having wasted perhaps 2-3 years of your life. You can continue doing research to try to figure out why you are getting conflicting results, perhaps succeeding, perhaps not, but in any case spending time and money you might not have. Or you can write a paper about the first three experiments and forget about the "bad" experiment. Some people take the latter route. This is best-known in pharmaceutical research, but it happens everywhere.

Even worse, perhaps you run an experiment, the results of which challenge the theory for you which you are known, an experiment which challenges the validity of your life's work. Who really wants to publish that?

What to Do

Quake's point is that you can't eliminate conflicts of interest from science. A baseball player needs to win games. A researcher needs to publish. As long as this is true, baseball players will have an incentive to take steroids and researchers will have incentives to "improve" their work. Any proposed solution that ignores these basic facts is doomed to fail

Are Scientists Really Cheating?

This article might have looked really gloomy. My point is simply that everybody has the incentive to cheat, not that everybody does. In the course of your life, there will be a number of instances in which it would be in your financial interests to murder somebody. Still, most people don't murder.

It is popular in some circles to assume most people are fundamentally bad and will do anything they can get away with, but my experience -- professionally and personally -- is otherwise.

Moreover, academic fraud is generally not in one's long-term interests. Even if it isn't exposed, others will fail to replicate your results and your theories will be disproved. Again, for a researcher, ultimately the best thing for your career is to be right. And fraud won't help you with that.


(Image sourced from blog.springsource.com)

A rash of scientific plagiarism?

Nature reports that Harold Garner of the University of Texas Southwestern Medical Center in Dallas has been scouring the medical literature using an automated text-matching software package to catch plagiarized articles.

A surprising number have been found. 181 papers have been classified as duplicates, sharing 85% of their text, on average, with a previous paper. One quarter of these are nearly 100% identical to a previous publication.

While it is troubling that anybody would be so brazen, the fact that they have gotten away with it so far says something: there are a lot of journals. And a lot of papers. For a plagiarist to be successful, it must be the case that neither the editor nor any of the referees have read the original article -- this despite the fact that referees are typically chosen because they are experts in the field the article addresses.

That, I think, is the big news: that it is possible to plagiarize so blatantly.

Incidentally, the Nature news brief suggests that the confirmed plagiarism is usually carried out in obscure journals. This means that the plagiarists are gaining relatively little for their effort, and the original authors are losing little.

That said

Garner's project has apparently identified 75,000 abstracts that seem highly similar. It's hard to tell what that means, so we'll have to wait for the full report.

An abstract is about 200 words long. PsychInfo currently lists 10,098 which contain the phrase "working memory." One would assume that, even if all of them are examples of independent work, many are highly similar just by random chance. So I hope to find out more about how "highly similar" is being operationalized in this project.

While I suspect that plagiarism is not a huge problem, I still think it is fantastic that people are attacking it with these modern tools. I think we will be seeing a lot more of this type of work. (Actually, come to think of it, a professor I had in 2002 actually used an automated plagiarism-catching software program to screen student homework, so this has been around for a while.)

Should we trust experiments on the Web?

When I first started doing Web-based experiments, a number of people in my own lab were skeptical as to whether I would get anything valuable out of them. Part of this was due to worries about method (How do you know the participants are paying attention? How do you know they are telling the truth?), but I think part of it was also a suspicion of the Internet in general, which, as we all know, is full of an awful lot of crap.

For this reason, I expected some difficulties getting my Web-based studies published. However, the first of these studies was accepted without much drama, and what concerns the reviewers did raise had nothing to do with the Web (granted that only one of the experiments in that paper was run online). Similarly, while the second study (run in collaboration with Tal Makovski) has run into some significant hurdles in getting published, none of them involved the fact that the experiments were all run online.

Until now. After major revisions and some new experiments, we submitted the paper to a new journal where we thought it would be well-received. Unfortunately, it was not, and many of the concerns involved the Web. Two of the reviewers clearly articulated that they just don't trust Web-based experiments. One went so far as to say that Web-based experiments should never be run unless there is absolutely no way to do the experiment in the lab.

(I would use direct quotes, but the reviewers certainly did not expect their comments to show up on a blog, anonymously or not. So you will have to take my word for it.)

Obviously, I trust Web-based experiments. I have written enough posts about why I think concerns are misguided, so I won't rehash that here. I am more interested in why exactly people have trouble with Web-based experiments as opposed to other methodologies.

Is it because the Web-based method is relatively new? Is it because the Internet is full of porn? Or is it simply the case that for any given method, there are a certain number of people who just don't trust it?

I have been doing street-corner surveying lately (a well-established method), and I can tell you that although it ultimately gives decent results, some very odd things happen along the way. But I suppose if, as a reviewer, I tried to reject a paper because I "just don't trust surveys," the action editor would override me.

Against peer review

In response to a recent post, an anonymous commenter wrote that

It would be scientific misconduct ... to make statements based on someone else's unpublished work ... Scientific results don't exist until they have been peer reviewed and published.

Peer review has become the gold standard of the scientific community. Bring up a scientific finding, and the first thing you may be asked is, "Ah, well, is this peer reviewed?" (For those who don't know, peer review means that, before the journal will publish a paper, one or more other scientists who study similar topics). There is now even a popular blog aggregater that focuses exclusively on blogging about peer reviewed research.

In the age of the Discovery Institute there are some good reasons to focus on peer reviewed research as a way of excluding quacks. It's a way of saying that this research has been vetted.

That said, when I read comments like the one above, I think the time has come to push back, and point out that peer review is not the arbiter of truth. Truth is the arbiter of truth, and peer review is merely a flawed tool we use to help get there.

Peer reviewers don't check to make sure the results are true. Peer reviewers do not typically replicate the experiment in question. They do not check the math. Most of what they do is check that the arguments are reasonable and that the experiment(s) were well designed. Peer reviewers do not necessarily even have to agree with a paper they accept. They may simply think the data are compelling and the arguments are worth hearing, even if they may be wrong.

Thus, peer review does a reasonable job of weeding out quacks. Luckily, most scientists are not quacks, so what does it do for the rest of us? I'm not sure, but I think a partial answer is that two minds are better than one. Reviewers often notice things that the authors missed -- not because the authors weren't smart, but because research is damn complicated and you can never think of everything.

Typically what happens, at least in psychology, is that the reviewers suggest additional analyses or additional experiments that would make the paper stronger. Based on those comments, the authors may run new experiments then revise the paper and resubmit. Peer reviewers, in this sense, aren't so much vetters or fact-checkers as editors. Peer review is a way of improving -- not perfecting -- an article.

So is it scientific misconduct to refer to unpublished work? I don't think so. It is dangerous, though, because there are good reasons (above) to be more confident of something that has gone through peer review. It is a bit impolite to refer to something that has not been published, because then your audience can't go look at it themselves. And that's the main point. Peer reviewers are not the judges of truth, but all of us are on the jury.

Publishing papers is slow

The bread and butter of scientific communication is the peer-reviewed journal. For those who are not familiar with the process, when a scientist (me for instance) wants to report some data, he writes it up and sends it to a journal. The editors of the journal ask a few other scientists who are experts in the same field to read the article and decide if it's any good.

This process has been criticized for being arbitrary and for being unable or unwilling to catch fraud. For all that, I honestly believe that peer review serves to improve the quality of papers. At least in psychology, it is rare for a journal to accept a paper on the first round. Instead, the reviewers suggest changes and additional experiments. Since they are experts in the field and bring a fresh eye to the problem, they often have good ideas.

There is one issue with peer review, however, that drives me nuts. That is how long the process takes. In January, a collaborator and I submitted short paper to a journal that promises extra-fast reviews of short papers. Three months later, we our expected rejection along with suggestions from the reviewers.

The thing is, in the three months that have passed, we've gotten busy with other things. I had to reread the paper a few times because I had forgotten all the details (for some reason, January feels like it was years ago). I spent the last week figuring out how to edit the experiment software, because it required some fancy programing that I had forgotten how to do.

Without further complaining, I'd like to announce the re-launch of The Video Experiment. If you have already participated (this is the only experiment I have ever run that involved a video), please do not participate in this version.* First of all, you'll be bored, because this is only a slight variation on the old experiment, and the video is the same. But more importantly, knowing what the experiment is about could affect your results.

That said, if you've never participated in the video experiment -- if you've never seen the "Bill et John" video or the "Kiwi" bird animation, you haven't participated -- please do so. It only takes 5-7 minutes, and it's easily the most entertaining experiment I've run online. Plus you get to see your own results at the end. With any luck, I can collect all the data we need within a few weeks, and then we can resubmit this paper.



*If you really want to participate, go ahead, but be sure to mark on the demographic form that you participated in the past.

It's a small world

Recently, Grrlscientist has written a couple posts about double-blind peer review (where the reviewers don't know who the authors are as well as the authors not knowing who the reviewers are). She hopes that more double-blind peer reviewing would diminish possible sex discrimination in peer review.

Aside from some controversy as to whether there actually is any prejudice against women in peer review, many writing comments have asked whether blind peer review isn't just a facade. After all, it's a very, very small world, and we all know who each other are. Not only can you usually tell who is writing a paper without looking at the author's names (based on who they cite, their research topic, and their theoretical perspective), it's not always difficult to figure out who the reviewers are once you read the review.

It's a small world in other ways, too. My section of the department here at Harvard (development) is currently interviewing 6 candidates for faculty positions and 4 prospective graduate students. The job candidates are from the following schools:

Duke, Yale, Max Planck, University of London, MIT, Stanford

The potential graduate students are from:

Columbia, Stanford, Yale, Johns Hopkins

This is a narrow enough set as it is, but notice the repetition of Stanford and Yale. We have particularly close ties with Yale. One of my office-mates is from Yale. One of our upper-level graduate students will be taking up a professorship at Yale in the fall in developmental psychology (and a good friend of mine, who is graduating from the vision lab this spring will be taking up a post-doctoral fellowship at Yale as well).

Consider the job candidates for a different position in the department (cognition, brain and behavior):

Yale, MIT, Giessen, Yale, University of Chicago

It's not really surprising that it's such a tight-knit community, but it's still interesting to observe.