Chemistry has its own problems with replication, according to Nature:
Scrounging chemicals and equipment in their spare time, a team of chemistry bloggers is trying to replicated published protocols for making molecules. The researchers want to check how easy it is to repeat the recipes that scientists report in papers ... Among the frustrations [chemists] have experienced with the chemical literature ... are claims that reactions yield products in greater amounts than seem reasonable, and scanty detail about specific conditions in which to run reactions. In some cases, reactions are reported which seem to good to be true - such as a 2009 paper which was corrected within 24 hours by web-savvy chemists live-blogging the experiment.
It's hard to tell from the article how common it is for a reaction simply not to be possible at all as opposed to simply produce less product than reported. Presumably either is problematic, but the causes would be different.

Given the recent excitement about (non-)replication, one has to wonder if this problem is more or less common than in the past. While my gut instinct is that replication was probably less of a problem in the earlier, smaller days of science, it's also quite possible that it's like many forms of violent crime: extremely rare today by historical standards, but we care much more about it.

1 comment:

P.D. Magnus said...

There are really two separate issues.
1. Claims which are really too good to be true. Those can't be replicated because they never happened the first time.
2. Claims which can't be replicated from the published details alone, because there is lots of craft knowledge involved. Replication is possible, but it requires either interacting with the original team or a lot of trial and error. SSK studies of tacit knowledge and trust show that this has always been a barrier to replication.