An article called Reproducibility: the risks of the replication drive just came out in Nature. In it, Mina Bissell makes some great points.
The main idea: replicating studies is hard. It’s easy to tweak something (without even knowing that you did it) and end up with different results. Because of this, it’s important not to cast doubt on the results of someone else’s experiment too quickly. Communicating with the lab who did the original study is important if you find yourself running into problems. Failure to replicate can have serious consequences: good scientists can lose credibility, promising lines of research may not be pursued, etc. Thus, attempts at replication should be taken seriously, and everyone should try to remain civil during the process.
OK. I think everyone probably agrees with that!
But there were parts of the article that made me a little uncomfortable. Bissell gives compelling examples of how tiny changes–using the same cell line, but from different laboratories, for ex.–can torpedo replication attempts. I too believe that this happens frequently, so no arguments there. But unlike Bissell, I see this as a major problem. If you can’t replicate a study using virtually (but not totally) identical conditions, how generalizable are the original results likely to be? How useful is an experiment that yields such shaky findings? If we can’t replicate findings in the lab, what are the odds that they will describe what’s happening out in the messy real world?
Bissell describes a comforting example in which exploring a failure to replicate under slightly different conditions yielded valuable scientific data. I’m sure there are serendipitous situations like that one, but I also suspect that they are few and far between. My suspicion is that, in most cases, when other labs fail to replicate an experiment after credible attempts to do so, there is probably a real problem with the original study. Either (1) because the original results were faulty in some way or (2) because the original results, though valid, are not at all robust. Either way, the science community needs to know. So in my eyes, the drive for replication remains vital and the risks are well worth it.