Science's 'irreproducibility crisis' is a public policy crisis too | TheHill: How much of a crisis? In 2005, Dr. John Ioannidis, now of Stanford, estimated that as much as half of published research findings in biomedicine are probably false. In some other fields it may be worse. The research on which concepts such as “stereotype threat,” “power poses,” and “implicit bias,” for example, reproduce badly if at all.
The scientific community is not lacking for those who tut-tut these findings. Skeptics say that just because someone tried to reproduce an experiment and failed doesn’t mean the original results were wrong. That’s a reasonable point, but evidence keeps mounting that the reproducibility crisis is real.
My colleagues and I at the National Association of Scholars think we know one of the deep causes of the crisis: bad statistics. Nearly all science these days depends on assessing the likelihood of a hypothesis. Mess up the connection between hypothesis and data, and the result is a “finding” that might as well be pulled out of thin air.
This is how that often happens. Researchers keep running statistical tests, one after the next, until they hit a fluke correlation. They then publish the correlation as though it is a positive result. That’s like the Texas farmer who shoots a hole in the side of his barn and then paints a bullseye around it.
The scientific community is not lacking for those who tut-tut these findings. Skeptics say that just because someone tried to reproduce an experiment and failed doesn’t mean the original results were wrong. That’s a reasonable point, but evidence keeps mounting that the reproducibility crisis is real.
My colleagues and I at the National Association of Scholars think we know one of the deep causes of the crisis: bad statistics. Nearly all science these days depends on assessing the likelihood of a hypothesis. Mess up the connection between hypothesis and data, and the result is a “finding” that might as well be pulled out of thin air.
This is how that often happens. Researchers keep running statistical tests, one after the next, until they hit a fluke correlation. They then publish the correlation as though it is a positive result. That’s like the Texas farmer who shoots a hole in the side of his barn and then paints a bullseye around it.