Are most research findings false?
Becoming wiser about using research in Occupational Therapy practice.
Having a strong skill set in critically appraising research is becoming more essential for OTs every year. But being critical should extend beyond questioning individual articles and include considering how the process of science unfolds.
In 2005, Dr. John Ioannidis a physician and scientist wrote highly cited article called “Why Most Published Research Findings Are False” (open access here). In the essay, John argues that most scientific findings (in all fields) are incorrect and he gives several suggestions on how we can better analyze findings.
Most research findings are probably (and demonstrably) false
The more something is studied, the less likely the results are to be positive.
The smaller the study, the less likely the results are to be true
The smaller the effect size, the less likely the results are to be true
Statistical significance does not equate to clinical or functional significance
The smaller the statistical significance, the less likely the results are to be true
Single studies are inconclusive and almost meaningless. Most single studies that have positive results are non-replicable.
Be aware of bias, the greater the conflict of interest or prejudice, the less likely a result is to be true
Non-standardized methodologies or flexible designs are bad.
The “hotter” the field, the less likely true. We often see “major excitement followed by severe disappointments in fields that draw wide attention”.
Reviews and meta analysis are only as good as the data they combine.
Claimed research findings may often be simply accurate measures of the prevailing bias
How can we improve our situation?
Rely on larger scale studies that look at a narrow, specific question which is likely to be true.
Be cautious about really large studies and epidemiological research because they “may be more likely to find a formally statistical significant difference for a trivial effect”.
Don’t “emphasize the statistically significant findings of any single team”.
We should move away from “chasing statistical significance” and should be testing relationships that are more likely to be true than untrue.
What does this mean?
The idea is not to ignore existing research, but to understand what it really tells us. Individual studies are prone to bias and being incorrect. At times, we’re better off focussing on overall trends and theoretical foundations rather than any one particular piece of evidence. For example, using “occupation-based” practice is probably the closest we can get to “evidence-based” practice. Trying to use specific interventions or programs is harder to justify, even with positive support from evidence because these findings are more likely to be false.
Confirmatory findings for specific interventions might also lead us astray by supporting modalities or interventions that are flashy but not functional. It’s happening in physiotherapy: many established modalities like ultrasound, IFC, manual therapy and more that have previously been supported by research are being demonstrated to be ineffective - and there is a renewed focus towards functional interventions like exercise. OT is not very different. Our field is filled with modalities and interventions that are probably less effective than our core interventions of meaningful activity.