A recent study casts doubt on the overall reliability of neuroscience, begging the question of whether any evidence produced by the field can overcome potential Daubert challenges. The study, published on the website Nature Reviews Neuroscience, asserts that the small sample sizes used in most academic neuroscience studies results in conclusions which lack credibility.
Sample sizes in a scientific studies have a direct effect on the statistical power of the study. When a sample size is large, both subtle and major effects are discernible within the collected data. When the sample size is small, only the larger effects will be shown with any reliability and smaller effects may be missed entirely. False positives may be recorded or worse the size of the effect may be exaggerated.
A statistical power of 80 percent is the desired goal in most studies. At this level, if the sample size is adequate and the effect is genuine the study would detect it 80 percent of the time. The study published in Nature Reviews Neuroscience reviewed 49 meta-analyses, studies of other studies, and concluded that within these 730 individual studies the median statistical power was below 20 percent. Human based neuroimaging studies within the study reached a median statistical power of only 8 percent.
Further, the small studies in the field are often not blind-tested and often the results have not been reproduced.
While the results of the paper have caused angst in the academic field of neuroscience, if the conclusions are correct there are major legal implications as well. The Daubert standard and its relevant factors call into question any conclusions reached by one of these studies with low statistical power. It appears that most of the testing is done on such a small scale the results have not been subject to peer review and a widespread acceptance within the scientific community has not been attained.
Neuroscience remains an area of tremendous interest and potential in both the scientific and legal communities yet the statistics discussed in Nature Reviews Neuroscience suggest we may be far away from tangible uses of the field.