December 29, 2008
Voodoo correlations in social brain studies:
I've just come across a bombshell of a paper that looked at numerous headline studies on the cognitive neuroscience of social interaction and found that many contained statistically impossible or spurious correlations between behaviour and brain activity.
The article is currently 'in press' for the journal Perspectives on Psychological Science but the preprint is available online as a pdf file.
Social cognitive neuroscience is a hot new area and many of the headline studies use fMRI brain imaging to look at how activity in the brain is correlated with social decision-making or perception.
This new analysis, led by neuroscientist Edward Vul, was inspired by the fact that some of these correlations seem to good to be true, and so the research team investigated. The abstract of their study is below, and it's powerful stuff - indicating that many of the results are due to flawed analyses.
If you're not familiar with neuroimaging research it might be useful to know that what a 'voxel' is before reading the abstract.
Essentially, brain scanners digitally divide the scanned area into a block of tiny boxes and each one of these is called a voxel (think 3D pixel).
This allows the scans to be analysed by comparing the activity or tissue density in each voxel to another measure - which could be the same voxel during another scan, or it could be something entirely different, such as a measure of emotion or social decision-making.
The newly emerging field of Social Neuroscience has drawn much attention in recent years, with high-profile studies frequently reporting extremely high (e.g., >.8) correlations between behavioral and self-report measures of personality or emotion and measures of brain activation obtained using fMRI. We show that these correlations often exceed what is statistically possible assuming the (evidently rather limited) reliability of both fMRI and personality/emotion measures. The implausibly high correlations are all the more puzzling because social-neuroscience method sections rarely contain sufficient detail to ascertain how these correlations were obtained.
We surveyed authors of 54 articles that reported findings of this kind to determine the details of their analyses. More than half acknowledged using a strategy that computes separate correlations for individual voxels, and reports means of just the subset of voxels exceeding chosen thresholds. We show how this non-independent analysis grossly inflates correlations, while yielding reassuring-looking scattergrams. This analysis technique was used to obtain the vast majority of the implausibly high correlations in our survey sample. In addition, we argue that other analysis problems likely created entirely spurious correlations in some cases.
We outline how the data from these studies could be reanalyzed with unbiased methods to provide the field with accurate estimates of the correlations in question. We urge authors to perform such reanalyses and to correct the scientific record.
The paper notes that some of the most widely-reported studies in recent years contain this flaw and this new paper has the potential to really shake up the world of social cognitive neuroscience.
pdf of preprint of 'Voodoo Correlations in Social Neuroscience'.
