I’ve just come across a bombshell of a paper that looked at numerous headline studies on the cognitive neuroscience of social interaction and found that many contained statistically impossible or spurious correlations between behaviour and brain activity.
The article is currently ‘in press’ for the journal Perspectives on Psychological Science but the preprint is available online as a pdf file.
Social cognitive neuroscience is a hot new area and many of the headline studies use fMRI brain imaging to look at how activity in the brain is correlated with social decision-making or perception.
This new analysis, led by neuroscientist Edward Vul, was inspired by the fact that some of these correlations seem to good to be true, and so the research team investigated. The abstract of their study is below, and it’s powerful stuff – indicating that many of the results are due to flawed analyses.
If you’re not familiar with neuroimaging research it might be useful to know that what a ‘voxel‘ is before reading the abstract.
Essentially, brain scanners digitally divide the scanned area into a block of tiny boxes and each one of these is called a voxel (think 3D pixel).
This allows the scans to be analysed by comparing the activity or tissue density in each voxel to another measure – which could be the same voxel during another scan, or it could be something entirely different, such as a measure of emotion or social decision-making.
The newly emerging field of Social Neuroscience has drawn much attention in recent years, with high-profile studies frequently reporting extremely high (e.g., >.8) correlations between behavioral and self-report measures of personality or emotion and measures of brain activation obtained using fMRI. We show that these correlations often exceed what is statistically possible assuming the (evidently rather limited) reliability of both fMRI and personality/emotion measures. The implausibly high correlations are all the more puzzling because social-neuroscience method sections rarely contain sufficient detail to ascertain how these correlations were obtained.
We surveyed authors of 54 articles that reported findings of this kind to determine the details of their analyses. More than half acknowledged using a strategy that computes separate correlations for individual voxels, and reports means of just the subset of voxels exceeding chosen thresholds. We show how this non-independent analysis grossly inflates correlations, while yielding reassuring-looking scattergrams. This analysis technique was used to obtain the vast majority of the implausibly high correlations in our survey sample. In addition, we argue that other analysis problems likely created entirely spurious correlations in some cases.
We outline how the data from these studies could be reanalyzed with unbiased methods to provide the field with accurate estimates of the correlations in question. We urge authors to perform such reanalyses and to correct the scientific record.
The paper notes that some of the most widely-reported studies in recent years contain this flaw and this new paper has the potential to really shake up the world of social cognitive neuroscience.
pdf of preprint of ‘Voodoo Correlations in Social Neuroscience’.
Mind Hacks 
about time
My 36 yr old girlfriend is laying in a hosp bed about 2 die.The doctor’s cannot find out what is wrong with her,her husband say’s is some type of psychosis,she told me what it is,she traveled 2 Jamaica 2 visit a man she met online,To make a long story short,The family of this man hated her because she was american,and did not want her 2 return 2 Jamaica 2 marry the brother- son.She returned back here 2 CT,and she fell out on the floor in excruciating pain,went 2 the doc,they couldnt find anything wrong w/her.She ended up in a wheelchair not being able 2 walk.Wich means no travling back 2 Jamaica.All of her organs are failing ,she is pretty much paralyzed and she is on life support.They still dont know how this happened.She told me they put roots on her,she knows this for a fact.Is thier anything I can do 2 remove these root’s or voodoo from her body?Please help,my number is 860-477-7175Thank You Lisa,She doesnt have much longer maby a few days.
Did you ever solve the mystery, I see that this was back in 2012 and I was just urios, and if you I’d like to talk more into depth about the situation .
i work in the field though am not involved directly – and i’m a fan of mind hacks. but the issues aren’t as clear cut as the authors of ‘voodoo correlations’ suggest – a response from some of the ‘red list’ authors is now online at http://www.bcn-nic.nl/replyVul.pdf. there is more as well at http://www.nature.com/news/2009/090114/pdf/457245a.pdf. i hope the owner as well as readers of the site will have a look.
I agree with the point made by Alex. The issues are definitely not as clear cut as implied by the authors of “voodoo correlations”. The response from the red list authors that Alex posted above makes at least as compelling of a case in favor of their approach. Moreover, an actual read through of the studies that the authors of “voodoo correlations” attack persuaded me that the approach taken by many of the red list studies was actually quite sound and definitely not nearly as inappropriate as one might gather from only reading the “voodoo” paper. The “voodoo” paper may be stretching it a bit. So take it with a grain of salt.
Another response paper to the voodoo correlation issue was recently posted on the web by some of the criticized authors. I definitely recommend checking it out as it makes a strong case that there were some flaws with the Vul et al methods, analysis, and conclusions. This is further evidence that some of the studies may have been misconstrued or unjustly criticized by Vul and colleagues. The response can be found at:
Click to access LiebermanBerkmanWager(invitedreply).pdf
Thanks for reading my blog:
http://www.blog.sethroberts.net/2008/12/28/voodoo-correlations-in-social-neuroscience/
We had done some research using MRI brain imaging as a means of tracking Nicotine addiction when developing our electronic cigarette product. We wanted to test how the brain interpreted the use of an electronic cigarette versus that of a traditional cigarette. The technology is unbelievable and what you call voxels, we actually referred to as cubix. We actually found that there was no major difference brain activity between e-cigarettes and tobacco cigs when conducting our studies. Very cool technology that wasn’t even a thought 10 years ago.
Johnny B
Director Of Product Development
Halo Electronic Cigarette Company
Why is there more evidence that some of the studies have been misconstrued or unjustly criticized by Vul and colleagues