Another possible hit this week for Professor Daryl Bem’s controversial “Feeling the Future” experiments, which found positive evidence for precognition, with the publication of a large-scale replication study which found no psi effect. The full paper, “Correcting the Past: Failures to Replicate Psi“, is freely available for download, and I recommend downloading it and having a good read. No doubt many will suffer (as I did) through the more technical descriptions of statistical analysis, but amongst that there is fascinating, respectful discussion of the Bem experiments and this latest replication attempt. In addition to the experiment results, the paper also features a meta-analysis of all replications attempted so far, which found again no significant evidence for precognition effects.
For those that can’t make it all the way through the paper, skeptic Steven Novella already has a large write-up of the new paper at Neurologica, in which he addresses many of the key points, as well as giving a broad overview of the entire Bem controversy.
Bem’s studies have not fared well in replication. Earlier this year Ritchie, Wiseman, and French published three independent replications of Bem’s 9th study, all negative. Of further interest is that the journal that originally published Bem’s article had declined to publish Ritchie et al’s paper claiming that they don’t publish replications. This decision (and editorial policy) was widely criticizes, as it reflects an undervaluing of replications.
It’s good to see that the journal has relented and agreed to publish a replication. Galak, LeBoeuf, Nelson, and Simmons should be commended, not only on their rigorous replication but their excellent article, which hits all the key points of this entire episode.
The researchers replicated experiments 8 and 9 of Bem (they chose these protocols because they were the most objective). They conducted 7 precise replications involving a total of 3,289 subjects (Bem’s studies involved 950 subjects). Six of the seven studies, when analyzed independently, were negative, while the last was slightly statistically significant. However, when the data are taken together, they are dead negative. The authors concluded that their experiments found no evidence for psi.
I leave it to more qualified minds than mine to authoritatively assess the merits of the paper. For what it’s worth, however, here’s my thoughts (caveats abounding):
I would imagine Bem would criticize this new replication on a key point, one which Novella glosses over in his assessment (in calling it a “rigorous” and “precise” replication) – that four of the seven experiments were done online, not in the lab. Additionally, one of the online experiments (#7) had roughly 2400 respondents, so across all 7 tests the amount of ‘lab’ results is only about 12%. This online aspect brings in a number of points of failure, from inattentiveness and distraction, right through to unintentional (by knowing about and thus being prepared for the ‘surprise’ test at the end) or intentional sabotage – it’s worth noting that the availability of the online test was passed around on skeptical forums such as the JREF and Rational Skepticism. Bem himself has criticized a previous paper from Galak et al. on this very point, saying when you do the test online, “you lose total control over it”. Interestingly, two of the three lab-based experiments done by the researchers had significantly lower p-values (p=0.04 and p=0.10) than the other tests.
Additionally, the researchers acknowledge explicitly that after reading Bem’s replication notes, they noticed that “there were at least three differences between our experiments (which followed the procedure described in Bem’s published paper) and the full procedure actually employed by Bem.” This included using a different word set to Bem for some tests. Steven Novella notes in his blog summary the importance of precise replications, saying that “a precise replication should have no degrees of freedom.” I find it hard to imagine how he reconciles this view with his support of the paper (describing it multiple times as a “precise” replication), and therefore that it “provides further evidence against psi as a real phenomenon, and specifically against the claims of Daryl Bem”.
Having said that, the paper itself does an admirable job of explaining these limitations/problems, and providing alternate analysis excluding some of these factors which appears to still show support for their conclusions. However, I had the distinct feeling that some of those exclusions were rather arbitrary (for instance, how to exclude possible sabotage), and in the end I’m not sure that they overcome the larger problem of (a) the massive online component and (b) the lack of precise replication, in terms of making this latest study acceptable as a true replication of Bem’s experiments. Nevertheless, fascinating reading and well worth your time.
You might also like…
