LiveScience is featuring an article on the unsuccessful replication of Daryl Bem's controversial precognition experiment by skeptics Richard Wiseman, Chris French and Stuart Ritchie, after their paper was posted on the open-access journal PLoS ONE:
Bad news for Miss Cleo and other alleged clairvoyants: A new study has failed to find evidence that psychic ability is real.
Skeptics may scoff at the finding as obvious, but the research is important because it refutes a study published in a psychological journal last year that claimed to find evidence of extrasensory perception. That research, conducted by Daryl Bem of Cornell University, triggered outrage in the psychological community when the Journal of Personality and Social Psychology announced in 2010 that the paper had been accepted for publication. Psychologists immediately leapt on Bem's statistics and methods, finding reasons how he may have come up with the unbelievable results.
But the real key to a strong scientific finding is reproducibility. If no other researchers can replicate a particular result, it's not likely that the result is real. So University of Edinburgh psychologist Stuart Ritchie and colleagues decided to mimic one of Bem's experiments almost exactly to see if they would also find evidence of psychic powers.
The results were clear. "We found nothing," Ritchie said.
The LiveScience report also quotes Daryl Bem as saying, in his official response to the negative replication, that this one failed study does not automatically invalidate his positive results - and that some attention should be paid to the alleged 'experimenter effect', where skeptical researchers get negative psi results, and neutral/pro-psi researchers get positive findings:
In a response to be published alongside Ritchie and his colleagues' research in the open-access journal PLoS ONE, Bem wrote that it was still "premature to conclude anything about the replicability" of his experiment. It takes multiple replication failures to definitively refute a study, he said.
Bem also suggested that because Ritchie, Wiseman and French are skeptical of psychic abilities, they might have unwittingly influenced their participants not to display any clairvoyance. (The computer-based design of the study, however, is supposed to help prevent researchers from biasing their participants.)
"This does not mean that psi[psychic phenomena] results are unverifiable by independent investigators, but that we must begin regarding the experimenter as a variable in the experiments that should be included in the research designs," Bem wrote.
Now, I think it may be a stretch for Bem to invoke the 'experimenter effect' explanation - unless we stretch it to some sort of telepathic or psychokinetic influence - as Bem himself set up the study protocol specifically to run via a computer in order to minimize the experimenter interaction.) And if we do that, I'm not sure how you could ever really study psi in a standard scientific manner. On the flipside, as I've noted previously, skeptics need to be careful of writing Bem's 'experimenter effect' concern off as 'special pleading', given that arch-skeptic (and lead author on the negative replication paper) Richard Wiseman himself has previously published papers discussing this possibility.
What I was amazed by though was what the LiveScience article *left out* from Bem's response...and also what Wiseman, French and Ritchie have thus far failed to share in every mention I've seen regarding their negative replication. According to Bem:
In their article, Ritchie et al. mention that their experiments were “pre-registered.” They are referring to an online registry set up by Wiseman himself, asking anyone planning a replication to pre-register it and then to provide him with the data when the study is completed. As he noted on the registration website: “We will carry out a meta-analysis of all registered studies…that have been completed by 1 December 2011.”
By the deadline, six studies attempting to replicate the Retroactive Recall effect had been completed, including the three failed replications reported by Ritchie et al. and two other replications, both of which successfully reproduced my original findings at statistically significant levels. (One of them was conducted in Italy using Italian words as stimuli.) Even though both successful studies were pre-registered on Wiseman’s registry and their results presumably known to Ritchie et al., they fail to mention them in this article. I consider this an important omission. (I also note that Ritchie et al., describe their replication attempt as three independent studies, but the total number of sessions they ran was the same as the number I ran in my own original experiment and its successful replication.) [my emphasis]
If Bem is correct in his claim, this means that both LiveScience and Wiseman/Ritchie/French are happily promoting their 'refutation', but failing to mention that Wiseman has also collected two other positive replications supporting Bem's results. This would certainly constitute "an important omission". I look forward to hearing more about this, hopefully when Richard Wiseman publishes the results of his meta-analysis of the pre-registered studies.
As an addendum, Chris French has written in The Guardian about his experience in trying to get the negative replication published in a journal, and what this might mean for the future of science journal publishing.