(Continued from part 3; the essentials of science:)

4. Peer Review and Replication

Peer review is the checking of articles and scientific work by other scientists in the same field.  Without peer review it is not science, in other words peer review is an essential element of science. It is not optional; you can’t claim to do science without it.  Replication is the repetition of one scientist’s experiment or study by other independent scientists to check the results.  It also is an essential element in science.  Stanovich¹ explains it this way:

“Scientific knowledge is public in a special sense…scientific knowledge does not exist solely in the mind of a particular individual. In an important sense, scientific knowledge does not exist at all until it has been submitted to the scientific community for criticism and empirical testing by others. Knowledge that is considered ’special’-the province of the thought processes of a particular individual, immune from scrutiny and criticism by others-can never have the status of scientific knowledge. Science makes the idea of public verifiability concrete via the procedure of replication. In order to be considered in the realm of science, a finding must be presented to the scientific community in a way that enables other scientists to attempt the same experiment and obtain the same results. When this occurs, we say the finding has been replicated… It ensures that a particular finding is not due simply to the errors or biases of a particular investigator. In short, for a finding to be accepted by the scientific community, it must be possible for someone other than the original investigator to duplicate it…

…one important way to distinguish charlatans and practitioners of pseudoscience from legitimate scientists is the former often bypass the normal channels of scientific publication and instead go straight to the media with ‘their findings.’ One ironclad criterion that will always work for the public when presented with scientific claims of uncertain validity is the question: Have the findings been published in a recognized scientific journal that uses some type of peer review procedure? The answer to this question will almost always separate pseudoscientific claims from the real thing…

Not all information in peer reviewed scientific journals is necessarily correct, but at least it has met a criterion of peer criticism and scrutiny. It is a minimal criterion, not a stringent one, because most scientific disciplines publish many different journals of varying quality. Most scientific ideas can get published somewhere in the legitimate literature if they meet some rudimentary standards. The idea that only a narrow range of data and theory can get published in science is false. This is an idea often suggested by purveyors of bogus remedies and therapies who try to convince the media and the public that they have been shut out of scientific outlets by a conspiracy of ‘orthodox science.’

But consider for a minute just how many legitimate outlets there are in a  field like psychology [between 100 and 200 journals are then listed on pages 12 to 14 of Stanovich’s book, although dozens more exist]. Virtually all halfway legitimate theories and experiments can find their way into this vast array of publication outlet.

…the failure of an idea, a theory, a claim, or a therapy to have adequate documentation in the peer reviewed literature of a scientific discipline is very diagnostic. Particularly when the lack of evidence is accompanied by a media campaign to publicize the claim, it is a sure sign that the idea, theory, or therapy is bogus…

The peer review process is far from perfect, but it is really the only consumer protection we have. To ignore it (or not be aware of it) is to leave ourselves at the mercy of the multimillion-dollar pseudoscience industries that are so good at manipulating the media to their own ends.” How To Think Straight About Psychology, Stanovich¹ (2001, p10-15)

Just a word to the wise: An appearance of peer review can be faked. Someone with a pseudoscientific psychotherapy can include accurate science in the field of neurology in their books. Neurology neither proves or disproves any psychotherapy.  Psychotherapies are tested with clinical type trials, not neurology. They then can hire someone from a local university, (or a friend or a patient with a relevant degree) and ask them to check the neurology science. Then they can thank the “scientist”  for helping them write the book in the credits: This is not peer review.

Peer review would involve looking not so much at neurology, but at such things as the miraculous claims of the therapy and the quality of the efficacy testing for the therapy, the real evidence for the therapy, etc.

¹ How to Think Straight About Psychology (6th Edition). (2001, IBSN 0-321-04713-3)  Keith E. Stanovich.  If this book is not easily available, try the latest  8th Edition (2006)