5_Testimonials
(from part 4, the essentials of science continued)
Testimonials and case study evidence
Stanovich1 writes:
“Case studies and testimonials are not useful at the later stages of scientific investigation because they cannot be used as confirming or disconfirming evidence in the test of a particular theory. The reason is case studies and testimonials are isolated events that lack the comparative information necessary to rule out alternative explanations.
The problem of relying on testimonial evidence is that if testimonials accumulate to support any specific remedy. All the competing remedies also have supporting testimonials. What we all want to know, of course is which remedy is best, and we cannot determine this by using testimonial evidence. As psychologist Ray Nickerson (1998) has said in his review of the cognitive processes we use to deceive ourselves:
‘Every practitioner of a form of pseudomedicine can point to a cadre of patients who will testify, in all sincerity, to have benefited from the treatment’ (p.192). Nickerson’s point is illustrated empirically in a study conducted by psychologist Anthony Greenwald and his colleagues (Greenwald, Spangenberg, Pratkanis, & Eskenazi, 1991).
In this study, the authors tested the usefulness of subliminal self help audiotapes (tapes that use messages below hearing threshold), which are commonly advertised in magazines and on television (Moore, 1995). They tested one tape program designed to improve memory and another to improve self esteem. After taking memory and self esteem tests, the subjects were given the tape and listened to it each day fro a month (the amount of time that the advertisers of the tapes said was sufficient to produce the advertised effects). Some subjects were given a self esteem tape labeled “self-esteem tape,” and the other subjects were given the memory tape labeled “memory tape.”
Importantly, however, two more groups of subjects were tested: one given a tape that was labeled “self esteem tape” but had the content of the memory tape and another given a tape that was labeled “memory tape” but had the content of the self esteem tape. These two conditions served as critical controls. What happened was that there was no improvement in actual memory or self esteem. However, there were differences in the
self-perceptions (testimonials) among the groups. Here, it was the label on he tape that was important, not the content. Both groups receiving a tape labeled “self-esteem” scored higher on the measure of self-perception of improvement on self esteem (even though one group had received memory content), and both groups receiving a tape labeled “memory” scored higher on the measure of self perception of improvement in memory (even though one group had received self-esteem content). In short these tapes generated plenty of testimonials despite the fact that their content had absolutely no effect on memory or self esteem (see also, Moore, 1995)”
How To Think Straight About Psychology, Stanovich¹ (p.59, 2001)
Everybody is vulnerable to the same psychological biases that are evident in this study, even those trained in science. Imagine for a moment that one of the participants above was a M.D. or Ph.D. from a good school (or imagine a celebrity like John Lennon or Tom Cruise). Imagine how convincing it would be to put that doctor’s (or celebrity) testimonial on the self-esteem tape company’s website or on the cover of their next book.
But how is this relevant to Primal Therapy? Stanovich uses an example that is obvious to most as a bogus product so that we can think of how it applies to things that are not so immediately obvious. With regard to Primal Therapy you have to ask yourself “Why haven’t cleverly designed studies with similar critical controls been used to measure the therapy?”. In more than 30 years of Dr Janov claiming to have a scientific therapy, there has been no independent, peer reviewed and replicated studies with the same kind of ingenious critical controls as described above. The question is, why have they been avoided? Why have the “experiments” done by Janov and his followers all been designed so as not to answer the questions the public really wants to know when choosing a therapy.
As far as I can tell almost every so called study done at Dr Janov’s primal center have been controlled and interpreted by Dr Janov or his fans who
also believe in primal therapy, and it is his interpretation, (his followers then adopt his interpretation), that then makes it to print in his books.
It is not that gathering case study data is wrong, what is objectionable is the misrepresentation that it is evidence or even proof of primal theory. In fact the very same data could be interpreted any number of ways from any other unfalsifiable theory.
Lets take a real example:
“[name omitted] told me during the 3 weeks intensive [name omitted] went through first line feelings and [he/she] had slowed down and felt rested, also that [his/her] skin has cleared. Now that shows the truth , that the therapy works.”
This is a testimonial, and although I know for sure it was meant sincerely, there are some biases and factors that affect all testimonials. To eliminate cognitive biases you have to conduct an experiment, with as many people as possible, and you repeat the experiment later. In this case it would resemble a clinical trial, where the variable you allow to change from group to group is the treatment type. You cannot eliminate bias by trying to be unbiased, that’s not how it works. Cognitive biases are natural and normal and everyone has them. Even good intelligent people have cognitive biases, and even with those people, an experiment is needed. (See “challenge to” section).
But let’s address the specific testimonial above. I have the permission from the person to discuss it, and he/she has told me some further information about it. The fact they felt rested could have been due to a whole host of things. The person had been working very hard at work (to save up money for therapy) right up to the day of the flight that brought them to therapy. In the three weeks, they were not allowed to work, and being in a beautiful place, with no work, maybe that was why they felt more rested. It’s not surprising the vital signs went down, especially after the nervousness of the first day.
Maybe his/her skin cleared due to less bacteria, or less stress due to no work, or due to the higher strength of sunshine. Maybe the person forgot that his/her skin cleared before therapy. Maybe it changed due to aging. Maybe the person’s intense belief in primal therapy, in fact “knowing with religious fervor that it worked ‘a priori’”, colored his or her testimony. There are many variables like this.
What happens is people become hyper sensitive to any change, real or imaginary, that happens in therapy. Whatever happens that is positive, we attribute it to therapy. But think about this, good and bad things happen to everybody throughout life. With any given placebo treatment, there will be something good that happened in that time period. If the placebo taker was told that the placebo could help any of the myriad of health and psychological problems, they are going to be able to find something to point to as proof.
People change with or without therapy, the key is to doing controlled studies (as they did when comparing cognitive therapy and medication for depression, see Abnormal Psychology, Barlow 2004). Now the treatment does not in of itself have to be mechanical and scientific, in a scientific study, it can be anything, physical, emotional, primalling or talking. It is only the measurement of the efficacy of such treatments that is controlled. And when we say the study needs to be controlled, we don’t mean the patient has to control feelings, the treatment is pretty much free to be whatever within ethical guidelines. It just means you control variables to eliminate most of the many cognitive biases.
1How to Think Straight About Psychology (6th Edition). (2001, IBSN 0-321-04713-3) Keith E. Stanovich. If this book is not easily available, try the latest 8th Edition (2006)
