Rocky Mountain Skeptics
The April First 1998 issue of the Journal of the American Medical Association (JAMA) contained a "Close Look at Therapeutic Touch" (pp.1005-1010). The article’s authors are L. Rosa (LR), E. Rosa (ER), L. Sarner (LS) and S. Barrett (SB). As is usual with JAMA when they have something particularly interesting in their current issue, the magazine itself was preceded by a press release which was distributed to media throughout the world.
By the time the actual magazine hit the streets, it had been upstaged by breathless reports on and in all the major media in the US and Canada. What caught the world’s attention was the purported proof that the basis of Therapeutic Touch (TT), the human energy field (HEF), does not exist. And the fact that put the study over the top for the media was the apparent intimate involvement of a young girl in the study. "Emily’s little experiment" was the wonder of the press and her face was seen everywhere for a day or so.
We do not undertake the task of critiquing Emily’s study lightly. She is the currently eleven year old daughter of LR and LS who are respectively the first and third authors on the study. We heartily concur and are delighted to note that the study, as published in JAMA, opened a small crack in the world’s certainty that TT and other tradition-based medical practices such as chiropractic, aroma therapy or acupuncture work.
However, the unequivocal conclusion drawn by JAMA in its Editor’s Note: "This simple, statistically valid study tests the theoretical basis for ‘Therapeutic Touch’: the ‘human energy field.’ This study found that such a field does not exist." is simply not so. The work reported in the JAMA article does not support that conclusion.
The public policy recommendations made by the editor, based on this one report, demand that questions be asked: "I believe that practitioners should disclose these results to patients, third-party payers should question whether they should pay for this procedure, and patients should save their money and refuse to pay for this procedure until or unless additional honest experimentation demonstrates an actual effect." Can or should a public policy statement be made based on one experiment? We have always chided TT experimenters for advocating policies based on a single favorable test. Should a single test, however welcome its conclusions, with many questionable aspects be treated any differently? In fact, is it not essential to point out inadequacies ourselves in work that apparently supports our beliefs in order to allay any suspicion of dishonesty on our part?
The only conclusion that can be drawn from this report is that twenty-one people who accepted the challenge to be tested by the authors failed to produce outcomes necessary to convince us that the HEF exists. They do not prove that the HEF does not exist.
In brief, the report speaks about "2 series of tests" (p 1007) which consisted of: 1) a group of tests administered to 15 individuals at their homes or offices over a period of several months, 2) 13 individuals which included 7 from the first group. This second group was tested the following year as part of a film production being made for Scientific American Frontiers while the first group was assembled for Emily’s 4th grade science project.
Some specific problems with the tests are:
Experimenters’ Bias
Nearly one-half of the paper consists of background material. Discussion of "Professional Recognition," "The TT Hypothesis," and "Literature Analysis" is a valuable asset to readers. It clearly states the historical and current status of the practice of TT. Much of what is said about Dolores Krieger, Dora Kunz and Martha Rogers paints a picture of people biased in favor of the TT hypothesis. "Although often presented as a scientific adaptation of ‘laying-on of hands,’ TT is imbued with metaphysical ideas." (p 1005); "Dora Kunz, who is considered TT’s co-developer, was president of the Theosophical Society of America from 1975 to 1987. She collaborated with Krieger on the early TT studies and claims to be a fifth-generation ‘sensitive’ and a ‘gifted healer.’ " (p 1006)The emphasis in these quotes is on words indicating theological rather than scientific origins of the practice. This description of the origins of TT has thus undoubtedly set the stage for a bias against TT among the science-based medical community who are the primary readers of JAMA. Furthermore, such phrases as "laying-on of hands," "metaphysical ideas," "Theosophy Society," "fifth-generation sensitive," "gifted healer" are not relevant to the experiment’s own design and findings. Clearly, some description of the background of TT is necessary. However, such biased words can scarcely contribute to an impartial evaluation of the experiment by JAMA readers.
LS and LR have published numerous articles debunking TT. They are members of or founders of groups whose purpose is to expose TT as unsound scientifically. They therefore have a well-established reputation in the skeptical community as adversaries of the TT practice as being science-based. Since ER is the daughter and step-daughter of LR and LS, it is difficult to imagine that she does not share her primary care giver’s attitudes about TT. Thus, it is highly likely that the experimental design and the experimenters themselves were biased against the existence of the HEF. Any honest experiment, therefore, should have been conducted, and data recorded by neutral experimenters.
Sampling Bias
Based on the description in the Methods section of the JAMA article, LR and LS found 25 TT practitioners (TTps) in Northeast Colorado "by searching for advertisements and following other leads," (p 1007). Of these twenty-five, twenty-one agreed to be tested. Nowhere in the Methods section is a description of any parameters established prior to the search for TTps that would have established objective criteria for them (e.g. number of hours trained, certificates, classes). It is clear that no random sampling was intended or even possible. This may be justified by the fact that, at the initial stage of this whole endeavor, the idea was simply to perform a 4th grade science project. Additionally, we have no understanding of what is meant by "other leads." After all anyone can claim to do HEF work, even those suffering from self delusion.
Nowhere are we told how contact was made. For the first group the subjects were informed that they would be participating in Emily’s 4th grade science fair project. For the second group, they were told that they would be videotaped for a possible broadcast.
Very sketchy reports of recruitment of TTps through advertisements and "other leads" is not substantially helpful to others who may try to replicate these experiments. Nor can these self-selected participants be considered a representative sample of any population. Skeptical experimenters, including the author of this article, have had almost no success in getting TTps to cooperate in any skeptics-led experiment of TT. The skeptical community would benefit from a more detailed description of the selection and agreement process between the experimenters and the test subjects.
Unclear Protocol
In the first set of trials we are told that: "15 practitioners were tested at their homes or offices on different days for a period of several months." Emily, at this time, was 9 or 10 years old. It is unlikely that she went alone to the test locations. Who drove her and helped her when she arrived? Was it LR, LS or both? Perhaps other skeptics went along. There appears to be no effort to create a controlled environment or even one that was similar in significant aspects to other experiment venues.
During the data-gathering phase of the experiment, the only attempt to control for any aspect of the experiment’s parameters is the coin-flipping procedure. We are told that Emily flipped a coin several times and 10 flips constituted a trial set. There was an opaque divider separating the TTp from Emily; a towel was placed over the arm of the TTp to permit "blinding". This aspect of the test procedure, while called "blinding" is actually the minimum requirement for obscuring the actual test from the participant.
Based on the description in the JAMA article, there was never any consideration given to an attempt to double-blind the experiment in any way. No matter the difficulties encountered in trying to double-blind the test, it should have been done in order for the experiment to have produced anything approaching valid test results.Once again we may ask whether – since this endeavor was intended to be a 4th grade science project both in conception and execution – it was suitable for publication in a journal that prides itself on its significant contributions to the science literature?
Each of the 15 sites visited was probably unique. Was any effort made to level the playing field for each test subject? Was the lighting similar? Was the time of day similar for each TTp? How many people were in the room while the test was being conducted? Were LR, LS or others in the room during the testing? Who recorded the result of the coin flip? Did anyone check to ensure that the appropriate hand was hovered over the test subject’s hand?
In film clips from the Scientific American Frontiers production, (more about this below) Emily is shown stopping the coin in its spin by placing her hand on it. This cannot be called a random toss when the experimenter is intervening in the process. When the TTp indicated a response, who documented it? Were still photos or video or audio recordings made during the first set of tests? There is no description of the documentation procedure.
The second set of tests was performed for a film crew from Scientific American Frontiers. The data collected at this single location were deemed equal to the first batch of points obtained from many separate locations. The two batches were simply added together. There is an enormous and significant difference between filming for a TV production with several camera crews, lights, director and others present, milling around and making sure that the film is useful for the show and a group of people arriving at a house or office with the purpose of performing some task. The film crew’s interest in the exactitude of the experimental procedures underway are, at best, incidental.
What is the relationship between the unspecified experimental procedure in the first experiment (before any idea of TV and JAMA entered the minds of any of the participants) and the second "procedure" on display primarily for the film crews?
Controls
The description of attempts to control for testable variables is very limited. The totality of the discussion consists of: "To examine whether air movement or body heat might be detectable by the experimental subjects, preliminary tests were performed on seven other subjects who had no training or belief in TT. Four were children who were unaware of the purpose of the test. Those results indicated that the apparatus prevented tactile cues from reaching the subject." (p 1008) This description does not constitute a control. If it was the same experimental protocol then what were the results? What about other possible cues? How were the controls selected? Did the experimenter go to the controls’ houses or did they all come to her house to be tested all in one session? If it was not the same protocol then what value is the control procedure?
The Journal of the American Medical Association is a well-established publication with a reputation for printing reliable information that physicians can use to cure diseases and save lives. It is among the most prestigious and accepted science-based magazines in the world. It is listened to; its publication is eagerly anticipated every week and the popular media frequently report on significant or interesting information that it contains.
It is therefore, doubly egregious, indeed, completely irresponsible, for JAMA editors to give space to work that, at the very best, can be described as competent for a 4th grade science project. As shown above, the quality of the research is exemplary of either very bad science or adequate school work. No matter how desperate we in the skeptical community are for a win in our column, JAMA, as a respected member of this community, did us no service by either the publication of a schoolgirl’s project or the subsequent over-promotion of the results and pronouncements about the works’ significance and policy implications.
And, consider the outcry from critics of TT if a nine year old child whose mother was Dolores Krieger had conducted the TT experiment which concluded that the HEF did exist.
Carla Selby is Vice President of Rocky Mountain Skeptics and heads up its therapeutic touch special interest group (tt-SIG). She has contributed extensively to RMS’s efforts in shedding light on the ritual of therapeutic touch. Her most recent published writings include: Science or Pseudoscience? Pentagon Grant Funds Alternative Health Study, Skeptical Inquirer, vol 20, no 4 and UAB Final Report of Therapeutic Touch — An Appraisal, Skeptical Inquirer, vol 21, no 3.
Read the Rocky Mountain Skeptics' disclaimer about the article.
Last updated: 11/09/00
You are visitor number
Webmaster: Aviva Rothschild
Copyright Rocky Mountain Skeptics (RMS), 1998.