inadequate research methods do not have validity and cannot contribute to knowledge.

Research Methodologies

The need to expand biomedical assumptions to include psychosocial and behavioral factors has become increasingly understood since Engel's 1977 comment. Indeed, the vast literature of studies that now includes psychosocial dimensions of disease outcomes, health, illness behavior, and correlates of well-being attests to wide acceptance of Engel's challenge. Many of these studies either encompass or cover aspects of some of the alternative therapies discussed in this report.

But other alternative systems and methods have not been adequately studied. One reason for this, according to various alternative medicine practitioners, is that conventional medicine researchers typically and inappropriately demand application of the "gold standard"—that is, prospective randomized clinical trials-when they are not appropriate. This demand occurs despite the availability of a range of suitable research methods from which to choose and the possibility that new methods will have to be identified to fit the situation. Sometimes the demand is for unusually large and complex designs intended to solicit multiple data rather than more appropriate, smaller designs that focus on first-step issues (Pavek, 1994).

Indeed, a review of published conventional research over the years indicates that prospective randomized clinical trials are not always possible or preferred. A 1990 report of the Institute of Medicine's Committee on Technological Innovation in Medicine supports this tacit reality and discusses methodological options:

It has also become clear that randomized controlled clinical trials are not necessarily practical or feasible for answering all clinical questions. Therefore, a variety of other methods, such as nonrandomized trials or observational methods, have been adopted to provide complementary information. Traditionally these methods were regarded as weaker than randomized clinical controlled clinical trials for clinical evaluation. Recent methodological advances, such as the use of non-classical statistics and the ability to link large-scale automated databases for analysis... are strengthening these approaches.

Issues in Evaluating Alternative
Medical Systems

Research design, even for conventional medicine, is a difficult and challenging process, even more so for alternative systems (Patel, 1987a). As should be abundantly clear from the introduction to the report, there is not one alternative medicine, but several. These consist of new approaches to patient care, new and unusual biomedical disease fighters, discrete methods of treatment, and systems of diagnostics and therapeutics that rely on and are governed by new paradigms.

Fortunately, helpful guidelines exist. The foremost guideline to keep in mind is that the basic goal of any investigation concerning a treatment for human beings is to determine whether the treatment makes a difference. Campbell and Stanley's classic monograph on experimental and quasi-experimental designs discusses lack of "internal validity"3 as the most serious threat to answering that fundamental question, commenting that "internal validity is the basic minimum without which any experiment is uninterpretable: did in fact the experimental treatments make a difference?" (Campbell and Stanley, 1963). The monograph lists several types of threats to internal validity, including

⚫ the possibility that events during the evaluation may unintentionally influence outcomes;

• the possibility that changes occurring naturally over time may be mistaken for treatment results; and

⚫ the risk that subject self-selection, rather than the treatment under study, caused the result or lack of result.

Sophisticated experimental designs eliminate or control threats to internal validity. Many threats-though not all-are controlled when studies compare two or more treatments by randomly assigning subjects to each of those treatments. In designs lacking random assignment, additional efforts are needed to bolster as much as possible the validity of the evaluation.

The second major concern for designing evaluations is external validity—that is, the ability to generalize the results of the evaluation to other populations and settings.

Other methodological concerns that have been raised address certain problems that are typically advanced in discussions of alternative medical research. Indeed, research in alternative medicine often appears fraught with conditions that seem uncontrollable or impossible to study. In many instances, however, the technique or a comparable one has already been studied, and published results can provide both encouragement and specific guidance.

As examples, there are useful approaches for studying music therapy provided in papers on how to assess the effects of music therapy in Alzheimer's disease (Aldridge, 1993). Eisenberg and colleagues (1993) reviewed numerous controlled studies assessing the effectiveness of cognitive behavioral techniques in managing hypertension. Methods for controlling or dealing with unwanted influences of subject, practitioner, and environment in learning situations can be found in such general texts as Complementary Methods for Research in Education (Jaeger, 1988), Research in Education (Best and Kahn, 1986), and Introduction to Educational Research (Charles, 1988).

Additional examples of research concerns in alternative medicine, along with proposed solutions, are presented below.

Measuring the Perspectives of Patients

Systems of health, as well as individual health care practitioners, vary in their approaches to health care, patients' decisionmaking, and intended outcomes (end points). Some systems and practitioners focus on quality-of-life issues as being paramount to surgical operations and chemical treatments. This emphasis on quality of life-and on patients making their own decisions-often is considered typical of alternative medicine. There is also a large conventional medicine literature dealing with quality-of-life issues.

Sometimes, difficult decisions must be made that are influenced by the views of practitioner, patient, and health care system. The choice might involve enduring long-term minor discomfort by not electing surgery or choosing a surgical procedure that will eliminate the discomfort but that carries a 2-percent chance of death. Or the choice might concern electing to have or to forego artificial life-sustaining procedures and equipment, such as resuscitation and heart-lung machines in terminal stages of disease.

Breast cancer research is an example of research that led to choices for patients. Under many circumstances, women may elect treatment for their breast cancer: today's choices are lumpectomy plus radiation therapy versus mastectomy. Men diagnosed with advanced prostate cancer also have medically equivalent treatment choices-surgery (orchiectomy) versus subcutaneous injections-with both approaches achieving the same goal of halting the male hormone that promotes prostate tumor growth. In these two examples, patient choice became possible when careful clinical research produced treatment options and then documented that the old and new treatments were equivalent in their effects.

Patients have even more choice in the absence of major or potentially fatal disease. Faced with symptoms of an enlarged prostate, for example, many men elect to endure their symptoms rather than undergo surgery. Recent Agency for Health Care Policy and Research (AHCPR) guidelines made this choice explicit for patients with enlarged prostates and increased national awareness of the need for patient involvement and sensitivity to quality-of-life issues. The guidelines were developed by a panel that was required to include health care consumer representation. AHCPR advised doctors and men with enlarged prostates (not prostate cancer) to curtail the use of two widely used diagnostic tests that frequently led to surgery and to rely instead on a questionnaire that quantifies how severely the condition affects the patient's quality of life.

Because some patients are comfortable with symptoms that other patients consider unacceptable, and because an enlarged prostate is not life threatening, the recommended focus on the patient's view of symptoms—rather than clinical measurement of prostate enlargement—is expected to lead to more appropriate treatment and decreased costs for surgery. (In 1992, before the guidelines, some 220,000 medicare patients received corrective prostate surgery performed at a cost of more than $1 billion [see sidebar].) Alternative practitioners note that increased use of patient choice and quality-of-life decisionmaking as exemplified by these guidelinesboth encourage financial savings and address psychosocial concerns that should not be neglected.

Disbelief

Disbelief is a factor not frequently addressed. Here, disbelief refers to the opinion of a physician, investigator, or research organization that a particular procedure or approach is ineffective. If this opinion is held by someone in a position to influence research funding or conduct, its impact can be widespread. In the view of alternative medicine researchers, the two typical ways in which they are affected include (1) outright, knee-jerk rejection of study proposals and (2) insistence on inappropriate and/or unnecessarily cumbersome study design.

A recent study provides corroboration concerning the alternative medicine researchers' perception of bias (Wilson et al., 1994). Wilson and his colleagues found that both medical and psychological researchers were more willing to overlook or disregard methodological flaws in studies that addressed "important" topics rather than "less serious" ones and to be more demanding in their standards for the latter.4 In an analogous way, scientists with little knowledge of or interest in alternative medical subjects could be expected not to take them seriously or to demand additional proofs.

Indeed, a frequent complaint of alternative medicine practitioners is that they are often obliged to conduct later stage studies even before preliminary information is gathered (Pavek, 1994). They also report the strong tendency of research institutions and methodologists to insist that very stringent controls be included in beginning studies-controls that would never be considered for early investigation of more conventional subjects. In other words, alternative medicine researchers protest that the standards of proof are raised for research on alternative practices.

In The Cancer Industry, Ralph Moss provides another good example of such misdirection and misperception. Moss examined the American Cancer Society's (ACS's) list of unproven methods for the 1970s and 1980s, which mentioned 70 practitioners and 63 methods. Although ACS had described its "unproven" list as containing

mostly unqualified practitioners and only a few researchers with appropriate degrees, Moss found that more than 70 percent held an M.D., Ph.D., or D.O. Further, more than 50 percent of the methods had never been investigated to prove whether or not they worked. Only 29 percent had received some investigation leading ACS to term them ineffective or "unproven" (Moss, 1989).

Perceiving these biases discourages alternative medicine researchers from attempting even preliminary studies. The discouraged investigator sees little point in proceeding if small, preliminary, information-gathering studies are so readily rejected. Likewise, large-scale, controlled trials demanded by those with such biases will likely not be funded.

Specific Testing Difficulties

The examples below serve to illustrate some of the controversy surrounding methodological decisionmaking for alternative medical systems.

Systematic therapeutic learning. Some alternative methods-such as biofeedback, meditation, imagery, and dance therapy-involve a learning process. With repetition, the person using them becomes more adept. Evaluating the benefit of such methods requires ensuring that a basic minimum of training is achieved by study subjects (i.e., the study must control for the amount of learning) and carefully selecting the appropriate research technique.

In studies of therapies involving learning, the research methodology usually includes appropriate control groups. Typically, the controls receive another intervention or none at all and would be students, patients, or clients of similar ages, talents, problems, interests, and whatever else is relevant to the process or technique under study. Many practitioners of alternative medicine consider the situation unethical if the control group receives no intervention. In addition, they note that subjects in the control group might be angry and frustrated (Goeble et al., 1993).