Page images
PDF
EPUB

been issued the CDC guidelines for the surveillance of nosocomial infections.

These steps are likely to improve the reliability of HCFA's generic quality screens over the next several years. Unlike the CDC personnel in the SENIC Project, however, PRO reviewers do not receive intensive training in the use of the guidelines, nor do they use diagnostic algorithms.

VALIDITY OF THE INDICATOR

Nosocomial (Hospital-Acquired)
Infections

Numerous studies link nosocomial infections to lengthened hospitalization, morbidity, and/or mortality (160,233,251,261,263,493,536,587,698). A prospective study of patients with indwelling bladder catheters in a teaching hospital, for example, found the development of urinary tract infections among these patients to be associated with a threefold increase in mortality (493). One analyst estimates that more than $2.8 billion in excess hospital charges are generated each year because of nosocomial infections (182). Because of the empirical association of nosocomial infections with adverse outcomes for patients, nosocomial infections have high face validity as an indicator of the quality of medical care.

Although the relationship between nosocomial infections and poor patient outcomes is well established, the link between inadequate/poor hospital care and the onset of infection is less clear. The fact that an infection is acquired in the hospital does not mean that it is caused by the hospital or by the poor quality of its practitioners. No available studies have examined or compared nosocomial infection rates in hospitals explicitly to examine the quality of providers. Numerous studies have published institutional nosocomial infection rates, however, as part of investigations of effective interventions, changes in rates over time, or the health and cost implications of hospital-acquired infections.

A review of the literature through 1975 identified 24 studies that published survey data on nosocomial infections in hospital populations

Moreover, the audit by physician advisors of cases that fail the screens is largely subjective. The SuperPRO has now started to analyze the reliability of PRO results for individual generic screen criteria. Depending on the findings of these analyses, further revisions of HCFA's generic quality screens may be necessary in the future.

(230). The prevalence of nosocomial infections in the hospital populations in these data ranged from 4.5 to 15.5 percent, and the incidence of such infections (infections per 100 discharges) varied from 3.1 to 14.1 percent. Community hospitals had lower reported nosocomial infection rates than referral, municipal, or chronic disease hospitals.

Comparisons of data from these studies tell little about the quality of care in the hospitals surveyed because, aside from measurement problems, the data are not adjusted for the hospitals' case mix or patients' severity of illness. Although most of the studies report nosocomial infections by site of infection, by service, and by procedure, the samples are too small to allow adequate stratification of the patient populations. Researchers attempting to calculate the impact of nosocomial infections on morbidity and costs usually compensate for confounding variables by matching infected patients with comparison subjects on as many attributes as possible. Although the results may be valid for the institution studied, it is very difficult to compare study results across institutions, even for seemingly similar subgroups of patients (e.g., all surgical patients or all patients with the same primary diagnosis). The authors of the literature review just mentioned attempted to compare the results of their matched subject study at Boston City Hospital with three other epidemiologic reports. Inconsistencies in results were attributed to possible further confounding variables among the patient populations (231).

The risk of acquiring a nosocomial infection is related to a number of factors in addition to the quality of providers. The likelihood of an infec

tion's occurring and its outcome depends more on patient susceptibilities than on the presence of the organism (49). Patients' underlying diseases, medical procedures, severity of illness at admission, hospital service, age, sex, race, and urgency of admission have all been found to be significant risk factors for nosocomial infection (96,232).

Understanding and adequately adjusting for such risk factors are critical to the use of nosocomial infections as a valid indicator of the quality of care. Moreover, the necessary adjustment factors for nosocomial infections may be different from those used to compare mortality statistics or other quality indicators. For example, one study, which compared urinary tract infections in small hospitals (under 75 beds) with infections in a large, teaching hospital, observed that the higher prevalence rate in the teaching hospital was due to the increased use of indwelling bladder catheters (95). With even a rudimentary understanding of case mix, it is not surprising that community hospitals have lower rates of nosocomial infections than teaching and municipal hospitals.

The SENIC Project provides valuable information, because the researchers attempted to control for patient risk and other intervening factors in their investigation of the efficacy of infectioncontrol programs. Using the large SENIC data base, the researchers determined estimates of the frequency of nosocomial infection by selected characteristics of patients (273). Hospital-related characteristics were controlled by using American Hospital Association survey data as proxies for changes in hospitals that could not be measured (272). And finally, differences in physicians' diagnostic practices (their propensities for ordering tests) were controlled by defining hospital-specific measures for use in analyses (272).

Because of confidentiality provisions, the SENIC Project data cannot be analyzed by hospital. Nevertheless, the research helps to validate nosocomial infection rates as quality indicators in several ways. First, the SENIC Project research

"Risks were significantly related to age, sex, service, duration of total and of preoperative hospitalization, presence of previous infection, types of underlying illnesses and operations, duration of surgery, and treatment with urinary catheters, continuous ventilatory support, or immunosuppressive medications (273).

ers have measured and quantified the patient risks and other variables that contribute to the outcome of hospital-acquired infection. This information could be used in further research to allow comparisons among hospital populations. The large, statistically valid data base developed in the SENIC Project could permit "norms" for nosocomial infection rates to be established by patient risk category."

Second, in concluding that one-third of all nosocomial infections could be prevented through surveillance and control programs, the SENIC Project demonstrates the potential of nosocomial infection rates to serve as an indicator of the quality of care across hospitals (272). One infectioncontrol program shown to be efficacious was the systematic feedback of surgical wound infection rates to the practicing surgeons. (In combination with an ongoing surveillance and control program, this program led to a 19-percent decrease in surgical wound infections.) Thus, changes in physicians' behavior, or the process of care, are associated with changes in nosocomial infection rates. Moreover, the extent to which hospitals establish and maintain effective infection control programs is an aspect of their quality of care.

There is no evidence that nosocomial infection rates are correlated to the general quality of health care institutions (external validity). In fact, there are well-defined inpatient groups who have very little risk of acquiring nosocomial infections, for example, pediatric, psychiatric, and rehabilitation patients (274). Nosocomial infections would not be valid indicators of the quality of care received by such patients.

Occurrence Screens

Most occurrence screens are based on the criteria established in the California Medical Insurance Feasibility Study, which reviewed a large sample of 1974 California hospital records. The study sought to identify potentially compensable events or medically caused patient disabilities. For

"CDC has another ongoing data collection system, the National Nosocomial Infections Surveillance System, that collects nosocomial infection rates from 85 volunteer hospitals. CDC is using these more recent data to develop risk indices by diagnosis-related groups and for surgical, critical care, and neonatal intensive care patients (305).

the purposes of this OTA assessment, the potentially compensable events identified in the study are synonymous with adverse events caused by poor-quality care. Investigators in the California study sampled hospital charts by service from a group of 23 hospitals stratified by size, ownership, and teaching status. Of the more than 20,000 charts reviewed by medical record auditors, approximately 50 percent failed the screens. The study investigators (all physicians) reviewed these records and concluded that 11 percent of those failing the screens constituted potentially compensable events (or 5.5 percent of all records reviewed) (432).

The California Medical Insurance Feasibility Study validated its 20 screening criteria as part of a controlled two-step screening and audit process for determining the incidence of potentially compensable events. It usefully identified potentially compensable events by medical specialty, location (e.g., 72 percent of the potentially compensable events occurred in the operating room), diagnosis and procedure, and by selected characteristics of patients. However, the study did not validate the screening criteria (by themselves) as quality indicators. In fact, on the basis of the published data, it is not possible to calculate the sensitivity or the specificity of the screening criteria in identifying either potentially compensable events or adverse events (potentially compensable events are a subset of adverse events that are medically caused). There is insufficient information about the patients' medical charts that passed the screens to determine these values. Moreover, of the records in the study that failed the screens, 81 percent were eliminated by the investigators because no medically or patient-caused disabilities were found upon further examination of the records. This high percentage indicates a substantial false-positive problem, whether the goal of the screens is identification of adverse events or identification of potentially compensable events. The two-step screening and audit process may be a valid and effective, yet very inefficient, method of identifying poor-quality care.

The California study did not examine the effectiveness of individual criteria in screening for potentially compensable events. Moreover, the determination by the physician investigators of

whether a potentially compensable event occurred was largely subjective (as is also true in the PRO program). The subjectivity of such assessments is a critical factor in the reliability of audit when more than just a few investigators are involved.

Research commissioned by New York State under recent medical malpractice reform legislation will update the results of the California study and help to ascertain the validity of occurrence screens. As part of a comprehensive study to find which patients suffered injuries in the course of their hospital treatment and which of these injuries were produced as a result of substandard treatment, the Harvard Medical Practice Study Group is reviewing the medical records of 30,000 patients hospitalized in New York in 1984. These records are being reviewed by medical record administrators using 17 screens derived from the 1974 California Medical Insurance Feasibility Study. The medical records that fail the screens are then subjected to further review by physicians to confirm the adverse event, to estimate the probability of causation, and finally to estimate the probability of negligence (283).

6

The results of the Harvard study commissioned by New York State could validate the relationship of the screening criteria (outcome measures) to poor-quality care (the process of medical management) if the data are directed to that purpose. The Harvard study may reaffirm the finding of the California study that occurrence screening as part of a two-step process involving screening and subsequent audit is a valid approach to quality assessment. The relationship of the screening criteria to the universe of adverse events or poorquality care in hospitals, however, will not be resolved adequately by the Harvard study. Because the medical records that do not fail the screens are not examined in depth, the true denominator number of adverse events remains unknown. The full-scale study began in mid-1987, and results are expected in early 1989.

"The California screens have been modified by the deletion of four criteria ("unplanned removal of an organ or part of an organ during an operative procedure," "wound infection on last full day prior to or day of discharge," "discharge with indwelling urinary catheter," and "parental analgesics last full day prior to discharge") and the addition of one criterion ("obstetric mishap or complication of abortion, labor, or delivery") (283).

The SuperPRO has evaluated the accuracy of HCFA's generic quality screens in finding quality problems. In a special study, the SuperPRO reviewed a sample of medical records from nine PROS for the period August 1986 through January 1987 (444). Just as the PROs do, the SuperPRO's nurse reviewers applied HCFA's generic screens and referred cases that failed to physician reviewers for determination of quality problems. In addition, the SuperPRO calculated how many false negatives the screening process yielded by sampling the records that had passed the generic screens. These records were re-reviewed by a physician to determine if there were quality problems.

The SuperPRO concluded that HCFA's generic screening process had a sensitivity (i.e., ability to identify cases with quality problems) of 49 percent and a specificity (i.e., ability to exclude cases without quality problems) of 73 percent (444). A sensitivity of less than 50 percent means the screening process was no better at detecting quality problems than chance. Because a small sample size (100 records) was used by the SuperPRO to determine the false negatives, the sensitivity finding may have some degree of error and may actually range between 37 and 70 percent. In any event, the SuperPRO researchers concluded the quality problems that were found through HCFA's generic screening process were more serious than the quality problems missed by the process.

The SuperPRO also evaluated individual screening criteria used in HCFA's generic screen, especially those criteria thought to be responsible for substantial numbers of false positives. The study recommended dropping several screening criteria (including one related to nosocomial infections) and modifying several others. HCFA's revisions of the generic quality screen for the 1988

90 PRO contract cycle were a response to these recommendations (see app. D).

The SuperPRO study is useful insofar as it relates to the validity and effectiveness of individual criteria, but it also has several shortcomings. The study's sample of Medicare cases, for example, is not a random sample; it is probably weighted toward problem cases. In addition to reviewing a mandatory random 3-percent sample of hospital discharges, PROS review cases based on a number of negotiated objectives. In selecting its re-review sample, the SuperPRO did not distinguish among the types of cases reviewed by the PROS. Moreover, the small sample size used in the special SuperPRO study does not permit reliable estimates of the validity of the screening process. The SuperPRO may undertake a larger analysis in the future.

[graphic][merged small]

FEASIBILITY OF USING THE Nosocomial (Hospital-Acquired) Infections

INDICATOR

The feasibility of obtaining nosocomial infection rates by standardizing data collection methods in all hospitals and maintaining reliability

over time is questionable (269,305). Relying on coded diagnoses from hospital discharge abstract systems would be an unreliable method of establishing infection rates across hospitals. At a minimum, thorough medical record review by trained personnel is essential for finding cases of

nosocomial infections. The PRO audit process involves such thorough chart review by nurse reviewers with followup by physician advisors.

An alternative to medical record review would be to establish new channels to obtain more reliable data. Currently, for example, all hospitals are required to have designated infection-control personnel and infection-control committees in order to be JCAHO accredited and to be eligible for Medicare and Medicaid reimbursement (424). Infection-control officers, usually nurse and sometimes physician epidemiologists, use ongoing surveillance techniques to find cases of nosocomial infections. If infection-control officers were required to use the standard definitions and guidelines provided by CDC, the data obtained by these personnel and utilized by the infection control committees could be channeled outside the institution for quality assessment purposes. CDC currently collects such data from approximately 85 volunteer hospitals in its National Nosocomial Infections Surveillance System (305).

Using rates only for selected sites of nosocomial infections, such as the bloodstream and surgical wounds, rather than combined rates of nosocomial infections at all sites, would minimize the measurement problem created by differing physician diagnostic practices. Bloodstream infections, which require only one verifying laboratory culture, have been suggested as one type of nosocomial infection for which reliable statistics could be gathered (305,699). Surgical wound infections do not require laboratory verification, although an impartial view of the wound in the operating room is necessary to determine the degree of contamination before and during the operation. Moreover, research has progressed furthest in understanding confounding patient risks for surgical wound infections. Data from the SENIC Project were analyzed using multiple logistic regression techniques (271). The researchers concluded that four risk factors predict a patient's probability of getting a surgical wound infection twice as well as the traditional classification of wound contamination alone: abdominal operation, operation lasting more than 2 hours, contaminated or dirty-infected operation, and three or more underlying diagnoses.

Occurrence Screens and
Incident Reporting

The use of occurrence screens and incident reporting by hospitals is widespread. The general availability of such systems was a primary reason for OTA's decision to study adverse events as a potential indicator of the quality of care. To the extent that occurrence screen and incident reporting systems are already in place, the additional costs of supplying information on adverse events to consumers could be minor as compared with costs of supplying information on other quality indicators. Moreover, poor patient outcomes are readily understandable by consumers and associated in the public mind with the quality of care.

Regulators are increasingly turning to occurrence screen and incident reporting systems to accomplish their goals in quality assurance. New York, and more recently Massachusetts, are collecting incident reports and, in turn, making selected information publicly available. Pennsylvania is implementing a statewide hospital discharge abstract system that includes information on the patient's severity of illness at admission and on several data elements normally considered occurrences or adverse outcomes. A primary purpose of Pennsylvania's data system is to inform the public about health care costs and quality. Several other States, including Colorado and Iowa, are pursuing approaches similar to Pennsylvania's. Thus, a number of State-level systems either already are, or soon will be, using statistics on adverse outcomes to inform consumers about the quality of hospitals.

On the national level, hospital-specific data generated by the PROS through the application of HCFA's generic quality screens are available to the public upon request to a PRO, subject only to hospital notification at least 30 days before disclosure (42 CFR 476.120,476.105 (1987)). Consumers can request information by hospital on screen failures, on quality problems identified during audit, or on both. As far as HCFA is aware, no such requests of PROS have been made to date (487). The Public Citizen Health Research Group contends that at least one PRO has refused to make similar types of outcome data available to

[ocr errors]
« PreviousContinue »