Results in Brief regulated under the Clean Air Act; risk assessment work was not evaluated because none was conducted specifically for developing this standard. GAO evaluated the extent to which accepted scientific and administrative practices were employed; GAO used criteria it developed from a search of the risk analysis literature. The final regulatory decision itself was not evaluated, however. The main strength of this study is the comprehensiveness of the examination of each case. The study's main limitation, which derives from its being a pilot, is the small number of cases. Because GAO examined only one case at each agency, it is not possible to generalize about the quality of the risk analysis work conducted by the agencies or the federal risk analysis process as a whole. The quality of the risk analysis work varied, fewer problems occurring in the risk assessment phases than in the risk management phases. The risk assessment work was generally adequate. FDA and OSHA were able to meet basic expectations for sound scientific performance in identifying and estimating the risks posed by specific hazards. However, the use of unverified assumptions to fill largely unavoidable gaps in the data resulted in great uncertainty about the magnitude of the risks. The risk management work in all three cases exhibited serious problems. The integration of policy options and risk assessment results was poorly documented, and the basis for regulatory decisions was unclear. Followup evaluations of the regulatory actions were not generally performed, largely because of cost or technical infeasibility. This means that the agencies cannot determine the risk reduction, if any, that is achieved by regulation. Principal Findings Risk Assessment The risk assessment work GAO evaluated generally met the technical and scientific criteria developed from the literature. The identification, characterization, and estimation of risk were well conducted by FDA and Executive Summary OSHA. They did a credible job of reviewing and evaluating the available evidence on a hazard. (See pages 22-31 and 44-51.) GAO did find some problems with risk assessment that were primarily Risk Management GAO found significant problems in the risk management work. In the FDA and EPA cases, the development and evaluation of risk management options and the actual decisionmaking process were poorly documented, so that it was difficult to determine the information the regulatory decisions had been based on. It was not always clear how regulatory options had been developed, how full the consideration of options had been, or how uncertainties associated with the options had been presented to decisionmakers. In the EPA case GAO examined, and possibly in the FDA case, the precision of estimates may not have been sufficient for choosing a final regulatory option from alternatives. (See pages 31-35, 51-55, and 65-77.) The extent and quality of the guidelines for risk management varied greatly, between and within the agencies. EPA had extensive guidelines for the development and evaluation of regulatory options, but it had no guidelines for decisionmaking. In contrast, OSHA had strong guidelines for decisionmaking but no guidelines for developing and evaluating risk management options. FDA had no guidelines for either options development or decisionmaking. (See pages 32-34, 52-54, 66, and 73-76.) None of the agencies had conducted follow-up evaluations of the regulations to determine if they were achieving the intended effects in reducing risk. An evaluation effort has been planned at EPA; an evaluation is not technically feasible for the type of case examined at FDA; and cost and other considerations prevent OSHA from planning such an effort. Thus, the efforts devoted to risk analysis on these three cases have resulted in regulations that are only presumed to be reducing adverse Recommendations Agency Comments effects on health. No knowledge of the actual health effects of these regulations is currently available. (See pages 35-36, 55, and 78.) GAO is making no recommendations because this report covers a pilot effort in which too few risk analysis cases were examined to permit generalization. The Department of Health and Human Services and the Department of Labor indicated general support for GAO's effort to evaluate the risk analysis process and help improve regulatory decisionmaking. However, all three agencies expressed a number of general concerns. Some comments are critical of the cases GAO Selected and suggest that the report holds the agencies accountable for the quality of scientific research that is beyond their control. Other comments assert that the report confuses the quality of the scientific work with the administrative process and that GAO's criteria are not universally applicable or valid. After carefully reviewing these issues, GAO believes these criticisms reflect a lack of understanding of GAO's intent in performing this study. Further, prior to the start of the evaluation, each agency reviewed the list from which its case was selected and was asked to raise objections to cases that it thought were inappropriate. No objections to the cases were raised. GAO agrees that no single case can represent an entire agency's risk analysis efforts, and the report does not make such representations. Moreover, while GAO does point out flaws in the scientific research the agencies relied on in their risk analyses, GAO does not hold the agencies accountable for these flaws. GAO disagrees that the report confuses the administrative process with the quality of the scientific work; on the contrary, the report purposely links the two dimensions. Process questions are important because without them, the quality of the research remains unknown; the documentary record is the prescribed means for demonstrating the quality of the research. Finally, GAO does not argue that the criteria it developed are universally applicable or that they exhaust the characteristics required for adequacy. Rather, they are appropriate for the cases examined and for other similar cases. These agencies' comments are discussed in detail in chapter 5 and |