[16] Kuh, E. and R. E. Welsch, "Energy and Econometric Models and Their Assessment for Policy: Some New Diagnostics Applied To Translog Energy Demand in Forecasting," Workshop Proceedings. [17] Mayer, L., "On a Perspective for Energy Model Validation," Workshop Proceedings. [18] Murphy, F. and H.J. Greenberg, "Validity as a Composite Measure of Goodness," Workshop Proceedings. [19] National Bureau of Standards, "Guidelines for Documentation of Computer Programs and Automated Data Systems, " FIPS 38, Washington, D.C., February 1976. [20] Nissen, D.H., "Impacts of Assessment on the Modeling Process," Workshop Proceedings. [21] Parikh, S.C.; W. Marcuse, T. Sparrow and D. Pilati, "Appropriate Assessment" and "Validation Issues, "1 Workshop Proceedings. [22] Richels, R.; and David Kresge, "Third Party Model Assessment" [23] Rubin, L., and F. Hopkins, "Validating the Hirst Residential Energy Use/Mid-Range Energy Forecasting System Interface," Workshop Proceedings. [24] Schweppe, F. and J. Gruhl, "Systematic Sensitivity Analysis Using Describing Functions Models, Workshop Proceedings. [25] Shaw, M., "Model Access and Documentation," Workshop Proceedings. [26] Stauffer, Jr., C.H., "Developing, Improving and Assessing the ICF's Coal and Electric Utilities Model," Workshop Proceedings. [27] Sweeney, J., "The Energy Modeling Forum," Workshop Proceedings. [28] U.S. Department of Energy, Energy Information Administration Annual Report to Congress 1978, Volume I, U.S. DOE, Washington, D.C., 1979. [29] Annual Report to Congress, Volume II (1977) Projections of [30] U.S. General Accounting Office, Guidelines for Model Evaluation (Exposure Draft), PAD-79-17, U.S. GAO, Washington, D.C., January 1979. [31] [32] An Evaluation of the Use of the Transfer Income Model Trim -- To Analyze Welfore Programs, PAD-28-14 U.S. GAO, Washington, D.C., November 25, 1977. Review of the 1974 Project Independence Evaluation System, OPA-76-20, U.S. GAO, Washington, D.C., April 21, 1976. [33] Weyant, J., "The Energy Modeling Forum and Model Assessments: Substitutes or Compliments," Workshop Proceedings. [34] Wood, D.O., "Model Assessment and the Policy Research Process: Current Practice and Future Promise," Workshop Proceedings. Model Assessment and the Policy Research Process: David 0. Wood M.I.T. Energy Laboratory and Sloan School of Management Introduction The rapid increase in the development and application of large-scale energy policy models since the 1973-74 OPEC oil embargo is unprecedented in the policy sciences. While other public policy areas, such as urban planning and water resources planning, have stimulated intensive modeling and model application efforts, energy policy modeling seems more visible and to have stimulated both the enthusiasm and concerns of broader constituencies. Visibility of energy policy modeling seems due both to the pervasiveness of energy in society and the perceived urgency of energy issues, and to active programs in government industry, foundations, and universities to develop and apply policy models in well publicized studies.* Such studies published in a form highlighting the role of energy policy models focus attention on models, sometimes at the expense of the analysis itself. The early identification of the FEA Project Independence Evaluation System (PIES) as the "pocket pistol of the President" gives some flavor of one type of concern about the role of models in the policy process. While government and quasi-government model-based policy studies have contributed significantly to model visibility, these applications have taken place in a broader context of scientific research and analysis of energy production and use. Prior to the embargo the NSF Research and Analysis for National Needs (RANN) program was sponsoring many energy related research projects, projects which were greatly stimulated by the *Examples would include the Project Independence Report [10]; the ERDA studies of National Research and Development [37]; the National Energy Out look-76 report [35]; the EIA Annual Administration Reports [17]; and the ERDA sponsored NAS study on Nuclear and Alternative Energy Systems; the Ford Foundation Energy Policy Project [60], and the report of the Nuclear Energy Policy Study Group [59]; Baughman and Joskow's analysis of the future of the U.S. nuclear industry [52]; Hudson and Jorgenson's analysis of the likely macro-economic and energy sector effects of alternative energy tax policies [57]; and the MIT Policy Study Group study of conditions for energy self-sufficiency [58]. focussing power of the embargo.* Subsequent to the embargo these modeling research activities were expanded with sponsorship from the DOE predecessor agencies, and from EPRI, various foundations, and universities. In addition to modeling research the policy interests of government and industry have stimulated the formation of commercial firms oriented toward providing model-based support for energy policy evaluation and analysis. Although generalizations are dangerous, as a rule these firms tended to organize and apply academic research results in modeling efforts and studies aimed at particular client groups and/or policy evaluation and analysis issues.** This large investment in energy research and modeling has been based upon, and has stimulated, confidence that policy models can make a major contribution in energy policy evaluation and analysis. That the essence of a policy issue is that differences in policy turn on value conflicts between two or more constituencies in resolving factual or analytical disputes and/or in interpretation of expected consequences of implementing particular policies is generally recognized; but even when value conflicts dominate, model proponents argued that systematic analysis and presentation of the results of alternative policies helps to make clear the nature and extent of the value conflict. But while expectations are high, the actual success of model-based policy evaluation and analysis has not yet been widely demonstrated and accepted. The sources of disappointment are not readily classified, but seem mostly related to perceived failures in the models themselves and in the policy modeling process. Caricaturing somewhat; analysts tend to find models unfocussed and lacking detail for the specific issues of *Much of the pre-embargo modeling effort is described and/or referenced in Macrakis [55]. **There are many examples. Firms represented at this conference which have been particularly prominent would include ICF, Inc., Decision Focus, Inc., Data Resources, Inc., and Dale Jorgenson Associates. interest, and difficult to "reconfigure" in a timely manner; further, assurances about the scientific validity of energy policy models have not been satisfactory. In contrast modelers are frustrated by the elusive and changing nature of the issues as posed by policy analysts, and sometimes even suspect the rationality of the policy process. Decision makers who rely on analysts as well as their various constituencies for inputs to policy making are confused and alarmed by conflicting analyses and are led to suspect the integrity of the modeling and analysis process. Finally the various constituencies potentially affected by model-based policy analysis seem suspicious that the modeling process may be indirectly resolving disputed factual, analytical, and value conflicts in a "blackbox" environment, especially when the model results are inconsistent with their policy positions. Important initiatives to deal with these concerns include efforts to ensure congressional oversight of government energy modeling and data development, and government and industry efforts to better organize model-based policy research and to ensure scientific review and analysis of policy models, with results communicated in a form accessible to all groups interested in energy policy research. The following remarks focus upon recent and current activities related to such policy model analysis, including a survey of the activities of government, industry, and universities relating to policy model evaluation and analysis, and some speculations about the future of these activities. |