most modeling methodologies. When compared with all other related models, a perceived weakness in REM may turn out to be a comparative advantage. An assessment report which focuses on the relative strengths and weakness of competing models offers more to potential users than is possible through individual assessment. Relations Between the Model Assessment Group, the Modelers, and the Model Assessment Sponsor Once the objectivity of the assessment group has been established, a protocol must be developed to allow for the interactions between the assessment and modeler groups. For the REM assessment, a detailed statement of work between EPRI and the MIT model assessment group laid out the schedule of activities, deliverables, and financial resources to be devoted to the project. Although it was realized that the cooperation of the modelers was essential to the assessment process, no formal contract or statement of understanding existed between the sponsor and the modelers. This resulted in the modelers spending significant unreimbursed resources in time and materials to participate in project review meetings, to review and comment on draft materials and to prepare formal comments on the REM assessment. It is now clear that in future assessments, arrangements should be made between the modelers and sponsors regarding the terms and conditions for modeler participation. Conclusions and Recommendations The project is a major step toward developing effective procedures for the independent evaluation of energy models. Independent model assessment is a critical element in making models more accessible and useful in the areas of technology assessment and policy analysis. Such "third-party" assess ments identify the weak points of a model's theoretical structure, empirical techniques, and implementation procedures. This information can then be used to employ the existing model most fruitfully and to develop future models. Rather than being an adversary proceeding, independent assessment is most effective when undertaken as a cooperative venture with support from and dialogue with the modeler. Independent assessment can increase confidence not only in a particular model but also in the credibility of the model developer, who has an important expert role to play in technology assessment and policy analysis. During its first year, MIT's Model Assessment Lab undertook overview assessments of the Baughman-Joskow Regionalized Electricity Model (REM) and the Wharton Macroeconomic Energy Model, and an indepth assessment of REM. The research has been successful and has played an important role in shaping EPRI's use of these models. Third-party verification and assessment has enabled EPRI to understand better the strengths and deficiencies of the models and to improve their use in planning research and making actual studies. At the present time, an indepth assessment is being made of the ICF Coal and Electric Utilities Model. This model is important to the electric utility industry because it is being used by the Department of Energy and the Environmental Protection Agency to assess the effects of alternative new source performance standards for coal-burning power plants. The prototype venture at MIT has proven the merit of the idea of a model assessment facility and justified the plan to institute such an operation on a continuing long term basis. Footnotes 1. Massachusetts Institute of Technology, Energy Laboratory. 1978. Independent Assessment of Energy Policy Models: Two Case Stu- Cambridge: MIT. 2. Systems Control, Inc. 1978. Applicability of Brookhaven National Institute. 3. Greenberger, M., Crenson, M. A., Crissey, B. L. 1976. Models in the Policy Process. New York: Russell Sage Foundation. 4. The discussion of the REM assessment is abstracted from Greenberger, M., and Richels, R. 1979. Assessing Energy Policy Models: Current State and Future Directions. Annual Review of Energy. 4. 5. Joskow, P. L., Baughman, M. L. 1976. The Future of the U. S. Electric Utility Industry. The Bell Journal of Economics. 7: 3-32. An Approach to Independent Model Assessment David T. Kresge* The M.I.T. Energy Laboratory has recently completed a study, funded by EPRI, dealing with the independent assessment of models used for energy policy analysis. The principal objectives of the project were to: - Provide assessments of two important energy systems models, the Baughman-Joskow Regionalized Electricity Model and the Wharton Annual Energy Model; Analyze these case studies to identify key organizational - Develop a framework for better understanding the general This paper draws on the experience gained through that assessment project to (1) present a general outline of the approaches to model assessment; (2) report on some of the specific lessons learned; and (3) make some suggestions for improvements in future assessment activities. 1. Approaches to Model Assessment: A Working Hypothesis The framework for energy model assessment which we propose as a working (and we hope workable) hypothesis contains four principal elements: (1) review of literature; (2) overview model assessment; (3) independent audit, and (4) in-depth assessment. Although these elements represent four distinct approaches to model assessment, they are most appropriately viewed as the stages in a comprehensive model assessment process. The approaches are interactive and complementary, and should not be viewed as mutually exclusive alternatives. A summary of the content and relationships among the approaches to energy model assessment is given in Figure 1. The assessment process must begin, of course, with an operational version of the energy model to be assessed. For a reasonably mature model, the available documentation should include at a minimum: a concise statement of the model's conceptual structure; a description of the procedures by which the model was empirically implemented (including a discussion of the underlying data bases); and a discussion of the results obtained when the model was applied to an analysis of the policy issues for which it was designed. * David T. Kresge, Associate Director, Joint Center for Urban Studies of the Massachusetts Institute of Technology and Harvard University, Cambridge, Massachusetts |