Now, if you are studying a particular policy area with a particular model, you would like your model to reflect the actual contention points. If the model, instead, essentially resolves those contention points; in other words, makes an assumption which fixes the way the dispute in an actual case is represented, then at least for exploring that difference of opinion it is useless. An additional concept that was used along with contention point, although I have not heard it referred to today, is the notion of a contention point being critical as opposed to noncritical. What that was meant to signify was if changing the assumption in the model alters the policy consequences, in other words leads to one policy action versus another, then that contention point is critical in the model. The contention point is not critical if it really does not matter which of two possible assumptions you make. Just to give you an example, in the model which was being assessed for which the concepts were originally devised, a contention point was whether or not the oil and gas industry was competitive. And that turned out to be a very critical contention point, although the assumption in the model was that it was competitive. Essentially, the contention had been resolved, so, therefore, unless you had a model which could explore a competitive industry versus a non-competitive industry, you could not really get at the root of the disagreement. In this case, the policy area was deregulation of natural gas prices. Dr. Stauffer (ICF): Marty, do you think, on balance, the process was a net benefit: a) to you; b) to the rest of the world? Then, was And finally, could it have been done more efficiently? it worth the cost? Dr. Baughman: Well, I am not sure that the book is closed at the present time, but my perception is at this point that, yes, it was a benefit to me in terms of organizing research priorities and even suggesting some new research areas. Since that happens to be one of the things that I pursue in my professional endeavors, that is worthwhile. To the other questions, I would have to let others judge. I feel that from what I have seen and the assessment report the MIT group has put together, that much of the mystery of the model, if there was any before, had to be removed as a result of this activity. I would like to see the activity continued. THE TEXAS NATIONAL ENERGY MODELING PROJECT: Milton L. Hollowayl INTRODUCTION Large-scale models are a product of our times and seem to be growing in the importance of their use in government policy work. Policymakers are using large-scale models and really have no alternative due to the complexity of potential impacts from policy decisions but at the same time they are sceptical of the reliability of forecasts and calculations from models. There is also great acceptance by the populace at large of results coming from computer analyses which in their minds seem to represent the epitome of technological solutions to problems. Analyses are somehow seen as more believable if they are based on computer technology. The major task that lies before us is to improve the usefulness of models and the judgments of professionals involved in public policy analysis. The central issue is the procedures by which the reliability of large-scale models can be established and made transparent--to distinguish between the influence of professional, subjective judgments and the influence of objective information that is reproducible by others. The Texas National Energy Modeling Project (TNEMP) has made some contribution to the goal of increased model credibility by transferring and operating the Midrange Energy Forecasting System. PROJECT PURPOSE The first purpose of the Texas National Energy Modeling Project (TNEMP) is to provide an independent evaluation of the Energy Information Administration's Midrange Energy Forecasting System (formerly known as PIES). evaluation will provide guidance to users of MREFS concerning the level of confidence one may have in the results of the models for government energy policy analysis purposes. The evaluation is critical, in the best sense of the term as used in scientific work, but also makes helpful suggestions for improvement in the model structure and in procedures used by the Energy Information Administration (EIA) for increasing model credibility. The second purpose of TNEMP is to provide recommendations to the Texas Energy Advisory Council concerning the maintenance of a national modeling system by the Council for purposes of evaluating Texas impacts within a consistent national modeling framework. As a result of the exercise, we have first-hand experience with MREFS, as well as DRI's Macro and Energy models. are able to compare these models, their structure and results with various Texas models at the Texas Energy Advisory Council, the University of Houston and the University of Texas for purposes of assessing their relative usefulness We 1 Dr. Holloway is Executive Director, Texas Energy Advisory Council, Austin, Texas, who serves as Project Director for the Texas National Energy Modeling Project. for energy policy analyses and possible joint use or integration. TNEMP participants have experience in still other modeling efforts, as well as experience in model evaluation, institutional arrangements for housing models and procedures for making the best use of resources to achieve successful model development, use and credibility. TNEMP is well suited for the second purpose. The evaluation of Texas impacts of national energy policy decisions must necessarily be done within a consistent national framework since a major portion of the nation's energy production and processing as well as corporate management of these for the nation as a whole is in Texas. A significant fraction of the nation's energy is also consumed in Texas, especially natural Texas based energy analyses must, therefore, be national to properly reflect Texas' interests and its role in the nation's energy future. order to be of use in the national energy policy process Texas based analyses must be centered in a credible national modeling and analysis framework. In A third purpose of TNEMP is to raise the level of attention concerning the current uses, practices, potential as well as the current abuses, and potential dangers of modeling for purposes of developing public policy and clarifying important issues to the citizens of a democratic society. increased reliance of policymakers, high level advisors, and the voting public on expert opinion, based in part on large-scale data bases and models, in an era of complex problems and policy prescriptions, requires that policymakers and high level experts gain a better understanding of current modeling reliability and practices. TNEMP is intended to help achieve this goal. Two additional factors are important side effects of the TNEMP exercise. First, we have advanced by some degree the art of model evaluation by achieving, among other things, the transfer and operation of a large and complex model. Hopefully others will benefit from both our successes and our mistakes for model evaluation is far from being a well-developed discipline. Second, we hope that some insight has been gained into what kind of institutional setting is appropriate for third party model evaluation efforts. The current institutional arrangements for the building, operation and maintenance of large models are critically lacking in efforts aimed at validation of relations, assumptions, data verification and model documentation. The quest is for the development of model evaluation institutions and/or incentives to bring about a timely devotion of model developers to validation and verification and to the provision of clarity and workability as model attributes. Third party evaluation can increase the probability of a model being accepted. It can enhance the transfer of science, technology, and statistical data to the policy decision process. ORGANIZATIONAL STRUCTURE FOR THE STUDY The organizational structure for the study consisted of four primary groups. First, the Texas Energy Advisory Council Executive Director provided project direction and the staff provided coordination, materials and other support. Second, the National Advisory Board provided advice on procedures, suggestions on methodology, evaluation of the Analysis Team results and general recommendations for a Texas national modeling capability. Third, the Analysis Team provided an evaluation of MREFS, made recommendations for improvements and alternatives, and made specific recommendations for Texas maintenance of a national modeling capability reflecting the impact on or by Texas. Fourth, the Supporting Institutions provided support by endorsing the objectives of the study, making review and comment, making data and facilities available, and funding the study. Figure 1 illustrates the working relationship of the four groups. Meetings were held periodically for the purpose of reporting progress of various studies to the National Advisory Board, other members of the Analysis Team, and the Supporting Institutions. This format provided opportunity for refinement of project objectives, definition of evaluation criteria, sharing of reference material, identification of weaknesses in the project study design, interaction of project participants with DOE personnel and overall guidance for the project director and individual principal investigators. In order to provide credibility of the project objectives and procedures, the National Advisory Board was asked to write an evaluation of the study to be published with the final report and their statement is included at the end of Volume I of the TNEMP report. In order to encourage maximum intergovernmental cooperation, EIA was offered the opportunity to comment on the project with the assurance that the remarks would also be published with the final report. EIA comments are also included at the end of Volume I. To achieve the objective of familiarizing key decision makers with the important findings of the project a meeting is now being arranged between members of the National Advisory Board, officials of EIA and the Department of Energy, and the Lieutenant Governor of Texas, who serves as Chairman of the Texas Energy Advisory Council. EVALUATION CRITERIA The reliability of information from the EIA large-scale energy model known as MREFS, currently being used for forecasting, policy evaluation and policy analysis, is our primary concern in this study. The determination of reliability is a difficult task; much has been written recently on the topic but not much is well defined and clarified at this time. We have made certain interpretations concerning the process and measures to be used in assessing reliability. In the usual understanding of reliability in scientific areas one would expect modeling results to be unambiguous, reproducible, and transferable. Such characteristics need to take on special definitions however, since we are dealing with a system involving the behavior of people (the economy with emphasis on the energy sector) and are making applications of models to describe alternative worlds, some of which will never exist for testing purposes. We also need to be specific with respect to the user of the information and in our case this will involve both other modelers and professionals as well as policy decisions makers and the public at large. It is also essential to distinguish uses of the models as between raising issues versus resolving issues. Models used for raising issues may rely more on hypothesis for their formulation whereas models used and designed for resolving issues will necessarily have to be based on accepted theory and/or laws. MREFS of the Energy Information Administration (EIA) is used primarily for the purpose of resolving energy issues and their use is for policymakers and the public at large. For that reason the evaluation criteria we have selected for TNEMP take on specific definitions. |