THE ENERGY MODELING FORUM AND MODEL ASSESSMENT: SUBSTITUTES OR COMPLEMENTS? John P. Weyanc Department of Operations Research Stanford University INTRODUCTION I have chosen quite a provocative title, I think, "The Energy Modeling Forum and Model Assessments: Substitutes or Complements?" First I have a confession to make. As is often the case with such titles, it is more provocative than it is an accurate summary description of what I am going to say. Indeed, I could pose the problem that I am going to address as to maximize the credibility and acceptability of energy policy models by adjusting the relative funding levels to model assessment and EMF-like activities within a fixed budget constraint. That I will not do. What I will do is to take a somewhat softer tack and talk in more general terms about the relative strengths and weaknesses and what I feel to be the appropriate roles of the two types of model analysis activities. In that regard, a more accurate title for my talk might be one that David Wood suggested to me recently, "The Energy Modeling Forum and Model Assessment: Where Do We Draw the Line?" I would like to say a little bit about my own personal perspective on this subject. First, it has been my privilege to have participated in the Energy Modeling Forum (EMF) activity from its inception, thanks to Bill Hogan and Jim Sweeney. Second, I have always had what I considered to be an implicit, obvious interest in model assessment activities and projects solely because I have felt that they were of great relevance to what we do at the EMF. In writing this paper, I have forced myself to make that relationship a bit more explicit. The first thing that I can think of in the literature chronologically that is relevant to this topic is the book by Greenberger, et al. entitled, "Models in the Policy Process" [1]. In Chapter 10 of that book, "Modeling and the Political Process," which is their conclusions and recommendations chapter, they point out that there indeed seems to be a problem with the actual use of policy models in general, and that there are lots of things that we can do to improve this situation. We could create better public education in modeling techniques; we could create more professional standards amongst the modelers; we could create more responsibility on the part of the decision makers in commissioning model studies; we could better define model assessment procedures and documentation standards; and finally, we could create bridges between energy model builders and model users. They astutely observe at that point, however, that if all of these courses of action were pursued simultaneously, none would get done very effectively. So they conclude that the model assessment activity is probably the highest payoff single activity. Indeed they imply, and I will say explicitly, that there is a lot of overlap between that particular course of action and the other four. The next piece of relevant literature is the paper Bill Hogan presented at the Lawrence Livermore conference on October 3rd of last year: "Energy Modeling: Building Understanding for Better Use" [2]. What I would like to recall from that talk are the three types of model evaluation activities he described: verification and validation for which - we are converging on standard definitions [3] -- and one Bill added to that, ventilation simply opening up the architecture and structure of the models to public scrutiny. We were also reminded in that talk that the purpose of modeling and analysis is not numbers, but insight. The third thing I would like to draw on is a lecture that David Wood gave at Stanford on November 9th of last year, entitled: "Model Assessments in the Policy Research Process" [4]; That lecture provoked both the title and substance of my comments. Finally, I have tried to draw on and in some cases integrate the comments and observations that I noted during the presentation of the other papers at the conference. Indeed, it turned out, in going through my notes, that almost everyone had something to say that was relevant to my topic. In the next section, I will quickly review the ongoing model assessment and EMF-like activities. Then I will recall a set of requisites for energy policy models that I will use to talk about the relative strengths and weaknesses of the different types of model evaluation and model analysis activities as currently practiced. Then I will move from that rather abstract discussion to a more operational accounting of what model assessment activities currently do that the EMF does not do, and conversely, what the EMF does that the model assessment activities don't do today. That leads quite naturally to an admission on my part that the dichotomy that I have created between assessment activities and the EMF is actually not a very precise one after all; there is a fair degree of overlap. There are many aspects of the EMF process, as now configured, that are really just different types of model assessment. Finally, I will talk a little bit about some alternative directions for the future for these two types of activities. ONGOING EMF AND MODEL ASSESSMENT ACTIVITIES I don't want to spend a lot of time reviewing the ongoing model assessment and EMF-like activities; especially since most of the other papers presented at the conference are focused on that task. Additionally, Greenberger [5] provides an insightful way of thinking about these two approaches to "model analysis." I am interested here only in model assessment (MA) activities that go all the way to the hands-on stage. There are several of those that we have heard about. The first one that comes to mind is the MIT Model Assessment Laboratory [6, 7, 8, 9]; the second is some of the work that they are doing on the Texas National Energy Modeling Project [10]; and the third is the model assessment work going on in DOE at the EIA [11]. The EMF-like activities, which I define as those not only doing standardized model comparisons, but also involving users in that comparison process including the Energy Modeling Forum [12], the Utility Modeling Forum [13], a serious desire at the Solar Energy Research Institute (SERI) to do a solar energy modeling forum, and an exercise carried out by the EIA in the preparation of the 1978 Administrator's Annual Report. USING MA AND EMF ACTIVITIES TO IDENTIFY ACCEPTABLE POLICY MODELS In his Stanford lecture David Wood first defined a policy model as one that can be used to address an issue where there are a lot of contention points, as opposed to a research model where we are just trying to learn about a system per se [4]. He then identified a short list of requisites for a policy model. I would like to use David's requisites as a framework for talking about what I feel are the comparative advantages of the three different types of model evaluation activities; verification, validation, and ventilation. Then, I will overlay on that my assessment of which of the two generic activities, that is EMF-like activities and model assessment activities, are most effective in accomplishing the three model evaluation functions. The first requisite for a good or acceptable policy model that Dave mentioned was that it must be based on good research results; that is, tried and tested principles, theory, and data. Here I think we are talking about what people have generically talked about as verification and validation [3]. Further, I think there was a hypothesis postulated by Dave Kresge yesterday [6] that the only was to do these things right is to do hands-on model assessment. Basically I agree with that, so I see in this first requisite that the way to determine whether an energy policy model is based on good research results is to do an in-depth model assessment. By that I mean hands-on, third-party independent review. The second thing that David thought was a requisite for a policy model is that it should include all the relevant policy options; It should have the correct slope, consider the things that are important in the policy debate, and include the relevant policy levers. To be able to deal with the policies that are actually being considered has been a problem in the past because many of the models really don't have the right policy levers. So when a debate comes up, which is usually quickly, it gets resolved very quickly too, and the model is not of much use. The model should also account for the important impacts of the alternative policies. The third requisite, which I would like to lump with the second, is that the model should include the appropriate inputs and outputs to enable one to analyze the alternative choices, the contention points under consideration. I personally feel that the kinds of activities that will best demonstrate and illuminate how the models stack up on the second and third requisites and help people evaluate the models in those regards, are basically ventilation-type activities. I think that is where the comparative advantage for ventilation activities are at this point in time. I perceive that there is a communication gap that exists right now in these areas that is important to mend before digging into the level of detail that one would go to in an in-depth, hands-on, third-party assessment. We should talk frequently in public about the policy levers and calculated impacts included in the various systems. This is why I believe that EMF-like activities have a comparative advantage in terms of increasing model acceptability and use through ventilation according to the second and third requisites at the present time. RELATIVE ROLES OF EMF AND MA ACTIVITIES: CURRENT PRACTICE In discussing the relative roles of EMF and model assessment activities some aspects of the current practice are useful reference points. In particular, it is important to identify the kinds of things that the model assessment activities do that the EMF does not, and vice versa. What do the model assessment activities do that the EMF doesn't? First, the EMF does not do comprehensive "overview assessments" like, say, the Model Assessment Laboratory [6]. We have done a pretty good job of telling the model users what the models as a set in each study are good for and, importantly, what they are not good for. But, we have not done enough about explaining to the model users what the relative strengths and weaknesses of the individual models are on a comparative basis, even strictly in terms of scope what is assumed exogenously and what is left endogenous to the model. And maybe we could add a little bit about how the endogenous part gets done without doing any validation. - The other thing that the Assessment Lab does that the Forum doesn't do is hands-on, third-party independent assessments. That policy was recommended at the workshop that help set up the Forum, and the model assessment project as well, in the summer of 1976 [14]. What does the EMF do that the model assessment projects don't at present? Number one, it involves model users directly in the modeling process, and that is a good way, I think, to accomplish ventilation. The forum process, which amounts to a very focused encounter group type of experience, accommodates the acculturation to models very expeditiously. And an important product of the Forum, and certainly a product in which the Forum has a comparative advantage over the MA activities, is alumni, an increasing community of, we hope, born again modelers and born again users. Another thing EMF does do that MA does not is standardized model comparisons. As I think we could gather from the several overviews of the Model Assessment Laboratory yesterday [6, 7, 8, 9], there are many people who feel that the assessment projects should do standardized model comparisons. Now, as Marty Baughman pointed out yesterday [8], there is in the present scheme an overview type comparison where the model that is being assessed is compared with published results from other similar models, and so forth. But, what I have in mind by comparative model assessments here is to actually run the same tests on more than one model. The benefit from doing this type of comparison is that you can identify differences in the explicit assumptions made a priori and the implications of those different assumptions ex post. The example that springs to my mind involves what is probably the model now most widely used by the policy and evaluation part of the Department of Energy [15]. During the course of the second EMF study, the representatives of that model observed that the projection of percentage electricity generation by coal from their model was different than everybody else's. They then ના |