Dr. Kresge: I think I would at least like to respond to it in passing, because it seems to me that the conclusion you draw from that, especially since in my other incarnations I am a modeler, it is very encouraging because it says, hey, don't assess me because I am inseparable from my model, and especially don't assess my model, because if you do you are doing it an injustice.

If we really believe what you say, then it seems to me that we either have to give up trying to assess anything, or you have to suggest an alternative procedure. Then on top of that, I don't see how that leads to a way of judging between competing models done by competent people that produce different results.

Dr. Greenberg: I just want to say one comment in response to that. I said I think it offers a limitation. I didn't say I think it renders it valueless.

Dr. Richels: Okay, I would like to add one more point. I have thought a lot about the problem of separating models from modelers, and I think in the last couple of years I have probably turned around 180 degrees. I question the feasibility of separating the model from the modeler for policy analysis. I don't question though the desirability of separating the model from the modeler for model assessment. And I think that the MIT group is intimately familiar with the Baughman-Joskow Model now, and one reason why we want to build in the participation of the modeler, Marty Baughman, in the assessment explicitly is to prevent us from going off in the wrong direction when indeed that happens. And, therefore, the differences are differences of opinion of modelers of analysis to particular aspects of the model and not whether the model is behaving in a particular manner.

Dr. Baughman: I agree with the remarks that Rich Richels just gave on this topic. I think that the question is a legitimate question. For example, in the in-depth assessment that MIT conducted, had I been present there were explanations for results--maybe not very good explanations, but explanations nonetheless--for why, when you jiggle those inputs, those outputs came out.

And so, to that extent, I think you are right that you can't separate the modeler as an analyst from the model. I think Rich gave the appropriate response. I think, as a result of this first activity, that it has been pretty well concluded that the modeler has to be included as part of this process, and so that is recognized in future activity.

Dr. Sweeney: In thinking about model assessment, I have decided that one of the most difficult things to catch is the implicit assumption, the difference in the world view between different modelers, or the world view incorporated in the model. Has the process that the MIT energy lab has gone through helped sort out some of these implicit assumptions? I will give an example from an energy model forum study that we went through. This is something that came up in a comparative study of a number of models.

The coal and transition study noted differences between geographic patterns among several of the models. It was traced back to the fact that an implicit assumption in one of the models was that there was a monopoly in the supply of coal.

I mean that was sort of implicitly in there and the modeler didn't understand that that was in the model. It was subsequently changed as part of the process.

Is there anything about the MIT energy assessment lab process that helps you get at that, or is it more in getting at sort of the guts of the model's explicit assumptions, the explicit coding issues?

Dr. Kresge: I think that there have been a couple of comments on that already. Marty made most of them, I believe. He was pointing out, that especially in the overview portion of the assessment, as opposed to the in-depth, comparative analysis inevitably is in there; you are using people who have substantial expertise in the field.

They are aware of other studies and those are brought into play. Rich made the point, that I think we all agree, that it would be very nice if the resources were available to do comparative analysis of key models within any given area. If we could afford to do comparative in-depth analysis of several coal models, that would be a very, very desirable thing, and hopefully that is something we are working toward.

Let me emphasize another point that is sort of peripherally related to yours, and Marty brought it up, and I would like to stress it. From the outset, we recognized that there were going to be points, like these differences in implicit assumptions, that would cause the assessors and the modelers to end up with irreconcilable differences. We would say that something was a limitation of the model and that the modeler would say was right. Το deal with that--at least at one level--we had in the initial experiment, and we have on a continuing basis, a very firm rule that in the final report there is a chapter there where the modeler has a chance to respond to the

assessment.

The report cannot be on an assessment basis only. There must be a chapter in there where the modeler can respond and say, look they have this world view. I have this other world view. We agree to disagree.

Apart from doing more than one model at a time, the only way I see the comparative assessment coming in is through the other studies that we are aware of. And, of course, it would be nice if there were more coordination with things like the forum.

Dr. Sweeney: Thank you. May I make a quick point to sum up what is at issue? When I say comparative model studies, I don't mean look at one study and say, well, that is pretty good because they include that, and this one doesn't include that. I mean standardizing the input to see how each of them behaves to a standardized set of inputs. That is what, I think, brought out the differences that we found.

Dr. Kresge: That is certainly what you would do, say, if you are doing side by side in-depth analysis. You would certainly do it by standardized experiments, so that that would automatically be in there.

Dr. Richels: I would like to add one point to what Dave said. I think he has brought up a very interesting feature. That is that by the time we get the final report at EPRI and we have had the final report there for a while and have been getting Marty's input all along, we do not take the opinion of the assessors as the gospel that factors into our decision when we are looking at a particular application of the model. In no way do we try to reach any kind of consensus between Marty Baughman and the assessment lab, nor do we encourage any consensus except over factual disputes. The kinds of inputs that arise through their disagreement is, in my opinion, the most valuable part of the process.

Dr. Manove (Boston University): I have been somewhat associated with MIT in their assessment. I have been sort of very disturbed by a few things I have heard and I just want to comment on them and also by something that I have done, which is the assessment project.

I have found myself in assessing the ICF model moving more and more toward verification and less and less toward validation, as you have defined it here. I think the reason that I have found myself doing this and my colleagues doing this is, this nonsense that we cannot know about the real world, and that all we can do is sort of give up and check each formula and see if it is consistent and check the data to see if anybody has made a mistake in adding up some numbers or compare this model with another one.

We can know about the real world and, if not, I think we should all quit and go home. What we have been saying here reminds me of people that say that our winning evolution is a theory, not a fact or people that might say that the assumption that the world was round, is somebody's model that explains the things we see. Sure, the round world is a model. Predicting an eclipse in 1992 is a model, but those are models that we know so well and we believe so closely that we call it a fact. We say that this is really true about the real world.

There are things that we do know about the world out there and what we ought to do in validating these models is sit down and figure out what do we really know. What do we really believe about the world. And in those few areas where we do know something, these models jive with what we know.

Now it will take some doing to figure out what we do know about the world but it can be done. One of the things that I think we do know, at least we believe strongly, is that the world is fairly continuous. That if you change a price by half of one percent, the quantities do not change by a factor of forty. That is why we do these activity analyses. That is one reason why you do it. You move something a little and if the model goes crazy, you say, hey, that must be wrong. We do know something about the world; we know this continuity thing.

For ex

There are things that we can test by ordinary empirical tests. ample, in the ICF model, they assume that the solution is going to be cost minimization for the whole country. Well, we can find out whether our costs are being minimized right now, by utility. We can do that kind of testing; we cannot find out for certain, but we can try to say something about the world.

I really think that there are some things that we know; there are some things that we know with a high degree of confidence and we ought to think about what we know and then we ought to really try to test validity, not just to compare things.

I think we ought to not be satisfied, either, with just testing validity of little pieces of the model; we have to test the validity of the whole model. If you test out each little piece as Hoff was describing, and sure you find out about transportation, but maybe the whole damn thing is wrong because your whole conception--the whole way you put it together-is wrong. You might be subject to some catastrophic error in your whole model. So, you have to do more than find out about a lot of little pieces; you have to ask big questions.

I think my experience with MIT has been not to ask the big questions. We have all been trying to push ourselves to ask big questions that we really believe this thing and does it really jive with what we know about the world. I hope that this kind of a "Gee, we cannot know anything about the world, anyway" attitude goes away.

That is my little speech.

Dr. Marcuse (Brookhaven): I assume that the reasons we do validations is really an attempt to improve models. Out of the validation process we hope to get improvements in the models, not weaken them.

I guess I would define improvement probably that the model supplied better information. Then the question is, what is better? I suppose the answer to better would be, is there some way it now gets used in the decision process in a way that it has an impact on making the decision. Hopefully, a better one.

The question I have is in what way have any of the models that they have assessed up to now been improved as a result of the validation?

Dr. Baughman: I want to respond, I guess, in a couple of ways to the comments and questions that you have made.

First of all, it strikes me that a model, I still agree, cannot be proven whether it is valid or invalid. A valid model, I think by definition, however, is one that is devoid of contention points, another word that has been used here today. And that if the model has no contention points, then it is pretty well accepted that the behavior of the model conforms to how everybody believes the real world behaves.

I still believe that whether or not it is valid still cannot be answered. F equals ma looked very good for a long, long time until something better came along, but it was a perfectly valid model for most of the applications to which it was put before relativity came along.

In terms of response to the question, how has the model been improved, there were several errors in programming that were brought to our attention as a result of efforts to reprogram the model. These have been corrected. I think that if I may paraphrase and briefly point out what I think MIT's report said, that if you look at the three basic components of the regionalized electricity model, in terms of their relative quality, probably the most interesting part of the model was the financial regulatory component; the part that was kind of so-so was demand; but, probably the weakest part in a comparative sense was the supply sub-model. That is useful input. I think they probably knew that, but many times that information goes subliminal and you do not explicitly discuss these things, especially with those who might be potential users of the model.

As a result of that, efforts have been made to reprogram much of the supply portion of the model. A lot of that is completed; some of it is still to be completed before some additional applications of the model are being made. We have and are responding to suggestions that were made in the

assessment.

Dr. Richels: I think that Marty has responded to one side of the coin and that we are looking for feedback from the modelers and, hopefully, an improved model or more useful model. The other side of the assessment process, though, is to aid the model users in the intelligent use of the model. I can think of several instances at EPRI over the last several months where we have had discussion involving the use of the BaughmanJoskow model. I can think of a few instances where the model assessment was quite helpful in assisting us in determining how we were going to use the model.

Dr. Greenberger (Johns Hopkins University): I just wanted to clarify this term contention point. At least, as it was originally intended.

It is not something that you want to eliminate in models, it is really something you look for in models. In particular, you look for points in models where there is an ambiguity, a possibility for different assumptions which corresponds to an area of disagreement in the policy field that you are studying so that the model gives you some way of exploring the different assumptions possible in tracing back to basic differences in points of view.