Dr. Ball: I wasn't talking in terms of credibility. Validity, in the sense that we are using it here, has to do with the concept of comparing things against reality. Validation implies, in the sense that we're

using it, that you are comparing the output directly against observed reality and seeing whether it's correct. Validity is often used in other senses as well, including your sense of credibility. But, I don't think we're talking about it in that sense.

I would suggest--I don't know whether this is what you're driving at-but if you used this multi-model or component concept that I was talking about, it would help you to maintain the credibility aspect purely by making what you do more open and transparent. Mainly, if you keep the parts of the model that are judgmental, the parts of the input assumptions that are judgmental, clear of the other mechanistic parts of the model and lay them out for people as clearly as possible, that's the only way I know of to be credible in that way. You'd simply have to open it up and let people take their choice. And if you can make it clear enough so that people can see what you're doing, they will have to make their own decision. Then your own credibility is not so much questioned in the process.

Mr. Woods:

I guess the thing is that there has been a tremendous amount of discussion. I'm looking at the calendar for the future, both today and tomorrow on whether there seems to be a focus on how do we mechanically go about back-fitting, etc., etc.

I get the feeling that credibility is the most critical thing, and the question of the validity--whether or not it meets all the mechanistic things--is sort of secondary. So the first thing that you must do when you set up the system is to establish credibility rather than this mechanistic validation process.

Dr. Ball:

I wouldn't maintain that. No. I would maintain that credibility is a meaningless thing, particularly in Government, in that you can't even attempt--. Openness is the only possible way that you

can deal with that part--.

Dr. House:

If you'd like to use the word "validation," that's what I really meant by "veracity" or "verisimilitude." For scientific validation, and I think that is what we were talking about here, that's a necessary but not a sufficient condition. Almost by definition, you'd have to have that in order to get comparative analysis to say that it turned out that way.

I guess the major split, and this is probably a good way to split it as almost any other, models at least should pass the first test. I'm absolutely certain that a large number of the models that I have looked at over the many years don't pass that first test. I mean, they don't do what they say they do and they don't do it except under the most rigorous conditions. I've seen more model outputs produced by a typewriter than I have by a computer. And, so, that's at least a necessary condition. Now the second condition of comparative analysis is some sort of objection to truth. I'm just saying that there are places where that falls apart.

Now, Dick and I have talked about this attempt to partition the types of models so that they might handle that. Some models fall very easily into it and others don't and maybe what you'd like to do is separate them and the like. But there are just some that can't fall into the second case because we just don't know enough to put them there. And, by the way, the classification we had out there, some type of models weren't designed to do that. I think your optimization work may be one. I quit. Thank you.

THIRD PARTY MODEL ASSESSMENT: A SPONSOR'S PERSPECTIVE

INTRODUCTION

Richard Richels

Electric Power Research Institute

The electric power industry has long been a sophisticated builder and user of models in planning capacity expansion, and in scheduling existing generation capacity to satisfy customers' energy demands at minimum cost. More recently the industry has provided support for developing large-scale models that encompass the interactions between the electricity sector of the economy and the rest of the energy-economic system. In addition, there are a number of important models which, although not used directly by the electric power industry, play a role in determining public policies that affect the industry. Sponsors of these more general models include private foundations, the National Science Foundation and government agencies.

lows:

A sampling of models relevant to the electric utility industry is as folThe Baughman-Joskow Regionalized Electricity Model, the Wharton Macroeconomic Energy Model, the Hudson-Jorgenson Macroeconomic Energy Model, the Gulf-SRI Energy System Model, the Brookhaven Energy System Optimization Model, the FEA Project Independence Evaluation System (PIES), the ETA-MACRO Model and the ICF Coal and Electric Utilities Model. Each of these

models includes an explicit representation of the electric power sector and, to varying degrees, all are being used in technology assessment and/or policy analysis relevant to the electric power industry. It is important for the electric utility industry to be certain that such models accurately represent the "real" world. This is the basic rationale behind the Electric Power Research Institute sponsoring the Model Verification and Assessment Project (RP 1015) at the MIT Energy Laboratory (1).

The model verification and assessment project was initiated on a trial basis

to test the practicality and usefulness of third-party model analysis. The Model Assessment Laboratory has three objectives.

It is intended to:

1. provide model users with evaluative information and understanding essential for the intelligent use of models;

2.

give model builders feedback signals helpful in correcting and improving the models; and

3. promote the development of state-of-the-art model assessment.

These three objectives are considered central to the health and usefulness of the energy modeling field and to the development of an infrastructure of supporting services and criticism.

In the past EPRI has turned to individual investigators for independent assessments of selected energy models. For example, in an analysis of the Brookhaven Models, the assessors were asked to evaluate the electric utility sector of the models and then to modify, extend and refine the models in relation to the existing "state-of-the-art" (2). Such assessments are typically "one-shot" efforts with little thought to developing a set of assessment criteria or establishing an assessment methodology which could form the foundation of future assessments. By consolidating the assessment function under a single organization, it is hoped that "life" and "continuity" can be brought to the assessment process and that the understanding and insights gained from past assessments could benefit future work.

Composition of the Assessment Laboratory

The model assessment facility employs two types of researchers, those that

form the infrastructure of the laboratory and provide the continuity among assessments and those that come on board for a particular assessment because of their expertise regarding certain aspects of the model undergoing assessment. The first category, the "model analyzer", represents a new type of researcher, "a highly skilled professional and astute practitioner of the art and science of third party model analysis" (3). Ideally he will be experienced both as a model builder and a model user but occupy a "middle position" while involved in the assessment process. The assessment group's reputation for fairness and objectivity depends heavily upon adherence to this condition. The model assessment process must provide for frequent interactions between the assessment and modeler groups. Modelers may feel uncomfortable about discussing model deficiencies, if they perceive the assessors as competitors or potential users.

Energy models employ the analytic methods of a variety of disciplines. A single modeling system may incorporate the techniques of several disciplines. If the laboratory is to maintain a capability for assessing a wide range of energy models, it must also have a strong resource base of researchers that can be drawn upon for specific assessments. Included in this group will be experts in mathematical programming, econometrics and related methods of statistical analysis as well as experts in various aspects of the electric utility industry and the energy sector in general. The composition of the assessment team at any given point will depend upon the characteristics of the model undergoing assessment.

Approaches to Assessment

The model assessment lab provides two types of assessments: (a) overview