The audit report was completed within two months. Since it was based on a preliminary version of the model, it was distributed only to the modelers and the model sponsor (EPRI). These procedures clearly demonstrate that the audit approach is sufficiently flexible and can be completed rapidly enough to incorporate its findings effectively in the model development process. 4. Concluding Observations

As a summary of the experience gained from the two assessments just discussed, I would like to outline an approach that seems generally appropriate for energy model assessment. An assessment must, of course, begin with a thorough review of the available literature dealing with the model in question. Next, it would proceed to an overview assessment that would use the technical documentation to carry out detailed analysis of the model's logic and implementation procedures. In most instances, the computer code would be a key element of the documentation used in the analysis. A major output from the overview assessment should be the identification of the model's critical points.

I would suggest that most assessments should next proceed to an independent audit and it is only after completion of the audit that a decision should be made about whether to go on to in-depth assessment. Because in-depth analysis involves direct hands-on operation of the model, it is substantially more time consuming and costly than an audit. An audit can function as a relatively inexpensive screening device and in some instances it will turn out that an in-depth assessment is not really required. And, in those cases where further in-depth analysis is called for, the audit will have sharpened the assessors thinking and will provide a sound basis for more intensive evaluation.

Generally a report would be issued upon completion of each stage of the assessment process (overview, audit, and in-depth). However, the experience to date indicates that an overview report, because it raises more questions than it answers, is best treated as an internal document. It facilitates communication among model builders, assessors, and sponsors but can be misinterpreted by readers not directly involved in the process. An audit report would generally be kept as an internal working document for a model still in the development phase but could be made publicly available for more mature models. Since indepth assessment would only be justified for models that are being used in policy applications, a report coming out of an in-depth assessment should certainly be publicly available.

As indicated by the dashed arrows in Figure 1 leading from the assessment reports back to the model itself, the assessment findings should be a part of the model development process. deed, this kind of feedback leading to improved policy models is likely to be one of the most significant contributions of the assessment activity. To facilitate this, I think it is important

that the modeler be involved in the assessment process from the outset. Also, the assessors have an obligation not merely to criticize the model but to suggest ways in which it can be improved. At the same time, the assessors should not themselves implement new "improved" versions of the models. By doing so, they would become competitive model builders and would not be able to function as objective model assessors.

Finally, I would like to suggest that, because model development is a continuous process, model assessment should not be terminated with the completion of an audit or in-depth report. When a model is improved, perhaps in response to the initial assessment, an updating of the assessment should be undertaken. This update would probably take the form of an audit and, because of the information base and expertise established during the initial assessment, the incremental cost would generally be very low. The credibility and reliability of energy policy models could, in my opinion, be significantly enhanced by thus including rigorous, objective assessment as a standard element of the modeling process.

REFLECTIONS ON THE MODEL ASSESSMENT PROCESS:

A MODELER'S PERSPECTIVE

Martin L. Baughman

Center for Energy Studies
The University of Texas at Austin

At the time I was approached about making the Regionalized Electricity Model (REM) available for assessment, we agreed that such an activity was desirable--indeed essential--for the advancement of the energy modeling profession. At the same time, I felt that the Regionalized Electricity Model was developed to a point where scrutiny by a third party would prove beneficial to further development of the model; it would, as well, make the model transparent, and thus useful, to potential users. And though it was not without some trepidation that I offered the model for assessment, I felt at the time that this particular model would be a good trial for the assessment laboratory.

The modelers still disagree in some areas with the MIT Group's presentation of the model; these areas are delineated later in the

Before setting forth these details, however, I would like to comment generally on the issues raised in this first independent assessment. MIT has labeled and listed these as follows:

(i) the extent to which the models being assessed should be
compared to similar models;

(ii) formalization of the relationships among the modelers,
the assessors, and the sponsors;

(iii) approaches to independent assessment; and

(iv) the nature and extent of in-depth independent assessment.

Individual vs. Comparative Model Assessment

What really is the distinction between an individual assessment and a comparative assessment? The distinction between these two modes of assessment is not made very clear in the MIT work. The MIT assessment of the Regionalized Electricity Model states on page 1-10 of the draft report: "In the present case it has not been possible to provide a comparative assessment between the Baughman-Joskow model and potential competitors." Although not explicit, this statement implies that what they attempted was an individual assessment, not a comparative assessment. But here a problem exists. The MIT Group states on page 3-56, in the section entitled Electricity Generation: Model Assessment:

It is worth noting that the generation simulators
used by electric utilities are considerably more
sophisticated than the electricity generation

model in REM. The utility models commonly employ
probabilistic simulation, incorporate many more
types of generating plants, and take into account
seasonal factors. The use of an annual load
duration curve in REM, although a reasonable
simplifying assumption for some purposes, undoubt-
edly restricts the applicability of the REM results.

The passage does not state for what applications the model is restricted as a result of the simplifying assumptions. The passage clearly states that REM cannot be used for some of the purposes of the more detailed utility models. I do not argue with the conclusion at all, but what I don't understand is why the statements exist in an individual model assessment.

Another example of the same point is the following: On page 3-12 of the MIT report, the first sentence of the section on the demand submodel entitled Overview Evaluation states:

The REM demand submodel generally represents the
state-of-the-art in overall energy demand modeling
at the time it was constructed.

This section of the report goes on:

REM does, however, differ in some details from
other efforts. It lacks, for example, the richness
of policy variables and technological specificity
found in other interfuel substitution models.

The report then proceeds to describe capabilities of other interfuel substitution models and contrasts these models with REM. Again, as in the case of the assessment of generation simulation, the MIT Group has used other models as a reference for comparison.

The report then goes on to compare and contrast the financial/regulatory submodel in REM with the Fishbein model. Again, the MIT Group has used the standard of another model to make comparisons and to facilitate the conduct of its assessment.

Now, what is the point of all this? The MIT Group claims they did not do a comparative model assessment. Yet, here are three obvious examples where other models were used as a reference for comparison. I argued from the inception of this assessment activity that a comparative model assessment was the only realistic perspective that could be adopted for assessment of large-scale energy policy models. Since the MIT Group has conducted its assessment, I feel even more strongly about this point. In fact I have concluded that a comparative assessment is really what the MIT Group conducted. Whereas the MIT Group might purport to have used reality as a frame of reference, in fact it