In summary, you must understand in great detail the substance of what is being modeled if you are to be capable of judging whether the model is properly structured, has valid data, is based on reasonable assumptions, and finally gets the right answer.

Step 2: Specify Issues to Be Addressed

The second step would be to understand the issues which the model was designed to address, or the questions the model was intended to answer.

No model is going to be designed to answer every possible question, so this must be specified beforehand. The ICF Coal and Electric Utilities Model (CEUM), for example, is a long-term model with a high level of detail, geographic and otherwise, in the coal and electric utility sector. This means that the model can address such issues as the effect of alternative new source performance standards on the coal and electric utilities industries. It can forecast emissions, it can forecast costs, it can forecast oil consumption, and it can forecast regional coal production. It can do leasing analyses, and it can do some tax and rebate analyses. As it turned out, we can use it to evaluate new technologies such as coal slurry pipelines. It can do basic sensitivity analyses like changes in severance taxes, wage rates, productivity and other coal parameters.

It will not handle short-term issues, or transitional issues such as the expected shutdown of considerable mining capacity in the near future as a result of the soon-to-be-promulgated strip-mining regulations. It won't deal with strikes. It won't deal with bad weather. It won't deal with rail car shortages. This is because it's a long-term model.

Similarly, it won't deal with broader issues. It won't deal with, for example, the trade-off between electricity and gas for home heating, and it won't deal with the effect of reduced U.S. oil imports and world oil prices.

Some of these concerns may be self-evident; but the point is that the model is designed to answer some questions and not others. Before we even start to look at the model structure, we must understand what it is trying to do, what it was designed to do.

Then the third step, still before we examine the model itself, is to understand how the issues the model is designed to analyze might affect the phenomenon being modeled, in this case the coal and electric utility markets.

As an example, if EPA were to make more stringent the new source performance standards for coal-fired power plants, we could expect that emissions on the new plants would be reduced and that the cost of the new plants would be increased. We would expect to see a shift to higher-sulfur coals. We would expect to see a change in the dispatch order. Most models won't allow the dispatch to change in response to the economics. In this particular new source performance standard problem, that is critical.

Stricter new performance standards could, for example, help create a shift to oil; as the costs of operating a coal-fired plant increase with more stringent emission standards, oil plants would become comparatively more economi

We could expect fewer shipments of Western coal to the East. Regional emissions could be expected to decrease generally, although the changes in emissions and cost by region may be very different. Power plant reliability could be expected to change, and that would affect the economics.

These are the kinds of changes one might expect to result fom a change in the new source performance standard. If the model was designed to address the new source performance standard issues, it should be structured to measure such changes.

Step 4: Analyze Structure of Model

Once we have examined these preliminary considerations, the fourth step then is to look at the model, itself and ask whether it is structured to pick up the kinds of dynamics we've been discussing.

For example, one of the first direct impacts we would see under the stricter standards discussed above is that a new plant is going to cost more. Its emissions rate is going to be reduced. Can we enter that in the model as a clean number in the form of an emission rate or a cost? Or must the model be modified in some way, such as the price differential between highand lowsulfur coal up to 40¢ per million Btu? When we do something like that, we're assuming an answer, not doing an analysis.

So, in evaluating the model, the first question is whether these direct effects can be entered as clean numbers, or whether an answer must be assumed.

The second question is whether the model is structured to pick up the dynamics and the trade-offs that we know are going to occur.

One of the dynamics mentioned earlier, economic dispatch, is critical for most coal and electric utility analyses. The model has to be able to dispatch generating capacity by region on an economic basis. That is how it is done in the real world.

Similarly, because we know the policy people are interested in Western coal production and how much coal comes East, we must know whether the model is set up to measure that with any precision. That turns out to fall heavily on regional disaggregation. If we represent the Midwest as a single geographic point, then the model will show that all a category of plants in the Midwest either will go to low-sulfur coal, or will not. In actuality, we know from other analyses that that line is going to shift back and forth between Iowa and Ohio, depending on circumstances. Thus, maybe Illinois will go to Western coal, but Indiana won't. Maybe western Kentucky will go to Western coal, and eastern Kentucky won't.

We need to know whether the model is set up to catch such specifics or whether it must represent that whole area as a single point, say, Indianapolis.

Finally, we need to know whether the model is structured to provide the output that we want. For example, we can measure emissions and costs by region, or regional production by sulfur contents? Are the outputs there or only some hints from which, as Peter House says, the forecast is created on the typewriter? We need to know these specifics.

This, then, is the fourth step: given an understanding of the industries, given an understanding of the questions we are trying to answer, and given an understanding of how these issues affect the industries, then, we need to know whether the model is structured appropriately, on the input side, on the internal structure side, and on the output side.

I think this is the key point in most assessments. If it can't pass this test, one might as well not proceed. Also, if one can't tell from the documentation, then the model similarly gets low marks.

Step 5: Review Critically Data Inputs

The next step after the structure is the data inputs. The first thing we would look for is whether the data are well-documented. By well-documented I don't mean saying that they got the power plant capacity from the FPC. Anybody who understands FPC power plant capacity data knows there is a great deal of noise there.

What the documentation should say is that the power plant capacity came from the FPC and probably 30 other sources. I think that the actual numbers used should be there, because there is so much room for interpretation. is so much noise in this kind of data, that the user has to be able to tell whether the mistakes were caught in the FPC data or whether EIA data mistakes that we know of are there.

And incidentally, I want to distinguish between models and modelers or analysts. There is a difference between them. A model by itself is fairly irrelevant. Part of evaluating the model is making sure that the people involved have taken adequate care of the data. There should be clear statements of the strengths and weaknesses. The modelers ought to furnish their judgments concerning whether the data they used were any good. They should state what data they think are critical and what difference it makes. I worry about people who call themselves modelers, creating beautiful structures, and ignoring the data. Because a good structure will depend on the data.

Step 6: Critically Review Assumptions

After we understand what we are trying to do, first we look at the structure, then the data, and finally the assumptions.

The assumptions should be clearly spelled out. They are usually called cenario specifications these days. Whatever they are called, they should be learly documented. They should be accompanied by a discussion of which ones ake a difference, and what kind of difference they make, and which ones are ost critical, and which ones are least certain.

None of the assumptions should be omitted, and they should all be presented ight up front. They should not be buried in the documentation.

Step 7: Analyze Model Forecasts

Thus far in the process of evaluation, we have gone through six steps ithout even looking at the model itself, without looking at any of the comuter code or any of the forecast measures. We are now ready for the final tep, which is to examine the model. This is not the first step; it is the ast step.

As we begin this step, we should approach the model from the context of an ssue. We don't want to examine the model in strictly abstract terms.

e acknowledge that it was designed to evaluate a given set of questions or ssues, and then we analyze its behavior on that issue.

In the third step we defined what we expected to happen when certain poliy alternatives were played out. Now we want to see whether the forecasts ehaved as we expected. If not, we should be able to explain why not. For xample, one analysis of the new source performance standards involved forecasting what should happen if EPA set up the most stringent standards it could evise. In general, we would have expected this to result in lower emisions. But the opposite turned out to be the case. The explanation was that s the cost of a new plant increased, economic dispatch would cause the existng oil and coal-fired plants to take on higher loads. These plants have very uch higher emission rates than the new coal plants. Therefore, the system ould end up having higher emissions than otherwise, because the increased missions from existing plants more than offset the reduced emissions from new lants.

This was an important finding and one which we had not expected. The ichness of the input data and output data from our model provided us the eans to understand this anomaly. The sound structure of the model resulted n an effect that was not anticipated but now is believed to be valid.

One other point, which relates again to the documentation, is that one hould be able to reproduce any forecast variable, using documentation, other orecast variables, and an understanding of the model structure. I consider his a critical point. This is an important way we can build confidence in a odel. It lets us know whether it's actually working the way it should.

If one can do this--and I and many of our clients and I can do so with our model--there is no need to review the computer code. One can determine whether the code is correct by analyzing the inputs and outputs. For those who are interested in the code--it can be reviewed. But such review is not necessary. I've never looked at the code to our model, and I'm convinced I understand thoroughly how the model operates.

Now, this seven-step process which I have defined is the process we used to build the model, and the one we apply for evaluation as we continue to refine the model.

Expected Findings of an Assessment

I want to summarize what I would expect to be the results of an assessment.

The first thing such an assessment should deal with is the quality of the documentation. This is the basis upon which people are asked to believe the answers. In order for them to be able to go through the process we've just outlined, they must see the documentation as their vehicle.

one is

The second

I should distinguish here between two types of documentation: aimed at the user, and the other at the operator. The first is keyed toward the person who will use the results, possibly to make decisions. is keyed to the operator, who wants to take the model and make it go. I believe the first type is overwhelmingly more important than the second. Indeed, for my own purposes--since I never intend to operate the model--the second is irrelevant. At any rate, the two types are distinct and separate.

Incidentally, there are examples of models used in government whose documentation is extremely inadequate. If we look at such a model and, as part of an assessment, attempt to duplicate a certain number from the model, we might find it impossible. When this happens, it is a failure of the documentation. And it happens frequently. The excuse given is that the modelers are too busy running the model to go back and document it. But the model just isn't very useful unless an outside assessor can determine whether he can believe it.

Secondly, an assessment should show what kinds of questions the model is designed to answer, and the kinds it is not designed to answer, and maybe some that are in between--that is, perhaps some that the model could answer if certain improvement opportunities were taken.

What

The assessment should deal then with the quality of the forecasts. are the strengths and weaknesses of the forecast we get for the questions which the model is designed to answer? Will it measure relative changes? How well? The structure is the key element in measuring relative changes. The data are the key elements for the absolute levels.